Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 1 of 44 Page ID #:33
`Case 8:11-cv-01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page1 of 44 Page ID #233
`
`EXHIBIT 1
`
`EXHIBIT 1
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 2 of 44 Page ID #:34
`Case 8“ “°”'°‘ 862‘”OC'JPR D°°”“"e”t "lllllllllllllllflllllllllllllll|llI1l1|I|l||I1||llll|||W|ll|||f|I|f‘34
`
`US006199060B1
`
`(12) United States Patent
`Gustman
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 6,199,060 B1
`*Mar. 6, 2001
`
`(54) METHOD AND APPARATUS MANAGEMENT
`OF MULTIMEDIA ASSETS
`
`OTHER PUBLICATIONS
`
`(75)
`
`Inventor: Samuel Gustman, Santa Monica, CA
`(US)
`
`.
`(*) Notice:
`
`(73) Assignee: Survivors of THW Shoah Visual
`History Foundation, Los Angeles, CA
`(US)
`.
`.
`.
`This patent issued on a continued pros-
`ecufion application filed under 37 CFR
`1.53(d), and is subject to the twenty year
`patent
`term provisions of 35 U.S.C.
`154(a)(2)'
`Subject to any disclaimer’ the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(21) Appl. No.: 09/076,504
`
`(22)
`
`Filed:
`
`May 12, 1993
`
`Related U_s_ Application Data
`
`(62) Division of application No. 08/678,727, filed on Jul. 10,
`1996; 110W Pat N0 5,813,014
`Int. Cl.7 ...................................................... G06F 17/30
`(51)
`(52) U.s. Cl.
`.......................... 707/3, 707/1, 707/2, 707/4,
`707/5; 707/104; 709/216
`(58) Field of Search .................................. .. 707/104, 1, 2,
`707/3’ 4, 5; 370/338’ 395’ 367’ 452’ 231;
`
`710/74; 712/220
`
`(56)
`
`.
`References Cited
`
`Us. PATENT DOCUMENTS
`
`5,192,999
`5,241,671
`
`................... .. 348/552
`3/1993 Graczyk et al.
`8/1993 Reed et al.
`......................... .. 707/104
`
`IEEE publication, “Database interoperation support in mul-
`timedia applications architecture and methodology” by W.
`A. Gray et al., The Institute of Electrical Engineers, Dept. of
`Computer Science, UK, pp. 2/1-2/6, Jan. 1998.*
`
`(List continued on next page.)
`
`Primary Examiner—Thomas G. Black
`Assistant Exam;-ne,,_Diane D. Mizrahi
`(74) Attorney, Agent, or Firm—The Hecker Law Group
`
`ABSTRACT
`(57)
`The invention is a generalized solution for management of
`Hlléltllnedla assets. Generailized interfaces age used between
`a
`rowser component, 1I1d€X1I1%S(:I'V1€I‘,
`arc ive server, rifiehr-
`tiary storage. manager, an met o p ayer components.
`e
`generalized interface defines a communication protocol that
`can be used by any browser, indexing server, archive server,
`tertiary storage manager, or method player component.
`Thus, system components supplied by multiple vendors can
`be interconnected to form a multimedia system that com-
`municates using the generalized interfaces of the invention.
`A browser component can accept
`input such as search
`requests and display multimedia data. An indexing server
`component includes a data catalogue that catalogues multi-
`media data. The catalogue can be used to identify catalogue
`elements associated with data that can satisfy a search
`req11?S‘~(/ffiarchlfecharilismi:“S‘?d‘° identify ‘hZ§.e.Sire‘i
`mutlme la
`am"
`Searc mec “Sm can Create a H10“
`fiéiiiueiéfvaéraiina§$iZ‘3e°§§i‘Zf?E§§§EZ§Z°$§§§“$07
`g
`'
`y
`age manager that manages the store containing the multi-
`
`in a display region of the
`method play which plays it
`browser. The invention retains the content and results of a
`.
`.
`search such that it is only necessary to perform a search
`once. Search elements are used to store the content of a
`search, i.e., search criteria. A segment element is used to
`Store the results of a Search.
`
`(List continued on next page.)
`
`18 Claims, 24 Drawing Sheets
`
`
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 3 of 44 Page ID #:35
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 3 of 44 Page ID #235
`
`US 6,199,060 B1
`Page 2
`
`5,649,186
`5,754,784 *
`5,774,698 *
`
`7/1997 Ferguson .............................. .. 707/10
`5/1998 Garland ......................... .. 395/200.49
`6/1998 Olnowich ........................... .. 395/500
`
`OTHER PUBLICATIONS
`“
`.
`.
`.
`.
`.
`IEEE pub11cat1en, Content based navrgatron 1n a epen
`hYP‘°~4m‘?d1a ‘°~41V}4°41441‘°~414 by RH Lewls 64 a1~>Th‘°~ 14154141149
`Of Electlcal Englneers, Department Of Electronics and Com-
`puter Science, UK pp. 5/1-5/6, Jan. 1995.*
`Musei Vaticani, Internet,
`(http://WWW.Virtua1.c0.i1/0rgs/ar-
`chival/bookjntljhtfn),
`1_10, Dec. 8,
`The Judaica Archival project, Internet, (http;//WWW.Virtua1_
`.co.i1/orgs/archiva/bookintrhtrn), pp 1-11, Apr. 1996.
`* cited by examiner
`
`US. PATENT DOCUMENTS
`
`2/1994 Enberg eta1~ ~~~~~~~~~~~~~~~~~~~~~~~ ~ 348/14
`2/1994 Glick et al.
`379/93
`3/1994 Bernstein et al.
`384/14
`4/1994 MacKa
`.......... ..
`345 328
`3/1995 Robisoz et al.
`381%”
`4/1995 Fujisawa et al.
`........................ 707/4
`6/1995 Baker ................................. .. 395/154
`7/1995 Dinwiddie, Jr. et al
`345/133
`7/1995 Bowen etal.
`...... ..
`370/352
`9/1995 Bergen et al.
`. . . . .
`. . . . . .. 707/9
`6/1996 Meske, Jr. et al.
`................ .. 709/206
`g:’a°£‘etI‘:f}‘;’£tir‘:11ér';';;i:""""
`7/1997 Antognini et al.
`..................... .. 707/9
`
`
`
`5,283,638
`5,283,819
`5,297,249
`5,307,456
`5,402,499
`5,404,506
`5,428,730 *
`5,434,592
`5,436,898
`5,450,581
`5,530,852
`
`5,649,185
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 4 of 44 Page ID #:36
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 4 of 44 Page ID #'36
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 1 of 24
`
`US 6,199,060 B1
`
`106
`
`104
`
`
`
`
`Tertiary
`Storage
`
`
`Manager
`
`
`
`102
`
`Method
`
`Player
`
`Archive
`Server
`
`110
`
`Index
`Server
`
`108
`
`Client and
`Method
`
`Player
`
`Hewlett Packard
`Hard ware Platform
`
`
`lntermedia
`
`Servher
`Ind
`W It
`Serve; H Archive Server
`
`and
`
`118
`
`
`
`Tertiary Storage
`Manager
`
`116
`
`4
`. . . . ¢ ; . » r — a . o o — . . . . . , ¢ . . . - - : » . o o « a a . » — . . ¢ — » . . . . . - - c ¢ o » « » « ¢ a ¢ o a a . o . » — o . a — — . tr
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 5 of 44 Page ID #:37
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 5 of 44 Page ID #237
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 2 of 24
`
`US 6,199,060 B1
`
`124
`
`
`
`
`Tertiary
`T :;::::§:,
`
`132
`
`126
`
`
`
`138
`
`Media
`
`130
`
`Index
`Server
`
`128
`
`§
`
`§Mainframe
`
`144
`
`150CV 150B
`
`Index
`Server
`
`§ 148
`2
`
`Server
`
`Archxve
`
`o . . . . . . . . . . — . . . . . . . . . . . . . « — . . . . . . . . . . . . . . . ¢ . ¢ , — - . . « . ¢ . . ¢ ¢ « . ¢ . . ¢ . - - ¢ . - uq
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 6 of 44 Page ID #:38
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 6 of 44 Page ID #238
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 3 0f 24
`
`US 6,199,060 B1
`
`. ; ; . . . . . . . . . . . . . . » » . . . . . , . . . . . . . , . . . . . . . . . . . . . . . . . . 4 . . . . . . . . . . , . . ..
`
`Method
`Player
`
` Tertiary
`
`' . . . . . . . . . . r ' . . . . . . . . . . . . . — . . . . . . . . . . . . . . « r p . . . y . . r . . . . r , . . . . . . . . . ..
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 7 of 44 Page ID #:39
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 7 of 44 Page ID #239
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 4 of 24
`
`US 6,199,060 B1
`
`219
`
`216
`
`I/O
`
`VIDEO AMP
`
`CRT
`
`213
`
`214
`
`CPU
`
`VIDEO MEMORY
`
`MAIN MEMORY
`
`210
`
`KEYBOARD
`
`MOUSE
`
`2“
`
`MASS STORAGE
`
`218
`
`217
`
`215
`
`212
`
`FIG. 2
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 8 of 44 Page ID #:40
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 8 of 44 Page ID #240
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 5 0f 24
`
`US 6,199,060 B1
`
`Browser kin.» Method
`
`314 y f 312
`
`316
`
`Indexing
`Server
`
`FIG. 3
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 9 of 44 Page ID #:41
`Filed 12/02/11
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Page 9 of 44 Page ID #:41
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 6 of 24
`
`US 6,199,060 B1
`
`Ehzase
`input data ID
`ID
`descriptive phrase
`In/Out Timeoodes
`setflceywords)
`set(images)
`senpersons)
`set(proposed keywords)
`
`Type
`ID
`label
`
`lhgggural Keyword
`I D
`keyword ID
`label
`Ianguuge of Choice
`Preferred Flag
`Characteristic
`
`
`
`
`Segment
`ID
`descriptive phrase
`sefiphrases)
`
`Keyword
`ID
`setflypes)
`set(thesa ural keywords)
`se!(parent keywords)
`sef(chiId keywords)
`se£(proposed keywords)
`
`lhgggural Egrgon
`ID
`Name
`
`set(proposed persons)
`Preferred Flag
`
`Exam:
`ID
`'1/P8
`sub-type
`In/Out Timecodes
`comment
`
`Person
`ID
`Primary Name
`Occupation
`Date of Birth
`set(proposed persons)
`
`424
`
`
` Proposed Person
`ID
`Name
`
`
`
`Image I Video
`ID
`In/Out Timewdes
`
`Loaztor
`
`Person ID
`Person Attribute
`
`FIG. 4A
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 10 of 44 Page ID #:42
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 10 of 44 Page ID #242
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 7 0f 24
`
`US 6,199,060 B1
`
` 41
`
`Proposed
`Keyword
`
`438
`
`
`
`412
`
`Thesaural
`
`Keyword
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 11 of 44 Page ID #:43
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 11 of 44 Page ID #243
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 8 0f 24
`
`US 6,199,060 B1
`
`486 \
`
`Thesaural
`Keyword One
`
`Thesaural
`Keyword Two
`
`Thesaural
`Keyword Three
`
`Thesaural
`Keyword Four
`
`4921:923
`
`4933
`493*‘
`
`496C
`4968
`
`496A
`
`4973
`497A
`
`483
`\
`
`KeYW°’d
`Instance
`one
`
`4900
`490C
`4908
`490A
`
`49115
`
`4918
`
`491A
`
`491D
`
`
`
`4948
`494A
`
`4953
`495A
`
`Keyword
`Instance
`Two
`
`FIG. 4C
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 12 of 44 Page ID #:44
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 12 of 44 Page ID #244
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 9 0f 24
`
`US 6,199,060 B1
`
`318
`
`Browser
`
`516
`
`\ Keyword
`
`314
`
`/
`
`518
`
`522
`
`\ Keyword Hierarchy
`
`520
`
`\ Person
`
`\ Thesaural Keyword
`
`524
`
`\ Type
`
`526
`
`\ Type Hierarchy
`
`Server
`
`Indexing
`
`FIG. 5
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 13 of 44 Page ID #:45
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 13 of 44 Page ID #245
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 10 of 24
`
`US 6,199,060 B1
`
`0
`
`602
`
`No_.@
`
`Yes
`
`606
`
`perform search
`
`Yes
`transmit command
`
`
`play command?
`
`FIG. 6
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 14 of 44 Page ID #:46
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 14 of 44 Page ID #246
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 11 of 24
`
`US 6,199,060 B1
`
`N0+@
`
`No->®
`
`
`
`search already
`exists?
`
`Yes
`
`
`
`segment
`associated with
`
`702
`
`704
`
`search?
`
`
`
`Yes
`
`retrieve query
`segment
`
`
`
`706
`
`708
`
`
`
`retrieve phrase ids
`associated with
`
`query segment
`
`
`
`
`
`
`
`FIG. 7A
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 15 of 44 Page ID #:47
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 15 of 44 Page ID #247
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 12 of 24
`
`US 6,199,060 B1
`
`722
`
`728
`
`
`
`background
`information
`
`
`
`search type
`and / or keyword
`
`search background
`
`person
`
`
`
`724
`
`h
`Seam person
`
`information
`
`
`
`
`
`
`phrase id(s)
`identified?
`
`N0-*©
`
`YES
`
`732
`
`create segment
`instance
`
`734
`
`create query
`elements
`
`FIG. 7B
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 16 of 44 Page ID #:48
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 16 of 44 Page ID #248
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 13 of 24
`
`US 6,199,060 B1
`
`802
`
`all search elements
`
`processed?
`
`Yes
`
`find all subelements
`of elements in set
`
`818
`
`add subelements
`to element set
`
`
`
`No
`
`
`
`NO
`
`get next element
`
`806
`
`search element
`attribute for
`search element
`
`816
`
`
`
`
`
`Yes ‘
`
`element in
`set?
`
`Yes
`
`N
`
`0
`
`8
`
`14
`
`add element to
`element set
`
`
`
`
`
`
`
`FIG. 8A
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 17 of 44 Page ID #:49
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 17 of 44 Page ID #249
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 14 of 24
`
`US 6,199,060 B1
`
`
`
`822
` search element
`all elements in
`Yes
`set processed?
`hierarchy?
`
`
`
` 826
`
`N0
`
`824
`
`set next element as
`current Search
`element
`
`Search element
`
`
`
`
`
`find all child elements for
`elements in
`found element set
`
`
`
`
`
`add child elements to
`found element set
`
`
`
`
`
`FIG. 8B
`
`
`
`attribute using
`search element
`
`828
`
`
`
`
`element
`instance
`found .7
`
`
`
`830
` element
`instance found
`in set?
`
`
`
`
`832
`
`add element
`
`instance to found
`
`element set
`
`
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 18 of 44 Page ID #:50
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 18 of 44 Page ID #250
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 15 of 24
`
`US 6,199,060 B1
`
`850
`
`860
`
`end
`
`Yes
`
`all keywords in
`set processed?
`
`
`
`N ‘’
`
`852
`
`set next keyword as
`current keyword
`
`
`
`
`search phrase
`elements for
`ord association
`
`ke
`
`8
`
`54
`
`856
`
`phrase instance
`found not in phrase
`set?
`
`
`No—>{CD
`
`
`
`Yes
`
`8
`
`58
`
`add phrase instance
`to phrase set
`
`
`
`FIG. 8C
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 19 of 44 Page ID #:51
`Filed 12/02/11
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Page 19 of 44 Page ID #:51
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 16 of 24
`
`US 6,199,060 B1
`
`902
`
`
`
`
`search background information
`using selection criteria to
`identify person instances
`
`904
`
`
`select segment
`instances for each
`
`erson instance 906
`
`
`
`
`
`create set of phrases
`associated with
`
`selected segments
`
`
`
`
`
`all phrases in
`set processed?
`
`
`
`
`No
`
`910
`
`918
`
`Yes
`
`end
`
`FIG. 9
`
`
`N0
`
`912
`
`get next phrase
`
`914
`
`
`
`
`search phrase's
`keywords for
`s ecified content
`
`
` 916
`
`
`create set of phrases
`having specified
`content
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 20 of 44 Page ID #:52
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 20 of 44 Page ID #252
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 17 of 24
`
`US 6,199,060 B1
`
`1002
`
`
`
`identify segment
`instance(s) associated
`with person instance(s)
`
`1004
`
`
`
` identify phrase
`
`instance(s) associated
`
`with segment insance(s) 1006
`
`create set of phrases
`associated with
`
`
`
`
`
`
`selected segments
`
`content search?
`
`No~§®
`
`
`
`Yes
`
`1010
`
`all phrases in set
`processed?
`
`
`Yes->®
`
`No
`
`PIG. 10A
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 21 of 44 Page ID #:53
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 21 of 44 Page ID #253
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 18 of 24
`
`US 6,199,060 B1
`
`
`
`get next phrase
`
`1012
`
`1014
`
`
`
`
`
`
`
`search phrase's
`keywords for
`s ecified content
`
` 1016
`
`create set of phrases
`having specified
`content
`
`FIG. 10B
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 22 of 44 Page ID #:54
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 22 of 44 Page ID #254
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 19 of 24
`
`US 6,199,060 B1
`
`416
`
`FIG. 11A
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 23 of 44 Page ID #:55
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 23 of 44 Page ID #255
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 20 of 24
`
`US 6,199,060 B1
`
`Qua
`ID
`
`1102
`
`Qugg. Element Record
`
`Query ID
`Query Element ID
`ID
`AndOr
`Element Order
`
`Query Element ID
`
`Q2eraL£Len;mt
`
`1104
`
`FIG. 11B
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 24 of 44 Page ID #:56
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 24 of 44 Page ID #256
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 21 of 24
`
`US 6,199,060 B1
`
`1208
`
`Ehtase
`
`Bluzase
`
`
`
`1212
`
`
`
`descriptive phrase ‘
`
`
`Storage
`input data ID
`
`input data path
`
`ID
`ID
`
`
` In/Out Timecodes In/Out Timecodes
`
`set(ke3/wards)
`
`Key I-‘rume
`set(images)
`
`
`set(persons)
`
`set(proposed keywords)
` 1210
`set(proposad persons)
`
`Cache 1
`
`Cache N .‘.
`
` 1214
`
`
`
` Cache
`Manager
`
`FIG. 12
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 25 of 44 Page ID #:57
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 25 of 44 Page ID #257
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 22 of 24
`
`US 6,199,060 B1
`
`
`
`
`
`
`retrieve
`
`multimedia data
`
`start thread
`
`1300
`
`1302
`
`1304
`
`transfer data
`
`to thread
`
`
`
`
`
`1306
`
`
`
`thread prepares and
`transfers data to browser
`
`
`No_.@
`
`1
`
`
`
`end of data?
`
`Yes
`
`1310
`
`terminate thread
`
`
`
`FIG. 13A
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 26 of 44 Page ID #:58
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 26 of 44 Page ID #258
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 23 of 24
`
`US 6,199,060 B1
`
`No
`
`
`
`
`
`
`play command?
`
`stop command?
`
`
`
`process play
`command
`
`FIG. 13B
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 27 of 44 Page ID #:59
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Filed 12/02/11 Page 27 of 44 Page ID #259
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 24 of 24
`
`US 6,199,060 B1
`
`1402
`
`ses
`
`U
`
`v
`Browser
`()b]ect
`
`1404
`
`Ngitsrd
`(jkjexi
`
`.
`
`1418
`
`1416
`
`1414
`
`595 -—'
`
`Object
`
`F”
`Object
`
`Uses
`
`\/
`Phrase
`Sequence
`
`1412
`
`Uses
`
`Uses
`
`Contains
`
`rase
`
`Ph
`Object
`
`M10
`
`FIG. 14
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 28 of 44 Page ID #:60
`Filed 12/02/11
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Page 28 of 44 Page ID #260
`
`US 6,199,060 B1
`
`1
`METHOD AND APPARATUS MANAGEMENT
`OF MULTIMEDIA ASSETS
`
`The invention relates to a system of managing multime-
`dia assets. This is a divisional of application Ser. No.
`08/678,727, filed Jul. 10, 1996 and now U.S Pat. No.
`5/813,014 filed Sep. 22, 1998.
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`
`2. Background
`Increasingly, computer systems are being used to present
`multimedia material. Such material is usually in the form of
`text, graphics, video, animation, and sound. Two or more of
`these data types are usually combined to form the multime-
`dia data presented by the computer system. A computer
`system that is used to present multimedia material is called
`a multimedia system.
`A multimedia system should include components that
`allow input,
`information retrieval, and display. FIG. 1A
`provides an illustration of components for a multimedia
`system. Client 110 includes an interface capability to accept
`user input. The user input can specify criteria for identifying
`multimedia data. Method player 102 converts the media data
`from its storage version to a playback version for use by
`client 110.
`
`Index server 108, archive server 106, and tertiary storage
`104 provide storage and retrieval capabilities. Index server
`108 is used to store and retrieve text data. Information
`entered in the user interface can be used to retrieve textual
`
`information stored by index server 108. The user input might
`identify a video segment using a title that is associated with
`the video segment using index server 108, for example. The
`multimedia data is stored on a physical storage device. A
`retrieval mechanism is provided to retrieve data from a
`physical storage device. Archive server 106 locates the
`multimedia data wherever it is stored and interfaces with
`
`tertiary storage manager 105 to access the media data.
`Current multimedia systems attempt to provide some or
`all of the components described in FIG. 1A. However, the
`components provided by these systems are merged to form
`a single component thereby making it impossible to separate
`the merged components into the discrete components
`described in FIG. 1A. This architecture is disadvantageous
`for at least two reasons: 1) there is no ability to replace a less
`capable component with another, more capable component;
`2) it forces each system to run on a single hardware platform.
`Each system provides vendor-specific implementations in
`which at least one of the vendor-specific components used in
`these systems renders the system less viable as a solution for
`multimedia applications. Since, the services are merged into
`a single server, there is no ability to remove a less viable
`component and substitute a more capable component in its
`place. For example, a vendor may offer a multimedia system
`that includes its premiere DBMS product as an index server
`along with a less capable archive server or client. All of the
`components are intertwined such that it is impossible to
`substitute another archive server or client for the existing
`archive server and client. The following provides a brief
`summary of some existing multimedia systems.
`FIG. 1B illustrates one multimedia system having an
`intermedia server, client and index server combined as a
`single component that runs on one, monolithic hardware
`platform. The system is not portable to another hardware
`platform. It runs on a single hardware platform (i.e., plat-
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`
`form 112, a Hewlett Packard platform). The system includes
`intermedia server 118, index server 116, and client 114.
`Client 114 is used to make a multimedia request to inter-
`media server 118. Intermedia server 118 queries index server
`116 for information to locate the requested data. Intermedia
`server 118 uses the data returned from index server 116 to
`request the data from its archive server and tertiary storage
`management subsystems.
`The archive server and tertiary storage management sub-
`systems are integrated in Intermedia server 118. Client and
`method player 114 and index server 116 are interconnected
`with intermedia server 118 to form a single component. The
`interconnections are hardwired such that it is impossible to
`replace one of the existing components. Another aspect of
`the design of this system results in the inability to substitute
`another component. Client 114 has a built-in method player
`functionality. Thus, the system expects the client component
`to include a method player. Yet, most client components that
`are offered by vendors do not include a method player.
`Therefore, it is impossible to use a different client or method
`player component with intermedia server 118.
`A multimedia system having a hub is illustrated in FIG.
`1C. Such a system is provided by Oracle. The hub is
`provided by media server 130. The system runs on a specific
`hardware platform (hardware platform 138, an N-Cube
`hardware platform) and is not portable to other platforms.
`Media server 130 acts as a hub that uses a proprietary
`interface to communicate with the other services. Compo-
`nents with which media server 130 can communicate are
`
`method player 122, tertiary storage manager 124, and client
`120 (via lines 134, 132, and 136, respectively). These
`components are merged into a single component thus mak-
`ing it impossible to replace a component. The component
`formed by media server 130, method player 122, tertiary
`storage manager 124, index server 128, archive server 126
`and client 120 must run on a single hardware platform,
`platform 138. Further, while index server 128 is a powerful
`database management system, client 120, archive server
`126, method player 122 and tertiary storage 124 offer weak
`solutions.
`
`tertiary storage
`A system having an index server,
`manager, and archive server is illustrated in FIG. 1D. Such
`a system is provided by IBM. The system offers index server
`148 (e.g., IBM’s DB2 or Oracle’s DBMS), archive server
`146, and tertiary storage manager 144 in an integrated
`system that runs on mainframe 140. Mainframe 140 is an
`IBM SP2 hardware platform comprised of two RS6000
`machines. Mainframe 140 is comprised of processing units
`144 in which components (e.g., index server 148) can run.
`Tertiary storage 144 is connected to one of processing units
`144. A request for multimedia content from a client
`is
`processed by index server 148. Index server 148 provides
`selection information that identifies the content which is
`
`retrieved from tertiary storage 144. The system is built to run
`in a mainframe environment using IBM hardware. Further,
`the system does not include a client or method player. Index
`server 148, tertiary storage manager 144 and archive server
`146 are combined as a single component such that it is
`impossible to replace one or more of them.
`A system having a kernel and data blades that interface
`with the kernel is illustrated in FIG. 1E. Such a system is
`provided by Informix. As illustrated below,
`the system
`includes kernel 170 that acts as a hub. Kernel 170 is
`
`supplemented by a plurality of data blades 160A—160H.
`Data blades 160A—160H communicate with the kernel using
`a proprietary interface. A data blade is a complex data type
`that is defined using a programming language such as “C”.
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 29 of 44 Page ID #:61
`Filed 12/02/11
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Page 29 of 44 Page ID #:61
`
`US 6,199,060 B1
`
`3
`
`Each data blade (e.g., 160A—160H) is attached to kernel 170
`and integrated into data dictionary 172. Data blades
`160A—160H can then be used over a file system. Kernel 170
`includes a index. Thus, another component must communi-
`cate with the index via a proprietary interface provided by a
`data blade (e.g., data blades 160A—160H). Data blades
`160H, 160B, and 160D provide a proprietary interface to
`method player 162,
`tertiary storage manager 164, and
`archive server 166, respectively. The components provided
`by this system are merged to form a single component that
`use a proprietary interface to communicate. The component
`combination runs in a single hardware platform 174. Data
`dictionary 172 can become large and cumbersome.
`In
`addition, a fault that occurs in one data blade that is included
`in data dictionary 172 causes a fault for the entire system.
`This system construction is not fault tolerant and is unac-
`ceptable for a production environment.
`A pre-packaged system having interconnected system
`components with hardwired, proprietary interconnections is
`illustrated in FIG. 1F. Such a system is provided by Cine-
`base. System 178 includes a component formed by method
`player 182, tertiary storage manager 184, archive server 186,
`index server 188, and client 190. There is no clear delinea-
`tion between components. Further,
`there are no clearly
`defined lines of communication between the components.
`Component interconnections are hardwired, and it is there-
`fore impossible to substitute components that can commu-
`nicate using the existing connections for the existing com-
`ponents. There is no ability to split the component into
`discrete components such that replacement component can
`be substituted for one of the existing components. Further,
`it
`is impossible to split
`the combined component
`into
`separate components that can run on multiple hardware
`platforms. The combined component offers a weak solution.
`The archive server is fast and efficient. Several
`tertiary
`storage managers are also provided. The method players are
`capable. However, the client and index server components
`provide a weak solution.
`None of these systems illustrated in FIGS. 1B—1F provide
`a general cataloguing capability that can catalogue any type
`of multimedia data. In addition, none of the systems pro-
`vided a viable multimedia system solution. That is, none of
`the systems provide viable options for each of the multime-
`dia components identified in FIG. 1A. All of the systems
`merge the components identified in FIG. 1A into a single,
`component that makes it impossible to replace one of the
`components. Further, by combining the components into a
`single component, each system must run on a single hard-
`ware platform. Further,
`there is no vendor-independent
`interface available to integrate components from different
`vendors to construct a optimum multimedia system.
`A number of prior art patents that describe software
`and/or hardware systems are provided below. These systems
`do not provide a general cataloguing capability or interfaces
`to interconnect the multimedia system components identi-
`fied in FIG. 1A.
`
`A computer system having a television circuit that is
`capable of receiving television signals and forwarding these
`signals to the computer system’s audio and video sub-
`systems is described in U.S. Pat. Nos. 5,192,999 (Graczyk et
`al.,
`issued on Mar. 9, 1993), 5,283,638 (Engberg et al.,
`issued Feb. 1, 1994), and 5,283,819(Glick et al., issued Feb.
`1, 1994). Video signals are received for display on the
`computer system’s monitor. Audio signals are output using
`the computer system’s speaker(s). A remote control device
`is used to control the television circuit within the computer
`system. Acomputer system having the television circuit can
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`be used in combination with a data/fax/voice modem circuit
`for telephonic transmission of television signals for telecon-
`ferencing purposes.
`A service for managing hypertext/hypermedia links is
`described in U.S. Pat. No. 5,297,249, Bernstein et al., issued
`on Mar. 22, 1994. A Link Manager Services (LMS) com-
`ponent provides a universal End User Interface (EUI)
`including menus and dialog boxes that can be called by an
`application program when a link, or marker, is selected by an
`application user. The LMS accesses a database that contains
`information about markers and their associated “presenters”
`(e.g., application or program). In addition, the LMS includes
`a viewer capability for viewing and maintaining existing
`links and creating new links. The LMS is used at runtime to
`generate menus and dialog boxes and manage the hypertext/
`hypermedia links.
`A system that interconnects audio-video equipment such
`as video tape record or video production switcher using one
`or more “AV LANs” in U.S. Pat. No. 5,307,456, Mackay,
`issued on Apr. 26, 1994. Generic device commands are
`communicated over an AV LAN to a device translator. The
`
`device translator translates the generic command into a
`device-specific command. The AV LANs are interconnected
`using bridges. The AV LANs can further be used to inter-
`connect workstations, minicomputers, mainframes and per-
`sonal computers: The workstations can be used to display
`information about the resources attached to an AV LAN.
`
`Acontroller apparatus for use with a computer system that
`provides programmed volume control of multiple analog
`input signals such as CD-ROM, telephony, MIDI, PCM,
`FM, and microphone in U.S. Pat. No. 5,402,499, Robison et
`al., issued on Mar. 28, 1995. Additionally, the controller can
`sum multiple inputs and produce multiple audio outputs. The
`controller provides telephony support such that it receives
`voice, DTMF and caller ID signals and sends telephonic
`audio signals.
`A multimedia system that provides a multimedia device
`control program (MMDCP) that interfaces with a multime-
`dia application program (MMAP) and a multimedia presen-
`tation manager (MMPM) in OS/2 in U.S. Pat. No. 5,428,
`730, Baker et al., issued on Jun. 27, 1995. The MMDCP
`provides a standardized user interface. Because many of the
`control functions are performed by the MMDCP, a MMAP
`can be simplified. The MMDCP creates an instance of a
`control panel for use with a multimedia device. A control
`panel instance includes UI widgets (e.g., buttons) that allow
`a user to control the associated device via the control panel
`instance and the device driver.
`
`An expansion unit to implement multimedia capabilities
`on an existing computer system (e.g., IBM PS/2) in U.S. Pat.
`No. 5,434,592, Dinwiddie, Jr. et al., issued on Jul. 18, 1995.
`The expansion unit is connected to the computer system via
`an expansion slot and to the computer system’s monitor. The
`expansion slot controls the video presentation displayed on
`the monitor.
`
`Amultimedia network bus (MMNB), multimedia network
`controller (MMNC) and protocol for transmitting isochro-
`nous and packet data between a backbone or wide area
`network and multiple local area network configurations in
`U.S. Pat. No. 5,436,898, Bowen et al., issued on Jul. 25,
`1995. The MMNC allocates isochronous channels and
`
`packet channels to users over the MMNB taking into
`account the time delay requirements of various data types
`(e.g., voice, moving images, graphics, or text). The MMNB
`and MMNC is for use with multimedia applications that
`require a varying mix of isochronous data and packet data
`transfers.
`
`

`
`Case 8:11-cv-01862-DOC-JPR Document 1-1 Filed 12/02/11 Page 30 of 44 Page ID #:62
`Filed 12/02/11
`Case 8:11-cv—01862-DOC-JPR Document 1-1
`Page 30 of 44 Page ID #262
`
`US 6,199,060 B1
`
`5
`SUMMARY OF THE INVENTION
`
`The invention is a generalized solution for management of
`multimedia assets. Generalized interfaces are used between
`a browser component, indexing server, archive server, ter-
`tiary storage manager, and method player components. The
`generalized interface defines a communication protocol that
`can be used by any browser, indexing server, archive server,
`tertiary storage manager, or method player component.
`Thus, system components supplied by multiple vendors can
`be interconnected to form a multimedia system that com-
`municates using the generalized interfaces of the invention.
`The browser component can accept input such as search
`requests and display multimedia data. Search requ