Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 1 of 44 Page ID #:46
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 1 of 44 Page ID #246
`
`EXHIBIT 1
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 2 of 44 Page ID #:47
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 2 of 44 Page ID #247
`lllllllllllllllllllllllllll1....allllllllllllllllllllllllllllllllllllllll
`
`USOO6199060B1
`
`(12) United States Patent
`Gustman
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 6,199,060 B1
`*Mar. 6, 2001
`
`(54) METHOD AND APPARATUS MANAGEMENT
`OF MULTIMEDIA ASSETS
`
`(75)
`
`‘
`Inventor: Samuel Gustman, Santa Monica, CA
`(US)
`
`.
`(*) Notlce:
`
`(73) Assignee: Survivors of THW Shoah Visual
`History Foundation, Los Angeles, CA
`(US)
`_
`_
`p
`This patent Issued on a continued pros-
`ccufion “PP1i_Cafi°n filed under 37 CFR
`l.53(d), and IS subject to the twenty year
`patent
`term provisions of 35 U.S.C.
`1'54(a)(2)'
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`USC. 154(b) by 0 days.
`
`(21) Appl. No.: 09/076,504
`
`(22) Filed;
`
`May 12, 1993
`
`Related U,s_ Application Data
`
`(62) Division of application No. 08/678,727, filed on Jul. 10,
`1995,11°‘VP31-N0»5,313,014-
`Int. CI.7 ...................................................... G06F 17/30
`(51)
`(52) U.S. Cl. .......................... 707/3; 707/1; 707/2; 707/4;
`707/5; 707/104; 7091,21 6
`(53) Field of Search .................................... 707/104, 1, 2,
`707/3’ 4’ 5; 370/338’ 395’ 367’ 452’ 231;
`395/500, 2007, 38, 49, 79; 345/352; 709/216;
`710/74_ 712/220
`’
`
`(56)
`
`References Cited
`U5. PATENT DOCUMENTS
`348/5_2
`31993 G
`k M1
`3
`/
`raczy e
`.
`.....................
`S/1993 Reed et al.
`........................... 707/104
`
`*
`
`5 192 999
`,
`,
`5,241,671
`
`OTHER PUBLICATIONS
`
`IEEE publication, “Database interoperation support in mul-
`timcdia applications architecture and methodology” by W.
`A Gray et al., The Institute of Electrical Engineers, Dept. of
`Computer Science, UK, pp. 2./1-2/6, Jan. l998.*
`
`(List continued on next page.)
`
`Primary Examirzer—Thomas G. Black
`Assistant Exam;-ne,_Dianc D’ Mizrahi
`(74) Attorney, Agent, or Firm~—Tl1e Hecker Law Group
`
`ABSTRACT
`(57)
`The invention is a generalized solution for management of
`multimedia assets. Generalized interfaces are used between
`a browser component, indexing server, archive server, ter-
`tiary’ storage manager, and method player components. The
`generalized interface defines a communication protocol that
`can be used by any browser, indexing server, archive server,
`tertiary storage manager, or method player component.
`Thus, system components supplied by multiple vendors can
`be interconnected to form a multimedia system that com-
`municates using the generalized interfaces of the invention.
`A browser component can accept
`input such as search
`requests and display multimedia data. An indexing server
`component includes a data catalogue that catalogues multi—
`media data. The catalogue can be used to identify catalogue
`elements associated with data that can satisfy a search
`“=°1“C5*~ A SWC11 m°°h*mi5m is used 1° idmfy 111° desired
`mummedla dam‘ Aswch m°°ha“‘““ C” “re” “dd‘“°“a1
`catalogues that are a subset of the catalogue provided on the
`;”:°’;1‘2§asec‘f‘°&’1‘af]‘; '5‘ tergirinsfifij
`mgedia datg The tertiar bstora e ma a e
`etlilges the cor—
`.
`‘
`.
`. Y
`1?
`. 1‘ 3 ’ ‘ “CV
`.
`rcspondmg multimedia data ‘from its store and sends it to a
`method play which plays it in a display region of the
`browser. The invention retains the content and results of a
`search such that it is only necessary to perform a search
`once. Search elements are used to store the content of a
`search, re, search criteria. A segment element is used to
`h _
`h_
`f
`'
`_
`Store t e rcsu S O a Search
`
`(List continued on next page.)
`
`18 Claims, 24 Drawing Sheets
`
`
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 3 of 44 Page ID #:48
`Case 2:11—cv—10701—DOC—JPR Document 1-1
`Filed 12/27/11 Page 3 of 44 Page ID #248
`
`US 6,199,060 B1
`Page 2
`
`5,649,186
`5,754,784 *
`5,774,698 *
`
`7/1997 Ferguson ......................... 707/10
`
`. 395/200.49
`5/1998 Garland
`
`6/1998 Olnowich ........................... .. 395/500
`OTHER PUBLICAIIONS
`IEEE publication, “Content based navigation in a open
`,
`.
`”
`.
`,
`hypermedra environment by RH. Lewis eta1.,.TheInst1tute
`0fE16Ct1Cal Engmccrs, D6partD16nt0fE1€Ctr0111CS and C0111-
`puter Science, UK pp. 5/1-5/6, Jan. 1995.*
`Musei Vaticani, Internet, (http2//www.virtual.co.il/orgs/an
`chiva]/bookjntr.htJ:n), pp 1-10, Dec. 8, 1994.
`The Judaica Archival Project, Internet, (http://www.Virtua1—
`£0_fl/Orgs/archiva/bookinmhun)’ pp 1_l1) Apr‘ 1996
`* cited by examiner
`
`US, PATENT DOCUMENTS
`
`2/1994 F/“berg eta‘.
`2/1994 Glick et al.
`3/1994 Bcrnstcin ct al.
`3‘/.1994 Md".Kdy """"""
`/1995 Robrsonetal.
`4/1995 Fujisawa et aI_
`6/1995 Baker ...................
`7/1995 Dinwiddic, Jr. ct al.
`7/1995 Bowen eta].
`.......
`9/1995 Bergen et a1.
`5/1995 Meskey 11 Gt 31-
`3/1997 Dockter et al.
`5/1997 Bradcn—Hardcr ct al.
`7/1997 Antognini et al.
`
`
`
`348/14
`379/93
`334/14
`" 345/328
`. 381/119
`707/4
`.. 395/154
`. 345/133
`. 370/352
`.
`707/9
`09/305
`-
`. 707/102
`. 707/102
`707/9
`
`522932638
`5,283,819
`5,297,249
`gfiggfigg
`,
`_,
`5,404-506
`5,423,730 *
`5,434,592
`5,436,898
`5,450,531
`5,530.-352
`5,608,900
`5,630,121
`5,649,185
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 4 of 44 Page ID #:49
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 4 of 44 Page ID #249
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 1 of 24
`
`US 6,199,060 B1
`
`104
`
`
`
`Tertiary
`
`Storage
`
`
`Manager
`
`
`
`102
`
`Method
`
`Player
`
`110
`
`106
`
`Hewlett Packard
`Hardware Platform
`
`Archive
`Server
`
`Index
`Serv er
`
`108
`
`Client and
`Method
`Player
`
`Index
`Server
`
`‘
`
`’
`
`116
`
`'
`
`118
`
`
`
`
`
`In termedia
`Server
`with
`Archive Server
`and
`
`
`
`Tertiary Storage
`Manager
`
`
`
`4
`, . . . . » . A — a » . — . — . » . . . ; , . o a . r : u 4 « 4 ¢ ; . . p — . . . . — . . . . . — . « « a a « . : 4 4 a . . . . . . . . . . . . . . ..
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 5 of 44 Page ID #:50
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 5 of 44 Page ID #250
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 2 of 24
`
`US 6,199,060 B1
`
`E 120
`
`:
`
`§
`
`Index
`Server
`
`123
`
`Tertiary
`Storage
`
`Mainframe
`144
`
`144
`
`Eertlary
`torage
`
` /(150161
`5 V
`
`15013
`
`146
`
`5
`
`3
`5
`
`3
`
`138
`
`3
`
`E
`;
`
`5
`
`1
`5
`p
`
`
`
`;
`
`5
`
`§
`; 148
`
`1
`3
`140 x;
`:\_
`
`Index
`Server
`
`. . . . . . . . . . . . . . , , . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . r - - - . — ¢ e o . a - ¢ - ¢ ¢ .'
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 6 of 44 Page ID #:51
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 6 of 44 Page ID #251
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 3 of 24
`
`US 6,199,060 B1
`
`,......................................................................................................... -. ,
`I
`5
`'
`162
`I
`u
`I
`p
`I
`.
`I
`I
`.
`p
`I
`-
`I
`I
`u
`'
`r
`I
`r
`'
`r
`I
`.
`I
`v
`:
`'
`o
`I
`u
`2
`'
`p
`'
`I
`I
`'
`:-
`’
`'
`r
`'
`I
`I
`.
`I
`p
`I
`.
`I
`1
`:
`'
`n
`I
`»
`I
`I
`,
`.
`'
`r
`I
`r
`I
`.-
`I
`I
`I
`p
`I
`r
`'
`r
`I
`r
`I
`r
`I
`.
`.
`.
`;
`I
`1
`t
`;
`
`.I
`
`.I
`
`.1
`
`,I
`
`.-
`
`.I
`
`. 5
`
`174
`
`
`
`.
`Archlve
`Server
`
`'~ , . . . . . . . . . V . . — V . . . . . . . . . . . . . . . . . . . . . — — . . . . . . ¢ . . . o . . ¢ . . . o « . . . — . . . . — — . . , . . . . .. . . . . . . . . . . . . . . . . . . . 4 .. . . . . . . . r4.
`
`
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 7 of 44 Page ID #:52
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 7 of 44 Page ID #252
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 4 of 24
`
`US 6,199,060 B1
`
`219
`
`216
`
`1/0
`
`VIDEO AMP
`
`CRT
`
`213
`
`214
`
`217
`
`215
`
`CPU
`
`'
`
`VIDEO MEMORY
`
`MAIN MEMORY
`
`A
`
`210
`
`218
`
`KEYBOARD
`
`211
`
`MOUSE
`
`MASS STORAGE
`
`FIG. 2
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 8 of 44 Page ID #:53
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 8 of 44 Page ID #253
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 5 of 24
`
`US 6,199,060 B1
`
`318
`
`310
`Browser +4.1.’ Method
`
`314 V / 312
`
`16
`
`FIG. 3
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 9 of 44 Page ID #:54
`Case 2:11—cv—10701—DOC—JPR Document 1-1
`Filed 12/27/11 Page 9 of 44 Page ID #254
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 6 of 24
`
`US 6,199,060 B1
`
`1211:1122
`input data ID
`ID
`descriptive pkmse
`In/Out Timemdes
`sefikeyuxmis)
`setfimages)
`seflpersons)
`ser(proposed keywords)
`set(proposed persons)
`
`
`
`label
`
`Segment
`ID
`des/:n'ph've phrase
`seflphmses)
`
`Kgyword
`ID
`semypes)
`szlflhesa um! keywords)
`se!(parenf keywords)
`sef(child keywords)
`
`set(proposed keywords)
`lnmtar
`
`urul
`
`ID
`keyword ID
`label
`Uznguuge of Chcrict
`Prefirrzd Flag
`Characteristfz
`
`Image I Video
`ID
`In/Out Timecodes
`
`rm in
`
`:1
`Person 1-D
`Person Am-ibute
`
`In/out Timeaodes
`comment
`
`Person
`ID
`Prirruzry Name
`Occupation
`Date of Birth
`set(proposed persons)
`
`41 8
`
`
`
`
`424
`
`414
`
`
` Progosed Person
`ID
`Name
`
`Label
`
` 426
`
`FIG. 4A
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 10 of 44 Page ID #:55
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 10 of 44 Page ID #255
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 7 of 24
`
`US 6,199,060 B1
`
`404
`
`
`
`414
`
`410
`
`420
`
`424
`
`3
`
`
`
`
`
`
`
`Proposed
`Keyword
`
`438
`
`Thesa ural
`
`Keyword
`
`446
`
`416
`
`436
`
`6
`
`
`
`PYOPOSEC1
`Person
`
`
`
`
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 11 of 44 Page ID #:56
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 11 of 44 Page ID #256
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 8 of 24
`
`US 6,199,060 B1
`
`486 \
`
`Thesaural
`Keyword One
`
`Thesaural
`Keyword Two
`
`Thesaural
`Keyword Three
`
`Thesaural
`Keyword Four
`
`92:92:;
`4
`
`4933
`493A
`
`496C
`
`4963
`496A
`
`4973
`497A
`
`4911;
`
`
`
`491E910
`4913
`
`491A
`
`488
`\
`
`Keyword
`Instance
`one
`
`490D
`490C
`“OB
`490A
`
`4948
`49 4A
`
`4953
`495A
`
`ord
`Ke
`Yw
`Instance
`Two
`
`FIG. 4C
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 12 of 44 Page ID #:57
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 12 of 44 Page ID #257
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 9 of 24
`
`US 6,199,060 B1
`
`314
`
`_/
`
`318
`
`Browser
`
`502
`
`\
`
`Data
`
`510
`
`\ File
`
`512
`
`514
`
`516
`
`\ Segment
`
`\ Phrase
`
`\ Keyword
`
`518
`
`522
`
`-\ Keyword Hierarchy
`
`520
`
`\ Person
`
`\ Thesaural Keyword
`524
`
`\ Type
`
`526
`
`*
`\ Type Hierarchy
`
`
`
`FIG. 5
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 13 of 44 Page ID #:58
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 13 of 44 Page ID #258
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 10 of 24
`
`US 6,199,060 B1
`
`602
`
`No—>®
`
`Yes
`
`606
`
`
`
`
`
`
`Yes
`
`perform search
`
`FIG. 6
`
`Search
`.
`
`play command?
`
`operauon?
`transmit command
`
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 14 of 44 Page ID #:59
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 14 of 44 Page ID #259
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 11 of 24
`
`US 6,199,060 B1
`
`N0—>®
`
`New
`
`
`
`search already
`exists?
`
`Yes
`
`702
`
`704
`
`
`segment
`associated with
`search?
`
`
`
`
`Yes
`
`retrieve query
`segment
`
`706
`
`708
`
`
`
`retrieve phrase ids
`associated with
`
`
`
`query segment
`
`
`
`
`
`FIG. 7A
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 15 of 44 Page ID #:60
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 15 of 44 Page ID #260
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 12 of 24
`
`US 6,199,060 B1
`
`722
`
`person
`
`background
`information
`
`
`
`Search erson
`P
`
`728
`
`search background
`information
`
`phrase id(s)
`identified?
`
`
`
`
`
`
`FIG. 7B
`
`
`
`Y95
`
`732
`
`create segment
`instance
`
`create query
`elements
`
`734
`
`
`
`
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 16 of 44 Page ID #:61
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 16 of 44 Page ID #261
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 13 of 24
`
`US 6,199,060 B1
`
`802 ,
`
`all search elements
`
`processed?
`
`Yes
`
`find all subelements
`of elements in set
`
`816
`
`
`
`NO
`
`ge max
`
`818
`
`add subelemen ts
`
`
`
`806
`
`Search element
`attribute for
`search element
`
`
`
`No
`
`
`
`Yes ‘
`
`element in
`set?
`
`Yes
`
`No
`
`8
`
`14
`
`add element to
`element set
`
`
`
`
`
`
`
`FIG. 8A
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 17 of 44 Page ID #:62
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 17 of 44 Page ID #262
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 14 of 24
`
`US 6,199,060 B1
`
`Yes
`
`search element
`hierarchy?
`
`find all child elements for
`elements in
`found element set
`
`822
` all elements in
`set processed?
`
`N0
`
`824
`
`set next element as
`current search
`gement
`
`326
`
`
`
`
`
`
`
`
`Search element
`
`attribute using
`Search element
`
`828
`
`add child elements to
`
`found element set
`
`No
`
`FIG. 8B
`
`830
` element
`instance found
`in set?
`
`
`
`
`
`832
`
`add element
`
`instance to found
`
`element set
`
`
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 18 of 44 Page ID #:63
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 18 of 44 Page ID #263
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 15 of 24
`
`US 6,199,060 B1
`
`850
`
`all keywords in
`set processed?
`
`
`
`N“
`
`852
`
`Yes
`
`end
`
`set next keyword as
`current keyword
`
`-
`
`8
`
`54
`
`search phrase
`elements for
`
`keyword association
`
`860
`
`
`
`856
`
`phrase instance
`found not in phrase
`
`set?
`
`
`Yes
`
`No-—>®
`
`85
`
`8
`
`add phrase instance
`to phrase set
`
`
`
`FIG. ac
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 19 of 44 Page ID #:64
`Filed 12/27/11
`Case 2:11—cv—10701—DOC—JPR Document 1-1
`Page 19 of 44 Page ID #264
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 16 of 24
`
`US 6,199,060 B1
`
`
`
`
`select segment
`instances for each
`
`erson instance
` 906
`
`
`
`
`
`create set of phrases
`associated with
`
`
`
`selected segments
`
`
`
`content
`search?
`
`No
`
`902
`
`search background information
`using selection criteria to
`identify person instances
`
`904
`
`910
`
`918
`
`Yes
`
`end
`
`
`
`
`
`
`
`all phrases in
`set processed?
`
`
`
`N0
`
`912
`
`get next phrase
`
`914
`
`
`
`search phrases
`keywords for
`s ecified content
`
`
`
`916
`
`create set of phrases
`having specified
`Content
`
`
`
`
`FIG. 9
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 20 of 44 Page ID #:65
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 20 of 44 Page ID #265
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 17 of 24
`
`US 6,199,060 B1
`
`1002
`
`identify segment
`instance(s) associated
`with person instance(s)
`
`
`
`
`
`
`1004
`
`
`
` identify phrase
`
`instance(s) associated
`with segment insance(s)
`
`associated with
`
`selected segments
`
`
`
`
`
`
` create set of phrases
` content search?
`
`No->®
`
`1010
`
`Yes
`
`
`
`all phrases in set
`
`processed? Yes—>®
`
`No
`
`FIG. 10A
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 21 of 44 Page ID #:66
`Filed 12/27/11
`Case 2:11—cv—10701—DOC—JPR Document 1-1
`Page 21 of 44 Page ID #266
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 18 of 24
`
`US 6,199,060 B1
`
`
` search phrase's
`
`get next phrase
`
`
`
`keywords for
`s ecified content
`
`1012
`
`1014
`
`
`
` 1016
`
`
`create set of phrases
`having specified
`content
`
`PIG. 10B
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 22 of 44 Page ID #:67
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 22 of 44 Page ID #267
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 19 of 24
`
`US 6,199,060 B1
`
`
`
`‘’404
`
`1
`
`FIG. 11A
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 23 of 44 Page ID #:68
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 23 of 44 Page ID #268
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 20 of 24
`
`US 6,199,060 B1
`
`Qu_eru
`ID
`
`1102
`
`Quggg Element Record
`
`Query ID
`Query Element ID
`ID
`AndOt
`Element Order
`
`
`
`
`
`
`1106
`
`1104
`
`Q.!;:xy_EArme_r1t
`
`Query Element ID
`
`PIG. 11B
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 24 of 44 Page ID #:69
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 24 of 44 Page ID #269
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 21 of 24
`
`US 6,199,060 B1
`
`1208
`
`Bhmsa
`
`Blmzsz
`
`Storage
`input data ID
`input data path
`
`ID
`ID
`
`
`
`
`
`
`
`
`
`
`
`
`descriptive phrase .
` In/Out Timecodes
`In/Out Timecodes
`sedkzywords)
`Key Pram:
`
`1212
`setfimages)
`set(persom;)
`
`setqzropased keywords)
`
`se£(praposcd persons)
`
`1210
`
`Cache 1
`
`
`
` Cache
`Manager
` 1214
`
`Cache N
`
`FIG. 12
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 25 of 44 Page ID #:70
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 25 of 44 Page ID #270
`
`U.S. Patent
`
`.. Mar. 6,2001
`
`Sheet 22 of 24
`
`US 6,199,060 B1
`
`
`
`
`'
`
`retrieve
`
`multimedia data
`
` 1300
`
`
`1302
`
`1304
`
`start thread
`
`transfer data
`to thread
`
`1306
`
`
`
`
`1
`
`1310
`
`
`
`
`
`thread prepares and
`transfers data to browser
`
`
`
`end of data?
`
`Yes
`
`terminate thread
`
`
`
`PIG. 13A
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 26 of 44 Page ID #:71
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 26 of 44 Page ID #271
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 23 of 24
`
`US 6,199,060 B1
`
`
`
`play command?
`
`Yes
`
`stop command?
`
`
`
`Yes—@
`
`
`
`
`No
`
`process play
`command
`
`FIG. 13B
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 27 of 44 Page ID #:72
`Case 2:11-cv—10701-DOC-JPR Document 1-1
`Filed 12/27/11 Page 27 of 44 Page ID #272
`
`U.S. Patent
`
`Mar. 6, 2001
`
`Sheet 24 of 24
`
`US 6,199,060 B1
`
`1402
`
`ses
`
`U
`
`-
`Browser
`Cltnect
`
`1404
`
`hgleatygfi
`()kfiect
`
`.
`
`Uses
`
`Uses
`
`1416
`
`\
`
`1418 ‘i‘l|Ii!ii¥EN|||I”
`
`U595 -———-
`
`Player
`object
`
`Phrase
`EC!
`Moaéigager
`
`-
`
`1414
`
`Uses
`
`Uses
`
`Uses
`
`1412
`
`\/
`Phrase
`Segsjzgce
`
`Contains
`
`rase
`
`Ph
`Object
`
`1410
`
`FIG. 14
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 28 of 44 Page ID #:73
`Filed 12/27/11
`Case 2:11—cv—10701—DOC—JPR Document 1-1
`Page 28 of 44 Page ID #273
`
`US 6, 199,
`
`060 B1
`
`.
`1
`METHOD AND APPARATUS MANAGEMENT
`or MULTIMEDIA ASSETS
`
`The invention relates to a system of managing multime-
`dia asscts. This is a divisional of application Ser. No.
`08/678,727, filed Jul. 10, 1996 and now U.S Pat. No.
`5/813,014 filed Sep. 22, 1998.
`
`BACKGROUND OF THE INVENTION
`
`10
`
`1. Field of the Invention
`2. Background
`Increasingly, computer systems are being used to present
`multimedia material. Such material is usually in the form of
`text, graphics, video, animation, and sound. Two or more of 15
`these data types are usually combined to form the multime-
`dia data presented by the computer system. A computer
`system that is used to present multimedia material is called
`a multimedia system.
`A multimedia system should include components that 20
`allow input,
`information retrieval, and display. FIG. 1A
`provides an illustration of components for a multimedia
`system. Client 110 includes an interface capability to accept
`user input. The user input can specify criteria for identifying
`multimedia data. Method player 102 converts the media data
`from its storage version to a playback version for use by
`client 110.
`
`Index server 108, archive server 106, and tertiary storage
`104 provide storage and retrieval capabilities. Index server
`108 is used to store and retrieve text data. Information
`entered in the user interface can be used to retrieve textual
`information stored by index server 108. The user input might
`identify a video segment using a title that is associated with
`the video segment using index server 108, for example. The
`multimedia data is stored on a physical storage device. A
`retrieval mechanism is provided to retrieve data from a
`physical storage device. Archive server 106 locates the
`multimedia data wherever it is stored and interfaces with
`tertiary storage manager 105 to access the media data.
`Current multimedia systems attempt to provide some or
`all of the components described in FIG. 1A. However, the
`components provided by these systems are merged to form
`a single component thereby making it impossible to separate
`the merged components into the discrete components
`described in FIG. 1A. This architecture is disadvantageous
`for at least two reasons: 1) there is no ability to replace a less
`capable component with another, more capable component;
`2) it forces each system to run on a single hardware platform.
`Each system provides vendor—specific implementations in
`which at least one of the vendor-specific components used in
`these systems renders the system less viable as a solution for
`multimedia applications. Since, the services are merged into
`a single server, there is no ability to remove a less viable
`component and substitute a more capable component in its
`place. For example, a vendor may offer a multimedia system
`that includes its premiere DBMS product as an index server
`along with a less capable archive server or client. All of the
`components are intertwined such that it is impossible to
`substitute another archive server or client for the existing
`archive server and client. The following provides a brief
`summary of some existing multimedia systems.
`FIG. IR illustrates one multimedia system having an
`intermedia server, client and index server combined as a
`single component that rtms on one, monolithic hardware
`platform. The system is not portable to another hardware
`platform. It runs on a single hardware platform (i.e., plat-
`
`30
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`form 112, a Hewlett Packard platform). The system includes
`intermedia server 118, index server 116, and client 114.
`Client 114 is used to make a multimedia request to inter-
`media server 118. Intermedia server 118 queries index server
`116 for information to locate the requested data. Intermedia
`server 118 uses the data returned from index server 116 to
`request the data from its archive server and tertiary storage
`management subsystems.
`The archive server and tertiary storage management sub-
`systems are integrated in Intermedia server 118. Client and
`method player 114 and index server 116 are interconnected
`with intermedia server 118 to form a single component. The
`interconnections are hardwired such that it is impossible to
`replace one of the existing components. Another aspect of
`the design of this system results in the inability to substitute
`another component. Client 114 has a built-in method player
`functionality. Thus, the system expects the client component
`to include a method player. Yet, most client components that
`are ollered by vendors do not include a method player.
`Therefore, it is impossible to use a different client or method
`player component with intermedia server 118.
`A multimedia system having a hub is illustrated in FIG.
`1C. Such a system is provided by Oracle. The hub is
`provided by media server 130. The system runs on a specific
`hardware platform (hardware platform 138, an N-Cube
`hardware platform) and is not portable to other platforms.
`Media server 130 acts as a hub that uses a proprietary
`interface to communicate with the other services. Compo-
`nents with which media server 130 can communicate are
`method player 122, tertiary storage manager 124, and client
`120 (via lines 134, 132, and 136, respectively). These
`componenLs are merged into a single component thus mak-
`ing it impossible to replace a component. The component
`formed by media server 130, method player 122, tertiary
`storage manager 124, index server 128, archive server 126
`and client 120 must run on a single hardware platform,
`platform 138. Further, while index server 128 is a powerful
`database management system, client 120, archive server
`126, method player 122 and tertiary storage 124 oller weak
`solutions.
`
`tertiary storage
`A system having an, index server,
`manager, and archive server is illustrated in FIG. 1D. Such
`a system is provided by IBM. The system oll'ers index sewer
`148 (e.g., IBM’s DB2 or Oracle’s DBMS), archive server
`146, and tertiary storage manager 144 in an integrated
`system that runs on mainframe 140. Mainframe 140 is an
`IBM SP2 hardware platform comprised of two RS6000
`machines. Mainframe 140 is comprised of processing units
`144 in which components (eg., index server 148) can run.
`Tertiary storage 144 is connected to one of processing units
`144. A request for multimedia content from a client is
`processed by index server 148. Index server 148 provides
`selection information that identifies the content which is
`retrieved from tertiary storage 144. The system is built to run
`in a mainframe environment using IBM hardware. Further,
`the system does not include a client or method player. Index
`server 148, tertiary storage manager 144 and archive server
`146 are combined as a single component such that it is
`impossible to replace one or more of them.
`A system having a kernel and data blades that interface
`with the kernel is illustrated in FIG. 1E. Such a system is
`provided by lnformix. As illustrated below,
`the system
`includes kernel 170 that acts as a hub. Kernel 170 is
`supplemented by a plurality of data blades 160A—-160H.
`Data blades 160A—160H communicate with the kernel using
`a proprietary interface. A data blade is a complex data type
`that is defined using a programming language such as “C”.
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 29 of 44 Page ID #:74
`Filed 12/27/11
`Case 2:11—cv—10701—DOC—JPR Document 1-1
`Page 29 of 44 Page ID #274
`
`US 6,199,060 B1
`
`3
`Each data blade (e.g., 160A—160H) is attached to kernel 170
`and integrated into data dictionary 172. Data blades
`160A—160H can then be used over a file system. Kernel 170
`includes a index. Thus, another component must communi-
`cate with the index via a proprietary interface provided by a
`data blade (e.g., data blades 160A—-160H). Data blades
`160H, 160B, and 160D provide a proprietary interface to
`method player 162,
`tertiary storage manager 164, and
`archive sewer 166, respectively. The components provided
`by this system are merged to form a single component that
`use a proprietary interface to communicate. The component
`combination runs in a single hardware platform 174. Data
`dictionary 172 can become large and cumbersome.
`In
`addition, a fault that occurs in one data blade that is included
`in data dictionary 172 causes a fault for the entire system.
`This system commotion is not fault tolerant and is unac-
`ceptable for a production environment.
`A pre-packaged system having interconnected system
`components with hardwired, proprietary interconnections is
`illustrated in FIG. 1b‘. Such a system is provided by Cine-
`base. System 178 includes a component formed by method
`player 182., tertiary storage manager 184, archive server 186,
`index server 188, and client 190. There is no clear delinea-
`tion between components. Further,
`there are no clearly
`defined lines of communication between the components.
`Component interconnections are hardwired, and it is there-
`fore impossible to substitute components that can commu-
`nicate using the existing connections for the existing com-
`ponents. There is no ability to split the component into
`discrete components such that replacement component can
`be substituted for one of the existing components. Further,
`it
`is impossible to split
`the combined component
`into
`separate components that can run on multiple hardware
`platforms. The combined component offers a weak solution.
`The archive server is fast and ellicient. Several
`tertiary
`storage managers are also provided. The method players are
`capable. However, the client and index server components
`provide a weak solution.
`None of these systems illustrated in FIGS. 1B—1F provide
`a general cataloguing capability that can catalogue any type
`of multimedia data. In addition, none of the systems pro-
`vided a viable multimedia system solution. That is, none of
`the systems provide viable options for each of the multime-
`dia components identified in FIG. 1A All of the systems
`merge the components identified in FIG. 1A into a single,
`component that makes it impossible to replace one of the
`components. Further, by combining the components into a
`single component, each system must run on a single hard-
`ware platform. Further,
`there is no vendor—independent
`interface available to integrate components from different
`vendors to construct at optimum multimedia system.
`A number of prior art patents that describe software
`and/or hardware systems are provided below. These systems
`do not provide a general cataloguing capability or interfaces
`to interconnect the multimedia system components identi-
`fied in FIG. 1A.
`
`~
`
`A computer system having a television circuit that is
`cap able of receiving television signals and forwarding these
`signals to the computer system’s audio and video sub-
`systems is described in U.S. Pat. Nos. 5,192,999 (Graczyk et
`al., issued on Mar. 9, 1993), 5,283,638 (Engberg et al.,
`issued Feb. 1, 1994), and 5,283,819(Glick et al., issued Feb.
`1, 1994). Video signals are received for display on the
`computer system’s monitor. Audio signals are output using
`the computer system’s speaker(s). A remote control device
`is used to control the television circuit within the computer
`system. A computer system having the television circuit can
`
`4
`be used in combination with a data/fax/voice modem circuit
`for telephonic transmission of television signals for telecon-
`ferencing purposes.
`A service for managing hypertext/hypermedia links is
`described in US. Pat. No. 5,297,249, Bernstein et al., issued
`on Mar. 22, 1994. A Link Manager Services (LMS) com-
`ponent provides a universal End User Interface (EUI)
`including menus and dialog boxes that can be called by an
`application program when a link, or marker, is selected by an
`application user. The LMS accesses a database that contains
`information about markers and their associated “presenters”
`(e.g., application or program). In addition, the LMS includes
`a viewer capability for viewing and maintaining existing
`links and creating new links. The LMS is used at runtime to
`generate menus and dialog boxes and manage the hypertext/
`hypermedia links.
`A system that interconnects audio-video equipment such
`as video tape record or video production switcher using one
`or more “AV LANS” in U.S. Pat. No. 5,307,456, Mackay,
`issued on Apr. 26, 1994. Generic device commands are
`communicated over an AV LAN to a device translator. The
`device translator translates the generic command into a
`device-specific command. The AV LANs are interconnected
`using bridges. The AV I.ANs can further be used to inter-
`connect workstations, minicomputers, mainframes and per-
`sonal computers: The workstations can be used to display
`information about the resources attached to an AV LAN.
`
`A controller apparatus for use with a computer system that
`provides programmed volume control of multiple analog
`input signals such as CD-ROM,
`telephony, MIDI, PCM,
`FM, and microphone in U.S. Pat. No. 5,402,499, Robison et
`al., issued on Mar. 28, 1995. Additionally, the controller can
`sum multiple inputs and produce multiple audio outputs. The
`controller provides telephony support such that it receives
`voice, DTMF and caller ID signals and sends telephonic
`audio signals.
`A multimedia system that provides a multimedia device
`control program (MMDCP) that interfaces with a multime-
`dia application program (MMAP) and a multimedia presen-
`tation manager (MMPM) in OS/2 in U.S. Pat. No. 5,428,
`730, Baker et al,, issued on Jun. 27, 1995. The MMDCP
`provides a standardized user interface. Because many of the
`control functions are performed by the MMDCP, a MMAP
`can be simplified. The MMDCP creates an instance of a
`control panel for use with a multimedia device. A control
`panel instance includes UI widgets (e.g., buttons) that allow
`a user to control the associated device via the control panel
`instance and the device driver.
`
`An expansion unit to implement multimedia capabilities
`on an existing computer system (e.g., IBM PS/2) in US. Pat.
`No. 5,434,592, Dinwiddie, Jr. et a1., issued on Jul. 18, 1995 .
`The expansion unit is connected to the computer system via
`an expansion slot and to the computer system’s monitor. The
`expansion slot controls the video presentation displayed on
`the monitor.
`
`A multimedia network bus (MMNB), multimedia network
`controller (MMNC) and protocol for transmitting isochro-
`nous and packet data between a backbone or wide area
`network and multiple local area network configurations in
`U.S. Pat. No. 5,436,898, Bowen et 211., issued on Jul. 25,
`1995. The MMNC allocates isochronous channels and
`packet channels to users over the MMNB taking into
`account the time delay requirements of various data types
`(e.g., voice, moving images, graphics, or text). The MMNB
`and MMNC is for use with multimedia applications that
`require a varying mix of isochronous data and packet data
`transfers.
`
`10
`
`‘I5
`
`30
`
`45
`
`50
`
`55
`
`60
`
`65
`
`

`
`Case 2:11-cv-10701-DOC-JPR Document 1-1 Filed 12/27/11 Page 30 of 44 Page ID #:75
`Filed 12/27/11
`Case 2:11—cv—10701—DOC—JPR Document 1-1
`Page 30 of 44 Page ID #275
`
`US 6,199,060 B1
`.
`
`—3
`
`SUMMARY OF THE INVENTION
`
`The invention is a generalized solution for management of
`multimedia assets. Generalized interfaces are used between
`a browser component, indexing server, archive server, ter-
`tiary storage