(cid:9) (cid:9)
`
`METHOD AND APPARATUS MANAGEMENT
`(54) METHOD AND APPARATUS MANAGEMENT
`OF MULTIMEDIA ASSETS
`OF MULTIMEDIA ASSETS
`
`(75) Inventor: Samuel Gustman, Santa Monica, CA
`Inventor: Samuel Gustman, Santa Monica, CA
`(US)
`(US)
`
`(73) Assignee: Survivors of THW Shoah Visual
`Assignee: Survivors of THW Shoah Visual
`History Foundation, Los Angeles, CA
`History Foundation, Los Angeles, CA
`(Us)
`(US)
`
`_
`Notice: (cid:9)
`(*) Notlce:
`
`_
`_
`_
`This patent issued on a continued pros-
`Th15_ Pawnt lssufid on a Con?rmed Pros‘
`ecution application filed under 37 CFR
`ecunon apphcanon ?led under 37 CFR
`1.53(d), and is subject to the twenty year
`1.53(d), and is subject to the tWenty year
`patent term provisions of 35 U.S.C.
`patent term provisions of 35 U.S.C.
`154(a)(2).
`
`154(a)(2).
`
`Subject to any disclaimer the term of this
`Subject to any disclaimer, the term of this
`patent is extended or adgusted under 35
`patent is extended or adjusted under 35
`US'C' 154(k)) by 0 days'
`U.S.C. 154(b) by 0 days.
`
`(21) Appl. No.: 09/076,504
`(21) Appl' NO‘: 09/076’504
`
`(22) Filed;
`(22) Filed: (cid:9)
`
`May 12, 1998
`May 12, 1998
`
`Related U_S_ Application Data
`Related U.S. Application Data
`
`(62) Division of application No. 08/678,727, filed on Jul. 10,
`(62) Division of application No. 08/678,727, ?led on Jul. 10,
`1996, now Pat. No. 5,813,014.
`1996, HOW Pat- NO- 5,813,014-
`(51) Int. Cl.7 .................................................... .. G06F 17/30
`(51)
`Int. C1.7
` GO6F 17/30
`(52) US. Cl. ........................ .. 707/3; 707/1; 707/2; 707/4;
`(52) U.S. Cl.
` 707/3; 707/1; 707/2; 707/4;
`707/5 707/104 709/216
`707/5; 707/104; 709/216
`’
`707/104 1 2
`(58) Field of Search
` 707/104, 1, 2,
`(58) Field of Search (cid:9)
`367 452’ 231?
`707/3
`707/3, 4, 5; 370/338, 395, 367, 452, 231;
`395/500 2’00’7 ’38 49 7’9_ 34’5/352’_ 706/216
`395/500, 200.7, 38, 49, 79; 345/352; 709/216;
`’
`'
`’
`’
`’
`’
`710/74 712/226
`710/74; 712/220
`’
`
`(56)
`(56)
`
`References Cited
`References Cited
`
`5 192 999
`5,192,999 (cid:9)
`5,241,671
`5,241,671 (cid:9)
`
`,
`
`,
`
`U.S. PATENT DOCUMENTS
`[)3 PATENT DOCUMENTS
`348/552
`3/1993 G
`k t
`1
` 348/552
`3/1993 Graczyk et al. (cid:9)
`8/1993 Reed it al. ......................... .. 707/104
`8/1993 Reed et al. (cid:9)
` 707/104
`
`e a .
`
`................... ..
`
`racZ
`
`11111111110111111 (cid:9)
`1111111111111111111111111111111
`11
`0
`US006199060B1
`US 6,199,060 B1
`(to) Patent No.: (cid:9)
`(10) Patent N0.:
`US 6,199,060 B1
`*Mar. 6, 2001
`(45) Date of Patent: (cid:9)
`(45) Date of Patent:
`*Mar. 6, 2001
`
`01111
`
`191
`91
`
`(12) United States Patent
`(12) United States Patent
`Gustman
`Gustman
`
`OTHER PUBLICATIONS
`OTHER PUBLICATIONS
`
`IEEE publication, “Database interoperation support in mul
`IEEE publication, "Database interoperation support in mul-
`timedia applications architecture and methodology” by W.
`timedia applications architecture and methodology" by W.
`A. Gray et al., The Institute of Electrical Engineers, Dept. of
`A. Gray et al., The Institute of Electrical Engineers, Dept. of
`Computer Science, UK, pp. 2/1—2/6, Jan. 1998.*
`Computer Science, UK, pp. 2/1-2/6, Jan. 1998.*
`
`(List continued on next page.)
`(List continued on next page.)
`
`Primary Examiner—Thomas G. Black
`Primary Examiner—Thomas G. Black
`Assistant Examiner—Diane D. MiZrahi
`Assistant Examiner—Diane D. Mizrahi
`(74) Attorney, Agent, or Firm—The Hecker LaW Group
`(74) Attorney, Agent, or Firm—The Hecker Law Group
`
`(57) (cid:9)
`(57)
`
`ABSTRACT
`ABSTRACT
`
`.
`
`.
`
`.
`
`.
`
`.
`
`The invention is a generalized solution for management of
`The invention is a generalized solution for management of
`multimedia assets. Generalized interfaces are used between
`multimedia assets. Generalized interfaces are used betWeen
`a browser component, indexing server, archive server, ter-
`a broWser component, indexing server, archive server, ter
`tiary storage manager, and method player components. The
`tiary storage manager, and method player components. The
`generalized interface de?nes a communication protocol that
`generalized interface defines a communication protocol that
`can be used by any browser, indexing server, archive server,
`can be used by any broWser, indexing server, archive server,
`tertiary storage manager, or method player component.
`tertiary storage manager, or method player component.
`Thus, system components supplied by multiple vendors can
`Thus, system components supplied by multiple vendors can
`be interconnected to form a multimedia system that com-
`be interconnected to form a multimedia system ‘that com
`municates using the generalized interfaces of the invention.
`mumcates using the generalized interfaces of the invention.
`A browser component can accept input such as search
`A broWser component can accept input such as search
`requests and display multimedia data. An indexing server
`requests and display multimedia data. An indexing server
`component includes a data catalogue that catalogues multi
`component includes a data catalogue that catalogues multi-
`media data. The catalogue can be used to identify catalogue
`media data. The catalogue can be used to identify catalogue
`elements associated with data that can satisfy a search
`elements associated With data that can satisfy a search
`request. A search mechanism is used to identify the desired
`requfzst'
`Search mechanism is uséd to identify the d_e_sired
`multimedia data. A search mechanism can create additional
`muhlmedla data" A Search mechamsm can Create. addmonal
`catalogues that are a subset of the catalogue provided on the
`catalogues that are a subset of the catalogue provided on the
`indexing server. An archive server identifies a tertiary stor-
`indexing server. An archive server identi?es a tertiary stor
`age manager that manages the store containing the multi-
`age manager that manages the store containing the multi
`media data. The tertiary storage manager retrieves the cor-
`media data. The tertiary storage manager retrieves the cor
`responding multimedia data from its store and sends it to a
`responding multimedia data from its store and sends it to a
`method play which plays it in a display region of the
`method play Which plays it in a display region of the
`browser. The invention retains the content and results of a
`broWser. The invention retains the content and results of a
`search such that it is only necessary to perform a search
`search such that it is only necessary to perform a search
`once. Search elements are used to store the content of a
`once. Search elements are used to store the content of a
`search, i.e., search criteria. A segment element is used to
`search, i.e., search criteria. A segment element is used to
`store the results of a search.
`Store the results of a Search‘
`
`(List continued on next page.)
`(List continued on next page.)
`
`18 Claims, 24 Drawing Sheets
`18 Claims, 24 Drawing Sheets
`
`418
`418
`
`426
`426
`
`Element
`
`1
`
`Page 1 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`

`

`US 6,199,060 B1
`US 6,199,060 B1
`Page 2
`Page 2
`
`U.S. PATENT DOCUMENTS
`US. PATENT DOCUMENTS
`
`_'
`
` 707/10
`7/1997 (cid:9) Ferguson (cid:9)
`5,649,186 (cid:9)
`7/1997 Ferguson .............................. .. 707/10
`5,649,186
`5,754,784 * (cid:9) 5/1998 (cid:9) Garland (cid:9)
` 395/200.49
`5,754,784 * 5/1998 Garland
`.. 395/20049
`5,774,698 * (cid:9) 6/1998 Olnowich (cid:9)
` 395/500
`5,774,698 * 6/1998 Olnowich ........................... .. 395/500
`
`OTHER PUBLICATIONS
`OTHER PUBLICATIONS
`
`348/14
`
`2/1994 Enberg et al. (cid:9)
`5,283,638
`2/1994 Enberg er a1- ----------------------- -- 348/14
`5,283,638
`2/1994 Glick et al. (cid:9)
`
`379/93
`5,283,819
`2/1994 Glick et a1.
`379/93
`5,283,819
`3/1994 Bernstein et al. (cid:9)
`
`384/14
`5,297,249
`3/1994 Bernstein et a1. .
`384/14
`5,297,249
`4/1994
`MacKay (cid:9)
`
`345/328
`5,307,456
`IEEE publication, "Content based navigation in a open
`IEEE publicatibn, “Content based neivigation in a Qpen
`Ill/Iss5‘)?
`"" "
`3/1995
`Robison et al. (cid:9)
`
`381/119
`5,402,499
`hypermedia environment" by P.H. Lewis et al., The Institute
`hypermedia environment by RH. Lewis et al.,~The Institute
`4/1995 Fujisawa et aL __
`7O7/4
`574047506
`4/1995 Fujisawa et al. (cid:9)
`
`707/4
`5,404,506
`of Electical Engineers, Department of Electronics and Com-
`5,428,730 * 6/1995 Baker (cid:9)
`
`395/154
`of ElectlcalEnglneers?epartmentof Electronlcs and Com
`5,428,730 * 6/1995 Baker .................. ..
`. 395/154
`puter Science, UK pp. 5/1-5/6, Jan. 1995.*
`5,434,592
`7/1995 Dinwiddie, Jr. et al. (cid:9)
`
`345/133
`puter Science, UK pp. 5/1—5/6, Jan. 1995*
`5,434,592
`7/1995 Dinwiddie, Jr. et a1
`. 345/133
`5,436,898
`7/1995 Bowen et al. (cid:9)
`
`370/352 Musei Vaticani, Internet, (http://www.virtual.co.il/orgs/ar-
`5,436,898
`7/1995 Bowen et a1. ..................... .. 370/352
`Musei Vaticani, Internet, (http://WWW.virtua1.c0.il/orgs/ar
`5,450,581
`9/1995 Bergen et al. (cid:9)
`
`707/9
`chival/bookintr.htm), pp 1-10, Dec. 8,1994.
`5,450,581
`
`
`
`
`1_10, 136C' 8, chival/bookintr_htm), The Judaica Archival Project, Internet, (http://WWW.virtua1
`
`5,530,852
`6/1996 Meske, Jr. et al. (cid:9)
`
`709/206
`The Judaica Archival Project, Internet, (http://www.virtual-
`5,608,900
`3/1997 Dockter et al. (cid:9)
`
`707/102
`.co.il/orgs/archiva/bookintr.htm), pp 1-11, Apr. 1996.
`.c0.rl/orgs/archrva/bookintrhtrn), pp 1 11, Apr. 1996.
`576307121
`5/1997 Braden_Harder et a1‘
`' 7O7/1O2
`5,630,121
`5/1997 Braden-Harder et al. (cid:9)
`
`707/102
`5,649,185
`7/1997 Antognini et al. (cid:9)
`
`707/9
`* cited by examiner
`5,649,185
`7/1997 Antognini et a1. ..................... .. 707/9
`* cited by examiner
`
`......................... .. 707/9 Bergen CI 8.1. 115465112?’ Jr~tet1a1~ -
`
`,
`
`,
`
`0c er e a. ..... ..
`
`.
`
`-
`
`-
`
`-
`
`_
`
`Page 2 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent (cid:9)
`U.S. Patent
`
`Mar. 6, 2001 (cid:9)
`Mar. 6, 2001
`Mar. 6, 2001 (cid:9)
`
`Sheet 1 of 24
`Sheet 1 0f 24
`Sheet 1 of 24
`
`US 6,199,060 B1
`US 6,199,060 B1
`US 6,199,060 B1
`
`102
`
`104
`
`Method
`Player
`
`Tertiary
`Storage
`Manager
`
`110
`110
`110
`
`106
`106
`106
`
`Archive
`Server
`
`Index
`Server
`
`108
`108
`108
`
`FIG. 1A
`FIG. 1A
`
`Client and
`Method
`Player
`
`112
`112
`/
`/
`
`Hewlett Packard
`Hewlett Packard
`Hewlett Packard
`Hardware Platform
`Hardware Platform
`Hard ware Platform
`
`118
`118
`118
`
`Intermedia
`lntermedia
`in termedia
`Server
`Server
`Server
`with
`with
`w ith
`Archive Server
`Archive Server
`Archive Server
`and
`and
`and
`Tertiary Storage
`Tertiary Storage
`Tertiary Storage
`Manager
`Manager
`Manager
`
`Index
`Server
`
`‘ ,
`
`116
`116
`116
`
`FIG. 1B
`FIG. 1B
`
`Page 3 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`
`U.S. Patent
`
`120
`
`Mar. 6, 2001 (cid:9)
`Mar. 6, 2001
`
`Sheet 2 of 24
`Sheet 2 0f 24
`
`US 6,199,060 B1
`US 6,199,060 B1
`
`124
`124
`
`138
`
`132
`
`Tertiary
`Storage
`Manager
`
`126
`
`130
`
`FIG. 1C
`
`Mainframe
`144
`14-4
`
`148
`
`140-,
`\
`
`FIG. 1D
`
`Page 4 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Mar. 6, 2001
`Mar. 6, 2001
`
`Sheet 3 0f 24
`Sheet 3 of 24 (cid:9)
`
`US 6,199,060 B1
`US 6,199,060 B1
`
`"""""""""
`162
`
`Method
`Player
`
`Method Player
`
`160G
`
`160F
`
`172
`
`160H
`
`160A
`160B
`
`160C
`170
`
`Tertiary
`Tertiary
`Storage
`Storage
`Manager
`Manager
`
`25,174
`
`164
`
`'
`
`164
`
`160E
`
`160D
`
`166
`
`Archive
`Arch iv 9
`Server
`Server
`
`a a a a a a a a , a a a a ' a , . a 1 a 1 v4
`
`178
`
`178 N. N.
`E
`
`190
`
`FIG. lE
`
`182
`182
`
`Method
`Player
`
`188
`
`
`
`Index Server
`
`Archive
`
`
`
`Tertiary Sim-age
`
`Server
`
`anager
`
`186
`186
`
`184
`
`'
`
`' . 1 1 1 1 1 1 1 v v r v . i 1 I r 1 I ' 1 v - 1 a a 1 v 1 I 1 1 a ' a . v 1 - 1 1 r p I 1 1 1 a ' r 1 1 1 1 r 1 r 1 1 . 1 r - a - .1
`
`FIG. 1F
`
`Page 5 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Mar. 6, 2001
`Mar. 6, 2001
`
`Sheet 4 of 24 (cid:9)
`Sheet 4 0f 24
`
`US 6,199,060 B1
`US 6,199,060 B1
`
`219
`219
`
`216
`216
`/
`
`VIDEO AMP
`VIDEO AMP
`
`CRT
`
`1
`217
`217
`/
`
`213
`213
`
`,..........r.
`
`214
`214
`/
`
`CPU
`
`VIDEO MEMORY
`VIDEO MEMORY
`
`MAIN MEMORY
`MAIN MEMORY
`
`215
`/
`
`l
`
`lir
`KEYBOARD
`KEYBOARD
`
`210
`
`(cid:9)Y
`
`iiimiinrIMIIIIIIIIIIIMINIMMIk
`
`__T__
`
`218
`218
`/
`
`211
`MOUSE /
`
`1/212
`212
`
`MASS STORAGE
`
`FIG. 2
`FIG. 2
`
`Page 6 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Mar. 6, 2001
`Mar. 6, 2001
`
`Sheet 5 of 24 (cid:9)
`Sheet 5 0f 24
`
`US 6,199,060 B1
`US 6,199,060 B1
`
`318
`
`310
`
`Browser FL» 9 Player
`314 y‘ /
`2
`\
`
`312
`
`316
`316
`
`Indexing
`Server
`
`306
`
`FIG. 3
`FIG. 3
`
`Page 7 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Mar. 6, 2001
`Mar. 6, 2001
`
`Sheet 6 of 24 (cid:9)
`Sheet 6 0f 24
`
`US 6,199,060 B1
`US 6,199,060 B1
`
`Segment
`Segment
`ID
`ID
`descriptive phrase
`descriptive phrase
`set( phrases)
`set(phrases)
`
`404
`404
`
`’
`
`Phrase
`'
`t d t ID
`input data ID
`f3,“ a a
`ID
`descriptive phrase
`descriptive phrase
`In/Out Timewdes
`In/Out Timecodes
`se?lfeywrds)
`set(keywords)
`set(images)
`“'(mmges)
`set(persons)
`selfpersons)
`set( proposed keywords)
`set(proposed keywords)
`set( proposed persons)
`set(proposed persons)
`
`410
`410
`
`1/
`
`Type
`Lupe
`ID
`ID
`Label
`Label
`
`Keyword
`Keyword
`ID
`ID
`sel?ypes)
`set(types)
`set(thesaurnl keywords)
`set(thesaural keywords)
`se!( parent keywords)
`set(parent keywords)
`se?child keywords)
`sehchild keywords)
`set(proposed keywords)
`set( r0 osed 1: words)
`P P
`91'
`
`422
`422 (cid:9)
`Thesaural Person
`Ih?emllezan ’
`ID
`ID
`Name
`Name
`Preferred Flag
`Preferred F lag
`
`Thesaural Keyword
`W11
`W11
`ID
`ID
`keyword ID
`keyword ID
`Label
`Label
`language of Choice
`Language of Choice
`Preferred Flag
`Preferred Flag
`Characteristic
`CImracterisnt
`
`908
`408
`
`I
`
`418
`418
`
`406
`406
`
`416
`416
`Y
`
`Em!!!
`Event
`ID
`ID
`type
`type
`sub-type
`sub-type
`In/Out Timecodes
`In/Out Timecodes
`comment
`comment
`
`Person
`Person
`ID
`ID
`Primary Name
`Primary Name
`Occupation
`Occupation
`Date of Birth
`Date of Birth
`set(proposed persons)
`set( proposed persons)
`
`412
`412
`
`414
`414
`/
`Proposed Person
`Emmi MMSM
`ID
`ID
`I D
`Name
`Name
`Label
`
`424
`424
`
`Image 1 Video
`Ima e I Video
`ID
`ID
`‘
`In/Out Timecodes
`In/Oul Trmewdes
`Locator
`Lomtor
`
`420
`420 (cid:9)
`/
`
`person Information
`u m ‘- n
`Person ID
`Person ID
`Person Attribute
`Person Attribute
`
`426
`426
`
`,1
`
`FIG. 4A
`FIG. 4A
`
`Page 8 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent (cid:9)
`U.S. Patent
`
`Mar. 6, 2001
`Mar. 6, 2001
`Mar. 6, 2001
`
`Sheet 7 of 24 (cid:9)
`Sheet 7 0f 24
`Sheet 7 of 24 (cid:9)
`
`US 6,199,060 B1
`US 6,199,060 B1
`US 6,199,060 B1
`
`Segment 447-» Segment
`450
`
`428$ T
`
`r 406
`
`Phrase
`
`430
`
`432 \
`
`410 (cid:9)
`410
`/
`K8 0rd
`yw
`
`420 (cid:9)
`420
`
`424
`424
`
`; 418
`'/
`
`Image/
`Video
`
`Proposed
`Person
`
`Person
`
`f
`
`444
`
`242
`
`434
`
`414
`414 (cid:9)
`
`Proposed
`Proposed
`Proposed
`Keyword
`Keyword
`Keyword
`
`438
`
`436
`
`426
`426
`426
`/
`
`422
`/
`
`Thesaural
`PEL'SOII
`
`Person
`Information
`
`412
`
`410
`
`Thesaural
`Keyword
`
`K
`d
`eywor
`
`416
`Y J
`
`T
`ype
`
`446
`446
`
`416
`416
`
`Type
`Type
`
`FIG. 4B
`FIG. 4B
`FIG. 4B
`
`Page 9 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent (cid:9)
`U.S. Patent
`
`Mar. 6, 2001
`Mar. 6, 2001
`Mar. 6, 2001
`
`Sheet 8 of 24 (cid:9)
`Sheet 8 of 24 (cid:9)
`Sheet 8 0f 24
`
`US 6,199,060 B1
`US 6,199,060 B1
`US 6,199,060 B1
`
`Thesaural (cid:9)
`486 \ Thesaural
`Thesaural (cid:9)
`486 N (cid:9)
`486 N (cid:9)
`Keyword One
`Keyword One (cid:9)
`Keyword One (cid:9)
`
`Thesaural
`Thesaural
`Thesaural
`Keyword Two
`Keyword Two
`Keyword Two
`
`Thesaural
`Thesaural
`Thesaural
`Keyword Three
`Keyword Three
`Keyword Three
`
`Thesaural
`Thesaural
`Thesaural
`Keyword Four
`Keyword Four
`Keyword Four
`
`488
`488 (cid:9)
`488
`\
`
`490D
`4900
`490C
`49°C
`490B
`4908
`Keyword
`Keyword
`Keyword 490A
`490A
`Instance
`Instance
`Instance
`one
`One
`One
`
`491E
`491E
`491E
`4910
`491D
`491D
`491C
`491C
`4918
`491B
`491B
`491A
`491A
`491A
`
`49213
`
`4933
`
`4943
`494A
`
`4958
`495A
`
`Keyword
`Keyword
`Keyword
`Instance
`Instance
`Instance
`Two
`Two
`Two
`
`496C
`496C
`496C
`496B
`496B
`4968
`
`4 96A
`
`4 96A
`496A
`
`4973
`497A
`
`FIG. 4C
`FIG. 4C
`FIG. 4C
`
`Page 10 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Mar. 6, 2001
`Mar. 6, 2001
`
`Sheet 9 of 24 (cid:9)
`Sheet 9 0f 24
`
`US 6,199,060 B1
`US 6,199,060 B1
`
`318
`
`Browser
`
`Data
`
`510 -.\- File
`512
`
`"N Segment
`514 ..‘
`‘• Phrase
`
`/ 314
`314
`/
`
`516
`\ Keyword
`516 '- Keyword
`518 -,
`518 \ Keyword Hierarchy
`‘• Keyword Hierarchy
`520
`520
`
`Person
`\ Person
`
`522 .-
`522
`‘• Thesaural Keyword
`\ Thesaural Keyword
`524 -
`
`524 \ Type
`N Type
`526 -'
`526 \ Type Hierarchy
``• Type Hierarchy
`
`Request
`
`504
`
`Indexing
`Server
`
`316
`
`FIG. 5
`FIG. 5
`
`Page 11 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent (cid:9)
`U.S. Patent
`
`Mar. 6, 2001
`Mar. 6, 2001
`Mar. 6, 2001
`
`Sheet 10 of 24
`Sheet 10 of 24 (cid:9)
`Sheet 10 0f 24
`
`US 6,199,060 B1
`US 6,199,060 B1
`US 6,199,060 B1
`
`602
`
`No
`No
`
`606
`606
`606
`,
`/
`Yes-b perform search
`perform search
`perform search
`Yes
`
`608
`
`610
`610
`610
`z
`,
`/
`transmit command
`Yes
`transmit command
`Yes
`Yes-i transmit command -—>
`
`Yes
`Yes
`Yes
`
`Search
`operatlon?
`
`N 0
`
`No
`No
`No
`
`FIG. 6
`FIG. 6
`FIG. 6
`
`Page 12 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent (cid:9)
`U.S. Patent
`
`Mar. 6, 2001
`Mar. 6, 2001
`Mar. 6, 2001
`
`Sheet 11 0f 24
`Sheet 11 of 24 (cid:9)
`Sheet 11 of 24 (cid:9)
`
`US 6,199,060 B1
`US 6,199,060 B1
`US 6,199,060 B1
`
`search already
`exists?
`
`
`
`No No
`
`Yes
`Yes
`Yes
`
`segment
`associated with
`search?
`
`No
`No
`
`Yes (cid:9)
`Yes
`Yes
`i
`retrieve query
`retrieve query
`retrieve query
`segment
`segment
`segment
`
`706
`706
`1
`
`i
`
`708
`
`708
`708
`
`I
`
`retrieve phrase ids
`retrieve phrase ids
`retrieve phrase ids
`associated with
`associated with
`associated with
`query segment
`query segment
`query segment
`
`FIG. 7A
`FIG. 7A
`FIG. 7A
`
`Page 13 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent (cid:9)
`U.S. Patent
`
`Mar. 6, 2001
`Mar. 6, 2001
`Mar. 6, 2001
`
`Sheet 12 0f 24
`Sheet 12 of 24 (cid:9)
`Sheet 12 of 24 (cid:9)
`
`US 6,199,060 B1
`US 6,199,060 B1
`US 6,199,060 B1
`
`person
`person
`person
`
`722
`722
`722
`background
`background
`background
`— information
`
`information
`— information
`
`
`724
`724
`724
`y
`/
`/
`
`h
`search person
`search person
`Seam person
`
`type/
`type/
`type/
`keyword
`keyword
`keyword
`i
`"fk
`4
`search type
`search type
`search type
`and/or keyword
`and/ or keyword
`and/or keyword
`
`726
`726
`726
`1
`
`728
`728
`728
`_.•
`/
`search background
`search background
`search background
`information
`information
`information
`
`730
`730
`
`phrase id(s)
`identified?
`
`Yes
`Yes
`YES
`V
`create segment
`instance
`
`732
`732
`732
`1
`
`1
`
`m I
`
`734
`734
`
`create query
`elements
`
`736
`
`FIG. 7B
`FIG. 7B
`FIG. 7B
`
`Page 14 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent (cid:9)
`U.S. Patent
`
`Mar. 6, 2001
`Mar. 6, 2001
`Mar. 6, 2001
`
`Sheet 13 0f 24
`Sheet 13 of 24 (cid:9)
`Sheet 13 of 24 (cid:9)
`
`US 6,199,060 B1
`US 6,199,060 B1
`US 6,199,060 B1
`
`all search elements
`processed?
`
`802
`
`816
`816
`816
`L
`find all subelements
`find all subelements
`find all subelements
`Yes
`Yes
`Yes_’ of elements in set
`of elements in set
`of elements in set
`
`No
`No
`NO
`V
`V
`v
`
`804
`804
`z
`/
`
`get next element
`get next element
`get next element
`
`806
`806
`/ 806
`z
`
`l
`search element
`search element
`search element
`attribute for
`attribute for
`attribute for
`search element
`search element
`search element
`
`818
`
`l
`818
`818
`z
`V (cid:9)
`z
`V (cid:9)
`/
`add subelements
`add subelements
`add subelements
`to element set
`to element set
`t0 element set
`
`820
`820
`820
`
`C end )
`end
`
`810
`810
`810
`
`No——
`
`Yes
`Yes
`Yes ——>
`
`Yes •
`Yes
`Yes ‘
`
`element in
`set?
`
`
`No
`
`No
`N 0
`V
`ir
`ir
`add element to
`add element to
`add element to
`element set
`element set
`element set
`
`814
`814
`814
`z
`1
`z
`
`5
`
`FIG. 8A
`FIG. 8A
`FIG. 8A
`
`Page 15 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent (cid:9)
`U.S. Patent
`
`Mar. 6, 2001
`Mar. 6, 2001
`Mar. 6, 2001
`
`Sheet 14 of 24 (cid:9)
`Sheet 14 of 24 (cid:9)
`Sheet 14 0f 24
`
`US 6,199,060 B1
`US 6,199,060 B1
`US 6,199,060 B1
`
`822
`
`all elements in
`all elements in
`all elements in
`set processed?
`set processed?
`set processed?
`
`Yes
`Yes
`Yes
`
`search element
`hierarchy?
`
`No—
`No—
`
`No
`No
`No
`824
`824
`824
`V
`z
`r
`1
`set next element as
`set next element as
`set next element as
`current search
`current search
`current search
`element
`element
`element
`l
`search element
`search element
`search element
`attribute using
`attribute using
`attribute using
`search element
`search element
`search element
`
`826
`, 826
`826
`r
`
`element
`instance
`found ?
`
`Yes
`Yes
`
`828
`
`830
`830
`
`q-Yes
`Yes
`
`element
`instance found
`in set?
`
`832
`832
`1
`
`N0
`+
`add element
`add element
`add element
`instance to found
`instance to found
`instance to found
`element set
`element set
`element set
`
`Yes
`Yes
`Yes
`l
`836
`836
`836
`4
`Ilk
`/
`/
`I
`find all child elements for
`ind all child elements for
`find all child elements for
`elements in
`elements in
`elements in
`found element set
`found element set
`found element set
`
`i
`
`838
`
`838
`838
`/
`add child elements to
`add child elements to
`add child elements to
`found element set
`found element set
`found element set
`
`4 (cid:9)
`
`end
`end
`
`840
`840
`840
`
`)
`)
`
`FIG. 8B
`FIG. 8B
`FIG. 8B
`
`Page 16 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent (cid:9)
`U.S. Patent
`
`Mar. 6, 2001
`Mar. 6, 2001 (cid:9)Mar. 6, 2001 (cid:9)
`
`
`Sheet 15 0f 24
`Sheet 15 of 24 (cid:9)Sheet 15 of 24 (cid:9)
`
`
`US 6,199,060 B1
`US 6,199,060 B1
`US 6,199,060 B1
`
`850
`
`all keywords in
`set processed?
`
`860
`860
`860
`
`Yes (cid:9)
`Yes
`
`end )
`II{ end )
`
`No
`No
`N 0
`852
`852
`852
`V
`/
`
`set next keyword as set next keyword as
`set next keyword as
`current keyword
`
`current keyword current keyword
`0
`search phrase
`search phrase
`search phrase
`elements for
`elements for
`elements for
`keyword association
`
`keyword association keyword association
`
`854
`854
`,
`
`a /
`
`856
`
`phrase instance
`phrase instance
`phrase instance
`
`found not in phrase found not in phrase
`found not in phrase
`set?
`set?
`set?
`
`Yes
`Yes
`Yes
`858
`j
`858
`858
`,
`/
`add phrase instance
`
`add phrase instance add phrase instance
`to phrase set
`to phrase set
`to phrase set
`
`FIG. SC
`FIG. 8C
`FIG. 8C
`
`Page 17 of 43
`
`MINDGEEK EXHIBIT 1001
`
`

`

`U.S. Patent (cid:9)
`U.S. Patent (cid:9)
`U.S. Patent
`
`Mar. 6, 2001
`Mar. 6, 2001
`Mar. 6, 2001
`
`Sheet 16 0f 24
`Sheet 16 of 24 (cid:9)
`Sheet 16 of 24 (cid:9)
`
`US 6,199,060 B1
`US 6,199,060 B1
`US 6,199,060 B1
`
`start
`start
`
`902
`902
`902
`i
`i
`/
`search background information
`search background information
`search background information
`using selection criteria to
`using selection criteria to
`using selection criteria to
`identify person instances
`identify person instances
`identify person instances
`l
`904
`904
`904
`/
`select segment
`select segment
`select segment
`instances for each
`instances for each
`instances for each
`person instance
`person instance
`person instance
`
`906
`906
`906
`V
`/
`create set of phrases
`create set of phrases
`create set of phrases
`associated with
`associated with
`associated with
`selected segments
`selected segments
`selected segments
`
`a Yes
`
`Yes
`
`all phrases in
`set processed?
`
`910
`
`Yes
`
`918
`
`No
`No
`No
`Y
`
`912
`912
`912
`z
`/
`-
`
`get next phrase
`get next phrase
`get next phrase
`
`914
`914
`914
`z
`/
`
`l
`search phrase's
`search phrase's
`search phrase's
`keywords for
`keywords for
`keywords for
`specified content
`specified content
`specified content
`+
`916
`916
`916
`/
`create set of phrases
`create set of phrases
`create set of phrases
`having specified
`having specified
`having specified
`content
`content
`content
`
`FIG. 9
`FIG. 9
`FIG. 9
`
`Page 18 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Mar. 6, 2001
`Mar. 6, 2001
`
`Sheet 17 0f 24
`Sheet 17 of 24 (cid:9)
`
`US 6,199,060 B1
`US 6,199,060 B1
`
`[ (cid:9)
`
`start
`
`
`
`1002
`1002
`i
`1
`
`I
`
`identify segment
`identify segment
`instance(s) associated
`instance(s) associated
`with person instance(s)
`with person instance(s)
`l
`1004
`4
`1004
`identify phrase
`identify phrase
`instance(s) associated
`instance(s) associated
`with segment insance(s)
`with segment insance(s)
`‘
`1006
`1006
`r
`/
`create set of phrases
`create set of phrases
`associated with
`associated with
`selected segments
`selected segments
`
`No
`
`No»@
`
`content search?
`
`Yes
`Yes
`
`all phrases in set
`processed?
`
`1010
`Yes
`
`Yes ->@
`
`FIG. 10A
`PIG. 10A
`
`Page 19 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`

`

`
`U.S. Patent (cid:9)U.S. Patent (cid:9)
`U.S. Patent
`
`Mar. 6, 2001
`Mar. 6, 2001
`Mar. 6, 2001
`
`Sheet 18 of 24 (cid:9)
`Sheet 18 0f 24
`Sheet 18 of 24 (cid:9)
`
`US 6,199,060 B1
`US 6,199,060 B1
`US 6,199,060 B1
`
`1012
`1012
`1012
`A'
`A'
`
`1014
`1014
`1014
`/
`
`get next phrase
`get next phrase
`get next phrase
`l
`V
`search phrase's
`search phrase's
`search phrase's
`keywords for
`keywords for
`keywords for
`specified content
`specified content
`specified content
`l
`1016
`1016
`1
`create set of phrases
`create set of phrases
`create set of phrases
`having specified
`having specified
`having specified
`content
`content
`content
`
`1016
`
`V 1018
`1018
`1018
`•
`( end )
`end
`
`FIG. 10B
`FIG. 10B
`FIG. 10B
`
`Page 20 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent (cid:9)
`
`Mar. 6, 2001
`Mar. 6, 2001
`
`Sheet 19 of 24
`Sheet 19 of 24 (cid:9)
`
`US 6,199,060 B1
`US 6,199,060 B1
`
`Person
`Person
`
`Person
`Person
`Information
`Information
`
`426
`426
`
`0
`0
`1 1102 128
`
`
`1 128
`
`
`1102
`
`444
`444
`
`0
`
`1116
`11111'''
`
`- Query
`Query (cid:9)
`Element
`Element
`
`(-
`
`1118 (cid:9)
`1118 (cid:9)
`
`0
`0
`/ 1130 (cid:9)
`/ 1130 (cid:9)
`
`Keyword
`Keyword
`
`Phrase (cid:9)
`Phrase (cid:9)
`
`/ 428
`
`1
`
`Segment
`Segment
`
`430
`
`0
`
`404
`404
`
`418
`418
`
`/ 432
`/ 432
`
`
`
`1104 (cid:9)
`
`FIG. liA
`FIG. liA
`
`Page 21 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`(cid:9)
`

`

`
`
`U.S. Patent (cid:9)U.S. Patent (cid:9)
`
`
`
`Mar. 6, 2001 Mar. 6, 2001
`
`
`
`Sheet 20 of 24 (cid:9)Sheet 20 of 24 (cid:9)
`
`
`
`US 6,199,060 B1 US 6,199,060 B1
`
`
`
`Query Query
`
`
`
`ID ID
`
`
`
`(cid:9) 1102 (cid:9) 1102
`
`
`
`Query Element Record Query Element Record
`
`
`
`(cid:9)[1106 (cid:9)[1106
`
`
`Query ID Query ID
`
`Query Element ID Query Element ID
`
`ID ID
`
`AndOr AndOr
`
`Element Order Element Order
`
`
`Query Element Query Element
`
`Query Element ID Query Element ID
`
`
`
`1104 1104
`
`
`
`FIG. 11B FIG. 11B
`
`Page 22 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`

`

`
`
`U.S. Patent (cid:9)U.S. Patent (cid:9)
`
`
`
`Mar. 6, 2001 Mar. 6, 2001
`
`
`
`Sheet 21 of 24 (cid:9)Sheet 21 of 24 (cid:9)
`
`
`
`US 6,199,060 B1 US 6,199,060 B1
`
`
`
`406 406
`
`
`
`1206 1206
`
`Phrase Phrase
`
`
`input data ID input data ID
`
`
`
`ID ID
`
`
`
`descriptive phrase descriptive phrase
`
`
`
`In/Out Timecodes In/Out Timecodes
`
`
`set(keywords) set(keywords)
`
`set(images) set(images)
`
`set(persons) set(persons)
`
`set(proposed keywords) set(proposed keywords)
`
`set(proposed persons) set(proposed persons)
`
`
`Phrase Phrase
`
`input data path input data path
`
`
`
`ID ID
`
`
`
`In/Out Timecodes In/Out Timecodes
`
`
`
`Key Frame Key Frame
`
`
`
`1208 1208
`
`
`
`Storage Storage
`
`
`
`1210 1210
`
`
`
`Cache 1 Cache 1
`
`
`- Cache - Cache
`
`Manager Manager
`
`
`
`1214 1214
`
`
`
`FIG. 12 FIG. 12
`
`Page 23 of 43
`
`MINDGEEK EXHIBIT 1001
`
`

`

`U.S. Patent (cid:9)
`U.S. Patent (cid:9)
`
`Mar. 6, 2001
`Mar. 6, 2001
`
`Sheet 22 of 24
`Sheet 22 of 24 (cid:9)
`
`US 6,199,060 B1
`US 6,199,060 B1
`
`start
`start
`
`(cid:9)1
`(cid:9)1
`1300
`1300
`,
`.,
`
`¤
`¤
`retrieve
`retrieve
`multimedia data
`multimedia data
`
`¤
`
`1302
`1302
`z
`z
`
`start thread
`start thread
`
`1304
`1304
`z
`z
`
`¤
`transfer data
`transfer data
`to thread
`to thread
`
`1306
`1306
`.
`V (cid:9)
`thread prepares and
`thread prepares and
`transfers data to browser
`transfers data to browser
`
`1308
`1308
`
`No
`No
`
`Yes
`Yes
`
`1310
`1310
`
`FIG. 13A
`FIG. 13A
`
`Page 24 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`
`
`U.S. Patent (cid:9)U.S. Patent (cid:9)
`
`
`
`Mar. 6, 2001 Mar. 6, 2001
`
`
`
`Sheet 23 of 24 (cid:9)Sheet 23 of 24 (cid:9)
`
`
`
`US 6,199,060 B1 US 6,199,060 B1
`
`
`
`play command? play command?
`
`
`
`Yes Yes
`
`
`
`Yes Yes
`
`
`
`1318 1318
`
`
`process play process play
`
`command command
`
`
`
`FIG. 13B FIG. 13B
`
`Page 25 of 43
`
`MINDGEEK EXHIBIT 1001
`
`

`

`
`
`U.S. Patent (cid:9)U.S. Patent (cid:9)
`
`
`
`Mar. 6, 2001 Mar. 6, 2001
`
`
`
`Sheet 24 of 24 (cid:9)Sheet 24 of 24 (cid:9)
`
`
`
`US 6,199,060 B1 US 6,199,060 B1
`
`
`
`1402 1402
`
`
`
`Uses Uses
`
`
`
`1404 1404
`
`
`
`1418 1418
`
`
`
`Uses Uses
`
`
`
`Uses Uses
`
`
`
`Uses Uses
`
`
`
`1416 1416
`
`
`
`1414 1414
`
`
`
`Uses Uses
`
`
`
`Uses Uses
`
`
`
`Uses Uses
`
`
`
`1412 1412
`
`
`
`1410 1410
`
`
`
`- Contains - - Contains -
`
`
`
`FIG. 14 FIG. 14
`
`Page 26 of 43
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`

`

`1 1
`
`
`METHOD AND APPARATUS MANAGEMENT METHOD AND APPARATUS MANAGEMENT
`
`OF MULTIMEDIA ASSETS OF MULTIMEDIA ASSETS
`
`
`
`US 6,199,060 B1 US 6,199,060 B1
`
`
`The invention relates to a system of managing multime-The invention relates to a system of managing multime-
`
`dia assets. This is a divisional of application Ser. No. dia assets. This is a divisional of application Ser. No.
`
`08/678,727, filed Jul. 10, 1996 and now U.S Pat. No. 08/678,727, filed Jul. 10, 1996 and now U.S Pat. No.
`
`5/813,014 filed Sep. 22, 1998. 5/813,014 filed Sep. 22, 1998.
`
`
`
`BACKGROUND OF THE INVENTION BACKGROUND OF THE INVENTION
`
`
`1. Field of the Invention 1. Field of the Invention
`
`2. Background 2. Background
`
`Increasingly, computer systems are being used to present Increasingly, computer systems are being used to present
`
`multimedia material. Such material is usually in the form of multimedia material. Such material is usually in the form of
`
`text, graphics, video, animation, and sound. Two or more of text, graphics, video, animation, and sound. Two or more of
`
`these data types are usually combined to form the multime-these data types are usually combined to form the multime-
`
`dia data presented by the computer system. A computer dia data presented by the computer system. A computer
`
`system that is used to present multimedia material is called system that is used to present multimedia material is called
`
`a multimedia system. a multimedia system.
`
`A multimedia system should include components that A multimedia system should include components that
`
`allow input, information retrieval, and display. FIG. 1A allow input, information retrieval, and display. FIG. 1A
`
`provides an illustration of components for a multimedia provides an illustration of components for a multimedia
`
`system. Client 110 includes an interface capability to accept system. Client 110 includes an interface capability to accept
`
`user input. The user input can specify criteria for identifying user input. The user input can specify criteria for identifying
`
`multimedia data. Method player 102 converts the media data multimedia data. Method player 102 converts the media data
`
`from its storage version to a playback version for use by from its storage version to a playback version for use by
`
`client 110. client 110.
`
`Index server 108, archive server 106, and tertiary storage Index server 108, archive server 106, and tertiary storage
`
`104 provide storage and retrieval capabilities. Index server 104 provide storage and retrieval capabilities. Index server
`
`108 is used to store and retrieve text data. Information 108 is used to store and retrieve text data. Information
`
`entered in the user interface can be used to retrieve textual entered in the user interface can be used to retrieve textual
`
`information stored by index server 108. The user input might information stored by index server 108. The user input might
`
`identify a video segment using a title that is associated with identify a video segment using a title that is associated with
`
`the video segment using index server 108, for example. The the video segment using index server 108, for example. The
`
`multimedia data is stored on a physical storage device. A multimedia data is stored on a physical storage device. A
`
`retrieval mechanism is provided to retrieve data from a retrieval mechanism is provided to retrieve data from a
`
`physical st