(12) United States Patent
`(12) United States Patent
`Gustman
`Gustman
`
`11111111111111111111111111fpR111111111111111 (cid:9)
`US006212527B1
`US 6,212,527 B1
`US 6,212,527 B1
`*Apr. 3, 2001
`*Apr. 3, 2001
`
`(to) Patent No.: (cid:9)
`(10) Patent N0.:
`(45) Date of Patent:
`(45) Date of Patent: (cid:9)
`
`01111
`
`(54) METHOD AND APPARATUS FOR
`(54) METHOD AND APPARATUS FOR
`CATALOGUING MULTIMEDIA DATA
`CATALOGUING MULTIMEDIA DATA
`
`Inventor: Samuel Gustman, Santa Monica, CA
`(75)
`(75) Inventor: Samuel Gustman, Santa Monica, CA
`(Us)
`(US)
`
`*
`_
`( * ) Notice:
`(
`) Notlcei
`
`.
`_
`.
`.
`.
`(73) Assignee: Survivors of the Shoah Visual History
`(73) Asslgnee" lszurvlzl'ofs 02311921102‘? vlélx‘lgslsmry
`Foundation, Los Angeles, CA (US)
`Oun am“, 05
`ge 65>
`(
`)
`_
`_
`_
`This patent issued on a continued pros-
`Th1s_ Pawnt {$511901 on a Con?rmed PYOS-
`ecution application filed under 37 CFR
`ecution application ?led under 37 CFR
`1.53(d), and is subject to the twenty year
`1.53(d), and is subject to the tWenty year
`patent (cid:9)
`term (cid:9) provisions (cid:9) of 35 (cid:9) U.S.C.
`patent term provisions of 35 U_S_C_
`154(a)(2).
`154(21)(2)_
`
`Subject to any disclaimer, the term of this
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`gusenct $532)???) (21f adjusted under 35
`U.S.C. 154(b) by 0 days.
`'
`'
`'
`y
`ays'
`
`~
`
`~
`
`(21) Appl. No.: 09/184,921
`(21) Appl. N0.: 09/184,921
`
`(22) Filed: (cid:9)
`22 Filed:
`(
`)
`
`Nov. 2, 1998
`Nov. 2 1998
`’
`Related US Application Data
`Related U.S. Application Data
`
`_
`_
`_
`_
`(63) Continuation of application No. 08/680,504, filed on Jul. 8,
`(63) Continuation of application No. 08/680,504, ?led on Jul. 8,
`1996, now Pat. No. 5,832,495.
`1996: HOW Pat- N°~ 578327495
`(51)
`Int. C1.7
` GO6F 17/30
`(51) Int. ci.7 .................................................... .. G06F 17/30
`(52) U S C]
`707/102. 707/1
`(52) U.S. Cl. (cid:9)
` 707/102; 707/1
`(58) Field of Search
`707h02 1
`(58) Field of Search (cid:9)
` 707/102, 1
`""""""""""""""""""""" "
`’
`References Cited
`References Cited
`
`(56)
`(56)
`
`U.S. PATENT DOCUMENTS
`US. PATENT DOCUMENTS
`4/1991 Makkuni et a1
`5 010 500
`4/1991 Makkuni et al. .
`5,010,500 (cid:9)
`395/8O7
`6/1992 Beitel et a1
`'
`'
`5’119’474
` 395/807
`6/1992 Beitel et al. (cid:9)
`5,119,474 (cid:9)
`6/1992 Trautman _' iiiiiiiiiiiiiiiiiiiiiiiiiiii 395/140
`571217470
`6/1992 Trautman (cid:9)
` 395/140
`5,121,470 (cid:9)
`6/1992 Kasahara et al. .................. .. 395/357
`5,123,088
`6/1992 Kasahara et al. (cid:9)
` 395/357
`5,123,088 (cid:9)
`5/1993 Shimada et al. (cid:9)
` 395/615
`5,210,868 (cid:9)
`5/1993 Shimada et al. ................... .. 395/615
`5,210,868
`5,241,671 * 8/1993 Reed et al. (cid:9)
` 707/104
`5,241,671 * 8/1993 Reed et al. ......................... .. 707/104
`
`395/62
`
`12/1993 Beitel et al. (cid:9)
`5,274,758
`12/1993 Beitel et al. ......................... .. 395/62
`5,274,758
`5,278,946
`395/62
`1/1994 Shimada et al. (cid:9)
`
`1/1994 Shimada et al
`395/62
`5,278,946
`395/338
`1/1994 Kuga et al. (cid:9)
`
`5,280,573
`1/1994 Kuga et al
`.. 395/338
`5,280,573
`379/67
`4/1994 Zucker (cid:9)
`
`5,303,296
`4/1994 zufker ~ ~ ~ ~ ~ ~ ~ ~ ~
`~ ~ ~ -- 379/67
`573037296
`4/1995 Fujisawa et al. (cid:9)
`
`395/604
`5,404,506
`4/1995 FllJlSaWa et al.
`.. 395/604
`5,404,506
`4/1995 Oren et al. (cid:9)
`
`395/601
`5,408,655
`4/1995 Oren et al. ......................... .. 395/601
`5,408,655
`5,414,644 *
`5/1995
`Seaman et al. (cid:9)
`
`702/183
`5,414,644 * 5/1995 Seaman et al. .................... .. 702/183
`5,428,774
`6/1995 Takahashietal. ..
`.. 395/612
`5,428,774
`6/1995
`Takahashi et al. (cid:9)
`
`395/612
`5,461,611
`10/1995 Drake, Jr. et al.
`370/420
`5,461,611
`10/1995 Drake, Jr. et al. (cid:9)
`
`370/420
`5,493,677
`2/1996 Balogh et al. (cid:9)
`
`395/615
`5,493,677
`2/1996 Balogh et al. ..................... .. 395/615
`5,515,490
`5/1996 Buchanan et al.
`345/302
`5,515,490
`5/1996 Buchanan et al. (cid:9)
`
`345/302
`5,608,900
`3/1997 Dockter et al. ..
`395/613
`5,608,900
`3/1997 Dockter et al. (cid:9)
`
`395/613
`5,613,909 * 3/1997 Stelovsky
`463/1
`5,613,909 (cid:9) * 3/1997 Stelovsky (cid:9)
`
`463/1
`3/1997 Liu Sheng et al. (cid:9)
`
`5,615,112
`395/615
`5,615,112
`3/1997 Liu Sheng et a .
`395/615
`5,617,119
`4/1997 Briggs et al. (cid:9)
`
`395/611
`5,617,119
`4/1997 Briggs et al. ......... ..
`395/611
`5,630,121 (cid:9) * 5/1997 Braden-Harder et al. (cid:9)
`
`707/102
`5,630,121 * 5/1997 Braden-Harder et al. ......... .. 707/102
`5,630,125 (cid:9) * 5/1997 Zellweger (cid:9)
`
`707/103
`* 1g;
`gellwegert ~~~~~~~~~~~~~~~~~~~~~
`18?
`5,682,330
`10/1997 Seaman et al. (cid:9)
`
`364/551.01
`5,724,605
`3/1998 Wissner ...... ..
`395/806
`5,724,605
`3/1998
`Wissner (cid:9)
`
`395/806
`5,832,495 *
`11/1998
`Gustman (cid:9)
`
`707/103
`5,832,495 * 11/1998 Gustman
`707/103
`5,867,709 * 2/1999 Lang et al. (cid:9)
`
`707/1
`5,867,709 * 2/1999 Lang et al. ............................. .. 707/1
`_
`_
`* cited by examiner
`* cited by examiner
`Primary Examiner—Wayne Amsbury
`Primary Examiner—Wayne Amsbury
`(74) Attorney, Agent, or Firm—The Hecker Law Group
`(74) Attorney, Agent, or Firm—The Hecker LaW Group
`ABSTRACT
`(57) (cid:9)
`
`,
`
`,
`
`eaman e a .
`
`.
`
`.
`
`(57)
`
`ABSTRACT
`
`The invention catalogues data such as multimedia data. A
`The invention catalogues data such as multimedia data. A
`catalogue is a collection of one or more catalogue elements.
`Catalogue is a Collection of one Or more Catalogue elements
`An index is used to access a catalogue. An element of a
`An index is used to access a catalogue. An element of a
`catalogue has one or more attributes. An attribute provides
`catalogue has one or more attributes. An attribute provides
`information that can be used to search for, answer questions
`information that can be used to search for, ansWer questions
`about, and navigate through a catalogue. An attribute of a
`about, and navigate through a catalogue. An attribute of a
`catalogue element can be an element that has attributes. A
`catalogue element can be an element that has attributes. A
`catalogue element attribute that is an element is referred to
`catalogue element attribute that is an element is referred to
`as an attribute element. Attribute elements and attributes are
`as an attribute element. Attribute elements and attributes are
`used to build an index that can be used to facilitate catalogue
`used to build an index that can be used to facilitate catalogue
`access. Within a catalogue, smaller catalogues can be cre-
`access. Within a catalogue, smaller catalogues can be cre
`ated by, for example, querying and user designation.
`ated by, for eXamplea querying and user designation
`
`18 Claims, 34 Drawing Sheets
`18 Claims, 34 Drawing Sheets
`
`, 204
`
`_,- 204
`
`Segment <47» Segment
`25D
`
`Z25 \
`
`‘ 1—_
`
`Z06
`
`Phrase if
`
`232 \
`
`230 i
`
`214
`
`210
`
`220
`
`114
`
`v 218
`V
`
`1225323
`
`Keyword
`
`[3135:
`
`1112?? Pm“
`
`235
`
`242
`
`\/\ 244
`
`234 \
`
`236
`
`212
`
`v 210
`v /
`
`216
`
`222
`/
`
`$55813‘
`
`W 4112:?‘
`
`Type
`
`Page 1 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Apr. 3, 2001
`Apr. 3, 2001
`
`Sheet 1 0f 34
`Sheet 1 of 34 (cid:9)
`
`US 6,212,527 B1
`US 6,212,527 B1
`
`119
`
`116
`116
`/
`
`VIDEO AMP
`
`CRT
`
`VIIIMIIIIIMIIIIITIMININIMINIIIft
`
`113
`113
`/
`
`114
`/
`
`CPU
`
`VIDEO MEMORY
`VIDEO MEMORY
`
`115
`115
`MAIN MEMORY /
`MAIN MEMORY
`
`111•11111111111MINZIIIIIIMMINIMINI\ (cid:9)
`
`.111111•1111111111MITIIIIIIIINIM
`
`110
`110
`
`KEYBOARD
`KEYBOARD
`
`111
`111 MASS STORAGE
`MASS STORAGE
`MOUSE /
`
`112
`112
`/
`
`FIG. 1
`FIG. 1
`
`Page 2 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Apr. 3, 2001
`Apr. 3, 2001
`
`Sheet 2 0f 34
`Sheet 2 of 34 (cid:9)
`
`US 6,212,527 B1
`US 6,212,527 B1
`
`204
`204
`
`Segment
`
`206
`
`208
`
`206 E 208
`Keyword E 218
`E
`
`Phrase
`
`210 (cid:9)
`210
`
`218
`
`212
`212
`,1
`Thesaural
`Thesaural
`Keyword
`
`220
`220
`1/
`Image/
`Video
`
`214
`214
`I)
`Proposed
`Proposed
`Keyword
`Keyword
`
`222 (cid:9)
`222
`
`Thesaural
`Person
`
`216
`216
`
`224
`224
`/
`proposed
`Person
`
`226
`226
`/ 2
`
`Person
`Information
`
`FIG. 2A
`FIG. 2A
`
`Page 3 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Apr. 3, 2001
`Apr. 3, 2001
`
`Sheet 3 of 34 (cid:9)
`Sheet 3 0f 34
`
`US 6,212,527 B1
`US 6,212,527 B1
`
`204
`204
`
`Segment
`Segment
`
`232 \
`
`Segment
`
`228
`
`T
`
`204
`
`250
`
`206
`
`230 i
`
`214
`214
`
`210
`210
`/
`
`220
`220
`1
`
`224
`224
`
`218
`21s
`
`Proposed
`Proposed
`Keyword
`Keyword
`
`Keyword
`
`Image/
`Video
`
`Proposed
`Proposed
`Person
`Person
`
`Person
`
`238
`
`I 212
`
`Thesaural
`Keyword
`
`'mr1.7 (cid:9)
`
`
`242 .,
`
`.....,,'N 244
`244
`
`234 -\
`
`y 210
`V /
`
`216
`
`Keyword
`
`Type
`
`Thesaural
`Thesaural
`Person
`Person
`
`Person
`Information
`
`246 -4
`
`216
`
`Type
`Type
`
`226
`
`FIG. 2B
`FIG. 2B
`
`Page 4 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Apr. 3, 2001
`Apr. 3, 2001
`
`Sheet 4 0f 34
`Sheet 4 of 34 (cid:9)
`
`US 6,212,527 B1
`US 6,212,527 B1
`
`280 (cid:9)
`280
`/
`/ (cid:9)
`
`S1 (cid:9)
`
`282 (cid:9)
`282
`/
`/ (cid:9)
`
`S2
`
`258 (cid:9)
`258
`
`I
`I (cid:9)
`TO P1 T1 P2 T2
`I
`I (cid:9)
`l
`l
`|
`
`/ (cid:9)
`260
`260 (cid:9)
`
`/
`/
`262
`262
`
`P3
`
`/
`/
`264
`264
`
`T3 (cid:9)
`I (cid:9)
`|
`
`I
`P4
` T4
`I
`|
`
`.
`l
`/
`266
`266
`
`P5 T5
`I
`.
`l
`/
`l
`268
`268
`
`P6
`.
`/
`l
`270
`270
`
`|
`
`l
`
`l
`272
`
`S3
`
`• • •
`
`284
`284
`l
`/
`
`Pn Tn
`|
`
`I
`Tn-
`1
`1
`i
`l
`274
`274
`
`Chapter Segment = (cid:9)
`Chapter Segment =
`
`Pl+P2 +P3
`P1 + P2 + P3
`P4 + P5
`P4 + P5
`P6 +P7 +...+Pn
`
`Testimony Segment = S1 + S2 + S3
`Testimony Segment = 51 + S2 + S3
`
`General Segment = P1 + P4 + P5 + P6 + Pn
`General Segment = P1 + F4 + P5 + P6 + Pn
`
`FIG. 2C
`FIG. 2C
`
`Page 5 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Apr. 3, 2001
`Apr. 3, 2001
`
`Sheet 5 of 34 (cid:9)
`Sheet 5 of 34
`
`US 6,212,527 B1
`US 6,212,527 B1
`
`Segment
`mm
`ID
`ID
`descriptive phrase
`descriptive phrase
`set(phrases)
`set(phrases)
`
`204
`’
`
`Phrase
`in at data ID
`input data ID
`,1;
`ID
`descriptive phrase
`descriptive phrase
`In/Out Timecodes
`In/Out Timecodes
`SBKWUOMS)
`set(keywords)
`set (images)
`sef(images)
`set(persons)
`set(persons)
`set(proposed keywords)
`set(proposed keywords)
`set(proposed persons)
`set(proposed persons)
`
`206
`206
`208
`' M ,208
`Event
`ID
`ID
`type
`type
`sub-type
`sub-type
`In/Out Timecodes
`In/Out Timecodes
`comment
`comment
`
`210
`210
`/
`
`Keyword
`Kauai!
`ID
`ID
`set( types)
`set(types)
`setIthesaural keywords)
`set(thesaural keywords)
`set(parent keywords)
`set(parent keywords)
`set(child keywords)
`set(child keywords)
`set(proposed keywords)
`set(proposed keywords)
`
`Type
`lune
`ID
`I D
`IAbel
`Label
`
`218
`218
`
`216
`216
`’
`
`Person
`Balm
`ID
`ID
`Primary Name
`Primary Name
`Occupation
`Occupation
`Date of Birth
`Date of Birth
`set(prop0sed persons)
`set(proposed persons)
`
`Thesaural Person
`l|':.: " '1
`ID
`ID
`Name
`Name
`Preferred Flag
`Preferred Flag
`
`222
`222
`/
`
`Thesaural Keyword
`...
`ID
`ID
`keyword ID
`keyword ID
`Label
`Label
`Language of Choice
`Language of Choice
`Preferred Flag
`Preferred Flag
`Characteristic
`Characteristic
`
`212
`l 212
`
`224
` 224
`
`214
`214
`/ {
`Proposed Person
`PrgpgsgdPersgn
`ID
`ID
`Name
`Name
`
`‘III 't\
`
`I!
`
`ID
`Label
`
`Image I Video
`1'5."
`5"
`ID
`ID
`In/Out Timecodes
`III/011i Tl'm?wdes
`Locator
`Locator
`
`220
`220 (cid:9)
`k
`
`Person Information
`" '1 Ill“! '1
`Person ID
`Person ID
`Person Attribute
`Person Attribute
`
`226
` 226
`/
`
`FIG. 2D
`FIG. ZD
`
`Page 6 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Apr. 3, 2001
`Apr. 3, 2001
`
`Sheet 6 of 34 (cid:9)
`Sheet 6 0f 34
`
`US 6,212,527 B1
`US 6,212,527 B1
`
`286 N. (cid:9)
`Thesaural
`286 \ Thesaural
`Keyword One
`Keyword One
`
`Thesaural
`Thesaural
`Keyword Two
`Keyword Two
`
`Thesaural
`Thesaural
`Keyword Three
`Keyword Three
`
`Thesaural
`Thesaural
`Keyword Four
`Keyword Four
`
`288 (cid:9)
`288
`\
`
`290D
`29013
`290C
`290C
`290B
`2903
`Keyword 290A
`Keyword 290A
`Instance
`Instance
`One
`
`291E
`291E
`291D
`291D
`291C
`2918
`291B
`291A
`
`292B
`292A
`
`293B
`293B
`293A
`293A
`
`one
`
`294B
`
`294B (cid:9)
`294A
`294A (cid:9)
`
`Keyword
`Keyword
`Instance
`Instance
`Two
`Two
`
`2955
`
`295B
`295A
`295A
`
`296C
`296B
`296B
`296A
`296A
`
`
`2971 3$
`2973
`.
`297A
`297A
`
`FIG. 2E
`FIG. 2B
`
`Page 7 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Apr. 3, 2001
`Apr. 3, 2001
`
`Sheet 7 of 34 (cid:9)
`Sheet 7 0f 34
`
`US 6,212,527 B1
`US 6,212,527 B1
`
`302
`302
`
`F
`r
`
`308
`308
`/
`
`301
` 301
`
`304
`304
`.1/ (cid:9)
`“{[Questkmir
`Questions
`w-303
`+ QA f
` QA
`306
`306
`
`-
`
`303
`
`1'
`310
`320
`318
`310 320
`318
`/
`/
`[Delete/Mergejl Segment I
`Segment I
`Delete/Merge
`
`300
`300
`j
`I Save Phrase |
`Save Phrase I
`
`F
`312
`312
`
`7'
`314
`314
`
`305
`305
`
`Exit
`
`316 /
`316
`
`FIG. 3A
`FIG. 3A
`
`Page 8 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Apr. 3, 2001
`Apr. 3, 2001
`
`Sheet 8 0f 34
`Sheet 8 of 34 (cid:9)
`
`US 6,212,527 B1
`US 6,212,527 B1
`
`322 (cid:9)
`322
`
`.. .
`
`304A
`
`334A (cid:9)
`334A
`
`
`327 (cid:9)
`327
`
`\
`
`\
`
`I
`
`334N
`334M
`\
`X
`
`3
`
`304
`304
`/
`
`322 /
`322 / (cid:9)
`
`323 /
`323 / (cid:9)
`
`324 /
`324
`
`325 (cid:9)
`325 /
`
`326 / (cid:9)
`326 /
`
`327 (cid:9) /
`327 /
`
`306
`306
`/
`Z_
`
`FIG. 3B
`FIG. 3B
`
`Page 9 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Apr. 3, 2001
`Apr. 3, 2001
`
`Sheet 9 of 34 (cid:9)
`Sheet 9 0f 34
`
`US 6,212,527 B1
`US 6,212,527 B1
`
`340
`340
`1
`/
`
`310
`310
`/
`
`n (cid:9)
`El
`
`D,
`9
`
`342
`342
`/
`/1
`
`| |
`|
`
`. ,........ eeee
`
`__144._. (cid:9)
`44 (cid:9)
`?g <4 m
`r. (cid:9)
`
`_.1:401..._._ (cid:9)
`
`Dbl
`
`ODD
`» ODD
`.<.m00 _
`--..---...t
`—
`
`AO. (cid:9)
`Q
`—
`......--„,. (cid:9)
`444„. (cid:9)
`[ : (cid:9)
`444
`
`344
`Still Frame
`Still Frame / 344
`
`Go to Phrase i 346
`Go to Phrase / 346
`
`Go to Last / 348
`Go to Last
`348
`
`FIG. 3C
`FIG. 3C
`
`Page 10 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`
`Apr. 3, 2001
`
`Sheet 10 of 34 (cid:9)
`
`US 6,212,527 B1
`
`350B
`
`350A
`
`314
`
`A
`—
`=
`
`354
`
`a
`b
`c
`d
`e
`f
`g
`h
`i
`
`350C
`
`k
`1
`m
`n
`o
`P
`q
`r
`s
`t
`u
`v
`w
`X
`Y
`z
`
`352
`
`354A
`
`354B
`
`354C
`
`354D
`
`354E
`
`354F
`
`Keyword Alphabeticl
`
`Keyword
`
`Type
`
`\1/4 war crimes (cid:9)
`
`Chasidim (cid:9)
`
`Hasidic Jews (cid:9)
`
`'• Germany (cid:9)
`
`l
`
`• (cid:9)
`k...
`
`world (cid:9)
`
`World War 11 (cid:9)
`
`356A --t
`• war crimes
`356B N
`• Jews
`356C N
`\ Jewish religion
`356D -,
`\ country
`356E -,
`•• place
`
`356F
`
`\• war
`
`350E
`
`350D
`/1
`
`< "
`
`> V
`
`FIG. 3D
`
`Page 11 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Apr. 3, 2001
`Apr. 3, 2001
`
`Sheet 11 0f 34
`Sheet 11 of 34 (cid:9)
`
`US 6,212,527 B1
`US 6,212,527 B1
`
`316
`
`364
`366
`360
`362
`368
`/ (cid:9)
`/ / / (cid:9)
`/ (cid:9)
`Segment/Notes (cid:9)
`Keywords (cid:9)
`Proposed
`Start (cid:9)
`End (cid:9)
`372A
`WaratIONAMMIMSAMMMSKOMENNERVO. .AWSNAMMMOSSMOIV
`A
`374A
`~ 1 Born December 5, 1924, in
`- \-- 1 Born December 5, 1924, in Lodz... 01:00:00:00 01:08:48:21 brothers *homess'sisters dating life
`01:00:00z00 01:08:48z21 brothers "homes'sisters dating life
`
`f -
`
`.. y
`
`370
`/
`
`.
`
`teachers * child
`teachers ‘ child
`
`V
`
`374B
`"\- (cid:9)
`'~
`374C
`-..\- 3
`
`2 Family was happy. Parents were... 01:08:48:21 01:12:49:15 behavior ' students (cid:9)
`2 Family was happy. Parents were... 01:08:48:21 01:12:49t15 behavior ‘ students
`
`372B
`
`' \ gslINCSOrtgslia.
`
`<1. I
`
`FIG. 3E
`FIG. 3B
`
`Page 12 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Apr. 3, 2001
`Apr. 3, 2001
`
`Sheet 12 0f 34
`Sheet 12 of 34 (cid:9)
`
`US 6,212,527 B1
`US 6,212,527 B1
`
`308
`308
`
`382
`
`384
`
`386
`
`388
`
`Interviewer
`
`Videographer
`
`Fact
`
` Other
`
`FIG. 3F
`FIG. 3F
`
`Page 13 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Apr. 3, 2001
`Apr. 3, 2001
`
`Sheet 13 0f 34
`Sheet 13 of 34 (cid:9)
`
`US 6,212,527 B1
`US 6,212,527 B1
`
`402
`402
`
`\\
`
`404
`
`.
`
`.
`
`4°6
`404 Keyword from Hilite I New Keyword
`406
`New 1:1’
`Proposed Item
`Proposed Item
`Keywords for this Phrase I Type
`Keywords for this Phrase Type
`
`P
`408 /
`408
`
`< I ‘"I
`gg
`< (cid:9)
`Proposed Keyword
`Proposed Keyword
`
`Comment
`Comment
`
`i
`410 /
`410
`
`<I "I
`< (cid:9)
`oo
`Word Used
`Word Used
`
`I Preferred Term
`I Preferred Term
`
`e
`412 /
`412
`
`< I 00I
`04
`< (cid:9)
`
`A
`A
`
`i
`at
`
`|>
`>
`
`< 00
`
`__
`V <IWI
`V
`I Keyword Hierarchy I
`Keyword Hierarchy
`A
`A Keyword
`Keyword
`et —
`i
`
`I
`
`—
`V
`
`I>
`
`>
`
`A
`A.
`es
`.1
`
`V
`V
`
`>
`>
`
`A
`1A
`V
`T
`418
`418
`1
`c
`/
`A
`—Q—
`Et
`_
`
`V
`T
`l>
`
`>
`
`4/16
`416
`'I
`
`I Type
`Type
`
`A
`A
`
`I
`Et
`
`v
`V
`
`<I WI
`< (cid:9)
`CKI
`
`|
`I
`
`I >
`>
`
`414
`414
`
`FIG. 4
`FIG. 4
`
`Page 14 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Apr. 3, 2001
`Apr. 3, 2001
`
`Sheet 14 of 34 (cid:9)
`Sheet 14 0f 34
`
`US 6,212,527 B1
`US 6,212,527 B1
`
`502
`502
`./
`
`522
`522 \
`
`518 (cid:9)
`520
`516
`518
`520
`516
`/
`_/ /
`/ '1
`i_1_ Did the interviewer appear on camera @ Yes
`(L) Yes
`Did the interviewer appear on camera
`512A
`and properly introduce him/herself,
`and properly introduce him/herself,
`512A
`\ the interviewee, and the interview? 0 No
`the interviewee, and the interview?
`O N°
`A
`A
`I
`V
`-—1 V
`5 8 520
`516
`51B
`520
`516
`/
`/ /
`L2. Did the interviewer ask open-ended
`0 Yes
`Did the interviewer ask open-ended
`questions, allowing the interview to
`uestions, allowin the interview to
`51213
`5128 \ tfllow?
`g
`0 No
`flow?
`O No
`A
`L
`
`522 \
`522
`
`‘
`
`a
`
`_
`LI.
`I ,,,,,,,,,,,,,
`A [II
`b
`_
`§
`i5-
`4414 Ott ac l
`Den CO 1000
`Q“ Q4 m M is » DD’
`m“
`y`en
`
`| |
`
`I segmemsl [ "Q I P Q" i
`
`Segments
`
`532
`
`I
`534
`
`l
`536
`
`506
`Still Frame l/ 506
`Still Frame /
`/ 508 (cid:9)
`Golo Phrasel/ 508
`Go to Phrase
`
`6° ‘oust
`Go to Last
`‘
`534
`504
`
`V
`T
`518 520
`516 (cid:9)
`516
`518
`520
`/
`
`/ /
`510 3
`510 12_ (cid:9)
`'
`Q Did the interviewer attempt to clarify 0 Yes
`Did the interviewer attempt to clarify
`0 Yes
`details without disrupting the
`512C
`details without disrupting the
`512C
`\ interview?
`@ No
`O No
`interview?
`522 \
`522
`
`.
`
`A
`
`V
`
`FIG. 5
`FIG. 5
`
`Page 15 of 49
`
`MINDGEEK EXHIBIT 1001
`
`

`

`U.S. Patent (cid:9)
`
`Apr. 3, 2001 (cid:9)
`
`Sheet 15 of 34 (cid:9)
`
`US 6,212,527 B1
`
`(cid:9) 602
`
`(cid:9)/. 604
`
`610
`
`620
`
`TC_In +-?
`
`•
`Interviewer
`
`TC_In +-?
`
`•
`Videographer
`
`A
`—
`
`\
`624
`
`\
`626
`
`\
`622
`
`A—
`
`
`
`\
`612
`
`\
`614
`
`\
`616
`
`< I
`
`V
`
`I>
`630
`
`<
`
`Ti
`
`>
`640
`
`TC_In +-?
`
`Factual
`
`TC_In +-?
`
`Other
`
`‘1
`632
`
`634
`
`\
`636
`
`\
`642
`
`\
`644
`
`\
`646
`
`A
`—
`'..•
`
`V
`
`A
`—
`,T'..
`
`V
`
`>
`
`606
`N
`
`A
`
`V
`
`656
`
`<
`
`>
`
`<
`
`FIG. 6
`
`Page 16 of 49
`
`MINDGEEK EXHIBIT 1001
`
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Apr. 3, 2001
`Apr. 3, 2001
`
`Sheet 16 of 34 (cid:9)
`Sheet 16 0f 34
`
`US 6,212,527 B1
`US 6,212,527 B1
`
`704
`704
`\
`
`Unclassified Item
`Unclassified ltem
`714
`A
`Jewish nonviolent resistance
`A
`714
`Jewish nonviolent resistance
`Nazi courts
`a @ N311 Col-"t5
`occupations
`Occupations
`personal property
`personal property
`
`....._
`
`742
`L 742
`
`702
`702
`\‘
`
`Item
`Item (cid:9)
`actions -/
`744
`actions
`,— 744
`antisemitic actions
`antisemitic actions /
`746 (cid:9)
`anti-Jewish measures and legislation
`746 \ anti-Jewish measures and legislation
`antisemitic comments and behavior
`antisemitic comments and behavior
`attacks on Jewish homes
`attacks on Jewish homes
`attacks on synagogues
`attacks on synagogues
`desecration of cemetaries
`desecration of cemetaries
`bystander responses to Holocaust
`bystander responses to l-lollocaust
`744 •••... deadly actions
`744 \ deadly actions
`Nazi government actions
`Nazi government actions
`violent actions
`violent actions
`amenities (cid:9)
`./ 742
`amenities
`742
`camp personnel /
`camp personnel
`character and behavior
`character and behavior
`
` 752
`de
`/' 752
`
`A
`A
`
`is
`a
`_
`
`_
`
`734 -...,,..
`734 \
`
`V
`V
`
`>
`|>
`
`I
`
`.
`
`732
`732
`
`@
`
`716
`716
`
`Delete
`
`718
`718
`
`Declassify
`
`720
`720
`
`Find `
`
`722
`712
`
`734
`
`\“ GE)
`< pp (cid:9)
`|>
`< WI
`
`V
`V
`
`<|w|
`90
`
`/.
`732
`732
`
`FIG. 7A
`FIG. 7A
`
`Page 17 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Apr. 3, 2001
`Apr. 3, 2001
`
`Sheet 17 0f 34
`Sheet 17 of 34 (cid:9)
`
`US 6,212,527 B1
`US 6,212,527 B1
`
`Item
`Item..772
`/ 772
`
`actions /
`actions
`ghetto life (cid:9)
`ghetto life
`air raids
`air raids
`774\ anti‘Nazi demonstrations
`anti-Nazi demonstrations
`774 (cid:9)
`apartments in teh ghettos
`apartments in teh ghettos
`Aryan side
`Aryan side
`bartering
`bartering
`beggars in ghettos
`beggars in ghettos
`bunkers in ghettos
`bunkers in ghettos
`curfew in the ghettos
`curfew in the ghettos
`
`cto
`
`732
`732
`
`702
`702
`\\
`
`A
`A
`sa
`a
`—
`
`704
`704
`\
`
`Unclassified Item
`Unclassified Item
`
`Nizoz / 782
`Nizoz
`, 782
`
`714
`714
`
`716
`
`718
`718
`
`A
`A
`se
`a
`__
`
`734 -,s..
`734 \
`
`716 IE Declassify
`
`720
`
`720 @
`
`Find
`
`722
`722
`
`734
`734 \
`
`I.
`
`V
`
`Find ID
`
`< 99
`
`<1
`V
`
`>
`
`/
`
`732
`732
`
`FIG. 7B
`FIG. 7B
`
`Page 18 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Apr. 3, 2001
`Apr. 3, 2001
`
`Sheet 18 0f 34
`Sheet 18 of 34 (cid:9)
`
`US 6,212,527 B1
`US 6,212,527 B1
`
`No
`
`804
`
`806
`806
`
`Yes@
`
`810
`
`Yes~> create segment
`
`8
`
`8
`
`a E3 93 (D rh O
`/
`
`Yes—>
`
`create event
`
`Yes
`Yes —> create image
`create image
`
`Yes
`Yes
`
`finished?
`
`N 0
`
`N 0
`
`C1’ Bate
`event?
`
`N 0
`
`create
`image?
`
`N 0
`
`FIG. 8A
`F.G. 8A
`
`Page 19 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`U.S. Patent
`
`Apr. 3, 2001
`Apr. 3, 2001
`
`Sheet 19 of 34 (cid:9)
`Sheet 19 0f 34
`
`US 6,212,527 B1
`US 6,212,527 B1
`
`820
`820
`
`create
`thesaural
`item?
`
`824
`
`/
`
`te
`Yes
`Yes
`Crea .
`+ thesaural ltem
`
`828
`,
`
`Yes-P
`Yes
`
`create
`Create.
`proposed item
`proposed 1tem
`
`create
`proposed
`item?
`
`No
`
`828
`
`delete/ merge?
`
`Yes
`Yes —>
`
`delete/merge
`delete / mer 2
`.
`g
`instance
`lnstance
`
`<5; <5 5 5
`
`830
`,
`I
`
`create phrase?
`
`832
`
`Yes (cid:9)
`Yes —->
`—11
`
`create phrase
`create phrase
`
`834
`,
`’
`
`FIG. 8B
`FIG. 8B
`
`Page 20 of 49
`
`MINDGEEK EXHIBIT 1001
`
`

`

`U.S. Patent (cid:9)
`
`Apr. 3, 2001
`
`Sheet 20 of 34 (cid:9)
`
`US 6,212,527 B1
`
`start
`
`V (cid:9)
`
`902
`z
`
`assign segment ID
`
`904
`,
`
`obtain
`descriptive phrase
`
`create segment
`instance
`
`906
`
`908
`,
`
`create element
`relationships
`
`910
`
`end 3
`
`FIG. 9A
`
`Page 21 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`
`Apr. 3, 2001
`
`Sheet 21 of 34 (cid:9)
`
`US 6,212,527 B1
`
`start
`
`assign ID
`
`922
`
`924
`z
`
`¤
`obtain start and
`end timecodes
`
`926
`z
`
`obtain
`descriptive phrase
`
`928
`r
`
`¤
`spell check
`descriptive phrase
`
`930
`z
`identify items in
`descriptive phrase
`
`¤
`
`932
`z
`
`create instance
`
`934
`r
`
`create element
`relationships
`
`936
`
`( end
`
`FIG. 9B
`
`Page 22 of 49
`
`MINDGEEK EXHIBIT 1001
`
`

`

`U.S. Patent (cid:9)
`
`Apr. 3, 2001
`
`Sheet 22 of 34 (cid:9)
`
`US 6,212,527 B1
`
`start
`
`1004
`
`1006
`
`locate instance
`of index element
`
`obtain thesaural
`term
`
`1008
`./
`
`create thesaural
`instance
`
`1010
`z
`
`create element
`relationships
`
`( end )
`
`FIG. 10
`
`Page 23 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`
`Apr. 3, 2001
`
`Sheet 23 of 34 (cid:9)
`
`US 6,212,527 B1
`
`start
`
`1102
`z
`
`¤
`identify
`proposed item
`
`¤
`
`1104
`z
`
`enter comment
`
`1106
`
`create proposed
`element instance
`
`1108
`z
`
`create element
`relationships
`
`1110
`
`(cid:9))
`
`end
`
`FIG. 11
`
`Page 24 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`
`Apr. 3, 2001
`
`Sheet 24 of 34 (cid:9)
`
`US 6,212,527 B1
`
`start
`
`obtain index
`element to be
`deleted
`
`1202
`,
`
`1204
`,
`
`locate related
`index elements
`
`1206
`
`Yes
`lir
`reassign/delete
`related elements
`
`1208
`z
`
`1212
`z
`
`No¤
`
`raise error
`
`Yes
`
`1214
`
`delete instance
`
`.4 (cid:9)
`
`C. end
`
`1216
`/
`-1
`
`FIG. 12
`
`Page 25 of 49
`
`MINDGEEK EXHIBIT 1001
`
`

`

`U.S. Patent (cid:9)
`
`Apr. 3, 2001
`
`Sheet 25 of 34 (cid:9)
`
`US 6,212,527 B1
`
`start
`
`1302
`z
`
`identify type
`
`¤
`
`1304
`
`identify sub-type
`
`¤
`
`1306
`
`obtain timecode
`
`1308
`
`obtain comment
`
`¤
`
`1310
`
`assign ID
`
`¤
`
`312
`
`create instance
`
`1314
`
`create element
`relationships
`
`1316
`
`end
`
`FIG. 13A
`
`Page 26 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`
`Apr. 3, 2001
`
`Sheet 26 of 34 (cid:9)
`
`US 6,212,527 B1
`
`start
`
`assign ID
`
`1322
`
`V (cid:9)
`
`1324
`z
`
`obtain timecode
`
`1326
`A'
`
`V (cid:9)
`obtain resource
`locator
`
`create instance
`
`1328
`,
`
`1330
`z
`
`create element
`relationships
`
`1332
`
`( end
`
`FIG. 13B
`
`Page 27 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`
`Apr. 3, 2001
`
`Sheet 27 of 34 (cid:9)
`
`US 6,212,527 B1
`
`No
`
`Yes
`
`Yes
`
`1410
`
`FIG. 14A
`
`Page 28 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`
`Apr. 3, 2001
`
`Sheet 28 of 34 (cid:9)
`
`US 6,212,527 B1
`
`1
`
`No
`
`Yes
`
`1414
`,
`
`delete children
`
`¤
`
`1416
`z
`
`delete item
`
`1418
`z
`
`¤
`update
`managment screens
`
`.4 (cid:9)
`
`end
`
`1420
`i
`
`FIG. 14B
`
`Page 29 of 49
`
`MINDGEEK EXHIBIT 1001
`
`

`

`U.S. Patent (cid:9)
`
`Apr. 3, 2001
`
`Sheet 29 of 34 (cid:9)
`
`US 6,212,527 B1
`
`1504
`
`Yes
`
`1506
`
`get next child
`instance
`
`1510
`z
`
`identify any
`children of instance
`
`FIG. 15A
`
`Page 30 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`
`Apr. 3, 2001
`
`Sheet 30 of 34 (cid:9)
`
`US 6,212,527 B1
`
`No
`
`Yes
`
`1514
`z
`
`delete children
`
`V (cid:9)
`
`1516
`z
`
`delete instance
`
`C)
`
`FIG. 15B
`
`Page 31 of 49
`
`MINDGEEK EXHIBIT 1001
`
`

`

`U.S. Patent (cid:9)
`
`Apr. 3, 2001
`
`Sheet 31 of 34 (cid:9)
`
`US 6,212,527 B1
`
`1604
`,
`
`raise error
`
`1606
`A,
`
`1608
`z
`
`find parent
`
`remove child
`from parent
`
`1606
`,
`
`V (cid:9)
`insert item in
`unclassified list
`
`4 (cid:9)
`
`1608
`I
`end )
`
`FIG. 16
`
`Page 32 of 49
`
`MINDGEEK EXHIBIT 1001
`
`

`

`U.S. Patent (cid:9)
`
`Apr. 3, 2001
`
`Sheet 32 of 34 (cid:9)
`
`US 6,212,527 B1
`
`start
`
`1702
`
`dragged item
`from classified
`list?
`
`Yes
`
`No
`
`1712
`
`get ID of old parent
`
`1714
`z
`
`get ID of new parent
`
`1716
`
`update inheritance
`
`.4 (cid:9)
`
`1718
`/
`end )
`
`1706
`z
`
`add to inheritance
`
`1710
`/
`
`raise error
`
`FIG. 17
`
`Page 33 of 49
`
`MINDGEEK EXHIBIT 1001
`
`

`

`U.S. Patent (cid:9)
`
`Apr. 3, 2001
`
`Sheet 33 of 34 (cid:9)
`
`US 6,212,527 B1
`
`1802A
`
`1802B
`
`1802C
`
`1804A
`
`Catalogue A
`
`180413
`
`Catalogue fl
`
`1804C
`
`Catalogue C
`
`FIG. 18A
`
`Page 34 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`
`Apr. 3, 2001
`
`Sheet 34 of 34 (cid:9)
`
`US 6,212,527 B1
`
`1814
`
`Catalogue
`
`1812
`
`Keyword
`and Type
`Hierarchies
`
`/
`Keyword
`and Type
`Hierarchies
`/
`1818B
`
`1818A
`
`Keyword
`and Type
`Hierarchies
`
`1818C
`
`FIG. 18B
`
`Page 35 of 49
`
`MINDGEEK EXHIBIT 1001
`
`(cid:9)
`

`

`US 6,212,527 B1
`
`1
`METHOD AND APPARATUS FOR
`CATALOGUING MULTIMEDIA DATA
`
`This is a continuation of application Ser. No. 08/680,504,
`filed Jul. 8, 1996, now U.S. Pat. No. 5,832,495.
`
`BACKGROUND OF THE INVENTION
`1. Field of the Invention
`This invention relates to the cataloguing multimedia data
`and including storage and retrieval mechanisms.
`2. Background
`Increasingly, computer systems are being used to present
`multimedia material. Such material is usually in the form of
`text, graphics, video, animation, and sound. Two or more of
`these data types are usually combined to form the multime-
`dia data presented by the computer system. A computer
`system that is used to present multimedia material is called
`a multimedia system. A problem with prior art multimedia
`systems is an inability to search and retrieve multimedia
`data.
`One prior art multimedia system uses a disk operating
`system that includes a file system for storing and retrieving
`files containing multimedia data. The file system catalogues
`the files based on the names given to the files. The file
`system can be used to retrieve a file that contains multimedia
`data based on the file's name. Other than the extent to which
`the file name identifies content, the file system does not
`provide the ability to retrieve multimedia information based
`on the content of the data. The search system provided by a
`file system is therefore inadequate to search with greater
`detail than that provided in a file name. A number of prior
`art multimedia systems are described below.
`A search system is described in U.S. Pat. No. 5,241,671,
`Reed et al., issued on Aug. 31, 1993 relates to a multimedia
`system that includes a database that is comprised of words,
`phrases, numbers, letters, maps, charts, pictures, moving
`images, animations, and audio information. A search capa-
`bility is provided that provides a series of entry paths for
`locating information in the database. An entry path allows a
`user to enter a search request that consists of a set of valid
`terms or stop terms. A stop term is a term that exists on a stop
`term list and may be the words "the" or "a", for example.
`Valid terms are linked to related terms using a stem index.
`A stem index contains a root term and a set of stems for each
`term that is related to the root word. For example, the word
`leaf is linked to terms "leaves" and "leafing".
`The creating and displaying of navigators for locating and
`accessing visual/image information is described in U.S. Pat.
`No. 5,123,088, Kasahara et al., issued on Jun. 16, 1992.
`Image information is categorized and linked in a circular list
`and ordered according to their attributes. When an image
`unit is displayed, the linked image units can be displayed as
`reduced images, or navigators. A user can navigate through
`a network of linked image units by selecting the desired
`navigator.
`A system for database retrieval wherein entries in differ-
`ent databases are retrieved by a process of matching key
`words of the databases is described in U.S. Pat. No. 5,210,
`868, Shimada et al., issued on May 11, 1993. Examples of
`two such databases are a mapping database and a customer
`attribute database. A dictionary is used to separate a keyword
`from a first database into common and proper noun subparts.
`Common and proper noun synonyms are inferred according
`to a set of rules. The synonyms are combined using a
`combination rule and then compared with keywords in a
`second database to generate a final matching result.
`
`5 (cid:9)
`
`2
`A system for handling multimedia using entity and rela-
`tion objects is described in U.S. Pat. No. 5,278,946, Shimada
`et al. issued on Jan. 11, 1994. An entity object defines
`methods and properties for entities such as a building, road,
`railroad, and boundary. A relation object defines method and
`properties for relationships between entities. A user model
`and system model can be coupled to generate a digest of
`multimedia data.
`A system for storing and retrieving digital images is
`10 described in U.S. Pat. No. 5,493,677, Balogh et al., issued on
`Feb. 20, 1996. A caption or other metadata can be associated
`with a digital image. A natural language capability removes
`ambiguities from the metadata input by a user prior to its
`storage. The natural language capability determines matches
`15 between a user query and the stored metadata. The system
`allows a user to select an image, review licensing terms for
`the selected image, and order the image.
`A repetitive analysis event system that accesses data using
`a time-based number is described in U.S. Pat. No. 5,414,644,
`20 Seaman et al., issued on May 9, 1995. The system uses an
`information library that consists of visual data storage and a
`textual database for storing written descriptions of the visual
`data and a glossary of keywords that identify repetitive
`events or behavior. A behavioral label is used to define a
`25 behavioral activity. A series of images or video clips are
`associated with the behavioral label. A user can retrieve
`images by identifying a subject, a behavioral activity, or
`other type of descriptive text. A chronological timeline is
`used to control the order in which the images are displayed.
`30 That is, the images are displayed in sequential order using
`the time-based number.
`A knowledge based information retrieval system is
`described in U.S. Pat. No. 5,404,506, Fujisawa et al., issued
`on Apr. 4, 1995. The system provides a visual interface for
`35 local searching and a natural language interpreter for global
`search. The natural language interpreter is used to infer the
`meaning of a noun phrase or a nominal phrase. The inferred
`meaning is used to retrieve information.
`The search capabilities in the patents identified above do
`not