(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2002/0069218A1
`(43) Pub. Date:
`Jun. 6, 2002
`Sull et al.
`
`US 2002006921.8A1
`
`(54) SYSTEM AND METHOD FOR INDEXING,
`SEARCHING, IDENTIFYING, AND EDITING
`PORTIONS OF ELECTRONIC MULTIMEDIA
`FILES
`(76) Inventors: Sanghoon Sull, Seoul (KR);
`Hyeokman Kim, Seoul (KR);
`Hyungseok Choi, Seoul (KR); Min
`Gyo Chung, Sungnam City (KR);
`Ja-Cheon Yoon, Seoul (KR);
`Jeongtaek Oh, Seoul (KR); Sangwook
`Lee, Seoul (KR); S. Moon-Ho Song,
`Seoul (KR); Jung Rim Kim, Seoul
`(KR); Keansub Lee, Suwon City (KR);
`Seong Soo Chun, Songnam City (KR);
`Sangwook Oh, Cheju City (KR);
`Yunam Kim, Cheju City (KR)
`Correspondence Address:
`Ronald L. Chichester
`Baker Botts L.L.P.
`One Shell Plaza
`910 Louisiana Street
`Houston, TX 77002-4995 (US)
`Appl. No.:
`09/911,293
`
`Filed:
`
`Jul. 23, 2001
`Related U.S. Application Data
`(63) Non-provisional of provisional application No.
`60/221,394, filed on Jul. 24, 2000. Non-provisional of
`
`(21)
`(22)
`
`provisional application No. 60/221,843, filed on Jul.
`28, 2000. Non-provisional of provisional application
`No. 60/222,373, filed on Jul. 31, 2000. Non-provi
`sional of provisional application No. 60/271,908,
`filed on Feb. 27, 2001. Non-provisional of provisional
`application No. 60/291,728, filed on May 17, 2001.
`
`Publication Classification
`
`(51) Int. Cl." ..................................................... G06F 15/00
`(52) U.S. Cl. .......................................................... 707/501.1
`
`(57)
`
`ABSTRACT
`
`A method and System are provided for tagging, indexing,
`Searching, retrieving, manipulating, and editing Video
`images on a wide area network Such as the Internet. A first
`Set of methods is provided for enabling users to add book
`marks to multimedia files, Such as movies, and audio files,
`Such as music. The multimedia bookmark facilitates the
`Searching of portions or Segments of multimedia files, par
`ticularly when used in conjunction with a Search engine.
`Additional methods are provided that reformat a video
`image for use on a variety of devices that have a wide range
`of resolutions by Selecting Some material (in the case of
`Smaller resolutions) or more material (in the case of larger
`resolutions) from the same multimedia file. Still more meth
`ods are provided for interrogating images that contain tex
`tual information (in graphical form) So that the text may be
`copied to a tag or bookmark that can itself be indexed and
`Searched to facilitate later retrieval via a Search engine.
`
`. . .
`. . . . .
`.
`.
`.
`.
`.
`.
`Options Directory
`Edit Go Bookmarks
`File
`Location:
`
`.
`
`.
`
`- 202
`
`. -IX
`
`.
`
`-----
`
`
`
`208
`
`List of wfaltimedia Bookmarks
`222
`24
`H-226
`
`Positional
`Information
`
`Bositional
`information
`
`intell
`information
`l
`
`Citcut
`inforcuation
`
`^ 200
`
`-1-
`
`Amazon v. Audio Pod
`US Patent 9,319,720
`Amazon EX-1005
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 1 of 65
`
`US 2002/0069218A1
`
`
`
`Go Bookmarks Options Directory
`
`Multimedia Content Presentation
`
`List of Conventional Bookmarks
`
`Positional
`information
`
`Title
`
`Figure I
`(Prior Art)
`
`-2-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 2 of 65
`
`US 2002/0069218A1
`
`Edit Go Bookmarks Options Directory
`File
`Location:
`
`202
`Link
`
`204
`
`2O6
`
`
`
`List of Multimedia Bookmarks
`
`208
`
`Positional
`Information
`
`Content
`Information
`
`Content
`Information
`
`Figure 2
`
`200
`
`-3-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 3 of 65
`
`US 2002/0069218A1
`
`Content Information saved in a Multimedia Bookmark
`
`214
`
`Query
`
`
`
`332
`
`Multimedia Search
`Engine
`
`Search
`st
`
`310
`
`Retrieved Position
`
`Play from Retrieved Position
`336
`
`Multimedia Player
`
`Multimedia Database
`
`Multimedia Content
`Metadata
`
`Multimedia Content
`Metadata
`OOOOOD de DO De
`
`
`
`
`
`Figure 3
`
`-4-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 4 of 65
`
`US 2002/0069218A1
`
`
`
`-5-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 5 of 65
`
`US 2002/0069218A1
`
`514
`
`SO2
`
`500
`
`SOLIrce COItent
`
`H sample-80.asf
`(master)
`
`O
`O
`li-m- sample-80.rm
`:
`
`
`
`
`
`of-t-t,
`
`s
`
`o=t -i.
`
`S.
`
`O
`
`--H sample-28.rm
`
`O
`
`:
`
`* The media time point P is a start time of the segment A in the master file.
`
`Figure 5
`
`-6-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 6 of 65
`
`US 2002/0069218A1
`
`Variations of
`Multimedia Content 1
`
`Metadata of
`Multimedia Content 1
`(ID: metadatal)
`media profiles
`
`606
`
`file
`
`- - - - -
`
`628 -
`' N file,
`
`630
`
`632
`
`- -
`
`602
`
`608
`
`618
`
`612
`
`620
`
`Metadata of
`Multimedia Content 2
`(ID: metadata2)
`
`604
`
`Variations of
`Multimedia Content 2
`
`
`
`-7-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 7 of 65
`
`US 2002/0069218A1
`
`MMBookmark List of a User
`
`
`
`Positional Information
`
`Content Information
`
`Figure 7
`
`702
`704
`
`706
`
`708
`
`710
`
`... 712
`
`Yio
`
`-8-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 8 of 65
`
`US 2002/0069218A1
`
`O
`O
`
`100
`100
`
`(a) P = P, = 100
`
`P
`
`A
`
`P.
`
`"" file
`playback file
`Ge.
`
`802
`
`P. bookmarked file
`97
`-3
`H-1 - Y -->
`100
`playback file
`
`2.
`
`804
`
`P,
`
`2.
`5
`H-
`
`100
`100
`
`bookmarked file - 806
`playback file
`A -->
`P.
`(c) P, = 102, P = 102+(5-21) = 105
`
`B
`
`-
`
`- - - -
`
`- - -
`
`- -
`
`- -
`
`- -
`
`-
`
`- -
`
`- -
`
`- -
`
`-
`
`- - - -
`
`- -
`
`- - - - -
`
`- -
`
`- - - - - -P.T.I.,
`
`2
`100
`h
`bookmarked file - 808
`|- - - >
`/
`-3
`97
`:
`playback file
`- H -->
`P
`
`2
`
`O
`
`100
`
`b
`
`bookmarked file
`
`810
`
`playback file
`
`Figure 8
`
`P.
`(e) P = 105, P = 105-(2-5D = 102
`
`is 80
`
`-9-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 9 of 65
`
`US 2002/0069218A1
`
`
`
`Edit
`File
`Location:
`
`Go Bookmarks
`
`Options Directory
`
`902
`
`930
`
`94O 952
`
`xt
`
`936Y 938
`
`942
`
`
`
`
`
`
`
`
`
`
`
`Multimedia Content
`Presentation
`
`
`
`
`
`8
`
`8
`8
`
`934
`
`
`
`O Variation
`V. Variation 2
`Variation 3
`------------------------
`V" 910
`
`908
`
`
`
`
`
`950
`
`t
`
`- 956
`
`954
`
`\ 928
`
`—- 912
`
`v- 916
`
`900
`
`-10-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 10 of 65
`
`US 2002/0069218A1
`
`MM - 1002
`PROCESS
`-
`
`- 1004
`
`SELECT MM CoNTENT TO
`PLAY
`
`
`
`
`
`
`
`006
`
`DISPLAY MULTIMEDIA
`PLAYER
`
`1008
`
`YES
`
`
`
`
`
`MULTIPLE
`ARATIONS2
`
`Figure 10
`
`1000
`
`- 1010
`-
`- -4-
`b. DISPLAY WARIATION LIST
`w
`-4- O12
`USE DEFAULT WARIATION
`
`DISPLAY LIST OF MMBOOKMARKS BY /
`USING THEIR CONTENT INFO, AND
`BOOKMARK CONTROLS
`
`104
`
`- 106
`
`/ SELECT
`NCONTROL
`
`CHANGE
`VARATION
`
`SELECT VARIATION
`
`- 1020
`
`
`
`V
`
`1022
`
`1024
`
`WCR OR BM
`CONTROL
`
`GOTO
`SELECTED
`WCR CONTROLS FUNCTIONS
`
`
`
`
`
`
`
`
`
`
`
`BOOKMARKCONTROLS
`
`-
`
`
`
`1026
`
`1030
`
`1034
`
`1038
`
`ADD-BM
`
`PLAY-BM
`
`ADD-BM
`TITLE
`
`-
`
`042
`
`1046 -/
`
`ity- 1028
`Figure 11
`
`y - 1032
`Figure 12
`
`y- O
`036
`Figure 13
`
`1040
`
`..y
`Figure 14
`
`1044
`1048
`
`(F)
`Figure 15
`
`(G)
`Figure 16
`
`-11-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 11 of 65 US 2002/0069218A1
`
`1028
`
`(B)
`SUSPEND MMPLA
`YBACK
`1104
`
`1102
`
`GET URI
`
`1100
`
`- 1106
`YES
`BM Ess" GET BOOKMARKED POSITION
`AYALABLE
`NO
`y
`- 1110
`COMPUTE Av FEATUREAT
`BOOKMARKED POSITION
`
`108
`
`METADATA
`EXIST?
`
`
`
`NO
`
`1 1114
`- FIND SEGMENT CORRESPONDING
`TO BM POSITION IN METADATA
`--
`
`NO
`
`ANNOTATED TEXT
`VAILABLE
`y YES
`GET ANNOTATED TEXT
`
`- 1116
`
`1118
`
`120
`
`MEDIA PROFILES
`AVAILABLE
`
`
`
`YES
`(MULTIPLE VARIATIONS)
`L GET METADATA ID
`
`
`
`
`
`
`
`1122
`
`Figure II
`
`- 124
`
`Y
`SAVE URI, BOOKMARKED POSITION,
`AV FEATURE, ANNOTATED TEXT,
`METADATA ID IN PERSISTENT STORAGE
`y
`- 1126
`
`REDISPLAY LIST OF MMBOOKMARKS
`BYUSING THEIR CONTENT INFO, AND
`BOOKMARK CONTROLS
`Y - 1128
`RESUMEMM PLAYBACK
`
`- 1610
`
`-12-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 12 of 65 US 2002/0069218A1
`
`1032
`
`A- 1202
`
`1200
`
`READ URL, BOOKMARKED PostION
`AND METADATA D FROMPERSISTENT
`STORAGE
`
`1204
`
`
`
`
`
`
`
`BM POSITION
`AVAILABLE
`
`1206
`NO
`
`
`
`- 1208
`YES -y
`
`1044
`
`120
`
`NO
`START MM PLAYBACK
`FROM BEGINNING —-
`
`METADATA ID
`AVAILABLE2
`NO
`
`1212
`
`YES
`(MULTIPLE WARIATIONS)
`4
`d ENTIFY BOOKMARKED
`AND PLAYBACK FILES
`
`
`
`214
`
`216
`
`READ THEIR OFFSETS
`FROMMETADATA
`
`128
`
`1220
`
`ADJUST BOOKMARKED POSITION
`USING OFFSETS
`
`V
`START MM PLAYBACK FROM
`ADJUSTED BOOKMARKED POSITION
`
`y
`
`Figure 12
`
`Y
`--- 'eza
`START MM PLAYBACK FROM
`BOOKMARKED POSITION
`
`?
`
`l
`
`1222
`
`--
`
`- 1610
`
`-13-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 13 of 65 US 2002/0069218A1
`
`- 1036
`
`D
`
`1302
`
`1300
`
`Dr. SELECTED MM BOOKMARK
`FROMPERSISTENT STORAGE
`
`- 1304
`
`REDISPLAY LIST OF MMBOOKMARKS
`BY USING THEIR CONTENT INFO. AND
`BOOKMARK CONTROLS
`
`610
`
`Figure 13
`
`040
`
`1400
`
`- 1402
`ENABLE BOOKMARK TITLE INPUT
`
`1406
`
`USE DEFAULT TITLE
`
`
`
`1408
`
`-
`
`REDISPLAY LIST OF MMBOOKMARKS
`BY USING THEIR CONTENT INFO. AND
`BOOKMARK CONTROLS
`
`- 1610
`
`Figure 14
`
`-14-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 14 of 65 US 2002/0069218A1
`
`- 1044
`
`(e)
`y
`READ CONTENT INFO. FROM
`PERSISTENT STORAGE
`
`so
`
`/ ''
`
`O
`
`Figure 15
`
`
`
`YES
`
`GET ANNOTATED TEXT
`FROM CONTENT INFO.
`
`NO
`1508
`BM OR non- YES
`RETRIEVED POSITION
`YAVAIRE -
`
`METADATA EXIST?
`
`NO
`
`NO
`
`1510
`
`1512
`
`
`
`
`
`
`
`
`
`PERFORM CONTENT-BASEDMM
`SEARCH USING AW FEATURE AS
`QUERY INPUT
`--
`
`FIND SEGMENT CORRESPONDING
`TO POSITION IN METADATA
`
`- 1514
`
`ANNOTATED TEXT
`AVAILABLE
`
`-- 1516
`
`PERFORM TEXT-BASED MM
`SEARCHUSING ANNOTATED
`TEXT AS QUERY INPUT
`
`- 518
`
`1520
`
`1522
`
`PLAYBACK2
`
`NO
`
`
`
`
`
`
`
`MORE
`EARCH2
`
`YES
`
`1530
`YES
`
`
`
`|
`
`NO
`
`1610
`
`GET CONTENT INFO. OF
`SELECTED SEGMENT
`
`COMPUTE AV FEATURES OF
`RETRIEVED SEGMENTSAT
`THER RETRIEVED POSITIONS
`y
`DISPLAY RETRIEVED SEGMENTS
`BY USING THEIR CONTENT INFO.
`J
`1526
`1524 — 1
`SELECT ONE OF RETRIEVED
`
`site"GMENTS
`
`te
`
`y
`PLAY SELECTED SEGMENT FROM ITS
`BEGINNING (RETRIEVED POSITION)
`
`1528
`
`-15-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 15 of 65 US 2002/0069218A1
`
`1600
`
`104.8
`
`1602
`
`READ BOOKMARK INFO (URI, BM
`POSITION, METADATA ID, AV AND
`TEXTUAL FEATURES) FROM
`PERSISTENT STORAGE
`
`/ 1604
`
`ENABLE EMAIL PROGRAM
`
`A
`y
`WRITE MAIL TO PERSON
`
`1606
`
`A- 1608
`
`SEND EMAIL WITH
`BOOKMARK INFO,
`
`1016
`
`SELECT
`CONTROL
`
`
`
`
`
`DIFFERENT MM
`CONTENT2
`
`1002
`
`MM y
`PROCESS
`
`Figure 16
`
`-16-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 16 of 65
`
`US 2002/0069218A1
`
`1700
`
`/
`
`SAVEMM
`BOOKMARK
`
`Y
`ENABLE EMAIL PROGRAM
`
`1702
`
`1704
`
`1706
`
`SELECT AND READ MESSAGE
`
`-- 1708
`NO
`
`1710
`
`INCLUDEMM
`BOOKMARK
`
`
`
`
`
`SELECT PLAY
`CONTROL
`
`ENABLE MMBOOKMARK
`PROGRAM
`
`1712
`
`1714
`
`SAVE DELIVERED BOOKMARK INFO
`F MESSAGE IN PERSISTENT STORAGE
`O
`
`-
`
`/ 176
`DISPLAY LIST OF MM BOOKMARKS
`BY USING THEIR CONTENT INFO,
`AND BOOKMARK CONTROLS
`
`-L--L
`
`m
`
`1718
`
`SELECT PLAY-BM CONTROL
`INTERNALLY
`
`1032
`
`Figure 17
`
`-17-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 17 of 65 US 2002/0069218A1
`
`
`
`
`
`
`
`1802
`
`1808
`
`
`
`
`
`
`
`V
`1800
`
`Figure 18
`
`-18-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 18 of 65 US 2002/0069218A1
`
`1910
`
`1918
`
`1922
`
`1924
`
`1928
`
`
`
`Figure 19
`
`-19-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 19 of 65 US 2002/0069218A1
`
`
`
`2022
`
`2024
`
`2030
`
`Figure 20
`
`-20-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 20 of 65 US 2002/0069218A1
`
`
`
`Figure 21
`
`-21-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 21 of 65 US 2002/0069218A1
`
`
`
`-22-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 22 of 65 US 2002/0069218A1
`
`
`
`-23-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 23 of 65 US 2002/0069218A1
`
`
`
`(x, y) = (X, aX)
`
`-24-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 24 of 65 US 2002/0069218A1
`
`2500
`
`
`
`START
`("NU")
`
`2502
`
`i
`
`
`
`A NEW USER ("NU") ENTERS P2P NETWORK
`
`2504
`
`"NU" MULTICASTS "PNG" TO ANNOUNCE ITS PRESENCE
`
`WAIT TO RECEIVE "PONG" FROM NODES ON THE NETWORK
`- 2510
`
`-u-
`
`A
`
`2508
`
`2524
`
`"NU" KEEPSTRACK OF NODES THAT SENT "PONG" N ORDER
`TO RETAIN A LIST OF ACTIVE NODES FOR CONNECTION
`
`
`
`"SN" EXECUTES VISUAL
`SEARCHENGINE USING ------
`THE QUERY MESSAGE
`
`"NU" INITIATES ASEARCH REQUEST BY
`MULTICASTING A QUERY MESSAGE TO THE
`NETWORK
`y
`
`Y
`
`- -
`
`-
`
`-l
`
`Yi
`SEAR. Es TNur -
`
`"NU" RECEIVES SEARCH RESULTMESSAGE
`CONTAINING IP ADDRESS OF SOURCE NODE ("SN")
`AND A LIST OF FILE SIZES AND NAMES
`
`252
`
`254
`
`2516
`
`"NU". MAKES A CONNECTION TO NODE "SN" AND
`DOWNLOAD IMAGE FILES
`
`
`
`ANOTHER
`SEARCH2
`
`
`
`LEAVE THE P2P NETWORK
`
`-- 2522
`
`END
`
`Figure 25
`
`-25-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 25 of 65 US 2002/0069218A1
`
`2605
`
`2606
`
`2607
`
`Figure 26
`
`
`
`
`
`Caption Text
`
`Figure 27(a)
`
`270
`
`
`
`Figure 27(b)
`
`2708
`
`2716 - -
`
`Figure 27(c)
`
`Figure 27
`
`-26-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 26 of 65 US 2002/0069218A1
`
`2802
`
`IEER
`2804 - Figure 28(a)
`i:-eal / 2808
`John
`StG
`A 280
`SaikoWs,
`68.
`Tiger Woods
`/
`Figure 28(b)
`
`2804
`
`282
`
`
`
`2902
`
`2904
`
`2906
`
`2908 -
`
`2.910
`
`Figure 28
`
`Original feature space I
`
`Haar transform space
`
`294
`
`2916
`
`298
`
`2920
`
`Haar
`Transform
`
`-------v -
`
`2930
`
`-----
`
`2930
`
`2930
`
`Figure 29
`
`-27-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 27 of 65
`
`US 2002/0069218A1
`
`
`
`enenb3O
`
`9009
`
`#7008ºoenembuB
`
`-28-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 28 of 65 US 2002/0069218A1
`
`
`
`-29-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 29 of 65
`
`US 2002/0069218A1
`
`
`
`
`
`
`
`ZOZE
`
`ZOZÊ
`
`-30-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 30 of 65
`
`US 2002/0069218A1
`
`0988
`
`099
`
`
`
`099
`
`-31-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 31 of 65 US 2002/0069218A1
`
`
`
`
`
`
`
`r
`cr)
`
`S.
`w
`CY
`
`-32-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 32 of 65 US 2002/0069218A1
`
`É
`
`
`
`od
`en ly
`t
`
`-33-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 33 of 65 US 2002/0069218A1
`
`Figure 36
`
`<!-- This is listing for metafile. .xml of the videol stored at server -->
`<Metadata
`<segment id=" seg 7" title=" segment 7" duration="580">
`r
`<MediaVRI> //www.video. serveri/vided 1 </Mediau Ri>
`<start Times O & AstartTimes kMediaDuration> 580 </MediaDuration
`<Keyframe> .
`.
`.
`. </Keyframe> <Annotation > ' ' ' ' </Annotation>
`
`<segment id="segs" title= "segment 5" duration="300">
`<StartTimes O (</StartTimes <MediaDuration> 300 </MediaDuration>
`<Keyframe> - - - - C/Keyfrantes <Annotation > .
`.
`.
`. </Annotation>
`
`<segment id=" seg. 1" title=" segment L" duration="100">
`<StartTimes C & StartTimes <MediaDuration> 100 </MediaDuration>
`<Keyframes - - - - (</Keyframe> <Annotation> * * * * CAAnnotation>
`</Segment>
`<Segment id="seg 2" title="segment 2" duration="200">
`<StartTimes 100 </StartTimes <MediaDuration> 200 </MediaDuration >
`<Keyframe> - - - - z/Keyframe> <Annotation: '
`' ' ' </Annotation >
`</Segment>
`</Segment>
`
`<segment id="seg 6" title=" segment 6" duration="280">
`<startTimes 300 <A StartTimes <MediaDuration> 280 </MediaDuration>
`<Keyframe> . . . . </Keyframe> <Annotation >
`</Annotation>
`<Segment id="seg 3" titles" segment 3" durations "110">
`<StartTimes 300 </StartTimes <MediaPurations 110 </MediaDuration>
`<Keyframe> * * * * </Keyframe> <Annotation> -
`- - - (</Annotation>
`</Segment>
`<Segment id="seg 4" titles" segment 4" durations "170">
`<startTimes 410 </StartTimes <MediaDurations 170 </MediaDuration>
`<Keyframe> .
`.
`. . a? Keyframe> <Annotation> * * * * </Annotation>
`</Segiment>
`</Segment>
`
`</Segment>
`</Metadata>
`
`-34-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 34 of 65 US 2002/0069218A1
`
`Figure 37
`
`<!-- This is listing for metafile2.xml of the video2 stored at serverl -->
`<Metadata>
`<Segment id="sege" title="segraent e" duration="380">
`<Medial JRIs fA www.video. server/video2 <AMediaQRIs
`<startTimes O &A StartTimes <Mediaturation> 380 </MediaDurations
`<Keyframe>
`</Keyframe> <Annotation > - - - - C /Annotation>
`
`<Segment id="segd" title=" segment d" duration="230">
`<StartTimes O </Start Times <Mediaturation> 230 </MediaDuration>
`<Keyframe> - - - - C /Keyframe> <Annotation > - - - - g/Annotations
`
`<Segment id="sega" title=" segment a" durations: "50">
`<StartTime 0 </StartTimes kMedia Duration 50 &/MediaDuration
`<Keyframe>
`</Keyframe> <Annotation>
`&/Annotation>
`</Segment>
`<Segment id=" seg b" titles" segment b" duration="180">
`<StartTime: 50 </StartTimes <MediaDuration > 180 </MediaDuration>
`<Keyframe> * * * * </Keyframe> <Annotation: .
`.
`.
`. g (Annotation>
`</Segment>
`</Segment>
`
`<Segment id="seg c" title=" segment c" duration="150">
`<StartTimes 230 </StartTimes <MediaDuration> 150 </MediaPuration>
`<Keyframe> - - - - -WKeyframe> <Annotation > - " " " ( / Annotation>
`</Segment>
`
`</Seguent>
`</Metadatas
`
`-35-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 35 of 65 US 2002/0069218A1
`
`Figure 38
`
`<!-- This is listing for metafile3.xiinl of the video.3 stored at server2. -->
`<Metadata>
`<Segment id="seg H" title=" segment H" duration="620">
`<MediatJRI> //www.video. server2/video.3 </MedialJRI>
`<StartTime> 0 </StartTimes <MediaDuration> 620 </MediaDuration:-
`<Keyframe> .
`. .
`. </Keyframe> <Annotation> - - - - (</Annotations
`
`<segment id="seg F" title= "segment F" duration="410">
`<StartTimes O C /StartTime> <Medialduration> 41C </MediaDuration>
`<Keyframe> .
`.
`. . </Keyframe> <Annotation> .
`. . . </Annotations
`
`<Segment id="seg A" title=" segment A" duration="80">
`<StartTimes 0 </StartTimes <MediaDuration> 80 </MediaDuration>
`<Keyframe>
`</Keyframe> <Annotation> .
`.
`.
`. C/Annotations
`</Segment>
`<segment id="seg B" title=" segment B" duration="150">
`<start Time > 80 </Start Time > <MediaDuration > 150 </MediaDuration>
`<Keyframe>
`C/Keyframe> <Annotations .
`. .
`. </Annotation>
`</Segment>
`<Segment id="seg C" title=" segment C" duration: "180">
`<StartTimes 230 </StartTimes <MediaDuration> 180 </MediaDurations
`<Keyframe> ' ' ' ' </Keyframe> <Annotation> .
`.
`.
`. </Annotation>
`</Segment>
`</Segment>
`<Segment id="seg G" title= "segment G" duration="210">
`kStartTime 410 </StartTime <MediaDuration> 210 </MediaDuration:
`<Keyframe> * * * * </Keyframe> <Annotations - - - - C/Annotation>
`<Segment id="seg D" title=" segment D" duration="110">
`kStartTime> 410 </StartTimes <MediaDuration> 110 </MediaDuration>
`<Keyframe> * * * * </Keyframe> <Annotation>
`Z/Annotation>
`</Segment>
`< Segment id="seg E" title="segment E" duration="100">
`<StartTimed 520 </StartTimes <MediaDuration> 100 </MediaDuration>
`<Keyframe> - - - - (</Keyframe> <Annotation> - - - - ZAnnotation>
`</Segment>
`</Segment>
`</Segment>
`</Metadata>
`
`-36-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 36 of 65 US 2002/0069218A1
`
`Figure 39
`
`< -- This is listing for metafile4.xml of the virtually edited vided -->
`< -- according to the "copy metadata of selected segment" method. -->
`<Metadatae
`<Segment id="seg (2)" title= "segment (2)" duration="710">
`<Keyframe> - - - - (</Keyframe> <Annotation > - -
`- - (</Annotation>
`
`<segment id="seg 5" title=" segment 5" duration="300">
`<MediauRI> //www.video. server1/video. </Medial JRI>
`<StartTimes O &/Start Times <MediaDurations 3OO &/MediaDurations
`<Keyframe> .
`.
`. . </Keyframe> <Annotation> * * * * </Annotation>
`<Segment id=" segl" title=" segment 1." duration="100">
`<StartTimed O </StartTimes <MediaDurations 100 </MediaDuration >
`<Keyframe> .
`.
`.
`. </Keyframes <Annotation> .
`.
`.
`. </Annotation>
`</Segment>
`<Segment id=" seg2' title="segment 2" duration="200">
`<StartTitme> 100 </StartTimes <MediaDurations 200 </MediaDuration>
`<Keyframe>
`</Keyframe> <Annotation>
`c/Annotation>
`</Segiment>
`</Segment>
`
`<Segment id="seg o' title= "segment c' duration="l50">
`9.
`SSSC
`<MediauRI> //www.video. server1/video2 </MediauRI>
`< StartTimes 230 </StartTime <MediaDuration> 150 </MediaDuration>
`<Keyframe> . . . . </Keyframe> <Annotation> * * * * </Annotation>
`</Segment>
`<segment id="seg (D" title=" segment CD" duration="26o">
`<Medial JRI> /www.video. server2/video.3 </MediaVRI>
`<Keyframe - - - - (</Keyframe> <Annotation> * * * * </Annotation>
`
`<Segment id="seg A" title=" segment A" duration="80">
`<StartTimes O &/StartTimes <Mediaturation > 80 </MediaDuration>
`<Keyframe> * * * * </Keyframe> <Annotation>
`</Annotation>
`</Segment >
`<Segment id="seg C" title="segment C" duration="180">
`<StartTime 23 0 < y StartTimes <MediaDurations 18O </Media uration>
`<Keyframe> .
`. . . </Keyframe> <Annotation> * * * * </Annotation>
`</Segment>
`</Segment>
`
`</Segment>
`</Metadata>
`
`-37-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 37 of 65 US 2002/0069218A1
`
`Figure 40
`
`<!-- This is listing for metafile4 - Xml of the virtually edited video -->
`<! -- according to the "write URI of selected segment" method. -->
`<Metadata>
`<Segment id="seg (2)" title="segment (2)" duration="710">
`<Keyframe> . .
`.
`. </Keyframe> <Annotation> .
`.
`. . </Annotation>
`
`<Segiment xlink: form="simple" shows="embed"
`href="//www.video. server1/metafile. .xml#id (seg 5)">
`<MedialJRI> //www.video. server1/video1 </MedialJRI>
`</Segment>
`<Segment xlink: form="simple" show="embed."
`href="//www.video. serveri/metafile2.xml#id (seg c)">
`<Medial JRI> //www.video. server1/video2 </Media URIs
`</Segment>
`<Segment id="seg G" title= "segment (D" duration="260">
`<MedialURIs W/www.video. server2/video.3 </Media URI>
`<Keyframe> ' '
`' ' </Keyframe> <Annotation> .
`.
`.
`. </Annotation>
`
`<Segment xlink: form="simple" show = "embed"
`hiref="//www.video. server2/metafile3.xml#id (seg A)">
`</Segment>
`<Segment xlink: forms "simple" shows="embed"
`href="//www.video. server2/metafile3.xml#id (seg C)">
`</Segment>
`</Segment>
`</Segment>
`</Metadata >
`
`-38-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 38 of 65 US 2002/0069218A1
`
`Figure 41
`
`<!-- This is play list for a root Segment of metafile4.xml -->
`<Playlist>
`<MediaSegment>
`<Medial URIs //www.video. serverl/video1 </Media URIs
`<StartTimes 0 </StartTimex <MediaDurations 300 </MediaDurations
`</MediaSegment>
`
`<MediaSegment>
`<Media JRIs //www.video. server1/video2 </MedialJRI>
`<StartTimed 230 </Start Times <MediaDuration> 150 </MediaDuration>
`</MediaSegment>
`<MediaSegment>
`<MediauRI> //www.video. server2/video.3 </MedialJRid
`<StartTimes 0 </StartTime> <MediaDuration> 80 </MediaDuration>
`</MediaSegment>
`<MediaSegment>
`<Medial JRI> //www.video. server2/video.3 </MedialJRI>
`<StartTime> 230 </StartTimes <MediaDuration> 180 </MediaDurations
`</MediaSegment>
`</Playlist>
`
`-39-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 39 of 65
`
`US 2002/0069218A1
`
`
`
`
`
`(HOLICIGH OFICIIA TVOJLALA
`
`
`
`-40-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 40 of 65 US 2002/0069218A1
`
`
`
`4306
`
`4308
`
`4310
`
`4312
`
`4314
`
`Content Value:
`1.0
`
`4330
`
`4304
`
`4324
`
`4322
`
`-41-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 41 of 65 US 2002/0069218A1
`
`
`
`4406
`
`4408
`
`4410
`
`4412 4414
`
`-42-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 42 of 65
`
`US 2002/0069218A1
`
`
`
`
`
`ç† 9.1m81. I
`
`%
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`-43-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 43 of 65 US 2002/0069218A1
`
`Video frame
`
`4604
`
`Adaptive window
`
`Focus of Attention
`
`
`
`Adjusting window position and size
`
`Figure 46
`
`-44-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 44 of 65 US 2002/0069218A1
`
`
`
`4702
`
`4708 4706
`
`-45-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 45 of 65
`
`US 2002/0069218A1
`
`4836
`
`Digital Video
`
`4818
`
`4820
`
`f
`
`4822
`
`|
`
`4824
`
`
`
`Key frame
`lecti
`d
`SeleCOn a
`feature vector
`extraction
`
`Key Frames
`and Feature
`Vectors
`
`
`
`PCA
`
`Dimension
`reduced
`Feature
`vectors
`7
`4840
`
`LBG
`clustering
`
`Clusters and
`their centroids Compute
`distance
`graphs
`
`4842
`
`
`
`Dimension-reduced
`Feature vectors
`
`
`
`Clusters and
`their centroids
`
`7.
`4808 4806
`
`4810
`
`Hypershell Search
`(Indexing Module) 4812
`
`Query Image/frame
`(query feature vector)
`
`N
`
`4834
`
`/
`
`
`
`Find the nearest cluster(s) to
`the query vector
`
`Perform hypershell
`intersection (basic,
`partitoned, or partioned
`dynamic)
`
`4800
`
`Rank all the feature vectors in
`the intersected regions
`
`Show the ranked result on
`the display
`
`Hypershell Search
`(Query Module)
`
`Figure 48
`
`-46-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 46 of 65 US 2002/0069218A1
`
`4910
`
`Positional
`Information
`
`4920
`
`Content
`information
`
`4930
`
`Metadata
`
`4902
`
`Video Bookmark
`
`f
`
`4900
`
`4932
`4934
`Figure 49
`
`4936
`
`
`
`5004
`
`Recommendation Engine
`
`Video
`Bookmark 1
`
`Video
`Bookmark 2
`
`Video
`Bookmark in
`
`Video Bookmark Folder
`
`5006
`
`5008
`
`\
`
`5010
`
`5000
`
`Figure 50
`
`-47-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 47 of 65 US 2002/0069218A1
`
`
`
`EPG Channel
`
`5100
`M
`
`5102
`
`
`
`Video Bookmark
`Process
`
`Video Bookmark
`Folder
`
`5106
`
`N
`5108
`
`Figure 51
`
`5210
`
`
`
`
`
`Video Bookmark
`Process
`
`
`
`Video Bookmark
`Folder
`
`52O6
`
`5208
`
`Figure 52
`
`-48-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 48 of 65 US 2002/0069218A1
`
`5336
`
`5302
`
`Facsimile
`Machine
`
`
`
`5334
`
`- ruck
`
`- a
`
`Communications
`Tower
`
`Scanner
`
`5332
`
`/
`
`
`
`
`
`
`
`
`
`5350
`/
`
`elevision
`
`5340
`
`CS
`5308
`
`
`
`se
`
`Satellite dish
`
`
`
`5312
`
`Workstation
`
`
`
`
`
`
`
`5322
`
`5324
`
`Video
`
`Laptop
`
`Sever
`
`5314
`
`5338
`Set-Top Box
`5330
`
`Digitizing pad
`
`5326
`
`Telephone
`
`Server 5320
`
`Figure 53
`
`5300
`
`-49-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 49 of 65 US 2002/0069218A1
`
`dequeue
`
`54.04
`
`54O2
`
`540
`6
`
`
`
`
`
`
`
`Indexing
`Server
`
`5410
`
`Figure 54
`
`-50-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 50 of 65 US 2002/0069218A1
`
`5518
`
`5506
`5510
`
`5512
`
`Termination
`Position
`
`Figure 55
`
`5604
`
`
`
`5614
`
`A.
`
`R (Rewind Interval)
`
`563O
`
`a- - -
`
`- as a
`
`a arra-ur-- a-- a-- a-- as as a
`
`a re-aa- - -
`
`-5628
`
`5608
`
`Beginning
`
`5610
`
`5622
`
`5618
`
`Termination
`Position
`5624
`
`End
`
`5632
`
`Figure 56
`
`-51-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 51 of 65 US 2002/0069218A1
`
`57OO Y
`
`u-1
`5702
`
`Start
`(Static Server)
`
`5704
`
`Run Scene Detection Algorithm and Obtain a List
`of Scene Boundaries in Advance
`
`Bookmark
`
`RequestType?
`
`57O6
`
`Playback
`
`5712
`
`57.14
`
`Save a Bookmark
`(Termination Position)
`
`5708
`
`5710
`
`Receive Playback Request
`from Client
`
`Decide Rewind Scope
`
`Send to Client the
`Precomputed List of Refresh
`Frames and Previous
`Termination Postion
`
`5716
`
`
`
`
`
`Random
`
`
`
`Frame
`Selection?
`
`Scene Boundary
`
`5718
`
`572O
`
`Perform Random
`Selection Over
`Rewind Scope
`
`Regular
`
`V
`
`Perform Regular
`Selection Over
`Rewind Scope
`
`Select Scene
`Bounaries incident
`Upon Rewind
`p S cope
`
`5722
`
`Figure 57
`
`-52-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 52 of 65 US 2002/0069218A1
`
`
`
`/
`
`\-5802
`
`Bookmark
`
`Request Type?
`
`Playback
`
`Send a Bookmark
`Request to the Server
`
`5806
`
`Send a Playback
`Request to the Server
`
`--
`
`58O8
`
`l
`
`
`
`
`
`Wait to Receive the
`Refresh Frames
`
`5810
`
`Slide Show
`
`Frame
`Presentation?
`
`Storyboard
`
`Present Refresh
`Frames in Slide Show
`Style
`
`
`
`
`
`Present Refresh
`Frames in Storyboard
`Style
`
`5816
`
`
`
`5814
`
`- Y -
`Continue Playing from
`the Previous
`Terminated Position
`- 5820
`
`5822
`
`End
`Connection?
`
`
`
`No
`
`Figure 58
`
`5824
`
`-53-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 53 of 65 US 2002/0069218A1
`
`5900 Y
`-
`
`Y-5902
`Start
`(
`(Dynamic Server/
`
`y
`
`
`
`Bookmark
`
`
`
`Request Type?
`
`Playback
`
`5904
`
`5906
`
`
`
`|creat AG
`
`Receiv
`
`REas ind Scope
`
`eCIce ReWi
`
`
`
`5912
`
`5910
`
`5914
`
`Frame
`Selection?
`
`
`
`Regular
`
`Perform Random Selection
`Over Rewind Scope
`
`Perform Regular Selection
`Over Rewind Scope
`
`5908
`
`Run Scene Detection Algorithm
`Upon Video Segment Bounded
`by Rewind Scope and Obtain
`List of Scene Boundaries
`
`5916
`
`5918
`
`Send to Client the List of Refresh Frames
`Previous Termination Position
`and
`
`
`
`Figure 59
`
`-54-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 54 of 65 US 2002/0069218A1
`
`6002
`
`Root: Whole Video
`
`6004
`
`6O20
`
`6022
`
`6024
`
`6026
`
`3rd Round
`
`4th Round
`
`Tiger Woods
`
`NAS,
`
`Tiger Woods
`9th Hole
`
`Tiger Woods
`18th Hole
`
`Interview
`
`6021
`
`6032
`
`6034
`
`Figure 60
`
`-55-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 55 of 65 US 2002/0069218A1
`
`
`
`6106
`
`f
`6108
`6110
`
`6116
`6114
`6112
`
`Figure 61
`
`-56-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 56 of 65 US 2002/0069218A1
`
`
`
`E.
`s
`:
`
`6208
`6210
`
`6216
`6214
`6212
`
`Figure 62
`
`-57-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 57 of 65 US 2002/0069218A1
`
`
`
`Figure 63
`
`-58-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 58 of 65 US 2002/0069218A1
`
`
`
`as e s e a c e o O p 9
`
`N 6408
`
`-59-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 59 of 65 US 2002/0069218A1
`
`6502
`
`
`
`6618
`
`6608
`
`6506
`
`6604
`
`6622
`
`Figure 66
`
`-60-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 60 of 65
`
`US 2002/0069218A1
`
`
`
`Figure 67
`
`-61-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 61 of 65 US 2002/0069218A1
`
`6804
`
`68O2
`
`
`
`
`
`Metadata
`DB
`
`
`
`
`
`Video
`Streams
`
`68O8
`
`6810
`
`\
`6812
`
`Figure 68
`
`682O
`
`68OO
`
`-62-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 62 of 65 US 2002/0069218A1
`
`Metadata Server
`
`
`
`
`
`
`
`Metadata Agent
`
`Metadata
`DB
`
`6908
`
`691O
`
`6912
`
`6930
`
`Figure 69
`
`6900
`
`-63-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 63 of 65 US 2002/0069218A1
`
`
`
`7004
`
`Figure 70
`
`-64-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002. Sheet 64 of 65 US 2002/0069218A1
`
`7104
`
`7102
`
`7122
`
`
`
`Figure 72
`
`-65-
`
`

`

`Patent Application Publication
`
`Jun. 6, 2002 Sheet 65 of 65 US 2002/0069218A1
`
`
`
`Figure 73
`
`-66-
`
`

`

`US 2002/0069218 A1
`
`Jun. 6, 2002
`
`SYSTEMAND METHOD FOR INDEXING,
`SEARCHING, IDENTIFYING, AND EDITING
`PORTIONS OF ELECTRONIC MULTIMEDIA
`FILES
`
`BACKGROUND OF THE INVENTION
`0001) 1. Field of the Invention
`0002 The present invention relates generally to marking
`multimedia files. More Specifically, the present invention
`relates to applying or inserting tags into multimedia files for
`indexing and Searching, as well as for editing portions of
`multimedia files, all to facilitate the Storing, Searching, and
`retrieving of the multimedia information.
`0003 2. Background of the Related Art
`0004) 1. Multimedia Bookmarks
`0005 With the phenomenal growth of the Internet, the
`amount of multimedia content that can be accessed by the
`public has virtually exploded. There are occasions where a
`user who once accessed particular multimedia content needs
`or desires to access the content again at a later time, possibly
`at or from a different place. For example, in the case of data
`interruption due to a poor network condition, the user may
`be required to access the content again. In another case, a
`user who once viewed multimedia content at work may want
`to continue to view the content at home. Most users would
`want to restart accessing the content from the point where
`they had left off. Moreover, Subsequent access may be
`initiated by a different user in an exchange of information
`between users. Unfortunately, multimedia content is repre
`Sented in a streaming file format So that a user has to view
`the file from the beginning in order to look for the exact
`point where the first user left off.
`0006. In order to save the time involved in browsing the
`data from the beginning, the concept of a bookmark may be
`used. A conventional bookmark marks a document Such as
`a static web page for later retrieval by Saving a link (address)
`to the document. For example, Internet browserS Support a
`bookmark facility by Saving an address called a Uniform
`Resource Identifier (URI) to a particular file. Internet
`Explorer, manufactured by the Microsoft Corporation of
`Redmond, Wash., uses the term "favorite' to describe a
`Similar concept.
`0007 Conventional bookmarks, however, store only the
`information related to the location of a file, Such as the
`directory name with a file name, a Universal Resource
`Locator (URL), or the URI. The files referred to by con
`ventional bookmarks are treated in the Same way regardless
`of the data formats for Storing the content. Typically, a
`Simple link is used for multimedia content also. For
`example, to link to a multimedia content file through the
`Internet, a URI is