Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 1 of 25 Page ID #:51
`Case 8:11-cv—01985—DOC-JPR Document 1-1
`Filed 12/27/11 Page 1 of 25 Page ID #251
`
`U.S. Patent
`
`sep. 22, 1998
`
`Sheet 5 of 24
`
`5,813,014
`
`318
`
`316
`
`Server
`
`Indexing
`
`FIG. 3
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 2 of 25 Page ID #:52
`Case 8:11-cv—01985-DOC-JPR Document 1-1
`Filed 12/27/11 Page 2 of 25 Page ID #252
`
`U.S. Patent
`
`Sep. 22, 1998
`
`Sheet 6 of 24
`
`5,813,014
`
` Ehmse
`input data ID
`:9
`
`descriptive phrase
`
`Ifl/0“? T5’-‘i900-9755
`35mf‘!f““"d5»‘
`
`sartmagzs)
`
`setfpersm-15}
`
`setfpropased keyunardsj
`
`
`
`
`
`
`
`
`
`nine
`ID
`Law
`
`Sigma:
`ID
`descriptive phrase
`setfplmzses)
`
`ID
`uflypesj
`setflhzsaural keywords)
`selfparem keywords)
`se1‘<'chiId keywords}
`sefipropascd Jcqfwords)
`
`
`
`
`sel(pa-oposed persons)
`
`
`Emznt
`ID
`WP“
`.
`53"‘-""’!P‘v'
`In/Out Tnnemdzs
`mmmznt
`
`fiergon
`N
`if‘.
`rmmy_ mm
`O°‘"I'*"”"’"
`Data: of Birth
`seffpraposed persons}
`
`[ABET
`
`424-
`
`
` 3;.-Jgosed Person
`
`ID
`Nnrn-E
`
`Persrm ID
`Person Attribute
`
`FIG. 4A
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 3 of 25 Page ID #:53
`Case 8:11-cv—01985—DOC-JPR Document 1-1
`Filed 12/27/11 Page 3 of 25 Page ID #253
`
`U.S. Patent
`
`Sep. 22, 1993
`
`Sheet 7 of 24
`
`5,813,014
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 4 of 25 Page ID #:54
`Case 8:11-cv—01985-DOC-JPR Document 1-1
`Filed 12/27/11 Page 4 of 25 Page ID #254
`
`U. S. Patent
`
`Sep. 22, 1993
`
`Sheet 8 of 24
`
`5,813,014
`
`435 \
`
`Thesaural
`Keyword One
`
`Thesaural
`Keyword Two
`
`Thesau:-at
`Keyword Three
`
`Thes-aural
`Keyword Four
`
`4913
`4925-
`
`4938
`493A
`
`496C
`
`4963
`495-“
`
`4973
`497A
`
`43;:
`
`Keyword
`Instance
`one
`
`4900
`490C
`4903
`49“
`
`4913
`
`491A
`
`49112
`MD
`
`
`
`4943
`494A
`
`4953
`495A
`
`Keyword
`Instance
`Two
`
`FIG. 4C
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 5 of 25 Page ID #:55
`Case 8:11-cv—01985—DOC-JPR Document 1-1
`Filed 12/27/11 Page 5 of 25 Page ID #255
`
`U.S. Patent
`
`Sep. 22, 1993
`
`Sheet 9 of 24
`
`5,813,014
`
`318
`
`
`
`515
`
`'
`\ Keyword
`
`314
`
`f
`
`518 \ Keyword Hierarchy
`520
`
`\ Person
`
`225 \ Thesaural Keyword
`
`524
`
`\ Type
`
`526 \ Type Hierarchy
`
` FIG. 5
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 6 of 25 Page ID #:56
`Case 8:11-’cv—01985—DOC-JPR Document 1--1
`Filed 12/27/11 Page 6 of 25 Page ID #256
`
`U.S. Patent
`
`Sep.22,1998 '
`
`Sheet 10 of 24
`
`5,813,014
`
`602
`
`Yes
`
`606
`
`
`
`
`
`
`Yes
`
`perform search
`
`FIG. 6‘
`
`search
`,
`
`play command?
`
`operatmn?
`transmit command
`
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 7 of 25 Page ID #:57
`Case 8:11-cv—01985-DOC-JPR Document 1-1
`Filed 12/27/11 Page 7 of 25 Page ID #257
`
`U.S. Patent
`
`Sep.22,1998
`
`Sheet 11 of 24
`
`-
`
`5,813,014
`
`NM-(D
`
`NHQ
`
`
`
`search already
`exists?
`
`Yes
`
`702
`
`704
`
`
`
`segment
`associated with
`search?
`
`Yes
`
`
`
`706
`
`708
`
`retrieve query
`segment
`
`
`
`
`retrieve phrase ids
`associated with
`
`
`query segment
`
`
`
`
`FIG. 7A
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 8 of 25 Page ID #:58
`Case 8:11-cv—01985—DOC-JPR Document 1-1
`‘Filed 12/27/11 Page 8 of 25 Page ID #258
`
`U.S. Patent
`
`Sep. 22, 1998
`
`Sheet 12 of 24
`
`5,813,014
`
`722
`
`728
`
`
`
`person
`
`
`
`background
`infurma tion
`
`
`
`and! or keyword
`
`search background
`
`
`
`phrase id(s)
`identified?
`
`Yes
`
`732
`
`
`
`create query
`elements
`
`
`
`FIG. 7B -
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 9 of 25 Page ID #:59
`Case 8:11-cv—01985—DOC-JPR Document 1-1
`Filed 12/27/11 Page 9 of 25 Page ID #259
`
`U.S. Patent
`
`Sep. 22, 1993
`
`Sheet 13 of 24
`
`5,813,014
`
`802
`
`T 81
`
`all seardt elements
`processed?
`
`find all subelements
`of elements in set
`
`Yes
`
`\
`
`818
`
`6
`
`
`add subelements
`to element set
`
`806
`
`
`
`
`search element
`attribute for
`search element
`
`Yes
`
`element in
`set?
`
`No
`
`Yes
`
`
`
`
`
`
`
`
`N o
`
`8
`
`14
`
`add element to
`element set
`
`FIG. 8A
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 10 of 25 Page ID #:60
`Case 8:11-cv—01985—DOC-JPR Document 1-1
`Filed 12/27/11 Page 10 of 25 Page ID #260
`
`U.S. Patent
`
`Sep.22, 1998
`
`Sheet 14 of 24
`
`5,813,014
`
`822
` all elements in
`
`set processed?
`
`
`Yes
`
`834
`
`search element
`hierarchy?
`
`
`
`
`
`set next element as
`current Search
`element
`
`Search element
`
`attribute using
`
`830
` element
`instance found
`in set?
`
`832
`
` add element
`instance to found
`element set
`
`search element
`
`
`
`
`
`
`find all Child elements £01‘
`elements in
`found element set
`
`
`
`add child elements to
`, found element set
`
`FIG. 8B
`
`

`
`Case 8:11-cv—0198'5-DOC-JPR Document 1-1
`Filed 12/27/11 Page 11 of 25 Page ID #261
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 11 of 25 Page ID #:61
`
`U.S. Patent
`
`Sep.22,1998
`
`Sheet 15 of 24
`
`5,813,014
`
`860
`
`end
`
`Yes
`
`all keywords in
`set processed?
`
`
`No
`
`850
`
`852
`
`search phrase
`elements for
`ord association
`
`
`ke
`
`
`
`856
`
`
`
`phrase instance
`found not in phrase
`set?
`
`
`
`Yes
`
`358
`
`add phrase instance
`to phrase set
`
`
`
`FIG. 8C
`
`
`
`
`set next keyword as
`current keyword
`
`854
`
`
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 12 of 25 Page ID #:62
`Filed 12/27/11
`Case 8:11—cv—01985—DOC-JPR Document 1-1
`Page 12 of 25 Page ID #262
`
`U.S. Patent
`
`Sep. 22, 1993
`
`Sheet 16 of 24
`
`5,813,014
`
`902
`
`search background information
`using selection criteria to
`identify person instances
`
`
`
`
`
`
`erson instance
`
`select segment
`instances for each
`
`904 .
`
`906
`
`
`
`create set of phrases
`associated with
`
`selected segments
`
`
`
`918
`
`Yes
`
`end
`
`
`
`12
`
`9
`
`9
`
`
`
`14
`
`
`
`all phrases in
`set processed?
`
`
`
`No
`
`Search phrase's
`keywords for
`s ecified content
`
`916
`
`
`
`create set of phrases
`having specified
`
`content
`
`
`
`FIG. 9
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 13 of 25 Page ID #:63
`Case 8:11-cv—01985—DOC-JPR Document 1-1
`Filed 12/27/11 Page 13- of 25 Page ID #263
`
`1 U.S. Patent
`
`Sep.22,1998
`
`Sheet 17 of 24
`
`5,813,014
`
`1002
`
`
`
`identify segment
`instance(s) associated
`with person instance(s)
`
`1004
`
`
`
`
`
`
`
`
`identify phrase
`
`instance(s) associated
`with segment insance(s)
`
`1006
`
`create set of phrases
`associated with
`
`
`
`selected segments
`
`
`content search?
`
`No—>®
`
`Yes
`
`
`
`
`1010
`
`
`
`all phrases in set
`processed?
`
`No
`
`Yes«>®
`
`FIG. IOA
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 14 of 25 Page ID #:64
`Case 8:11-cv—01985-DOC-JPR Document 1-1
`Filed 12/27/11 Page 14 of 25 Page ID #264
`
`US. Patent
`
`Sep. 22, 1998
`
`Sheet 13 of 24
`
`5,813,014
`
`_.
`
`
`
`search phrases
`keywords for
`s ecifiecl content
`
`12
`
`10
`
`10
`
`14
`
`
`
`1016
`
`create set of phrases
`having specified
`Content
`
`FIG. 10B
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 15 of 25 Page ID #:65
`Case 8:11-cv—01985—DOC-JPR Document 1-1
`Filed 12/27/11 Page 15 of 25 Page ID #265
`
`U.S. Patent
`
`Sep.22,1998
`
`Sheet 19 of 24
`
`5,813,014
`
`PIG. 11A
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 16 of 25 Page ID #:66
`Case 8:11-cv—01985—DOC-JPR Document 1-1
`Filed 12/27/11 Page 16 of 25 Page ID #266
`
`U.S. Patent
`
`Sep.22,1998
`
`Sheet 20 of 24
`
`5,813,014
`
`
`
`Qu¢_1-3 fitment Record
`
`1106
`
`
`
`
`
`
`Query ID
`Query Element ID
`ID
`A.nd0r
`Element Order
`
`
`
`£2.m'_r1LE.!:m.mt
`
`Query Element ID
`
`PIG. 11B
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 17 of 25 Page ID #:67
`Case 8:11-cv—01985-DOC—JPR Document1-1
`Fi|ed12/27/11
`Page 17 of 25 Page ID #267
`
`U.S.a Patent
`
`Sep. 22, 1998
`
`Sheet 21 of 24
`
`5,813,014
`
`1208
`
` Ehzas:
`ID
`
`Rhtnsz
`
` Storage
`
`in}:-u§ data path
`
`
`
`input data ID
`ID
`
`
`
`
`
`descriptive phrase
`
`In/Out Timecndcs
`In/Out Timecoda
`
`5et(ke_1,rwonis)
`Key Frame
`settimages)
`settpersons)
`seffpraposad key-words)
`sef(prapa5::d persansl
`
`
`
`
`1212
`
`1210
`
`Cache 1
`
`1214 -’
`
`FIG. 12
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 18 of 25 Page ID #:68
`Case 8:11-cv—01985—DOC-JPR "Document 1-1
`Filed 12/27/11 Page 18 of 25 Page ID #268
`
`U.S. Patent
`
`Sep.22,1998.
`
`Sheet 22 of 24 '
`
`5,813,014
`
`
`
`
`
`
`1300
`
`retrieve
`
`multimedia data
`
`start thread
`
`transfer data
`to thread
`
`. 1302
`
`
`
`13
`
`O4
`
`1306
`
`thread prepares and
`transfers data to browser
`
`_
`
`1
`
`end of data?
`
`Yes
`
`1310
`
`terminate thread
`
`
`No..@
`
`
`PIG. 13A
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 19 of 25 Page ID #:69
`Case 8:11-cv—01985—DOC-JPR Document 1-1
`Filed 12/27/11 Page 19 of 25 Page ID #269
`
`U.S. Patent
`
`Sep.22,1993
`
`Sheet 23 of 24
`
`5,813,014
`
`play Command?
`
`No
`
`
`
`
`stop command?
`
`process play
`command
`
`
`
`
`FIG. 13B
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 20 of 25 Page ID #:70
`Case 8:11-cv—01985-DOC-JPR Document 1-1
`Filed 12/27/11 Page 20 of 25 Page ID #270
`
`U.S. Patent
`
`Sep.22,1998
`
`Sheet 24 0f24
`
`'
`
`5,813,014
`
`Browser
`
`1402
`
`-
`
`USES
`
`1404
`
`Method
`fife‘
`
`1413
`
`_
`C11?
`Object
`
`U
`
`55
`
`Player
`Obie“
`
`Uses
`
`1412
`
`Phrase
`
`Uses
`
`Uses
`
`1416
`
`U55
`
`\
`Phrase
`Manager
`Obs-e(',
`
`1414
`
`1
`
`1410
`
`Contains ———
`
`phrase
`Object
`
`FIG. 14
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 21 of 25 Page ID #:71
`Case 8:11-cv—01985-DOC-JPR Document 1-1
`Filed 12/27/11 Page 21 of 25 Page ID #271
`
`5,813,014
`
`1
`METHOD AND APPARATUS FOR
`MANAGEMENT OF MULTIMEDIA ASSETS
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`
`The invention relates to a system of managing multimedia
`assets.
`
`2. Background
`Increasingly, computer systems are being used to present
`multimedia material. Such material is usually in the form of
`text, graphics, video, animation, and sound. Two or more of
`these data types are usually combined to form the multirne-
`dia data presented by the computer system. A computer
`system that is used to present multimedia material is called
`a multimedia system.
`A multimedia system should include components that
`allow input, information retrieval, and dispiay. FIG. 1A
`provides an illustration of components for a multimedia
`system. Client 110 includes an interface capability to accept
`user input. The user input can specify criteria for identifying
`multimedia data. Method player 102 converts the media data
`from its storage version to a playback version for use by
`client 110.
`
`Index server 108, archive server 106, and tertiary storage
`1134 provide storage and retrieval capabilities. Index server
`108 is used to store and retrieve text data. Information
`entered in the user interface can be used to retrieve textual
`information stored by index server 108. The user input might
`identify a video segment using a title that is associated with
`the video segment using index server 108, for example. The
`multimedia data is stored on a physical storage device. A
`retrieval mechanism is provided to retrieve data from a
`physical storage device. Archive server 106 locates the
`multimedia data wherever it is stored and interfaces with
`tertiary storage manager 105 to access the media data.
`Current multimedia systems attempt to provide some or
`all of the components described in FIG. 1A. However, the
`components provided by these systems are merged to form
`a single component thereby making it impossible to separate
`the merged components into the discrete components
`described in FIG. 1A. This architecture is disadvantageous
`for at least two reasons: 1) there is no ability to replace a less
`capable component with another, more capable component;
`2.) it forces each system to run on a single hardware platform.
`Each system provides vendor-specific implementations in
`which at least one of the vendor-specific components used in
`these systems renders the system less viable as a solution for
`multimedia applications. Since, the services are merged into
`a single server, there is no ability to remove a less viable
`component and substitute a more capable component in its
`place. For example, a vendor may ofier a multimedia system
`that includes its premiere DBMS product as an index server
`along with a less capable archive server or client. All of the
`components are intertwined such that it is impossible to
`substitute another archive server or client for the existing
`archive server and client. The following provides a brief
`summary of some existing multimedia systems.
`FIG. 1B illustrates one multimedia system having an
`intermedia server, client and index server combined as a
`single component that runs on one, monolithic hardware
`platform. The system is not portable to another hardware
`platform. It runs on a single hardware platform (i.e., plat-
`form 112, a Hewlett Packard platform). The system includes
`intermedia server 118, index server 1.16, and client 114.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`Client 114 is used to make a multimedia request to inter-
`media server 118. Intermedia server 118 queries index server
`116 for information to locate the requested data. Intermedia
`server 118 uses the data returned from index server 116 to
`request the data from its archive server and tertiary storage
`management subsystems.
`The archive server and tertiary storage management sub-
`systems are integrated in Intermedia server 118. Client and
`method player 114 and index server 116 are interconnected
`with interniedia server 118 to form a single component. The
`interconnections are hardwired such that it is impossible to
`replace one of the existing components. Another aspect of
`the design of this system results in the inability to substitute
`another component. Client 114 has a built-in method player
`functionality. Thus, the system expects the client component
`to include a method player. Yet, most client components that
`are offered by vendors do not include a method player.
`Therefore, it is impossible to use a difierent client or method
`player component with intermedia server 118.
`A multimedia system having a hub is illustrated in FIG.
`1C. Such a system is provided by Oracle. The hub is
`provided by media server 130. The system runs on a specific
`hardware platform (hardware platform 138, an N«Cube
`hardware platform) and is not portable to other platforms.
`Media server 130 acts as a hub that uses a proprietary
`interface to communicate with the other services. Compo-
`nents with which media server 130 can communicate are
`method player 122, tertiary storage manager 124, and client
`12!} (via lines 134, 132, and 136, respectively). These
`components are merged into a single component thus mak-
`ing it impossible to replace a component. The component
`formed by media server 130, method player 122, tertiary
`storage manager 124, index server 128, archive server 126
`and client 120 must run on a single hardware platform,
`platform 138. Further, While index server 128 is a powerful
`database management system, client 120, archive server
`126, method player 122 and tertiary storage 124 offer weak
`solutions.
`
`tertiary storage
`A system having an index server,
`manager, and archive server is illustrated in FIG. 1D. Such
`a system is provided by IBM. The system ofiers index server
`148 (e.g., IBM’s DB2 or Oracle’s DBMS), archive server
`146, and tertiary storage manager 144 in an integrated
`system that runs on mainframe 140. Mainframe 140 is an
`IBM SP2 hardware platform comprised of two RS6000
`machines. Mainframe 140 is comprised of processing units
`144 in which components (e.g., index server 148) can run.
`Tertiary storage 144 is connected to one of processing units
`144. A request for multimedia content from a client is
`processed by index server 148. Index server 148 provides
`selection information that identifies the content which is
`retrieved from tertiary storage 144. The system is built to run
`in a mainframe environment using IBM hardware. Further,
`the system does not include a client or method player. Index
`server 148, tertiary storage manager 144 and archive server
`146 are combined as a single component such that it is
`impossible to replace one or more of them.
`A system having a kernel and data blades that interface
`with the kernel is illustrated in FIG. IE. Such a system is
`provided by Informix. As illustrated below,
`the system
`includes kernel 170 that acts as a hub. Kernel 170 is
`supplemented by a plurality of data blades 160A~160H.
`Data blades 160A—16OH communicate with the kernel using
`a proprietary interface. A data blade is a complex data type
`that is defined using a programming language such as “C”.
`Each data blade (e.g., 160A—160H) is attached to kernel 170
`and integrated into data dictionary 172. Data blades
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 22 of 25 Page ID #:72
`Filed 12/27/11
`Case 8:11—cv—01985—DOC-JPR Document 1-1
`Page 22 of 25 Page ID #272
`
`5,813,014
`
`3
`160A—160H can then be used over a file system. Kernel 170
`includes a index. Thus, another component must communi-
`cate with the index via a proprietary interface provided by a
`data blade (e.g., data blades 160A—16Dl-1). Data blades
`160E, 160B, and 160D provide a proprietary interface to
`method player 162,
`tertiary storage manager 164, and
`archive server 166, respectively. The components provided
`by this system are merged to form a single component that
`use a proprietary interface to communicate. The component
`combination runs in a single hardware platform 174. Data
`dictionary 1'72 can become large and cumbersome.
`In
`addition, a fault that occurs in one data blade that is included
`in data dictionary 172 causes a fault for the entire system.
`This system construction is not fault tolerant and is unac-
`ceptable for a production environment.
`A pre-packaged system having interconnected system
`components with hardwired, proprietary interconnections is
`illustrated in FIG. 1F. Such a system is provided by Cine-
`base. System 178 includes a component formed by method
`player 182, tertiary storage manager 184, archive server 186,
`index server 188, and client 190. There is no clear delinea-
`tion between components. Further,
`there are no clearly
`defined lines of communication between the components.
`Component interconnections are hardwired, and it is there-
`fore impossible to substitute components that can commu-
`nicate using the existing connections for the existing com-
`ponents. There is no ability to split the component into
`discrete components such that replacement component can
`be substituted for one of the existing components. Further,
`it
`is impossible to split the combined component
`into
`separate components that can run on multiple hardware
`platforms. The combined component olfers a weak solution.
`The archive server is fast and eflicient. Several tertiary
`storage managers are also provided. The method players are
`capable. However, the client and index server components
`provide a weak solution.
`None of these systems illustrated in FIGS. LB-1F provide
`a general cataloguing capability that can catalogue any type
`of multimedia data. In addition, none of the systems pro-
`vided a viable multirnedia system solution. That is, none of
`the systems provide viable options for each of the multime-
`dia components identified in FIG. 1A. All of the systems
`merge the components identified in FIG. 1A into a single,
`component that makes it impossible to replace one of the
`components. Further, by combining the components into a
`single component, each system must run on a single hard-
`ware platform. Further,
`there is no vendor-independent
`interface available to integrate components from different
`vendors to construct a optimum multimedia system.
`A number of prior art patents that describe software
`and/or hardware systems are provided below. These systems
`do not provide a general cataioguing capability or interfaces
`to interconnect the multimedia system components identi-
`fied in FIG. 1A.
`
`A computer system having a television circuit that is
`capable of receiving television signals and forwarding these
`signals to the computer system’s audio and video sub-
`systems is described in U.S. Pat. No. 5,192,999 (Graczyk et
`a1.,
`issued on Mar. 9, 19931 U.S. Pat. No. 5,283,638
`(Engberg et al., issued Feb. 1, 1994), and U.S. Pat. No.
`5,283,819 (Glick et al., issued Feb. 1, 1994). Video signals
`are received for display on the computer systern’s monitor.
`Audio signals are output using the computer system’s
`spealrer{s). A remote control device is used to control the
`television circuit within the computer system. A computer
`system having the television circuit can be used in combi-
`nation with a data/fax/voice modem circuit for telephonic
`transmission of television signals for teleconferencing pur-
`poses.
`
`4
`A service for managing hypertext/hypermedia links is
`described in U.S. Pat. No. 5,297,249, Bernstein et a1., issued
`on Mar. 22, 1994. A Link Manager Services (LMS) com-
`ponent provides a universal End User Interface
`including menus and dialog boxes that can be called by an
`application program when a link, or marker, is selected by an
`application user. The LMS accesses a database that contains
`information about markers and their associated “presenters”
`(e.g., application or program). In addition, the LMS includes
`a viewer capability for viewing and maintaining existing
`links and creating new links. The LMS is used at runtime to
`generate menus and dialog boxes and manage the hypertext]
`hypermedia links.
`A system that interconnects audio-video equipment such
`as video tape record or video production switcher using one
`or more “AV LANS” in U.S. Pat. No. 5,307,456, Mackay,
`issued on Apr. 26, 1994. Generic device commands are
`communicated over anAV LAN to a device translator. The
`device translator translates the generic command into a
`device-specific command. The AV LANS are interconnected
`using bridges. The AV [ANS can further be used to inter-
`connect workstations, minicomputers, mainframes and per-
`sonal computers. The workstations can be used to display
`information about the resources attached to an AV L_AN.
`Acontroller apparatus for use with a computer system that
`provides programmed volume control of multiple analog
`input signals such as CD-ROM, telephony, MIDI, PCM,
`FM, and microphone in U.S. Pat. No. 5,402,499, Robison et
`al., issued on Mar. 28, 1995. Additionally, the controller can
`sum multiple inputs and produce multiple audio outputs. The
`controller provides telephony support such that it receives
`voice, DTMF and caller ID signals and sends telephonic
`audio signals.
`A multimedia system that provides a multimedia device
`control program (MMDCP) that interfaces with a multime-
`dia application program (MMAP) and a multimedia presen-
`tation manager (MMPM) in OS/2 in U.S. Pat. No. 5,428,
`730, Baker et al., issued on Jun. 27, 1995. The MMDCP
`provides a standardized user interface. Because’ many of the
`control functions are performed by the MMDCP, a MMAP
`can be simplified. The MMDCP creates an instance of a
`control panel for use with a multimedia device. A control
`panel instance includes UI widgets (e.g., buttons) that allow
`a user to control the associated device via the control panel
`instance and the device driver.
`
`An expansion unit to implement multimedia capabilities
`on an existing computer system (e.g., IBM PS/'2) in U.S. Pat.
`No. 5,434,592, Dinwiddie, Jr. et 211., issued on Jul. 18, 1995.
`The expansion unit is connected to the computer system via
`an expansion slot and to the computer system’s monitor. The
`expansion slot controls the video presentation displayed on
`the monitor.
`
`Amultirnedia network bus (MMNB), multimedia network
`controller (MMNC) and protocol for transmitting isochro-
`nous and packet data between a backbone or wide area
`network and multiple local area network configurations in
`U.S. Pat. No. 5,436,898, Bowen et. al., issued on .Iul. 25,
`1995. The MMNC allocates isochronous channels and
`packet channels to users over the MMNB taking into
`account the time deiay requirements of various data types
`(e.g., voice, moving images, graphics, or text). The MMNB
`and MMNC is for use with multimedia applications that
`require a varying mix of isochronous data and packet data
`transfers.
`
`SUMMARY OF THE INVENTION
`
`The invention is a generalized solution for management of
`multimedia assets. Generalized interfaces are used between
`
`10
`
`15
`
`20
`
`30
`
`35
`
`40
`
`45
`
`SO
`
`60
`
`65
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 23 of 25 Page ID #:73
`Case 8:11—cv—01985—DOC-JPR Document 1-1
`Filed 12/27/11 Page 23 of 25 Page ID #273
`
`5,813,014
`
`5
`a browser component, indexing server, archive server, ter-
`tiary storage manager, and method player components. The
`generalized interface defines a communication protocol that
`can be used by any browser, indexing server, archive server,
`tertiary storage manager, or method player component.
`Thus, system components supplied by multiple vendors can
`be interconnected to form a multimedia system that com-
`municates using the generalized interfaces of the invention.
`The browser component can accept input such as search
`requests and display multimedia data. Search requests are
`processed by the browser. The indexing server component
`includes :1 data catalogue that catatogues multimedia data.
`The catalogue can be used to identify multimedia data that
`can satisfy a search request entered using the browser
`componenfs user interface. Search mechanisms provided by
`the browser send requests to the indexing server to query the
`catalogue and obtain attribute information associated with
`the catalogue. A search mechanism can create additional
`catalogues that are a subset of the catalogue provided on the
`indexing server.
`The indexing server’s multimedia catalogue includes one
`or more catalogue elements which can be a complex mul-
`timedia asset. Acomplex multimedia asset can consist of one
`or more attribute elements. An attribute element
`is an
`attribute that can have attributes. A catalogue element is
`associated with a portion of multimedia data (e.g., one or
`more frames of video data). The result of a search operation
`performed by the browser identifies a set of catalogue
`elements that can satisfy a search request. Each catalogue
`element has an associated identifier (ID) (e.g., an integer ID)
`that uniquely identifies the catalogue element. A set of IDs
`that represent the set of catalogue elements identified in a
`search operation are sent to the archive server component for
`retrieval of the associated multimedia data.
`The invention retains the content and results of a search
`such that it is only necessary to perform a search once.
`Thereafter, the results of the search can be retrieved without
`performing the search. Search elements are used to store the
`content of a search, i.e., search criteria. Search elements
`include a query and a query element. Query element
`instances contain the se arch criteria. A query instance iden-
`tifies a particular query. Aqucry instance can contain one or
`more query element instances.
`A segment element is used to store the results of a search.
`Asegment element is a multimedia asset. Thus, for example,
`when a set of catalogue elements is identified from a search
`operation, a segment element is instantiated. An attribute
`element of the segment element contains each catalogue
`element identified in the search operation. A relationship is
`formed between the segment element and a query instance.
`When a search request is entered via the browser’s user
`interface,
`the query element instances associated with a
`query instance are examined to determine whether the
`current Search is a reoccurrence of a previous search. If so,
`the relationship formed between the query instance and the
`segment instance is used to retrieve the segment that con-
`tains the catalogue elements identified in the search opera-
`tion performed for the query instance. Each catalogue ele-
`ment oontaiued in the segment can then be retrieved and sent
`to the archive server. The archive server identifies the
`tertiary storage manager that manages the store containing
`the multimedia data. The tertiary storage manager retrieves
`the corresponding multimedia data from its store and sends
`it to the method play which plays it in a display region of the
`browser.
`
`Multiple types of catalogue elements can be used with the
`invention. Preferably, however, one type of catalogue ele-
`
`6
`ment that is referred to as a phrase is used in the invention.
`Aphrase is associated with a portion of multimedia data. A
`phrase has a plurality of attributes some of which are
`attribute elements (i.e., an attribute that has associated
`attributes). The attribute elements that are attributes of a
`phrase include keyword, person, image, video (e.g., docu-
`mentary footage), proposed person, and proposed keyword.
`To locate multimedia data associated with a phrase, the
`archive server component retains information for each
`phrase in the catalogue such as the ID and a pointer to the
`physical storage location of the data associated with the
`phrase. When the archive server receives a set of phrase IDs,
`it uses the physical storage pointer associated with each
`phrase to locate the multimedia data. A phrase object is
`created to retain the multimedia data associated with a
`phrase. Multimedia data retrieved from permanent storage
`into phrase object instances.
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1A provides an illustration of components for a
`multimedia system.
`'
`FIGS. 1B-—1F illustrate pre-packaged multimedia systems
`having interconnected system Components with hardwired,
`proprietary interconnections.
`FIG. 2 provides an example of a general purpose com-
`puter to be used with the present invention.
`FIG. 3 illustrates an embodiment of the invention having
`generalized interfaces between system components.
`FIG. 4A provides an example of elements in the indexing
`template and their attributes according to an embodiment of
`the invention.
`
`10
`
`15
`
`30
`
`FIG. 4B illustrates relationships formed between the
`elements identified in FIG. 4A according to an embodiment
`of the invention.
`
`35
`
`40
`
`45
`
`SD
`
`60
`
`65
`
`FIG. 4C provides an illustration of instances of keyword
`410 and a plurality of associated label attributes according to
`an embodiment of the invention.
`FIG. 5 provides an overview of an interface between a
`browser component and an indexing server component
`according to an embodiment of the invention.
`FIG. 6 provides a process ftow for handling search and
`display operations according to an embodiment of the inven-
`tion.
`
`FIGS. 7A—7B provide a process flow for processing a
`search request according to an embodiment of the invention.
`FIGS. 8A—8B provide a type and keyword search process
`flow according to an embodiment of the invention.
`FIG. 8C provides a process flow for identifying instances
`of phrase 406 that are associated with instances of keyword
`410 contained in a search set created by a search operation
`according to an embodiment of the invention.
`FIG. 9 provides a person infonnation search process flow
`according to an embodiment of the invention.
`FIGS. 10A—l0B provide a person search process flow
`according to an embodiment of the invention.
`FIGS. 11A—11B illustrate search elements, relationships,
`and attributes used to process and retain search requests
`according to an embodiment of the invention.
`FIG. 12 illustrates use of the phrase element to access
`multimedia data in accordance with an embodiment of the
`invention.
`
`FIGS. 13A—-I313 provide a process flow for handling a
`play command invocation received from a browser accord-
`ing to an embodiment of the invention.
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-1 Filed 12/27/11 Page 24 of 25 Page ID #:74
`Filed 12/27/11
`Case 8:11-cv-01985-DOC-JPR Document 1 -1
`Page 24 of 25 Page ID #274
`
`5,813,014
`
`7
`FIG. 14 provides an example of asset management objects
`according to an embodiment of the invention.
`
`DETAILED DESCRIPTION OF THE
`INVENTION
`
`Amethod and apparatus for managing multimedia assets
`is described. In the following description, numerous specific
`details are set forth in order to provide a more thorough
`description of the present invention. It will be apparent,
`however, to one slciiled in the art, that the present invention
`may be practiced without these specific details. In other
`instances, Well-lcrrown features have not been described in
`detail so as not to obscure the invention.
`
`An embodiment of the invention can be implemented as
`computer software in the form of computer readable pro-
`gram code executed on a general purpose computer such as
`illustrated in FIG. 2. A keyboard 210 and mouse 211 are
`coupled to a bi-directional system bus 218. The keyboard
`and mouse are for introducing user input to the computer
`system and communicating that user input to central pro-
`cessing unit (CPU) 213. Other suitable input devices may be
`used in addition to, or in place of, the mouse 211 and
`keyboard 210. L0 (inputioutput) unit 219 coupled to
`bi-directional system bus 213 represents such I/O elements
`as a printer, network communications card, modem, A./V
`(audio/video) I/O, etc.
`The computer system of FIG. 2 also includes a video
`memor