Case 8:11-cv-01985-DOC-JPR Document 1-2 Filed 12/27/11 Page 1 of 25 Page ID #:76
`Case 8:11-cv—01985-DOC—JPR Documentt-2
`Fi|ed12/27/11
`Page 1 of 25* Page ID #276
`
`5,813,014
`
`11
`495B, 496C, and 497Acan be the preferred French, English,
`German, and Russian instances (respectively) for Keyword
`Instance Two.
`Referring to FIG. 4A, type 416 is associated with key-
`word 410. Type 416 provides attribute information for
`keyword 410. Type 416 can be used to include instances of
`keyword 410 in a classification or category. In other words,
`an instance of keyword 410 is an instantiation of an instance
`of type 416. For example, an instance of keyword 410
`having an attribute of “Ford Bronco” could be associated
`with a type instance having an attribute of “car”. Another
`instance of keyword 410 having an attribute of “Mustang”
`can also be associated with the same instance of type 416.
`Both instances of keyword 410 are instances of a car. One
`or more instances of type 416 can be associated with an
`instance of keyword 410. In the preferred embodiment, a
`hierarchy is established for instances of type 416. An
`instance of type 416 can be a parent to or a child of another
`other instances of type 416. An instance of keyword 419 that
`is associated with an instance of type 416 is also associated
`with the hierarchy of the instance of type 416.
`Other attribute elements that can be associated with an
`input data fragment via phrase 406 include person 418, and
`image 420. Person 418 identifies an individual associated
`with an input data fragment. In the previous example, a
`personal life experience may Contain a reference to a person.
`An instance of person 418 can be used to identify the
`reference. Person information 426 provides attribute infor-
`mation for an instance of person 418. An instance of image
`420 is used for data such as a still photograph that
`is
`referenced in the input data.
`In the preferred embodiment of the invention, some
`elements, such as keyword 410 and person 418, must be
`approved before becoming actual
`instances. Prior to
`approval,
`the instances are considered to be proposed
`instances. For example, proposed keyword 414 and pro-
`posed person 424 are attribute elements used to identify
`instances of keyword 410 and person 418 that have not yet
`been approved as actual instances. Proposed instances are
`reviewed and a determination is made whether to transform
`the proposed attribute element into an actual attribute ele-
`ment or to otherwise dispose of the proposed attribute
`element.
`Person Information 426 is an attribute element associated
`with person 418. A “one-to-one” relationship (relationship
`436) exists between person information 426 and person 418.
`Person information 426 contains attributes for person 418.
`The attributes of person information 426 contain informa-
`tion for a person having an instance of person 418.
`Events can also be associated with input data. Each event
`becomes an instance of event 408. As previously described,
`input data can be decomposed into input data fragments each
`of which is associated with an instance of phrase 406. Input
`data can also be decomposed into input data fragments that
`are associated with instances of event 408. A type attribute
`is associated with event 408. Examples of an event type in
`the preferred embodiment include a segment, phrase, break
`between tapes, quality assurance details, facts, and miscel-
`laneous _(or other). An event can be used to access the
`associated input data fragment. An instance of event 408 can
`be used to access an input data fragment. For example, an
`instance of event 408 of type phrase can be used to locate the
`input data fragment associated with an instance of phrase
`406.
`
`Another example of an event type is a quality assurance
`event. In the preferred embodiment of the invention, a
`quality assurance mechanism can be used to monitor the
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`12
`quality of the input data and provide feedback. Quality
`assurance events are used to mark the input data. An event
`can mark a positive, negative, or neutral quality assurance
`event. For example, video input data is being collected in
`multiple interviews. Each interview can be reviewed to
`identify parts of the interview process that are noteworthy.
`Where, for example, an interviewer does not follow-up with
`an interviewee to obtain additional details, a negative quality
`assurance event can be created. A positive event can be
`similarly created. An event
`that
`is neither positive nor
`negative (i.e., informational or neutral) can also be created.
`A report of quality assurance events can be generated and
`used to provide feedback to the persons involved in collect-
`ing the input data.
`Relationships of Elements
`In the preferred embodiment, catalogue and attribute
`elements are interrelated. Relationships are formed between
`two or more elements using the invention. FIG. 4B illus-
`trates relationships formed between the elements identified
`in FIG. 4A according to an embodiment of the invention. A
`“many” relationship is signified using a double arrow. A
`“one” relationship is identified using a single arrow. Rela-
`tionship 428, for example, is a “many-to-many” relation-
`ship. That is, one or more instances of segment 404 can be
`related to many instances of phrase 406.Alternatively stated,
`segment 404 contains one or more instances of phrase 406.
`One instance of phrase 406 can be related to multiple
`instances of segment 404. That is, an instance of phrase 406
`is contained within one or more instances of segment 404.
`As illustrated by relationship 446, one or more instances of
`type 416 can be related to other instances of type 416.
`A “many-to-many” relationship (relationship 430) exists
`between phrase 406 and proposed keyword 414, keyword
`410, image/video 420, proposed person 424 and person 418.
`An instance of phrase 406 can be related to a set of proposed
`keywords, a set of keywords, a set of images and/or video,
`a set of proposed persons, and a set of persons, each set
`having zero or more members. Further, an instance of
`proposed keyword 414, keyword 410, image 420, proposed
`person 424 or person 413 can be related to more than one
`instance of phrase 406.
`Relationship 438 illustrates a “many-to-many” relation-
`ship between keyword 410 and thesaural keyword 412. An
`instance ofkeyword 410 can be associated with one or more
`instances of thesaural keyword 412. The same instance of
`thesaural keyword 412 can be associated with one or more
`instances of keyword 410.
`As previously stated, instances of type 416 can be inter-
`related with other instances of type 416 via a type hierarchy.
`Relationship 444 identifies an instance of type 416 as a
`parent or child of another instance of type 416. Similarly, the
`instances of keyword 410 are interrelated via a keyword
`hierarchy. Keyword 410 can be related to other instances of
`keyword 410 via relationship 442. Relationship 442 identi-
`fies an instance of keyword 410 as a parent or child of
`another instance of keyword 410. Relationship 444 relates
`keyword 410 and type 416. That is, one instance of keyword
`410 is related to an instance of type 416. Conversely, an
`instance of type 416 can be associated with multiple
`instances of keyword 410.
`Further, an instance of keyword 410 can be related to
`many instances of type 416 via relationships 442 and 446.
`That is, an instance of keyword 410 has a type that is
`associated with an instance of type 416. In addition, the
`instance of keyword 410 inherits the types associated with
`the children of its associated instance of type 416.
`Person 418 and person information 426 have a “one-to-
`one” relationship via relationship 428. Person 418 and
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-2 Filed 12/27/11 Page 2 of 25 Page ID #:77
`Case 8:11—cv—01985-DOC—JPR' Document 1-2
`Filed 12/27/11
`Page 2 of 25 Page ID #277
`
`5,813,014
`
`13
`thesaural person 422 are related via relationship 434. Person
`418 can be associated with multiple instances of thesaural
`person 422. An instance of thesaural person 422 can be
`related to multiple instances of person 418 via relationship
`434.
`
`Segment 404 is a container element. That is, as illustrated
`by relationship 428, segment 404 can contain multiple
`instances of phrase 406. Segment 404 is defined by me set
`of elements that it contains. For example, segment 404 is, for
`example, a chapter segment, a testimony segment, or a
`general segment. Instances of phrase 406 can be grouped in
`the order in which they occur in the input data in a chapter
`segment. As a testimony segment, segment 404 contains a
`grouping of instances of 404 associated with the input data.
`For example, a testimony segment can contain all instances
`of segment 404 that are associated with a videotaped inter-
`view. Relationship 450 illustrates the relationship between
`instances of segment 404 (i.e., a testimony segment) that act
`as a container for other instances of segment 404. A general
`segment contains a set of instances of phrase 406 that are not
`necessarily related to particular input data. A general seg-
`ment can be a collection of phrases that meet a certain
`criteria. For example, a general segment can contain
`instances of phrase 406 that are related to an instance of
`keyword 410 having a value of “teacher”.
`Segment 404 therefore identifies a group of catalogue
`elements (e.g., phrase 406. An instance of segment 404 can
`identify all catalogue element instances. Other instances of
`segment 404 can identify a subset of catalogue elements.
`Thus, for example, an instance of segment 404 can identify
`all instances of phrase 406 or a some subset of all of the
`instances of phrase 406. The set including all instances of
`phrase 406 is a catalogue. Asmallcr catalogues that contain
`a subset of all instances of phrase 406 is also a catalogue.
`Within a catalogue, a smaller catalogue can be created by,
`for example, a query operation or user designation.
`Aset of catalogue elements can be identified by querying
`the attribute elements, for example.Aquery operation can be
`performed on the attribute elements to examine other
`attribute elements associated with a catalogue element. A
`query operation identifies a set of cataloguing elements (e.g.,
`instances of phrase 406) that satisfy the criteria specified in
`the query. Aset of cataloguing elements identified in a query
`are grouped in an instance of segment 404. Auser can also
`specify a collection of phrases 406 that can be grouped in an
`instance of segment 404.
`Attributes
`
`FIG. 4Aprovides examples of attributes for catalogue and
`attribute elements according to an embodiment of the inven-
`tion. Segment 404 contains an identifier (ID), a descriptive
`phrase, and a set of phrases, for example. The phrases related
`to an instance of segment 404 are included in the segment
`instance’s set of phrases. A set is formed by creating
`relationships between the elements. FIG. 4B illustrates
`examples of the relationships that exist between elements in
`an embodiment of the invention. The relationships that form
`a set can be implemented using any of the known techniques
`known in the art. For example,
`the relationships can be
`implemented in a programming language using pointers. In
`a relational database management system, for example,.the
`relationships can be formed using relations and primary and
`foreign keys.
`Referring to FIG. 4A, phrase 406 includes an input data
`ID (e.g., identifies the input data hom which the phrase was
`generated), an ID, a descriptive phrase, In/Out timecodes
`(i.e., a corresponding location within the input data), a set of
`keywords, images, persons, proposed keywords, and pro-
`
`10
`
`15
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`14
`posed persons. Keyword 410 includes an ID, and sets of
`types, thesaural keywords, child keywords and parent key-
`words. The child and parent keyword set form relationships
`for the keyword hierarchy. The set of thesaural keywords
`related to keyword 410 contain keyword values or labels for
`keyword instance.
`'
`Person 418 includes an ID, a primary name, an
`occupation, date of birth, and a set of proposed persons.
`Person information 426 contains a person ID for die asso-
`ciated instance of person 418. Person information 426
`contains one or more attributes for the associated instance of
`person 418. The attribute information can vary depending on
`the multimedia information being catalogued. For example,
`the catalogued multimedia data may consist of interviews
`with individuals. An instance of person 418 can be instan-
`tiated and associated with an interviewee. Person informa-
`tion 426 associated with the instance of person 418 can then
`include biographical inforrnation of the interviewee. The
`multimedia data videotaped sporting events. In this case, an
`instance of person 418 can be created for a person associated
`with the sporting event (e.g., player, referee, and broadcast-
`ing personnel). An instance of person information 426
`associated with the instance of person 418 can include
`statistical information associated with the participant.
`An event 408 includes an ID, type (eg., segment, phrase,
`interviewer, videographer, fact, or other), sub-type (e.g., a
`positive, negative, or informational event), timecodes, and a
`comment (or descriptive note).
`Thesaural keyword 412 includes an ID, a keyword [D
`(i.e., the ID for an instance of keyword 410 for which the
`thesaural keyword instance is an alternative), a label (i.e.,
`the value of the keyword instance to which the thesaural
`instance is related), a language of choice identifier (or
`language ID), a preferred flag, and a characteristic (or class).
`If set, the preferred flag specifies that the thesaural keyword
`instance is the preferred alternative for the related keyword
`instance in the language specified by the language ID. The
`characteristic attribute further defines the thesaural keyword
`instance. It can be used to identify that thesaural keyword
`instance is a slang word, for example.
`An ID, timecode and locator are included as attributes for
`image 420. The locator attribute is used to locate the
`digitized image,
`for example. Proposed keyword 414
`includes an ID and a label. It is also possible to include the
`attributes contained in keyword 410 in proposed keyword
`414. Thus, the user that is proposing a new keyword can
`enter as much information regarding the proposed keyword.
`Proposed person 424 includes an ID and name attribute.
`Lilce proposed keyword 414, the attributes associated with
`person 418 can be included in proposed person 424. Type
`416 includes an ID and a label.
`Elements and their relationships can be managed using a
`cataloguing mechanism and a relationship management
`mechanism. The cataloguing mechanism includes a user
`interface that
`includes a series of screens. During
`cataloguing, a user (e.g., a cataloguer) reviews the input data
`and causes elements to be instantiated and associated with
`the input data and other elements. Elements that already
`exist can be associated with the input data during catalogu-
`ing. In addition, a cataloguer can propose new elements and
`relationships. The relationship management facility is used
`to review the elements and relationships proposed by a
`cataloguer. The relationship management facility can also be
`used to create new elements and relationships.
`Browser and Indexing Server Interface
`Browser 318 and indexing server 316 interact to query a
`multimedia catalogue and identify catalogue elements (eg,
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-2 Filed 12/27/11 Page 3 of 25 Page ID #:78
`Case 8:11-cv—01985-DOC—JPR Document 1-2 "Filed 12/27/11
`Page 3 of 25 Page ID #278
`
`5,813,014
`
`15
`instances of phrase 406) that satisfy specified criteria. Inter-
`face 314 provides a protocol to facilitate communication
`between browser 318 and indexing server 316. Interface 314
`includes program interfaces to query the catalogue and
`attribute elements associated with the catalogue. FIG. 5
`illustrates an overview of interface 314 between browser
`318 and indexing server 316 according to an embodiment of
`the invention.
`Interface 314 provides a mechanism for browser 318 to
`communicate request 504 to indexing server 316 and for
`indexing server 316 to transmit data 502 to browser 318 in
`response to request 504. Various groupings of routines are
`provided by interface 314 to query the catalogue and its
`attributes and attribute element instances. One group of
`routines that returns data associated with a general segment
`is file routines 510. File routines 510 includes routines that
`return the number of phrases, keywords,
`thesanral
`keywords, person keywords, or segments in a general seg-
`ment. Other routines return the phrases, keywords, thesaural
`keywords, person keywords or segments in a general seg-
`ment. Routines are provided in segment routines 512 that
`return similar information as those in file 510 for other
`segment types.
`Phrase routines 514 contains routines that retrieve infor-
`mation associated with phrase 406 such as in or out time-
`codes for, or the number or instances of keywords in, an
`instance of phrase 406. Other routines in phrase routines 514
`search the descriptive phrase attribute of phrase 4-06 to
`identify instances of phrase 406 that contain a search string.
`The number or actual instances of phrase 406 that contain
`the search string can be returned to browser 313.
`Keyword group 516 contains routines that identify the
`instance of type 416 associated with an instance of keyword
`410. Routines in this group can return the number or
`instances of segment 404 (e.g., general segment or other
`segment) or phrase 406 that are associated with an instance
`of keyword 410. Other routines identify instances of key-
`word 410 having an associated label or a descriptive phrase
`that contains a Search string.
`The keyword hierarchy is queried using routines in key-
`word hierarchy group 518. Routines in this group can return
`the number or actual instances of keyword 410 that are either
`a parent or child of another instance of keyword 410. Person
`group 520 contains routines for identifying an instance of
`person 418 associated with a general segment or vice versa.
`The number or actual
`instances of person 418 can be
`returned that match first and/or last name search strings.
`Thesaural keyword group 522 contains routines that returns
`attribute information (eg, ID or label for an instance of
`thesaural keyword 412). A search string can be compared
`against the label attribute in instances of thesaural keyword
`412 to return the number or actual instances of thesaural
`keyword 412.
`Instances of type 416 can be queried using type group 524
`and type hierarchy group 526. Type group 524 contains
`routines that return attribute information (eg, label and
`description attributes) for instances of tnue 416. Using other
`routines in this group, a search string can be compared
`against the label or description attribute in instances of type
`416 to return the number or actual instances of type 416. The
`type hierarchy is queried using routines in type hierarchy
`group 526. Routines in this group can return the number or
`actual instances of type 416 that are either a parent or child
`of another instance of type 416.
`The section entitled Browser-Indexing Server Interface
`Routines contains a more detailed discussion of the routines
`contained in each group. It should be apparent that other
`
`10
`
`15
`
`30
`
`35
`
`40
`
`45
`
`5D
`
`55
`
`60
`
`65
`
`16
`groups and routines can be included in interface 314 to query
`and retrieve data from a catalogue.
`Browser
`Browser 318 includes a user interface that includes dis-
`play and input regions. Any user interface can be used with
`browser 318. For example, a graphical user interface can be
`provided that displays values associated with instances of
`type 416 and/or keyword 410. Using the interface, a user can
`enter Search criteria (e.g., a search string) that can be used
`to identify instances of type 416 or keyword 410. The
`instances selected based on the search criteria can be dis-
`played in the user interface for review. A drag and drop
`capability can be used with the user interface to drag criteria
`into a search string. In addition to a user interface for
`entering search criteria, browser 318 includes a user inter-
`face for displaying the multimedia data identified in a Search
`operation. Browser 318 processes search input and multi-
`media display operations. FIG. 6 provides a process flow for
`handling search and display operations according to an
`embodiment of the invention.
`
`At step 602 (i.e., “input”.-"’), a determination is made
`whether input is received by browser 318. If not, processing
`continues at step 602 to wait for input. At step 604 (i.e.,
`"search operation?”), a determination is made whether the
`input is a search operation. If it is, processing continues at
`step 606 to perform the search and returns to step 602 to
`accept other input. If the input is not a search operation,
`processing continues at step 608. At step 608 (i.e., “play
`commandl"), a determination is made whether the display
`input
`is a play command (e.g., forward,
`reverse, fast
`forward, fast reverse, or jrunp). If it is, processing continues
`at step 610 to transmit the play command to archive server
`306 and processing continues at step 602 to await other
`input. If it is not a play command, processing continues at
`step 602 to accept other input.
`Perform Search
`At step 606 of FIG. 6, browser 318 performs a search
`based on a determination (in step 604) that the input is a
`search request. FIGS. 7A-7B provide a process flow for
`processing a search request according to an embodiment of
`the invention.
`_
`The invention stores previous searches and the results of
`the previous searches. The results of a Search form a
`sub-catalogue, or collection of catalogue elements. Thus,
`when a search is entered, processing determines whether the
`search has already been performed. If so, the search results
`are retrieved. Therefore, there is no need to repeat a search.
`If the search is a new search, browser 318 performs the
`search. When search input is received, browser 313 deter-
`mines whether a the type of search requested and initiates
`the search.
`
`_ At step 702 (i.e., “search already exists‘.‘’’), a determina-
`tion is made whether the search specified in the input already
`exists. If the search already exists, processing continues at
`step 704 to identify the instance of segment 404 that was
`created for the search. If no instance of segment 404- is
`found, processing ends at step 736. If an instance of segment
`404 is found, processing continues at step 706 to retrieve it.
`At step 708, the instances of phrase 406 associated with the
`segment instance are identified (e.g., the associated phrase
`ids are retrieved from indexing server 316 using GEt_
`Phrases_In_Segment routine in segment group 512 of
`interface 314). Processing ends at step 736.
`If it is determined at step 702 that the search has not
`already been performed, processing continues at step 722 to
`perfonzn the search. At step 722 (i.e., type?”), the type of
`search is determined by browser 318. The type of search can
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-2 Filed 12/27/11 Page 4 of 25 Page ID #:79
`Case 8:11-cv—01985-DOC—JPR Document1-2
`Fi|ed12/27/11
`Page 4 of 25 Page ID #279
`
`5,813,014
`
`17
`be determined, for example, from the selections made by the
`user when specifying the search criteria. Aperson search can
`be invoked by selecting a “search for person” control button
`and entering a person’s name, for example. [fit is a search
`of the instances of person 418, processing invokes a search
`person flow at step 724. If it is determined that a type or
`keyword search is specified, processing continues at step
`726 to invoke a type/keyword search. If it is determined at
`step 722 that the search involves a search of instances of
`person information 426, processing continues at step 728 to
`invoke a search of instances of person information 426.
`Asearch invoked in steps 724, 726, or 728 can return a set
`of phrase ids (none or more phrase ids). Adetermination is
`made at step 730 (ie, “phrase id(s) identified?”), whether
`the Search operation yielded any instances of phrase 406. If
`not, processing ends at step 736. If so, processing continues
`at step 732 to create a segment instance. Instances of
`relationship 428 are formed between the segment instance
`and the instances of phrase 406 identified in the search
`operation. At step 734, the search criteria is stored in query
`elements. These elements can be queried in a subsequent
`invocation of this process flow (at step 702) to determine
`whether the search already exists. Search elements are
`discussed in more detail
`in the section entitled Search
`
`Elements and Objects. Processing ends at step 736.
`Type or Keyword Search
`In addition to a person search, browser 318 can perform
`a type and/or keyword clement search. If browser 318
`determines at step 722 of FIG. 7B that the search operation
`is a type or keyword element search, browser 318 invokes
`the search. FIGS. 8A—8B provide a type and keyword search
`process flow according to an embodiment of the invention.
`The search request identifies the search criteria used to
`identify the instances of type 416 or keyword 410 (e.g., the
`search input specifies values for the label attributes of type
`416 or keyword 410). Browser 318 can be used to display
`instances of type 416 and/or keyword 410. Adrag and drop
`facility can be used to drag an instance into the search. A
`conjunction (e.g., “and” or “or”) can be selected to combine
`more than one instance. Each instance of a type 416 or
`keyword 410 that is dragged into a search represents an
`element of the search. Each element of the search criteria is
`processed to identify the instances of type 416 or keyword
`410 that satisfy the search request. At step 802 (i.e., “all
`search elements processed‘?”), a determination is made
`whether all of the search elements have been processed. If
`not processing continues at step 894 to get the next element
`of the search criteria (e.g., a potential label attribute value).
`At step 806, the search element is compared against the
`corresponding attribute (e.g., label attribute) of the element
`instances being searched (i.e.,
`instances of type 416 or
`keyword 4-10).
`At step 810 (i.e., “found?”), a determination is made
`whether any element instances were found to match the
`search element, or criteria. If not, processing continues at
`step 802 to process any remaining search elements. If
`element instances were found (e.g., one or more i.nstances of
`keyword 410 met the criteria specified in the current search
`element), processing continues at step 812 to determine
`whether the element instances are already contained in a set
`of found element instances. If so, processing continues at
`step 802 to process any remaining search elements. If not,
`processing continues at step 814 to add the element
`instances to the search set and processing continues at step
`802 to process any remaining search elements.
`If it is determined at step 802 that there are no more search
`elements, processing continues at step 816 to identify ele-
`
`10
`
`15
`
`25
`
`35
`
`4D
`
`50
`
`S5
`
`60
`
`65
`
`18
`ment instances contained in a hierarchy of element instances
`(e.g., subtypes of instances of type 416 or child instances of
`keyword 410). At step 816, element instances that are child
`instances of the element instances contained in the search set
`are identified. At step 818, the child instances are added to
`the search set, if they are not already contained in the set.
`Processing ends at step 820.
`Related Keyword/'i‘ype Search
`Relationship 444 exists between keyword 410 and type
`416 (see FIG. 4B). Relationship 444 can be used to identify
`a second search set using the Search set created in FIG. 8A.
`Using relationship 444, a set of keyword instances can be
`identified from a set of types and vice versa. For example,
`if the search operation performed in FIG. 8A yielded a
`search set that contains one or more instances of type 416,
`relationship 444 can be queried to obtain a set of instances
`of keyword 410 associated (via instances of relationship
`444) with instances of type 416 contained in the search set.
`Similarly, instances of type 416 can be identified, using
`relationship 444, from a search set created in FIG. 8A that
`contains instances of keyword 410. FIG. 8B contains a
`process flow for creating a second search set using the search
`set containing instances of type 416 or keyword 410 and
`relationship 444.
`At step 822 (Le, “all elements in set processed”), a
`determination is made whether all of the element instances
`in the search set are processed. If not, processing continues
`at step 824 to get the next element instance (e.g., an instance
`of type 416 or keyword 410). At step 826, the element
`instance associated with the current element instance via
`relationship 444 is located. For example, an instance of
`keyword 410 is identified that is associated (via an instance
`of relationship 444) with an instance of type 416 in the
`search set. Or, an instance of type 416 is identified that is
`associated with an instance of keyword 410 in the search set.
`At step 828 (i.e., “element instance found?”), a determi-
`nation is made Whether the element instance is found in step
`826. If not processing continues at step 822 to process any
`remaining search set elernent instances. If so, processing
`continues at step 830 (i.e., element instance found in set?" ,
`to determine whether the related element instance already
`exists in the related clement search set being created. If so,
`processing continues at step 828 to process any remaining
`search set elements. if it does not already exist in the related
`clement search set, processing continues at step 832 to add
`the related element to the related clement search set. Pro-
`cessing continues at step 822 to process any remaining
`elements in the search set.
`If it is determined at step 822 that all elements in the
`search set have been processed, processing continues to
`search the related elements’ element hierarchy (e.g., the type
`hierarchy, if the related element search set contains instances
`of type 416). At step 836, element instances that are chiid
`instances of the related element instances contained in the
`related clement search set are identified. At step 838, the
`child instances are added to the related clement search set,
`if they are not already contained in the set. Processing ends
`at step 840.
`The process flows of FIGS. 8A—8B can yield instances of
`keyword 410. A set of keyword instances can be used to
`identify a set of catalogue elements (e.g., phrase 4-06) that
`can be used to reference the multimedia data. FIG. 8C
`provides a process flow for identifying instances of phrase
`406 that are associated with instances of keyword 410
`contained in a search set created by a search operation
`according to an embodiment of the invention.
`At step 850 (i.e., “all keywords in set processed‘?”), a
`determination is made whether there are any instances of
`
`

`
`Case 8:11-cv-01985-DOC-JPR Document 1-2 Filed 12/27/11 Page 5 of 25 Page ID #:80
`Case 8:11-cv—01985-DOC—JPR Document1-2
`Fi|ed12/27/11
`Page 5 of 25 Page ID #280
`
`5,813,014
`
`19
`keyword 410 that remain to be processed. If not, processing
`ends at step 860. If so, processing continues at step 852 to
`get the next instance of keyword 410 in the search set. At
`step 854, a search is performed to identify instances of
`phrase 406 that are related (via relationship 430) to the
`current
`instance of keyword 410. For example, Get“
`PhraSes_l.n__Testin1ony_Keyword can be called in key-
`word group 516 of interface 314 for each testimony instance.
`At step S56 (i.e., “phrase instance found not in phrase set?"),
`a determination is made Whether the instances of phrase 406
`identified in step 852 already exist in the set of phrases. If
`so, processing continues at step 850 to process any remain-
`ing keyword instances. IE not, processing continues at step
`858 to add the phrase instance(s) found in step 854 in the set
`of phrases. Processing continues at step 850 to process any
`remaining keyword instances.
`Person Information Search
`Person information 426 contains attribute information for
`instances of person 418. For example, person information
`426 can contain bi.rth date and place attribute information for
`an instance of person 418. A search operation can be
`performed on person information 426 to identify instances
`of person 418 having certain attributes. An instance of
`person 418 can be associated with an instance of segment
`404 (e.g., a general segment) via relationship 432. The
`instance of segment 404 associated with the instance of
`person 418 can be used to identify the instances of phrase
`406 contained within the segment
`instance. Further, an
`instance of person 418 can be used to identify associated
`instances o