United States Patent (19)
`Stelovsky
`
`I||||I||||III
`USOO5613909A
`5,613,909
`11
`Patent Number:
`Mar. 25, 1997
`(45) Date of Patent:
`
`54 TDME-SEGMENTED MULTIMEDIA GAME
`PLAYING AND AUTHORNG SYSTEM
`
`76 Inventor: Jan Stelovsky, 2049 Oswald St.,
`Honolulu, Hi. 96816
`
`21 Appl. No.: 278,946
`22 Filed:
`Jul. 21, 1994
`(51) Int. Cl. ............................................. A63F 9/22
`52 U.S. Cl. ................................... 463/1, 463/30; 463/40;
`434/307 A; 434/309; 434/323; 434/327
`58) Field of Search ..................................... 273/433,434,
`273/435, 436, 437, 439, 85 G, DIG. 28,
`460, 430, 432, 434/307 A, 307, 308, 309,
`315, 316, 323,327; 364/412
`
`56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`4,305,131 12/1981 Best ........................................ 364.f410
`4,360,345 1/1982 Hon .....
`434/323
`4,634,386
`1/1987 Tanaki ...............
`434/323
`434/309
`4,712,180 12/1987 Fujiyama et al...
`4,798,543
`1/1989 Spiece ................
`... 434/323
`... 434/323
`5,065,345 11/1991 Knowles et al. ...
`5,106,097 4/1992 Levine .................................... 273f460
`5,250,747 10/1993 Tsumura .....
`434/307 A
`5,267,865 12/1993 Lee et al. ................................ 434/323
`5,377,997
`1/1995 Wilden et al. .
`... 273f434
`5,393,073 2/1995 Best ........................................ 273f434
`5,395,123
`3/1995 Kondo ....
`... 434/307 A
`5,397,853 3/1995 Koguchi .............................. 434/307 A
`
`5,447.438 9/1995 Wantanabe et al................. 434/307 A
`Primary Examiner-Jessica Harrison
`Assistant Examiner Mark A. Sager
`Attorney, Agent, or Firm-Leighton K. Chong
`57)
`ABSTRACT
`An interactive system is provided for playing a game,
`educational or instructional sequence in conjunction with a
`prerecorded multimedia presentation consisting of at least
`motion video, sound and accompanying text. The multime
`dia presentation is partitioned into time segments. The game
`sequence is also partitioned into task units that are synchro
`nized with the time segments. The time address markers for
`the beginning and ending of each of the time segments
`initialize, execute and close each task unit of a programmed
`game sequence. A corresponding authoring system allows an
`author to design and implement the time-segmentation of the
`multimedia presentation and its functional interface with the
`game sequence. This system can be used for motivating
`instructional games based on music video, movies, animated
`cartoons and karaoke that provide the player with the same
`excitement and challenge of the popular video games. The
`games can contain educational contents and be used for
`teaching and training reading comprehension and foreign
`language instruction. Similarly, motivating tests based on
`this concept can accompany training in other subjects. The
`provided multimedia games can be the basis of many
`products in the emerging "edutainment' industry that con
`bines education and entertainment.
`
`16 Claims, 18 Drawing Sheets
`
`camera WCR
`
`other input and
`output devices
`
`pointing device
`
`laser disk
`
`
`
`segment 1
`
`ent N
`
`t0
`
`t
`
`tN
`
`te
`
`microphone
`
`P
`R
`E
`video
`S
`audio
`E
`N
`text
`T
`A other
`media
`
`O
`N
`
`choices
`
`-1-
`
`Amazon v. Audio Pod
`US Patent 9,319,720
`Amazon EX-1044
`
`

`

`U.S. Patent
`
`Mar. 25, 1997
`
`Sheet 1 of 18
`
`5,613,909
`
`camera f WCR
`
`other input and
`output devices
`
`f
`
`
`
`
`
`s
`
`Convert
`
`recorder
`
`Q
`\
`microphone
`op
`
`a
`
`modem
`r-,
`
`cin/ S
`as,
`
`A
`
`TW
`
`A7
`pointing device .
`
`277
`7G-sy
`laser disk
`
`FIG. 1
`
`segment 1
`
`segment N
`
`P
`
`video
`audio
`S
`E
`N text
`T
`A other
`T
`media
`9
`
`time
`
`G
`
`N Y
`
`N N
`
`input
`
`
`
`feedback
`
`-2-
`
`

`

`U.S. Patent
`
`Mar, 25, 1997
`
`Sheet 2 of 18
`
`5,613,909
`
`
`
`display elements
`fetch presentation
`
`
`
`
`
`FIG 3a
`
`-3-
`
`

`

`U.S. Patent
`
`Mar. 25, 1997
`
`Sheet 3 of 18
`
`5,613,909
`
`
`
`
`
`
`
`highlight text (I)
`T = T(1)
`play segment (I)
`
`
`
`
`
`play resources (I)
`
`FIG. 3b
`
`-4-
`
`

`

`U.S. Patent
`
`Mar 25, 1997
`
`Sheet 4 of 18
`
`5,613,909
`
`START GAME
`
`END GAME
`
`
`
`eS
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG. 3C
`
`-5-
`
`

`

`U.S. Patent
`
`Mar 25, 1997
`
`Sheet 5 of 18
`
`5,613,909
`
`record
`ecord video \\record audioWWother media VW enter text
`
`enter
`CSOLICeS
`
`synchronize
`
`P R
`O
`P
`
`b
`N
`
`
`
`a
`
`(o)
`D
`mass storage
`R
`U (e.g. cd-rom)
`BS U
`T A
`G
`O
`
`
`
`
`
`text track
`
`segment
`text
`
`segmented
`presentation
`
`N
`segment
`eSOl CeS
`
`l
`
`
`
`MULTIMEDIA GAME
`
`FIG 4
`
`-6-
`
`

`

`U.S. Patent
`
`Mar. 25, 1997
`
`Sheet 6 of 18
`
`5,613,909
`
`
`
`Lyrics
`
`Operation Area
`
`FIG. 5
`
`-7-
`
`

`

`U.S. Patent
`
`Mar. 25, 1997
`
`Sheet 7 of 18
`
`5,613,909
`
`
`
`DREAM
`LAWYER
`Marie Kara-Oke
`
`somebody
`desperately
`
`...
`...
`...
`
`| ||
`
`|
`|
`|
`|
`I need a lawyer to give me
`The -law
`That will - always
`
`...
`
`...
`
`- EE
`
`FIG. 6
`
`-8-
`
`

`

`U.S. Patent
`
`Mar. 25, 1997
`
`Sheet 8 of 18
`
`5,613,909
`
`
`
`
`
`
`
`
`
`paused?
`
`O
`pause music video
`
`
`
`
`
`FIG 7a
`
`-9-
`
`

`

`U.S. Patent
`
`Mar 25, 1997
`
`Sheet 9 of 18
`
`5,613,909
`
`START GAME
`
`END GAME
`
`fetch tiles (I)
`display tiles (I)
`Score = 0
`display Score
`T = T(0)
`- O
`start music video
`
`I = +1
`
`highlight text (I)
`
`E (G)
`
`O
`
`O
`
`G
`
`yes
`
`eS
`
`V = value (K, I, T)
`
`
`
`
`
`yes
`hide tile (K)
`
`display Score
`
`
`
`FIG. 7b
`
`-10-
`
`

`

`U.S. Patent
`
`Mar 25, 1997
`
`Sheet 10 of 18
`
`5,613,909
`
`
`
`
`
`
`
`START
`RECORD (I)
`
`
`
`
`
`<G>
`
`yes
`stop recording (I)
`
`
`
`END
`RECORD (I)
`
`
`
`
`
`
`
`
`
`
`
`
`
`T = T(0)
`
`play song
`with/out singer's voice
`
`highlight text (I)
`run RECORD (I)
`dehighlight text (I)
`
`
`
`O
`run FEEDBACK
`
`END
`KARAOKE
`
`
`
`
`
`
`
`
`
`
`
`FIG. 8a.
`
`-11-
`
`

`

`U.S. Patent
`
`Mar 25, 1997
`
`Sheet 11 of 18
`
`5,613,909
`
`
`
`
`
`
`
`START
`RECORDING (I)
`
`T = T(I)
`play song from T(I)
`without singer's voice
`play recording (I)
`
`(OD O
`
`yes
`stop recording (I)
`
`
`
`
`
`END
`RECORDING (I)
`
`
`
`
`
`
`
`
`
`
`
`START
`FEEDBACK
`
`
`
`
`
`highlight text (I)
`run RECORDING (I)
`
`
`
`O
`
`compare
`Oeaker2
`
`run SPEAKER (I)
`
`
`
`compare
`inger?
`
`run SINGER (I)
`
`dehighlight text (I)
`
`
`
`O
`
`END
`FEEDBACK
`
`FIG. 8b
`
`-12-
`
`

`

`U.S. Patent
`
`Mar. 25, 1997
`
`Sheet 12 of 18
`
`5,613,909
`
`START
`SPEAKER (I)
`
`
`
`
`
`T = T(I)
`start native speaker's voice (I)
`
`
`
`<G>
`
`yes
`stop voice (I)
`
`
`
`
`
`START
`SONG (I)
`
`
`
`
`
`
`
`T = T(I)
`play song from T(T)
`with singer's voice track
`
`(G>.
`
`yes
`
`
`
`
`
`
`
`
`
`END
`SPEAKER (I)
`
`
`
`END
`SONG (I)
`
`FIG. 8c
`
`-13-
`
`

`

`U.S. Patent
`U.S. Patent
`
`Mar. 25, 1997
`Mar. 25, 1997
`
`Sheet 13 of 18
`Sheet 13 of 18
`
`5,613,909
`5,613,909
`
`START
`
`
`
`
`
`REMOTE
`
`
`
`
`
`
`
`
`no
`
`yes
`
`discard movie
`
`
`FIG. 9a
`FIG. 9a
`
`-14-
`
`-14-
`
`

`

`U.S. Patent
`
`Mar. 25, 1997
`
`Sheet 14 of 18
`
`5,613,909
`
`START
`REMOTE
`GAME
`
`END
`REMOTE
`GAME
`
`hide text (I)
`
`O
`
`hide tile (I, K)
`O
`
`
`
`
`
`
`
`I = --1
`display text (I)
`
`
`
`
`
`display tile (I, K)
`
`yes
`
`
`
`FIG 9b.
`
`-15-
`
`

`

`U.S. Patent
`
`Mar 25, 1997
`
`Sheet 15 of 18
`
`5,613,909
`
`Movie fills the screen
`
`
`
`Score
`
`Dialog
`
`Choice
`
`FIG 10
`
`
`
`management of segments
`add
`
`editing of boundaries
`
`management of choices
`
`link feedback
`link text
`
`-16-
`
`

`

`U.S. Patent
`
`Mar. 25, 1997
`
`Sheet 16 of 18
`
`5,613,909
`
`SEGMENTS
`it frames:
`
`
`
`range: label:
`
`EVENTS
`in range:
`
`original:
`
`
`
`
`
`TXT TRACK
`original transcript:
`I need a lawyer to give me
`the kind of law
`that lasts always
`
`transcript with blanks:
`a lawyer to give me
`law
`the
`that - always
`
`VIDEO FILE:jan:Top 10careyavi
`
`duration: 125520 volume 255
`
`
`
`FIG. 12
`
`-17-
`
`

`

`U.S. Patent
`
`Mar 25, 1997
`
`Sheet 17 of 18
`
`5,613,909
`
`Current Segment
`start point
`replay limit
`
`end point
`
`- N -
`
`St Adjacent Segment end
`
`FIG. 13a
`
`Current Segment
`replay limit
`end
`-1
`
`Start
`A
`
`St Adjacent segment end
`
`FIG. 13b
`
`
`
`11520
`
`14839 O at least
`Oatmost
`
`replay
`O segment
`Odelta(s) 16
`O from/to
`
`FIG. 14
`
`-18-
`
`

`

`U.S. Patent
`
`Mar. 25, 1997
`
`Sheet 18 of 18
`
`5,613,909
`
`
`
`
`
`
`
`Update Segment
`New Segment
`Delete Segment
`Init Segments
`Update Event
`New Event
`Delete Event
`Init Events
`Associate
`Check
`
`FIG. 15
`
`-19-
`
`

`

`1.
`TIME-SEGMENTED MULTIMEDIA GAME
`PLAYING AND AUTHORNG SYSTEM
`
`5,613,909
`
`10
`
`15
`
`20
`
`25
`
`1. BACKGROUND OF THE INVENTION
`Movies, music videos and karaoke systems are some of
`the most successful forms of modern entertainment. Movies
`and music videos are essentially pre-recorded multimedia
`presentations with several synchronized tracks. A track is a
`recorded sequence of specific media type, such as motion
`video, audio, and text. (Examples of text tracks include
`movie subtitles and closed captioning.)
`Karaoke systems are also multimedia presentations with
`prerecorded video and music audio tracks synchronized with
`a text display of lyrics, allowing the user to create a new
`vocal track in real time by singing into the microphone.
`Video and electronic games are another popular enter
`tainment form which uses video images, sound, and some
`times text to challenge a player to successfully execute a
`winning sequence of actions within a given time. Most video
`games impose time limits on the user's input. Often, the
`sequencing of video scenes is determined by the player's
`input. Some computerized games have been tied to real time
`constraints, such as for playing simultaneously with real
`time events.
`Other types of computerized games used for educational
`purposes include narrative or task sequences based on
`multiple choice, composition, fill-in-the-blanks or other
`related tasks. There are also instructional games based on
`motion video presentation which are accompanied by a text
`track, such as family of products from HyperGlott M or
`subtitled films used for language instruction.
`The object of the present invention is to combine the high
`entertainment value of movies, music videos, and other
`forms of multimedia presentations with the functional value
`of video and computerized games for educational and
`instructional tools.
`
`30
`
`35
`
`45
`
`2
`2.1. DESCRIPTION OF THE DRAWINGS
`The invention is described below in conjunction with the
`appended drawings, as follows:
`FIG. 1 shows the overall system architecture.
`FIG. 2 shows the linking of multimedia presentation to the
`multimedia game.
`FIG. 3a shows the overall control flow diagram of basic
`Multimedia Game.
`FIG. 3b shows the control flow diagram of explore mode
`in basic Multimedia Game.
`FIG.3c shows the control flow diagram of game mode in
`basic Multimedia Game.
`FIG. 4 shows the basic production process of the Multi
`media Game.
`FIG. 5 shows the screen layout of the Music Video Game.
`FIG. 6 shows the screen snapshot from the Music Video
`Game.
`FIG. 7a shows the control flow diagram of explore mode
`in Music Video Game.
`FIG.7b shows the control flow diagram of game mode in
`Music Video Game.
`FIG. 8a shows the global control flow diagrams of game
`mode in Karaoke Game.
`FIG. 8b shows the control flow diagrams of game mode
`in Karaoke Game (cont.).
`FIG. 8c shows the control flow diagrams of game mode
`in Karaoke Game (cont.).
`FIG. 9a shows the overall control flow diagram of Remote
`Movie Game.
`FIG.9b shows the control flow diagram of game mode in
`Remote Movie Game.
`FIG. 10 shows the screen layout of the Remote Movie
`Gain.
`FIG. 11 shows the modules of Segmentation Authoring
`System.
`FIG. 12 shows the screen snapshot from the Segmentation
`Authoring System.
`FIG. 13a shows the graphical pane and its icons.
`FIG. 13b shows the graphical pane when editing the end
`point.
`FIG. 14 shows the lower panes of the segment specifica
`tion area.
`FIG. 15 shows the menu commands of Segmentation
`Authoring System.
`
`2. SUMMARY OF THE INVENTION
`The present invention provides an interactive system for
`playing a game, educational or instructional sequence in
`conjunction with a prerecorded multimedia presentation
`consisting of at least motion video, sound and accompany
`ing text. The multimedia presentation is partitioned into time
`segments. The game sequence is also partitioned into task
`units that are synchronized with the time segments. The time
`address markers for the beginning and ending of each of the
`time segments initialize, execute and close each task unit of
`a programmed game sequence.
`This system can be used for motivating instructional
`games based on music video, movies, animated cartoons and
`karaoke that provide the player with the same excitement
`and challenge of the popular video games. These games can
`contain educational contents and be used for teaching and
`training reading comprehension and foreign language
`instruction. Similarly, motivating tests based on this concept
`can accompany training in other subjects. The invention can
`be the basis of many products in the emerging "edutain
`ment' industry that combines education and entertainment.
`The invention also includes a corresponding authoring
`system which allows an author to design and implement the
`time-segmentation of the multimedia presentation and its
`functional interface with the game sequence.
`
`50
`
`55
`
`60
`
`65
`
`3. DETAILED DESCRIPTION OF INVENTION
`3.1. Overall System Architecture
`Referring to FIG. 1, the overall system architecture pref
`erably includes:
`computer terminal or game player component
`mass storage:
`hard disk
`CD-ROM or tape
`multimedia input sources, including:
`VCR+video frame digitizer
`CD-ROM
`audio player/recorder
`laser disk player
`keyboard or other means of text entry
`microphone
`
`-20-
`
`

`

`5,613,909
`
`3
`sound or music input device with audio digitizer
`input/game controls:
`keyboard
`mouse, joystick or other pointing device
`voice command microphone, speech digitizer
`output/display devices:
`monitor
`audio speakers
`printer
`recorder, VCR, audio
`This system architecture can be used for production as
`well as for replay. The computer can also be in the form of
`a game player (e.g. SegaTM) connected to a TV monitor. The
`computer preferably incorporates hardware that can either
`digitize audio and video (e.g. audio and frame digitizer
`boards) or play it directly through the speakers and in a
`window on the screen or a separate video monitor. If the
`multimedia game is used without recording, the camera/
`video/audio recording equipment and microphone compo
`nents are not needed. Depending on the storage medium
`used for distributing the game, a CD-ROM drive, hard disk,
`laser disk player or another device is used. If broadcast
`services or network distribution is used, then a modem,
`Ethernet, or connection to an electronic gateway is provided.
`3.1 Time Segmenting of Multimedia Presentation
`In the basic process of this invention, a multimedia
`presentation is an arbitrary network of prerecorded media
`sequences where at least one sequence is dependent on time
`(i.e. the sequence is clocked while the presentation is
`playing.) The time-dependent sequence is composed of
`tracks that are synchronized with respect to a common time
`axis (hereinafter "multimedia presentation”). The basic track
`consists of video display images and is synchronized with at
`least one other track that consists of audio or text display.
`The types of video tracks include, but are not limited to,
`motion video, animation, or a sequence of still images, such
`as a slide show. The types of audio tracks include, but are not
`limited to, speech (such as dialog), song, or music. The types
`of textual track include, but are not limited to, the transcript
`of a sound track, such as transcript of movie dialog, song
`lyrics, a text narrative, or other textual works.
`The synchronization among the individual tracks can be
`either continuous or discrete. In continuous synchronization,
`a track starts at a given point of time and continues until a
`given point of time is reached (e.g. motion video and audio).
`In discrete synchronization, a media asset is displayed at a
`given point of time (e.g. still image appears, text section is
`displayed or highlighted) and its display is altered at another
`given point of time (e.g. still image disappears, text section
`is erased or de-highlighted).
`The multimedia presentation is segmented with respect to
`specific beginning and ending points of segments on the
`time axis, i.e. there are one or more points of time that
`partition the time axis into time segments. Typically, the
`multimedia presentation can be played either as a whole or
`in a segment-dependent fashion, for instance, when a pause
`or another media is played in between two time segments, or
`when the user is allowed to select and play an arbitrary time
`segment.
`Other tracks of any other media format, such as motion
`video, audio, sequence of still images, or text can be
`associated with a multimedia presentation and be synchro
`nized with respect to the presentation's time or its segments
`or be independent of its time axis. Such tracks can be
`provided or input by the user. An example of user's own
`input is recorded speech, video, or sound.
`
`45
`
`50
`
`55
`
`60
`
`65
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`4
`3.2. Multimedia Game Playing System
`For purposes of this invention, a game, teaching, or
`instructional tool (hereinafter "multimedia game”) is a soft
`ware program that accepts a user's input during the time a
`time-segmented multimedia presentation is played, inter
`prets the input as a selection among possible choices,
`evaluates the choice and provides the user with feedback
`about the value of the choice s/he has selected.
`Referring to FIG. 2, the basic functional linking of a
`time-segmented multimedia presentation and a game is
`accomplished in the following way. The multimedia game
`has two layers, 1) the multimedia presentation, and 2) the
`game program. The presentation has tracks that are synchro
`nized and partitioned into N time segments. Without loss of
`generality, one can assume that the time segments are
`contiguous, i.e. each time segment I starts at time T(I-1) and
`ends at T(I), and the presentation starts at time T(0) and ends
`at T(n). (If there is a gap in between two time segments or
`two segments overlap, one can introduce an additional time
`segment that represents the gap or the overlap.)
`By linking to the starting and ending times of each time
`segment of the presentation played, the game layer presents
`to the user the choices associated with each time segment. It
`then interprets the user's input as a selection of a choice,
`evaluates the selected choice and gives the user a feedback
`that reflects the resulting value.
`Referring to FIG. 3a, control of the multimedia game
`proceeds in the following steps. First, initial screen elements
`are displayed and the multimedia game is retrieved and
`initialized. Then the user can choose to enter the "explore'
`mode or the "game” mode (see below). Note that while the
`"explore" mode can give the user a preview of the presen
`tation, it does not need to be included in the multimedia
`game. Finally, when the user decides to quit, the multimedia
`game is terminated.
`Referring to FIG.3b, the explore mode of the multimedia
`game proceeds in the following way. First, the section of the
`text linked to the time segment is highlighted, and the time
`segment of the presentation is retrieved and played. If there
`is no user input, the presentation continues playing until its
`end is reached. Then the text is de-highlighted and the
`presentation proceeds with the next time segment. If the user
`does not interact with the game, all of the time segments of
`the presentation will be played in sequence. The user can,
`however, interrupt the play and choose an arbitrary time
`segment. This can be realized, for instance, by clicking on
`any desired text section. Then an associated media asset (e.g.
`native speaker's pronunciation of the segment's text) can be
`played and the presentation resumes starting with the
`selected segment. Other forms of interaction with the user
`are possible, such as playing only one segment at a time or
`playing the time segments in randomly shuffled order.
`The control flow diagram in FIG. 3c illustrates the core
`multimedia game-playing algorithm for an example of a
`game in which a user selects a choice related to the text
`accompanying a multimedia presentation. It depicts the case
`that time segments are played in sequence without interrup
`tion. At the beginning of each time segment, the linked
`section of the text track is highlighted, and the choices
`available to the user for this segment are fetched. Then these
`choices can be presented to the user and the time segment of
`the presentation starts playing. While the time segment of
`the presentation is played, the multimedia game accepts user
`input and interprets it as a selection of a choice. The game
`program then fetches the value of the selected choice with
`respect to the current time segment and gives the user a
`sensory feedback of this value. Once the time segment
`
`-21-
`
`

`

`5
`
`10
`
`15
`
`20
`
`25
`
`S
`reaches its ending point, the text is de-highlighted and the
`game proceeds with the next time segment.
`If choices are presented to the user, they either can be
`displayed on the monitor or played as sound. For instance,
`the choices can be displayed as a menu in the "game' area
`of the screen. While at present only visual and audio output
`are available, any future sensory output is feasible. The
`choices can be presented either in sequence or several at a
`time. They can dynamically change according to time or the
`user's input. The set of choices can vary depending on the
`segment currently played and can include "distractor
`choices that do not fit into the current context.
`The user can input in numerous ways, e.g. click with a
`mouse on a choice within the menu, type a key that identifies
`a choice, select a choice with a joystick or pointer, push a
`software or hardware button while a choice is presented,
`identify a choice using a microphone or any other input
`device. The input can also be continuous, as in the recording
`of the user's voice.
`The evaluation of user's choices usually depends on the
`duration of the current segment. (i.e. the value assigned to at
`least one user's choice changes either at the start time or at
`the end time of at least one segment.) Each choice is
`evaluated with respect to the current time and segment
`index.
`The evaluation can result in one or more numbers, or an
`identification of a media element (e.g. image or sound), or
`simply in either a "correct' or an "incorrect' feedback, or
`any combination there of. The value of the user's choice can
`also depend on other factors, such as the history of the user's
`performance. For instance, if several choices are related to
`the same segment, selecting a choice might influence the
`evaluation of subsequent choices, e.g. the user might be
`required to select the choices in a certain order to receive the
`maximum score or an incorrect choice might decrease the
`value of the next correct choice.
`The feedback to the user can be visual, such as incre
`menting a score or uncovering part of an image. It can be
`audio, such as a fanfare music, or any other sensory input.
`Note that a value that might be identified by omission of
`feedback, for instance the game program might offer feed
`back for all values except for the "incorrect' value. The user
`then can interpret the lack offeedback as an evaluation of his
`or her choice. The feedback can be tied to the multimedia
`presentation and influence its timing and sequencing. For
`instance, the current track can be repeated, the text track can
`be modified to reflect the user's choice or another media
`track can be played before the next segment starts playing.
`There are several other variations on the schema depicted
`in FIG. 3c. For instance, the sequence of time segments can
`be shuffled instead of being played in its natural order. The
`choices can be displayed one at a time instead of in a menu.
`Also, all choices can be displayed in the beginning of the
`game. Choice of several variants can be offered to the user
`to reflect the difficulty of the game. Moreover one or more
`tracks of the presentation can be altered within the game
`mode.
`3.3. Authoring System for Linking the Multimedia Game to
`Its Presentation
`The construction of a time-segmented multimedia pre
`sentation and its linking with the multimedia game can be
`greatly simplified using a dedicated authoring system (Seg
`mentation Authoring System or "SAS' hereinafter) that
`facilitates the identification of the points in time where a
`segment starts and the point where it ends. Such a system
`can also simplify the specification of additional resources
`linked to each time segment, the specification of user's
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`5,613,909
`
`6
`choices linked to each segment, and the value and feedback
`linked to each choice.
`Conventional video editing tools such as Premiere
`produced by Adobe Systems, Inc. of Mountain View, Calif.,
`and products of Avid Technology, Inc., of Tewksbury, Mass.,
`allow the author to synchronize moving images and sound
`tracks. Several of these tools allow the author to specify and
`export “Edit Decision Lists' (EDL) that define start and end
`points ("in" and "out” points) of a scene in time units (such
`as SMPTETM time code). These tools aim at composing
`video and audio segments into one linear sequence rather
`than splitting one linear sequence into consecutive time
`segments. Therefore, they do not facilitate the editing of a
`boundary point in relation to its neighbor time segment.
`While these tools allow the author to replay only one time
`segment and some limit the replay to the "working space',
`they do not relate the replay time to the boundary point being
`edited. These tools also lack provisions for synchronization
`of EDLs with other resources, such as intervals of a separate
`text or other media types or events. Multimedia authoring
`tools such as Authorware M or DirectorTM., produced by
`MacroMedia of San Francisco, Calif., allow the author to
`place events on a time axis (called "score”), but they have
`no facilities to associate events directly with time segments.
`Therefore, a modified user interface is used here for the
`task of defining the start and end points of segments. For
`instance, the task of splitting human speech into segments is
`particularly sensitive to the exact determination of those two
`boundary points.
`Referring to FIG. 4, the authoring production of a mul
`timedia game proceeds in the following steps.
`1) The author produces a synchronized multimedia pre
`sentation. The recording and synchronization can be
`accomplished with well known hardware and software
`systems:
`audio and video can be recorded with microphone, tape
`recorder and camcorder, such as these produced by
`SonyTM Corp. of Tokyo, Japan
`audio and video can be digitized using a video grabber
`board (e.g. Media 100TM from Data Translations, Inc.
`of Marlboro, Mass.) or recorded in analog form on a
`laserdisk (as provided by laser disk service bureaus)
`if necessary, audio and video tracks can be synchro
`nized using well known linear editing systems such
`as PremiereTM produced by Adobe Systems, Inc. of
`Mountain View, Calif.
`the digital components of multimedia can be recorded
`on a CD-ROM (e.g. using CD-ROM burner available
`from Eastman Kodak Co. of Rochester, N.Y.)
`on-line services provide downloading of files (e.g.
`Prodigy, Compuserve or Internet).
`2) Using the SAS, the author partitions the multimedia
`presentation into time segments according to predomi
`nant time units, e.g. measures of Song, sound bites, or
`action sequences in a movie.
`3) Sections of a text track are linked to the time segments.
`4) Additional media resources are linked to each of the
`time segments.
`5) Sets of choices available to the user are specified and
`linked to each of the time segments.
`6) Values are linked to the choices.
`7) A feedback is linked to the value assigned to each
`choice.
`8) The multimedia game is recorded onto a mass storage
`media, such as CD-ROM.
`9) The multimedia game is distributed to the user via retail
`outlets, broadcast channels, or on-line services.
`
`-22-
`
`

`

`5,613,909
`
`10
`
`15
`
`7
`10) The multimedia game is played by the user.
`The box labeled "Segmentation Authoring Tool” in FIG.
`4 outlines the role that the SAS plays in the production
`process (steps 2 to 7).
`The SAS is particularly useful for editing a time-seg
`mented presentation to obtain a optional multimedia game.
`As shown in FIG. 12, SAS displays the set of segments
`already specified and ordered according to their start points.
`It allows the author to select one current segment from the
`sequence of segments. The current segment can be then
`edited. The author can insert a new segment at an arbitrary
`position in the sequence (e.g. before or after the current
`segment), and delete the current segment.
`To facilitate editing an individual segment, the system
`displays the current segment's start and end points sepa
`rately. This way, the author can select which one of these
`“boundary points' s/he wants to edit. The coordinate of the
`current boundary point can be moved with respect to the
`time axis. Since the end point of a segment is closely related
`to the start point of the next segment, the SAS' user interface
`allows the author to ensure that changes of the boundary
`point of one segment are immediately reflected in the
`changes of the corresponding boundary point of the adjacent
`segment. The relationship between the two corresponding
`points includes keeping the distance constant (keeping the
`distance equal to zero means that the end of one segment
`will be always the start point of its neighbor), or ensuring
`that the points remain at least a minimum distance (e.g. to
`prevent overlapping segments) or ensuring that the points
`maintain at most a maximum distance (keeping the points
`close together can simplify editing of the next segment).
`Conventional editing systems allow the author to select
`and replay a time interval. SAS adapts this functionality to
`segment-based editing allowing the author to replay the
`current segment. However, to specify the boundary point of
`35
`a lengthy segment, it is inconvenient to replay the entire
`segment. Therefore, SAS allows the author to enter a maxi
`mum replay time AT labeled "delta' on the screen. If the
`start point T is being modified, the replay interval is always
`from T to (T-AT) independent of T. If the end point T is
`being modified, the replay interval is always from (T-AT)
`to T independent of the value of T. The boundary points
`can be edited using either a textual/numeric or graphical
`interface as described herein.
`SAS can also be used to simplify the synchronization of
`the text track with the motion video and sound track. It
`allows the author to enter text into a text field, select a
`portion of the text and assign it to a segment. Similarly, SAS
`allows the author to select a portion of the text track and
`associate it with choice available to the user (or an "event').
`SAS also supports the identification of additional
`resources associated with each segment and each event.
`Examples of such resources include additional discrete or
`continuous media tracks, such as icons, still images, audio,
`motion video tracks and hypertext links leading to informa
`tion associated with the segment or the event. These addi
`tional resources can be independent, constitute a predefined
`sequence, or be tied to a time point in between the start and
`end point of the segment (respective the segment to which
`the event belongs). The interfaces described herein can be
`employed to help the author specify a time point or time
`interval.
`As shown in FIG. 12, the output of SAS consists of
`segment and event tables. The segment table contains the
`segments' attributes, such as the boundaries of each segment
`with respect to the time axis, the indices of the segment's
`text interval within the text track and the identification of
`
`65
`
`8
`additional resources associated with each segment. The
`event table contains the events' attributes, such as the
`segment to which each event belongs, the indices of the
`element's text interval within the text track and the identi
`fication of additional resources associated with each event.
`4.1. Description of a Music Video Game
`A Music Video Game application will be now described
`to illustrate a preferred implementation of the invention. The
`Music Video Game uses a music video as a multimedia
`presentation and combines it with