(12) United States Patent
`Srinivasan et al.
`
`USOO6357042B2
`US 6,357,042 B2
`(10) Patent No.:
`Mar. 12, 2002
`(45) Date of Patent:
`
`(54) METHOD AND APPARATUS FOR
`MULTIPLEXING SEPARATELY-AUTHORED
`METADATA FOR INSERTION INTO AVIDEO
`DATA STREAM
`
`(76) Inventors: Anand Srinivasan, 12718 Torrey Bluff
`Dr. #155; Mehul Y Shah; Indranil
`Chakraborty, both of 12633 El
`Camino Real #3408; Mohan Mardikar,
`12640 Torrey Bluff Dr. #7; P Venkat
`Rangan, 13011 Callcott Way; Kamal
`Bhadada, 12782 Torrey Bluff Dr. #103,
`all of San Diego, CA (US) 92130
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(*) Notice:
`
`(21) Appl. No.: 09/235,781
`(22) Filed:
`Jan. 22, 1999
`Related U.S. Application Data
`(63) Continuation-in-part of application No. 09/225,250, filed on
`Jan. 4, 1999, which is a continuation-in-part of application
`No. 09/218,738, filed on Dec. 22, 1998, which is a continu
`ation-in-part of application No. 09/168,315, filed on Oct. 7,
`1998, which is a continuation-in-part of application No.
`09/160,365, filed on Sep. 24, 1998, which is a continuation
`in-part of application No. 09/154,532, filed on Sep. 16,
`1998.
`(51) Int. Cl. .......................... H04N 7/025; H04N 7/10;
`H04N 7/173; G06F 3/00; G06F 13/00
`(52) U.S. Cl. .......................... 725/32; 72.5/119; 345/328
`(58) Field of Search .......................... 709/217; 345/327,
`345/328; 348/12, 13, 10, 6, 7; 455/6.1,
`6.2, 6.3, 3.1, 3.2; 370/503–519; 725/119,
`143, 32,36; G06F 15/167; HO4N 1/00,
`1/17, 7/025, 7/11
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`
`
`5,424,770 A * 6/1995 Schmelzer et al. ............ 34.8/9
`5,793.980 A * 8/1998 Glaser et al. ............... 709/231
`5,889,950 A
`3/1999 Kuzma .......................... 34.8/6
`5,894,328 A
`4/1999 Tahara et al. ............... 348/423
`5,917,830 A * 6/1999 Chen et al. .................. 455/6.3
`6,002,393 A * 12/1999 Hite et al. .................. 345/327
`6,018,768 A
`1/2000 Ullman et al. ................ 348/12
`6,034,746 A
`3/2000 Desai et al. ................ 348/423
`6,035,332 A * 3/2000 Ingrassia, Jr. et al. ...... 709/218
`* cited by examiner
`Primary Examiner Andrew Faile
`ASSistant Examiner Vivek Srivastava
`(74) Attorney, Agent, or Firm Donald R. Boys; Central
`Coast Patent Agency, Inc.
`(57)
`ABSTRACT
`An authoring System for interactive Video has two or more
`authoring Stations for providing authored metadata to be
`related to a main Video data Stream and a multiplexer for
`relating authored metadata from the authoring Sources to the
`main Video data Stream. The authoring Stations annotate
`created metadata with presentation time stamps (PTS) from
`the main Video Stream, and the multiplexer relates the
`metadata to the main video stream by the PTS signatures. In
`analog streams PTS may be created and integrated. In Some
`embodiments there may be multiple and cascaded Systems,
`and Some Sources may be stored Sources. Various methods
`are disclosed for monitoring and compensating time differ
`ences among Sources to ensure time coordination in end
`product. In different embodiments transport of metadata to
`an end user Station is provided by Internet Streaming, VBI
`insertion or by Internet downloading. User equipment is
`enhanced with hardware and Software to coordinate and
`present authored material with the main data Stream.
`21 Claims, 18 Drawing Sheets
`
`12
`
`
`
`Video Head End
`
`-
`
`
`
`
`
`Object Tracking
`Module
`
`

`

`U.S. Patent
`
`Mar. 12, 2002
`
`Sheet 1 of 18
`
`US 6,357,042 B2
`
`US. Patent
`
`Mar. 12, 2002
`
`Sheet 1 0f 18
`
`US 6,357,042 B2
`
`I `81-I
`Fig.I
`
`11
`
`
`
`ObjectTrackingModule
`
`
`
`
`
`
`
`K.VideoHeadEnd
`
`12
`
`ZI
`
`15
`
`S I
`
`TTI-1028, Page 2
`
`

`

`U.S. Patent
`
`Mar. 12, 2002.
`
`Sheet 2 of 18
`
`US 6,357,042 B2
`
`US. Patent
`
`Mar. 12, 2002
`
`Sheet 2 0f 18
`
`US 6,357,042 B2
`
`
`
`Fig.2
`
`s
`
`TTI-1028, Page 3
`
`

`

`U.S. Patent
`
`Mar. 12, 2002
`
`Sheet 3 of 18
`
`US 6,357,042 B2
`
`US. Patent
`
`Mar. 12, 2002
`
`Sheet 3 0f 18
`
`US 6,357,042 132
`
`
`
`‘————_—__
`
`hm
`
`TTI-1028, Page 4
`
`
`

`

`U.S. Patent
`
`Mar. 12, 2002.
`
`Sheet 4 of 18
`
`US 6,357,042 B2
`
`US. Patent
`
`Mar. 12, 2002
`
`Sheet 4 0f 18
`
`US 6,357,042 B2
`
`33
`
`
`
`
`
`
`
`
`
`
`
`
`
`Fig.4
`
`III”
`|-|--|--
`II”
`:
`
`TTI-1028, Page 5
`
`

`

`U.S. Patent
`
`Mar. 12, 2002
`
`Sheet 5 of 18
`
`US 6,357,042 B2
`
`US. Patent
`
`Mar. 12, 2002
`
`Sheet 5 0f 18
`
`US 6,357,042 B2
`
`
`
`ç (81-I
`Fig.5
`
`._1
`
`IEV)
`O
`D4
`"O5.)H
`U
`.5.)
`3H
`D4V
`CL
`
`AE0
`
`TTI-1028, Page 6
`
`

`

`U.S. Patent
`
`Mar. 12, 2002
`
`Sheet 6 of 18
`
`US 6,357,042 B2
`
`US. Patent
`
`Mar. 12, 2002
`
`Sheet 6 0f 18
`
`US 6,357,042 B2
`
`69
`
`9 I
`
`13
`
`
`
`
`
`me
`
`:1.
`.9:
`2‘55
`:30
`"0°08
`oxm
`2M0
`35
`SW”
`4.853
`On“
`
`S9
`
`lg.6
`
`TTI-1028, Page 7
`
`

`

`U.S. Patent
`
`US. Patent
`
`Mar. 12, 2002
`
`Sheet 7 0f 18
`
`US 6,357,042 132
`
`Gwv
`
`mm.
`
`«E»
`
`$3985
`
`Eupmxm
`
`Du
`
`9»
`
`mm
`
`NNE
`
`
`
`«mm
`
`
`
`839mmwEhofiz<
`
`Mm
`
`mm
`
`SS
`
`L
`m
`m
`
`Q99
`
`hm
`
`L
`S
`
`93E83>
`
`tam
`
`mw
`
`TTI-1028, Page 8
`
`
`
`

`

`U.S. Patent
`
`US 6,357,042 B2
`
`US. Patent
`
`Mar. 12, 2002
`
`Sheet 8 0f 18
`
`US 6,357,042 132
`
`mm
`
`mm.
`
`_—m—mmmm$—
`
`30
`
`03
`
`Momfio
`
`Stowwccofis<
`
`mo
`
`[FF,,
`
`TTI-1028, Page 9
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Mar. 12, 2002
`
`Sheet 9 of 18
`
`US 6,357,042 B2
`
`US. Patent
`
`Mar. 12, 2002
`
`Sheet 9 0f 18
`
`US 6,357,042 132
`
`
`
`m.uE
`
` uofiCBEEmD
`
`IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
`
`:
`
`TTI-1028, Page 10
`
`
`

`

`U.S. Patent
`
`Mar. 12, 2002.
`
`Sheet 10 of 18
`
`US 6,357,042 B2
`
`
`
`Reader/Frame Counter
`
`Number/Marker
`Generator
`
`Writer/Data Inserter
`
`Pixel Command
`
`To Brodcast System
`
`TO WAN
`
`Fig. 10
`
`

`

`U.S. Patent
`
`Mar. 12, 2002
`
`Sheet 11 of 18
`
`US 6,357,042 B2
`
`103
`
`105
`
`107
`
`109
`
`1 11
`
`Monitor and identify
`frames in streams
`as needed.
`
`
`
`
`
`Generate binary
`signatures
`
`
`
`Write binary data
`into VBI/HBI
`
`
`
`
`
`
`
`Select pixels for
`alteration and
`activate
`
`Insert markers
`as needed
`
`Fig. 11
`
`

`

`U.S. Patent
`
`Mar. 12, 2002
`
`Sheet 12 of 18
`
`US 6,357,042 B2
`
`
`
`©InpOWN
`
`KeIds[GI
`
`LÕI1nduI
`
`qnduILÕI
`
`6 I I
`
`/~ || A,
`JLdIO
`
`====
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Mar. 12, 2002
`
`Sheet 13 0f 18
`
`US 6,357,042 B2
`
`
`
`Capture and load
`Separate data streams
`into pipeline
`
`Search for markers
`in Video broadcast
`data Stream
`
`Search for frame
`numbers in
`annotation Stream
`
`Determine timing
`error factor between
`StreamS
`
`Buffer streams to
`reduce error factor
`
`Repeate steps 145
`-151 for each
`marker found
`
`Fig. 13
`
`

`

`U.S. Patent
`
`Mar. 12, 2002
`
`Sheet 14 of 18
`
`US 6,357,042 B2
`
`US. Patent
`
`Mar. 12, 2002
`
`Sheet 14 0f 18
`
`US 6,357,042 132
`
`5-,.
`
`
`
`fig. 4
`
`
`
`TTI-1028, Page 15
`
`

`

`U.S. Patent
`
`Mar. 12, 2002
`
`Sheet 15 of 18
`
`US 6,357,042 B2
`
`US. Patent
`
`Mar. 12, 2002
`
`Sheet 15 0f 18
`
`US 6,357,042 B2
`
`
`
`i
`
`01:01:52
`
`38:3:38
`
`8:33:33
`
`“(‘9 5:»: the firm mme-n '
`
`00:20:03? .
`
`‘
`
`,
`
`I:
`
`fifig.£§
`ig.
`
`TTL1028,Page16
`
`

`

`U.S. Patent
`
`Mar. 12, 2002
`
`Sheet 16 of 18
`
`US 6,357,042 B2
`
`US. Patent
`
`Mar. 12, 2002
`
`Sheet 16 0f 18
`
`US 6,357,042 132
`
`
`
`“2,50pr
`
`3a
`
`W78258Z358me
`
`3ME
`
`IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIJ
`
`TTI-1028, Page 17
`
`
`
`

`

`U.S. Patent
`
`Mar. 12, 2002
`
`Sheet 17 of 18
`
`US 6,357,042 B2
`
`243
`
`245
`
`
`
`User Logs On to Service
`
`Ad Server Notified and
`Accesses User-Profile Data
`
`Ad Server Executes Ad
`Schedule to Integrate Ads
`With Video, Based on
`Profile Data
`
`Fig. 17
`
`

`

`U.S. Patent
`
`Mar. 12, 2002
`
`Sheet 18 of 18
`
`US 6,357,042 B2
`
`/ 90
`
`Smom
`
`m
`
`001
`
`
`
`BmmmMI
`aAl'-
`
`US. Patent
`
`20mn,
`
`r.maflofisdcNQ4watofi=<m02.3-5th
`
`mvm
`
`
`
`6S
`
`«W,
`
`m
`
`m.3.me
`
`?I '81-I
`
`
`
`
`
`NaBRO—Eon.BEBE@22585%::5
`
`
`
`UuEmubmBEBE
`
`Umommom
`
`SN
`
`
`
`Emfimnooztongs;33302
`
`
`
`
`
`th
`LSZ
`
`TTL1028,Page19
`
`
`

`

`US 6,357,042 B2
`
`1
`METHOD AND APPARATUS FOR
`MULTIPLEXING SEPARATELY-AUTHORED
`METADATA FOR INSERTION INTO AVIDEO
`DATA STREAM
`
`CROSS-REFERENCE TO RELATED
`DOCUMENTS
`The present invention is a continuation-in-part (CIP) of
`copending patent application Ser. No. 09/225,250, filed Jan.
`4, 1999 which is a CIP of Ser. No. 09/218,738, filed Dec. 22,
`1998, which is a CIP of Ser. No. 09/168,315, filed Oct. 7,
`1998, which is a CIP of copending application Ser. No.
`09/160,365, filed Sep. 24, 1998 which is a CIP of Ser. No.
`09/154,532, filed Sep. 16, 1998.
`FIELD OF THE INVENTION
`The present invention is in the field of Video broadcasting,
`and pertains more particularly to methods and apparatus for
`multiplexing Separately-authored metadata for coordination
`with a main Video data Stream.
`
`15
`
`2
`Set-top boxes and computer-integrated TVS adapted as
`described above typically have inputs for such as a TV
`antenna (analog), cable TV (analog or digital), more recently
`direct-satellite TV (digital), and may also connect to Video
`cassette recorders and to mass Storage devices Such as hard
`disk drives and CD-ROM drives to provide a capability for
`uploading Video data from Such devices and presenting the
`dynamic result as a display on the TV Screen.
`The present inventors have noted that with the coupling of
`computer technology with TV, many capabilities familiar to
`computer users have been made available to TV users. For
`example, ability to provide text annotation for TV presen
`tations is considerably enhanced. Computer techniques Such
`a Pix-on-Pix are now available, wherein separate TV pre
`Sentations may be made in Separate windows, or overlaid
`windows on the display Screen. Separate windows may also
`Support display from Separate Sources, Such as an analog TV
`program in one window, a computer game in another, and a
`Video conference in a third.
`With the technologies described above becoming more
`available in the market place, it has become desirable to
`further integrate the technologies described So that a user
`Viewing a Video presentation might be enabled to gather
`additional information about a specific image entity or
`entities portrayed in a Video through interactive method. An
`ultimate goal is to provide a means for advertisers to
`promote and Sell products through user interaction in a way
`that minimizes Steps required by Such a user to access
`additional information regarding traditionally advertised
`products Such as through commercials and the like.
`In typical prior art Video authoring Systems, end users
`receive a Single video stream that contains the Video data and
`any added annotated data Such as Subtitling, sponsor logos,
`information blocks, and the like. However, it is desirable to
`build upon the goal Stated in the preceding paragraph above,
`by having Separate Streams, one containing video data, and
`the other containing annotative data, that may arrive at and
`end user's location via different delivery media and be
`displayed Synchronously on a Suitable display Screen.
`An authoring System, known to the inventor, may provide
`image tracking coordinates along with various further
`annotation, and may deliver Separate Streams via Separate
`carriers to an end user. Also known to the inventor is a
`System for providing a means of applying a signature and
`asSociative frame identification to the Separate Streams
`respectively before broadcast So that both Streams may later
`be re-synchronized at the user's end. Such a System is
`likewise described under the cross-referencing Section.
`In current art commercial programming, various compa
`nies may purchase advertising blockS or time slots from a
`content provider. The content provider then edits-in Such
`commercials to the appropriate slots before broadcasting.
`Typically, Such commercial ads may be local to an area of
`broadcast and are limited in profiling to those general
`demographics associated with a range or geography of local
`Viewers. For example, in a broadcast football game, com
`mercials may be geared to appealing to a general profile of
`a Sports fan. For a cable channel carrying exclusively
`Women's programming, advertisements would be geared
`more toward women in general. The profiling or focusing of
`advertisement a company can do is thus quite limited.
`A System known to the inventors and disclosed in this
`patent application under the sub-heading below titled “Per
`sonalized and Interactive Ad System/Network” provides in
`one embodiment an Internet-connected Subscription Server
`running an ad-engine in the form of a Software application
`
`25
`
`35
`
`40
`
`BACKGROUND OF THE INVENTION
`With continuing development of new and better ways of
`delivering television and other Video presentations to end
`users, and parallel development of computerized informa
`tion Systems, Such as the Internet and the associated World
`Wide Web (WWW), there have been concerted efforts to
`integrate various Systems to provide enhanced information
`delivery and entertainment Systems. For example, develop
`erS are introducing integrated Systems combining TVs with
`computer subsystems, so a TV may be used as a WEB
`browser, or a PC may be used for enhanced TV viewing.
`In Some Systems computer elements, Such as a CPU,
`memory, and the like, are built into the familiar chassis of a
`TV set. In such a system, the TV screen becomes the display
`monitor in the computer mode. In Such a System, conven
`tional TV elements and circuitry are incorporated along with
`the computer elements, and capability is provided for a user
`to Switch modes, or to view recorded or broadcast video with
`added computer interaction. One may thus, with a properly
`equipped System, Select to view analog TV programs, digital
`TV programs, conventional cable TV, satellite TV, pay TV
`from various sources, and browse the WWW as well,
`displaying WEB pages and interacting with on-screen fields
`and relational Systems for jumping to related information,
`databases, and other WEB pages. The capabilities are often
`integrated into a Single display, that is, one may view a
`broadcast presentation and also have a window on the
`display for WEB interaction.
`In Some other Systems, computer elements are provided in
`an enclosure separate from the TV, often referred to in the art
`as a Set-top box. Set-top box Systems have an advantage for
`providers in that they may be connected to conventional
`television sets, so end users don’t have to buy a new TV
`55
`along with the computer elements.
`In Such integrated Systems, whether in a single enclosure
`or as Set-top box Systems, user input is typically through a
`hand-held device quite Similar to a familiar remote
`controller, usually having infra-red communication with the
`Set-top box or a receiver in the integrated TV. For computer
`modes, Such as WEB browsing, a cursor is displayed on the
`TV Screen, and cursor manipulation is provided by buttons
`or other familiar pointer apparatus on the remote. Select
`buttons are also provided in the remote to perform the
`familiar function of Such buttons on a pointer device, like a
`mouse or trackball more familiar to computer users.
`
`45
`
`50
`
`60
`
`65
`
`

`

`US 6,357,042 B2
`
`3
`that has ability to Select Video ads according to user profile
`and to Stream Such ads to a user along with a main Video data
`Stream. In Some cases the ads are interactive. In Systems
`wherein the main Video and Such video ads are Sent by a
`common carrier, Such as an Internet connection, the ads are
`inserted in the main video stream in the form of video
`metadata.
`It is desirable that that more than one authoring Station or
`System may be used when creating metadata for delivery to
`an end user, because there are a variety of functions that may
`be implemented through metadata. For example, it is desir
`able that Separate authoring Stations will be used in hyper
`Video authoring, Such as in providing object tracking
`coordinates, creating hot spots (hyperlinks) in a video,
`providing interactive regions for tracked objects, inserting
`URLs, providing review markers by Scene authoring, and
`So on. Scene authoring based on Scene-change-detection
`technology (SCDT) has several purposes, Such as providing
`thumbnails as bookmarks for users to Select and review
`particular portions of Video presentations, and for markers
`for ad insertion or insertion of other information. In addition,
`Separate ad information may be authored by yet additional
`authors and provided as metadata for delivery to an end user.
`While combination of live video and live annotation
`Streams is treated herein concerning hyper-video authoring
`and delivery, it is generally understood that in the live case,
`annotation Streams may be timed to run in Sync alongside or
`over a main video Steam. This process is performed at the
`provider's end. However, the presence of possibly two or
`more Separately-authored annotation data-Sets wherein the
`method of delivery is not necessarily in real time, requires
`a more comprehensive approach.
`What is clearly needed is a method and apparatus for
`merging Separately-authored Sets of metadata Such that the
`metadata is associated appropriately to a correct frame
`location in a main Video. Such a method and apparatus
`would serve to ad flexibility to the authoring process and to
`simplify delivery methods.
`
`15
`
`25
`
`35
`
`40
`
`45
`
`4
`embodiments the PTS-enhanced metadata is stored to be
`downloaded as needed by a user.
`In various embodiments of the invention the authoring
`Stations may include one or more of Scene authoring, hyper
`Video authoring, and ad authoring Stations. At the user end
`the user System is enhanced with Software for displaying the
`main Video data Stream and the authored metadata according
`to the PTS.
`In another aspect of the invention a method for coordi
`nating authored video metadata with a main Video data
`Stream is provided, comprising steps of (a) ensuring the
`main video data stream has a presentation time Stamp (PTS);
`(b) feeding the digital main video data Stream to authoring
`Stations; (c) authoring matadata at the authoring Stations;
`and (d) marking the metadata with presentation time stamps
`(PTS) from the main video data stream.
`This method may further comprise a step for multiplexing
`authored metadata with the main Video data Stream, wherein
`the multiplexer places the metadata in relation to the main
`video data stream according to the PTS. There may also be
`multiple Sources of Video fed to the multiplexer as well as
`multiple metadata Streams for a Video, and a step as well for
`compensating for real-time differences between the multiple
`Sources. In the compensating Step, presentation time Stamps
`(PTS) may be amended according to source time differences.
`In Some cases the PTS-enhanced metadata is streamed
`over the Internet to an end user. In other cases the PTS
`enhanced metadata is inserted into Video blanking intervals
`(VBI) of an analog stream according to the PTS. In still other
`cases the PTS-enhanced metadata is stored to be down
`loaded as needed by a user. The authoring Stations may
`include one or more of Scene authoring, hyper-video
`authoring, and ad authoring Stations, and analog streams
`may be accommodated in Some embodiments by conversion
`to a digital format before authoring and multiplexing, and in
`others by integrating a PTS with the analog stream. Also, at
`the final user's end, there is Software for rendering the main
`Video data Stream and authored metadata according to PTS.
`In yet another aspect of the invention a digital Video
`multiplexing System is provided comprising metadata inputs
`from Video authoring Stations, an input for a main digital
`Video data Stream; and an output to a Video transport
`interface. The multiplexer notes presentation time Stamps
`asSociated with authored metadata, and places the authored
`metadata relative to the main Video data Stream for transport
`to end users. The multiplexing System may have multiple
`Video data Stream inputs, and one or more of the inputs may
`be from a stored source. There may also be multiple video
`data Stream inputs from multiple Sources, and the multi
`plexer monitorS real time clocks of the Sources and uses the
`information to compensate one or both of the multiple
`Streams. In case of real-time differences the multiplexer
`compensates incoming Streams by buffering one or more of
`the Streams. The multiplexer may also compensate incoming
`Streams by amending the presentation time Stamps of one or
`more of the Streams.
`In embodiments of the invention as taught in enabling
`detail below, for the first time it is possible to annotate one
`or more main Video streams, either analog or digital Streams,
`and to enhance the Streams with authored metadata in a
`manner that multiple inputs may be made and fully coordi
`nated to be completely useful when finally delivered to the
`end user, and many interactive functions not previously
`known in the art are provided.
`BRIEF DESCRIPTION OF THE DRAWING
`FIGURES
`FIG. 1 is a block diagram illustrating an authoring System
`incorporating an image tracking module according to an
`embodiment of the present invention.
`
`SUMMARY OF THE INVENTION
`In a preferred embodiment of the present invention an
`authoring System for interactive video, comprising a Video
`feed providing a main Video presentation Stream; two or
`more authoring Stations coupled to the Video feed providing
`authoring functions creating metadata for enhancing the
`main Video stream; and a multiplexer for coordinating
`authored metadata with the main Video Stream. The author
`ing stations may note a presentation time Stamp (PTS) of
`Video frames or any other time Stamp and incorporate it in
`the authored metadata for matching the metadata with the
`main Video presentation Stream.
`In various embodiments there is a multiplexer for com
`bining authored metadata with the main Video data Stream,
`and the multiplexer places the metadata in relation to the
`main video data stream according to the PTS. The multi
`plexer in Some cases receives multiple Video Streams as well
`as the authored metadata, and time clocks are monitored for
`Separate Stream Sources and clocks are adjusted to compen
`Sate for real-time differences in Sources. One or more of the
`Stream Sources may be from a stored Source. In Some
`embodiments PTS values are rewritten in one or more
`Streams to compensate for perceived time differences. Also
`in Some embodiments PTS-enhanced metadata is streamed
`over the Internet to an end user, and in others the PTS
`65
`enhanced metadata is inserted into Video blanking intervals
`(VBI) of an analog stream according to the PTS. In still other
`
`50
`
`55
`
`60
`
`

`

`S
`FIG. 2 is an illustration of a display Screen depicting a
`tracked image entity according to an embodiment of the
`present invention.
`FIG. 3 is an enlarged view of the image entity of FIG. 2
`wherein key pixels are visible.
`FIG. 4 is a exemplary view of a value table containing
`values of Signature pixels according to an embodiment of the
`present invention.
`FIG. 5 is a motion diagram illustrating an image tracking
`process according to an embodiment of the present inven
`tion.
`FIG. 6 is a block diagram illustrating a data-stream
`generator and Set-up function of the tracking module of FIG.
`1 according to an embodiment of the present invention.
`FIG. 7 is a block diagram illustrating a real-time video
`authoring and delivery System according to an embodiment
`of the present invention.
`FIG. 8 is a block diagram illustrating multiple authoring
`Station architecture according to an embodiment of the
`present invention.
`FIG. 9 is a block diagram illustrating an exemplary
`modular architecture of a single authoring Station according
`to an embodiment of the present invention.
`FIG. 10 is a block diagram illustrating a signature appli
`cation apparatus at the authoring end according to an
`embodiment of the present invention.
`FIG. 11 is a process flow chart illustrating logical Steps of
`providing a signature at the authoring end according to an
`embodiment of the present invention.
`FIG. 12 is a block diagram illustrating a data capture and
`Synchronization device at the user's end according to an
`embodiment of the present invention.
`FIG. 13 is a Process flow chart illustrating logical steps
`for capturing and Synchronizing Separate Video Streams for
`user display and interaction according to an embodiment of
`the present invention.
`FIGS. 14 and 15 are illustrations of a graphic user
`interface of a Video editor according to an embodiment of
`the present invention.
`FIG. 16 is an overview of an exemplary ISP-based video
`delivery network Supporting personalized and interactive
`Video advertising according to an embodiment of the present
`invention.
`FIG. 17 is a process flow chart illustrating video-ad server
`function and interaction according to an embodiment of the
`present invention.
`FIG. 18 is a diagram of a multiplexing System according
`to an embodiment of the present invention.
`DESCRIPTION OF THE PREFERRED
`EMBODIMENTS
`According to a preferred embodiment of the present
`invention, a method and apparatus is provided which allows
`a programmer functioning in a Video editing mode to initiate
`tracking of any image entity or entities in a Video Stream,
`after which initiation tracking may be automatic, wherein
`appropriate coordinate-tracking data associated with the
`image entity or entities is provided Synchronized with the
`original data Stream So that Such entities may later be
`rendered identifiable and interaction-capable to an end user.
`By image entity is meant any perSon or thing depicted in a
`Video display, Such as a player in a Sports game, and actor
`in a play, a car in a car race, and So on.
`FIG. 1 is a block diagram illustrating an authoring Station
`11 adapted for accepting a Video data Stream 16 from a Video
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 6,357,042 B2
`
`6
`head end 15 as is known in the art. The original video data
`Source 12 may be any recorded or live Source and is not
`limited as to protocol. Authoring Station 11 includes a
`display 14 and an object tracking module 13 which is
`adapted to automatically track image entities in a Video
`presentation, and to Synchronize the tracking result with the
`Video data Stream.
`The Overall purpose of the authoring Station is addition of
`innovative material to the Video data Stream, Such as text
`overlay, graphic icons and logos for advertisement, Some of
`which may be associated with identity and address data to
`allow a viewer at a computerized end Station to access
`advertisements and other data which may be associated with
`individual entities in the video presentation. Advertisements
`may, for example, be associated with a tracked object. Also
`the text annotations could either be set to track along with an
`object, or appear in a fixed position anywhere on the Screen,
`as they are typical in broadcasts today.
`In FIG. 1 input data stream 15 to tracking module 13 is a
`Stream of Successive bitmapped frames in a normalized
`resolution, required by the tracking module. The authoring
`Station can be based on Virtually any Sort of computer
`platform and operating System, and in a preferred
`embodiment, a PC station running MS Windows is used, in
`which case the input Stream 16, regardless of protocol, is
`converted to bitmapped frames displayed typically at 30
`frames per second with a resolution of 352 by 240 pixels.
`The use and operation of the tracking module is disclosed
`below in enabling detail, and outputs both the original data
`Stream 19 and an annotation Stream, which in this case
`comprises the data of one or more tracks for moving entities
`in the video display. These two streams may be combined or
`Separately provided to downstream processes and
`equipment, and other annotations and alterations may be
`made to the data Stream in the authoring Station, as described
`above. Output 21 from authoring station 11 is meant to be a
`general indication of data output, and not to indicate that
`there is a single data Stream. There may be multiple Streams
`of various protocol.
`An authoring Station Such as Station 11 comprises Soft
`ware that may process various media (analog or digital)
`including Video presentations or Streams, both live and
`recorded. For example, included Video Stream types includ
`ing but not limited to common formats Such as Audio Video
`Interleave (AVI) and Moving Pictures Experts Group
`(MPEG). Video source 12 may embody any video source
`that may be known in the art such as a CD-ROM, Satellite
`TV, cable TV, VCR, Internet Server, and so on. Video source
`12 may provide prerecorded video or live broadcast video.
`Also, future new formats of Video Streams shall be consid
`ered equivalent in the Sense of this invention.
`As described briefly above, the inventor has found that a
`Suitable computer platform for performing the tracking
`process enabled by tracking module 13 is a PC/VDU run
`ning Windows with a central processing unit (CPU) oper
`ating at at least 300 megahertz and having at least 64
`megabytes of random access memory (RAM). Video fre
`quency or a frame speed of 30 frames per second (FPS) is
`utilized in this example as a Suitable and compatible stan
`dard for processing. The display size of the Video picture in
`this instance is 352x240 pixels, which is a reduced resolu
`tion from that normally displayed in a Video presentation on
`most end use Stations, but provides a Suitable format for the
`present tracking process. It will, however, be apparent to the
`skilled artisan that the example parameters presented herein
`are not limiting to the Success of practicing the present
`
`

`

`US 6,357,042 B2
`
`15
`
`35
`
`40
`
`7
`invention. Other Video frame Speeds, Video display sizes,
`computer platform types and Speeds, and Software types
`may be employed in other embodiments of the present
`invention. The only requirement is that the processor be
`powerful enough to enable the tracking process at the
`necessary frame Speed.
`Tracking module 13 is a Software module provided as a
`component of a Windows driver in this particular embodi
`ment. In other embodiments, module 13 may be provided as
`a driver or as a component of a driver for another platform
`such as may be used as an alternative to WindowsTM.
`Module 13 interfaces with other software components
`executing on Station 11 for the purpose of adding the
`functionality of the present invention which is the ability to
`track a moving image entity within a presented Video Stream.
`Tracking module 13 is adapted to determine a pixel
`Signature associated with a visual entity to be tracked in a
`Video display, and then to determine the path of the pixel
`Signature in the display as the path of the image entity to be
`tracked.
`Once the Video stream is in the domain of tracking module
`13, an image entity in a fist bitmap image is Selected to be
`tracked, and further frame-by-frame movement of the image
`entity is automatically tracked. Tracking data associated
`25
`with a tracked image entity is associated with the bitmap
`Video Stream via a Second Synchronous data-stream com
`prising the frame-by-frame coordinates of the tracked image
`entity. This output video stream comprises the original
`Stream plus the Synchronous data Stream that contains the
`tracking data and is illustrated (via directional arrow) as
`leaving module.
`The end result of tracking process of the present invention
`is a Series of coordinates of an assumed center point of a
`tracking object associated with the image entity Selected to
`be tracked. Through additional editing processes a moving
`region associated with the image entity in a display may be
`made to be interactive and identifiable to an end user. User
`interaction with Such an image entity during viewing of a
`Video can be programmed to provide additional network
`Stored information about that entity to Suitable customer
`premises equipment (CPE) adapted to receive and display
`the information. Such further information may be displayed,
`for example, as an overlay on the display of the dynamic
`Video containing the Subject image entity. In this way,
`advertisers, product promoters, or the like may present
`information to end users based on user interaction with an
`asSociated entity in a dynamic Video display.
`FIG. 2 is an illustration of a display Screen depicting an
`image entity 27 to be tracked according to an embodiment
`of the present invention. PC/VDU display 14 shows a
`bitmap image Stream during the tracking process. Screen 14
`is a display Screen as viewed by a programmer or editor
`performing the tracking process.
`To begin, an editor Selects, typically by well-known
`cursor technique, a tracking box 29 of pre-Selected shape
`and size, and places (drag-and-drop, click-on etc.) the
`Selected box over an image entity to be tracked, with the
`center point of the box Substantially centered on the