(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2007/0250901 A1
`(43) Pub. Date:
`Oct. 25, 2007
`McIntire et al.
`
`US 2007025.0901A1
`
`(54) METHOD AND APPARATUS FOR
`ANNOTATING MEDIA STREAMS
`
`(52) U.S. Cl. .................. 725/146; 715/500.1; 715/501.1;
`725/145
`
`(76) Inventors:
`
`John P. McIntire, Palo Alto, CA
`(US); David W.J.
`Stringer-Calvert, Santa Clara, CA
`(US)
`Correspondence Address:
`Kin-Wah Tong, Esq.
`Patterson & Sheridan, LLP
`Suite 100, 595 Shrewsbury Avenue
`Shrewsbury, NJ 07702
`
`(21) Appl. No.:
`
`11/694,836
`
`(22) Filed:
`
`Mar. 30, 2007
`Related U.S. Application Data
`(60) Provisional application No. 60/788,367, filed on Mar.
`30, 2006, provisional application No. 60/788,368,
`filed on Mar. 30, 2006, provisional application No.
`60/747,379, filed on May 16, 2006, provisional appli
`cation No. 60/804,091, filed on Jun. 6, 2006, provi
`sional application No. 60/866,803, filed on Nov. 21,
`2006, provisional application No. 60/871,705, filed
`on Dec. 22, 2006.
`
`Publication Classification
`
`(51) Int. Cl.
`H04N 7/6
`G06F 7700
`G06F 5/00
`
`(2006.01)
`(2006.01)
`(2006.01)
`
`ABSTRACT
`(57)
`In one embodiment, the invention is a method and apparatus
`for annotating media streams. One embodiment of a method
`for facilitating creation of an annotated media stream by a
`user includes receiving the media stream from the user,
`mapping at least one item of Supplemental content to at least
`a portion of the media stream to produce the annotated
`media stream, and storing the annotated media stream. In
`another embodiment, a method for distributing an annotated
`media stream provided by a user includes receiving the
`annotated media stream from the user, where the annotated
`media stream includes at least one item of Supplemental
`content mapped to at least a portion of a media stream to
`produce the annotated media stream, storing the annotated
`media stream, receiving a request from a viewer to view the
`annotated media stream, and distributing the annotated
`media stream to the viewer. In another embodiment, a
`method for distributing an annotated media stream provided
`by a user includes distributing the annotated media stream to
`a viewer, where the annotated media stream includes at least
`one item of Supplemental content mapped to at least a
`portion of a media stream to produce the annotated media
`stream, and collecting a commission when the annotated
`media stream is viewed by the viewer. In one embodiment,
`the collecting requires further action on a part of the viewer
`(such as selection of a hyperlink contained in the at least one
`item of Supplemental content or consummation of a com
`mercial transaction).
`
`to
`
`
`
`VIEWER/BUYER
`104
`
`STREAM ANNOTATION
`PROVIDER
`112
`ANNoTA
`SELERS SUPPLEMENTAL CONTENT )
`ANNOTATION SERVICES
`4.
`
`
`
`ACCESS PROVIDERS
`>
`106
`
`ADBUYERS
`108
`
`CONTENT CREATORS
`102
`
`
`
`
`
`
`
`
`
`Page i
`
`AMAZON EX. 1004
`Amazon v. CustomPlay
`US Patent No. 9,124,950
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 1 of 48
`
`US 2007/0250901 A1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`MEDIASTREAMS
`
`PROPERTY
`
`Z?SHETTES
`
`Page ii
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 2 of 48
`
`US 2007/025.0901 A1
`
`200 N
`
`IDENTIFY ONE ORMORE ARTICLESTO
`APPEAR IN MEDIA STREAM, AND TAG
`EACH IDENTIFIED ARTICLE WITH A UNIQUE TAG
`
`204
`
`CREATE OR UPDATE DATABASE OF
`TAGS AND ASSOCIATED ARTICLES
`
`2O6
`
`DETECT AND READ TAGS DURING FILMING
`
`208
`
`ASSIGN UNIOUEDS TO EACHSEGMENT OF
`MEDIA STREAMDURING FILMING
`
`210
`
`
`
`
`
`CREATE MAPPING OF ARTICLES TO SEGMENTS
`NACCORDANCE WITH READ TAGS AND
`ASSIGNED SEGMENT DENTIFFERS
`
`212
`
`FIG. 2
`
`Page iii
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 3 of 48
`
`US 2007/025.0901 A1
`
`302
`
`
`
`
`
`
`
`READ ARTICLE TAGS FROM ONE ORMORE
`LOCATIONS IN 3-D SPACE
`
`304
`
`DETERMINE 3-D POSITION OF AT LEAST ONE
`ARTICLE IN ACCORDANCE WITH READINGS
`
`306
`
`
`
`DETERMINE SUBSET OF 3-D SPACE
`DEFINED BY MAGE CAPTURING DEVICE
`
`308
`
`SELECT TAGGED ARTICLE
`
`310
`
`DOES
`ARTICLE
`APPEAR IN SUBSET
`SPACE
`
`NO
`
`MARKARTICLE
`AS "ACTIVE"
`
`MARKARTICLE
`AS "INACTIVE"
`
`316
`
`SELECT
`ANOTHER
`ARTICLE
`
`
`
`
`
`FIG. 3
`
`320
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Page iv
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 4 of 48
`
`US 2007/025.0901 A1
`
`CREATE OR OBTAIN ENCYCLOPEDIA OF
`MAGES ARTICLES AND INDIVIDUALS THAT
`MAY APPEAR IN MEDIA STREAM
`
`404
`
`OBTAN SEGMENT DENT FERENCODED
`MEDIA STREAM
`
`406
`
`PROCESS MEDIA STREAMNACCORDANCE
`WITH AVIDEO RECOGNITION SYSTEM,
`USING ENCYCLOPEDIA OF IMAGES
`
`408
`
`
`
`USING RECOGNITION RESULTS, ASSOCIATE
`ONE ORMORE SEGMENTS DENTIFIERS WITH
`ONE OR MORE ARTICLE IDS FOR ARTICLES
`APPEARNG THEREN
`
`- 410
`
`FIG. 4
`
`Page v
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 5 of 48
`
`US 2007/025.0901 A1
`
`
`
`CREATE OR OBTAIN ENCYCLOPEDIA ARTICLES
`AND INDIVIDUALS THAT MAY APPEAR
`NMEDIA STREAM
`
`504
`
`OBTAN SEGMENT DENTIFIERID-ENCODED
`MEDIA STREAM
`
`506
`
`PROCESS MEDIA STREAMNACCORDANCE
`WITH A SPEECH RECOGNITION SYSTEM,
`USINGENCYCLOPEDIA
`
`USING RECOGNITION RESULTS, ASSOCIATE
`ONE OR MORE SEGMENT DENTIFIERS WITH
`ONE OR MORE ARTICLE DS FOR ARTICLES
`APPEARNG THEREN
`
`508
`
`510
`
`FIG. 5
`
`Page vi
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 6 of 48
`
`US 2007/025.0901 A1
`
`600 N
`
`602
`
`
`
`OBTAIN CODEDMEDIA STREAM
`
`604
`
`
`
`
`
`
`
`
`
`MAP AT LEAST ONE SEGMENT DENTIFIER
`OF THE MEDIA STREAM TO AN TEM OF
`SUPPLEMENTAL CONTENT ENACCORDANCE
`WITH THE CODING OF THE STREAM
`
`606
`
`STORE MAPPNG
`
`
`
`610
`
`FIG. 6
`
`Page vii
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 7 of 48
`
`US 2007/025.0901 A1
`
`702
`
`
`
`RECEIVE MEDIA STREAM
`
`704
`
`
`
`
`
`RECEIVE SUPPLEMENTAL CONTENT
`ASSOCATED WITH MEDIA STREAM
`
`706
`
`SYNCHRONIZE SUPPLEMENT CONTENT
`WITH MEDIA STREAM
`
`708
`
`- - - - - - - - - - - - - - - - - -
`SYNCHRONIZE SYSTEM-SELECTED
`SUPPLEMENTAL CONTENT WITH n- 70
`MEDIA STREAM
`- - - - - - - - - - - - - - - - -
`
`
`
`SAVE MEDASTREAM
`
`712
`
`714
`
`FIG. 7
`
`Page viii
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 8 of 48
`
`US 2007/025.0901 A1
`
`800 N
`
`RECEIVE AMAPPING FROM SEGMENT DENTIFIERS
`OF AMEDIA STREAM TO TEMS OF SUPPLEMENTAL CONTENT
`
`
`
`
`
`
`
`DISPLAY MEDIA STREAM
`
`RECEIVE FRST SIGNAL FROM VIEWER OF
`MEDIA STREAM
`
`
`
`808
`
`DETERMINE SEGMENT DENTIFIER
`CORRESPONDING TO SIGNAL
`
`810
`
`DISPLAY SUPPLEMENTAL CONTENT MAPPED
`TO CORRESPONDING SEGMENT DENTIFFER
`
`RECEIVE SUBSEOUENT SIGNAL FROM VIEWER
`
`816
`
`NO
`
`RETURN
`TO MEDIA
`STREAM
`p
`
`PURCHASE
`REQUESTED
`p
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`CONSUMMATE DESIRED TRANSACTION
`
`FIG. 8
`
`Page ix
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 9 of 48
`
`US 2007/025.0901 A1
`
`DISPLAY MEDASTREAM
`
`904
`
`RECEIVE FIRST SIGNAL FROM VIEWER OF
`MEDIA STREAM
`
`906
`
`
`
`
`
`IDENTIFY ONE OR MORE ARTICLES
`APPEARNG IN MEDIA STREAM
`
`908
`
`RETRIEVE INFORMATIONABOUT - 910
`THE ARTICLE(S)
`
`DISPLAY RETREVED INFORMATION
`
`912
`
`RECEIVE SUBSEOUENT SIGNAL FROM VIEWER
`
`914
`
`
`
`YES
`
`
`
`RETURN
`TO MEDIA
`STREAM
`
`916
`
`NO
`
`
`
`
`
`PURCHASE
`RECQUESTED
`
`918
`
`NO
`
`YES
`
`
`
`CONSUMMATE DESRED TRANSACTION
`
`920
`
`FIG. 9
`
`Page x
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 10 of 48
`
`US 2007/025.0901 A1
`
`
`
`W/TOO ZÁX
`
`Page xi
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 11 of 48 US 2007/0250901 A1
`
`AF
`
`1102
`
`1100
`
`/
`
`RECEIVE AMAPPNG FROM ONE ORMORE
`KEYWORDS TO TEM OF SUPPLEMENTAL CONTENT
`
`1104
`
`1106
`
`1108
`
`1110
`
`1112
`
`1114
`
`1116
`
`DISPLAY MEDIA STREAM
`
`RECEIVE FIRST SPEECH SIGNAL FROM
`VIEWER OF MEDIA STREAM
`
`PROCESS FIRST SPEECH SIGNAL
`TO RECOGNIZE VIEWER COMMAND
`
`MATCH VIEWER COMMAND TO
`ONE OR MORE KEYWORDS
`
`DISPLAY SUPPLEMENTAL CONTENT
`MAPPING TO CORRESPONDING KEYWORD(S)
`
`RECEIVE SUBSEQUENT SIGNAL FROM VIEWER
`
`
`
`
`
`YES
`
`
`
`
`
`RETURN
`TO MEDIA
`STREAM?
`
`1118
`NO
`
`
`
`
`
`
`
`
`
`
`
`PURCHASE
`REQUESTED
`
`1120
`
`NO
`
`1122
`
`CONSUMMATE DESRED TRANSACTION
`
`F.G. 11
`
`Page xii
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 12 of 48 US 2007/0250901 A1
`
`A.
`
`1202
`
`/
`
`DISPLAY MEDIA STREAM
`
`1200
`
`1204
`
`RECEIVE FIRST SPEECH SIGNAL FROM
`VIEWER OF MEDIA STREAM
`
`1206
`
`PROCESS FIRST SPEECH SIGNAL TO
`RECOGNIZE A VIEWER COMMAND
`
`QUERY REMOTE SERVER IN ACCORDANCE
`WITH VIEWER COMMAND
`
`1208
`
`1210
`
`DISPLAY QUERY RESULTS
`
`RECEIVE SUBSEQUENT SIGNAL FROM VIEWER
`
`RETURN
`TO MEDIA
`STREAM?
`
`PURCHASE
`REQUESTED
`
`
`
`CONSUMMATE DESIRED TRANSACTION
`
`1220
`
`F.G. 12
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Page xiii
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 13 of 48 US 2007/0250901 A1
`
`
`
`-
`
`s
`S 1
`to Q
`O
`O) St
`CD
`
`>
`1.
`S
`L
`s
`
`S
`
`Page xiv
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 14 of 48 US 2007/0250901 A1
`
`1400
`
`1404
`
`1406
`
`1408
`
`1410
`
`
`
`
`
`
`
`
`
`OBTAN VIEWABLE
`MEDIA STREAM
`
`DIVIDE MEDIASTREAMINTO
`A PLURALTY OF SEGMENTS
`
`MAP AT LEAST ONE SEGMENT
`IDENTIFIER TO AN
`INTERMEDIATE NODE
`
`STORE FIRST MAPPING
`
`1412
`
`FIG. 14
`
`Page xv
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 15 of 48 US 2007/0250901 A1
`
`(STARD 1502
`
`/
`
`1500
`
`RECEIVE MEDIA STREAM
`
`DISPLAY MEDIA STREAM
`
`RECEIVE FIRST SIGNAL
`FROM VIEWER OF MEDIA STREAM
`
`DETERMINE SEGMENT DENTIFIER
`CORRESPONDING TO SIGNAL
`
`I
`
`1504
`
`1506
`
`1508
`
`1510
`
`DETERMINE SUPPLEMENTAL CONTENT
`MAPPED FROM CORRESPONDING
`SEGMENT DENTIFIER,
`WANTERMEDIATE NODE
`
`1511
`
`1512
`
`DISPLAY RETRIEVED
`SUPPLEMENTAL CONTENT
`
`1514
`
`1516
`
`
`
`
`
`RECEIVE SUBSEQUENT
`SIGNAL FROM VIEWER
`
`RETURN
`TO MEDIA
`STREAM
`2
`
`
`
`PURCHASE
`REQUESTED
`?
`
`
`
`1520
`
`CONSUMMATE DESRED TRANSACTION
`
`
`
`FIG. 15
`
`Page xvi
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 16 of 48 US 2007/0250901 A1
`
`1600
`
`/
`
`1604
`
`1606
`
`1608
`
`
`
`
`
`
`
`
`
`
`
`CREATE MEDIA STREAM
`
`ASSOCATE SUPPLEMENTAL
`CONTENT WITH MEDIA STREAM
`
`UPLOAD MEDIA STREAM AND
`ASSOCATIONS TO ACCESS PROVIDER
`
`FIG. 16
`
`Page xvii
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 17 of 48 US 2007/0250901 A1
`
`1700
`
`1704
`
`PROVIDE MEDIA STREAM
`FOR VIEWING
`
`
`
`
`
`RECEIVE VIEWER SIGNAL
`
`
`
`PROVIDE SUPPLEMENTAL CONTENT
`
`
`
`1708
`
`1710
`
`1712
`
`1714
`
`FIG. 17
`
`Page xviii
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 18 of 48 US 2007/0250901 A1
`
`1800
`
`/
`
`1802
`
`PLAY MEDIA STREAM
`
`DISPLAY ASSOCATED
`SUPPLEMENTAL CONTENT
`
`RECEIVE VIEWER SIGNA
`
`DISPLAY ADDITIONAL INFORMATION
`
`1812
`
`1804
`
`1806
`
`1808
`
`1810
`
`F.G. 18
`
`Page xix
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 19 of 48 US 2007/0250901 A1
`
`
`
`92
`C
`CD
`S
`E
`O
`O
`
`Page xx
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 20 of 48
`
`US 2007/025.0901 A1
`
`006|×
`
`
`
`Z555i,
`
`Page xxi
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 21 of 48 US 2007/0250901 A1
`
`
`
`
`
`O
`O
`O)
`
`3
`C
`CS
`O
`o
`S.
`O df; |
`
`2
`2
`S
`O
`O
`
`CD
`CD
`C
`CS
`?h
`o
`S.
`O
`CO
`
`N
`e
`&
`S.
`
`2
`2
`S
`(3
`
`N
`s
`&
`8
`
`?h
`O
`v
`
`CD
`II -
`
`O
`O)
`N
`
`CD
`-
`
`Page xxii
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 22 of 48
`
`US 2007/025.0901 A1
`
`
`
`XYZ - 100
`
`2002
`
`D
`
`N
`
`20022
`
`20023
`
`2000
`
`XYZ-210
`
`
`
`
`
`
`
`s
`
`2002n-1
`
`2002
`
`FIG. 20
`
`Page xxiii
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 23 of 48 US 2007/0250901 A1
`
`s
`
`
`
`
`
`Page xxiv
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 24 of 48
`
`US 2007/025.0901 A1
`
`
`
`ENON: OISTIWN MAOT:NO||O\/ ÅCIE WOO : EXHNEE)
`
`
`
`
`
`00||-ZXX
`
`
`
`
`
`
`
`
`
`
`
`00ZZ
`
`DDDDDDDDDDDDDDDDDDDD
`
`DDDDDDDDDDD
`
`D
`
`DDDDDDDDDD
`
`Page xxv
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 25 of 48 US 2007/0250901 A1
`
`
`
`S
`
`CY)
`CN
`
`
`
`H
`CO
`O
`
`w
`
`i
`?h
`Z O
`
`9 R
`O H
`
`V
`
`a
`
`CO
`
`3
`
`or
`
`CO
`
`CO
`C/D
`aC
`
`(5
`
`O.
`CO.
`d d
`
`Page xxvi
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 26 of 48
`
`US 2007/025.0901 A1
`
`
`
`Page xxvii
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 27 of 48
`
`US 2007/025.0901 A1
`
`
`
`
`
`
`
`
`
`
`009 Z
`
`00||
`
`Page xxviii
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 28 of 48
`
`US 2007/025.0901 A1
`
`
`
`
`
`Page xxix
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 29 of 48
`
`US 2007/025.0901 A1
`
`00/Z
`
`C|| || NEWSDES
`
`
`
`
`
`
`
`
`
`Page xxx
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 30 of 48 US 2007/0250901 A1
`
`
`
`
`
`
`
`
`
`s
`
`
`
`Page xxxi
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 31 of 48
`
`US 2007/025.0901 A1
`
`
`
`
`
`
`
`
`
`
`
`
`
`C]] (HENWEI/\ + CI| . || NEWSDES
`
`006Z
`
`Page xxxii
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 32 of 48 US 2007/0250901 A1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Page xxxiii
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 33 of 48
`
`US 2007/025.0901 A1
`
`
`
`A + CILNEW?ES
`
`Page xxxiv
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 34 of 48 US 2007/0250901 A1
`
`3200
`
`
`
`3300
`
`BRAND X PANTS
`BUY NOW: 135.00
`
`FIG. 33
`
`Page xxxv
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 35 of 48 US 2007/0250901 A1
`
`3200
`
`3400
`
`3400
`
`3400
`
`3400
`
`3400
`
`3400n
`
`FIG. 34
`
`
`
`3200
`
`S
`
`()
`
`THRASHING
`AT MOUNTAINX
`
`a FA
`
`Page xxxvi
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 36 of 48 US 2007/0250901 A1
`
`
`
`3602
`
`SUPPLEMENTAL CONTENT
`(ALTERNATE 3600- 3600 )
`
`MAPPING M1
`
`MAPPING M2
`
`MAPPING M3
`
`O
`
`(
`O
`
`3600
`
`36002
`
`3600
`
`FIG. 36
`
`Page xxxvii
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 37 of 48 US 2007/0250901 A1
`
`
`
`3602
`
`VIEWER
`RESPONSE
`
`3700
`1
`
`3700
`2
`
`3700
`3
`
`3600
`
`36002
`
`3600
`
`FIG. 37
`
`Page xxxviii
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 38 of 48 US 2007/0250901 A1
`
`
`
`3602
`
`IT'S THE
`PANTS
`
`3800
`
`3800
`
`3800
`
`3802
`
`38022
`
`38023
`
`FIG 38
`
`Page xxxix
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 39 of 48
`
`US 2007/025.0901 A1
`
`
`
`0069
`
`Page xl
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 40 of 48
`
`US 2007/025.0901 A1
`
`
`
`
`
`
`
`
`
`
`
`”B??OW NHWET ?OEGIA HñOÅ OL LNB LNOO TWINEWETddnS CICIW OLSHEHLO MOTIV OLEMIT nO\ q InOW
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`”EHOW NHWET ?OBC]|/\ \![\OÅ 01 INHINOO TWINEWETd&nS (IGW OL EXIT nO\ GTnOM
`
`
`
`
`
`
`
`
`
`ON @SEX O
`
`
`
`ON OSEX (Q)
`
`\_0007
`
`Page xli
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 41 of 48
`
`US 2007/025.0901 A1
`
`
`
`(~~~
`
`
`
`HOHWES LNB LNOO TVILNEWEIdcins
`
`
`
`
`
`Page xlii
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 42 of 48
`
`US 2007/0250901 A1
`
`
`
`Page xliii
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 43 of 48
`
`US 2007/025.0901 A1
`
`
`
`Page xliv
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 44 of 48
`
`US 2007/025.0901 A1
`
`
`
`| WH |XIS X CINV}{8 N
`
`67999999916000 : Od?@
`
`(~~~
`
`Page xlv
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 45 of 48
`
`US 2007/025.0901 A1
`
`
`
`spueoqMOUS X pueug
`
`Page xlvi
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 46 of 48
`
`US 2007/025.0901 A1
`
`
`
`
`
`Page xlvii
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 47 of 48 US 2007/0250901 A1
`
`4702
`
`ANNOTATION PROVIDER / ACCESS PROVIDER
`
`
`
`
`
`
`
`MEDIA
`STREAM
`
`
`
`
`
`SEGMENT ID
`
`FIG. 47
`
`Page xlviii
`
`

`

`Patent Application Publication
`
`Oct. 25, 2007 Sheet 48 of 48 US 2007/0250901 A1
`
`ANNOTATION PROVIDER / ACCESS PROVIDER
`
`4702
`
`
`
`
`
`
`
`
`
`
`
`DEVICE D
`
`MEDIA
`STREAM
`
`FIG. 48
`
`4800
`
`Page xlix
`
`

`

`US 2007/025.0901 A1
`
`Oct. 25, 2007
`
`METHOD AND APPARATUS FOR
`ANNOTATING MEDIA STREAMS
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`0001. This application claims the benefits of U.S. Provi
`sional Patent Application Ser. No. 60/788,367, filed Mar. 30,
`2006; Ser. No. 60/788,368, filed Mar. 30, 2006; Ser. No.
`60/747,379, filed May 16, 2006; Ser. No. 60/804,091, filed
`Jun. 6, 2006: Ser. No. 60/866,803, filed Nov. 21, 2006; and
`Ser. No. 60/871,705, filed Dec. 22, 2006, all of which are
`herein incorporated by reference in their entireties.
`
`FIELD OF THE INVENTION
`0002 The present invention relates generally to adver
`tising and relates more particularly to advertising relating to
`viewable media streams.
`
`BACKGROUND OF THE INVENTION
`0003. With the increasing use by television viewers of
`digital video recorders (DVRs) to avoid traditional commer
`cial advertising on television and in similar media, product
`placement is increasingly becoming important as a tool for
`advertisers to reach consumers. For example, an advertiser
`may arrange for a particular article to be used in a movie by
`a popular actress, with the expectation that Such use will
`project a positive product image to consumers and thereby
`increase sales.
`0004. However, product placement is not without its
`drawbacks. For one, artistic concerns typically dictate that
`product placement be substantially less obtrusive than tra
`ditional commercial advertising (e.g., prominent and
`repeated display of the name of the article is generally
`avoided), thus, it may not be clear to viewers what specific
`article or brand is being depicted. Additionally, even if the
`article is identifiable, a viewer's potential interest in learning
`more about or in purchasing the article may be eroded by the
`time delay between the article being depicted and the viewer
`having access to the information or to a purchasing oppor
`tunity.
`0005. Several methods have been proposed for allowing
`consumers to interact with media depicting product place
`ments, for example to retrieve additional information about
`a depicted article on demand. However, the enabling tech
`nology is still in its relatively early stages, and the user
`interfaces are therefore typically cumbersome. Moreover,
`the implementation methodologies tend to be computation
`ally complex and therefore quite expensive. In addition, if
`the advertising information relating to a product placement
`is embedded in the media stream itself, it is generally
`impractical or impossible to add, modify or delete this
`information. This decreases the value of the advertising over
`time, due to changes in various characteristics of the placed
`article, such as the article's retail price and/or availability.
`0006 Moreover, these same concerns often carry over to
`other popular but less traditional forms of media. For
`instance, services that provide on-demand access to video
`content over the Internet or another distribution channel
`(e.g., wherein a filmmaker creates and then uploads a video
`to an access provider and the access provider then makes the
`video available for viewing by others, potentially including
`the general public) are becoming increasingly popular. In
`Some cases, the access provider provides additional func
`
`tionality, Such as enabling viewers to search for specific
`video content of interest, ranking the videos by viewer
`popularity, and allowing the community of viewers to rate
`and post commentary about particular videos.
`0007 Providing such services is expensive, particularly
`with respect to the storage costs and bandwidth transmission
`costs associated with presenting significant volumes of
`Video content to a large audience. Therefore, a robust
`revenue stream is required to make a video access provider
`financially viable and sustainable. Present access providers
`attempt to generate revenue by various means including by
`charging users a video hosting fee, by charging viewers a
`Subscription or pay-per-view fee, or, most popularly, by
`presenting advertising information to the viewing audience
`via pre-video or post-video commercials, banner ads, pop-up
`ads, and other types of advertising. In the latter case, revenue
`models for online video access providers typically face the
`same challenge that traditional media faces: how to associate
`effective advertising with the video content while not
`appearing to intrude on the viewer experience or on the
`creative integrity of the filmmaker.
`0008. In naive advertising models, the advertising that is
`presented to viewers of online video is selected randomly
`and without reference to the subject matter or other charac
`teristics of the specific video content being viewed. In more
`Sophisticated models, access providers strive to improve the
`relevancy of the advertising they present by leveraging the
`data and processes that are intended primarily to enable
`viewers to find video content of interest. This can be
`accomplished, for example, by referencing information from
`the video itself (generated, for example, using speech rec
`ognition, text recognition, or other technologies for deter
`mining the Subject matter of a given video) and/or by
`referencing the search terms that have been input by viewers
`in their efforts to locate a specific video.
`0009. Although such methods have proven to be some
`what effective in enabling video “search” in the online video
`context, the subject matter-related information that is
`obtained and/or utilized by such methods is generally too
`inaccurate, too misleading, too vague, too specific, and/or
`too voluminous to enable the effective selection and presen
`tation of relevant advertising.
`0010. The deficiencies described above are compounded
`by the fact that effective advertising targeting requires
`subject matter information to be comprehended within the
`context of the overall viewing experience. Currently, the
`automated systems that are necessary to deal with the
`tremendous volume of videos that are hosted by successful
`online video access providers (particularly with respect to
`“user-generated content) are incapable of addressing Such
`contextual elements. For example, even if an automated
`system were to recognize that a given video contained
`Subject matter regarding “buying a new sport utility vehicle
`(SUV), the system might fail to comprehend and/or take
`into account the fact that the context of the subject matter
`was a warning about the adverse impact of SUVs on global
`warming (and, as such, represented an undesirable adver
`tising placement for an SUV manufacturer).
`0011
`Furthermore, current advertising targeting systems
`lack the capacity to address the important connection
`between the subject matter and context of a video and the
`demographics and interests of likely viewers of Such video.
`For example, absent specific instructions that have been
`formulated in advance (and are, therefore, very costly to
`
`Page 1
`
`

`

`US 2007/025.0901 A1
`
`Oct. 25, 2007
`
`provide on a large scale), current systems are incapable of
`Surmising that a viewer who is interested in viewing a video
`warning against purchasing an SUV might also be interested
`in viewing advertising about seemingly unrelated Subject
`matter such as organic foods, energy-saving lightbulbs, trips
`to see the melting glaciers of Kilimanjaro or the campaign
`of a local “Green” politician.
`0012 Finally, many of the methods that are typically used
`by access providers to present advertising (e.g., pre-roll,
`pop-up, or overlay advertising) are intrusive and/or disrup
`tive of the viewing experience, thereby interfering with the
`filmmaker's creative control over the presentation of his or
`her video and marring the quality of the viewer's viewing
`experience.
`0013 For these and other reasons, advertising methods
`conventionally used by online video access providers cause
`filmmakers to be less willing to create or share videos,
`viewers to be less willing to watch videos, and advertisers to
`place a lesser value on the advertising “inventory' offered
`for sale by the access providers. As a result, online video
`access providers fail to maximize the potential revenue
`opportunity represented by their viewing audiences.
`0014 Thus, there is a need in the art for a method and
`apparatus for annotating media streams such as television,
`hosted online video and the like to provide more effective,
`user-friendly advertising.
`
`SUMMARY OF THE INVENTION
`In one embodiment, the invention is a method and
`0.015.
`apparatus for annotating media streams. One embodiment of
`a method for facilitating creation of an annotated media
`stream by a user includes receiving the media stream from
`the user, mapping at least one item of Supplemental content
`to at least a portion of the media stream to produce the
`annotated media stream, and storing the annotated media
`Stream.
`0016. In another embodiment, a method for distributing
`an annotated media stream provided by a user includes
`receiving the annotated media stream from the user, where
`the annotated media stream includes at least one item of
`Supplemental content mapped to at least a portion of a media
`stream to produce the annotated media stream, storing the
`annotated media stream, receiving a request from a viewer
`to view the annotated media stream, and distributing the
`annotated media stream to the viewer.
`0017. In another embodiment, a method for distributing
`an annotated media stream provided by a user includes
`distributing the annotated media stream to a viewer, where
`the annotated media stream includes at least one item of
`Supplemental content mapped to at least a portion of a media
`stream to produce the annotated media stream, and collect
`ing a commission when the annotated media stream is
`viewed by the viewer. In one embodiment, the collecting
`requires further action on a part of the viewer (such as
`selection of a hyperlink contained in the at least one item of
`Supplemental content or consummation of a commercial
`transaction).
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`0018. The teaching of the present invention can be
`readily understood by considering the following detailed
`description in conjunction with the accompanying drawings,
`in which:
`
`0019 FIG. 1 is a schematic diagram illustrating one
`embodiment of a television value chain, according to the
`present invention;
`0020 FIG. 2 is a flow diagram illustrating one embodi
`ment of a method for producing coded media streams;
`0021
`FIG. 3 is a flow diagram illustrating one embodi
`ment of a method for detecting and readings tags on articles
`appearing in a media stream, according to the present
`invention;
`0022 FIG. 4 is a flow diagram illustrating another
`embodiment of a method for producing coded media
`Streams;
`0023 FIG. 5 is a flow diagram illustrating another
`embodiment of a method for producing coded media
`Streams;
`0024 FIG. 6 is a flow diagram illustrating one embodi
`ment of a method for annotating media streams, according
`to the present invention;
`0025 FIG. 7 is a flow diagram illustrating another
`embodiment of a method for annotating media streams,
`according to the present invention;
`0026 FIG. 8 is a flow diagram illustrating one embodi
`ment of a method for providing annotated media streams to
`viewers, according to the present invention;
`0027 FIG. 9 is a flow diagram illustrating a second
`embodiment of a method for providing annotated media
`streams to viewers, according to the present invention;
`0028 FIG. 10 is a schematic diagram illustrating an
`exemplary display for presenting retrieved information
`about articles appearing in a portion of a media stream;
`0029 FIG. 11 is a flow diagram illustrating a third
`embodiment of a method for providing annotated media
`streams to viewers, according to the present invention;
`0030 FIG. 12 is a flow diagram illustrating a fourth
`embodiment of a method for providing annotated media
`streams to viewers, according to the present invention;
`0031
`FIG. 13 is a high level block diagram of the media
`stream annotation method that is implemented using a
`general purpose computing device;
`0032 FIG. 14 is a flow diagram illustrating another
`embodiment of a method for annotating media streams,
`according to the present invention;
`0033 FIG. 15 is a flow diagram illustrating a fifth
`embodiment of a method for providing annotated media
`streams to viewers, according to the present invention;
`0034 FIG. 16 is a flow diagram illustrating one embodi
`ment of a method for associating Supplemental content with
`a media stream, according to the present invention;
`0035 FIG. 17 is a flow diagram illustrating one embodi
`ment of a method for providing a service for hosting a media
`stream with at least one mapping to Supplemental content;
`0036 FIG. 18 is a flow diagram illustrating one embodi
`ment of a method for displaying a media stream with
`mapped Supplemental content to a viewer;
`0037 FIG. 19 depicts an exemplary display device:
`0038 FIG. 20 illustrates an exemplary media stream that
`has been logically divided into a plurality of segments;
`0039 FIG. 21 illustrates an exemplary mapping in which
`segment identifiers identifying segments of the media stream
`of FIG. 20 are mapped to items of supplemental content;
`0040 FIG. 22 illustrates an exemplary mapping in which
`segments of the media stream illustrated in FIG. 20 are
`associated with Supplemental information;
`
`Page 2
`
`

`

`US 2007/025.0901 A1
`
`Oct. 25, 2007
`
`FIG. 23 illustrates an exemplary filming location in
`0041
`which a plurality of articles are tagged;
`0042 FIG. 24 depicts an image capturing device with
`integrated RFID sensor system;
`0043 FIG. 25, for example, illustrates an exemplary tag
`log that identifies the active tags associated with each
`segment identifier identifying a media stream segment;
`0044 FIG. 26 illustrates an exemplary set-top box that
`interacts with user signals provided via a remote control;
`004.5
`FIG. 27 illustrates the system of FIG. 26 after the
`Supplemental content mapped to the segment identifier has
`been retrieved and displayed to the viewer via the display
`device;
`0046 FIG. 28 illustrates the system of FIG. 26 after the
`viewer has indicated interest in a single article for which
`Supplemental content is mapped;
`0047 FIG. 29 illustrates the use of viewer-driven con
`straints in the display of Supplemental content;
`0048 FIG. 30 illustrates the results of using viewer
`driven constraints to select Supplemental content;
`0049 FIG. 31 illustrates an exemplary system in which a
`viewer signal indicates that the viewer wishes to “book
`mark Supplemental content mapped to the current seg
`ment's segment identifier for later viewing:
`0050 FIG. 32 illustrates an exemplary media stream that
`has been loaded onto a computing device;
`0051
`FIG. 33 illustrates an exemplary item of supple
`mental content that may be mapped to the media stream
`illustrated in FIG. 32:
`0052 FIG. 34 illustrates the exemplary media stream of
`FIG. 32, where the media stream has been logically divided
`into temporal segments to facilitate mapping of Supplemen
`tal content;
`0053 FIG. 35 illustrates the exemplary media stream of
`FIG. 32, wherein an item of supplemental content is mapped
`Such that the item of Supplemental content is displayed
`simultaneously with at least a segment of the media stream;
`0054 FIG. 36 illustrates three items of similar supple
`mental content for potential mapping to a segment identifier
`associated with a segment of a media stream;
`0055 FIG. 37 illustrates click-through counters for each
`respective item of rotated Supplemental content illustrated in
`FIG. 36:
`0056 FIG. 38 illustrates click-through counters for dif
`ferent respective items of rotated Supplemental content;
`0057 FIG. 39 illustrates the dynamics of an exemplary
`syndication team, according to the present invention;
`0058 FIG. 40 illustrates an exemplary initial display for
`presentation to a content creator;
`0059 FIG. 41 illustrates an exemplary display that may
`be presented to an advertiser to allow search of media
`streams associated with a given item of Supplemental con
`tent;
`0060 FIG. 42 illustrates an exemplary display in which
`results of a search such as that enabled by the display of FIG.
`41 are presented;
`0061
`FIG. 43 illustrates an exemplary display in which
`an advertiser is enabled to block association of an item of
`Supplemental content with a given media stream;
`0062 FIG. 44 illustrates an exemplary scenario in which
`the ability to provide an item of supplemental content
`relating to a given article appearing a media stream is
`auctioned off to various sellers;
`
`0063 FIG. 45 illustrates an exemplary display in which
`the Supplemental content associated with a media stream is
`used to generate related banner advertising:
`0064 FIG. 46 illustrates the use of an initial annotated
`media stream across various platforms including a Social
`networking website, a web log and a portable media player;
`0065 FIG. 47 illustrates another application of aspects of
`the present invention; and
`0.066
`FIG. 48 illust