US009697230B2
`
`(12) United States Patent
`Houh et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 9,697.230 B2
`Jul. 4, 2017
`
`(54) METHODS AND APPARATUS FOR
`DYNAMIC PRESENTATION OF
`ADVERTISING, FACTUAL AND
`INFORMATIONAL CONTENT USING
`ENHANCED METADATA IN
`SEARCH-DRIVEN MEDIA APPLICATIONS
`(75) Inventors: Henry Houh, Lexington, MA (US);
`sey Nathan Stern, Belmont, MA
`(US)
`(73) Assignee: CXENSE ASA, Olso (NO)
`(*) Notice:
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 953 days.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`5,613,034 A
`3/1997 Ney et al. ...................... 395.26
`5,613,036 A
`3/1997 Strong ............
`395.252
`6,006,265 A * 12/1999 Rangan et al. ............... TO9,226
`6,064,959 A
`5/2000 Young et al.
`...? A 858 sing s al. ............... 704/257
`rue et al.
`6, 157,912 A 12/2000 Kneser et al. ................ 704/27O
`6,345,253 B1
`2/2002 Viswanathan
`6,418,431 B1
`7/2002 Mahajan et al. .................. 7O7/4
`6,484,136 B1
`11/2002 Kanevsky et al. ................ TO4/9
`6,501,833 B2 12/2002 Phillips et al. ............ 379.88.07
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`(21) Appl. No.: 11/395,608
`
`(22) Filed:
`
`Mar. 31, 2006
`
`EP
`JP
`
`6, 2000
`1 008 931 A2
`9, 2004
`200435O253
`(Continued)
`
`(65)
`
`Prior Publication Data
`
`OTHER PUBLICATIONS
`
`May 10, 2007
`US 2007/010676O A1
`Related U.S. Application Data
`(60) Provisional application No. 60/736,124, filed on Nov.
`9, 2005.
`(51) Int. Cl.
`G06F 5/16
`G06F 7/30
`(52) U.S. Cl.
`CPC. G06F 17/30247 (2013.01); G06F 17/30787
`(2013.01); G06F 17730796 (2013.01); G06F
`17/30817 (2013.01); G06F 17/30852
`(2013.01)
`
`(2006.01)
`(2006.01)
`
`(58) Field of Classification Search
`USPC ..... 707/3, 10, 4, 5, 102; 705/50, 51, 57, 59;
`725/119, 134, 135, 136, 46, 9, 37
`See application file for complete search history.
`
`International Search Report for International Application No. PCT/
`US2006/043561, Date of Mailing May 9, 2007 (3 pages).
`(Continued)
`Primary Examiner — Brian Whipple
`(74) Attorney, Agent, or Firm — Tarter Krinsky & Drogin
`LLP
`
`ABSTRACT
`(57)
`A computerized method and apparatus is disclosed for
`dynamic presentation of advertising, factual, informational
`content and combinations thereof. In particular, the adver
`tising content is dynamically presented according to the
`playback of corresponding segments identified within a
`media file or stream.
`
`20 Claims, 18 Drawing Sheets
`
`
`
`Timed Ad Placement Agent
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 001
`
`

`

`US 9,697.230 B2
`Page 2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`6,546,427 B1 * 4/2003 Ehrlich et al. ................ TO9,231
`6,611,803 B1
`8/2003 Furuyama et al. ........... TO4,254
`6,671,692 B1
`12/2003 Marpe et al.
`6,687,697 B2
`2/2004 Collins-Thompson et al. .. 707/6
`6,691,123 B1
`2/2004 Gulliksen
`6,697.796 B2
`2/2004 Kermani
`6,728,673 B2
`4/2004 Furuyama et al. ........... TO4,254
`6,728,763 B1
`4/2004 Chen .
`. . . . .
`TO9,219
`6,738,745 B1
`5/2004 Navratil et al. .............. 7041277
`6,748,375 B1
`6, 2004 Wong et al.
`6,768.999 B2
`7/2004 Prager et al.
`6,785,688 B2
`8, 2004 Abajian et al.
`6,816,858 B1
`1 1/2004 Coden et al. ..................... 707/5
`6,848,080 B1
`1/2005 Lee et al. ...................... 71.5/533
`6,856,997 B2
`2/2005 Lee et al.
`6,859,799 B1
`2, 2005 Yuen
`6,873,993 B2
`3/2005 Charlesworth et al.
`6,877,134 B1
`4/2005 Fuller et al.
`6,973,428 B2 13. ETA. et al. r TO4/251
`gi 8. R .2. eyt get a.
`7,120,582 B1
`10/2006 Young et al.
`7, 177,881 B2
`2/2007 Schwesig et al.
`709/204
`7,222,155 B1* 5/2007 Gebhardt et al.
`7,260,564 B1* 8/2007 Lynn et al. ....................... 707/3
`7,308,487 B1
`12/2007 Dansie et al.
`7,337.462 B2 * 2/2008 Dudkiewicz ...... GO6F 17/30017
`348/465
`
`9/2010 Houh et al.
`7,801.910 B2
`2001/0045962 A1 11, 2001 Lee et al. ...................... 345,745
`2001.0049826 A1 3
`12, 2001 Wilf
`20020052925 A1
`5/2002 Kim et al. .................... 709/217
`2002fOO69218 A1
`6, 2002 Sull et al.
`2002fOO996.95 A1
`7/2002 Abaiian et al.
`2002/0108112 A1
`8, 2002 Wallace et al.
`2002/0133398 A1* 9, 2002 Geller et al. .................... TO5/14
`2002/0143852 A1 10, 2002 Guo et al.
`TO9,201
`2003/O123841 A1* 7/2003 Jeannin ........................... 386/46
`2003/0171926 A1
`9, 2003 Suresh et al.
`704/270.1
`2004/0103433 A1
`5/2004 Regeard et al. ................ 7.25/53
`2004/01995.02 A1 10/2004 Wong et al.
`2004/0199507 Al
`10/2004 Tawa, Jr. .......................... 707/7
`2004/0205535 A1 10, 2004 Newman et al.
`2005/0033758 A1
`2/2005 Baxter .......................... 7O7/1OO
`2005/0086692 A1* 4/2005 Dudkiewicz et al. .......... T25/46
`2005, 009691.0 A1
`5, 2005 Watson et al.
`2005/O165771 A1
`7, 2005 GO et al.
`2005/O187965 A1
`8/2005 Abajian
`2005/0197724 A1
`9/2005 Neogi ............................. TOO/94
`2005, 0198570 A1* 9, 2005 Otsuka .............. GO6F 17,30787
`71.5/2O1
`
`9, 2005 Morton et al.
`2005/0216443 A1
`2005/0229118 A1 10, 2005 Chiu et al.
`2005/0234875 A1 10, 2005 Auerbach et al. ................ 707/3
`2005/0256867 A1 11, 2005 Walther et al. .
`707/5
`2006, OO15904 A1
`1/2006 Marcus ........
`725/46
`2006/0020662 A1
`1/2006 Robinson .
`709/203
`2006.0020971 A1
`1/2006 Poslinski .
`725/44
`2006,0047580 A1
`3, 2006 Saha ............
`os
`2006/0053156 A1
`3/2006 Kaushansky et al. ........ 707/102
`2006/0265421 A1 11/2006 Ranasinghe et al.
`2007,0005569 A1
`1/2007 Hurst-Hiller et al.
`2007/0041522 A1
`2/2007 Abella et al.
`2007/0078708 A1
`4/2007 Yu et al.
`2007/01OO787 A1
`5, 2007 Lim et al.
`2007/0106646 A1
`5, 2007 Stern et al.
`2007/0106660 A1
`5, 2007 Stern et al.
`2007/0106685 A1
`5, 2007 Houh et al.
`2007/0106693 A1
`5, 2007 Houh et al.
`2007/O118873 A1
`5, 2007 Houh et al.
`2007/0183741 A1* 8, 2007 Lerman ............. GO6F 17/30017
`386,249
`
`
`
`2008/0040224 A1
`
`2007/0233.879 A1* 10, 2007 Woods ................... G06Q 30/02
`709,227
`2/2008 Roker .................... G06Q 30/02
`TO5, 14.66
`
`2009, 0222442 A1
`
`9, 2009 Houh et al.
`
`FOREIGN PATENT DOCUMENTS
`
`2002O24865 A1
`KR
`WOO2,11123 A2
`WO
`WO WO 2005/0.04442 A1
`WO
`2007/056485 A2
`WO
`2007/056531 A1
`WO
`2007/056532 A1
`WO
`2007/056534 A1
`WO
`2007/056535 A2
`
`4/2002
`2/2002 ............. G1OL 17.00
`1, 2005
`5/2007
`5/2007
`5/2007
`5/2007
`5/2007
`
`OTHER PUBLICATIONS
`
`Van Thong, et al., "SPEECHBOT: An Experimental Speech-Based
`Search Engine for Multimedia Content in the Web.” Cambridge
`Research Laboratory, Technical Report Series, 19 pages, Jul. 2001.
`International Search Report for PCT/US2006/043683 dated Aug. 2,
`2007 (3 pages).
`Written Opinion of the International Searching Authority for PCT/
`US2006/043683 dated Aug. 2, 2007 (6 pages).
`Stuart, Anne, “SpeechBot: A Search Engine for Sound,” http://www.
`hpl.hp.com/cgi-bin/pfcgi, last visited on Nov. 6, 2006 (4 pages).
`“BBN Audio Indexer.” BBN Technologies, A Verizon Company,
`last visited on Nov. 6, 2006, http://web.archive.org/web/
`20020214165834.http://www.bbn.com/speech/audioindexer.html,
`approximately Feb. 14, 2002, as indicated on Wayback Machine
`website
`at
`http://web.archive.org/web/*/http://www.bbn.com/
`speech/audioindexer.html, last visited on Nov. 6, 2006 (3 pages).
`International Search Report for International Application No. PCT/
`US2006 043680, date of mailing Apr. 27, 2007 (3 pages).
`International Search Report for International Application No. PCT/
`US2006/043682, date of mailing Mar. 30, 2007, including Written
`Opinion of the International Searching Authority (10 pages total).
`International Search Report for PCT/US2006/043561, dated Apr.
`17, 2007 (3 pages).
`Lindblad, Christopher J., et al., “ViewStation Applications: Impli
`cations for Network Traffic.' IEEE Journal of Selected Areas in
`Communications, 1995.
`Little, T.D.C., et al., “A Digital On-Demand Video Service Sup
`porting Content-Based E. International Multimedia
`E.
`ence, Proceedings of the First ACM International Conference on
`Multimedia, Anaheim, CA, Sep. 1993, 10 pages.
`Van Thong, Jean-Manuel "SpeechBot: a Speech Recognition based
`Audio Indexing System for the Web”, 2000, 10 pages, available on
`internet at: http://apotheca.hpl.hp.com/ftp/pub/compaq/CRL/publi
`cations/mvt/speechbotRIAO2000.pdf.
`International Preliminary Report on Patentability received for PCT
`Patent Application No. PCT/US2006/043682, issued on May 14,
`2008, 6 pages.
`International Preliminary Report on Patentability received for PCT
`Patent Application No. PCT/US2006/043683, issued on May 14,
`2008, 6 pages
`--
`--
`s
`Bregler et al., “Video Rewrite: Driving Visual Speech with Audio'.
`ACM SIGGRAPH, 1997, pp. 1-8.
`International Preliminary Report on Patentability received for PCT
`atent Application No. PCT/US2006/043561, issued on May 14
`E6 R
`y 14,
`International Search Report and Written Opinion received for PCT
`Patent Application No. PCT/US2006/043679, mailed on Apr. 27.
`2007, 7 pages.
`International Preliminary Report on Patentability received for PCT
`Patent Application No. PCT/US2006/043679, issued on May 14,
`2008, 6 pages.
`International Preliminary Report on Patentability received for PCT
`Patent Application No. PCT/US2006/043680, issued on May 14,
`2008, 6 pages.
`
`* cited by examiner
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 002
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 1 of 18
`
`US 9,697.230 B2
`
`Content
`Descriptor
`
`
`
`
`
`27
`
`
`
`
`
`Media
`indexer 10
`
`
`
`Descriptor
`indexer 50
`
`F.G. 1A
`
`index
`
`40
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 003
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 2 of 18
`
`US 9,697.230 B2
`
`10 N.
`
`
`
`COntent
`Descriptor
`
`
`
`
`
`
`
`Media
`indexing
`Controller
`
`110
`
`Speech
`Recognition
`ProCeSSOr
`100a
`
`Natural
`Language
`Processor
`1 OOb
`
`Video Frame
`
`Analyzer ver 100C
`
`Non-speech
`Audio
`Analyzer
`1 OOC
`
`Marker
`Extractors
`
`1 OOe
`
`Embedded
`Metadata
`ProceSSOS
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Timed
`Word
`Segments
`
`105a
`
`Timed Audio
`Speech
`Segments
`
`105b
`
`Timed Video
`Segments
`
`105C
`Timed
`Non-speech
`Audio
`Segments
`
`105d
`
`Timed
`Marker
`Segments
`
`- 105e
`
`Attributes
`
`105f
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 004
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 3 of 18
`
`US 9,697,230 B2
`
`YACWSH><AYVWINNS><31LL><TaN> ooz7
`eGEZ4++++y<FONACIANOO><NOLLWENG><LIS34OGNa><15S44OLYVLS><adALLNAWOASHOIIdsOlanv><al'Das>
`
`
`
`
`
`IGEZaSEZpezOgezZqsez
`
`
`
`
`
`aGS¢EGG+YvYY<NOLLWYNG><13S44O'GON3><14S4IOLYVLS><4dALINSWOASO1GNVHO3adS-NON><dl'O4S>
`8GS2PSsze9G45¢
`
`
`
`
`
`
`
`eG9¢¢yy¢¢<NOILWaNd><LaSIdOGN3><LASIAOLYVLS><3dALLNAWOASYSNYVN><dl94S>
`
`
`
`
`eSbzyyy$‘<NOLLVYNG><LASIAOGNA><LASAAOLYVLS><3dALINSWOASOAGIA><dl'O3S>
`2G27qs7z-BGzzyy+++‘<JONAGISANOO><NOLLVYENG><LASAdOGNa><LISAIOLYVLS><GYOM><dl'D3AS>
`
`
`8G97¢pG9z9492qS97
`
`
`
`eSb7pcrzoG¢ZqSbz
`
`1S7zG27pgzz
`eGlZYt+oY<SALNGIYLIY
`PSLZIGLCQGl2
`
`
`éOld
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 005
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 005
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 4 of 18
`
`US 9,697,230 B2
`
`...in the 'state' of the union
`address tuesday night
`presidentbushy Outlined proposals
`
`- 320
`
`420
`
`
`
`Snippet
`Generator
`Module
`
`
`
`440
`
`FIG. 4
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 006
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 5 of 18
`
`US 9,697.230 B2
`
`Search
`
`510
`
`Obtain
`Metadata
`Results
`
`515
`
`585
`
`Get First
`
`Return Search
`
`525
`
`582
`
`535
`
`Get Next
`Metadata
`Result
`
`
`
`
`
`
`
`
`
`Parse Metadata
`for Content
`Segment info
`
`DOC
`Confidence OK?
`
`Determine
`SegmentType
`Preference
`
`Get First
`Segment info
`
`
`
`Segment
`Confidence OK?
`550
`
`
`
`Keyword(s)
`Found?
`
`
`
`
`
`
`
`
`
`
`
`580
`Add to Search
`Result
`
`Get Next
`Segment info
`
`F.G. 5
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 007
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 6 of 18
`
`US 9,697.230 B2
`
`
`
`
`
`
`
`S
`
`620
`
`F.G. 6A
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 008
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 7 of 18
`
`US 9,697.230 B2
`
`
`
`Present Search Snippet
`Including User Actuated
`Display Element
`
`710
`
`Determine Media Playback
`Offset in Response to
`Element Actuation
`
`720
`
`Compare Playback Offset
`with Timing Offsets of
`Media Content Segments
`
`730
`
`Playback Media Content
`Starting at Next Segment
`Offset
`
`740
`
`F.G. 6B
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 009
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 8 of 18
`
`US 9,697.230 B2
`
`
`
`| | | | | | |
`
`{
`
`+ – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – – –}
`
`“¿?|098(soldo! -6(3) squau69S
`
`
`
`
`G88|?oeeds opnw peu]] |
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 0010
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 9 of 18
`
`US 9,697.230 B2
`
`642
`
`644
`
`646
`
`648
`
`
`
`610
`?
`
`900a
`
`Facts
`
`900
`
`Ads
`
`information
`
`FIG. 7A
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 0011
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 10 of 18
`
`US 9,697.230 B2
`US 9,697,230 B2
`
`c06
`
`wor
`
`
`
`1206
`
`
`
`syods
`
`je007]
`
`
`
`yoegheig
` 1206
` [206
`Eauzo6|5206
`yuUeWUTeLe}Uy
`souisngoePZ206esGz06|©206
`
`3206
`
`snonunuod
`
`n
`N
`CD
`I
`
`Buruiy
`
`oy
`
`Out]
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 0012
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 0012
`
`
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 11 of 18
`
`US 9,697.230 B2
`
`3.
`
`i
`
`i
`
`
`
`l
`2
`
`C
`g
`
`o
`CCC
`
`
`
`w-
`
`S
`-
`
`
`
`
`
`o
`
`o,
`
`2 |H||
`
`
`
`s
`
`O
`C
`
`i
`
`i
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 0013
`
`

`

`U.S. Patent
`U.S. Patent
`
`Jul. 4, 2017
`
`v06vL6ZLG
`
`
`
`deyauibuyeulbug
`
`eyeqpyAouengjay
`
`|LL6ejeqJessy
`
`O16806906
`
`EVEpepoW
`
`paoueyuyospi//olpny
`
`w {|
`
`----|
`5| ?
`
`————————————————————————————————————————---- |
`~——Jo—_—
`
`(/)
`º+)
`eN?
`Sheet 12 of 18
`
`©|
`
`©|
`w?|
`
`US 9,697.230 B2
`US 9,697,230 B2
`
`V8Sls
`
`
`
`juabyjuawece|gpypew)
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 0014
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 0014
`
`
`
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 13 of 18
`
`US 9,697.230 B2
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Obtain Data Map Associating
`Ads To Content Segments
`940
`
`DOWnload Audio/Video Content
`With Enhanced Metadata
`942
`
`Start Playback Audio/Video
`Content
`944
`
`Track Playback Status
`Using Enhanced Metadata
`946
`
`Detect Transition From Previous
`To Current Content Segment
`948
`
`Notify Advertising Engine
`Of Current Content Segment
`950
`
`Determine Ads To Present
`During Current Content Segment
`Using Data Map
`952
`
`DOWnload Ads To Client
`For Presentation
`954
`
`Present Ads
`956
`
`FG. 8B
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 0015
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 14 of 18
`
`US 9,697.230 B2
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Obtain Data Map Associating
`Ads To Content Segments
`960
`
`DOWnload Audio/Video Content
`With Enhanced Metadata and Ads
`962
`
`Start Playback Audio/Video
`Content
`964
`
`Track Playback Status
`Using Enhanced Metadata
`966
`
`Detect Transition From Previous
`To Current Content Segment
`968
`
`Notify Advertising Engine
`Of Current Content Segment
`970
`
`Determine Ads To Present
`During Current Content Segment
`Using Data Map
`972
`
`Download Ad Identifiers to Client
`For Selecting Ads To Present
`974
`
`Select And Present Ads
`According To Ad Identifiers
`956
`
`FG. 8C
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 0016
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 15 of 18
`
`US 9,697.230 B2
`
`
`
`
`
`
`
`
`
`
`
`Obtain Data Map Associating
`Ads To Content Segments
`980
`
`DOWnload Audio/Video Content
`With Enhanced Metadata,
`Data Map And Ads
`982
`
`Start Playback Audio/Video
`Content
`984
`
`Track Playback Status
`Using Enhanced Metadata
`986
`
`Detect Transition From Previous
`To Current Content Segment
`988
`
`
`
`
`
`
`
`
`
`
`
`Determine Ads To Present
`During Current Content Segment
`Using Data Map
`990
`
`Present Ads
`992
`
`FIG. 8D
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 0017
`
`

`

`U.S. Patent
`
`r>< E
`
`----
`
`US 9,697.230 B2
`
`
`
`|Spy 0 || 6806906
`
`
`
`* ———————————————————————————————————————- - -
`
`
`
`
`
`
`
`?ue6v queuuepeld pwy pºul L
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 0018
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 17 of 18
`
`US 9,697.230 B2
`
`Obtain Data Map Associating
`Ads To Content Segments AND
`Enhanced Metadata For
`Audio/Video Content
`1000
`
`
`
`DOWnload Audio/Video Content
`1002
`
`Start Playback Audio/Video
`Content
`1004
`
`Track Playback Status
`Using Enhanced Metadata
`And Playback Status Indicators
`From Client
`1006
`
`Detect Transition From Previous
`To Current Content Segment
`1008
`
`Determine Ads To Present
`During Current Content Segment
`Using Data Map
`1010
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`DOWnload Ads To Client
`For Presentation
`1012
`
`Present Ads
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 0019
`
`

`

`U.S. Patent
`
`Jul. 4, 2017
`
`Sheet 18 of 18
`
`US 9,697.230 B2
`
`Obtain Data Map Associating
`Ads To Content Segments AND
`Enhanced Metadata For
`Audio/Video Content
`1100
`
`
`
`DOWnload Audio/Video Content
`AND ACS
`1102
`
`Start Playback Audio/Video
`Content
`1104
`
`Track Playback Status
`Using Enhanced Metadata
`And Playback Status indicators
`From Client
`1106
`
`Detect Transition From Previous
`To Current Content Segment
`1108
`
`Determine Ads To Present
`During Current Content Segment
`Using Data Map
`1110
`
`DOWnload Ad Identifiers To Client
`For Selecting Ads To Present
`1112
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Select And Present Ads
`According To Ad identifiers
`1114
`
`FIG. 9C
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 0020
`
`

`

`US 9,697,230 B2
`
`1.
`METHODS AND APPARATUS FOR
`DYNAMIC PRESENTATION OF
`ADVERTISING, FACTUAL AND
`INFORMATIONAL CONTENT USING
`ENHANCED METADATA IN
`SEARCH-DRIVEN MEDIA APPLICATIONS
`
`2
`binations thereof (hereinafter referred to as “advertising
`content generally). In particular, the advertising content is
`dynamically presented according to the playback of corre
`sponding segments identified within a media file or stream.
`According to one embodiment, the method and apparatus
`involves a client receiving media content and corresponding
`metadata for playback. The metadata identifies content seg
`ments and corresponding timing information derived from
`the media content using one or more media processing
`techniques. During playback of the media content, the client
`identifies the content segment in playback by comparing a
`playback status with the timing information from the meta
`data. Once the content segment is identified, the client
`presents a set of factual, informational, or advertising con
`tent associated with the content segment in playback.
`In Such embodiments, the method and apparatus can
`receive media content and corresponding metadata for play
`back, Such that the metadata identifies content segments and
`corresponding timing information derived from the media
`content using one or more media processing techniques.
`During playback of the media content, the content segment
`in playback can be identified by comparing a playback status
`with the timing information from the metadata. Once the
`content segment is identified, a set of factual, informational,
`or advertising content associated with the content segment in
`playback is presented.
`In Such embodiment, the method and apparatus can pro
`vide an identification of the content segment in playback to
`a server, and receive the set of factual, informational, or
`advertising content associated with the content segment in
`playback from the server.
`In such embodiments, the method and apparatus can
`receive a plurality of factual, informational, or advertising
`content from a server, provide an identification of the
`content segment in playback to the server, and receive a set
`of instructions or data from the server identifying the set of
`factual, informational, or advertising content from among
`the plurality of factual, informational, or advertising content
`for presentation.
`In Such embodiments, the method and apparatus can
`obtain a data map that associates a corresponding set of
`factual, informational, or advertising content with each of
`the content segments of the media content identified in the
`metadata. During playback of the media content, the content
`segment in playback is identified by comparing the playback
`status with the timing information from the metadata. Using
`the data map, the corresponding set of factual, informa
`tional, or advertising content associated with the content
`segment in playback is determined and presented concur
`rently or relative to the content segment in playback.
`In Such embodiment, the content segments can include
`word segments, audio speech segments, video segments and
`marker segments. For example, the content segments of the
`media content can correspond to a word, a phrase, a sen
`tence, a paragraph, a story, or a topic. The content segments
`of the media content can also correspond to a speaker
`identified during the one or more media processing tech
`niques.
`According to another embodiment, the method and appa
`ratus involves a server receiving obtaining metadata corre
`sponding to media content in playback at a client. The
`metadata identifies content segments and corresponding
`timing information derived from the media content using
`one or more media processing techniques. During playback
`of the media content, the server receives a playback status of
`the media content from the client. The server identifies the
`current content segment in playback by comparing the
`
`RELATED APPLICATIONS
`
`This application claims the benefit of U.S. Provisional
`Application No. 60/736,124, filed on Nov. 9, 2005. The
`entire teachings of the above application are incorporated
`herein by reference.
`
`10
`
`FIELD OF THE INVENTION
`
`15
`
`Aspects of the invention relate to methods and apparatus
`for generating and using enhanced metadata in search-driven
`applications.
`
`BACKGROUND OF THE INVENTION
`
`As the World Wide Web has emerged as a major research
`tool across all fields of study, the concept of metadata has
`become a crucial topic. Metadata, which can be broadly
`defined as “data about data,” refers to the searchable defi
`nitions used to locate information. This issue is particularly
`relevant to searches on the Web, where metatags may
`determine the ease with which a particular Web site is
`located by searchers. Metadata that are embedded with
`content is called embedded metadata. A data repository
`typically stores the metadata detached from the data.
`Results obtained from search engine queries are limited to
`metadata information stored in a data repository, referred to
`as an index. With respect to media files or streams, the
`metadata information that describes the audio content or the
`video content is typically limited to information provided by
`the content publisher. For example, the metadata informa
`tion associated with audio/video podcasts generally consists
`of a URL link to the podcast, title, and a brief summary of
`40
`its content. If this limited information fails to satisfy a search
`query, the search engine is not likely to provide the corre
`sponding audio/video podcast as a search result even if the
`actual content of the audio/video podcast satisfies the query.
`
`25
`
`30
`
`35
`
`45
`
`SUMMARY OF THE INVENTION
`
`According to one aspect, the invention features an auto
`mated method and apparatus for generating metadata
`enhanced for audio, video or both (“audio/video') search
`driven applications. The apparatus includes a media indexer
`that obtains an media file or stream (“media file/stream'),
`applies one or more automated media processing techniques
`to the media filefstream, combines the results of the media
`processing into metadata enhanced for audio/video search,
`and stores the enhanced metadata in a searchable index or
`other data repository. The media file/stream can be an
`audio/video podcast, for example. By generating or other
`wise obtaining Such enhanced metadata that identifies con
`tent segments and corresponding timing information from
`the underlying media content, a number of for audio/video
`search-driven applications can be implemented as described
`herein. The term “media' as referred to herein includes
`audio, video or both.
`According to another aspect, the invention features a
`computerized method and apparatus for dynamic presenta
`tion of advertising, factual, informational content and com
`
`50
`
`55
`
`60
`
`65
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 0021
`
`

`

`US 9,697,230 B2
`
`3
`playback status with the timing information from the meta
`data and then directs the client to present a set of factual,
`informational, or advertising content associated with the
`content segment in playback.
`In Such embodiments, the method and apparatus can
`obtain a data map that associates a corresponding set of
`factual, informational, or advertising content with each of
`the content segments of the media content identified in the
`metadata; and use the data map to determine the correspond
`ing set of factual, informational, or advertising content
`associated with the content segment in playback.
`In Such embodiments, the method and apparatus can
`direct the client to present the set by transmitting the set of
`factual, informational, or advertising content associated with
`the content segment in playback to the client for presenta
`tion.
`In Such embodiments, the method and apparatus can
`transmit a plurality of factual, informational, or advertising
`content from the server, and, Subsequent to identifying the
`content segment in playback, direct the client to present the
`set of factual, informational or advertising content by pro
`viding the client with a set of instructions or data identifying
`the set from among the plurality of factual, informational, or
`advertising content for presentation.
`In Such embodiments, the content segments can include
`word segments, audio speech segments, video segments and
`marker segments. For example, the content segments of the
`media content can correspond to a word, a phrase, a sen
`tence, a paragraph, a story, or a topic. The content segments
`of the media content can also correspond to a speaker
`identified during the one or more media processing tech
`niques.
`
`BRIEF DESCRIPTIONS OF THE DRAWINGS
`
`4
`FIG. 8A is a diagram illustrating an apparatus for pre
`senting a set of factual, informational, or advertising content
`associated with the content segment in playback.
`FIG. 8B is a flow diagram illustrating a computerized
`method for dynamic presentation of a set of factual, infor
`mational, or advertising content associated with the content
`segment in playback.
`FIG. 8C is a flow diagram illustrating another computer
`ized method for dynamic presentation of a set of factual,
`informational, or advertising content associated with the
`content segment in playback.
`FIG. 8D is a flow diagram illustrating another computer
`ized method for dynamic presentation of a set of factual,
`informational, or advertising content associated with the
`content segment in playback.
`FIG. 9A is a diagram illustrating an apparatus for pre
`senting a set of factual, informational, or advertising content
`associated with the content segment in playback.
`FIG. 9B is a flow diagram illustrating a computerized
`method for dynamic presentation of a set of factual, infor
`mational, or advertising content associated with the content
`segment in playback.
`FIG. 9C is a flow diagram illustrating another computer
`ized method for dynamic presentation of a set of factual,
`informational, or advertising content associated with the
`content segment in playback.
`
`DETAILED DESCRIPTION
`
`Generation of Enhanced Metadata for Audio/Video
`The invention features an automated method and appa
`ratus for generating metadata enhanced for audio/video
`search-driven applications. The apparatus includes a media
`indexer that obtains an media file/stream (e.g., audio/video
`podcasts), applies one or more automated media processing
`techniques to the media file/stream, combines the results of
`the media processing into metadata enhanced for audio/
`Video search, and stores the enhanced metadata in a search
`able index or other data repository.
`FIG. 1A is a diagram illustrating an apparatus and method
`for generating metadata enhanced for audio/video search
`driven applications. As shown, the media indexer 10 coop
`erates with a descriptor indexer 50 to generate the enhanced
`metadata 30. A content descriptor 25 is received and pro
`cessed by both the media indexer 10 and the descriptor
`indexer 50. For example, if the content descriptor 25 is a
`Really Simple Syndication (RSS) document, the metadata
`27 corresponding to one or more audio/video podcasts
`includes a title, Summary, and location (e.g., URL link) for
`each podcast. The descriptor indexer 50 extracts the descrip
`tor metadata 27 from the text and embedded metatags of the
`content descriptor 25 and outputs it to a combiner 60. The
`content descriptor 25 can also be a simple web page link to
`a media file. The link can contain information in the text of
`the link that describes the file and can also include attributes
`in the HTML that describe the target media file.
`In parallel, the media indexer 10 reads the metadata 27
`from the content descriptor 25 and downloads the audio/
`video podcast 20 from the identified location. The media
`indexer 10 applies one or more automated media processing
`techniques to the downloaded podcast and outputs the
`combined results to the combiner 60. At the combiner 60, the
`metadata information from the media indexer 10 and the
`descriptor indexer 50 are combined in a predetermined
`format to form the enhanced metadata 30. The enhanced
`metadata 30 is then stored in the index 40 accessible to
`search-driven applications such as those disclosed herein.
`
`10
`
`15
`
`25
`
`30
`
`35
`
`The foregoing and other objects, features and advantages
`of the invention will be apparent from the following more
`particular description of preferred embodiments of the
`invention, as illustrated in the accompanying drawings in
`which like reference characters refer to the same parts
`throughout the different views. The drawings are not nec
`essarily to Scale, emphasis instead being placed upon illus
`trating the principles of the invention.
`FIG. 1A is a diagram illustrating an apparatus and method
`for generating metadata enhanced for audio/video search
`driven applications.
`FIG. 1B is a diagram illustrating an example of a media
`indexer.
`FIG. 2 is a diagram illustrating an example of metadata
`enhanced for audio/video search-driven applications.
`FIG. 3 is a diagram illustrating an example of a search
`Snippet that enables user-directed navigation of underlying
`media content.
`FIGS. 4 and 5 are diagrams illustrating a computerized
`method and apparatus for generating search Snippets that
`enable user navigation of the underlying media content.
`FIG. 6A is a diagram illustrating another example of a
`search Snippet that enables user navigation of the underlying
`media content.
`FIGS. 6B and 6C are diagrams illustrating a method for
`navigating media content using the search Snippet of FIG.
`6A.
`FIGS. 7A-7C are diagrams that illustrate dynamic pre
`sentation of advertising content according to the playback of
`corresponding segments.
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Google Exhibit 1016 - Google v. CSI
`IPR2025-00877 - Page 0022
`
`

`

`US 9,697,230 B2
`
`5
`In other embodiments, the descriptor indexer 50 is
`optional and the enhanced metadata is generated by the
`media indexer 10.
`FIG. 1B is a diagram illustrating an example of a media
`indexer. As shown, the media indexer 10 includes a bank of
`media processors 100 that are managed by a media indexing
`controller 110. The media indexing controller 110 and each
`of the media processors 100 can be implemented, for
`example, using a Suitably programmed or dedicated proces
`Sor (e.g., a microprocessor or microcontroller), hardwired
`logic, Application Specific Integrated Circuit (ASIC), and a
`Programmable Logic Device (PLD) (e.g., Field Program
`mable Gate Array (FPGA)).
`A content descriptor 25 is fed into the media indexing
`controller 110, which allocates one or more appropriate
`media processors 100a .
`.
`. 100n to process the media
`files/streams 20 identified in the metadata 27. Each of the
`assigned media processors 100 obtains the media file/stream
`(e.g., audio/video podcast) and app