I IIIII IIIIIIII Ill lllll lllll lllll lllll lllll lllll lllll lllll 111111111111111111
`US006985932Bl
`
`(12) United States Patent
`Glaser et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 6,985,932 Bl
`Jan.10,2006
`
`(54) MULTIMEDIA COMMUNICATIONS SYSTEM
`AND METHOD FOR PROVIDING AUDIO ON
`DEMAND TO SUBSCRIBERS
`
`(75)
`
`Inventors: Robert D. Glaser, Seattle, WA (US);
`Mark O'Brien, Cambridge, MA (US);
`Thomas B. Boutell, Seattle, WA (US);
`Randy Glen Goldberg, Princeton, NJ
`(US)
`
`4,845,756 A
`
`7/1989 Seen et al. .................... 381/77
`
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`
`309298
`
`3/1989
`
`(Continued)
`
`OTHER PUBLICATIONS
`
`(73) Assignee: RealNetworks, Inc., Seattle, WA (US)
`
`NCSA Mosaic Version History, Jun. 1993.*
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by O days.
`
`(21) Appl. No.: 09/237,099
`
`(22) Filed:
`
`Jan.25, 1999
`
`Related U.S. Application Data
`
`(63) Continuation of application No. 08/347,582, filed on
`Nov. 30, 1994, now Pat. No. 5,793,980.
`
`(51)
`
`Int. Cl.
`G06F 15/16
`(2006.01)
`(52) U.S. Cl. ....................... 709/219; 709/231; 725/142
`(58) Field of Classification Search ................ 709/231,
`709/219, 217,247,232,227,213,216; 725/41,
`725/92,46, 135, 1, 142,2,105,91,87,89,
`725/94; 711/118
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`3,882,538 A
`3,990,710 A
`4,253,157 A
`4,504,705 A
`4,506,387 A
`4,581,484 A
`4,611,277 A
`4,658,093 A
`4,827,256 A
`
`5/1975 Lowe .......................... 386/92
`11/1976 Hughes
`. ... ... 707 /104.1
`2/1981 Kirschner et al.
`3/1985 Pilloud . ... ... .. ... ... ... ... .. . 381/77
`3/1985 Walter ........................ 359/118
`4/1986 Bendig .................... 379/88.13
`9/1986 Kemppainen et al.
`........ 703/27
`4/1987 Hellman ...................... 705/52
`5/1989 Yokoyama .................. 370/532
`
`(Continued)
`
`Primary Examiner-William C. Vaughn, Jr.
`(74)Attorney,Agent, or Firm-Steven C. Stewart; Schwabe,
`Williamson & Wyatt, P.C.
`
`(57)
`
`ABSTRACT
`
`An audio-on-demand communication system provides real(cid:173)
`time playback of audio data transferred via telephone lines
`or other communication links. One or more audio servers
`include memory banks which store compressed audio data.
`At the request of a user at a subscriber PC, an audio server
`transmits the compressed audio data over the communica(cid:173)
`tion link to the subscriber PC. The subscriber PC receives
`and decompresses the transmitted audio data in less than
`real-time using only the processing power of the CPU within
`the subscriber PC. According to one aspect of the present
`invention, high quality audio data compressed according to
`lossless compression techniques is transmitted together with
`normal quality audio data. According to another aspect of
`the present invention, metadata, or extra data, such as text,
`captions, still images, etc., is transmitted with audio data and
`is simultaneously displayed with corresponding audio data.
`The audio-on-demand system also provides a table of con(cid:173)
`tents indicating significant divisions in the audio clip to be
`played and allows the user immediate access to audio data
`at the listed divisions. According to a further aspect of the
`present invention, servers and subscriber PCs are dynami(cid:173)
`cally allocated based upon geographic location to provide
`the highest possible quality in the communication link.
`
`40 Claims, 17 Drawing Sheets
`
`AUDIO CONTROL CENTER
`
`SUBSCRIBER PC
`AUDIO BUFFERS
`
`,..-,ooo
`
`, , ,~
`
`i'\
`
`tttJ
`
`1050,--- -,
`
`RECEIVER
`
`J{J{J
`
`'
`'
`'
`'
`'
`'
`'---- J
`
`j
`
`SWITCH
`CONTROLLER
`1
`
`\1060
`
`3
`
`IOJO
`
`4
`
`5
`
`6
`
`-........,010
`
`/070-....__,
`
`METADATA
`
`METADATA BUFFERS
`
`11 !JO\
`
`I S\\1TCH CONTROLLER H
`
`:
`
`: ~\
`
`120
`
`\
`
`Netflix 1017 - Page 1
`
`

`
`US 6,985,932 Bl
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`4,899,299 A
`4,905,094 A
`4,920,432 A
`4,924,303 A
`4,941,123 A
`4,949,187 A
`4,963,995 A
`4,975,691 A
`4,987,529 A
`4,999,806 A
`5,001,580 A
`5,041,921 A
`5,051,822 A
`5,057,932 A
`5,109,482 A
`5,132,992 A *
`5,164,839 A
`5,191,573 A
`5,195,092 A *
`5,237,322 A
`5,247,347 A
`5,253,341 A
`5,262,875 A
`5,282,028 A
`5,283,819 A
`5,289,545 A
`5,297,249 A
`5,434,852 A *
`5,440,334 A
`5,497,502 A
`5,508,731 A *
`5,539,449 A *
`5,542,087 A *
`5,561,456 A *
`5,583,994 A *
`5,612,742 A *
`5,625,404 A *
`
`2/1990 MacPhail .................... 707/204
`2/1990 Pocock et al. .............. 386/106
`4/1990 Eggers et al.
`5/1990 Brandon et al.
`7/1990 Thompson .................. 709/217
`8/1990 Cohen
`10/1990 Lang . ... ... ... .. ... ... ... ... .. . 386/54
`12/1990 Lee ............................. 345/79
`1/1991 Craft et al.
`................. 710/113
`3/1991 Chernow et al.
`........... 717/177
`3/1991 Aranovsky et al. ...... 360/77.12
`8/1991 Scheffler
`9/1991 Rhoades ...................... 463/25
`10/1991 Lang .......................... 386/101
`4/1992 Bohrman .................... 345/723
`7/1992 Yurt et al.
`.................. 375/240
`11/1992 Lang . ... ... ... .. ... ... ... ... .. . 386/54
`3/1993 Hair
`... .. ... ... 340/825.5
`3/1993 Wilson et al.
`8/1993 Heberle ................. 340/870.13
`9/1993 Litteral et al. .............. 725/114
`10/1993 Rozmanith et al. ......... 709/219
`11/1993 Mincer et al.
`.............. 386/101
`1/1994 Johnson et al. ............. 725/139
`2/1994 Glick et al. .............. 379/93.01
`2/1994 Jestice ........................ 361/718
`3/1994 Berstein et al. ............... 381/77
`7/1995 La Porta et al. ............ 370/385
`8/1995 Walters et al. ................ 725/92
`3/1996 Castille
`4/1996 Kohorn ...................... 725/116
`7/1996 Blahut et al. ............... 370/396
`7/1996 Neimat et al. ................ 707/10
`10/1996 Yu .............................. 725/97
`12/1996 Rangan ...................... 709/219
`3/1997 Krause et al.
`......... 375/240.25
`4/1997 Grady et al. ................ 725/114
`
`5,629,732 A * 5/1997 Moskowitz et al.
`........ 725/102
`5,631,903 A * 5/1997 Dianda et al.
`.............. 370/401
`5,642,151 A * 6/1997 Nusbickel et al. .......... 725/105
`5,682,325 A * 10/1997 Lightfoot et al. ........... 340/825
`5,719,786 A * 2/1998 Nelson et al.
`.............. 345/716
`5,734,719 A * 3/1998 Tsevdos et al. . . . . . . . . . . . . . 700/234
`5,754,784 A * 5/1998 Garland et al. ............. 345/835
`5,758,085 A * 5/1998 Kouoheris et al. .......... 709/231
`5,790,176 A * 8/1998 Craig ......................... 725/106
`5,793,980 A * 8/1998 Glaser et al.
`............... 709/231
`5,822,537 A * 10/1998 Katseff et al.
`.............. 709/231
`5,835,667 A * 11/1998 Wactlar et al. . . . . . . . . . . . . . . . . 386/96
`5,848,234 A * 12/1998 Chernick et al.
`........... 709/202
`5,870,552 A * 2/1999 Dozier et al. ............... 709/219
`5,877,755 A
`3/1999 Hellhake
`5,951,646 A * 9/1999 Brandon ..................... 709/231
`5,978,567 A * 11/1999 Rebane et al. .............. 709/219
`6,151,634 A * 11/2000 Glaser et al.
`............... 709/236
`6,604,144 Bl*
`8/2003 Anders ....................... 709/231
`6,622,171 B2 * 9/2003 Gupta et al. ................ 709/231
`
`FOREIGN PATENT DOCUMENTS
`
`WO
`WO
`
`WO 93/16557
`WO 94/14273
`
`8/1993
`6/1994
`
`OTHER PUBLICATIONS
`
`Mosaic Quick Tour for Windows, Copyright 1994.*
`Stephen J. Bigelow, Modem Communications Standards,
`Electronics Now. pp. 35-42, Sep. 1994.
`PCT Written Opinion Mailed Nov. 14, 1996 International
`Application No. PCT/US95/14765.
`C. Topolcic; "experimental Internet Stream Protocol, Ver(cid:173)
`sion 2 (ST-II)"; Internet Working Group Request for Com(cid:173)
`ments RFC-1190 Oct. 1990.
`
`* cited by examiner
`
`Netflix 1017 - Page 2
`
`

`
`U.S. Patent
`
`Jan.10,2006
`
`Sheet 1 of 17
`
`US 6,985,932 Bl
`
`2 w
`0
`0
`2
`
`Netflix 1017 - Page 3
`
`

`
`U.S. Patent
`
`Jan.10,2006
`
`Sheet 2 of 17
`
`US 6,985,932 Bl
`
`,----------------L,
`
`210
`
`215
`
`120
`
`I
`I
`
`~~-~
`
`LIVE
`SOURCE
`
`RECORDED
`SOURCE
`
`A/D CONVERTOR
`
`DIGITAL
`SOURCE
`
`COMPRESSOR
`121
`~ _________ 230 _ _ _ _
`1
`I
`I
`
`DISK STORAGE
`
`I
`ARCHIVAL
`STORAGE
`.___...,... __ __, I I
`
`I
`
`I
`I
`I
`L_ -
`
`PRIMARY
`SERVER
`-
`-
`-
`
`-
`
`-
`
`L _
`
`235
`
`240
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`I 1
`I I
`I I
`- J _J
`
`NET TRANSPORT
`
`265
`
`TEMPORARY
`STORAGE
`
`PROXIMATE
`SERVER
`
`SUBSCRIBER
`P.C.
`
`260
`
`270
`
`NET
`TRANSPORT
`
`110
`
`110
`
`SUBSCRIBER
`P.C.
`
`NET
`TRANSPORT
`
`SUBSCRIBER
`P.C.
`
`270
`
`110
`
`250
`
`265
`
`F/C, 2A
`
`Netflix 1017 - Page 4
`
`

`
`272-- I
`
`FLOW
`CONTROL
`
`274J
`
`ERROR
`CORRECTING
`MODEM
`
`282
`
`2 r 284
`
`I
`
`I
`
`292
`
`TCP /IP
`PROTOCOL
`
`•
`
`TCP/IP
`PROTOCOL
`
`I"'-
`
`I~
`
`MODEM
`
`276
`
`278
`
`280
`
`TELEPHONE
`LINES
`
`286
`
`TELEPHONE
`LINES
`
`ERROR
`CORRECTING
`MODEM
`
`FLOW
`CONTROL
`
`288
`
`290
`
`MODEM
`
`TCP /IP
`PROTOCOL
`
`FIG, 28
`
`FIG, 2C
`
`294
`
`I\
`
`HIGH SPEED
`NETWORK
`(T1 OR
`OTHER FAST
`TRANSPORT)
`
`296
`
`TCP/IP
`PROTOCOL
`
`""
`
`•
`FIG, 20
`
`d •
`r:JJ.
`•
`~
`~ .....
`~ = .....
`
`~
`~
`?
`"'"" ~=
`
`N
`C
`C
`O'I
`
`'JJ. =(cid:173)~
`~ ....
`0 ....,
`"'""
`-..J
`
`~
`
`e
`rJ'J.
`-..a-..
`'° ~
`-..(Ji
`~
`N
`~
`i,-
`
`Netflix 1017 - Page 5
`
`

`
`U.S. Patent
`
`Jan. 10,2006
`
`Sheet 4 of 17
`
`US 6,985,932 Bl
`
`110
`
`SUBSCRIBER PC
`BUFFERS
`
`120
`
`300
`
`312
`
`AUDIO
`CONTROL -------,---1r--i
`CENTER
`
`RECEIVER
`
`DECODER
`
`320 335
`
`130
`
`338
`
`WAVE DRIVER
`
`~337
`
`CPU
`
`332
`
`330
`
`FIG 3
`
`Netflix 1017 - Page 6
`
`

`
`U.S. Patent
`
`Jan. 10,2006
`
`Sheet 5 of 17
`
`US 6,985,932 Bl
`
`402
`
`SEND MESSAGE
`TO DESIGNATED - - -C - - {
`ADDRESS
`
`FIG 4A
`
`404
`
`NO
`
`C
`
`SEND REQUEST
`MESAGE TO
`SERVER WITH
`CLIP NAME
`~ - -~ - - - - - - - - - - - - - - ;8
`406
`
`SEND MESSAGE
`TO DESIGNATED
`ADDRESS
`
`YES
`
`NO
`
`410
`
`END
`
`'-------lD
`
`SEND BEGIN
`MESSAGE
`TO SERVER
`
`416
`
`FLUSH BUFFERS AND
`IGNORE MESSAGES
`FROM SERVER UNTIL
`NEW DAT A COMES:
`ATTEND TO ANY
`PENDING MESSAGES
`FROM SUBSCRIBER
`
`SEND A
`SEEK
`MESSAGE
`
`YES
`
`~414
`
`Netflix 1017 - Page 7
`
`

`
`U.S. Patent
`
`Jan. 10,2006
`
`Sheet 6 of 17
`
`US 6,985,932 Bl
`
`421
`
`422
`
`SEND A
`PAUSE MESSAGE
`TO THE
`SERVER
`
`FILL
`BUFFERS
`
`D
`
`C
`
`425
`
`SEND A STOP
`MESSAGE TO THE
`SERVER
`
`427
`
`SEND AN
`END MESSAGE
`TO THE SERVER
`
`SEND MESSAGE
`TO DESIGNATED
`ADDRESS
`
`439
`
`440
`
`DECOMPRESS
`A
`BUFFER
`
`LOAD
`WAVE
`DRIVER
`
`YES
`
`¥50
`
`SET 1J,,1TIAL
`RAMP-UP I--~~~~~~~~
`BIT FALSE
`~ - -
`
`DROSP':_TOUT ~~
`FLAG
`
`48
`
`Netflix 1017 - Page 8
`
`

`
`U.S. Patent
`
`Jan. 10,2006
`
`Sheet 7 of 17
`
`US 6,985,932 Bl
`
`500
`
`BEGIN
`
`PLAY PART
`OF BLOCK
`
`YES
`
`520
`
`NO
`
`NO
`
`YES
`
`540
`
`DEQUEUE A
`BLOCK SO
`THAT IT GETS
`PLAYED
`
`FIG, 5
`
`Netflix 1017 - Page 9
`
`

`
`U.S. Patent
`
`Jan. 10,2006
`
`Sheet 8 of 17
`
`US 6,985,932 Bl
`
`600
`
`BEGIN
`.------~-------;D
`
`NO
`
`CALL UP
`REQUESTED
`CLIP
`
`SEND
`INITIALIZATION
`DATA
`
`NO
`
`YES
`
`FIG
`
`6A
`
`640
`
`SEND SEEK
`ACKNOWLDGE
`MESSAGE
`
`645
`
`GO TO ADDRESS
`REQUESTED BY
`SEEK MESSAGE
`
`C
`
`Netflix 1017 - Page 10
`
`

`
`U.S. Patent
`
`Jan.10,2006
`
`Sheet 9 of 17
`
`US 6,985,932 Bl
`
`675
`
`END
`
`FIG, 68
`
`TRANSMIT AUDIO
`DATA TO SUBSCRIBER
`
`INCREMENT
`
`i
`
`650
`
`678
`
`NO
`
`685
`
`690
`
`SEND TIME
`MESSAGE TO
`SUBSCRIBER
`
`RESET
`
`i
`
`Netflix 1017 - Page 11
`
`

`
`U.S. Patent
`
`Jan.10,2006
`
`Sheet 10 of 17
`
`US 6,985,932 Bl
`
`700
`
`/ 44 4
`
`710
`
`LOAD DATA
`INTO BUFFER
`
`720
`
`INDICATE
`TIME ON i - - - - - (cid:173)
`DISPLAY
`
`NO
`
`NO
`
`745
`
`YES
`
`INDICATE LENGTH
`OR TABLE OF
`CONTENTS ON
`VISUAL DISPLAY
`
`YES
`
`FLUSH
`BUFFERS
`
`YES
`
`735
`
`755
`
`END
`
`YES
`
`NO
`
`,,-----'---.(,,,,---712
`RETURN
`
`FIG, 7
`
`Netflix 1017 - Page 12
`
`

`
`8ro
`
`-
`-
`@JD URL: I file:
`
`file
`
`[RealAudio]
`Spyglass Mosaic -
`~ovigote
`~dit
`Windows
`.!:f.elp
`
`///CI/PN/OEMO.HTM
`
`I
`
`Rondr extols the virtues of PN's oudio-on-demond technologr
`Commander Will Riker is sarcastic yet menacing
`Captain Picard odresses the Replicator
`
`-I
`
`Remote Clips
`
`These clips ore being kept on o PNA server on halcyon.com
`accessed vio the Internet. Both the WinSock ond the server on
`be started before these clips con be retrieved.
`
`Dove Letterman and Dee-Dee exchange ~ks
`Notions of the World
`
`Remote Clips
`
`Mariners vs. Rorols
`
`This clip begins in the top of the 6th inning. A text display window
`is included, showing the game's progress.
`
`• I
`
`IT • -I
`...
`... ~ i['oyi i(ouse l I (top l 1 fforword
`860 810 820
`/
`1•11 I
`1•1
`/
`850.J
`Length (sec):~ Position (sec): I 111.0
`I
`
`Sound Playback
`
`830
`
`•
`
`~
`
`880 .-/ Display
`
`batter:
`bolls:
`strikes:
`outs:
`
`Tino Mortinez
`0
`0
`
`1
`
`Mariners
`
`0 8 1
`
`1 0
`
`Royals
`
`2 1 0
`
`0 1
`
`inning:
`
`top 6th
`R H E
`------ 2
`4 8
`------ 4
`8 1
`
`I\
`< >
`
`y
`N
`
`~ •
`I•
`
`I Dismiss Display l
`
`FIG BA
`
`/
`
`I
`,-840
`
`87{,
`
`890
`
`'5
`(8S
`
`d •
`r:JJ.
`•
`~
`~ .....
`~ = .....
`
`~
`~
`?
`"'"" ~=
`
`N
`C
`C
`O'I
`
`'JJ. =(cid:173)~
`~ ....
`"'""
`"'"" 0 ....,
`"'""
`-..J
`
`e
`rJ'J.
`-..a-..
`'° ~
`~
`N
`~
`i,-
`
`(It
`
`Netflix 1017 - Page 13
`
`

`
`U.S. Patent
`
`Jan. 10,2006
`
`Sheet 12 of 17
`
`US 6,985,932 Bl
`
`-
`-
`
`File
`
`Custom - C: \NETMANAG\ TCPIP.CFG
`Interface
`Services
`Qisconnect
`~etup
`tlelp
`'
`'-----a !5
`PPPO - COM1, 57600 baud
`
`Interface:
`
`...
`
`...
`
`A
`~
`
`Dial:
`
`3826245
`
`IP Address:
`
`198. 137.231.20
`
`Subnet Mask:
`
`255. 255. 255. 0
`
`Host Name:
`
`boutell
`
`Domain Name:
`
`TYPE
`od····.
`
`IP
`
`;;.;:::
`
`NAME
`ffppp{)tF ? .....
`
`-
`
`-I
`I Stop
`
`DOMAIN
`................... _ .... , .... , ...... _ ........................ ,:_::_ .. :·-:\:·
`•.. ,.
`::·1., . .,-w::: .. ']:.-4.·-. r-,----r:·"'··:T\:~Il·:t.::_::.·--·
`:'.':'.::':
`...
`Custom - C: \NETMANAG\ TCPIP.CFG
`I I Print I 1:: s~er :ri
`I I Save
`I Clear I I Close I
`t
`
`.........
`
`;c;,.;;.; .. ; .. ;
`
`, .................
`
`, -
`
`1 ...
`
`~
`
`f - - -
`
`--•
`
`I ..
`
`login: boutell
`boutell
`Password:********
`
`.. I
`
`I
`
`FIG BB
`
`Netflix 1017 - Page 14
`
`

`
`TRANSMISSION LINE 130
`
`BUFFERS
`315
`--SEND H.Q. DATA~ - - - - - - - - ; . . .__ - - - - - - - -~
`
`[ill [ill· I N I N I N I N I N I N I N I N I N I N I N I N I N I N I N I N I N I N I N I N I N I N I N I N ~
`
`~IME
`
`I
`L_
`
`I H
`
`II ffil. - -1 N-.-!-,N lr-N-.-1 N-,,-N.--1 N--.,-N--1 N--,-!-N r--1 N .... ,-N lr-N-.-1 N-,,-N ... , N--.,-N.--1 N--.,-N--1 N--,-1-N r--1 N .... ,-N lr-N'"T"'"I N-,~
`
`I
`L_
`
`D
`
`-- SEND N DATA
`•
`!L..J --J
`
`[ill [El --J
`
`·225
`
`N
`ITT•
`N
`N
`N
`N
`
`900
`
`MEMORY
`
`H
`H
`H
`H
`H
`H
`
`SERVER
`
`AUDIO CONTROL CENTER
`"240
`
`FIG 9
`
`1q I •IN IN IN IN IN IN IN I •IN IN IN IN IN IN IN IN IN IN IN IN I •IN I-(cid:173)
`-q
`w
`w
`w
`
`I
`:
`•
`!N!N!N!N!N!N!N!N!N!N!N!N!N!N!N!N~
`I
`I
`
`H
`
`• •
`•
`
`!N!N!N!N!N~
`I
`I
`!N!N!N!N!N!N!N~
`I
`I
`!N!N!N!N!N!N!N!N!N~
`I
`I
`
`H
`
`H
`
`H
`
`[ill (EJ --J
`
`[ill [ill --J
`
`DATA PATH-
`
`I
`
`I
`
`I
`
`H
`
`H
`
`H
`
`DE COMPRESSOR
`
`d •
`r:JJ.
`•
`~
`~ .....
`~ = .....
`
`~
`~
`?
`"'"" ~=
`
`N
`C
`C
`O'I
`
`'J'J. =(cid:173)~
`~ ....
`"'"" ~
`0 ....,
`"'""
`-..J
`
`e
`rJ'J.
`-..a-..
`'° ~
`~
`N
`~
`i,-
`
`(It
`
`Netflix 1017 - Page 15
`
`

`
`AUDIO CONTROL CENTER
`AUDIO DATA
`
`1005
`
`SUBSCRIBER PC
`AUDIO BUFFERS
`
`,-1000
`
`1
`
`2
`
`3
`
`4
`
`5
`
`6
`I SWITCH CONTROLLER:
`
`120
`
`-
`,)
`
`1-
`I
`I
`
`I
`
`- -1
`I
`I
`
`-I
`
`130\
`
`\
`
`I(/
`I
`/020 L -)-'
`
`1030
`
`-.........1010
`
`1
`
`2
`
`3
`
`4
`
`5
`
`6
`
`J 15 ---------._
`
`110
`
`1050
`
`,---b-:
`:
`:__
`RECEIVER~ :
`I~ I
`I ____ ~
`JOO
`
`SWITCH
`CONTROLLER
`f
`\
`
`1060
`
`!070~
`
`METADATA
`
`METADATA BUFFERS
`
`Fla 10
`
`d •
`r:JJ.
`•
`~
`~ .....
`~ = .....
`
`~
`~
`?
`"'"" ~=
`
`N
`C
`C
`O'I
`
`'JJ. =(cid:173)~
`~ ....
`"'""
`,i;;..
`0 ....,
`"'""
`-..J
`
`e
`rJ'J.
`-..a-..
`'° ~
`~
`N
`~
`i,-
`
`(It
`
`Netflix 1017 - Page 16
`
`

`
`U.S. Patent
`
`Jan.10,2006
`
`Sheet 15 of 17
`
`US 6,985,932 Bl
`
`120
`
`110
`
`AUDIO
`CONTROL
`CENTER
`
`!JO
`
`315
`
`310
`
`1110
`
`AUDIO
`BUFFER
`
`CPU
`
`HIGH
`QUALITY
`BUFFER
`
`SUBSCRIBER PC
`
`FIG 11
`
`120
`
`/
`
`AUDIO
`CONTROL
`CENTER
`
`(!JO
`
`110
`
`(
`
`AUDIO
`BUFFER
`I
`
`CPU
`
`I
`
`I/ ~ 315
`
`L---
`
`310
`
`v
`
`ADVANCE I/ L.---- !. '210
`BUFFER
`
`SUBSCRIBER PC
`
`FIG 12
`
`Netflix 1017 - Page 17
`
`

`
`U.S. Patent
`
`Jan.10,2006
`
`Sheet 16 of 17
`
`US 6,985,932 Bl
`
`AUDIO SERVER
`
`240
`
`DATA
`DATA
`DATA
`DATA
`DATA
`DATA
`DATA
`DATA
`DATA
`DATA
`DATA
`DATA
`DATA
`
`1330
`
`!JOO
`
`1310
`
`1320
`
`!JOO
`
`1320
`
`DATA
`
`TRANSMIT
`ACKNOWLEDGE
`
`FIG 13
`
`!JO
`
`SUBSCRIBER
`PC
`
`DATA
`DATA
`DATA
`DATA
`
`!JOO
`
`315
`
`Netflix 1017 - Page 18
`
`

`
`U.S. Patent
`
`Jan.10,2006
`
`Sheet 17 of 17
`
`US 6,985,932 Bl
`
`1 BYTE
`1 BYTE
`VARIABLE
`I REQUEST I LENGTH I FILE NAME
`1 BYTE
`\__1400 1 BYTE
`I BEGIN l
`, - - -A - - .
`, - - -A - - .
`I PAUSE J__
`14o4
`1 BYTE
`I EXTRAS NO I
`4 BYTES
`\....1408
`
`1402
`
`1 BYTE
`JXTRAS OK I
`1 BYTE
`
`1406
`
`SEEK
`
`1 BYTE
`
`TIME
`
`TIME DATA
`
`4 BYTES
`
`I
`\_ 1410
`
`TIME DATA
`
`FROM CLIENT
`TO SERVER
`
`F/C, 14
`
`2 BYTES
`1 BYTE
`j.___6_TI_M_E_+__._j __ 6_TI_ME_D_A_TA __ __,f-_ 1425
`1 BYTE
`2 BYTES
`
`---1420
`
`J 6 TIME -
`
`6 TIME DATA
`
`I
`FROM
`SERVER
`VARIABLE
`1 BYTE
`1 BYTE
`TO
`CLIENT , - - -A - - .~ - - J . . .__~ - -~~ -~
`I AUDIO
`I LENGTH I AUDIO DATA ~1435
`1 BYTE
`1 BYTE
`
`1~1430
`
`I SEEK ACKJ ~1445
`1440 4 BYTES
`1 BYTE
`
`LENGTH
`
`1 BYTE
`
`SIZE
`
`1 BYTE
`
`TEXT
`
`TIME DATA
`
`4 BYTES
`
`TIME DATA
`
`4 BYTES "-.___1455
`
`1 BYTE
`I ROWS
`1 BYTE
`
`1 BYTE
`
`I COLUMNS I
`
`VARIABLE
`
`TIME DATA ~ ROWS
`1460
`
`I TEXT DATA I
`
`Netflix 1017 - Page 19
`
`

`
`US 6,985,932 Bl
`
`1
`MULTIMEDIA COMMUNICATIONS SYSTEM
`AND METHOD FOR PROVIDING AUDIO ON
`DEMAND TO SUBSCRIBERS
`
`BACKGROUND OF THE INVENTION
`
`Priority Claim
`
`The present invention is a continuation of Ser. No. 08/347,
`582 U.S. Pat. No. 5,793,980, filed on Nov. 30, 1994.
`
`FIELD OF THE INVENTION
`
`The present invention relates to multimedia computer
`communication systems and, in particular, to communica(cid:173)
`tion systems which provide Audio-On-Demand services.
`
`DESCRIPTION OF THE RELATED ART
`
`In recent years, the computer industry has observed an
`increasing demand for versatility in the personal computer
`market. The average consumer is less interested in high
`computer performance such as increased memory and clock
`rates than in the everyday usefulness of a personal computer
`system. For example, parents may be interested in educa(cid:173)
`tional computer programs for their children which instruct
`using both visual and audio media. As a result, there has
`been an increasing demand for personal computers and
`computer networks which have multimedia capabilities.
`Among the most desirable multimedia capabilities are
`those associated with the transmission of audio information.
`A number of uses have been contemplated for transmission
`of audio information. For example, a user may want access
`to music or news, or may want to have a book read to them
`over their computer. Also, transmission of audio data pro(cid:173)
`vides much needed access to valuable information for visu(cid:173)
`ally impaired persons. Such multimedia communication
`systems which provide subscribers with selectable audio
`information are commonly called audio-on-demand sys(cid:173)
`tems.
`U.S. Pat. No. 5,132,992 issued to Yurt, et al., discloses an
`audio and video transmission and receiving system. The
`audio and video-on-demand system disclosed by Yurt, et al.,
`distributes video and/or audio information to multiple sub(cid:173)
`scriber units from a central source material library. Digital
`signal processing is used to compress data within the source
`material library so that such data can be transmitted over
`standard communication links such as a cable or satellite
`broadcast channel, or a standard telephone line to a receiver
`specified by subscriber service. The receiver subscriber unit
`includes a decompressor for decompressing data sent from
`the source materials library and playing back the decom(cid:173)
`pressed data by means of an audio or visual display.
`Although known audio-on-demand communication sys(cid:173)
`tems offer many significant benefits, such systems are still
`subject to a number of significant limitations. For instance,
`significant difficulties are encountered when attempting to
`provide real time audio playback over narrowband commu(cid:173)
`nication links such as a standard telephone line.
`
`SUMMARY OF THE INVENTION
`
`The present invention provides a real-time, audio-on(cid:173)
`demand system which may be implemented using only the
`processing capabilities of the CPU within a conventional
`personal computer. As detailed above, a number of signifi(cid:173)
`cant difficulties arise when attempting to provide real-time
`
`2
`audio-on-demand. It has been found that these difficulties
`are exacerbated when the subscriber receiving unit is a
`conventional personal computer having an Intel 486 micro(cid:173)
`processor, or processors of equivalent power, as a central
`5 processing unit. Of course, higher power processors could
`be used, but such systems would become prohibitively
`expensive and would not be available to the mainstream
`personal computer user. In order to compensate for lack of
`processing power, special hardware or other additional capa-
`10 bilities would be needed. The system of the present inven(cid:173)
`tion overcomes these difficulties so that real-time audio-on(cid:173)
`demand is available to
`the average consumer on an
`unmodified personal computer.
`In order to overcome the aforementioned difficulties, the
`15 system of the present invention employs an audio compres(cid:173)
`sion algorithm which provides audio compression on the
`order of 22:1. As is well known in the art, audio data in
`digitized format requires large amounts of memory space. It
`has been found that, in order to transmit digitized audio data
`20 so that a high quality audio signal is generated in real time,
`a data rate on the order of 22 kilobytes per second is
`typically necessary. However, current data rates achievable
`by most average cost modems on a reliable basis, fall in the
`range of 1.8 kilobytes (14.4 kilobits) per second. Conse-
`25 quently, the real-time, audio-on-demand system of the
`present invention provides a form of audio compression
`which allows digitized audio data to be transmitted over a
`conventional 14.4 kilobits per second modem connection.
`For purposes of practical implementation, it is preferable to
`30 use less than the maximum possible modem bandwidth
`when transmitting data. It has been found that very good
`performance can be obtained if the data transmission rate is
`about 1 kilobyte per second. Assuming a required data rate
`of 22 kilobytes per second and a transmission bandwidth of
`35 approximately 1 kilobyte per second, an audio compression
`of approximately 22 to 1 is required. Audio compression
`algorithms which may be used in accordance with the
`teachings of the present invention to provide audio com(cid:173)
`pression on the order of 22:1 are well known in the art. The
`40 EINTIA IS-54 standard, which is herein incorporated by
`reference, discloses an algorithm description such that one
`of ordinary skill in the art could implement a compression
`algorithm suitable for use in the present invention. Advan(cid:173)
`tageously, a preferred embodiment of the algorithm employs
`45 an adaptation of the IS-54 VSELP cellular compression
`algorithm compatible with the IS-54 VSELP cellular com(cid:173)
`pression algorithm available from MOTOROLA. Of course,
`it should be understood that in order to facilitate the com(cid:173)
`pression and transmission of digitized audio data, it may be
`50 advantageous to convert the compression algorithm from
`hexadecimal to binary (i.e., from ASCII data format to
`binary data format). Another preferred embodiment of the
`invention utilizes the code excited linear predication (CELP)
`coder, version 3.2, available from NTIS, U.S. Department of
`55 Commerce, 5285 Port Royal Rd., Springfield, Va., 22161
`(telephone number 703-487-4650). Another preferred
`embodiment implements the well known GSM coding algo(cid:173)
`rithm available through the European standards committee.
`Yet another preferred implementation uses a LPC-10 based
`60 coder described in a publication entitled "Digital Processing
`of Speech Signals," by L. R. Rabiner and R. W. Schafer,
`published by Prentice Hall, 1978. The aforementioned pub(cid:173)
`lic documents are herein incorporated by reference.
`Although the required data rates are achievable by means
`65 of the improved audio compression algorithm described
`above, certain difficulties are still inherent in a system which
`provides real time audio-on-demand without specialized
`
`Netflix 1017 - Page 20
`
`

`
`15
`
`3
`software. Further difficulties are encountered in computer
`systems which run high power applications programs such
`as computer systems which run in a MICROSOFT WIN(cid:173)
`DOWS environment. Specifically, it is still necessary to
`decompress and translate the audio data received into a 5
`format compatible with WINDOWS. This poses particular
`problems since a WINDOWS environment
`typically
`requires a great deal of processing power so that much of a
`CPU's time is spent in supporting the WINDOWS software.
`To overcome this difficulty, the system of the present inven- 10
`tion continually monitors requests issued by application
`programs which run concurrently with the audio-on-demand
`system of the present invention. In this manner, requests
`issued by the applications programs are processed rather
`than ignored in the system of the present invention.
`Furthermore, data buffers of reasonable size should be
`allocated within the dynamic random access memory
`(DRAM) of a conventional 486 Intel based personal com(cid:173)
`puter in order to avoid deleterious effects on computer
`performance. Thus, typically, buffer memories are allocated 20
`within the DRAM to have on the order of approximately 16
`or 32 kilobytes of storage. If digitized audio data is trans(cid:173)
`mitted and received within the data buffer at too fast a rate,
`the buffers would overflow causing the loss of significant
`portions of data and audio dropout. As is well known in the
`art, audio dropout is a phenomena wherein audio playback
`terminates for some noticeable time period and then resumes
`after this delay. On the other hand, if data was transmitted
`too slowly, then the buffers would empty out again resulting
`in significant dropout and degradation of audio quality. 30
`Thus, a number of significant difficulties are encountered
`when attempting to implement a real time audio-on-demand
`system within a 486 CPU based personal computer system,
`or other similar personal computer systems. Thus, the
`present invention provides a method of monitoring and
`regulating the flow of data between the server and the
`subscriber unit which insures that the buffers are constantly
`maintained at or near maximum capacity.
`In a further aspect of the invention, audio quality degra- 40
`dation may be compensated for through the data flow
`regulation of the present invention. This flow regulation
`constantly maintains the buffers at or near maximum capac-
`ity so that, in the event of a delay in the communication link,
`the subscriber unit can continue to play back audio already 45
`stored in the buffers until new audio data begins to arrive
`again. Also, the present invention employs a method of
`transmitting high quality audio data compressed using a
`lossless compression algorithm or a compression algorithm
`having a compression ratio which requires transmission at a 50
`rate greater than real time, at selected intervals so that brief
`passages of higher quality audio signals are produced at
`playback. In one embodiment, the user may select when a
`high quality passage is to be sent so that important pieces of
`audio data are played back clearly.
`In another aspect of the invention increased control over
`received audio data is provided for by transmitting selected
`significant portions of an audio clip being transmitted in
`anticipation that the user may desire to move immediately to
`a new position in the audio clip.
`In addition, versatility is added to the audio-on-demand
`system of the present invention by transmission of limited
`extra data, or "metadata," interleaved with the transmitted
`audio data. The metadata may include text, captions, still
`image data, high quality audio data, etc., and includes 65
`information so as to allow the subscriber to synchronize the
`metadata with significant events in the audio data. The
`
`55
`
`US 6,985,932 Bl
`
`4
`metadata is correlated with the audio data to provide a
`combined audio and visual experience.
`Furthermore, the present invention advantageously pro(cid:173)
`vides dynamic allocation of server/subscriber pairs to insure
`the best possible quality of communication links between
`the server and the subscriber.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 shows a simplified schematic block diagram of an
`audio-on-demand system constructed in accordance with the
`present invention.
`FIG. 2A is a more detailed schematic block diagram
`showing the main functional elements of the audio-on(cid:173)
`demand system of the present invention.
`FIGS. 2B-2D are schematic block diagrams showing the
`main functional elements of alternate embodiments of the
`net transports depicted in FIG. 2A.
`FIG. 3 is a schematic block diagram showing the main
`functional elements of a receiving subscriber audio unit such
`as a subscriber personal computer.
`FIGS. 4A and 4B together depict a control flow diagram
`showing the general method employed by the audio-on-
`25 demand system of the present invention to provide real time
`audio decoding within the CPU of the receiver subscriber
`audio unit.
`FIG. 5 is a subcontrol flow diagram showing the general
`operation of the wave driver of FIG. 3.
`FIGS. 6A and 6B together depict the general flow of
`control employed within the audio server of the present
`invention.
`FIG. 7 depicts a control flow diagram which details the
`35 method employed within the read data subroutine block of
`FIG. 4B.
`FIG. SA depicts the various displays observed on the
`video screen of the subscriber personal computer as the user
`selects an audio clip to be played from a menu, and selects
`various options while the audio clip is being played.
`FIG. SB depicts the various displays observed on the
`video screen of the subscriber personal computer as the user
`dials the server, logs into the server system, and initiates a
`disconnect.
`FIG. 9 is a schematic representation of an exemplary data
`transaction between a server and a subscriber unit which
`illustrates method used in the high quality transmission
`mode of the present invention.
`FIG. 10 is a simplified block diagram which depicts the
`main functional elements of an audio-on-demand system
`that provides real-time playback of audio data in addition to
`metadata which can be displayed in synchronism with
`corresponding audio data.
`FIG. 11 is a simplified block diagram which depicts the
`main functional elements of an audio-on-demand system
`that provides audio playback of selected portions of high
`quality audio data in real-time.
`FIG. 12 is a simplified block diagram which depicts the
`60 main functional elements of an audio-on-demand system
`that provides a table of contents indicating significant divi(cid:173)
`sions within a requested audio clip, and which provides for
`immediate playback of audio data at the divisions specified
`in the table of contents.
`FIG. 13 is a schematic representation of the method used
`in accordance with the present invention to manage the flow
`of data blocks from the server to the subscriber PC.
`
`Netflix 1017 - Page 21
`
`

`
`US 6,985,932 Bl
`
`5
`FIG. 14 illustrates the data structures of various data
`messages transmitted between the server and the subscriber
`PC in accordance with the teachings of the present inven(cid:173)
`tion.
`
`DETAILED DESCRIPTION OF IBE
`PREFERRED EMBODIMENT
`
`FIG. 1 shows a simplified schematic block diagram of an
`"audio-on-demand" system constructed in accordance with
`the present invention. The system 100 comprises a sub(cid:173)
`scriber personal computer (PC) 110 ( e.g., an IBM PC having
`a 486 Intel Microprocessor), having a video display 115. The
`subscriber PC 110 connects to an audio control center 120
`over telephone lines 130 via a modem 140.
`In operation, a user calls the audio control center 120 by
`means of the modem 140. The audio control center 120
`transmits a menu of possible selections over the telephone
`lines 130 to the personal computer 110 for display on the
`video display 115. The user may then select one of the
`available options displayed on the video display 115 of the
`computer 110. For example, the user may opt to listen to a
`song or hear a book read. Once the audio data has been
`transmitted, the modem 140 disconnects from the audio
`control center 120.
`FIGS. 2A-2D and FIG. 3 are schematic block diagrams
`which show, in greater detail, the main functional elements
`of the aud