(12)
`
`United States Patent
`Fuller et al.
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 6,711,622 B1
`Mar. 23, 2004
`
`US006711622B1
`
`(54) VIDEO AND AUDIO STREAMING FOR
`MULTIPLE USERS
`
`5,889,942 A * 3/1999 Orenshteyn .......... .. 395/187.01
`5,905,524 A * 5/1999 Sauer ........................ .. 348/15
`5,905,872 A
`5/1999 DeSimone et a1.
`
`
`
`
`
`(75) Inventors: Michael J. Fuller, Los GatoS, CA _ John J‘ Graham’ Saratoga’ CA
`
`
`
`, , 2 Z: 5,953,005 A * 9/1999 Liu .......................... .. 345/302 ~~~~~~~~~~~~~~~~~ ~~ glbatyant 0e Z e a . .
`
`
`
`
`
`
`
`
`
`
`
`
`
`_
`_
`.
`(73) Asslgnee- Broadware Technologles’ Inc-1
`Cupemno, CA(US)
`_
`_
`_
`_
`subJectto any dlsclalrnentheterm ofthls
`patent is extended or adjusted under 35
`U.S.C. 154(1)) byO days.
`
`_
`(*) Nome:
`
`_
`(21) APP1"N°"09/001’850
`
`(22) Filed:
`
`Dec. 31, 1997
`
`(51) Int. Cl.7 .............................................. .. G06F 15/16
`(52) U..S. Cl. ........................ .. 709/231; 707/505; 379/88
`(58) Fleld 0f Search ............................... .. 709/302, 204,
`709/227’ 218’ 231’ 707/10_4’ 10’ 39§/187'01f
`345/302’ 329’ 158’ 327’ 705/517’ 348/15’
`379/88
`
`(56)
`
`References Cited
`
`U'S' PATENT DOCUMENTS
`5,228,859 A * 7/1993 Rowe ....................... .. 434/118
`5,390,240 A * 2/1995 Sensney ................ .. 379/93.14
`5,512,938 A * 4/1996 Ohno ........................ .. 348/15
`
`2722:7222: : 5122; 5’724’355 A 3/1998 Bruno et a1 ;;; ............ 12321522
`
`
`
`5,751,280 A * 5/1998 Abbott et al. .......... .. 707/500.1
`5,764,916 A * 6/1998 Busey et a1_ ______________ __ 709/227
`5,778,181 A * 7/1998 Hidary et a1.
`709/218
`5,802,526 A * 9/1998 Fawcett et a1.
`707/104
`709/204
`5,828,839 A * 10/1998 Moncreiff ------- -
`578427015 A : 11/1998 Cunm? ct a1~ -
`709/104
`5,844,553 A 12/1998 Hao et a1‘ """" "
`345/329
`5,861,881 A * 1/1999 Freeman et a1. .......... .. 345/302
`5,880,792 A
`3/1999 Ward et a1.
`5,884,032 A * 3/1999 Bateman et a1. .......... .. 709/204
`5,889,506 A * 3/1999 Lopresti et a1. ........... .. 345/158
`
`5,960,173 A * 9/1999 Tang et a1. ............... .. 709/201
`6,006,241 A * 12/1999 Purnaveja et a1. ..
`707/512
`6,009,410 A * 12/1999 LeMole e161. ............. .. 705/14
`6,020,931 A
`2/2000 Bilbrey e161.
`345/327
`6,025,837 A
`2/2000 Matthews, III 6161.
`6,055,522 A * 4/2000 Krishna 6161. ........... .. 705/517
`6,091,407 A
`7/2000 Boetje et a1.
`6,151,609 A * 11/2000 Truong ..................... .. 707/505
`6,222,843 B1
`4/2001 Mauger
`379/88
`6,298,120 B1 * 10/2001 Civanlar et a1.
`6,452,933 B1 * 9/2002 Duf?eld et a1. ........... .. 370/415
`
`OTHER PUBLICATIONS
`
`.
`.
`and
`Performance
`Architectures:
`Server
`Video
`Scalability _'_Bemhardt, 6ta1‘(1994)ftp'eureCOmfr/ATM/
`papeI-SEURECOM/PapeI-S/breStBeI-nhardLpSgZ~*
`Realtime Audio and Video, Dec. 8, 1996; http:ei.cs.vt.edu/~
`WWWbtb/book/chap 11/introduction.*
`
`(List continued on neXt page.)
`
`Primary Examiner—Marc D. Thompson
`Assistant Examiner—Thong Vu
`(74) Attorney) Agent) or Firm—C001@y GOdWaId LLP
`
`(57)
`
`ABSTRACT
`
`A. video data to multiple users is described. In one of
`
`
`embodiment, the System Comprises a ?rst Client, a Second
`client and a server. The ?rst and second clients are executing
`broWsers. The server can communicate With the tWo clients.
`The server concurrently provides streaming audio and video
`data to both of the clients. Importantly, the server does not
`require the tWo broWsers to use a plug-in or a helper
`application to receive and use the streaming audio and video
`data
`'
`
`4 Claims, 5 Drawing Sheets
`
`[130
`/I80
`/— I00
`<— ClientSideq- Communications—h Server Side ————>
`
`V13‘;
`
`-
`
`gria??inggo
`
`V
`
`138 _‘ [1178 m gm
`
`S amin AudioandVideoDa
`
`‘3
`
`/
`
`Display
`
`'76
`
`‘
`
`'
`
`Browser
`
`1
`
`f1“ <~>\
`
`\ \
`
`chem
`j
`'12
`
`102
`
`V
`
`'I
`
`Internet
`m
`
`178 \‘ smaqlmgmdmm
`
`Video Data
`
`/
`If’,
`
`5
`
`Real-TimeServer HQ
`
`Video
`
`I Audio
`
`Videol’roxy Audiol’roxy
`
`5
`/I-04
`
`3:251,
`
`Browser
`Client
`
`144
`
`146
`r1
`
`148
`udio
`
`t
`
`149
`
`T
`.
`
`Limb WJ ‘5175 _
`Mike
`159-
`VldBU Input L '64 \ I66
`5431;121:1111
`
`111
`
`108
`
`l
`
`7158
`
`Netflix 1016 - Page 1
`
`

`
`US 6,711,622 B1
`Page 2
`
`OTHER PUBLICATIONS
`
`Instructions for using Netscape Navigator, downloaded from
`Netscape
`Website/../NetHelp/netscape/navigatr/nav.htrnl,
`printed on Dec. 14, 1997, 67 pages.
`Paek et al. “Video Server Retrieval Scheduling for Variable
`Bit Rate Scalable Video”, IEEE 1996, pp. 108—112.
`
`Huang et al., “The Mercuri Multirnedia Laboratory at Hon
`eyWell”, IEEE 1996, pp. 72—77.
`Liu et al., Performance of a Storage System for Supporting
`Different Video Types and Qualities, IEEE 1996, pp.
`1314—1331.
`
`* cited by eXarniner
`
`Netflix 1016 - Page 2
`
`

`
`U.S. Patent
`
`Mar. 23, 2004
`
`Sheet 1 0f5
`
`US 6,711,622 B1
`
`92
`.
`
`
`Q:
`
`v.ouou-1
`
`5582>Eao%=<maeafimE~:\o§_<mq..a§_m
`
`a00asun82>Ea“n.>
`
`_—
`
`
`
`
`
`
`
`AIl|I||l|IuEmShowTlmaoufiasafioo|VmoEmEmmaIIV
`
`a2\\%~\\2:|\
`
`
`
`HD>.HDmQDRV
`
`
`
`.CoEu2vuhfiw
`
`SmQ83>3Ba29%magnum.3~I\
`
`n2/
`
`uuwsoum
`
`I
`_____.___._._
`L._—.
`
`.__.._.._..L._ .L___..____..__.._______.............__.._..___.._.....__ __.._--.
`
`
`ESEoE=<bnoumouE> <wflnaobm
`
`
` .3~QJHotomofifiniomII..QN34
`
`6885N3
`
`N.b~r.w
`
`wasoE:<mfinaobw1/1/39:32>W:1111.1
`©m~E“E59:5
`
`
`3&9ODE>%m.N“.3
`
`nouuuanooEm
`
`StowBofium2
`
`n.._w=<S9:
`
`Emu83>
`
`Q39:9:3:
`
`M5m2
`
`__..___..__.._____.._._____
`
`2:
`
`32>
`
`%m_&m_D
`
`_._3u..3832>
`
`“£38m
`
`:~
`
`Netflix 1016 - Page 3
`
`Netflix 1016 - Page 3
`
`
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Mar. 23, 2004
`
`Sheet 2 0f 5
`
`US 6,711,622 B1
`
`Client Initiates HTTP Request
`
`l
`
`[210
`
`/— 220
`
`Web Server Supplies Decoding Instructions
`
`l
`
`Client Executes Instructions
`
`/— 230
`
`240 —\ +
`Client Makes Audio Request
`
`242—\
`
`i
`
`Web Server Makes Audio Request
`
`Real-Time Server Writes Audio Data
`246 —\
`i
`Web Server Transmits Audio Data
`
`+ /—250
`
`Client Makes Video Request
`
`l /— 252
`
`Web Server Makes Video Request
`
`254
`/—
`I
`Real-Time Server Writes Video Data
`l /— 256
`Web Server Transmits Video Data
`
`Yes
`
`248 —\
`
`‘
`
`I
`
`Yes I
`
`I
`
`+ /— 258
`
`Client Dccompresses Audio Data and
`Plays Audio
`
`Client Decomprcsscs Video Data and
`Plays Video
`
`Continue?
`
`Continue?
`
`Stop Audio
`
`Stop Video
`
`FIG. 2
`
`Netflix 1016 - Page 4
`
`

`
`U.S. Patent
`
`Mar. 23, 2004
`
`Sheet 3 0f5
`
`US 6,711,622 B1
`
`Bou:_>>
`
`.
`
`....kw.
`
`:o_mm_Em:mC..
`
`
`
`
`
`
`
`
`
`:a.:a.=5SE»u.95V..Q,E.Ew_uu&§..w2.._.fi2whoK:
`
`
`
`V35:o_s2om
`
`an DUB DUDDD mm}
`
`DEMo=.$_u>EV.2.5.u>5
`
`
`
`§_8_.:o=u:<
`
`Euzao9:3.28..
`
`£2.80£3.32E32m,_.O
`
`
`
`Sucau$.50.5__&8_oo.m
`
`EoEmE:o._,Sand
`
`8250mmm3ov=_>>
`
`
`
`Eunau._uuuu.._Em
`
`
`
`B3ficozazo=:owm<
`
`u.=.5mucuubwmom3on=_>»
`
`
`
`‘I>.~.0332u>_SE2E
`
`
`
`DMoon30305moEoD82>>52
`
`Netflix 1016 - Page 5
`
`Netflix 1016 - Page 5
`
`
`

`
`U.S. Patent
`
`Mar. 23, 2004
`
`Sheet 4 of 5
`
`US 6,711,622 B1
`
`bmaoum
`
`38:5
`
`§\A
`
`isEuasm
`
`5E26:am
`
`V2.
`
`:52..5C.752
`
`32>Eu
`
`
`
`So»umooau
`
`cozuuacou
`
`
`
`~22..32E:3.5090:30»338:
`
`Jenna!5-Caufl
`
`
`
`.£:i..._..£v.6.3«at22¢..2.£§:£...__.53..
`
`.u..ri<)uhI.1»‘v>.v..HF.I!|x51'l'..
`
`
`
`
`
`.4...“..,___.a..wk...,913T..1.kw,A.u......¢k._...........u.,€q..:,..H.w....4....‘.._....v.£,..J..o....:u?.Im.t5_..fl.u_..NGm.v.Dgram.
`
`
`
`
`
`
`
`
`
`..111.
`
`Netflix 1016 - Page 6
`
`Netflix 1016 - Page 6
`
`

`
`U.S. Patent
`
`Mar. 23, 2004
`
`Sheet 5 of5
`
`US 6,711,622 B1
`
`w .UFN
`
`Netflix 1016 - Page 7
`
`

`
`US 6,711,622 B1
`
`1
`VIDEO AND AUDIO STREAMING FOR
`MULTIPLE USERS
`
`1. RELATED APPLICATIONS
`This application incorporates by reference, and claims
`priority from, the US. provisional patent application
`entitled, “Streaming Audio and Video for Multiple Users,”
`serial No. 60/034,128, having inventors Michael J. Fuller
`and John J. Graham.
`
`2. THE BACKGROUND OF THE INVENTION
`
`a. The Field of the Invention
`This invention relates to the ?eld of netWork delivery of
`audio and video information. In particular, the invention
`relates to a system for delivering audio and video informa
`tion to multiple users.
`b. Background Information
`The Internet enables many different Ways of communi
`cating. Originally, the Internet Was used for the exchange of
`?les and electronic mail. As the capabilities of the Internet
`expand, other types of communications are enabled.
`Audio and video transmissions are an important area of
`the communications that the Internet enables. For example,
`many technologies support the transmission of digital video
`and/or audio signals over the Internet. An example of such
`a technology is QuicktimeTM, available from Apple
`Computer, Inc., of Cupertino Calif. Quicktime movies are
`?les that can be transmitted across the Internet. Quicktime
`provides both audio and video displays. Many other ?le
`formats alloW audio and video to be displayed on people’s
`computers.
`This paragraph describes an example use of a the Quick
`time technology. Auser Will have a broWser application that
`resides on his/her computer. The computer, acting as a client
`under the direction of the broWser application, Will connect
`to various World Wide Web (Web) servers. Each Web server
`Will typically serve hypertext markup language (HTML)
`?les to the clients. The ?les may include text, graphics
`references, and references to specialiZed ?les. Some of these
`specialiZed ?les can include audio and video information in
`Quicktime form. The clients can then play these audio and
`video ?les once they are doWnloaded using a Quicktime
`plug-in, a helper application, or Quicktime capabilities built
`into the broWser application. Aplug-in and a helper appli
`cation are described in greater detail beloW.
`Streaming audio and video, as a subset of all the types of
`audio and video that can be transmitted over the Internet,
`alloW people to broadcast long and/or live video and audio
`transmissions across the Internet. Streaming video and audio
`is video and audio digital data that is transmitted on a
`continuous basis. A client can access the data stream and
`regenerate the video images and audio signal as they are
`being transmitted. Streaming technology is particularly
`helpful Where the events are live, or Where the ?les Would
`be so large as to be a burden on the end users. Examples of
`Where streaming technology is particularly useful are for the
`display of conferences, sporting events, radio broadcasts,
`television broadcasts, and the like.
`RealNetWorks, Inc. of Seattle, Wash., provides a system
`for transmitting streaming audio and video signals to users
`over the Internet. RealNetWorks supplies a server that alloWs
`multiple users to simultaneously receive streaming audio
`and video.
`The real audio system requires that not only the client
`have additional softWare, but that the content provider have
`
`10
`
`15
`
`2
`a separate server from their normal Web server. For a client
`to receive a real audio broadcast, the client typically con
`nects through their broWser to a Web page With a reference
`to a real audio server. The client then accesses its separate
`real audio player program. The real audio player program
`then connects to the referenced real audio server. A signi?
`cant draWback to such an arrangement is that the user must
`doWnload the real audio player program. This program must
`then be installed on the user’s computer. This may cause a
`number of problems for the user. For example, if the user is
`behind a ?reWall, or some security program, the client may
`not be able to receive the broadcast from the server.
`Additionally, the installation of any program may have
`con?icts With other programs. The program has the disad
`vantage of being platform speci?c. This means that a dif
`ferent program must be developed and doWnloaded for each
`type of computer that is to be used to access RealNetWorks
`broadcasts. Additionally, the broadcasters of the streaming
`audio and video need to use the RealNetWorks server, Which
`is separate from the broadcasters’ World Wide Web server
`(also referred to as the Web server). This increases the
`broadcasters’ security problems because noW the broadcast
`ers must be concerned With tWo separate servers.
`Another example of a video and audio system that uses
`Internet like communications is the MBone. The MBone is
`a specialiZed communication netWork that alloWs for the
`distribution of streaming video and audio signals to multiple
`users. A specialiZed netWork is set up speci?cally to transmit
`MBone communications. A signi?cant draWback of this
`system is that users must be connected to the specialiZed
`netWork. Additionally, users Will be required to have spe
`cialiZed softWare on their computers to listen to and Watch
`MBone transmissions.
`A streaming video system, not requiring a user to doWn
`load a separate program, Was developed for a single user by
`John Graham of California. This single user broadcast
`technology alloWed a Web server to serve a single streaming
`video signal to a single client. Although the user did not need
`to doWnload a plug-in to see the video, only one user Was
`alloWed to access the video stream at a time. In this system,
`video information Was captured from a video camera and
`digitiZed. The digital video information Was then encapsu
`lated in a MIME encoded multipart data stream. The client
`received this data stream and reconstructed frames of the
`digital video.
`Therefore, What is desired is a platform independent video
`and audio streaming system that does not require the user to
`doWnload additional programs beyond the functionalities
`found in a broWser.
`
`3. A SUMMARY OF THE INVENTION
`
`55
`
`65
`
`A system and method of providing streaming audio and
`video data to multiple users is described. In one
`embodiment, the system comprises a ?rst client, a second
`client and a server. The ?rst and second clients are executing
`broWsers. The server can communicate With the tWo clients.
`The server concurrently provides streaming audio and video
`data to both of the clients. Importantly, the server does not
`require the tWo broWsers to use a plug-in or a helper
`application to receive and use the streaming audio and video
`data.
`In some embodiments of the invention, a broWser causes
`a client to request an HTML ?le from a Web server. The
`client receives the HTML ?le. The HTML ?le includes an
`HTML tag that directs the broWser to load one or more
`applets from the server. The broWser executes the applets
`
`Netflix 1016 - Page 8
`
`

`
`US 6,711,622 B1
`
`3
`causing the browser to request streaming audio and video
`from the Web server. That request may or may not include
`parameters giving information about the type of request
`being made. The Web server associates a server process With
`the request, given the parameters in the request. The Web
`server noti?es real-time audio and video process that stream
`ing audio and video information is needed. In response to the
`noti?cation, the real-time audio and video process stores
`encoded audio and video data in a shared memory location.
`The server process accesses the shared memory and inserts
`the audio and video data into one or more data streams. The
`client receives the data streams and reconstructs the audio
`and video signals using only the capabilities of the broWser.
`In some embodiments, a separate stream and server process
`is used for each of the audio and video data. These embodi
`ments alloW multiple clients to simultaneously receive the
`same audio and video data.
`Other embodiments of the invention include a Web server
`that can serve streaming audio and video information as Well
`as perform more usual Web server functions (such as,
`serving Web pages, performing ?le transfers, supporting
`secure communications). These embodiments have the
`advantage of alloWing the broadcasters and the users to set
`up their security con?gurations for one Web server, rather
`than tWo servers (a Web server and a streaming audio and
`video server).
`Although many details have been included in the descrip
`tion and the ?gures, the invention is de?ned by the scope of
`the claims. Only limitations found in those claims apply to
`the invention.
`
`4. A BRIEF DESCRIPTION OF THE DRAWINGS
`The ?gures illustrate the invention by Way of example,
`and not limitation. Like references indicate similar elements.
`FIG. 1 illustrates a system including one embodiment of
`a streaming audio and video system for multiple users Where
`client computers do not need plug-ins or helper programs.
`FIG. 2 illustrates an example method of streaming audio
`and video for multiple users.
`FIG. 3 illustrates a Web page that a user can use to access
`a streaming audio and video broadcast.
`FIG. 4 illustrates a Web page for selecting the bandWidth
`of the user’s Internet connection.
`FIG. 5 illustrates a Web page having streaming audio and
`video.
`
`5. THE DESCRIPTION
`The folloWing sections describe embodiments of the
`invention. The ?rst section provides de?nitions that Will help
`in the understanding of the remaining sections. The second
`section shoWs an example system that supports various
`embodiments of the invention. The third section describes an
`example method of using the invention. The fourth section
`illustrates an actual use of the streaming audio and video
`used in some embodiments of the invention. The last section
`revieWs additional alternative embodiments of the invention.
`a. De?nitions
`The folloWing de?nitions Will help in the understanding
`of the folloWing description.
`Client: a computer, process, program, or combination of
`computers, processes or programs, that can communi
`cate With a server.
`Server: a computer, process, program, or combination of
`computers, processes or programs, that can communi
`cate With a client. The server and the client can be
`executing on the same computer(s).
`
`15
`
`25
`
`35
`
`45
`
`55
`
`65
`
`4
`Web server: a server for serving at least Internet related
`requests. Example Web servers can serve HTML pages
`in response to HTTP requests from clients. Some Web
`servers can serve many different kinds of requests, e. g.,
`HTTPS, and/or FTP.
`BroWser: an application, program, process, or combina
`tion of applications, programs or processes, that alloW
`a client to make a request of a Web server and process
`the results of the request. The broWser may be part of
`a stand alone application or a set of programs that are
`integrated into the operating system of the client.
`Plug-in: plug-ins are external softWare programs that
`extend the capabilities of the broWser in a speci?c Way.
`For example, a plug-in can be used to give the broWser
`the ability to play audio samples or vieW video movies
`from Within the broWser.
`Helper application: like plug-ins, helper applications are
`external softWare programs. The broWser redirects
`some types of ?le to the helper applications. The helper
`applications alloW clients to process many different
`types of ?les on the Internet. When the broWser
`encounters a sound, image, or video ?le, the broWser
`hands off the data to the helper applications to run or
`display the ?le.
`JavaScript: a stand-alone programming language built
`into many broWsers. Primarily an extension to the
`Internet standard HTML language.
`Java: a programming language Whose programs (called
`applets) can travel over the Internet for use by clients.
`Whereas plug-ins and helper applications are tailored
`for a particular type of client, Java applets generally
`Work across the Internet regardless of the type of client.
`Many broWsers include Java capabilities so Java
`applets require no installation. Except Where noted,
`Java and JavaScript are interchangeable.
`b. System
`FIG. 1 illustrates a system including one embodiment of
`the invention Where audio and video is supplied over the
`Internet to multiple users. The folloWing paragraphs ?rst list
`the elements of FIG. 1, describe their interconnections, and
`then describe the various elements in greater detail.
`This paragraph lists the elements of FIG. 1. FIG. 1
`includes a system having three parts: a client side 100, a
`communications interface 180, and a server side 130. The
`client side 100 includes a client 112 and a client 111. The
`client 112 includes a broWser 102 having a video display
`area 103. The client 111 includes a broWser 108 and a video
`display area 104. The communications interface 180
`includes the Internet 185. The server side 130 includes a Web
`server 131, a shared memory 135, and a real-time server
`140. The Web server 131 includes tWo processes, a process
`138 and a process 139. The real-time server 140 includes a
`video module 144, an audio module 146, a video proxy 148,
`and an audio proxy 149. The real-time server 140 interfaces
`With a number of other elements. These elements include a
`video card 159, an input audio interface 162, and an HTTP
`connection to remote server 170. Various elements of FIG.
`1 have noW been listed.
`The folloWing paragraphs describe the interconnections
`betWeen the elements of FIG. 1. Beginning on the server side
`130, the video card 159 receives a video input 158 outputs
`a digital video signal to the video module 144. Similarly, an
`input audio interface 162 receives a mike input 164 and/or
`a line input 166 and outputs a digital audio signal to the
`audio module 146. The HTTP connection to remote server
`170 receives a streaming audio and video data 175. The
`
`Netflix 1016 - Page 9
`
`

`
`US 6,711,622 B1
`
`5
`HTTP connection to remote server 170 outputs the video
`data to the video proxy module 148 and the audio data to the
`audio proxy module 149. The real-time server 140 uses the
`data received by the various modules and stores portions of
`that data in the shared memory 135, after some manipulation
`of the data. The shared memory 135 is accessed by the Web
`server 131. The process 138 and the process 139 transmit
`and receive data to the communications interface 180. Thus,
`the couplings of the server side 130 have been described. In
`some embodiments, the process 138 and the process 139
`correspond to HTTPD processes.
`The communications interface 180 alloWs the client side
`100 to communicate With the server side 130. The commu
`nications interface, and the Internet 185 in particular, sup
`port many different types of connections by the client side
`100 and the server side 130. In the example communications
`inter face 180, the communications interface 180 includes
`the Internet 185. In particular, in this example, the process
`138 is communicating streaming audio and video data 176
`to the Internet 185. Similarly, the process 139 is communi
`cating the streaming audio and video data 178 to the Internet
`185.
`On the client side 100, the client 112 is communicating
`the streaming audio and video data 176 With the Internet
`185. Similarly, the client 111 is communicating With the
`Internet 185 to receive and manage the streaming audio and
`video data 178. The clients then communicate the informa
`tion to their respective broWser applications. The broWser
`applications the generate video images in their respective
`video display areas.
`Thus, the connections betWeen the various elements of
`FIG. 1 have been described. NoW the various elements Will
`be described in greater detail in the folloWing paragraphs.
`An example method of using these elements is described
`beloW in relation to FIG. 2.
`The server side 130 Will be described ?rst.
`The video input 158 represents a video signal that a user
`of such a system Wishes to broadcast to the clients on the
`client side 100. The video input 158 can include analog
`signals representing video information. The video card 159
`digitiZes the video input 158 to produce a digital video
`image. Various embodiments of the invention include Sun
`video cards available from Sun Microsystems, of Mountain
`VieW, Calif., and Parallax XVideo XtraTM video cards,
`available from Parallax Graphics, Inc., of Santa Clara, Calif.
`HoWever, What is important is that the real-time server 140,
`and in particular the video module 144, receives some sort
`of digitiZed video signal. The video module 144 is respon
`sible for providing the real-time server 140 With the video
`information that Will be broadcast to the client side 100. The
`video module 144 can convert the digitiZed video signals to
`be of better use to the rest of the system. For example, the
`video module 144 may, if not done by the video card 159,
`convert digital video data into a sequence of JPEG digital
`video frames. In any case, What is important is that the
`real-time server 140 receives digital video information in a
`format that it can use (example formats include, JPEG,
`MPEG, GIF, and AVI).
`Similarly, the input audio interface 162 alloWs for the
`input of analog audio signals and converts this input to
`digital audio signals. What is important is that the audio
`module 146 receives a digitiZed audio signal that can be used
`by the real-time server 140. The audio module 146 may
`convert the digitiZed audio signal into any of a number of
`formats, corresponding to any of a number of audio trans
`mission rates (examples include G711 and G723 audio
`compression formats).
`
`15
`
`25
`
`35
`
`45
`
`55
`
`65
`
`6
`The HTTP connection to remote server 170 represents an
`important advantage of one embodiment of the invention. In
`this embodiment, the HTTP connection to remote server 170
`alloWs the server side 130 to forWard broadcasts of audio
`and video signals from other streaming audio and video
`servers. In these uses, the server side 130 acts as a client to
`another server. The HTTP connection to remote server 170
`can receive video and audio signals being broadcast through
`the Internet 185 from another server. The HTTP connection
`to remote server 170 provides the digital video information
`from the other server to the video proxy module 148.
`Similarly, the HTTP connection to remote server 170 pro
`vides the audio data to the audio proxy module 149. The
`video proxy module 148 and the audio proxy module 149
`then supply the respective video and audio data to the
`real-time server 140.
`The real-time server 140 represents an application, or set
`of applications, executing on one or more computers, that
`prepares audio and video data for broadcasting to multiple
`users through the Web server 131. The real-time server 140
`takes the data from the various modules, processes the data,
`and stores the processed data in the shared memory 135. The
`real-time server 140 can perform compression, and other
`manipulations of the data, to reduce the processing burden
`on the Web server 131. For example, in some embodiments
`of the invention, the real-time server 140 receives digitiZed
`video data and compresses that data into JPEG images.
`These JPEG images are sequenced digital frames of video.
`Similarly, for the audio data, the real-time server 140 breaks
`the audio information into one-half second time periods of
`audio data (other embodiments use other time periods).
`These one-half second time periods of data are stored in the
`shared memory 135. The real-time server 140 can also
`compress the audio information into one of a number of
`various compressed audio signals (e.g., G.711 and/or G.723
`audio compression formats). In some embodiments of the
`invention, the real-time server can broadcast audio and video
`from multiple sources to multiple clients.
`The shared memory 135 represents a shared storage area
`for use by the real-time server to store audio and video data
`for access by the Web server 131. In one embodiment the
`shared memory 135 has a locking and semaphore usage
`scheme to ensure that the real-time server 140 is not Writing
`data into the shared memory 135 While the Web server 131
`is accessing that data. In some embodiments, the sema
`phores act as noti?ers to indicate that neW data in the shared
`memory 135 is available for use by the Web server 131. In
`some embodiments, the video data and the audio data are
`stored in different shared memory locations.
`The Web server 131 communicates data over the Internet
`185 using one or more communications protocols. In some
`embodiments of the invention, these protocols include
`HTTP (Hypertext Transfer Protocol), TCP (Transmission
`Control Protocol) and UDP (User Datagram Protocol). The
`Web server 131 represents an application, including one or
`more processes, for communicating over the Internet 185. In
`one embodiment, the Web server 131 includes an Apache
`Web server. Each of the processes in the Web server 131
`represents one or more processes for serving streaming
`audio and video data to the client side 100. In some
`embodiments, the Web server 131 transmits the video data as
`a multipart MIME (multi-purpose Internet mail extensions)
`encoded ?le for decoding directly by the broWser or as
`compressed video information for decoding by an applet run
`in the broWser. The Web server 131 transmits the audio data
`as compressed audio data for decoding by an applet run in
`the broWser.
`
`Netflix 1016 - Page 10
`
`

`
`US 6,711,622 B1
`
`7
`In some embodiments, the Web server 131 initiates a
`separate process for each audio and video connection made
`from client side 100. Thus, for one client receiving stream
`ing audio and video data, tWo processes Would be started
`Within the Web server 131. The ?rst process Would supply
`video data and the second process Would supply audio data.
`The processes access the shared memory 135 and serve the
`data across the Internet to the respective client.
`In some embodiments, the Web server 131 initiates at least
`one process for each client. This provides important advan
`tages in some embodiments of the invention. In particular,
`because the Web server 131 is serving the data directly
`through processes it created, server side 130 users need not
`Worry about security issues beyond those already faced With
`their Web server 131. Thus, these embodiments of the
`invention have a loWer chance of interfering With client side
`100 ?re Walls and have a loWer chance of having a server
`side 130 security problem.
`Other embodiments of the invention include separate
`Common GateWay Interface (CGI) programs for audio and
`video. These CGI programs are used by the Web server 131
`to serve the streaming audio and video data. These CGI
`programs are not necessarily integrated as tightly to the Web
`server 131 as the process 138 and the process 139. HoWever,
`a CGI program alloWs for the easy extension of many
`different types of Web servers.
`The communications interface 180 permits communica
`tions betWeen the server side 130 and the client side 100. In
`this example, the Internet 185 supports the communications.
`Other embodiments of the invention support other commu
`nications interfaces. For example, the Internet 185 can be
`replaced by a local area netWork, a Wide are netWork, a
`proprietary telecommunications and networking
`infrastructure, or some other communications interface.
`What is important is that the server side 130 can commu
`nicate With the client side 100. The communications inter
`face 180 can also include combinations of the above
`described technologies. For example, the Internet 185 can
`include a Web server to Which the clients on the client side
`communicate through to access the Internet 185. The client
`side 100 can be on a local area netWork that is connected
`through a server, or router, to the Internet 185.
`The client side 100 represents the consumers of the
`streaming audio and video data. In this example, the tWo
`clients are receiving separated streaming audio and video
`data signals. Other embodiments of the invention support
`many more clients.
`The client 112 represents a computer, such as a PC
`compatible computer, running a broWser application 102.
`For video display, the broWser application 102 can include
`a Netscape NavigatorTM or CommunicatorTM program for
`“multipart/x-mixed-replace MIME type video,” or a
`Microsoft Internet ExplorerTM 3.0 or later for a Java based
`video transmission. In some embodiments, the Java based
`video transmission applet parses the multipart/x-mixed
`replace MIME type video. For audio, the broWser applica
`tion 102 can include any broWser that supports Java and/or
`J avaScript.
`The broWser application 102 is responsible for receiving
`the streaming audio and video data 176 and reconstructing
`an audio and video signal suitable for the end user. In this
`example, the video display area 103 displays the recon
`structed video information received from the video input
`158 at the real-time server 140. Similarly, the client 111 is
`executing the broWser 108. The broWser 108 is displaying
`the same video signal in the video display area 104. The
`client 11 represents another computer executing a broWser
`application.
`
`10
`
`15
`
`25
`
`35
`
`45
`
`55
`
`65
`
`8
`Various embodiments of the invention have modi?cations
`to the system shoWn in FIG. 1. Some of these variations are