(12) United States Patent
`Gupta et al.
`
`USOO6622171B2
`US 6,622,171 B2
`Sep. 16, 2003
`
`(10) Patent No.:
`(45) Date of Patent:
`
`(73) ASSignee: Mison Corporation, Redmond, WA EP
`EP
`EP
`
`5,414,455 A 5/1995 Hooper et al. ................. 348/7
`5,420,801. A * 5/1995 Dockter et al. ............. 34.5/501
`5.434,848 A 7/1995 Chimento, Jr. et al. ..... 370/232
`5,455,910 A 10/1995 Johnson et al. .......... 707/500.1
`5,481,542 A 1/1996 Logston et al. ............. 725/131
`(List continued on next page.)
`FOREIGN PATENT DOCUMENTS
`O 669 587 A2 * 1/1994
`O605115
`7/1994
`O653.884
`5/1995
`O 669587 A2 SE
`O676898
`10/1995
`O746,158
`12/1996
`O 812 112 A2 * 10/1997
`WO 94/O1964
`* 1/1994
`0-812112 A2 10/1997 .......... HO4N/7/173
`WO 98/37698
`* 8/1998
`OTHER PUBLICATIONS
`H.J. Chen et al., “A Scalable Viedo-on-Demand Service for
`the Provision of VCR-Like Functions," IEEE, May 15,
`, pp
`
`EP
`EP
`EP
`WO
`WO
`WO
`
`1995, pp. 65-72.
`
`(List continued on next page.)
`
`Primary Examiner Mark R. Powell
`Assistant Examiner William C. Vaughn, Jr.
`(74) Attorney, Agent, or Firm-Lee & Hayes, PLLC
`(57)
`ABSTRACT
`Multimedia content is Streamed over a network System from
`a server computer to a client computer. The client allows a
`user to enter a variable playback Speed and varies the Speed
`at which the multimedia content is rendered at the client.
`Time-Scale modification technology is used to maintain the
`original pitch of any audio content, thereby maintaining its
`intelligibility.
`
`- -
`
`-
`
`-
`
`32 Claims, 9 Drawing Sheets
`
`(54) MULTIMEDIA TIMELINE MODIFICATION
`IN NETWORKED CLIENT/SERVER
`SYSTEMS
`
`(75) Inventors: Anoop Gupta, Woodinville, WA (US);
`Nshare D. Omoigui, Redmond, WA
`
`(*) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`O
`
`-1 - O
`
`- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
`
`(56)
`
`(21) Appl. No.: 09/153,664
`(22) Filed:
`Sep. 15, 1998
`(65)
`Prior Publication Data
`US 2002/0038374A1 Mar. 28, 2002
`G06F 15/16
`(51) Int. Cl."
`(52) U.S. Cl. ....................... 709/231; 709/246; 709/219;
`709/232; 345/704; 34.5/748
`(58) Field of Search ................................. 709/219, 217,
`709/231, 233, 234, 232, 247, 225, 226,
`248,246, 245, 263; 725/41, 43, 90,94,
`96; 345/700, 704, 725, 748
`References Cited
`U.S. PATENT DOCUMENTS
`a
`4,931,950 A 6/1990 Isle et al. ................... 364/513
`5,050,161 A 9/1991 Golestani.................... 370/60
`5,119,474 A : 6/1992 Beitel et al. ................ 395/154
`52.7 A 13. E. E. - - - - - - - - - - - - - - - - .f
`5,309,562 A 5/1994 Li ................
`...
`5,313,454 A 5/1994 Bustini et al. ................ 370/13
`5,341,474 A 8/1994 Gelman et al. ...
`... 395/200
`
`2
`
`2
`
`CC. C. a.
`
`- - -
`
`
`
`- 130
`- Y-... x
`Select Muftimedia
`Content
`
`Accept speed
`Designation
`
`w
`
`Requested
`Selected Content
`
`--
`
`32
`
`134
`
`—
`y
`dy Composite
`
`Media Stream
`
`-- 138
`
`- i38
`
`Stream to Client
`
`14
`
`Receive streaming
`Content
`
`Modify Timelines
`
`144 -
`
`y
`
`Render Time
`Modified Content
`
`-1-
`
`Amazon v. Audio Pod
`US Patent 10,805,111
`Amazon EX-1061
`
`

`

`US 6,622,171 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`
`
`1/2000 Cannon et al. ............. 709/231
`6,014,706 A
`6,023,731 A 2/2000 Chawla ...................... 709/231
`6,032,130 A
`2/2000 Alloul et al. ................. 705/27
`6,035,341 A 3/2000 Nunally et al. ............. 709/253
`6,041,345 A 3/2000 Levi et al. .....
`... 709/217
`6,049,823 A 4/2000 Hwang .............
`709/218
`6,064,794. A 5/2000 McLaren et al. .
`... 386/68
`6,118,450 A 9/2000 Proehl et al. .....
`... 345/810
`6,118,817 A 9/2000 Wang ...............
`... 375/240
`6,128,653 A 10/2000 Del Val et al. ............. 709/219
`6,133,920 A 10/2000 deCarmo et al. ........... 345/840
`6,144,375. A 11/2000 Jain et al. ............ ... 345/302
`6,148,304. A 11/2000 de Vries et al. ............ 707/104
`6,154,771. A 11/2000 Rangan et al. .............. 709/217
`6,166,314. A 12/2000 Weinstock et al.......... 84/483.1
`6,169,843 B1
`1/2001 Lenihan et al. ............... 386/46
`6,173,317 B1
`1/2001 Chaddha et al. ..... ... 709/219
`6,204.840 B1
`3/2001 Petelycky et al.
`... 345/302
`6,215,910 B1
`4/2001 Chaddha .............. ... 382/253
`6,230,172 B1
`5/2001 Purnaveja et al. .......... 707/512
`6,233,389 B1
`5/2001 Barton et al. ................. 386/46
`6,332,147 B1 * 12/2001 Moran et al. ..
`707/500.1
`6,345,293 B1 * 2/2002 Chaddha ........
`707/104.1
`6,392,651 B1 * 5/2002 Stradley ...........
`345/473
`6,424,792 B1 * 7/2002 Tsukagoshi et al. .......... 386/95
`6,453,336 B1 * 9/2002 Beyda et al. ............... 709/204
`
`2/1996 Macera et al. .............. 709/249
`5,490,252. A
`5,504,744 A 4/1996 Adams et al.
`... 370/232
`5,519,701. A 5/1996 Colmant et al.
`... 370/412
`5,521,630 A 5/1996 Chen et al. ...
`... 725/90
`5,533,021 A 7/1996 Branstad et al.
`... 370/396
`5,537,408 A * 7/1996 Branstad et al. .
`... 370/473
`5,541.955 A 7/1996 Jacobsmeyer ....
`... 375/222
`5,559,942 A
`9/1996 Gough et al. .
`... 395/155
`5,566,175 A 10/1996 Davis ........
`... 370/468
`5,574,724. A 11/1996 Bales et al. ...
`... 370/410
`5,614,940 A 3/1997 Cobbley et al. ............... 348/7
`5,617,423 A 4/1997 Li et al. .......
`... 370/426
`5,623,690 A 4/1997 Palmer et al.
`... 707/500.1
`5,625,405 A 4/1997 Dulac et al. .
`... 725/93
`5,640,320 A * 6/1997 Jackson et al.
`700/87
`5,642,497 A * 6/1997 Crary et al. ..
`... 703/23
`5,664.227 A 9/1997 Mauldin et al.
`... 395/778
`5,692.213 A * 11/1997 Goldberg et al
`707/500.1
`5,692,313 A * 12/1997 Ikeda et al.
`... 34/319
`5,717,691. A
`2/1998 Dighe et al.
`... 370/401
`5,717.869 A * 2/1998 Moran et al. .
`... 345/339
`5,719,786 A * 2/1998 Nelson et al.
`... 709/219
`5,721,829. A
`2/1998 Dunn et al. ................. 709/219
`5,742,347 A
`4/1998 Kandlur et al. ........ 375/240.25
`5,768,533 A 6/1998 Ran
`5,786.814 A 7/1998 Moran et al. ............... 345/328
`5,793.980 A * 8/1998 Glaser et al. ....
`... 709/231
`Microsoft Corporation and RealNetworks, Inc., Advanced
`5,794.210 A 8/1998 Goldhaber et al.
`... 705/14
`Streaming Format (ASF) Specification, Feb. 26, 1998, Pub
`5,794.249 A 8/1998 Orsolini et al. ..
`... 707/104
`lic Specification Version 1.0, 55 pages.
`5,801,685 A * 9/1998 Miller et al...
`... 345/302
`Lynda Hardman et al., “Multimedia authoring paradigns,”
`5,808,662 A * 9/1998 Kinney et al.
`348/14.1
`Authoring and Application of Hypermedia-Based Use
`E. A : s
`al
`"ji.
`r-Interfaces, IEE Colloquim, The Institution of Electrical
`5.82ss48 A 10/1998 MacCormack et al. 395/200.77
`5,832,437 A * 11/1998 Nishiguchi et al. ......... 702s Engineers 1995, pp. 8/1-8/3.
`5.835,495. A 11/1998 Ferriere ...................... 370465
`John David N. Dionisio and Alfonso F. Cardenas, “A Unified
`5.835,667 A 11/1998 Wactlar et al................. 386/96
`Data Model for Representing Multimedia, Timeline, and
`5,838,906 A 11/1998 Doyle et al. ..
`395/200.32
`Simulation Data," IEEE Transactions. On Knowledge and
`5,842,172 A 11/1998 Wilson ....................... 704/503
`Data Engineering, vol. 10, No. 5, Sep./Oct. 1998, pp.
`5,852,705 A * 12/1998 Hanko et al. ................. 386/92
`746-767.
`SEA :
`t
`S. - - - - - - i.
`3.
`Informedia-Internet References, http://www.informedi
`2Y- - -2
`Orter et al. ................
`f
`a.cs.cmu.edu, date unknown.
`5,870.755 A
`2/1999 Stevens et al. ............. 707/104
`-- - -
`5,873,735 A
`2/1999 Yamada et al.
`... 434/316
`GSM Full Rate Speech Transcoding.” ETS1/PT 12, Feb.
`5,880,788 A * 3/1999 Bregler ....................... 345/473
`1992, pp. 1-93.
`5,892,506 A
`4/1999 Hermanson ................. 345/302
`“Speech Codec for the European Mobile Radio System, P.
`5,893,053 A * 4/1999 Trueblood ..........
`... 702/187
`Vary et al., 1998, pp. 227–230.
`s: A :
`Badar et al. ........3 s:
`Arons, Barry, “SpeechSkimmer: A System for Interactively
`2- Y -
`an el al. . . . . . . . . . . . . . . . .
`ss
`5,918,002 A 6/1999 Klemets et al. ..
`395/182.16
`Skimming Recorded Speech”, ACM Transactions on Com
`5,930,787. A 7/1999 Minakuchi et al. ............ o,
`puter-Human, vol. 4. No. 1, pp. 338.
`5.941,936 A * 8/1999 Taylor ............................... 11
`Internet Reference “An Annotated Bibliography of Interac
`5,953,506 A * 9/1999 Kalra et al.
`... 709/231
`tive Speech User Interfaces by Barry Arons' HTTP://bar
`5,956,716 A 9/1999 Kenner et al. ................ 707/10
`ons.www.media.mit.edu/people/barons/Arons Annotated
`5,995,941 A 11/1999 Maquire et al. .............. 705/10
`Bibliography.html. Date unknown.
`5.999,979 A * 12/1999 Vellanki et al...
`... 709/232
`6,006,241. A 12/1999 Purnaveja et al. .......... 707/512
`* cited by examiner
`
`OTHER PUBLICATIONS
`
`-2-
`
`

`

`U.S. Patent
`
`Sep. 16, 2003
`
`Sheet 1 of 9
`
`US 6,622,171 B2
`
`
`
`Streaming
`Data Files
`
`Streaming Data
`
`-3-
`
`

`

`U.S. Patent
`
`Sep. 16, 2003
`
`Sheet 2 of 9
`
`US 6,622,171 B2
`
`
`
`
`
`
`
`Lv
`
`i
`
`87
`
`_OPPlA
`
`Jaydepy
`
`
`
`eojy[e907
`
`
`
`
`
`
`
`
`sngWweajskg
`
`
`
` Lg—YJOMaN©
`
`eeWeyUy
`
`Uod
`
`WOMAN
`jeuag
`
`\
`
`asnoyy
`
`BOeLaIU]
`
`
`
`
`
` e
`
`
`/paeoghsypjis
`
`
`
` uoieaddy|Buyeiedo
`
`SIOMION
`
`swelB0ig
`
`~
`
`uojeoddy
`
`
`
`pueoghey
`
`
`
`
`
`BalySpPlIAAWapoyy
`
`A
`
`
`
`ZSGS
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
` suelBoldWua}sAS
`
`Ge
`
`
`
`yupBuissacolg
`jeeeSeceeeecie iereieiereieaed
`.)
`ZE sainpoyy6LyWelGolY
`
`waysAs
`9€uoneoddy
`
`J8UIO!
`
`Se
`
` Buyeiedo Kowa,weyskS
`
`swelbold
`
`-4-
`
`-4-
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Sep. 16, 2003
`
`Sheet 3 of 9
`
`US 6,622,171 B2
`
`10
`
`Composite
`Media
`Stream
`
`
`
`104
`
`
`
`105-
`
`Audio
`
`Demultiplex
`
`108
`
`110 -
`
`
`
`
`
`
`
`
`
`Audio Time
`MOdification
`
`Video Time
`Modification
`
`
`
`
`
`Audio Render
`
`Video Render
`
`-5-
`
`

`

`U.S. Patent
`
`Sep. 16, 2003
`
`Sheet 4 of 9
`
`US 6,622,171 B2
`
`Select Multimedia
`Content
`
`Accept Speed
`Designation
`
`Requested
`Selected Content
`
`130
`
`132
`
`134
`
`
`
`
`
`
`
`14O
`
`142
`
`144
`
`Receive Streaming
`Content
`
`Modify Timelines
`
`Render Time
`Modified Content
`
`Identify Composite
`Media Stream
`
`138
`
`Stream to Client
`
`-6-
`
`

`

`U.S. Patent
`
`Sep. 16, 2003
`
`Sheet 5 of 9
`
`US 6,622,171 B2
`
`200
`
`MultiMedia Player
`202 NFile view Play Favorites Help
`
`X
`
`204 N-
`
`Media Screen
`
`206
`
`4
`21
`
`
`
`
`
`21 O <
`
`A. 212(1) 220), 212(3)
`it as 224-1
`D. x10 D x 1.25 D x1.50
`
`222
`
`- 1 - 208
`
`-
`
`Clip:
`Author:
`Copyright:
`Stopped:
`
`22O
`218
`216
`
`OO:36 / OO:36
`
`-7-
`
`

`

`U.S. Patent
`
`Sep. 16, 2003
`
`Sheet 6 of 9
`
`US 6,622,171 B2
`
`240.
`
`MultiMedia Player
`202 NFile View Play Favorites Help
`
`X
`
`204 - N
`
`Media Screen
`
`242
`
`--
`2O6 <
`
`210 -
`
`O.5
`
`IKKKKDD (
`2.5
`248
`- 246
`ShOW:
`Clip:
`Author:
`Copyright:
`Stopped:
`
`N- 226
`
`1 1-208
`
`OO:36 / OO:36
`
`-8-
`
`

`

`U.S. Patent
`
`Sep. 16, 2003
`
`Sheet 7 of 9
`
`US 6,622,171 B2
`
`260
`
`MultiMedia Player
`202 - File View Play Favorites Help
`
`DX
`
`204
`
`Media Screen
`
`N- 226
`
`262
`
`266
`
`206
`
`Y
`
`- -
`
`(
`
`- - -
`
`--a
`
`X10
`
`X15
`
`Copyri
`Stopped:
`
`OO:36 / OO:36
`
`-9-
`
`

`

`U.S. Patent
`
`Sep. 16, 2003
`
`Sheet 8 of 9
`
`US 6,622,171 B2
`
`
`
`303
`
`Client
`
`314
`
`Client
`
`-10-
`
`

`

`U.S. Patent
`
`Sep. 16, 2003
`
`Sheet 9 of 9
`
`US 6,622,171 B2
`
`32O
`
`330
`
`Server
`
`
`
`324
`
`Client
`
`334
`
`Client
`
`-11-
`
`

`

`US 6,622,171 B2
`
`1
`MULTIMEDIA TIMELINE MODIFICATION
`IN NETWORKED CLIENT/SERVER
`SYSTEMS
`
`TECHNICAL FIELD
`This invention relates to networked client/server Systems
`and to methods of Streaming and rendering multimedia
`content in Such Systems.
`BACKGROUND OF THE INVENTION
`Multimedia Streaming-the continuous delivery of Syn
`chronized media data like Video, audio, text, and
`animation-is a critical link in the digital multimedia revo
`lution. Today, Streaming media is primarily about Video and
`audio, but a richer, broader digital media era is emerging
`with a profound and growing impact on the Internet and
`digital broadcasting.
`Synchronized media means multiple media objects that
`share a common timeline. Video and audio are examples of
`Synchronized media-each is a separate data Stream with its
`own data Structure, but the two data Streams are played back
`in Synchronization with each other. Virtually any media type
`can have a timeline. For example, an image object can
`change like an animated .gif file: text can change and move,
`and animation and digital effects happen over time. This
`concept of Synchronizing multiple media types is gaining
`greater meaning and currency with the emergence of more
`Sophisticated media composition frameworks implied by
`MPEG-4, Dynamic HTML, and other media playback envi
`rOnmentS.
`The term “streaming” is used to indicate that the data
`representing the various media types is provided over a
`network to a client computer on a real-time, as-needed basis,
`rather than being pre-delivered in its entirety before play
`back. Thus, the client computer renderS Streaming data as it
`is received from a network Server, rather than waiting for an
`entire “file to be delivered.
`The widespread availability of Streaming multimedia
`enables a variety of informational content that was not
`previously available over the Internet or other computer
`networks. Live content is one significant example of Such
`content. Using Streaming multimedia, audio, Video, or
`audio/visual coverage of noteworthy events can be broadcast
`over the Internet as the events unfold. Similarly, television
`and radio Stations can transmit their live content over the
`Internet.
`In comparison to text-based or paper-based presentations,
`Streaming multimedia presentations are very effective in
`certain Situations. Audio/visual presentations, for example,
`are able to capture and convey many Subtle factors that are
`not perceivable from paper-based documents. Even when
`the content is a spoken presentation, an audio/visual record
`ing captures gestures, facial expressions, and various speech
`nuances that cannot be discerned from text or even from Still
`photographs.
`Although Streaming multimedia content compares favor
`ably with textual content in most regards, one disadvantage
`is that it requires significant time for viewing. It cannot be
`"skimmed' like textual content. Thus, information consum
`ers are forced to choose between the efficiency of the written
`word and the richness of the multimedia experience.
`The invention described below addresses this disadvan
`tage of prior art Streaming multimedia content, allowing
`more efficient multimedia perusal of Streaming multimedia
`presentations than has previously been possible.
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`SUMMARY OF THE INVENTION
`The invention utilizes time-Scale modification So that a
`user can vary the Speed of Streaming content without
`destroying its intelligibility. In accordance with the
`invention, a user Selects multimedia content from a menu
`presented at a network client computer. In addition, the user
`Selects a speed factor, indicating the Speed at which the
`multimedia should be rendered relative to its default speed.
`In response to these Selections by the user, the network
`client contacts a network Server and initiates a composite
`media Stream from the Server to the client. AS it receives the
`media Stream, the client processes the individual media
`Streams of the composite Stream, altering their timelines in
`accordance with the user's Speed designation. Time-Scale
`modification techniques are used to alter audio Streams to
`maintain their original pitch.
`In alternative embodiments, timelines are altered at the
`Server, before Streaming to the client. To accomplish this, the
`Server either alters the timelines dynamically, as Specific
`content is requested, or Selects from a number of prestored
`media Streams having pre-altered timelines.
`The invention includes methods of adapting to limited
`bandwidth situations by composing or Selecting composite
`Streams having differing degrees of quality, and/or by com
`posing or Selecting Streams with timelines that are altered to
`closely correspond with whatever Speed factor has been
`chosen. In one embodiment of the invention, certain media
`Streams, Such as audio Streams, take precedence over other
`Streams Such as Video streams. In this embodiment of the
`invention, the audio stream is sent with an unaltered
`timeline, at a rate Sufficient to Satisfy the consumption
`requirements of the client, given the current Speed factor.
`The video is then degraded in quality to reduce its
`bandwidth, so that it can be streamed in whatever bandwidth
`is not require by the audio.
`BRIEF DESCRIPTION OF THE DRAWINGS
`FIG. 1 is a block diagram of a networked client/server
`System in accordance with the invention.
`FIG. 2 is a block diagram of a networked computer that
`can be used to implement either a Server or a client in
`accordance with the invention.
`FIG. 3 is a block diagram illustrating communications and
`rendering of a composite media Stream in accordance with
`the invention.
`FIG. 4 is a flowchart illustrating methodological aspects
`of the invention.
`FIG. 5 is a diagrammatic illustration of a graphical user
`interface window having a time-Scale-modification tool for
`changing a playback Speed of Streaming multimedia content
`according to one implementation.
`FIG. 6 is a diagrammatic illustration of a graphical user
`interface window having a time-scale-modification tool
`according to a Second implementation.
`FIG. 7 is a diagrammatic illustration of a graphical user
`interface window having a time-scale-modification tool
`according to a third implementation.
`FIG. 8 is a block diagram illustrating one embodiment in
`which a plurality of timeline-altered media Streams are
`Stored at and provided from a Server.
`FIG. 9 is a block diagram illustrating another embodiment
`in which a plurality of timeline-altered media Streams are
`Stored at and provided from a Server.
`FIG. 10 is a block diagram illustrating yet another
`embodiment in which a plurality of timeline-altered media
`Streams are Stored at and provided from a Server.
`
`-12-
`
`

`

`3
`FIG. 11 is a block diagram illustrating yet another
`embodiment in which a plurality of timeline-altered media
`Streams are Stored at and provided from a Server.
`DETAILED DESCRIPTION
`General Network Structure
`FIG. 1 shows a client/server network system and envi
`ronment in accordance with the invention. Generally, the
`System includes a network Server computer 10 and a plu
`rality of network client computers 11. The computers com
`municate with each other over a data communications
`network. The communications network in FIG. 1 comprises
`a public network 12 Such as the Internet. The data commu
`nications network might also include local-area networks
`and private wide-area networks.
`Server computer 10 has access to Streaming media content
`in the form of different composite media Streams. Some
`composite media Streams might be Stored as files in a
`database or other file Storage System 13. Other composite
`media Streams might be Supplied to the Server on a "live”
`basis from other data Source components through dedicated
`communications channels or through the Internet itself.
`Streaming Media
`In this discussion, the term “composite media Stream”
`describes Synchronized Streaming data that represents a
`Segment of multimedia content. The composite media
`Stream has a timeline that establishes the Speed at which the
`content is rendered. The composite media Stream can be
`rendered to produce a plurality of different types of user
`perceivable media, including Synchronized audio or Sound,
`Video graphics or motion pictures, animation, textual
`content, command Script Sequences, or other media types
`that convey time-varying information or content in a way
`that can be sensed and perceived by a human. A composite
`media Stream comprises a plurality of individual media
`Streams representing the multimedia content. Each of the
`individual media Streams corresponds to and represents a
`different media type and each of the media Streams can be
`rendered by a network client to produce a user-perceivable
`presentation using a particular presentation medium. The
`individual media Streams have their own timelines, which
`are Synchronized with each other So that the media Streams
`can be rendered simultaneously for a coordinated multime
`dia presentation.
`There are various Standards for Streaming media content
`and composite media Streams. The "Advanced Streaming
`Format (ASF) is an example of Such a standard, including
`both accepted versions of the Standard and proposed Stan
`dards for future adoption. ASF specifies the way in which
`multimedia contentis Stored, Streamed, and presented by the
`tools, Servers, and clients of various multimedia Vendors.
`ASF provides benefits Such as local and network playback,
`extensible media types, component download, Scalable
`media types, prioritization of Streams, multiple language
`Support, environment independence, rich inter-stream
`relationships, and expandability. Further details about ASF
`are available from Microsoft Corporation of Redmond,
`Wash.
`Regardless of the Streaming format used, an individual
`data Stream contains a Sequence of digital data units that are
`rendered individually, in Sequence, to produce an image,
`Sound, or Some other Stimuli that is perceived by a human to
`be continuously varying. For example, an audio data Stream
`comprises a Sequence of Sample values that are converted to
`a pitch and Volume to produce continuously varying Sound.
`A video data Stream comprises a Sequence of digitally
`Specified graphics frames that are rendered in Sequence to
`produce a moving picture.
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 6,622,171 B2
`
`15
`
`25
`
`4
`Typically, the individual data Streams of a composite
`media Stream are interleaved in a Single Sequence of data
`packets. Various types of data compression might be used
`within a particular data format to reduce communications
`bandwidth requirements.
`The sequential data units (Such as audio Sample values or
`video frames) are associated with both delivery times and
`presentation times, relative to an arbitrary Start time. The
`delivery time of a data unit indicates when the data unit
`should be delivered to a rendering client. The presentation
`time indicates when the value Should be actually rendered.
`Normally, the delivery time of a data unit precedes the
`presentation time.
`The presentation times determine the actual Speed of
`playback. For data Streams representing actual events or
`performances, the presentation times correspond to the rela
`tive times at which the data Samples were actually recorded.
`The presentation times of the various different individual
`data Streams are consistent with each other So that the
`Streams remain coordinated and Synchronized during play
`back.
`Exemplary Computer Environment
`In the discussion below, the invention will be described in
`the general context of computer-executable instructions,
`Such as program modules, being executed by one or more
`conventional personal computers. Generally, program mod
`ules include routines, programs, objects, components, data
`Structures, etc. that perform particular tasks or implement
`particular abstract data types. Moreover, those skilled in the
`art will appreciate that the invention may be practiced with
`other computer System configurations, including hand-held
`devices, multiprocessor Systems, microprocessor-based or
`programmable consumer electronics, network PCs,
`minicomputers, mainframe computers, and the like. In a
`distributed computer environment, program modules may be
`located in both local and remote memory Storage devices.
`FIG.2 shows a general example of a computer 20 that can
`be used as a network node or host in accordance with the
`invention. Computer 20 is shown as an example of a
`computer that can perform the functions of either Server
`computer 10 or a client computer 11.
`Computer 20 includes one or more processors or process
`ing units 21, a System memory 22, and a System buS 23 that
`couples various System components including the System
`memory to processors 21.
`The system bus 23 may be any of several types of bus
`Structures including a memory bus or memory controller, a
`peripheral bus, and a local bus using any of a variety of bus
`architectures. The System memory includes read only
`memory (ROM) 24 and random access memory (RAM) 25.
`Abasic input/output system 26 (BIOS), containing the basic
`routines that help to transfer information between elements
`within Server computer 20, Such as during Start-up, is Stored
`in ROM 24. Computer 20 further includes a hard disk drive
`27 for reading from and writing to a hard disk, not shown,
`a magnetic disk drive 28 for reading from or writing to a
`removable magnetic disk 29, and an optical disk drive 30 for
`reading from or writing to a removable optical disk 31 Such
`as a CD ROM or other optical media. The hard disk drive 27,
`magnetic disk drive 28, and optical disk drive 30 are
`connected to the system bus 23 by an SCSI interface 32 or
`Some other appropriate interface. The drives and their asso
`ciated computer-readable media provide nonvolatile Storage
`of computer readable instructions, data structures, program
`modules and other data for computer 20. Although the
`exemplary environment described herein employs a hard
`disk, a removable magnetic disk 29 and a removable optical
`
`-13-
`
`

`

`S
`disk 31, it should be appreciated by those skilled in the art
`that other types of computer readable media which can Store
`data that is accessible by a computer, Such as magnetic
`cassettes, flash memory cards, digital Video disks, random
`access memories (RAMs) read only memories (ROM), and
`the like, may also be used in the exemplary operating
`environment.
`A number of program modules may be Stored on the hard
`disk, magnetic disk 29 optical disk31, ROM 24, or RAM 25,
`including an operating System 35, one or more application
`programs 36, other program modules 37, and program data
`38. A user may enter commands and information into
`computer 20 through input devices such as keyboard 40 and
`pointing device 42. Other input devices (not shown) may
`include a microphone, joystick, game pad, Satellite dish,
`Scanner, or the like. These and other input devices are
`connected to the processing unit 21 through an interface 46
`that is coupled to the system bus. A monitor 47 or other type
`of display device is also connected to the System buS 23 via
`an interface, Such as a Video adapter 48. In addition to the
`monitor, personal computers typically include other periph
`eral output devices (not shown) Such as Speakers and print
`CS.
`Computer 20 operates in a networked environment using
`logical connections to one or more remote computers, Such
`as a remote computer 49. The remote computer 49 may be
`another personal computer, a Server, a router, a network PC,
`a peer device or other common network node, and typically
`includes many or all of the elements described above
`relative to computer 20, although only a memory Storage
`device 50 has been illustrated in FIG. 2. The logical con
`nections depicted in FIG. 2 include a local area network
`(LAN) 51 and a wide area network (WAN) 52. Such
`networking environments are commonplace in offices,
`enterprise-wide computer networks, intranets, and the Inter
`net. In the described embodiment of the invention, remote
`computer 49 executes an Internet Web browser program
`such as the “Internet Explorer Web browser manufactured
`and distributed by Microsoft Corporation of Redmond,
`Wash.
`When used in a LAN networking environment, computer
`20 is connected to the local network 51 through a network
`interface or adapter 53. When used in a WAN networking
`environment, computer 20 typically includes a modem 54 or
`other means for establishing communications over the wide
`area network 52, Such as the Internet. The modem 54, which
`may be internal or external, is connected to the System bus
`23 via a serial port interface 33. In a networked environment,
`program modules depicted relative to the personal computer
`20, or portions thereof, may be Stored in the remote memory
`Storage device. It will be appreciated that the network
`connections shown are exemplary and other means of estab
`lishing a communications link between the computerS may
`be used.
`Generally, the data processors of computer 20 are pro
`grammed by means of instructions Stored at different times
`in the various computer-readable Storage media of the com
`puter. Programs and operating Systems are typically
`distributed, for example, on floppy disks or CD-ROMs.
`From there, they are installed or loaded into the Secondary
`memory of a computer. At execution, they are loaded at least
`partially into the computer's primary electronic memory.
`The invention described herein includes these and other
`various types of computer-readable Storage media when
`Such media contain instructions or programs for implement
`ing the Steps described below in conjunction with a micro
`processor or other data processor. The invention also
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 6,622,171 B2
`
`15
`
`25
`
`6
`includes the computer itself when programmed according to
`the methods and techniques described below. Furthermore,
`certain Sub-components of the computer may be pro
`grammed to perform the functions and Steps described
`below. The invention includes such sub-components when
`they are programmed as described. In addition, the invention
`described herein includes data Structures, described below,
`as embodied on various types of memory media.
`For purposes of illustration, programs and other execut
`able program components Such as the operating System are
`illustrated herein as discrete blocks, although it is recog
`nized that Such programs and components reside at various
`times in different Storage components of the computer, and
`are executed by the data processor(s) of the computer.
`Client-Based Multimedia Time-Scale Modification
`AS shown in FIG. 1, a network System in accordance with
`the invention includes a network server 10 from which a
`plurality of composite media Streams are available. In Some
`cases, the composite media Streams are actually Stored by
`server 10. In other cases, server 10 obtains the composite
`media Streams from other network Sources or devices.
`The system also includes network clients 11. Generally,
`the network clients are responsive to user input to Select or
`request identified composite media Streams. In response to a
`request for a composite media Stream, Server 10 Streams the
`requested composite media Stream to the network client in
`accordance with Some known format Such as ASF. The client
`renders the data Streams to produce the multimedia content.
`In accordance with the invention, a network client also
`accepts a Speed designation from a human user. The Speed
`designation is preferably a Speed factor relative to the
`original or default playback Speed of the Selected multime
`dia Stream. For example, a speed factor of 1.2 indicates that
`the composite media Stream is to be rendered at 1.2 times its
`original or default Speed, thereby achieving time compres
`Sion. A Speed factor of 0.8 indicates that the composite
`media Stream is to be rendered at 0.8 times its original or
`default Speed, thereby achieving time expansion.
`In response to the Speed designation from the user, the
`system modifies the timelines of the individual media
`Streams of the composite media Stream, while keeping the
`timelines Synchronized with each other and while maintain
`ing the original pitch of any audio produced from audio
`Streams. In one embodiment of the invention, Such timeline
`modification is performed by the network client. In other
`embodiments of the invention, the timeline modification can
`be performed at the network server, before the media
`Streams are Streamed to the network client.
`Timeline modification changes the timeline of the
`received data Streams in accordance with the user Speed
`designation to achieve either time compression or time
`expansion. With Some types of media, Such as Video streams,
`this involves either omitting Selected frames or modifying
`the presentation times of the individual data units or video
`frames. In other cases, Such as with audio Streams, the
`time-modification is more difficult-Simply changing the
`presentation times would alter the pitch of the original audio
`and make it unintelligible. Accordingly, Some type of audio
`processing technique is used to time-compress or time
`expand audio Streams, while maintaining the original pitch
`of the audio-thereby maintaining the intelligibility of the
`audio.
`There are vario