Ulllted States Patent [19]
`Levi et al.
`
`US006041345A
`[11] Patent Number:
`[45] Date of Patent:
`
`6,041,345
`Mar. 21, 2000
`
`[54] ACTIVE STREAM FORMAT FOR HOLDING
`MULTIPLE MEDIA STREAMS
`
`07245600 9/1995 Japan ............................. .. H04L 1/00
`OTHER PUBLICATIONS
`
`[75] Inventors: Steven P. Levi, Redmond; ‘Mark D.
`ganAllitll‘éelgik segngilg'gllg; M-
`"We >
`6 men ;
`a
`'
`Knowltom Benevue> an of Wash'
`_
`_
`_
`_
`[73] Asslgnee' sglcgosoft Corporatlon’ Redmond’
`as ~
`
`[21] Appl- N05 08/813,151
`
`Huang, J__H_, et a1” “MHTP_A Multimedia High_speed
`Transport Protocol”, Proceedings, IEEE GLOBECOM ’92,
`vol. 3, p. 1364—1368 (Dec. 6—9, 1992).
`LaPorta, T.F., et al., “The MultiStream Protocol: A Highly
`Flexible High—Speed Transport Protocol”, IEEE Journal on
`Selected Areas in Communications, 11, p. 519—530 (May
`1993).
`Sarginson, P.A., “MPEG—2: A Tutorial Introduction to the
`Systems Layer”, IEE Colloquim on MPEG—What It Is and
`
`[60]
`
`Mar- 7’
`Related US Application Data
`Provisional application NO_ 60/013,029’ Man 8’ 1996’ and
`provisional application No. 60/028,789, Oct. 21, 1996.
`7
`
`...........................................
`
`What It ISI’L’I, SchatZmayr, R., et al., “Providing Support for Data Transfer 1,
`
`
`in a NeW Networking Environment”, Multimedia Transport
`and Teleservices. International Cost 237 Works Proceed
`ings, Vienna, Austria, p. 241—255, (NOV. 13, 1994).
`Ohta, N., Packet Video: Modeling and Signal Processing,
`NOrWOOd, MA: Artech House, Inc‘, 144453, (1994)‘
`
`[58] Field of Search ................... .. 348/38, 441; 370/232,
`370/17, 94.3, 94.1, 312, 474, 329; 375/46;
`371/374, 40; 369/275.3; 386/111; 395/325;
`709/217; 380/14; 711/202; 455/4.2, 34.1
`
`Primary Examiner—Frank J. Asta
`Assistant Examiner—Thong Vu
`Attorney, Agent,
`or Firm—SchWegman, Lundberg,
`Woessner & Kluth, PA.
`
`[56]
`
`References Cited
`
`[57]
`
`ABSTRACT
`
`U_S_ PATENT DOCUMENTS
`_
`_
`lvaslleskl et {11' """"""""""" "
`I?
`5’506’847 41996 s?fzgigkit a '
`“gm/9g 3
`5:559:88 9/1996 Shimizu
`371/374
`576047843
`2/1997 Shaw et aL
`395/101
`5,625,877
`4/1997 Dunn et a1_ _____ __
`__ 455/34_1
`5,654,962
`8/1997 Rostoker et a1, _
`370/232
`5,668,803
`9/1997 Tymes et al.
`370/312
`5,671,226
`9/ 1997 Murakani et al
`370/474
`5,708,961
`1/1998 Hylton 'et a1. .......................... .. 455/4.2
`27;;
`glhlft‘iml
`"""""""""""""""" "
`370629
`577747461
`6/1998 Hyden et a1‘
`370629
`577747481
`6/1998 Hyden et a1‘ _
`5,842,224 11/1998 Fenner .................................. .. 711/202
`5,911,776
`6/1999 Guck ..................................... .. 709/217
`
`,
`
`,
`
`a1 ra e a .
`
`.
`
`FOREIGN PATENT DOCUMENTS
`
`An active stream format is de?ned and adopted for a logical
`structure that encapsulates multiple data streams. The data
`streams may be of different media. The data of the data
`streams is partitioned into packets that are suitable for
`transmission over a'transport medium. The packets may
`include error correcting information. The packets may also
`include clock licenses for d1ctat1ng the advancement of a
`clock When the data streams are rendered. The format of
`ASF facilitates ?exibility and choice of packet siZe and in
`specifying maximum bit rate at Which data may be rendered.
`Error Concealment Strategies may be employed in the pack
`etiZation of data to distribute portions of samples to multiple
`packets. Propelrqty inforntllation 'may be replicatedrlatrlldfstored
`in separate pac et's to en 'ance 1ts error to erance.
`e ormat
`faPlhFates dynaml? de?nmon of medla types and the Pack‘
`@HZaHOH of data In Such dynamlcally de?ned data types
`Within the format.
`
`.
`
`.
`
`.
`
`0 753 954 A2 1/1997 European Pat. Off. ...... .. H04L 29/06
`
`21 Claims, 24 Drawing Sheets
`
`J/ 70
`
`Jr 72
`
`Source
`
`Destination
`
`76
`
`
`
`Storage ASF
`
`K 77
`
`I’
`
`Renderers
`
`Netflix 1032 - Page 1
`
`

`
`U.S. Patent
`
`Mar. 21,2000
`
`Sheet 1 0f 24
`
`6,041,345
`
`sosncumu?
`
`830m
`
`/
`
`~ .ME
`
`r
`j A
`
`/
`
`Netflix 1032 - Page 2
`
`

`
`U.S. Patent
`
`Mar. 21,2000
`
`Sheet 2 0f 24
`
`6,041,345
`
`/ 20
`
`Build ASF stream
`
`d
`
`Transmit ASF stream
`
`l
`
`d
`
`Render media streams
`in ASP stream
`
`Fig. 2
`
`Netflix 1032 - Page 3
`
`

`
`U.S. Patent
`
`Mar. 21, 2000
`
`Sheet 3 0f 24
`
`6,041,345
`
`ASP 16‘ \
`
`Header Section
`
`4/‘ 28
`
`header__object
`
`_/”‘ 3-?
`
`properties_object _/" 34
`
`stream _properties_object__ /" 35
`
`content_description_objecL/" 38
`
`marker_object _/" 40
`
`error__correction_object _/" 42
`clock_object
`__/"44
`scn'pt_command_object _/" 4-5
`
`codec_object
`
`_/" 45
`
`Data Section
`
`J’- 30
`
`data_object
`
`__/" 47
`
`packets
`
`__/" 43
`
`index__0bj ect
`
`J’- 49
`
`Fig. 3
`
`Netflix 1032 - Page 4
`
`

`
`U.S. Patent
`
`Mar. 21,2000
`
`Sheet 4 0f 24
`
`6,041,345
`
`header_0bj ect 32 \
`object_id
`__/ 50
`
`size
`__/ 52
`number_headers __/ 54
`alignment
`__/ 55
`
`architecture
`
`__/ 57
`
`properties_object 34 \
`0bject_id
`__/ 56
`
`size
`
`__/ 58
`
`multimedia_stream_id _/ 50
`
`total_size
`
`__/ 62
`
`created
`
`___/ 64
`
`num_packets __/ 6‘5
`
`play_duration __/ 66
`
`send_duration
`
`___/ 67
`
`preroll
`?ags
`
`__/ 68
`_/ 70
`
`min __packet_size __/ 72
`
`max _packet_size __/ 74
`
`maximum_bit_rate
`
`F ig. 4
`
`F ig. 5
`
`Netflix 1032 - Page 5
`
`

`
`U.S. Patent
`
`Mar. 21, 2000
`
`Sheet 5 0f 24
`
`6,041,345
`
`(
`
`Begin
`
`,
`
`V
`
`Identify size of
`smallest packet in
`data section
`
`78
`“f
`
`/' 80
`
`Store size of
`smallest packet in
`min _packet_size
`
`l
`
`Identify size of
`largest packet in data
`section
`
`82
`“f
`
`84
`“f
`
`Store size of
`largest packet in
`max_packet_size
`
`Return
`
`Fig. 6A
`
`Netflix 1032 - Page 6
`
`

`
`U.S. Patent
`
`Mar. 21,2000
`
`Sheet 6 of 24
`
`6,041,345
`
`28
`
`30
`
`48
`
`48
`
`48
`
`48
`
`48
`
`48
`
`48
`
`48
`
`ASF1 83
`
`Header
`Section
`
`
`
`
`ASF2 85
`
`Section
`
`
`
`Fig. 6B
`
`Netflix 1032 - Page 7
`
`Netflix 1032 - Page 7
`
`

`
`U.S. Patent
`
`Mar. 21, 2000
`
`Sheet 7 of
`24
`
`6,041,345
`
`stream _properties__obj ect
`
`36 \ /_ 86.
`object_id
`-
`f 88
`a
`/ .90
`stream
`e
`/_ g2
`JYP
`error_concealment_strateg§'
`/' .94
`a
`f .96
`
`size
`
`offset
`
`type_speci?c_len
`
`stream_number
`
`/ .98
`error_,concealment_len _
`/- 100
`d
`/- 102
`type_speci?c_data entries‘
`/- 104
`error_concealment__data "'
`entnes
`
`Fig. 7
`
`Netflix 1032 - Page 8
`
`

`
`U.S. Patent
`
`Mar. 21,2000
`
`Sheet 8 0f 24
`
`6,041,345
`
`Sample 106'
`
`Fig. 8
`
`Netflix 1032 - Page 9
`
`

`
`U.S. Patent
`
`Mar. 21,2000
`
`Sheet 9 0f 24
`
`6,041,345
`
`content__description__obj ect
`
`38 \
`object_id
`
`size
`
`__/ 7 70
`
`__/‘ 112
`
`title_len
`
`_/ 774
`
`author__len
`
`__/ 775
`
`copyright__len v/ 775
`
`description_1en
`
`__/‘ 777
`
`rating_len
`
`_/ 773
`
`title entries
`
`__/ 77-9
`
`author entries
`
`__/' 120
`
`copyright entries _/ 72 7
`
`description entn'es __/' 72?
`
`rating entries
`
`___/ 723
`
`Fig. 9
`
`Netflix 1032 - Page 10
`
`

`
`U.S. Patent
`
`Mar. 21,2000
`
`Sheet 10 0f 24
`
`6,041,345
`
`marker__obj ect 40 \
`
`object_id
`
`__/' 726'
`
`size
`
`__/' 725'
`
`marker_id
`
`_/ 730
`
`num_entries _/ 7-32
`
`entry_alignment
`
`__ /' 734
`
`name_1en
`
`__/' 735
`
`name
`
`6/‘ 738
`
`marker_data
`
`/' 740
`
`Fig. 10A
`
`Netflix 1032 - Page 11
`
`

`
`U.S. Patent
`
`Mar. 21,2000
`
`Sheet 11 0f 24
`
`6,041,345
`
`marker entries 741 \
`
`offset
`
`time
`
`__/ 742
`
`__/ 744
`
`entry_len
`
`__/' 746
`
`entry_data
`
`__/ 748
`
`Fig. 108
`
`Netflix 1032 - Page 12
`
`

`
`U.S. Patent
`
`Mar. 21,2000
`
`Sheet 12 0f 24
`
`6,041,345
`
`error__correction_objsct
`42 \
`object__id
`
`d/ 150
`
`size
`
`/ 152
`
`error_correction_id _/ 754
`
`error_correction_data_1en_/‘ 755
`error-correction_data entrieg/ 7-55
`
`Fig. 11
`
`Netflix 1032 - Page 13
`
`

`
`U.S. Patent
`
`Mar. 21,2000
`
`Sheet 13 0f 24
`
`6,041,345
`
`Apply error correcting /' 160
`to interleave packets, _'
`including storage of
`ECCs in packets
`
`l
`
`Store information
`regarding error
`correcting
`methodology in header
`
`/' 162
`'
`
`l
`
`Destination computer / 164
`accesses error
`correcting information
`in header for playback
`
`Fig. 12
`
`Netflix 1032 - Page 14
`
`

`
`U.S. Patent
`
`Mar. 21,2000
`
`Sheet 14 0f 24
`
`6,041,345
`
`clock_object 44 \
`object_id
`__/' 755
`
`SiZ€
`
`q/
`
`packet_clock_type __/' 770
`
`packet_clock_size __/' 772
`
`clock_speci?c_1en
`
`clock_speci?c_data
`
`Fig. 13
`
`Netflix 1032 - Page 15
`
`

`
`U.S. Patent
`
`Mar. 21, 2000
`
`Sheet 15 0f 24
`
`6,041,345
`
`script_c0mmand_obj ect
`45 \
`object_id
`
`size
`
`v/ 178
`
`___/ 780
`
`command_id
`
`__/' 782
`
`num__commands __/ 784
`
`nmn_types
`
`d/ 186‘
`
`type_names
`
`__/' 788
`
`command_entry 6/- 790
`
`Fig. 14A
`
`Netflix 1032 - Page 16
`
`

`
`U.S. Patent
`
`Mar. 21,2000
`
`Sheet 16 0f 24
`
`6,041,345
`
`type_names_struc
`7.92 \
`type_name_len ’/ 7-94
`
`type_names
`
`__/ 796
`
`Fig. 148
`
`command_entry element
`7.98 \
`time
`
`v/ 200
`
`type
`
`/- 202
`
`parameter
`
`__/ 204
`
`Fig. 140
`
`Netflix 1032 - Page 17
`
`

`
`U.S. Patent
`
`Mar. 21,2000
`
`Sheet 17 0f 24
`
`6,041,345
`
`codec_object 46 \
`object_id
`__/ 206
`
`size
`
`__/' 205
`
`codec_ID
`
`__/' 270
`
`codec_em1'y_1en
`
`__/' 272
`
`c0dec_entry
`
`__/' 274
`
`Fig. 15A
`
`Netflix 1032 - Page 18
`
`

`
`U.S. Patent
`
`Mar. 21, 2000
`
`Sheet 18 0f 24
`
`6,041,345
`
`codecEntry element
`216 \
`type
`
`d/ 218
`
`name_len
`
`6/ 220
`
`name
`
`d/ 222
`
`descripti0n_len _/ 224
`
`description
`
`__/ 226
`
`cbinf0_len
`
`__/ 228
`
`cbinfo
`
`/ 230
`
`Fig. 15B
`
`Netflix 1032 - Page 19
`
`

`
`U.S. Patent
`
`Mar. 21,2000
`
`Sheet 19 0f 24
`
`6,041,345
`
`data_object 47 \
`object_id
`__/‘ 232
`
`size
`
`__/‘ 234
`
`multi__stream_id d/ 236‘
`
`num_packets
`
`__/‘ 238
`
`alignment
`
`_/ 240
`
`packet_alignment “/ 242
`
`Fig. 16
`
`Netflix 1032 - Page 20
`
`

`
`U.S. Patent
`
`Mar. 21, 2000
`
`Sheet 20 of 24
`
`6,041,345
`
`packet 48
`
`
`
`stream_flag
`
`256‘
`
`258
`
`260
`
`262
`
`264
`
`266‘
`
`268
`
`
`
`
`
`
`
`Fig. 1 7
`
`Netflix 1032 - Page 21
`
`
`
`
`
`
`
`
`244
`
`246
`
`248
`
`250
`
`2.52
`
`254
`
`
`
`
`
`
`
`Netflix 1032 - Page 21
`
`

`
`U.S. Patent
`
`Mar. 21,2000
`
`Sheet 21 of 24
`
`6,041,345
`
`
`270
`
`272
`
`274
`
`276'
`
`278
`
`
`
`im'tial_stIucture 244
`
`packet_len_type
`
`paddmg len type
`
`_tYPe
`
`multiple_pay1oads_present
`
`
`
`
`
`Fig. 18A
`
`Netflix 1032 - Page 22
`
`Netflix 1032 - Page 22
`
`

`
`U.S. Patent
`
`Mar. 21,2000
`
`Sheet 22 of 24
`
`6,041,345
`
`initial_structure 244
`
`280
`
`270
`
`error_correction_present
`
`error_correction_len_type ‘ 284
`opaque_data
`‘ 235
`error_correction_data_length ‘ 255
`
`error_correction_data_len
`
`error_correction_data
`
`opaque
`
`reserved
`
`packet_1en_type
`
`padding_1en_type
`
`sequence
`
`type
`
`multiple_payloads_present
`
`290
`
`2-92
`
`300
`
`302
`
`304
`
`272
`
`274
`
`275
`
`275
`
`Fig. 18B
`
`Netflix 1032 - Page 23
`
`Netflix 1032 - Page 23
`
`

`
`U.S. Patent
`
`Mar. 21,2000
`
`Sheet 23 of 24
`
`6,041,345
`
`
`314
`
`316'
`
`378
`
`320
`
`322
`
`323
`
`325
`
`
`
`payload_struc 266‘
`
`
`
`
`
`
`
`Fig. 19
`
`Netflix 1032 - Page 24
`
`Netflix 1032 - Page 24
`
`

`
`U.S. Patent
`
`Mar. 21, 2000
`
`Sheet 24 of 24
`
`6,041,345
`
`index_obj ect 4.9
`
`object_id
`
`
`
`
`
`324
`
`326‘
`
`328
`
`
`
`
`338A
` 340A
`
`
`
`
`
`
`
`
`
`3383
`
`338N
`
`340N
`
`Fig. 20
`
`Netflix 1032 - Page 25
`
`330
`
`332
`
`334
`
`336
`
`time_delta
`
`index_info array
`
`packet_count
`
`
`
`
`
`
`
`Netflix 1032 - Page 25
`
`

`
`6,041,345
`
`1
`ACTIVE STREAM FORMAT FOR HOLDING
`MULTIPLE MEDIA STREAMS
`
`This application claims benefit of Provisional Applica-
`tion Ser. No. 60/013,029 filed Mar. 8, 1996, and a provi-
`sional of Ser. No. 60/028,789 filed Oct. 21, 1996.
`
`TECHNICAL FIELD
`
`The present invention relates generally to data processing
`systems and more particularly to an active stream format for
`holding multiple media streams.
`
`BACKGROUND OF THE INVENTION
`
`Conventional file and/or stream formats for transmitting
`multiple data streams of varying media are limited in several
`respects. First, these formats are generally limited in the
`packet sizes that are available for encapsulating data. Such
`formats, if they specify packets, specify the packets as a
`given fixed size. Another limitation of such formats is that
`they do not facilitate the use of error correction codes. A
`further weakness of these conventional formats is that they
`do not provide flexibility in timing models for rendering the
`data encapsulated within the format. An additional limitation
`with such formats is that they are not well adapted for
`different transport mediums that have different levels of
`reliability and different transmission capabilities.
`
`SUMMARY OF THE INVENTION
`
`In accordance with a first aspect of the present invention,
`a computer system has a logical structure for encapsulating
`multiple streams of data that are partitioned into packets for
`holding samples of data from the multiple data streams. A
`method of incorporating error correction into the logical
`structure is performed on the computer system. In accor-
`dance with this method, a portion of at least one packet is
`designated for holding error correcting data. The error
`correcting data is then stored in the designated portion of the
`packet.
`In accordance with another aspect of the present
`invention, multiple streams of data are stored in packets and
`error correcting data is stored in at least some of the packets.
`The packets are encapsulated into a larger stream and
`information regarding what error correcting methods are
`employed for the packets is also stored in the packets.
`In accordance with yet another aspect of the present
`invention, samples of data from multiple data streams are
`stored in packets, and replicas of information are stored in
`at least some of the packets. A flag is set in each of the
`packets that holds replicas to indicate that the packets hold
`the replicas. The packets are encapsulated into a larger
`logical structure and transmitted to a destination.
`In accordance with a further aspect of the present
`invention, a logical structure is provided for encapsulating
`multiple streams of data where the streams of data are stored
`in packets. Clock licenses that dictate advancement of a
`clock are stored in multiple ones of the packets. The logical
`structure is transmitted from a source computer to a desti-
`nation computer. The clock is advanced at the destination
`computer as dictated by the clock license for each packet
`that holds a clock license in response to the receipt or
`processing of the packet at the destination computer.
`In accordance with an additional aspect of the present
`invention, a stream format is provided for encapsulating
`multiple streams of data. The stream format includes a field
`for specifying a packet size for holding samples of the
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`multiple streams of data. In a logical structure that adopts the
`stream format, a value is stored in the field that corresponds
`to the desired packet size. Packets of the desired size are
`stored within the logical structure and the logical structure is
`transmitted over a transport medium to the destination.
`In accordance with a further aspect of the present
`invention, a stream format is provided for encapsulating
`multiple streams of data. A field is included in a logical
`structure that adopts the stream format for holding a value
`that specifies a maximum bit rate at which the multiple
`streams may be rendered at the destination. Avalue is stored
`in the field and the logical structure is transmitted over a
`transport medium to a destination.
`In accordance with another aspect of the present
`invention, a stream format is provided for encapsulating
`multiple data streams and a new media type is dynamically
`defined. An identifier of the media type is stored in a logical
`structure that adopts the stream format and packets of the
`new media type are stored in the logical structure.
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`A preferred embodiment of the present invention will be
`described below relative to the following drawings.
`FIG. 1 is a block diagram illustrating a computer system
`that is suitable for practicing the preferred embodiment of
`the present invention.
`FIG. 2 is a flowchart illustrating use of the ASF stream in
`accordance with a preferred embodiment of the present
`invention.
`
`FIG. 3 is a block diagram illustrating the components of
`the ASF stream.
`
`FIG. 4 is a block diagram illustrating the format of the
`header_object.
`FIG. 5 is a block diagram illustrating the format of the
`properties_object.
`illustrating the steps that are
`FIG. 6A is a flowchart
`performed to fill in packet size fields within the ASF stream.
`FIG. 6B is a diagram illustrating different packet sizes and
`respective ASF streams.
`FIG. 7 is a block diagram illustrating the format of the
`streamipropertiesiobject.
`FIG. 8 is a diagram that illustrates the partitioning of a
`sample for storage in multiple packets.
`FIG. 9 is a diagram that illustrates the format of the
`content_description_object.
`FIG. 10A is a diagram illustrating the format of the
`markeriobject.
`FIG. 10B is a diagram illustrating the format of a marker
`entry.
`FIG. 11 is a diagram illustrating the format of the error_
`correctioniobject.
`FIG. 12 is flowchart illustrating the steps that are per-
`formed to utilize error correcting information in accordance
`with a preferred embodiment of the present invention.
`FIG. 13 is a diagram illustrating format of the clocl<_
`object.
`FIG. 14A is a diagram illustrating the format of the
`script_command_object.
`FIG. 14B is a diagram illustrating the format of a type_
`names_struc.
`
`FIG. 14C is a diagram illustrating the format of a
`commandientry.
`FIG. 15A is a diagram illustrating the format of the
`codec_object.
`
`Netflix 1032 - Page 26
`
`Netflix 1032 - Page 26
`
`

`
`6,041,345
`
`3
`FIG. 15B is a diagram of a CodecEntry.
`FIG. 16 is a diagram illustrating the format of the data_
`object.
`FIG. 17 illustrates the format of a packet.
`FIG. 18A illustrates a first format
`that
`structure may assume.
`FIG. 18B illustrates a second format that the initial_
`structure may assume.
`FIG. 19 illustrates the format of a payload_struc.
`FIG. 20 is a diagram illustrating the format of the index_
`object.
`
`the initial_
`
`DETAILED DESCRIPTION OF THE
`INVENTION
`
`invention
`The preferred embodiment of the present
`employs an active stream format (ASF) for holding multiple
`media streams. ASF is well suited for storage of multimedia
`streams as well as transmission of multiple media streams
`over a transport medium. ASF is constructed to encapsulate
`diverse multimedia streams and facilitates optimal interleav-
`ing of respective media streams. ASF specifies the packeti-
`zation of data and provides flexibility in choosing packet
`sizes. In addition, ASF enables the specification of a maxi-
`mum data transmission rate. As such, the packetization and
`transmission of media streams may be tailored to facilitate
`the bandwidth limitations of the system on which media
`streams are stored or transmitted.
`ASF facilitates the use of error correction and error
`
`concealment techniques on the media streams. In unreliable
`transport mediums, such error correction and error conceal-
`ment is highly beneficial. ASF is independent of media types
`and is extensible to handle newly defined media types. ASF
`supports flexible timing approaches and allows an author of
`an ASF stream to specify the synchronization of events. ASF
`supports synchronized rendering using a variety of synchro-
`nization clock types and provides index information which
`can be used as markers for lookup to provide playback
`features such as fast forward and fast reverse.
`
`FIG. 1 is a block diagram of an illustrative system for
`practicing the preferred embodiment of the present inven-
`tion. FIG. 2 is a flowchart that illustrates the steps that are
`performed in the illustrative embodiment of FIG. 1. An ASF
`stream 16 is built by an author (step 20 in FIG. 2) and stored
`on a storage 14 on a source computer 10. As will be
`described in more detail below, ASF allows the author to
`design the stream for a most efficient storage based on the
`type of source computer 10 on which it is stored. Sometime
`later, the ASF stream 16 is transferred over a transport media
`17, such as a network connection, to a destination computer
`12 (step 24 in FIG. 2). The destination computer 12 includes
`a number of renderers 18 for rendering the media types that
`are present within the ASF stream 16. For example, the ASF
`stream 16 may include audio-type data and video-type data.
`The renderers 18 at the destination 12 include an audio
`
`renderer and a video renderer. The renderers may begin
`rendering data as soon as they receive data prior to the
`complete transmission of the entire ASF stream 16 (see step
`26 in FIG. 2). The renderers need not immediately render the
`data, but rather may render the data at a later point in time.
`FIG. 3 depicts the basic logical organization of an ASF
`stream 16. It is up to the author to fill in the contents of the
`ASF stream in accordance with this format. The ASF stream
`16 is divisible into a header section 28, a data section 30 and
`an index section 49. In general, the header section is first
`transmitted from the source computer 10 to the destination
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`computer 12 so that the destination computer may process
`the information within the header section. Subsequently, the
`data section 30 is transmitted from the source computer 10
`to the destination computer 12 on a packet-by-packet basis
`and the index section 49 is transmitted. The header section
`28 includes a number of objects that describe the ASF stream
`16 in aggregate. The header section 28 includes a header_
`object 32 that identifies the beginning of the ASF header
`section 28 and specifies the number of objects contained
`within the header section. FIG. 4 depicts the format of the
`header_object 32 in more detail. The header_object 32
`includes an objectiid field 50 that holds a UUID for the
`header_object. The UUID is an identifier. The header_
`object 32 also includes a size field 52 that specifies a 64-bit
`quantity that describes the size of the header section 28 in
`bytes. The headeriobject 32 additionally includes a
`number_headers field 54 that holds a 32-bit number that
`specifies a count of the objects contained within the header
`section that follow the header_object 32. An alignment field
`55 specifies packing alignment of objects within the header
`(e.g., byte alignment or word alignment). The architecture
`field 57 identifies the computer architecture type of the data
`section 30 at the index section 49. The architecture field 57
`specifies the architecture of these sections as little endian or
`big endian.
`The headeriobj ect 32 is followed in the header section 28
`by a properties_object 34, such as depicted in FIG. 5. The
`propertiesiobject 34 describes properties about the ASF
`stream 16. As can be seen in FIG. 5, the properties_object
`34 includes an objectiid field 56 that holds a UUID and a
`size field 58 that specifies the size of the properties_object
`34. The propertiesiobject 34 also includes a multimedia,
`stream_id field 60 that contains a UUID that identifies a
`multimedia ASF stream. Atotal_size field 62 is included in
`the properties_object 34 to hold a 64-bit value that
`expresses the size of the entire ASF multimedia stream.
`The propertiesiobj ect 34 also holds a created field 64 that
`holds a timestamp that specifies when the ASF stream was
`created. A num_packet field 65 holds a 64-bit value that
`defines the number of packets in the data section 30. A
`play_duration field 66 holds a 32-bit number that specifies
`the play duration of the entire ASF stream in 100-
`nanosecond units. For example, if the ASF stream 16 holds
`a movie, the duration field 66 may hold the duration of the
`movie. The play_duration field 66 is followed by a send_
`duration field 67 that corresponds to send the ASF stream in
`100-nanosecond units. A preroll
`field 68 specifies the
`amount of time to buffer data before starting to play, and the
`flags field 70 holds 32-bits of bit flags.
`The properties_object 34 includes a min_packet_size
`field 72 and a max_packet_size field 74. These fields 72
`and 74 specify the size of the smallest and largest packets 48
`in the data section 30, respectively. These fields help to
`determine if the ASF stream 16 is playable from servers that
`are constrained by packet size. For constant bit rate streams,
`these values are set to have the same values. Amaximum_
`bit_rate field 76 holds a value that specifies the maximum
`instantaneous bit rate (in bits per second) of the ASF stream.
`FIG. 6A is a flowchart illustrating how these values are
`identified and assigned during authoring of the ASF stream
`16. First, the size of the smallest packet in the data section
`30 is identified (step 78 in FIG. 6A). The size of the smallest
`packet is stored in the min_packet_size field 72 (step 80 in
`FIG. 6A). The size of the largest packet in the data section
`30 is identified (step 82 in FIG. 6A), and the size is assigned
`to the maxipacketisize field 74 (step 84 in FIG. 6A).
`One of the beneficial features of ASF is its ability for
`facilitating different packet sizes for data of multiple media
`
`Netflix 1032 - Page 27
`
`Netflix 1032 - Page 27
`
`

`
`6,041,345
`
`5
`streams. FIG. 6B shows one example of two different
`streams 83 and 85. In stream 83, each of the packets is
`chosen to have a size of 512 bytes, whereas in stream 85
`each of the packets 48 holds 256 bytes. The decision as to
`the size of the packets may be influenced by the speed of the
`transport mechanism over which the ASF stream is to be
`transmitted, the protocol adopted by the transport medium,
`and the reliability of the transport medium.
`As mentioned above, the properties_object 34 holds a
`value in the maximumibitirate field 76 that specifies an
`instantaneous maximum bit rate in bits per second that is
`required to play the ASF stream 16. The inclusion of this
`field 76 helps to identify the requirements necessary to play
`the ASF stream 16.
`
`The header section 28 (FIG. 3) must also include at least
`one stream_properties_obj ect 36. The stream_properties_
`object 36 is associated with a particular type of media stream
`that is encapsulated within the ASF stream 16. For example,
`one of the stream_properties_objects 36 in the header
`section 28 may be associated with an audio stream, while
`another such object is associated with a video stream. FIG.
`7 depicts a format for such stream_properties_objects 36.
`Each stream_properties_object 36 includes an object_id
`field 86 for holding a UUID for the object and a size field 88
`for holding a value that specifies the size of the object in
`bytes. A stream_type field 90 holds a value that identifies
`the media type of the associated stream.
`least three
`The streamipropertiesiobject 36 holds at
`fields 92, 98 and 104 for holding information relating to
`error concealment strategies. In general, ASF facilitates the
`use of error concealment strategies that seek to reduce the
`effect of losing information regarding a given sample of
`media data. An example of an error concealment strategy is
`depicted in FIG. 8. A sample 106 is divided into four
`sections S1, S2, S3 and S4. When the sample is incorporated
`into packets in the ASF stream, the samples are distributed
`into separate packets P1, P2, P3 and P4 so that if any of the
`packets are lost, the amount of data that is lost relative to the
`sample is not as great, and techniques, such as interpolation,
`may be applied to conceal the error. Each sample has a
`number of associated properties that describe how big the
`sample is, how the sample should be presented to a viewer,
`and what the sample holds. Since the loss of the property
`information could prevent the reconstruction of the sample,
`the properties information for the entire sample is incorpo-
`rated with the portions of the sample in the packets.
`The error_concealment_strategy field 92 holds a UUID
`that
`identifies the error concealment strategy that
`is
`employed by the associated stream. The error_
`concealmentilen field 98 describes the number of bytes in
`an error concealment data block that is held in the error_
`concealmentidata entries 104. The properties associated
`with the error concealment strategy are placed in the error_
`concealment_data entries 104. The number of entries will
`vary depending upon the error concealment strategy that is
`adopted.
`The stream_properties_object 36 includes a stream_
`number field 100 that holds an alias to a stream instance. The
`
`stream_properties_obj ect 36 also includes an offset field 94
`that holds an offset value to the stream in milliseconds. This
`
`value is added to all of the timestamps of the samples in the
`associated stream to account for the offset of the stream with
`
`to the timeline of the program that renders the
`respect
`stream. Lastly,
`the stream_properties_object 36 holds a
`type_specific_len field 96 that holds a value that describes
`the number of bytes in the type_specific_data entries 102.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`The type_specific_data entries 102 hold properties values
`that are associated with the stream type.
`The header section 28 (FIG. 3) may also include a number
`of optional objects 38, 40, 42, 44, 45 and 46. These optional
`objects include a content_description_object 38 that holds
`information such as the title, author, copyright information,
`and ratings information regarding the ASF stream. This
`information may be useful and necessary in instances
`wherein the ASF stream 16 is a movie or other artistic work.
`The contentidescriptioniobject 38 includes an objectiid
`field 110 and a size field 112 like the other objects in the
`header section 28. A titleilen field 114 specifies the size in
`bytes of the title entries 119 that hold character data for the
`title of the ASF stream 16. An authorilen field 115 specifies
`the size in bytes of the author entries 120 which hold the
`characters that specify the author of the ASF stream 16. The
`copyright_len field 116 holds the value that specifies the
`length in bytes of the copyright entries 121 that hold
`copyright information regarding the ASF stream 16. The
`descriptionilen field 117 holds a value that specifies the
`length in bytes of the description entries 122. The descrip-
`tion entries 122 hold a narrative description of the ASF
`stream 16. Lastly, the rating_len field 118 specifies a size in
`bytes of the rating entries 123 that hold rating information
`(e.g., X, R, PG-13) for the ASF stream content.
`The header section 28 may include a marker_object 40.
`The marker_object 40 holds a pointer to a specific time
`within the data section 30. The marker_object enables a
`user to quickly jump forward or backward to specific data
`points (e.g., audio tracks) that are designated by markers
`held within the marker_object 40.
`FIG. 10A shows the marker_object 40 in more detail. The
`marker_obj ect 40 includes an object_id field 126 that holds
`a UUID, and a size field 128 specifies the size of the
`markeriobject in bytes. A markeriid field 130 contains a
`UUID that identifies the marker data strategy, and a num_
`entries field 132 specifies the number of marker entries in the
`marker_object 40. An entry_alignment field 134 identifies
`the byte alignment of the marker data, and a name_len field
`136 specifies how many Unicode characters are held in the
`name field 138, which holds the name of the marker_object
`40. Lastly, the markeridata field 140 holds the markers in
`a table. Each marker has an associated entry in the table.
`FIG. 10B shows the format of a marker entry 141 such as
`found in the markeridata field 140. An offset field 142 holds
`an offset in bytes from the start of packets in the data_obj ect
`47 indicating the position of the marker entry 141. A time
`field 144 specifies a time stamp for the marker entry 141. An
`entry_len field 146 specifies the size of an entry_data field
`148, which is an array holding the data for the marker entry.
`The header section 28 may also include an error_
`correctioniobject 42 for an error correction method that is
`employed in the ASF stream. Up to four error correction
`methods may be defined for the ASF stream 16 and, thus, up
`to four error_correction_objects 42 may be stored within
`the header section 28 of the ASF stream 16. FIG. 11 depicts
`the format of the erroricorrectioniobject 42.
`The error_correction_object 42 includes an object_id
`field 150 and a size field 152, like those described above for
`the other objects in the header section 28. The error_
`correction_object 42 also includes an error_correction_id
`154 that holds UUID that identifies the error correcting
`methodology associated with the object 42. The error_
`correctionidatailen field 156 specifies the length in bytes
`of the error_correction_data entries 158 that hold octets for
`error correction. The error_correction_object 42 is used by
`the destination computer 12 (FIG. 1) in playing the ASF
`stream 16.
`
`Netflix 1032 - Page 28
`
`Netflix 1032 - Page 28
`
`

`
`6,041,345
`
`7
`FIG. 12 depicts a flowchart of how error correcting may
`be applied in the preferred embodiment of the present
`invention. In particular, an error correction methodology
`such as an N+1 parity scheme, is applied to one or more
`streams within the ASF stream 16 (step 160 in FIG. 12).
`Information regarding the error correcting methodology is
`then