I 1111111111111111 11111 lllll lllll 111111111111111 1111111111 1111111111 11111111
`US006957350Bl
`
`(12) United States Patent
`Demos
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 6,957,350 Bl
`Oct. 18, 2005
`
`(54) ENCRYPTED AND WATERMARKED
`TEMPORALAND RESOLUTION LAYERING
`IN ADVANCED TELEVISION
`
`(75)
`
`Inventor: Gary A. Demos, Culver City, CA (US)
`
`(73) Assignee: Dolby Laboratories Licensing
`Corporation, San Francisco, CA (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by O days.
`
`(21) Appl. No.: 09/541,701
`
`(22) Filed:
`
`Apr. 3, 2000
`
`Related U.S. Application Data
`
`(63) Continuation-in-part of application No. 09/442,595,
`filed on Nov. 17, 1999, now abandoned, which is a
`continuation of application No. 09/217,151, filed on
`Dec. 21, 1998, now Pat. No. 5,988,863, which is a
`continuation of application No. 08/594,815, filed on
`Jan. 30, 1996, now Pat. No. 5,852,565.
`
`Int. Cl.7 .............................................. H04N 7/167
`(51)
`(52) U.S. Cl. ...................... 713/203; 380/201; 380/203;
`360/60; 369/84; 705/57
`(58) Field of Search ................................ 380/201, 203;
`360/60; 369/84; 705/57; 375/240.1, 240.11,
`375/240.25; 348/429.1, 477
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`5,742,343 A *
`6,069,914 A
`6,236,727 Bl*
`6,332,194 Bl
`2001/0028725 Al *
`
`4/1998 Haskell et al. ......... 375/240.15
`5/2000 Cox
`5/2001 Ciacelli et al.
`12/2001 Bloom et al.
`10/2001 Nakagawa .................. 382/100
`
`............. 380/212
`
`OIBER PUBLICATIONS
`
`Larry Bloomfield, Copy Protection-deja Vu, Broadcast
`Engineering, Oct. 1998, vol. 40, Iss. 11, p. 14, 2pages.*
`
`Janet Pinkerton, Dealscope Consumer Marketplace,
`Philadelphia, Jan. 1999, vol. 41, Iss. 1, p. 32, 1 pgs.*
`Demos, Gary A. "Temporal and resolution layering in
`advanced television." DemoGraFX Coporation, Nov. 27,
`1995, pp. 4-12.
`Demos, Gary A "A comparision of hierarchical high defini(cid:173)
`tion imagery coding schema." DemoGraFX Coporation,
`1992 IEEE, pp. 68-74.
`
`* cited by examiner
`
`Primary Examiner-Ayaz Sheikh
`Assistant Examiner-Taghi T. Arani
`(74) Attorney, Agent, or Firm-Fish & Richardson P.C.
`
`(57)
`
`ABSTRACT
`
`A method and apparatus for image compression using tem(cid:173)
`poral and resolution layering of compressed image frames,
`and which provides encryption and watermarking capabili(cid:173)
`ties. In particular, layered compression allows a form of
`modularized decomposition of an image that supports flex(cid:173)
`ible encryption and watermarking techniques. Using layered
`compression, the base layer and various internal components
`of the base layer can be used to encrypt a compressed
`layered movie data stream. By using such a layered subset
`of the bits, the entire picture stream can be made unrecog(cid:173)
`nizable by encrypting only a small fraction of the bits of the
`entire stream. A variety of encryption algorithms and
`strengths can be applied to various portions of the layered
`stream, including enhancement layers. Encryption algo(cid:173)
`rithms or keys can be changed at each slice boundary as
`well, to provide greater intertwining of the encryption and
`the picture stream. Watermarking tracks lost or stolen copies
`back to the source, so that the nature of the method of theft
`can be determined and so that those involved in a theft can
`be identified. Watermarking preferably uses low order bits in
`certain coefficients in certain frames of a layered compres(cid:173)
`sion movie stream to provide reliable identification while
`being invisible or nearly invisible to the eye. An enhance(cid:173)
`ment layer can also have its own unique identifying water(cid:173)
`mark structure.
`
`39 Claims, 10 Drawing Sheets
`
`,--1aoo
`
`Select unit to
`encrypt
`
`,--~~__,1306
`
`Netflix, Inc. and Hulu, LLC - Ex. 1006, Page 0001
`IPR2020-00614 (Netflix, Inc. and Hulu, LLC v. DivX, LLC)
`
`

`

`U.S. Patent
`
`Oct. 18, 2005
`
`Sheet 1 of 10
`
`US 6,957,350 Bl
`
`D
`
`D
`
`36 Frames Per Second
`
`D
`
`11 711
`
`11211
`
`DODD (cid:143)
`
`60 Frames Per Second
`
`11311
`I
`
`D
`
`3-2
`Pulldown
`
`11211
`
`D
`
`24 Frames Per Second
`
`I
`
`FIG. 1
`
`Netflix, Inc. and Hulu, LLC - Ex. 1006, Page 0002
`IPR2020-00614 (Netflix, Inc. and Hulu, LLC v. DivX, LLC)
`
`

`

`U.S. Patent
`
`Oct. 18, 2005
`
`Sheet 2 of 10
`
`US 6,957,350 Bl
`
`p
`
`B
`
`Base Temporal Layer @ 36 Hz
`p
`p
`I
`4 Frame
`~ PSpacing _.,.
`I
`
`B
`
`B
`
`p
`
`I
`
`B
`
`B
`
`B
`
`P
`
`B
`
`B
`
`B
`
`P
`
`B
`
`B
`
`B
`
`P
`
`B
`
`B
`
`B
`
`B
`
`B
`
`B
`
`Temporal Enhancement Layer, Yielding 72 Hz
`
`FIG. 2
`
`p
`
`p
`
`Base Temporal Layer @ 36 Hz
`p
`p
`p
`
`p
`
`p
`
`I
`2 Frame
`P Spacing_.,.
`
`I-
`
`I
`
`B
`
`P
`
`B
`
`P
`
`B
`
`P
`
`B
`
`P
`
`B
`
`P
`
`B
`
`P
`
`B
`
`B
`
`B
`
`B
`
`B
`
`B
`
`Temporal Enhancement Layer, Yielding 72 Hz
`
`FIG. 3
`
`Netflix, Inc. and Hulu, LLC - Ex. 1006, Page 0003
`IPR2020-00614 (Netflix, Inc. and Hulu, LLC v. DivX, LLC)
`
`

`

`U.S. Patent
`
`Oct. 18, 2005
`
`Sheet 3 of 10
`
`US 6,957,350 Bl
`
`50
`
`60Hz
`t
`36Hz
`+
`
`..-------+I 36 Hz Base Layer __
`MPEG-2 Decoder
`
`_P____.__B .............. _P_ ..... B ___ , ______ -J J ft2 - - - - - - :~ :~
`
`72 Hz
`
`1
`I . - - - - - - - ,
`I
`Temporal
`,
`Enhancement
`:
`To 72 Hz
`1
`I
`Optional
`:
`'---------------------~
`FIG. 4
`
`60 Hz Interlaced
`Cameras
`
`Other60 Hz
`Interlaced
`Sources
`(e.g. non-film
`
`video tape) @'62
`
`36Hz
`
`De-Inter/acer
`
`64
`Frame Rate
`Conversion
`
`Converter
`
`72Hz
`
`FIG. 5
`
`Netflix, Inc. and Hulu, LLC - Ex. 1006, Page 0004
`IPR2020-00614 (Netflix, Inc. and Hulu, LLC v. DivX, LLC)
`
`

`

`i,-
`~
`Q
`(It
`~
`-...,l
`(It
`\0
`O'I
`rJJ.
`e
`
`'"""' 0
`0 ....,
`~ ....
`'JJ. =(cid:173)~
`
`,i;;..
`
`~
`0
`N
`~CIO
`'"""'
`:-'"
`(')
`0
`
`~ = ......
`
`~ ......
`~
`r'1
`~
`
`I
`I
`1
`:
`I
`
`•
`
`1024x 432
`2.35: 1
`
`-
`
`---------=
`
`l
`
`540
`
`4!0
`
`....___ 960 X 480
`
`2: 1
`
`640x512--
`
`5:4
`
`1.87: 1
`960x 512
`
`FIG. 6
`
`-+
`
`-
`
`.• •
`
`4: 3 i
`
`680 x 512
`
`------1.5: 1 Stretch-------
`
`I
`l..-640x480
`I
`
`4:3
`
`I
`I
`I•
`I
`I
`
`I
`I
`I
`
`---7---f _______ ......_ ______ !
`
`---
`1024x512 =---------Sr
`
`-
`
`-----
`
`2: 1
`
`Netflix, Inc. and Hulu, LLC - Ex. 1006, Page 0005
`IPR2020-00614 (Netflix, Inc. and Hulu, LLC v. DivX, LLC)
`
`

`

`U.S. Patent
`
`Oct. 18, 2005
`
`Sheet 5 of 10
`
`US 6,957,350 Bl
`
`2 :1
`2048x 1024
`
`1.5: 1
`1536 X1024
`
`2.4: 1
`2048 X 853
`
`t
`
`I I
`I I
`I I
`LJ __
`
`MASTER References
`@2k x 1 k Level
`
`I - - - - - - - - - - -L - - - I
`I I
`I I
`1152 x 864
`: :
`ll.,_J4:3
`Safe Area
`I I
`I I
`I I
`I I
`
`+a-1-
`I
`I
`I I
`I I
`I I
`II
`II
`I I
`I I
`I I
`I I
`: :
`I I
`I I I I.++-+--+,
`11 LL
`
`1894x 1024
`1.85: 1
`
`2: 1
`536x 768
`
`1820x 1024
`1. 78: 1
`
`1280x 1024
`5:4
`
`1365x 1024
`4:3
`
`1408x 1024
`1.37: 1
`
`x4/3
`Enhancement
`Layer
`
`x2
`Enhancement
`Layer
`
`I - - - - - - - - - - - I
`t
`I I
`I I
`864 x 648
`I I
`I I
`I !+---=J 4 : 3
`I I
`Sate Area
`I I
`I I
`I I
`I I
`1 I
`I
`1 I________ -- I
`
`I
`
`11
`11
`11
`11
`11
`11
`11
`
`1024x 768
`4:3
`
`I 1-.= t -=-=-=-=1-:
`11
`1 J.--576x432
`I
`11
`
`4: 3
`I
`I
`I
`11
`: I
`Sate Area
`I
`I
`11
`__!-=--=--=--=--=--=--=- J. W-
`
`11
`
`x3/2
`Enhancement
`Layer
`
`Netflix, Inc. and Hulu, LLC - Ex. 1006, Page 0006
`IPR2020-00614 (Netflix, Inc. and Hulu, LLC v. DivX, LLC)
`
`

`

`U.S. Patent
`
`Oct. 18, 2005
`
`Sheet 6 of 10
`
`US 6,957,350 Bl
`
`80
`
`2k x 1k
`Original
`Image
`
`Filter to
`1/2 Resolution
`- - - -(cid:141)
`
`r81
`
`1k x 512
`Base Layer
`
`Compress
`
`c58_2_
`
`l MPEG-2
`l MPEG-2
`Decompress 83
`
`Send
`Base
`Layer
`
`85
`
`2k x 1 k Enlargement
`of 1k x 512
`Base Layer
`
`Top Octave
`
`Sharpness
`
`I
`Weight ------0
`
`(0.25 typ.)
`
`\
`
`Expand!
`
`rB4
`
`2k x 1 k Enlargement
`ot1kx512
`Decompressed
`
`;
`
`Enhancement
`Difference
`
`Genter
`Weighting
`
`86
`
`2kx 1k
`I
`I Enhancement Layer I
`I Source
`I
`[ ________ J
`
`MPEG-2
`_co_m_p_re_s_s __ ~1 _ _ __
`Send Resolutiom
`Enhancement
`Layer
`
`FIG. 8
`
`Netflix, Inc. and Hulu, LLC - Ex. 1006, Page 0007
`IPR2020-00614 (Netflix, Inc. and Hulu, LLC v. DivX, LLC)
`
`

`

`U.S. Patent
`
`Oct. 18, 2005
`
`Sheet 7 of 10
`
`US 6,957,350 Bl
`
`MPEG-2
`Base
`Layer ~ Decompress
`(8 mb, typ.)
`
`Base Layer
`1k X 512
`
`, '
`Expand
`
`91
`
`Expanded Base Layer
`2kx1k
`
`Enhancement
`Layer
`(10 mb, typ.)
`
`MPEG-2
`Decompress
`92
`Decoded Enhancement
`Layer
`2kx 1k
`
`93
`A1V High Resolution
`Image
`2kx 1k
`
`FIG. 9
`
`Netflix, Inc. and Hulu, LLC - Ex. 1006, Page 0008
`IPR2020-00614 (Netflix, Inc. and Hulu, LLC v. DivX, LLC)
`
`

`

`U.S. Patent
`
`Oct. 18, 2005
`
`Sheet 8 of 10
`
`US 6,957,350 Bl
`
`Base Resolution 1 1 DD
`Resolution: 1 k x 512
`Rate: 24 fps & 36 fps
`No B Frame Capability
`18 MPixels/sec.
`+ Base Resolution
`Temporal Enhancement r 102
`1k X 512
`72Hz
`
`1k X 512
`24 or36Hz
`
`Resolution: 1k x 512
`Rate: 36 B frames/sec
`18 Mpixels/sec.
`
`Base Layer
`MPEG-2
`
`( 5mb, typ.)
`I
`I
`I
`I
`4
`1
`L ______ .__ __ _,.
`
`B-Frames
`(3 mb, typ.)
`
`Enhancement
`Layer
`MPEG-2
`(6 mb typ.)
`
`B-Frames
`(4 mb typ.)
`
`..-'-
`
`1
`
`+
`Resolution Enhancement r 104
`Resolution: 2k x 1 k
`2k x 1 k
`Rate: 24 fps & 36 fps
`24 or 36 Hz
`No B Frame Capability
`~
`72 MPixels/sec.
`
`+
`_______ ........
`High Resolution •
`Temporal Enhancement
`
`Resolution: 2k x 1 k
`Rate: 36 B frames/sec
`72 MPixels/sec
`
`106
`2kx 1k
`72Hz
`
`FIG. 10
`
`Netflix, Inc. and Hulu, LLC - Ex. 1006, Page 0009
`IPR2020-00614 (Netflix, Inc. and Hulu, LLC v. DivX, LLC)
`
`

`

`U.S. Patent
`
`Oct. 18, 2005
`
`Sheet 9 of 10
`
`US 6,957,350 Bl
`
`High Influence Is Better For Encryption
`
`(more)
`I
`n
`I
`I
`u
`e
`n
`C
`e
`(less)
`
`Low Influence Is Better For Watermaking
`Time
`FIG. 11
`
`r1200
`
`1200
`I
`
`X
`
`FIG. 12A
`
`/1204
`
`/1204
`
`r1200
`/1204
`
`FIG. 128
`
`Netflix, Inc. and Hulu, LLC - Ex. 1006, Page 0010
`IPR2020-00614 (Netflix, Inc. and Hulu, LLC v. DivX, LLC)
`
`

`

`U.S. Patent
`
`Oct. 18, 2005
`
`Sheet 10 of 10
`
`US 6,957,350 Bl
`
`,-1300
`
`-----
`
`Select unit to
`encrypt
`
`,-1302
`
`r 1400
`
`Select
`encryption
`algorithm
`
`- Select unit to
`-
`watermark
`
`,
`
`,-1304
`
`,
`
`,1 402
`
`Generate
`Key(s)
`
`Select
`watermark
`technique(s)
`
`11306
`
`,-1 401
`
`'
`
`Encrypt unit
`
`Apply
`watermark
`
`FIG. 13
`
`FIG. 14
`
`Netflix, Inc. and Hulu, LLC - Ex. 1006, Page 0011
`IPR2020-00614 (Netflix, Inc. and Hulu, LLC v. DivX, LLC)
`
`

`

`US 6,957,350 Bl
`
`1
`ENCRYPTED AND WATERMARKED
`TEMPORAL AND RESOLUTION LAYERING
`IN ADVANCED TELEVISION
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`This application is a continuation-in-part application of
`and claims priority to U.S. application Ser. No. 09/442,595
`filed on Nov. 17, 1999, now abnd., which was a continuation
`of U.S. application Ser. No. 09/217,151 filed on Dec. 21,
`1998, now U.S. Pat. No. 5,988,863, which was a continu(cid:173)
`ation of U.S. application Ser. No. 08/594,815 filed Jan. 30,
`1996, (now U.S. Pat. No. 5,852,565, issued Dec. 22, 1998).
`
`TECHNICAL FIELD
`
`This invention relates to electronic communication sys(cid:173)
`tems, and more particularly to an advanced electronic tele(cid:173)
`vision system having temporal and resolution layering of
`compressed image frames, and which provides encryption
`and watermarking capabilities.
`
`BACKGROUND
`
`The United States presently uses the NTSC standard for
`television transmissions. However, proposals have been
`made to replace the NTSC standard with an Advanced
`Television standard. For example, it has been proposed that
`the U.S. adopt digital standard-definition and advanced
`television formats at rates of 24 Hz, 30 Hz, 60 Hz, and 60
`Hz interlaced. It is apparent that these rates are intended to
`continue ( and thus be compatible with) the existing NTSC
`television display rate of 60 Hz (or 59.94 Hz). It is also
`apparent that "3-2 pulldown" is intended for display on 60
`Hz displays when presenting movies, which have a temporal
`rate of 24 frames per second (fps). However, while the above
`proposal provides a menu of possible formats from which to
`select, each format only encodes and decodes a single
`resolution and frame rate. Because the display or motion
`rates of these formats are not integrally related to each other,
`conversion from one to another is difficult.
`Further, this proposal does not provide a crucial capability
`of compatibility with computer displays. These proposed
`image motion rates are based upon historical rates which 45
`date back to the early part of this century. If a "clean-slate"
`were to be made, it is unlikely that these rates would be
`chosen. In the computer industry, where displays could
`utilize any rate over the last decade, rates in the 70 to 80 Hz
`range have proven optimal, with 72 and 75 Hz being the 50
`most common rates. Unfortunately, the proposed rates of 30
`and 60 Hz lack useful interoperability with 72 or 75 Hz,
`resulting in degraded temporal performance.
`In addition, it is being suggested by some in the field that
`frame interlace is required, due to a claimed need to have 55
`about 1000 lines of resolution at high frame rates, but based
`upon the notion that such images cannot be compressed
`within the available 18-19 mbits/second of a conventional 6
`MHz broadcast television channel.
`It would be much more desirable if a single signal format
`were to be adopted, containing within it all of the desired
`standard and high definition resolutions. However, to do so
`within the bandwidth constraints of a conventional 6 MHz
`broadcast television channel requires compression ( or "scal(cid:173)
`ability") of both frame rate (temporal) and resolution (spa- 65
`tial). One method specifically intended to provide for such
`scalability is the MPEG-2 standard. Unfortunately, the tern-
`
`2
`poral and spatial scalability features specified within the
`MPEG-2 standard are not sufficiently efficient to accommo(cid:173)
`date the needs of advanced television for the U.S. Thus, the
`proposal for advanced television for the U.S. is based upon
`the premise that temporal (frame rate) and spatial (resolu(cid:173)
`tion) layering are inefficient, and therefore discrete formats
`are necessary.
`In addition to the above issues, the inventor has identified
`a need to protect and manage the use of valuable copyrighted
`10 audio and video media such as digital movies. The viability
`of entire technologies for movie data delivery can hinge on
`the ability to protect and manage usage. As the quality of
`digital compressed movie masters approaches that of the
`original work, the need for protection and management
`15 methodologies becomes a crucial requirement.
`In approaching a system architecture for digital content
`protection and management, it would be very beneficial to
`have a variety of tools and techniques which can be applied
`in a modular and flexible way. Most commercial encryption
`20 systems have been eventually compromised. It is therefore
`necessary to architect any protection system to be suffi(cid:173)
`ciently flexible as to adapt and strengthen itself if and when
`it is compromised. It is also valuable to place informational
`clues into each copy via watermarking of symbols and/or
`25 serial number information in order to pinpoint the source
`and method by which the security has been compromised.
`Movie distribution digitally to movie theaters is becoming
`feasible. The high value copies of new movies have long
`been a target for theft or copying of today's film prints.
`30 Digital media such as DVD have attempted crude encryption
`and authorization schemes (such as DIVX). Analog cable
`scramblers have been in use from the beginning to enable
`charging for premium cable channels and pay-per-view
`events and movies. However these crude scramblers have
`35 been broadly compromised.
`One reason that digital and analog video systems have
`tolerated such poor security systems is that the value of the
`secondary video release and the loss due to pirating is a
`relatively small proportion of the market. However, for
`40 digital first-run movies, for valuable live events, and for high
`resolution images to the home and business (via forms of
`HDTV), robust security systems become a requirement.
`The present invention overcomes these and other prob(cid:173)
`lems of current digital content protection systems.
`
`SUMMARY
`
`The present invention provides a method and apparatus
`for image compression which demonstrably achieves better
`than 1000-line resolution image compression at high frame
`rates with high quality. It also achieves both temporal and
`resolution scalability at this resolution at high frame rates
`within the available bandwidth of a conventional television
`broadcast channel. The
`inventive
`technique efficiently
`achieves over twice the compression ratio being proposed
`for advanced television while providing for flexible encryp-
`tion and watermarking techniques.
`Image material is preferably captured at an initial or
`primary framing rate of 72 fps. An MPEG-2 data stream is
`60 then generated, comprising:
`(1) a base layer, preferably encoded using only MPEG-2 P
`frames, comprising a low resolution ( e.g., 1024x512
`pixels), low frame rate (24 or 36 Hz) bitstream;
`(2) an optional base resolution temporal enhancement layer,
`encoded using only MPEG-2 B frames, comprising a low
`resolution (e.g., 1024x512 pixels), high frame rate (72
`Hz) bitstream;
`
`Netflix, Inc. and Hulu, LLC - Ex. 1006, Page 0012
`IPR2020-00614 (Netflix, Inc. and Hulu, LLC v. DivX, LLC)
`
`

`

`US 6,957,350 Bl
`
`4
`FIG. 3 is a second preferred MPEG-2 coding pattern.
`FIG. 4 is a block diagram showing temporal layer decod(cid:173)
`ing in accordance with the preferred embodiment of the
`present invention.
`FIG. 5 is a block diagram showing 60 Hz interlaced input
`to a converter that can output both 36 Hz and 72 Hz frames.
`FIG. 6 is a diagram showing a "master template" for a
`base MPEG-2 layer at 24 or 36 Hz.
`FIG. 7 is a diagram showing enhancement of a base
`10 resolution template using hierarchical resolution scalability
`utilizing MPEG-2.
`FIG. 8 is a diagram showing the preferred layered reso(cid:173)
`lution encoding process.
`FIG. 9 is a diagram showing the preferred layered reso-
`15 lution decoding process.
`FIG. 10 is a block diagram showing a combination of
`resolution and temporal scalable options for a decoder in
`accordance with the present invention.
`FIG. 11 is a diagram showing the scope of encryption and
`20 watermarking as a function of unit dependency.
`FIGS. 12A and 12B show diagrams of image frames with
`different types of watermarks.
`FIG. 13 is a flowchart showing one method of applying
`the encryption techniques of the invention.
`FIG. 14 is a flowchart showing one method of applying
`the watermarking techniques of the invention.
`Like reference symbols in the various drawings indicate
`like elements.
`
`3
`(3) an optional base temporal high resolution enhancement
`layer, preferably encoded using only MPEG-2 P frames,
`comprising a high resolution (e.g., 2kxlk pixels), low
`frame rate (24 or 36 Hz) bitstream;
`( 4) an optional high resolution temporal enhancement layer,
`encoded using only MPEG-2 B frames, comprising a high
`resolution (e.g., 2kxlk pixels), high frame rate (72 Hz)
`bitstream.
`The invention provides a number of key technical
`attributes, allowing substantial improvement over current
`proposals, and including: replacement of numerous resolu(cid:173)
`tions and frame rates with a single layered resolution and
`frame rate; no need for interlace in order to achieve better
`than 1000-lines of resolution for 2 megapixel images at high
`frame rates (72 Hz) within a 6 MHz television channel;
`compatibility with computer displays through use of a
`primary framing rate of 72 fps; and greater robustness than
`the current unlayered format proposal for advanced televi(cid:173)
`sion, since all available bits may be allocated to a lower
`resolution base layer when "stressful" image material is
`encountered.
`The disclosed layered compression technology allows a
`form of modularized decomposition of an image. This
`modularity has additional benefits beyond allowing scalable
`decoding and better stress resilience. The modularity can be 25
`further tapped as a structure which supports flexible encryp(cid:173)
`tion and watermarking techniques. The function of encryp(cid:173)
`tion is to restrict viewing, performance, copying, or other
`use of audio/video shows unless one or more proper keys are
`applied to an authorized decryption system. The function of 30
`watermarking is to track lost or stolen copies back to a
`source, so that the nature of the method of theft can be
`determined to improve the security of the system, and so that
`those involved in the theft can be identified.
`Using layered compression, the base layer, and various 35
`internal components of the base layer (such as I frames and
`their DC coefficients, or motion vectors for P frames) can be
`used to encrypt a compressed layered movie stream. By
`using such a layered subset of the bits, the entire picture
`stream can be made unrecognizable (unless decrypted) by 40
`encrypting only a small fraction of the bits of the entire
`picture stream. Further, a variety of encryption algorithms
`and strengths can be applied to various portions of the
`layered stream, including the enhancement layers (which
`can be seen as a premium quality service, and encrypted 45
`specially). Encryption algorithms or keys can be changed at
`each slice boundary as well, to provide greater intertwining
`of the encryption and the image stream.
`The inventive layered compression structure can also be
`used for watermarking. The goal of watermarking is to be 50
`reliably identifiable to detection, yet be essentially invisible
`to the eye. For example, low order bits in DC coefficients in
`I frames would be invisible to the eye, but yet could be used
`to uniquely identify a particular picture stream with a
`watermark. Enhancement layers can also have their own 55
`unique identifying watermark structure.
`The details of one or more embodiments of the invention
`are set forth in the accompanying drawings and the descrip(cid:173)
`tion below. Other features, objects, and advantages of the
`invention will be apparent from the description and draw- 60
`ings, and from the claims.
`
`DESCRIPTION OF DRAWINGS
`
`FIG. 1 is a timing diagram showing the pulldown rates for 65
`24 fps and 36 fps material to be displayed at 60 Hz.
`FIG. 2 is a first preferred MPEG-2 coding pattern.
`
`DETAILED DESCRIPTION
`
`Throughout this description, the preferred embodiment
`and examples shown should be considered as exemplars,
`rather than as limitations on the present invention.
`
`Temporal and Resolution Layering
`
`Goals of a Temporal Rate Family
`After considering the problems of the prior art, and in
`pursuing the present invention, the following goals were
`defined for specifying the temporal characteristics of a
`future digital television system:
`Optimal presentation of the high resolution legacy of 24
`frame-per-second films.
`Smooth motion capture for rapidly moving image types,
`such as sports.
`Smooth motion presentation of sports and similar images
`on existing analog NTSC displays, as well as computer(cid:173)
`compatible displays operating at 72 or 75 Hz.
`Reasonable but more efficient motion capture of less(cid:173)
`rapidly-moving images, such as news and live drama.
`Reasonable presentation of all new digital types of images
`through a converter box onto existing NTSC displays.
`High quality presentation of all new digital types of
`images on computer-compatible displays.
`If 60 Hz digital standard or high resolution displays come
`into the market, reasonable or high quality presentation
`on these displays as well.
`Since 60 Hz and 72/75 Hz displays are fundamentally
`incompatible at any rate other than the movie rate of 24 Hz,
`the best situation would be if either 72/75 or 60 were
`eliminated as a display rate. Since 72 or 75 Hz is a required
`rate for N.1.1. (National Information Infrastructure) and
`computer applications, elimination of the 60 Hz rate as being
`fundamentally obsolete would be the most future-looking.
`However, there are many competing interests within the
`
`Netflix, Inc. and Hulu, LLC - Ex. 1006, Page 0013
`IPR2020-00614 (Netflix, Inc. and Hulu, LLC v. DivX, LLC)
`
`

`

`US 6,957,350 Bl
`
`5
`broadcasting and television equipment industries, and there
`is a strong demand that any new digital television infra(cid:173)
`structure be based on 60 Hz (and 30 Hz). This has lead to
`much heated debate between the television, broadcast, and
`computer industries.
`Further, the insistence by some interests in the broadcast
`and television industries on interlaced 60 Hz formats further
`widens the gap with computer display requirements. Since
`non-interlaced display is required for computer-like appli(cid:173)
`cations of digital television systems, a de-interlacer is 10
`required when interlaced signals are displayed. There is
`substantial debate about the cost and quality of de-interlac(cid:173)
`ers, since they would be needed in every such receiving
`device. Frame rate conversion, in addition to de-interlacing,
`further impacts cost and quality. For example, that NTSC 15
`to-from PAL converters continue to be very costly and yet
`conversion performance is not dependable for many com(cid:173)
`mon types of scenes. Since the issue of interlace is a
`complex and problematic subject, and in order to attempt to
`address the problems and issue of temporal rate, the inven- 20
`tion is described in the context of a digital television
`standard without interlace.
`
`6
`dropping frames or clipping the starts and ends of scenes to
`fit within a given broadcast scheduled. These operations can
`make the 3-2 pulldown process impossible to reverse, since
`there is both 59.94 Hz and 24 Hz motion. This can make the
`film very difficult to compress using the MPEG-2 standard.
`Fortunately, this problem is limited to existing NTSC(cid:173)
`resolution material, since there is no significant library of
`higher resolution digital film using 3-2 pulldown.
`Motion Blur. In order to further explore the issue of
`finding a common temporal rate higher than 24 Hz, it is
`useful to mention motion blur in the capture of moving
`images. Camera sensors and motion picture film are open to
`sensing a moving image for a portion of the duration of each
`frame. On motion picture cameras and many video cameras,
`the duration of this exposure is adjustable. Film cameras
`require a period of time to advance the film, and are usually
`limited to being open only about 210 out of 360 degrees, or
`a 58% duty cycle. On video cameras having CCD sensors,
`some portion of the frame time is often required to "read"
`the image from the sensor. This can vary from 10% to 50%
`of the frame time. In some sensors, an electronic shutter
`must be used to blank the light during this readout time.
`Thus, the "duty cycle" of CCD sensors usually varies from
`50 to 90%, and is adjustable in some cameras. The light
`25 shutter can sometimes be adjusted to further reduce the duty
`cycle, if desired. However, for both film and video, the most
`common sensor duty cycle duration is 50%.
`Preferred Rate. With this issue in mind, one can consider
`the use of only some of the frames from an image sequence
`captured at 60, 72, or 75 Hz. Utilizing one frame in two,
`three, four, etc., the subrates shown in TABLE 1 can be
`derived.
`
`Selecting Optimal Temporal Rates
`Beat Problems. Optimal presentation on a 72 or 75 Hz
`display will occur if a camera or simulated image is created
`having a motion rate equal to the display rate (72 or 75 Hz,
`respectively), and vice versa. Similarly, optimal motion
`fidelity on a 60 Hz display will result from a 60 Hz camera
`or simulated image. Use of 72 Hz or 75 Hz generation rates 30
`with 60 Hz displays results in a 12 Hz or 15 Hz beat
`frequency, respectively. This beat can be removed through
`motion analysis, but motion analysis is expensive and inex(cid:173)
`act, often leading to visible artifacts and temporal aliasing.
`In the absence of motion analysis, the beat frequency
`dominates the perceived display rate, making the 12 or 15
`Hz beat appear to provide less accurate motion than even 24
`Hz. Thus, 24 Hz forms a natural temporal common denomi(cid:173)
`nator between 60 and 72 Hz. Although 75 Hz has a slightly
`higher 15 Hz beat with 60 Hz, its motion is still not as
`smooth as 24 Hz, and there is no integral relationship
`between 75 Hz and 24 Hz unless the 24 Hz rate is increased
`to 25 Hz. (In European 50 Hz countries, movies are often
`played 4% fast at 25 Hz; this can be done to make film
`presentable on 75 Hz displays.)
`In the absence of motion analysis at each receiving
`device, 60 Hz motion on 72 or 75 Hz displays, and 75 or 72
`Hz motion on 60 Hz displays, will be less smooth than 24
`Hz images. Thus, neither 72/75 Hz nor 60 Hz motion is
`suitable for reaching a heterogeneous display population 50
`containing both 72 or 75 Hz and 60 Hz displays.
`3-2 Pulldown. A further complication in selecting an
`optimal frame rate occurs due to the use of "3-2 pulldown"
`combined with video effects during the telecine (film-to(cid:173)
`video) conversion process. During such conversions, the 3-2 55
`pulldown pattern repeats a first frame ( or field) 3 times, then
`the next frame 2 times, then the next frame 3 times, then the
`next frame 2 times, etc. This is how 24 fps film is presented
`on television at 60 Hz (actually, 59.94 Hz for NTSC color).
`That is, each of 12 pairs of 2 frames in one second of film 60
`is displayed 5 times, giving 60 images per second. The 3-2
`pulldown pattern is shown in FIG. 1.
`By some estimates, more than half of all film on video has
`substantial portions where adjustments have been made at
`the 59.94 Hz video field rate to the 24 fps film. Such 65
`adjustments include "pan-and-scan", color correction, and
`title scrolling. Further, many films are time-adjusted by
`
`TABLE 1
`
`Rate
`
`1/2 Rate
`
`1/3 Rate
`
`1/4 Rate
`
`1/5 Rate
`
`1/6 Rate
`
`75 Hz
`72 Hz
`60 Hz
`
`37.5
`36
`30
`
`25
`24
`20
`
`18.25
`18
`15
`
`15
`14.4
`12
`
`12.5
`12
`10
`
`35
`
`40
`
`The rate of 15 Hz is a unifying rate between 60 and 75 Hz.
`The rate of 12 Hz is a unifying rate between 60 and 72 Hz.
`However, the desire for a rate above 24 Hz eliminates these
`45 rates. 24 Hz is not common, but the use of 3-2 pulldown has
`come to be accepted by the industry for presentation on 60
`Hz displays. The only candidate rates are therefore 30, 36,
`and 37.5 Hz. Since 30 Hz has a 7.5 Hz beat with 75 Hz, and
`a 6 Hz beat with 72 Hz, it is not useful as a candidate.
`The motion rates of 36 and 37.5 Hz become prime
`candidates for smoother motion than 24 Hz material when
`presented on 60 and 72/75 Hz displays. Both of these rates
`are about 50% faster and smoother than 24 Hz. The rate of
`37.5 Hz is not suitable for use with either 60 or 72 Hz, so it
`must be eliminated, leaving only 36 Hz as having the desired
`temporal rate characteristics. (The motion rate of 37.5 Hz
`could be used if the 60 Hz display rate for television can be
`move 4% to 62.5 Hz. Given the interests behind 60 Hz, 62.5
`Hz appears unlikely-there are even those who propose the
`very obsolete 59.94 Hz rate for new television systems.
`However, if such a change were to be made, the other
`aspects of the present invention could be applied to the 37.5
`Hz rate.)
`The rates of 24, 36, 60, and 72 Hz are left as candidates
`for a temporal rate family. The rates of 72 and 60 Hz cannot
`be used for a distribution rate, since motion is less smooth
`when converting between these two rates than if 24 Hz is
`
`Netflix, Inc. and Hulu, LLC - Ex. 1006, Page 0014
`IPR2020-00614 (Netflix, Inc. and Hulu, LLC v. DivX, LLC)
`
`

`

`US 6,957,350 Bl
`
`7
`used as the distribution rate, as described above. By hypoth(cid:173)
`esis, we are looking for a rate faster than 24 Hz. Therefore,
`36 Hz is the prime candidate for a master, unifying motion
`capture and image distribution rate for use with 60 and 72/75
`Hz displays.
`As noted above, the 3-2 pulldown pattern for 24 Hz
`material repeats a first frame ( or field) 3 times, then the next
`frame 2 times, then the next frame 3 times, then the next
`frame 2 times, etc. When using 36 Hz, each pattern opti(cid:173)
`mally should be repeated in a 2-1-2 pattern. This can be seen 10
`in TABLE 2 and graphically in FIG. 1.
`
`8
`24 Hz on 60 Hz and 72 Hz displays. However, if the duty
`cycle is increased to be in the 75-90% range, then the 36 Hz
`samples would begin to approach the more common 50%
`duty cycle. Increasing the duty rate may be accomplished,
`for example, by using "backing store" CCD designs which
`have a short blanking time, yielding a high duty cycle. Other
`methods may be used, including dual CCD multiplexed
`designs.
`
`Modified MPEG-2 Compression
`For efficient storage and distribution, digital source mate(cid:173)
`rial havi