111111
`
`1111111111111111111111111111111111111111111111111111111111111
`US009503742B2
`
`c12) United States Patent
`Gutierrez Novelo
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 9,503,742 B2
`Nov. 22, 2016
`
`(54) SYSTEM AND METHOD FOR DECODING 3D
`STEREOSCOPIC DIGITAL VIDEO
`
`(56)
`
`(75)
`
`Inventor: Manuel Rafael Gutierrez Novelo,
`Nueva Santa Maria (MX)
`
`(73) Assignee: TD Vision Corporation S.A. de C.V.,
`Col. Nueva Santa Maria (MX)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 687 days.
`
`CN
`EP
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,579,413 A
`5,612,735 A
`5,619,256 A
`5,652,616 A
`
`11/1996 Bj0ntegaard
`3/1997 Haskell et al.
`4/1997 Haskell et al.
`7/1997 Chen eta!.
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`1450816 A
`10/2003
`2/1995
`0639031
`(Continued)
`
`OTHER PUBLICATIONS
`
`(21) Appl. No.: 12/837,421
`
`(22) Filed:
`
`Jul. 15, 2010
`
`(65)
`
`Prior Publication Data
`
`US 2010/0271462 Al
`
`Oct. 28, 2010
`
`Related U.S. Application Data
`
`(63)
`
`Continuation of application No. 11/510,262, filed on
`Aug. 25, 2006, which is a continuation of application
`No. PCT/MX2004/000012, filed on Feb. 27, 2004.
`
`(51)
`
`Int. Cl.
`H04N 7112
`H04N 19142
`
`(2006.01)
`(2014.01)
`(Continued)
`
`(52)
`
`(58)
`
`U.S. Cl.
`CPC .............. H04N 19142 (2014.11); H04N 19130
`(2014.11); H04N 19144 (2014.11); H04N
`191593 (2014.11); H04N 191597 (2014.11);
`H04N 19161 (2014.11); H04N 19/70 (2014.11)
`Field of Classification Search
`CPC .................................................... H04N 19/597
`USPC ............ 375/240.01, 240.14, 240.27, 240.23;
`348/43, 46
`IPC ......................................................... H04N 7/12
`See application file for complete search history.
`
`Tseng eta!., "Compatible Video Coding of Stereoscopic Sequences
`Using MPEG-2's Scalability and Interlaced Structure," Interna(cid:173)
`tional Workshop on HDTV '94, Oct. 1994, Torino, Italy.
`(Continued)
`
`Primary Examiner- Young Lee
`(74) Attorney, Agent, or Firm- Knobbe Martens Olson
`& Bear LLP
`(57)
`ABSTRACT
`Described herein is a MPEG-2 compatible stereoscopic
`3D-video image digital decoding method and system. In
`order to obtain 3D-images from a digital video stream,
`modifications are made to the current MPEG2 decoders, by
`means of software and hardware changes in different parts of
`the decoding process. Namely, the video_sequence struc(cid:173)
`tures of the video data stream are modified via software to
`include the necessary flags at the bit level of the image type
`in the TDVision® technology. Modifications are also made
`in the decoding processes as well as in decoding the infor(cid:173)
`mation via software and hardware, wherein a double output
`buffer is activated, a parallel and difference decoding selec(cid:173)
`tor is activated, the decompression process is executed, the
`corresponding output buffer is displayed; and the decoder is
`programmed via software to simultaneously receive and
`decode two independent program streams, each with an
`TDVision® stereoscopic identifier.
`
`1 Claim, 5 Drawing Sheets
`
`PRIMARY
`IMAGE
`OUTPUT
`
`57
`
`58
`
`52
`
`IPR2018-00534
`Sony EX1001 Page 1
`
`

`

`US 9,503,742 B2
`Page 2
`
`(51)
`
`Int. Cl.
`H04N 191597
`H04N 19130
`H04N 19161
`H04N 191593
`H04N 19144
`H04N 19/70
`
`(2014.01)
`(2014.01)
`(2014.01)
`(2014.01)
`(2014.01)
`(2014.01)
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,815,601 A
`5,886,736 A
`5,963,257 A
`5,978,515 A
`6,043,838 A
`6,055,012 A
`6,072,831 A
`6,097,759 A
`6,144,701 A
`6,148,109 A
`6,151,362 A
`6,292,588 B1
`6,345,123 B1
`6,370,193 B1
`6,370,276 B2
`6,377,625 B1
`6,456,432 B1
`6,633,676 B1
`6,636,644 B1
`6,658,056 B1
`6,661,914 B2
`6,665,445 B1
`6,678,331 B1
`6,678,424 B1
`7,136,415 B2
`7,277,121 B2
`7,636,088 B2
`2002/0009137 A1
`2003/0048354 A1
`2003/0095177 A1
`2003/0190079 A1
`2003/0202592 A1
`2004/0008893 A1
`2004/0027452 A1
`2004/0101043 A1
`2004/0120396 A1
`2004/0252186 A1
`2006/0133493 A1
`2007/0041442 A1
`2010/0039499 A1
`2010/0271463 A1
`
`9/1998 Katata eta!.
`3/1999 Chen
`10/1999 Katata eta!.
`1111999 Katata eta!.
`3/2000 Chen
`4/2000 Haskell eta!.
`6/2000 Chen
`8/2000 Murakami et a!.
`1112000 Chiang eta!.
`1112000 Boon eta!.
`1112000 Wang
`9/2001 Shen eta!.
`212002 Boon
`4/2002 Lee
`4/2002 Boon
`4/2002 Kim
`9/2002 Lazzaro et a!.
`10/2003 Kleihorst et a!.
`10/2003 Itokawa
`12/2003 Duruoz eta!.
`12/2003 Dufour
`12/2003 Boon
`112004 Moutin eta!.
`112004 Ferguson
`1112006 Yun eta!.
`10/2007 Mashitani et a!.
`12/2009 Nomura eta!.
`112002 Nelson eta!.
`3/2003 Takemoto et a!.
`5/2003 Yun eta!.
`10/2003 Penain eta!.
`10/2003 Sohn et al.
`112004 Itoi et a!.
`2/2004 Yun eta!.
`5/2004 Flack eta!.
`6/2004 Yun eta!.
`12/2004 Mashitani et a!.
`6/2006 Cho et al.
`2/2007 Novelo
`2/2010 Nomura eta!.
`10/2010 Novelo
`
`FOREIGN PATENT DOCUMENTS
`
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`KR
`wo
`wo
`
`07-240943
`07-296185
`09-139957
`09-237353
`11-069346
`11-113026
`11-509998
`2000-0207 57
`2001054140
`2002-519792
`2003-319419
`1997-0060973
`wo 9817068
`wo 03/092304
`
`9/1995
`1111995
`5/1997
`9/1997
`3/1999
`4/1999
`8/1999
`1/2000
`2/2001
`7/2002
`1112003
`8/1997
`4/1998
`1112003
`
`OTHER PUBLICATIONS
`
`Harman, "Home Based 3D Entertainment-An Overview," Pro(cid:173)
`ceedings of the International Conference on Image Processing,
`2000, vol. 1, Sep. 2000, pp. 1-4.
`
`Gotoh, H. "Gotoh Hiroshige's Monthly Report," DOSN Power
`Report, Japan, Impress Corporation, Jul. 1, 2003, vol. 13, No.7, pp.
`125-126.
`Nakasu, MPEG-2 Video Encoding Schemes and Their Features,
`Academic Literature, Etc. 2007-51282-007, Jun. 1, 2010, pp. 2-6.
`Nakasu, E. "Comprehensible MPEG2 and MPEG2 image coding
`system and feature," Video a, Dec. 1, 1992, Shasin-kogyo Publisher,
`vol. 15, No. 12, pp. 34-38.
`Notice of Reasons for Rejection for Japanese Patent Application No.
`P2005-512205 dated Jun. 22, 2010 by Japanese Patent Office.
`Notice of Reasons for Rejection for Japanese Patent Application No.
`P2007-500701 dated Jun. 1, 2010 by Japanese Patent Office.
`Notice of Reasons for Rejection for Japanese Patent Application No.
`P2007-500700 dated Jun. 1, 2010 by Japanese Patent Office.
`Notice of Request for Submission of Argument for Korean Patent
`Application No. 10-2006-7020051 dated Jul. 13, 2010.
`Notice of Request for Submission of Argument for Korean Patent
`Application No. 10-2006-7020050 dated Jul. 13, 2010.
`Office Action for Canadian Patent Application No. 2,557,534 dated
`Feb. 21, 2011 by Canadian Intellectual Property Office.
`Examination Report
`for European Patent Application No.
`04715594.0 dated Oct. 18, 2010 by European Patent Office.
`Examination Report
`for European Patent Application No.
`04715595.7 dated Oct. 18, 2010 by European Patent Office.
`First Office Action
`for Chinese Patent Application No.
`200480042753.3 dated Aug. 8, 2008 by State Intellectual Property
`Office of P.R. C.
`for Chinese Patent Application No.
`Second Office Action
`200480042753.3 dated Mar. 20, 2009 by State Intellectual Property
`Office of P.R. C.
`for Chinese Patent Application No.
`Third Office Action
`200480042753.3 dated Feb. 24, 2010 by State Intellectual Property
`Office of P.R. C.
`for Chinese Patent Application No.
`First Office Action
`2004800425881 dated Jun. 13, 2008 by State Intellectual Property
`Office of P.R. C.
`for Chinese Patent Application No.
`Second Office Action
`2004800425881 dated May 8, 2009 by State Intellectual Property
`Office of P.R. C.
`for Chinese Patent Application No.
`Third Office Action
`2004800425881 dated Mar. 18, 2010 by State Intellectual Property
`Office of P.R. C.
`for Chinese Patent Application No.
`First Office Action
`201010227674.7 dated Jul. 20, 2011 by State Intellectual Property
`Office of P.R. C.
`for Chinese Patent Application No.
`First Office Action
`201010221966.X dated Jun. 23, 2011 by State Intellectual Property
`Office of P.R. C.
`Notice of Reasons for Rejection for Japanese Patent Application No.
`P2007 -500701 dated Jan. 5, 2010 by Japanese Patent Office.
`Notice of Reasons for Rejection for Japanese Patent Application No.
`P2007 -500700 dated Jan. 12, 2010 by Japanese Patent Office.
`Final Rejection for Japanese Patent Application No. P200 1-500701
`dated Dec. 21, 2010 by Japanese Patent Office.
`Final Rejection for Japanese Patent Application No. P200 1-500700
`dated Dec. 21, 2010 by Japanese Patent Office.
`Notice of Final Rejection for Korean Patent Application No.
`10-2006-7020051 dated May 24, 2011.
`Notice of Final Rejection for Korean Patent Application No.
`10-2006-7020050 dated May 23, 2011.
`Notice of Request for Submission of Argument for Korean Patent
`Application No. 10-2011-7022035 dated Dec. 15, 2011.
`Notice of Request for Submission of Argument for Korean Patent
`Application No. 10-2011-7022011 dated Dec. 14, 2011.
`Chien, et al., Efficient Stereo Video Coding System For Immersive
`Teleconference With Two-Stage Hybrid Disparity Estimation Algo(cid:173)
`rithm, DSP/IC Design Lab, Graduate Institute of Electronics Engi(cid:173)
`neering and Department of Electrical Engineering, National Taiwan
`University 1, Sec. 4, Roosevelt Rd., Taipei, Taiwan, 2003.
`Jiang, et a!., A Hybrid Scheme for Low Bit-Rate Coding of Stereo
`Images, IEEE Transactions on Image Processing, vol. 11, No. 2,
`Feb. 2002.
`
`IPR2018-00534
`Sony EX1001 Page 2
`
`

`

`US 9,503,742 B2
`Page 3
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`Naito, 3D-HDTV Digital Coding Scheme Conforming To Mpeg-2
`With Full Use Of Disparity Compensation, Electronics and Com(cid:173)
`munications in Japan, Part 1, vol. 85, No.2, 2002, Translated from
`Denshi Joho Tsushin Gakkai Ronbunshi, vol. J83-B, No. 5, May
`2000, pp. 739-747.
`Olm, Encoding and Reconstruction of Multiview Video Objects,
`IEEE Signal Processing Magazine, May 1999, vol. 16, pp. 47-54.
`Perkins, Data Compression of Steropairs, IEEE Transactions on
`communications, vol. 40, No.4, Apr. 1992.
`Puri, et al., Basics of Stereoscopic Video, New Compression Results
`with MPEG-2 and a Proposal for MPEG-4, Signal Processing:
`Image Communication 10, 1997, pp. 201-234.
`Schertz, Source Coding of Stereoscopic Television Pictures, Institut
`fur Rundfunktechnik, FXG, 1992, 462-464.
`
`Seferidis, et a!., Improved Disparity Estimation in Stereoscopic
`Television, IEEE Xplore, vol. 29, No. 9, 1993.
`Sethuraman, et a!., A Multiresolution Framework For Stereoscopic
`Image Sequence Compression, Proc. of ICIP-95, vol. II, pp. 361-
`365, IEEE Computer Society Press, 1994.
`Siegel, et a!., Compression and Interpolation of 3D-Stereoscopic
`and Multi-View Video, The Rototics Institute, Carnegie Mellon
`University, Pittsburgh, PA 15213, SAE vol. 3012, 1997, 227-238.
`Tengecharoen, et a!., Stereo Image Compression by Quadrant
`Vector Quantization, Faculty of Engineering, and Research Center
`for Communications and Information Technology (ReCCIT), King
`Mongkut's Institute of Technology Ladkrabang, Bangkok 10520,
`Thailand, 2003.
`Yan, eta!., Stereo Video Coding Based on Frame Estimation and
`Interpolation, IEEE Transactions on Broadcasting, vol. 49, No. 1,
`Mar. 2003, pp. 14-21.
`Zhu, Fast Disparity Estimation Algorithms for Stereo Video Coding
`Based on MPEG-2, Bulletin of Science and technology, vol. 19, No.
`1, Jan. 31, 2003.
`
`IPR2018-00534
`Sony EX1001 Page 3
`
`

`

`U.S. Patent
`
`Nov. 22, 2016
`
`Sheet 1 of 5
`
`US 9,503,742 B2
`
`30
`
`SYSTEM
`
`COMPILER
`./
`
`32
`
`3
`
`FIG. 1
`
`IPR2018-00534
`Sony EX1001 Page 4
`
`

`

`U.S. Patent
`
`Nov. 22,2016
`
`Sheet 2 of 5
`
`US 9,503,742 B2
`
`MPEG 2-4 COMPATIBLE
`DECODER
`
`- 1
`
`I
`
`DECODING
`
`- -2
`
`I
`
`SOFTWARE
`ALGORITHM
`
`HARDWARE
`CHANGES
`
`L4
`
`Fig. 2
`
`IPR2018-00534
`Sony EX1001 Page 5
`
`

`

`U.S. Patent
`
`Nov. 22,2016
`
`Sheet 3 of 5
`
`US 9,503,742 B2
`
`sw
`DECODING
`PROCESS
`
`N
`0
`
`CODED ~
`DATA
`I _.
`
`0
`
`VLD
`DECODING
`
`INVERSE
`SCAN
`
`INVERSE
`QUANTIZATION
`MATRIX
`
`~
`
`I
`.....
`.....
`
`I
`.....
`N
`
`I .....
`
`v.>
`
`DECODED
`IMAGE
`
`MOTION
`COMPENSATION
`
`I .....
`
`Ol
`
`I .....
`
`()'1
`
`INVERSE
`COSINE
`TRANSFORM
`
`I
`
`Fig. 3
`
`IPR2018-00534
`Sony EX1001 Page 6
`
`

`

`U.S. Patent
`
`Nov. 22, 2016
`
`Sheet 4 of 5
`
`US 9,503,742 B2
`
`DECODING COMPILATION
`FORMAT
`
`VIDEO_SEQUENCE
`READING
`
`I SEQUENCE_HEADER
`I USER_DATA
`
`1-- 40
`
`~
`
`-4 1
`
`~42
`~ 43
`
`SEQUENCE_SCALABLE_EXTENSION -44
`
`I EXTRA_BIT_PICTURE
`
`PICTURE_CODING_EXTENSION
`
`~45
`
`1--- 46
`
`PICTURE_TEMPORAL_
`SCALABLE_EXTENSION()
`
`-47
`
`Fig. 4
`
`IPR2018-00534
`Sony EX1001 Page 7
`
`

`

`U.S. Patent
`
`Nov. 22,2016
`
`Sheet 5 of 5
`
`US 9,503,742 B2
`
`HWDECODING
`COMPILATION _so
`FORMAT ~
`
`51
`
`53
`I
`
`VIDEO_SEQUENCE
`
`IMAGE TYPE
`IDENTIFICATION
`
`T
`
`SEQUENCE HEADER
`IDENTIFICATION
`
`si
`
`NORMAL
`VIDEO
`STREAM
`
`I
`54
`
`55
`I
`
`TDVISION
`VIDEO ERROR
`CORRECTION
`
`PRIMARY
`IMAGE
`OUTPUT
`
`I
`57
`
`r-.
`
`56
`I
`
`OUTPUT
`BUFFER
`
`-
`
`5~
`SECONDARY
`IMAGE
`OUTPUT
`
`Fig. 5
`
`IPR2018-00534
`Sony EX1001 Page 8
`
`

`

`US 9,503,742 B2
`
`1
`SYSTEM AND METHOD FOR DECODING 3D
`STEREOSCOPIC DIGITAL VIDEO
`
`CROSS-REFERENCE TO RELATED
`APPLICATION
`
`This application is a continuation of copending U.S.
`application Ser. No. 11/510,262,
`titled "Stereoscopic
`3D-Video Image Digital Decoding System and Method,"
`filed Aug. 25, 2006, which is a continuation of PCT Appli(cid:173)
`cation No. PCT/MX2004/00012 filed on Feb. 27,2004 in the
`Spanish language. The disclosures of all the above-refer(cid:173)
`enced applications, publications, and patents are considered
`part of the disclosure of this application, and are incorpo- 15
`rated by reference herein in their entirety.
`
`FIELD OF THE INVENTION
`
`2
`In order to codify the images to be presented in a
`sequential manner and form video signals, each image needs
`to be divided in rows, where each line is in turn divided in
`picture elements or pixels, each pixel has two associated
`values, namely, luma and chroma. Luma represents the light
`intensity at each point, while luma represents the color as a
`function of a defined color space (RGB), which can be
`represented by three bytes.
`The images are displayed on a screen in a horizontal-
`10 vertical raster, top to bottom and left to right and so on,
`cyclically. The number oflines and frequency of the display
`can change as a function of the format, such as NTSC, PAL,
`or SECAM.
`The video signals can be digitized for storage in digital
`format, after being transmitted, received, and decoded to be
`displayed in a display device, such as a regular television set
`or the 3DVisor®, this process is known as analog-to-digital
`video signal coding-decoding.
`By definition, MPEG has two different methods for inter(cid:173)
`lacing video and audio in the system streams.
`The transport stream is used in systems with a greater
`error possibility, such as satellite systems, which are sus(cid:173)
`ceptible to interference. Each package is 188 bytes long,
`25 starting with an identification header, which makes recog(cid:173)
`nizing gaps and repairing errors possible. Various audio and
`video programs can be transmitted over the transport stream
`simultaneously on a single transport stream; due to the
`header, they can be independently and individually decoded
`30 and integrated into many programs.
`The program stream is used in systems with a lesser error
`possibility, as in DVD playing. In this case, the packages
`have a variable-length and a size substantially greater than
`the packages used in the transport stream. As a main
`35 characteristic, the program stream allows only a single
`program content.
`Even when the transport and program streams handle
`different packages, the video and audio formats are decoded
`in an identical form.
`In turn, there are three compression types, which are
`applied to the packages above, e.g. time prediction, com(cid:173)
`pression, and space compression.
`Decoding is associated to a lengthy mathematical process,
`which purpose is to decrease the information volume. The
`complete image of a full frame is divided by a unit called
`macro block, each macro block is made up of a 16 pixelsx 16
`pixels matrix, and is ordered and named top to bottom and
`left to right. Even with a matrix array on screen, the
`information sent over the information stream follows a
`special sequential sequence, i.e. the macro blocks are ordered
`in ascending order, this is, macroblockO, macroblock1, etc.
`A set of consecutive macro blocks represents a slice; there
`can be any number of macro blocks in a slice given that the
`macroblocks pertain to a single row. As with the macrob(cid:173)
`locks, the slices are numbered from left to right and bottom
`to top. The slices should cover the whole image, as this is a
`form in which MPEG2 compresses the video, a coded image
`not necessarily needs samples for each pixel. Some MPEG
`profiles require handling a rigid slice structure, by which the
`whole image should be covered.
`U.S. Pat. No. 5,963,257 granted on Oct. 5, 1999 to Katata
`et a!., protects a flat video image decoding device with
`means to separate the coded data by position areas and
`image form, bottom layer code, predictive coding top layer
`code, thus obtaining a hierarchical structure of the coded
`data; the decoder has means to separate the data coded in the
`hierarchical structure in order to obtain a high quality image.
`
`The present invention is related to stereoscopic video 20
`image display in the 3DVisor® device and, particularly, to
`a video image decoding method by means of a digital data
`compression system, which allows the storage of three(cid:173)
`dimensional information by using standardized compression
`techniques.
`
`BACKGROUND OF THE INVENTION
`
`Presently, data compression techniques are used in order
`to decrease the bits consumption in the representation of an
`image or a series of images. The standardization works were
`carried out by a group of experts of the International
`Standardization Organization. Presently, the methods are
`usually known as JPEG (Joint Photographic Expert Group),
`and MPEG (Moving Pictures Expert Group).
`A common characteristic of these techniques is that the
`image blocks are processed by means of the application of
`a transform adequate for the block, usually known as Dis(cid:173)
`crete Cosine Transform (DCT). The formed blocks are
`submitted to a quantization process, and then coded with a 40
`variable-length code.
`The variable-length code is a reversible process, which
`allows the exact reconstruction of that which has been coded
`with the variable-length code.
`The display of digital video signals includes a certain 45
`number of image frames (30 to 96 fps) displayed or repre(cid:173)
`sented successively at a 30 to 75Hz frequency. Each image
`frame is still an image formed by a pixels array, of the
`display resolution of a particular system. By example, the
`VHS system has a display resolution of 320 colunms and 50
`480 rows, the NTSC system has a display resolution of 720
`colunms and 486 rows, and the high definition television
`system (HDTV) has a display resolution of 1360 colunms
`and 1020 rows. In reference to a digitized form of low
`resolution, 320 colunms by 480 rows VHS format, a two- 55
`hour long movie could be equivalent to 100 gigabytes of
`digital video information. In comparison, a conventional
`compact optical disk has an approximate capacity of 0.6
`gigabytes, a magnetic hard disk has a 1-2 gigabyte capacity,
`and the present compact optical disks have a capacity of 8 60
`or more gigabytes.
`All images we watch at the cinema and TV screens are
`based on the principle of presenting complete images (static
`images, like photographs) at a great speed. When they are
`presented in a fast and sequential manner at a 30 frames per 65
`second speed (30 fps) we perceive them as an animated
`image due to the retention of the human eye.
`
`IPR2018-00534
`Sony EX1001 Page 9
`
`

`

`US 9,503,742 B2
`
`3
`U.S. Pat. No. 6,292,588 granted on Sep. 18, 2001 to Sherr
`eta!., protects a device and method for coding predictive flat
`images reconstructed and decoded from a small region, in
`such way that the data of the reconstructed flat image is
`generated from the sum of the small region image data and
`the optimal prediction data for said image. Said predictive
`decoding device for an image data stream includes a vari(cid:173)
`able-length code for unidimensional DCT coefficients. U.S.
`Pat. No. 6,370,276 granted on Apr. 9, 2002 to Boon, uses a
`decoding method similar to the above.
`U.S. Pat. No. 6,456,432 granted on Sep. 24, 2002 to
`Lazzaro et a!., protects a stereoscopic 3D-image display
`system, which takes images from two perspectives, displays
`them on a CRT, and multiplexes the images in a field-
`sequential manner with no flickering for both eyes of the
`observer.
`U.S. Pat. No. 6,658,056 granted on Dec. 2, 2003 to
`Duruoz et a!., protects a digital video decoder comprising a
`logical display section responding to a "proximal field"
`command to get a digital video field of designated locations
`in an output memory. The digital video display system is
`equipped with a MPEG2 video decoder. Images are decoded
`as a memory buffer, the memory buffer is optimized main(cid:173)
`taining compensation variable tables and accessing fixed 25
`memory pointer tables displayed as data fields.
`U.S. Pat. No. 6,665,445 granted on Dec. 16, 2003 to
`Boon, protects a data structure for image transmission, a flat
`images coding method and a flat images decoding method.
`The decoding method is comprised of two parts, the first part 30
`to codifY the image-form information data stream, the sec(cid:173)
`ond part is a decoding process for the pixel values of the
`image data stream, both parts can be switched of the flat
`image signal coding.
`U.S. Pat. No. 6,678,331 granted on Jan. 13, 2004 to 35
`Moutin eta!., protects a MPEG decoder, which uses a shared
`memory. Actually, the circuit includes a microprocessor, a
`MPEG decoder, which decodes a flat image sequence, and a
`common memory for the microprocessor, and the decoder. It
`also includes a circuit for evaluating the decoder delay, and 40
`a control circuit for determining the memory priority for the
`microprocessor or the decoder.
`U.S. Pat. No. 6,678,424 granted on Jan. 13, 2004 to
`Ferguson, protects a behavior model for a real-time human
`vision system; actually, it processes two image signals in 45
`two dimensions, one derived from the other, in different
`channels.
`
`4
`It is still another object of the present invention to provide
`a decoding method with the video_sequence process normal
`form, in such a way that when a TDVision® type image is
`found, the buffer of the last complete image is stored in the
`left or right channel buffers.
`It is also another object of the present invention to provide
`a decoding process in which two interdependent (difference)
`video signals can be sent within the same video_sequence,
`in which information decoding is applied and is stored as a
`10 B type frame.
`It is still another object of the present invention to provide
`a decoding process in which error correction is applied to the
`last obtained image when the movement and color correc(cid:173)
`tion vectors are applied.
`It is also an object of the present invention to program the
`decoder by software, to simultaneously receive and codifY
`two independent program streams.
`It is still another object of the present invention to provide
`a decoding system, which decodes the 3D-image informa-
`20 tion via hardware, in which a double output buffer is
`activated.
`It is another object of the present invention to provide a
`decoding system of 3D-image information, which activates
`an image-decoding selector in parallel and by differences.
`It is also another object of the present invention to provide
`a 3D-image information decoding system, which executes
`the decompression process and displays the corresponding
`output buffer.
`
`15
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 represents one embodiment of a technology map
`FIG. 2 shows a flowchart in which the steps of one
`embodiment of a process are outlined.
`FIG. 3 illustrates structures that can be modified and the
`video_sequence of the data stream in order to identifY the
`TDVision® technology image type at the bit level.
`FIG. 4 shows one embodiment of the compilation soft(cid:173)
`ware format for the TDVision® decoding method (40).
`FIG. 5 is a representation of one embodiment of the
`decoding compilation format of the hardware.
`
`DETAILED DESCRIPTION OF THE
`INVENTION
`
`BRIEF DESCRIPTION OF THE INVENTION
`
`It is an object of the present invention to provide a
`stereoscopic 3D-video image digital decoding system and
`method, comprised of changes in software and changes in
`hardware.
`It is an additional object of the present invention to
`provide a decoding method where the normal video_se(cid:173)
`quence process is applied to the coded image data, i.e.
`variable_length_decoding (VLD), inverse_scan; inverse
`quantization,
`inverse_discrete_cosine_transform (IDCT),
`and motion_compensation.
`It is also an object of the present invention to make
`changes in the software information for decoding the iden(cid:173)
`tification of the video format, 2D-images MPEG2 backward
`compatibility, discriminating a TDVision® type image, stor(cid:173)
`ing the last image buffer, applying information decoding, 65
`applying error correction and storing the results in the
`respective channel buffer.
`
`The combination of hardware and software algorithms
`makes possible the stereoscopic 3D-image information com(cid:173)
`pression, which are received as two independent video
`signals but with the same time_code, corresponding to the
`50 left and right signals coming from a 3Dvision® camera, by
`sending two simultaneous programs with stereoscopic pair
`identifiers, thus promoting the coding-decoding process.
`Also, two interdependent video signals can be handled by
`obtaining their difference, which is stored as a "B" type
`55 frame with the image type identifier. As the coding process
`was left open in order to promote the technological devel(cid:173)
`opment, it is only necessary to follow this decoding process,
`namely: apply variable-length decoding to the coded data
`where a substantial reduction is obtained, but a look-up table
`60 should be used to carry out decoding; apply an inverse scan
`process; apply an inverse quantization process in which each
`data is multiplied by a scalar; apply the inverse cosine
`transform function; apply error correction or motion com-
`pensation stage and eventually obtain the decoded image.
`The novel characteristics of this invention in connection
`with its structure and operation method will be better
`understood from the description of the accompanying fig-
`
`IPR2018-00534
`Sony EX1001 Page 10
`
`

`

`US 9,503,742 B2
`
`5
`ures, together with the attached specification, where similar
`numerals refer to similar parts and steps.
`FIG. 1 represents the technology map to which the subject
`object of the present invention pertains. It shows a stereo(cid:173)
`scopic 3D-image coding and decoding system and corre(cid:173)
`sponding method. The images come from a stereoscopic
`camera (32), the information compiled in (31) and are
`displayed in any adequate system (30) or (33). The infor(cid:173)
`mation is coded in (34) and then it can be transmitted to a
`system having an adequate previous decoding stage such as 10
`(35), which may be a cable system (36), a satellite system
`(37), a high definition television system (38) or a stereo(cid:173)
`scopic vision system such as TDVision®'s 3DVisors® (39).
`FIG. 2 shows a flowchart in which the steps of the process 15
`are outlined. The objective is to obtain three-dimensional
`images from a digital video stream by making modifications
`to the current MPEG2 decoders, and changes to software (3)
`and hardware (4) in the decoding process (2): the decoder (1)
`should be compatible with MPEG2-4.
`FIG. 3 outlines the structures that should be modified and
`the video_sequence of the data stream in order to identifY the
`TDVision® technology image type at the bit level.
`Each of the stages of the decoding process is detailed
`below (20):
`The coded data (10) are bytes with block information,
`macroblocks, fields, frames, and MPEG2 format video
`images.
`Variable_length_decoding (11) (VLC, Variable-length
`Decoder) is a compression algorithm in which the most
`frequent patterns are replaced by shorter codes and those
`occurring less frequently are replaced by longer codes. The
`compressed version of this information occupies less space
`and can be transmitted faster by networks. However, it is not
`an easily editable format and requires decompression using
`a look-up table.
`For example, the word BEETLE
`
`Therefore, the ASCII code for the word is:
`0100 0010 0110 0101 0110 0101 0111 01000 0110 1100
`0110 0101
`in VLC: 0000 0010 10 11 11 0100 00010 01 11.
`A substantial decrease is noted, however, in order to go
`back from VLC to the word 'Beetle' a search in the look-up
`table is needed to decode the bit stream, this is made by
`exact comparison of the read bits.
`Inverse scan (12): The information should be grouped by
`blocks, and by coding the information with the VLC a linear 55
`stream is obtained. The blocks are 8x8 data matrixes, so it
`is necessary to convert the linear information in a square 8x8
`matrix. This is made in a descending zigzag manner, top to
`bottom and left to right in both sequence types, depending
`on whether it is a progressive image or an interlaced image. 60
`Inverse Quantization (13): It consists simply in multiply(cid:173)
`ing each data value by a factor. When codified, most of the
`data in the blocks are quantized to remove information that
`the human eye is not able to perceive, the quantization
`allows to obtain a greater MPEG2 stream conversion, and it 65
`is also required to perform the inverse process (Inverse
`quantization) in the decoding process.
`
`6
`Inverse cosine transform (14) (IDCT, inverse discrete(cid:173)
`_cosine_transform): The data handled within each block
`pertain to the frequency domain, this inverse cosine trans(cid:173)
`form allows to return to the samples of the space domain.
`Once the data in the IDCT have been transformed, pixels,
`colors and color corrections can be obtained.
`Motion compensation (15) allows to correct some errors
`generated before the decoding stage of MPEG format,
`motion compensation takes as a reference a previous frame
`and calculates a motion vector relative to the pixels (it can
`calculate up to four vectors), and uses them to create a new
`image. This motion compensation is applied to the P and B
`type images, where the image position is located over a "t"
`time from the reference images. Additionally to the motion
`compensation, the error correction is also applied, as it is not
`enough to predict the position of a particular pixel, but a
`change in its color can also exist. Thus, the decoded image
`is obtained (16).
`To decode a P or B type image, the reference image is
`20 taken, the motion vectors are algebraically added to calcu(cid:173)
`late the next image, and finally the error correction data is
`applied, thus generating the decoded image successfully.
`Actually, in the video_sequence, two interdependent video
`signals exist, "R-L=delta, the delta difference is that stored
`25 as a B type stereoscopic pair frame with TDVision® iden(cid:173)
`tifier and which is constructed at the moment of decoding by
`differences from
`the
`image. This
`is, R-delta=L and
`L-delta=R, the left image is constructed from the difference
`with the right image, which in tum is constructed from the
`30 difference with the left image.
`The previous process is outlined in such a way that the left
`or right signal is taken, both are stored in a temporary buffer,
`then the difference between the left and right signals is
`calculated, and then it is coded as a B type image stored in
`35 the video_sequence to be later decoded by differences from
`said image.
`In the decoding process it can be deducted that the data
`inputted by the VLC stage are much smaller than the data
`outputted by the same stage.
`MPEG video sequence structure: This is the maximum
`structure used in the MPEG2 format and has the following
`format:
`Video sequence (Video_Sequence)
`Sequence header (Sequence_Header)
`Sequence extension (Sequence_Extension)
`User Data (0) and Extension (Extension_and_User_Data
`(0))
`Image group header (Group_of_Picture_Header)
`User Data (1) and Extension (Extension_and_User_Data
`50 (1))
`Image header (Picture_Header)
`Coded image extension (Picture_Coding_Extension)
`User Data (2) and Extensions (Extension_and_User_Data
`(2))
`Image Data (Picture_Data)
`Slice( Slice)
`Macroblock (Macroblock)
`Motion vectors