`Patentamt
`European
`Patent Office
`
`des brevets
`
`Office européen
`
`For official use only
`
`Request for grant of a European patent
`
`MKEY
`MKEY - EP12169160.4
`1 Application number:
`
`
`DREC
`DREC - 23 May 2012
`2 Date of receipt (Rule 35(2) EPC):
`
`RENA
`3 Date of receipt at EPO (Rule 35(4) EPC):
`
`
`
`
`4 Date of filing:
`
`5
`
`Grant of European patent, and examination of the application under
`Article 94, are hereby requested.
`
`5.1
`
`The applicant waives his right to be asked whether he wishes to
`proceed further with the application (Rule 70(2))
`
`Procedural language:
`
`Description and/or claims filed in:
`
`6
`
`Applicants or representative‘s reference
`
`Applicant 1
`
`Name:
`
`Address:
`
`10-1
`
`15-1
`
`16-1
`
`17-1
`
`17-1
`
`17-1
`
`State of residence or of principal place of business:
`
`Representative 1
`
`Name:
`
`Company:
`
`Address of place of business:
`
`Telephone:
`
`Fax:
`
`e-mail:
`
`E D e
`
`n
`
`
`en
`
`
`
`
`
`
` mail@vennershipley.co.uk
`
`
`
`JMH/51383EP2
`
`
`
`
`
`Tdvision Corporation S.A. DE C.V.
`
`Pina 201-A
`Col. Nueva
`
`DF 02800 Santa Maria
`
`Mexico
`
`
`Mexico
`
`
`
`Hewett Jonathan
`
`Venner Shipley LLP
`
`200 Aldersgate
`
`London EC1A 4HD
`
`United Kingdom
`
`(0)20 7600 4212
`
`
`(0)20 7600 4188
`
`EPO Form 1001 E
`
`Page 1 of 4
`lPR2018—00534
`
`Sony EX1004 Page 1
`
`IPR2018-00534
`Sony EX1004 Page 1
`
`
`
`Authorisation
`
`20
`
`Authorisation is attached.
`
`|nventor(s)
`
`23
`
`Inventor details filed separately
`
`24
`
`Title of invention
`
`Title of invention:
`
`25
`
`Declaration of priority (Rule 52)
`
`A declaration of priority is hereby made for the following applications
`
`E
`
`
`
`
`
`
`
`Method and System for Digital Decoding 3D
`Stereoscopic Video Images
`
`This application is a complete translation of the previous application
`
`It is not intended to file a (further) declaration of priority
`
`ED
`
`E
`
`25.2
`
`25.3
`
`26
`
`27
`
`27.1
`
`Reference to a previously filed application
`
`Divisional application
`
`Application number of earlier application:
`
`Date of Filing (Art. 80/Rule 40 EPC):
`
`Date of Examining Division‘s first communication in respect of the
`earliest application for which a communication has been issued (Rule
`36(1)(a) EPC):
`
`EP04715594.0
`
`
`27 February 2004
`
`
`18 October 2010
`
`
`
`
`
`28 Article 61(1)(b) application
`
`29
`
`Claims
`
`29.1
`
`29.2
`
`29.3
`
`30
`
`Figures
`
`Number of claims:
`
`D
`
`
`
`
`
`
`
`D as attached
`
`It is proposed that the abstract be published together with figure No.
`
`D as in the previously filed application (see Section 26.2)
`
`E The claims will be filed later
`
`
`
`
`
`
`
`EPO Form 1001 E
`
`Page 2 of 4
`|PR2018—00534
`
`Sony EX1004 Page 2
`
`IPR2018-00534
`Sony EX1004 Page 2
`
`
`
`31
`
`32
`
`33
`
`Designation of contracting states
`
`All the contracting states party to the EPC and valid for the parent application at the time of filing of this divisional application are deemed
`to be designated (see Article 76(2)).
`
`Different applicants for different contracting states
`
`Extension of the European patent
`
`This application is deemed to be a request to extend the European patent application and the European patent granted in respect of it to
`all non-contracting states to the EPC with which extension agreements are in force on the date on which the application is filed. However,
`the request is deemed withdrawn if the extension fee is not paid within the prescribed time limit.
`
`
`33.1
`
`It is currently intended to pay the extension fee(s) for the following
`states:
`
`
`
`
`
`Fl
`
`
`
`
`
`
`
`I:
`
`
`
`
`
`
`
`
`_
`_
`Not speCIerd
`
`
`
`
`
`
`
`
`28050158
`
`
`
`
`
`/Venner Shipley LLP/
`
`Biological material
`
`38
`
`Nucleotide and amino acid sequences
`
`The European patent application contains a sequence listing as part of
`the description
`
`The sequence listing is attached in computer-readable format in
`accordance with WIPO Standard ST.25
`
`The sequence listing is attached in PDF format
`
`Further indications
`
`39 Additional copies of the documents cited in the European search
`report are requested
`
`Number of additional sets of copies:
`
`40
`
`Refund of the search fee under to Article 9 of the Rules relating to
`Fees is requested
`
`Application or publication number of earlier search report:
`
`42
`
`Payment
`
`Mode of payment
`
`43 Refunds
`
`Any refunds should be made to EPO deposit account:
`
`Account holder:
`
`44-A Forms
`Details:
`System file name:
`
`
`
`Request
`as ep-request.pdf
`
`
`
`
`A-1
`
`A-2
`
`
`
`1- Designation 01' inventor
`
`1. Inventor
`
`as F1002-1.pdf
`
`EPO Form 1001 E
`
`Page 3 of 4
`|PR2018—00534
`
`Sony EX1004 Page 3
`
`IPR2018-00534
`Sony EX1004 Page 3
`
`
`
`
`
`Original file name:44-B Technical documents System file name:
`
`
`
`Specification
`20120523 Description for filing 51383ep2.PDF SPECEPO-1.PDF
`
`Description
`
`
`Specification
`20120523 Drawings for filing 51383ep2.PDF
`SPECEPO-2.PDF
`
`drawing(s)
`
`
`
`
`
`Specification
`
`20120523 Abstract for filing 51383ep2.PDF
`abstract
`
`SPECEPO-3.PDF
`
`3-1
`
`B-2
`
`B-3
`
`C-1
`
`45
`
`44-0 Other documents
`Original file name:
`System file name:
`
`
`
`
`Additional Representatives JMH.pdf
`
`OTHER-1.pdf
`
`
`
`Additional remesentatives
`
`
`
`General authorisation:
`
`
`
`
`
`46 Signature(s)
`
`Place:
`
`Date:
`
`London
`
`23 May 2012
`
`Signed by:
`
`[Jonathan Hewett/
`
`Capacity:
`
`(Representative)
`
`EPO Form 1001E
`
`Page 4 of 4
`lPR2018—00534
`
`Sony EX1004 Page 4
`
`IPR2018-00534
`Sony EX1004 Page 4
`
`
`
`1/5
`
`
`
`|PR2018—00534
`
`Sony EX1004 Page 5
`
`IPR2018-00534
`Sony EX1004 Page 5
`
`
`
`2/5
`
`
`
`MPEG 2-4 COMPATIBLE
`
`DECODER
`
`DECODING
`
`
`
`
`SOFTWARE
`ALGORITHM
`
`
`
`
`HARDWARE
`
`CHANGES
`
`|PR2018—00534
`
`Sony EX1004 Page 6
`
`IPR2018-00534
`Sony EX1004 Page 6
`
`
`
`3/5
`
`
`
` SW
`DECODING
`PROCESS
`
`
`OZ
`
`CODE
`D
`DATA
`
`VLD
`DECODING
`
`INVERSE
`SCAN
`
`
`INVERSE
`QUANTIEQION
`
`
`0L
`
`LL
`
`ZL
`
`MOTION
`COMPENSATION
`
`DECODED
`
`IMAGE
`
`9L
`
`TRANSFORM .x
`
`INVERSE
`COSINE
`
`
`
`
`.b.
`
`|PR2018—00534
`
`Sony EX1004 Page 7
`
`IPR2018-00534
`Sony EX1004 Page 7
`
`
`
`4/5
`
`
`
`DECODING COMPILATION
`FORMAT
`
`
`4O
`
`VIDEO_SEQUENCE
`READING
`
`_41
`
`42
`
`4s
`
`44
`
`45
`
`EXTRA BIT PICTURE
`
`46
`
`
`
`
`PICTURE__CODING_EXTENSION
`
`47
`
`PICTURE_TEMPORAL_
`SCALABLE__EXTENSION( )
`
`|PR2018—00534
`
`Sony EX1004 Page 8
`
`IPR2018-00534
`Sony EX1004 Page 8
`
`
`
`5/5
`
`
`
`PRIMARY
`
`HW DECODING
`
`
`
`
`NORMAL
`
`COMPILATION
`OUTPUT
`
`
`
`
`VIDEO
`
`FORMAT
`
`
`STREAM
`
`IMAGE
`
`
`
`
`
`IMAGE TYPE
`
`
`IDENTIFICATION
`OUTPUT
`VIDEO_SEQUENCE
`
`BUFFER
`
`
`
`TDVISION
`
`
`
`VIDEO ERROR
`
`
`
`SEQUENCE__HEADER
`
`CORRECTION
`IDENTIFICATION
`SECONDARY
`
`IMAGE
`
`OUTPUT
`
`
`IPR2018-00534
`
`Sony EX1004 Page 9
`
`IPR2018-00534
`Sony EX1004 Page 9
`
`
`
`ADDITIONAL REPRESENTATIVES
`
`READ, Matthew Charles
`
`JUMP, Timothy John Simon
`
`GILL, Sian Victoria
`
`GREY, Ian Michael
`
`PIOTROWICZ, Pawel Jan Andrzej
`
`DERRY, Paul Stefan
`
`ELEND, Almut Susanne
`
`COWLEY, Catherine
`
`WALASKI, Jan
`
`MAYS, Julie
`
`JOHANSSON, Anna Olivia
`
`BRUCE, Alexander
`
`HU'I'I‘ER, Anton
`
`HARRISON, Philip
`
`HEARE, Tanya
`
`JOHNSON, Stephen
`
`PATON, David
`
`BROWN, Alexander
`
`CHETTLE, John
`
`KENNEDY, Richard
`
`ANDERSON, Oliver
`
`SAMPSON, Eimear
`
`TAYLOR, David
`
`HAND LEY, Matthew
`
`RUSSELL, Tim
`
`all of Venner Shipley LLP, 20 Little Britain, London EC1A 7DH, United Kingdom
`
`|PR2018—00534
`
`Sony EX1004 Page 10
`
`IPR2018-00534
`Sony EX1004 Page 10
`
`
`
`Designation of inventor
`
`User reference:
`
`Application No:
`
`JMH/51383EP2
`
`Inventor
`
`GUTIERREZ NOVELO Manuel Rafael
`
`Pina 201-A
`Col. Nueva
`
`European patent: AS emlo er
`
`The applicant has acquired the right to the
`
`DF 02800 Santa Maria
`
`Mexico
`
`EPO Form 1002
`
`Page 1 of 1
`
`|PR2018—00534
`
`Sony EX1004 Page 11
`
`IPR2018-00534
`Sony EX1004 Page 11
`
`
`
`STEREOSCOPIC 3D-VIDEO IMAGE DIGITAL DECODING SYSTEM
`
`AND METHOD
`
`FIELD OF THE INVENTION
`
`The present
`
`invention is
`
`related to stereoscopic video
`
`image display in the 3DVisor® device and, particularly,
`
`to a video
`
`image decoding method by means of a digital data compression
`
`system, which allows the storage of three—dimensional information by
`
`using standardized compression techniques.
`
`BACKGROUND OF THE INVENTION
`
`Presently, data compression techniques are used in order
`
`to decrease the bits consumption in the representation of an image or
`
`a series of images. The standardization works were carried out by a
`
`group Of experts of the international Standardization Organization.
`
`Presently, the methods are usually known as JPEG (Joint Photographic
`
`Expert Group), and MPEG (Moving Pictures Expert Group).
`
`A common characteristic of these techniques is that the
`image blocks are processed by means of the application of a transform
`
`adequate for the block, usually known as Discrete Cosine Transform
`
`(DCT). The formed blocks are submitted to a quantization process, and
`
`then coded with a variable-length code.
`
`The variable-length code is a reversible process, which
`
`allows the exact reconstruction of that which has been coded with the
`
`10
`
`15
`
`20
`
`25
`
`variable-length code.
`
`The display of digital video signals includes a certain
`
`number of
`
`image frames (30 to 96 fps) displayed or represented
`
`successively at a 30 to 75 Hz frequency. Each image frame is still an
`
`|PR2018—00534
`
`Sony EX1004 Page 12
`
`IPR2018-00534
`Sony EX1004 Page 12
`
`
`
`image formed by 3 pixels array, according to the display resolution of a
`
`particular system. By example,
`
`the VHS system has
`
`a display
`
`resolution of 320 columns and 480 rows,
`
`the NTSC system has a
`
`display resolution of 720 columns and 486 rows, and the high definition
`
`television system (HDTV) has a display resolution of 1360 columns and
`
`1020 rows.
`
`In reference to a digitized form of low resolution, 320
`
`columns by 480 rows VHS format, a two—hour long movie could be
`
`equivalent to 100 gigabytes of digital video information. In comparison,
`
`a conventional compact optical disk has an approximate capacity of 0.6
`
`gigabytes, a magnetic hard disk has a 1-2 gigabyte capacity, and the
`
`present compact optical disks have a capacity of 8 or more gigabytes.
`
`All
`
`images we watch at the cinema and TV screens are
`
`based on the principle of presenting complete images (static images,
`
`like photographs) at a great speed. When they are presented in a fast
`
`and sequential manner at a 30 frames per second speed (30 fps) we
`
`perceive them as an animated image due to the retention of the human
`
`eye.
`
`In order
`
`to codify the images to be presented in a
`
`sequential manner and form video signals, each image needs to be
`
`divided in rows, where each line is in turn divided in picture elements
`
`or pixels, each pixel has two associated values, namely,
`
`luma and
`
`chroma. Luma represents the light intensity at each point, while luma
`
`represents the color as a function of a defined color space (RGB),
`
`which can be represented by three bytes.
`
`The images are displayed on a screen in a horizontal-
`
`vertical raster, top to bottom and left to right and so on, cyclically. The
`
`number of lines and frequency of the display can change as a function
`
`of the format, such as NTSC, PAL, or SECAM.
`
`10
`
`15
`
`20
`
`25
`
`|PR2018—00534
`
`Sony EX1004 Page 13
`
`IPR2018-00534
`Sony EX1004 Page 13
`
`
`
`The video signals can be digitized for storage in digital
`
`format, after being transmitted, received, and decoded to be displayed
`
`in a display device, such as a regular television set or the 3DVisor®,
`
`this process
`
`is known as
`
`analog-to—digital video signal coding-
`
`decoding.
`
`By definition, MPEG has
`
`two different methods
`
`for
`
`interlacing video and audio in the system streams.
`
`The transport stream is used in systems with a greater
`
`error possibility, such as satellite systems, which are susceptible to
`
`10
`
`interference. Each package is 188 bytes
`
`long,
`
`starting with an
`
`identification header, which makes recognizing gaps and repairing
`
`errors possible. Various audio and video programs can be transmitted
`
`over the transport stream simultaneously on a single transport stream;
`
`due to the header, they can be independently and individually decoded
`
`15
`
`and integrated into many programs.
`
`The program stream is used in systems with a lesser error
`
`possibility, as in DVD playing.
`
`In this case,
`
`the packages have a
`
`variable-length and a size substantially greater than the packages used
`
`in the transport stream. As a main characteristic, the program stream
`
`20
`
`allows only a single program content.
`
`Even when the transport and program streams handle
`
`different packages,
`
`the video and audio formats are decoded in an
`
`identical form.
`
`In turn,
`
`there are three compression types, which are
`
`25
`
`applied to the packages above, e.g. time prediction, compression, and
`
`space compression.
`
`Decoding is associated to a lengthy mathematical process,
`
`which purpose is to decrease the information volume. The complete
`
`lPR2018—00534
`
`Sony EX1004 Page 14
`
`IPR2018-00534
`Sony EX1004 Page 14
`
`
`
`image of a full frame is divided by a unit called macroblock, each
`
`macroblock is made up of a 16 pixels x 16 pixels matrix, and is ordered
`
`and named top to bottom and left to right. Even with a matrix array on
`
`screen,
`
`the information sent over the information stream follows a
`
`special sequential sequence,
`
`i.e.
`
`the macroblocks are ordered in
`
`ascending order, this is, macroblockO, macroblock1, etc.
`
`A set of consecutive macroblocks represents a slice; there
`
`can be any number of macroblocks
`
`in a
`
`slice given that
`
`the
`
`macroblocks pertain to a single row. As with the macroblocks,
`
`the
`
`slices are numbered from left to right and bottom to top. The slices
`
`must cover the whole image, as this is a form in which MPEG2
`
`compresses the video, a coded image not necessarily needs samples
`
`for each pixel. Some MPEG profiles require handling a rigid slice
`
`structure, by which the whole image should be covered.
`
`USP No. 5,963,257 granted on October 5th, 1999 to Katata
`
`et al., protects a flat video image decoding device with means to
`
`separate the coded data by position areas and image form, bottom
`
`layer code, predictive coding top layer code,
`
`thus obtaining a
`
`hierarchical structure of the coded data; the decoder has means to
`
`separate the data coded in the hierarchical structure in order to obtain
`
`a high quality image.
`
`USP No. 6,292,588 granted on September 18th, 2001 to
`
`Shen et al., protects a device and method for coding predictive flat
`
`images reconstructed and decoded from a small region,
`
`in such way
`
`that the data of the reconstructed flat image is generated from the sum
`
`of the small region image data and the optimal prediction data for said
`
`image. Said predictive decoding device for an image data stream
`
`includes a variable-length code for unidimensional DCT coefficients.
`
`10
`
`15
`
`20
`
`25
`
`lPR2018-00534
`
`Sony EX1004 Page 15
`
`IPR2018-00534
`Sony EX1004 Page 15
`
`
`
`USP No. 6,370,276 granted on April 9th, 2002 to Boon, uses a
`
`decoding method similar to the above.
`
`USP No. 6,456,432 granted on September 24th, 2002 to
`
`Lazzaro et al., protects a stereoscopic 3D-image display system, which
`
`takes images from two perspectives, displays them on a CRT, and
`
`multiplexes the images in a field-sequential manner with no flickering
`
`for both eyes of the observer.
`
`USP No. 6,658,056 granted on December 2, 2003 to
`Duruoz et al., protects a digital video decoder comprising a logical
`
`10
`
`display section responding to a “proximal
`
`field” command to get a
`
`digital video field of designated locations in an output memory. The
`
`digital video display system is equipped with a MPEG2 video decoder.
`
`Images are decoded as a memory buffer,
`
`the memory buffer
`
`is
`
`optimized maintaining compensation variable tables and accessing
`
`15
`
`fixed memory pointer tables displayed as data fields.
`
`USP No. 6,665,445 granted on December 16th, 2003 to
`
`Boon, protects a data structure for image transmission, a flat images
`
`coding method and a flat
`
`images decoding method. The decoding
`
`method is comprised of two parts, the first part to codify the image-
`
`form information data stream, the second part is a decoding process
`
`for the pixel values of the image data stream, both parts can be
`
`switched according to the flat image signal coding.
`
`USP No. 6,678,331 granted on January 13th, 2004 to
`
`Moutin et al., protects a MPEG decoder, which uses a shared memory.
`
`Actually, the circuit includes a microprocessor, a MPEG decoder, which
`
`decodes a flat
`
`image sequence, and a common memory for
`
`the
`
`microprocessor, and the decoder.
`
`It also includes
`
`a circuit
`
`for
`
`evaluating the decoder delay, and a control circuit for determining the
`
`20
`
`25
`
`|PR2018—00534
`
`Sony EX1004 Page 16
`
`IPR2018-00534
`Sony EX1004 Page 16
`
`
`
`memory priority for the microprocessor or the decoder.
`
`USP No. 6,678,424 granted on January 13th, 2004 to
`
`Ferguson, protects a behavior model
`
`for a real-time human vision
`
`system; actually,
`
`it processes two image signals in two dimensions,
`
`one derived from the other, in different channels.
`
`BRIEF DESCRIPTION OF THE INVENTION
`
`it
`
`is an object of
`
`the present
`
`invention to provide a
`
`stereoscopic 3D-video image digital decoding system and method,
`
`comprised of changes in software and changes in hardware.
`
`It is an additional object of the present invention to provide
`
`a decoding method where the normal video_sequence process is
`
`applied to the coded image data, i.e.variable_length_decoding
`
`(VLD),
`
`inverse__scan;
`
`inverse~quantization,
`
`inverse_discrete_cosine_transform (lDCT), and motion_compensation.
`
`it
`
`is also an object of the present
`
`invention to make
`
`changes in the software information for decoding the identification of
`
`the
`
`video
`
`format,
`
`2D—images MPEG2
`
`backward
`
`compatibility,
`
`discriminating a TDVision® type image, storing the last image buffer,
`
`applying information decoding, applying error correction and storing
`
`the results in the respective channel buffer.
`
`It is still another object of the present invention to provide
`
`a decoding method with the video_sequence process normal form,
`
`in
`
`such a way that when a TDVision® type image is found, the buffer of
`
`the last complete image is stored in the left or right channel buffers.
`
`It is also another object of the present invention to provide
`
`a decoding process in which two interdependent (difference) video
`
`signals can be sent within the same video_sequence,
`
`in which
`
`10
`
`15
`
`20
`
`25
`
`lPR2018—00534
`
`Sony EX1004 Page 17
`
`IPR2018-00534
`Sony EX1004 Page 17
`
`
`
`information decoding is applied and is stored as a B type frame.
`
`It is still another object of the present invention to provide
`
`a decoding process in which error correction is applied to the last
`
`obtained image when the movement and color correction vectors are
`
`apphed.
`
`It is also an object of the present invention to program the
`
`decoder by software,
`
`to simultaneously receive and codify two
`
`independent program streams.
`
`It is still another object of the present invention to provide
`
`a decoding system, which decodes the 3D-image information via
`
`hardware, in which a double output buffer is activated.
`
`It
`
`is another object of the present invention to provide a
`
`decoding system of 3D-image information, which activates an image—
`
`decoding selector in parallel and by differences.
`
`It is also another object of the present invention to provide
`
`a 3D-image information decoding system, which
`
`executes
`
`the
`
`decompression process and displays the corresponding output buffer.
`
`DETAILED DESCRIPTION OF THE INVENTION.
`
`The combination of hardware and software algorithms
`
`makes possible the stereoscopic 3D-image information compression,
`
`which are received as two independent video signals but with the same
`
`time_code, corresponding to the left and right signals coming from a
`
`3Dvision® camera,
`
`by sending two simultaneous programs with
`
`stereoscopic pair
`
`identifiers,
`
`thus promoting the coding—decoding
`
`process. Also,
`
`two interdependent video signals can be handled by
`
`obtaining their difference, which is stored as a “B” type frame with the
`
`image type identifier. As the coding process was left open in order to
`
`10
`
`15
`
`20
`
`25
`
`lPR2018—00534
`
`Sony EX1004 Page 18
`
`IPR2018-00534
`Sony EX1004 Page 18
`
`
`
`promote the technological development,
`
`it is only necessary to follow
`
`this decoding process, namely: apply variable-length decoding to the
`
`coded data where a substantial reduction is obtained, but a look-up
`
`table must be used to carry out decoding; apply an inverse scan
`
`process; apply an inverse quantization process in which each data is
`
`multiplied by a scalar; apply the inverse cosine transform function;
`
`apply error correction or motion compensation stage and eventually
`
`obtain the decoded image.
`
`The novel characteristics of this invention in connection
`
`with its structure and operation method will be better understood from
`
`the description of the accompanying figures, together with the attached
`
`specification, where similar numerals refer to similar parts and steps.
`
`Figure 1
`
`represents the technology map to which the
`
`subject object of
`
`the present
`
`invention pertains.
`
`It
`
`shows
`
`a
`
`stereoscopic 3D-image coding and decoding system and corresponding
`
`method. The images come from a stereoscopic camera (32),
`
`the
`
`information compiled in (31) and are displayed in any adequate system
`
`(30) or (33). The information is coded in (34) and then it can be
`
`transmitted to a system having an adequate previous decoding stage
`
`such as (35), which may be a cable system (36), a satellite system
`
`(37), a high definition television system (38) or a stereoscopic vision
`
`system such as TDVision®’s 3DVisors® (39).
`
`Figure 2 shows a flowchart
`
`in which the steps of the
`
`process are outlined. The objective is to obtain three-dimensional
`
`images from a digital video stream by making modifications to the
`
`current MPEGZ decoders, and changes to software (3) and hardware
`
`(4) in the decoding process (2): the decoder (1) must be compatible
`
`with MPEG2-4.
`
`10
`
`15
`
`20
`
`25
`
`|PR2018—00534
`
`Sony EX1004 Page 19
`
`IPR2018-00534
`Sony EX1004 Page 19
`
`
`
`Figure 3 outlines the structures that must be modified and
`
`the video_sequence of
`
`the data stream in order to identify the
`
`TDVision® technology image type at the bit level.
`
`Each of the stages of the decoding process is detailed
`
`below (20):
`
`The coded data (10) are bytes with block information,
`
`macroblocks, fields, frames, and MPEG2 format video images.
`
`Variable_length_decoding
`
`(11)
`
`(VLC, Variable-length
`
`Decoder)
`
`is a compression algorithm in which the most frequent
`
`patterns are replaced by shorter codes and those occurring less
`
`frequently are replaced by longer codes. The compressed version of
`
`this information occupies less space and can be transmitted faster by
`
`networks. However,
`
`it
`
`is not an easily editable format and requires
`
`decompression using a look-up table.
`
`For example, the word BEETLE
`
`Letter
`B
`E
`L
`T
`
`ASCII Code
`01000010
`0110 0101
`01101100
`0111 0100
`
`VLC
`0000 001010
`11
`000101
`0100
`
`Therefore, the ASCII code for the word is:
`
`0100 0010 0110 01010110 0101011101000 01101100 0110 0101
`
`in VLC: 0000 00101011110100 00010 0111.
`
`A substantial decrease is noted, however,
`
`in order to go
`
`back from VLC to the word 'Beetle' a search in the look-up table is
`
`needed to decode the bit stream, this is made by exact comparison of
`
`the read bits.
`
`Inverse scan (12): The information must be grouped by
`
`blocks, and by coding the information with the VLC a linear stream is
`
`10
`
`15
`
`20
`
`25
`
`lPR2018—00534
`
`Sony EX1004 Page 20
`
`IPR2018-00534
`Sony EX1004 Page 20
`
`
`
`10
`
`obtained. The blocks are 8x8 data matrixes, so it
`
`is necessary to
`
`convert the linear information in a square 8x8 matrix. This is made in a
`
`descending zigzag manner,
`
`top to bottom and left
`
`to right
`
`in both
`
`sequence types, depending on whether it is a progressive image or an
`
`interlaced image.
`
`Inverse Quantization (13):
`
`It consists simply in multiplying
`
`each data value by a factor. When codified, most of the data in the
`
`blocks are quantized to remove information that the human eye is not
`
`able to perceive, the quantization allows to obtain a greater MPEGZ
`
`stream conversion, and it
`
`is also required to perform the inverse
`
`process (Inverse quantization) in the decoding process.
`
`inverse
`
`cosine
`
`transform
`
`(14)
`
`(IDCT,
`
`inverse__discrete_cosine_transform): The data handled within each
`
`block pertain to the frequency domain, this inverse cosine transform
`
`allows to return to the samples of the space domain. Once the data in
`
`the lDCT have been transformed, pixels, colors and color corrections
`
`can be obtained.
`
`Motion compensation (15) allows to correct some errors
`
`generated before the decoding stage of MPEG format, motion
`
`compensation takes as a reference a previous frame and calculates a
`
`motion vector relative to the pixels (it can calculate up to four vectors),
`
`and uses them to create a new image. This motion compensation is
`
`applied to the P and B type images, where the image position is
`
`located over a "t" time from the reference images. Additionally to the
`
`motion compensation, the error correction is also applied, as it is not
`
`enough to predict the position of a particular pixel, but a change in its
`
`color can also exist. Thus, the decoded image is obtained (16).
`
`To decode a P or B type image, the reference image is
`
`10
`
`15
`
`20
`
`25
`
`lPR2018—00534
`
`Sony EX1004 Page 21
`
`IPR2018-00534
`Sony EX1004 Page 21
`
`
`
`11
`
`taken, the motion vectors are algebraically added to calculate the next
`
`image, and finally the error correction data is applied, thus generating
`
`the decoded image successfully. Actually,
`
`in the video_sequence, two
`
`interdependent video signals exist, “R-L= delta, the delta difference is
`
`that stored as a B type stereoscopic pair
`
`frame with TDVision®
`
`identifier and which is constructed at
`
`the moment of decoding by
`
`differences from the image. This is, R-delta= L and L-delta= R, the left
`
`image is constructed from the difference with the right image, which in
`
`turn is constructed from the difference with the left image.
`
`The previous process is outlined in such a way that the left
`
`or right signal is taken, both are stored in a temporary buffer, then the
`
`difference between the left and right signals is calculated, and then it is
`
`coded as a B type image stored in the video_sequence to be later
`
`decoded by differences from said image.
`
`In the decoding process it can be deducted that the data
`
`inputted by the VLC stage are much smaller than the data outputted by
`
`the same stage.
`
`MPEG video sequence structure: This is the maximum
`
`structure used in the MPEG2 format and has the following format:
`
`Video sequence (Video__Sequence)
`
`Sequence header (Sequence__Header)
`
`Sequence extension (Sequence_Extension)
`
`User Data (0) and Extension (Extension__and__User_Data
`
`(0))
`
`Image group header (Group__of_Picture_Header)
`
`User Data (1) and Extension (Extension_and_User__Data
`
`(1))
`
`Image header (Picture__Header)
`
`10
`
`15
`
`20
`
`25
`
`|PR2018—00534
`
`Sony EX1004 Page 22
`
`IPR2018-00534
`Sony EX1004 Page 22
`
`
`
`12
`
`Coded image extension (Picture_Coding_Extension)
`
`User Data (2) and Extensions (Extension~and_User_Data
`
`(2))
`
`Image Data (Picture__Data)
`
`Slice(Slice)
`
`Macroblock (Macroblock)
`
`Motion vectors (Motion_Vectors)
`
`Coded Block Pattern (Coded_B|ock_Pattern)
`
`Block (Block)
`
`Final Sequence Code (Sequence_end_Code)
`
`These structures make up the video sequence. A video
`
`sequence is applied for MPEG format,
`
`in order to differentiate each
`
`version there must be a validation that immediately after the sequence
`
`header,
`
`the sequence extension is present; should the sequence
`
`extension not follow the header, then the stream is in MPEG1 format.
`
`At
`
`the
`
`beginning
`
`of
`
`a
`
`video
`
`sequence,
`
`the
`
`sequence_header
`
`and
`
`sequencewextension
`
`appear
`
`in
`
`the
`
`video_sequence. The
`
`sequence_extension
`
`repetitions must
`
`be
`
`identical on the first try and the "s" repetitions of the sequence_header
`
`vary little compared to the first occurrence, only the portion defining
`
`the quantization matrixes should change. Having sequences repetition
`
`allows a random access to the video stream, i.e., if the decoder wants
`
`to start playing at the middle of the video stream this may be done, as
`
`it only needs to find the sequence_header and sequence_extension
`
`prior to that moment in order to decode the following images. This also
`
`happens for video streams that could not start from the beginning, such
`
`as a satellite decoder turned on after the transmission time.
`
`The full video signal coding-decoding process is comprised
`
`|PR2018—00534
`
`Sony EX1004 Page 23
`
`10
`
`15
`
`20
`
`25
`
`IPR2018-00534
`Sony EX1004 Page 23
`
`
`
`13
`
`of the following steps:
`
`Digitizing the video signals, which can be done in NTSC,
`
`PAL or SECAM format.
`
`Storing the video signal in digital form
`
`Transmitting the signals
`
`Recording the digital video stream in a physical media
`
`(DVD, VCD, MiniDV)
`
`Receiving the signals
`
`Playing the video stream
`
`Decoding the signal
`
`Displaying the signal
`
`It is essential to double the memory to be handled by the
`
`adequate DSP and have the possibility of disposing of up to 8 output
`
`buffers, which allow the previous and simultaneous representation of a
`
`stereoscopic image on a device such as TDVision®’s 3DVisor®
`
`Actually, two channels must be initialized when calling the
`
`programming APl of the DSP as, by example, the illustrative case of
`
`the Texas instruments TMS320062X DSP.
`
`MPEG2VDEC_create
`
`(const
`
`lMPEG2VDEC_fxns*fxns,
`
`10
`
`15
`
`20
`
`const MEPGZVDEC_Params* params).
`
`Where |MPEG2VDEC_fxns y MEPGZVDEC_Params are
`
`pointer structures defining the operation parameters for each video
`
`channel, e.g.:
`
`3DLhandle=MPEG2VDEC_create (fxns3DLEFT,Params3DLEFT).
`
`25
`
`3DRhandle=MPEGZVDEC_create(fxn33DRIGHT,ParamsSDRlGHT.
`
`Thereby enabling two video channels to be decoded and
`
`obtaining two video handlers, one for
`
`the left-right stereoscopic
`
`|PR2018—00534
`
`Sony EX1004 Page 24
`
`IPR2018-00534
`Sony EX1004 Page 24
`
`
`
`14
`
`channeL
`
`A double display output buffer is needed and by means of
`
`software,
`
`it will be defined which of the two buffers must display the
`
`output by calling the AP function:
`
`Namely,
`
`MPEG2VDEC_APPLY(3DRhandle,
`
`inputR1,
`
`inputRZ, inputR3, 3doutright_pb, 3doutright_fb).
`
`MPEG2VDEC_APPLY(3DLhandIe,
`
`inputL1,
`
`inputL2,
`
`inputL3, 3doutleft_pb, 3doutleft_fb).
`
`This same procedure can be implemented for any DSP,
`
`10
`
`microprocessor or electronic device with similar functions.
`
`Where 3DLhandle is the pointer to the handle returned by
`
`the DSP’s
`
`create
`
`function,
`
`the
`
`input1
`
`parameter
`
`is
`
`the
`
`FUNC_DECODE_FRAME or FUNC_START_PARA address,
`
`input2 is
`
`the pointer to the external input buffer address, and input3 is the size
`
`15
`
`of the external input buffer size.
`
`3doutleft_pb is the address of the parameter buffer and
`
`3doutleft_fb is the beginning of the output buffer where the decoded
`
`image will be stored.
`
`The timecode and timestamp will be used for output to the
`
`20
`
`final device in a sequential, synchronized manner.
`
`it is essential to double the memory to be handled by the
`
`DSP and have the possibility of disposing of up to 8 output buffers
`
`which allow the previous and simultaneous display of a stereoscopic
`
`image on a device such as TDVision® Corporation’s 3DVisor®.
`
`25
`
`The integration of software and hardware processes is
`
`carried out by devices known as DSP, which execute most of the
`
`hardware process. These DSP are programmed by a C and Assembly
`
`language hybrid provided by the manufacturer. Each DSP has its own
`
`|PR2018—00534
`
`Sony EX1004 Page 25
`
`IPR2018-00534
`Sony EX1004 Page 25
`
`
`
`15
`
`API, consisting of a functions list or procedure calls located in the DSP
`
`and called by software.
`
`With this reference information, the present application for
`
`MPEGZ format—compatible 3D-images decoding is made.
`
`Actually, at
`
`the beginning of a video sequence the
`
`sequence header (sequence_header) and the sequence extension
`
`always appear. The repetitions of the sequence extension must be
`
`identical to the first. On the contrary, the sequence header repetitions
`
`vary a little as compared to the first occurrence, only the portion
`
`10
`
`defining the quantization matrixes should change.
`
`Figure 4 shows the compilation software format for the
`
`TDVision® decoding method (40), where the video_sequence (41) of
`
`the digital stereoscopic image video stream is identified, which may be
`
`dependent or independent (parallel images),
`
`in the sequence_header
`
`(42).
`
`if the image is TDVision® then the double buffer is activated and
`
`the changes
`
`in
`
`the aspect_ratio_information are identified. The
`
`information corresponding to the image that can be found here is read
`
`in the user_data (43). The sequence_scalable_extension (44) identifies
`
`the information contained in it and the base and enhancement layers,
`
`the video_sequence can be located here, defines the scalable_mode
`
`and
`
`the
`
`layer
`
`identifier.
`
`extra_bit_picture
`
`(45)
`
`identifies
`
`the
`
`picture_estructure, picture_header and the picture_coding_extension
`
`(46) reads the “B” type images and if it
`
`is a TDVision® type image,
`
`then
`
`it
`
`decodes
`
`the
`
`second
`
`buffer.
`
`picture_temporal_scalable_extension ()
`
`(47),
`
`in
`
`case
`
`of having
`
`temporal scalability, is used to decode B