`35
`
`ABSTRACT
`
`MPEG-2 compatible stereoscopic 3D-video image digital
`
`decoding method and system, using its own coding algorithm. In order
`
`to obtain 3D-images from a digital video stream, modifications have
`
`5
`
`been made to the current MPEG2 decoders, by means of software and
`
`hardware changes in different parts of the decoding process. Namely,
`
`the video_sequence structures of the video data stream are modified
`
`via software to include the necessary flags at the bit level of the image
`
`type in the TDVision® technology; the modifications are made in the
`
`10
`
`decoding processes as well as in decoding the information via software
`
`and hardware, where a double output buffer is activated, a parallel and
`
`differences decoding selector is activated, the decompression process
`
`is executed, the corresponding output buffer is displayed; the decoder
`
`must be programmed
`
`via software to simultaneously receive and
`
`15
`
`decode two independent program streams, each with an TDVision®
`
`stereoscopic identifier.
`
`IPR2018-00534
`Sony EX1005 Page 1
`
`
`
`CA 02557534 2006-08-25
`30
`
`CLAIMS
`
`1.
`
`A stereoscopic 3D-video
`
`image digital decoding
`
`system and method, in which the structures of the video_sequence of
`
`the video data stream are modified via software,
`
`to
`
`include the
`
`5
`
`necessary flags at the bit level for the image type, characterized by
`
`only modifying the software and by using the user_data() section to
`
`store the error correction which allows to regenerate the stereoscopic
`
`video signal, thereby actually identifying the video format; applying a
`
`logical "and" for MPEG2 backward compatibility in case it is not a
`
`10
`
`TDVision® video; typically decoding by scanning the video_sequence;
`
`when the image is a TDVision® image:
`
`a)
`
`storing the last complete image buffer in the left or
`
`right channel buffer.
`
`b)
`
`applying the differences or parallel decoding for 8
`
`15
`
`type frame information,
`
`c)
`
`applying error correction to the last image obtained
`
`by applying the motion and color correction vectors,
`
`d)
`
`e)
`
`2.
`
`storing the results in their respective channel buffer,
`
`continuing with the video_sequence reading.
`
`Stereoscopic 3D-video
`
`image digital decoding
`
`method and system, in which the video_sequence structures of the
`
`video data stream are modified via software to include the necessary
`
`flags at the bit level of the image type according to Claim 1, further
`
`characterized by the decoder compilation format being as follows:
`
`a)
`
`b)
`
`reading video_sequence ,
`
`discriminating the sequence_header, if a TDVision®
`
`image is identified, then activating the double buffer,
`
`c)
`
`reading in the user data the image as if it was
`
`20
`
`25
`
`IPR2018-00534
`Sony EX1005 Page 2
`
`
`
`CA 02557534 2006-08-25
`31
`
`contained in said structure,
`
`d)
`
`adding
`
`in
`
`the
`
`sequence_scalable_extension
`
`information to the video_sequence MPEG, said information could be
`
`contained within said structure,
`
`e)
`
`finding in the picture_header the TDVision® image
`
`identifier in the extra_bit_picture,
`
`f)
`
`reading
`
`the
`
`"B"
`
`type
`
`image
`
`in
`
`the
`
`picture_coding_extension, and
`
`if
`
`it
`
`is a TDVision®
`
`type
`
`image,
`
`decoding then the second buffer,
`
`g)
`
`if the image is temporarily scalable, applying "B" to
`
`5
`
`10
`
`the decoder.
`
`3.
`
`Stereoscopic 3D-video
`
`images digital decoding
`
`method and system, in which the structures and the video_sequence of
`
`the video data stream are modified to include the necessary flags at
`
`15
`
`the bit
`
`level of
`
`the
`
`image
`
`type according
`
`to Claim 1,
`
`further
`
`characterized in that when the decoder detects a user_data() code, it
`searches the 32-bit 3DVision®_start_identifier = OxOOOABCD identifier,
`upon detecting this information a call is made to the special decoding
`
`function which compares the output buffer and applies it from the
`
`20
`
`current reading offset of the video_sequence.
`
`4.
`
`Stereoscopic 3D-video
`
`images digital decoding
`
`method and system, in which the video_sequence structures of the
`
`video data stream are modified via software to include the necessary
`
`flags at the bit level of the image type according to Claim 1, further
`
`25
`
`characterized in that the decoder must be programmed via software to
`
`simultaneously receive and decode two program streams.
`
`5.
`
`Stereoscopic 3D-video images digital decoding method
`
`and system, in which the video_sequence structures of the video data stream
`
`IPR2018-00534
`Sony EX1005 Page 3
`
`
`
`CA 02557534 2006-08-25
`32
`
`are modified via software to include the necessary flags at the bit level of the
`
`image
`
`type according
`
`to Claim 1,
`
`further characterized
`
`in
`
`that
`
`two
`
`interdependent video signals can be sent within the same video_sequence;
`
`said signals depending one form the other, and coming from a 3DVision®
`
`5
`
`camera; in terms of their algebraic addition (R-L=delta), each signal is stored
`
`as a B type frame, which decoding is by differences from one of them.
`
`6.
`
`Stereoscopic 3D-video
`
`images digital decoding
`
`method and system, in which the video_sequence structures of the
`
`video data stream are modified via software to include the necessary
`
`10
`
`flags at the bit level of the image type according to Claim 1, further
`
`characterized in that two independent video streams L and R are
`
`stored in simultaneous form, but being synchronized with the same
`
`time_code, and decoded and displayed in parallel.
`
`7.
`
`Stereoscopic 3D-video
`
`images digital decoding
`
`15
`
`method and system, in which the video_sequence structures of the
`
`video data stream are modified via hardware, characterized by the
`
`specific use of the structures, substructures and sequences belonging
`
`to the video_sequence to implement the MPEG2 backward-compatible
`
`TDVision® technology via hardware, in effect, discriminating whether it
`
`20
`
`is a 2S or 30 signal; activating a double output buffer (additional
`
`memory); activating a parallel decoding selector, activating a
`
`difference- decoding selector; executing the image decompression
`
`process, displaying the image in the corresponding output buffer;
`
`enabling the PICTURE_DATA3D() function, which is transparent for the
`
`25
`
`compatible MPEG2 readers.
`
`8.
`
`Stereoscopic 3D-video
`
`images digital decoding
`
`method and system, in which the video_sequence structures of the
`
`video data stream are modified via hardware according with Claim 7,
`
`IPR2018-00534
`Sony EX1005 Page 4
`
`
`
`CA 02557534 2006-08-25
`33
`
`characterized by the specific use of the structures, substructures and
`
`sequences belonging to the video_sequence in order to implement the
`
`MPEG2 backward-compatible TOVision® technology via hardware:
`
`a)
`
`sequence_header
`
`5
`
`aspect_ratio_information
`
`1001 n/a
`
`in TOVision®
`
`1010 4:3
`
`in TOVision®
`
`1011 16:9 in TOVision®
`
`1100 2.21:1 in TOVision®
`
`10
`
`a
`
`logical "and" with 0111 will be executed
`
`to obtain
`
`backward compatibility with 20 systems, where an instruction is sent to
`
`the OSP stating that the stereoscopic pair buffer (left or right) must be
`
`equal to the source;
`
`b)
`
`frame_rate_code
`
`15
`
`1001 24,000/1001 (23. 976) in TDVision® format
`
`1010 24 in TOVision® format
`
`1011 25 in TDVision® format
`
`1100 30,000/1001 (29.97) in TOVision® format
`
`1101 30 in TDVision® format
`
`20
`
`1110 50 in TOVision® format
`
`1111 60,000/1001 (59.94) in TOVision®format
`
`a
`
`logical "and" with 0111 will be executed
`
`to obtain
`
`backward compatibility with 20 systems, where an instruction is sent to
`
`the OSP stating that the stereoscopic pair buffer (left or right) must be
`
`25
`
`equal to the source;
`
`c)
`
`user_data()
`
`sequence_ scalable_ extension
`
`d)
`
`picture_header
`
`IPR2018-00534
`Sony EX1005 Page 5
`
`
`
`CA 02557534 2006-08-25
`34
`
`extra_bit_picture
`
`0 = TDVision®
`
`1 =normal
`
`e)
`
`picture_coding_extension
`
`5
`
`picture_structure
`
`00 = image in TDVision® format
`
`f)
`
`9.
`
`picture_temporal_scalable_extension().
`
`Stereoscopic 3D-video
`
`images digital decoding
`
`method and system, in which the video_sequence structures of the
`
`10
`
`video data stream are modified via hardware according with Claim 7,
`
`characterized
`
`in
`
`that, when
`
`the PICTURE_DATA3D()structure
`
`is
`
`recognized, it proceeds to read the information directly by the decoder,
`
`but it writes the information in a second output buffer also connected to
`
`a video output additional to that existing in the electronic display
`
`15
`
`device.
`
`10. Stereoscopic 3D-video
`
`images digital decoding
`
`method and system, in which the video_sequence structures of the
`
`video data stream are modified via hardware according with Claim 7,
`
`characterized in that, if the signal is of TDVision® type, it is identified if
`
`20
`
`it is a transport stream, program stream or left or right multiplexion at
`
`60 frames per second; when it is a transport stream it has backward
`
`compatibility in the current 20 coders; where an instruction is sent to
`
`the DSP stating that the stereoscopic pair buffer (left or right) must be
`
`equal to the source, having the ability to display the video without 3D
`
`25
`
`characteristics of TDVision®.
`
`IPR2018-00534
`Sony EX1005 Page 6
`
`
`
`CA 02557534 2006-08-25
`
`STEREOSCOPIC 3D-VIDEO IMAGE DIGITAL DECODING SYSTEM
`
`AND METHOD
`
`FIELD OF THE INVENTION
`
`5
`
`The present invention is related to stereoscopic video
`
`image display in the 30Visor® device and, particularly, to a video
`
`image decoding method by means of a digital data compression
`
`system, which allows the storage of three-dimensional information by
`
`using standardized compression techniques.
`
`10
`
`BACKGROUND OF THE INVENTION
`
`Presently, data compression techniques are used in order
`
`to decrease the bits consumption in the representation of an image or
`
`a series of images. The standardization works were carried out by a
`
`15
`
`group of experts of the International Standardization Organization.
`
`Presently, the methods are usually known as JPEG (Joint Photographic
`
`Expert Group}, and MPEG (Moving Pictures Expert Group).
`
`A common characteristic of these techniques is that the
`
`image blocks are processed by means of the application of a transform
`
`20
`
`adequate for the block, usually known as Discrete Cosine Transform
`
`(OCT). The formed blocks are submitted to a quantization process, and
`
`then coded with a variable-length code.
`
`The variable-length code is a reversible process, which
`
`allows the exact reconstruction of that which has been coded with the
`
`25
`
`variable-length code.
`
`The display of digital video signals includes a certain
`
`number of image frames (30 to 96 fps) displayed or represented
`
`successively at a 30 to 75 Hz frequency. Each image frame is still an
`
`IPR2018-00534
`Sony EX1005 Page 7
`
`
`
`CA 02557534 2006-08-25
`2
`
`image formed by a pixels array, according to the display resolution of a
`
`particular system. By example,
`
`the VHS system has a display
`
`resolution of 320 columns and 480 rows, the NTSC system has a
`
`display resolution of 720 columns and 486 rows, and the high definition
`
`5
`
`television system (HDTV) has a display resolution of 1360 columns and
`
`1020 rows.
`
`In reference to a digitized form of low resolution, 320
`
`columns by 480 rows VHS format, a two-hour long movie could be
`
`equivalent to 100 gigabytes of digital video information. In comparison,
`
`a conventional compact optical disk has an approximate capacity of 0.6
`
`10
`
`gigabytes, a magnetic hard disk has a 1-2 gigabyte capacity, and the
`
`present compact optical disks have a capacity of 8 or more gigabytes.
`
`All images we watch at the cinema and TV screens are
`
`based on the principle of presenting complete images (static images,
`
`like photographs) at a great speed. When they are presented in a fast
`
`15
`
`and sequential manner at a 30 frames per second speed (30 fps) we
`
`perceive them as an animated image due to the retention of the human
`
`eye.
`
`In order to codify the
`
`images to be presented
`
`in a
`
`sequential manner and form video signals, each image needs to be
`
`20
`
`divided in rows, where each line is in turn divided in picture elements
`
`or pixels, each pixel has two associated values, namely, luma and
`
`chroma. Luma represents the light intensity at each point, while luma
`
`represents the color as a function of a defined color space (RGB),
`
`which can be represented by three bytes.
`
`25
`
`The images are displayed on a screen in a horizontal-
`
`vertical raster, top to bottom and left to right and so on, cyclically. The
`
`number of lines and frequency of the display can change as a function
`
`of the format, such as NTSC, PAL, or SECAM.
`
`IPR2018-00534
`Sony EX1005 Page 8
`
`
`
`CA 02557534 2006-08-25
`3
`
`The video signals can be digitized for storage in digital
`
`format, after being transmitted, received, and decoded to be displayed
`
`in a display device, such as a regular television set or the 3DVisor®,
`
`this process
`
`is known as analog-to-digital video signal coding-
`
`5
`
`decoding.
`
`By definition, MPEG has
`
`two different methods
`
`for
`
`interlacing video and audio in the system streams.
`
`The transport stream is used in systems with a greater
`
`error possibility, such as satellite systems, which are susceptible to
`
`10
`
`interference. Each package
`
`is 188 bytes
`
`long, starting with an
`
`identification header, which makes recognizing gaps and repairing
`
`errors possible. Various audio and video programs can be transmitted
`
`over the transport stream simultaneously on a single transport stream;
`
`due to the header, they can be independently and individually decoded
`
`15
`
`and integrated into many programs.
`
`The program stream is used in systems with a lesser error
`
`possibility, as in DVD playing.
`
`In this case, the packages have a
`
`variable-length and a size substantially greater than the packages used
`
`in the transport stream. As a main characteristic, the program stream
`
`20
`
`allows only a single program content.
`
`Even when the transport and program streams handle
`
`different packages, the video and audio formats are decoded in an
`
`identical form.
`
`In turn, there are three compression types, which are
`
`25
`
`applied to the packages above, e.g. time prediction, compression, and
`
`space compression.
`
`Decoding is associated to a lengthy mathematical process,
`
`which purpose is to decrease the information volume. The complete
`
`IPR2018-00534
`Sony EX1005 Page 9
`
`
`
`CA 02557534 2006-08-25
`4
`
`image of a full frame is divided by a unit called macroblock, each
`
`macroblock is made up of a 16 pixels x 16 pixels matrix, and is ordered
`
`and named top to bottom and left to right. Even with a matrix array on
`
`screen, the information sent over the information stream follows a
`
`5
`
`special sequential sequence,
`
`i.e.
`
`the macro blocks are ordered in
`
`ascending order, this is, macroblockO, macroblock1, etc.
`
`A set of consecutive macroblocks represents a slice; there
`
`can be any number of macroblocks
`
`in a slice given
`
`that
`
`the
`
`macroblocks pertain to a single row. As with the macroblocks, the
`
`10
`
`slices are numbered from left to right and bottom to top. The slices
`
`must cover the whole image, as this is a form
`
`in which MPEG2
`
`compresses the video, a coded image not necessarily needs samples
`
`for each pixel. Some MPEG profiles require handling a rigid slice
`
`structure, by which the whole image should be covered.
`
`15
`
`USP No. 5,963,257 granted on October 5th, 1999 to Katata
`
`et al., protects a flat video image decoding device with means to
`
`separate the coded data by position areas and image form, bottom
`
`layer code, predictive coding
`
`top
`
`layer code,
`
`thus obtaining a
`
`hierarchical structure of the coded data; the decoder has means to
`
`20
`
`separate the data coded in the hierarchical structure in order to obtain
`
`a high quality image.
`
`USP No. 6,292,588 granted on September 18th, 2001 to
`
`Shen et al., protects a device and method for coding predictive flat
`
`images reconstructed and decoded from a small region, in such way
`
`25
`
`that the data of the reconstructed flat image is generated from the sum
`
`of the small region image data and the optimal prediction data for said
`
`image. Said predictive decoding device for an image data stream
`
`includes a variable-length code for unidimensional OCT coefficients.
`
`IPR2018-00534
`Sony EX1005 Page 10
`
`
`
`CA 02557534 2006-08-25
`5
`
`USP No. 6,370,276 granted on April 9th, 2002 to Boon, uses a
`
`decoding method similar to the above.
`
`USP No. 6,456,432 granted on September 24th, 2002 to
`
`Lazzaro et al., protects a stereoscopic 3D-image display system, which
`
`5
`
`takes images from two perspectives, displays them on a CRT, and
`
`multiplexes the images in a field-sequential manner with no flickering
`
`for both eyes of the observer.
`
`USP No. 6,658,056 granted on December 2, 2003 to
`
`Duruoz et al., protects a digital video decoder comprising a logical
`
`10
`
`display section responding to a "proximal field" command to get a
`
`digital video field of designated locations in an output memory. The
`
`digital video display system is equipped with a MPEG2 video decoder.
`
`Images are decoded as a memory buffer,
`
`the memory buffer is
`
`optimized maintaining compensation variable tables and access1ng
`
`15
`
`fixed memory pointer tables displayed as data fields.
`
`USP No. 6,665,445 granted on December 16th, 2003 to
`
`Boon, protects a data structure for image transmission, a flat images
`
`coding method and a flat images decoding method. The decoding
`
`method is comprised of two parts, the first part to codify the image-
`
`20
`
`form information data stream, the second part is a decoding process
`
`for the pixel values of the image data stream, both parts can be
`
`switched according to the flat image signal coding.
`
`USP No. 6,678,331 granted on January 13th, 2004 to
`
`Moutin et al., protects a MPEG decoder, which uses a shared memory.
`
`25
`
`Actually, the circuit includes a microprocessor, a MPEG decoder, which
`
`decodes a flat image sequence, and a common memory for the
`
`microprocessor, and
`
`the decoder.
`
`It also
`
`includes a circuit
`
`for
`
`evaluating the decoder delay, and a control circuit for determining the
`
`IPR2018-00534
`Sony EX1005 Page 11
`
`
`
`CA 02557534 2006-08-25
`6
`
`memory priority for the microprocessor or the decoder.
`
`USP No. 6,678,424 granted on January 13th, 2004 to
`
`Ferguson, protects a behavior model for a real-time human vision
`
`system; actually, it processes two image signals in two dimensions,
`
`5
`
`one derived from the other, in different channels.
`
`BRIEF DESCRIPTION OF THE INVENTION
`
`It is an object of the present invention to provide a
`
`stereoscopic 3D-video image digital decoding system and method,
`
`10
`
`comprised of changes in software and changes in hardware.
`
`It is an additional object of the present invention to provide
`
`a decoding method where the normal video_sequence process is
`
`applied to the coded image data, i.e. variable_length_decoding
`
`(VLD),
`
`inverse_scan;
`
`inverse_ quantization,
`
`15
`
`inverse_discrete_cosine_transform (IDCT), and motion_compensation.
`
`It is also an object of the present invention to make
`
`changes in the software information for decoding the identification of
`
`the video
`
`format, 20-images MPEG2 backward compatibility,
`
`discriminating a TDVision® type image, storing the last image buffer,
`
`20
`
`applying information decoding, applying error correction and storing
`
`the results in the respective channel buffer.
`
`It is still another object of the present invention to provide
`
`a decoding method with the video_sequence process normal form, in
`
`such a way that when a TDVision® type image is found, the buffer of
`
`25
`
`the last complete image is stored in the left or right channel buffers.
`
`It is also another object of the present invention to provide
`
`a decoding process in which two interdependent (difference) video
`
`signals can be sent within
`
`the same video_sequence,
`
`in which
`
`IPR2018-00534
`Sony EX1005 Page 12
`
`
`
`CA 02557534 2006-08-25
`7
`
`information decoding is applied and is stored as a B type frame.
`
`It is still another object of the present invention to provide
`
`a decoding process in which error correction is applied to the last
`
`obtained image when the movement and color correction vectors are
`
`5
`
`applied.
`
`It is also an object of the present invention to program the
`
`decoder by software,
`
`to simultaneously receive and codify
`
`two
`
`independent program streams.
`
`It is still another object of the present invention to provide
`
`10
`
`a decoding system, which decodes the 3D-image information via
`
`hardware, in which a double output buffer is activated.
`
`It is another object of the present invention to provide a
`
`decoding system of 3D-image information, which activates an image(cid:173)
`
`decoding selector in parallel and by differences.
`
`15
`
`It is also another object of the present invention to provide
`
`a 3D-image information decoding system, which
`
`executes
`
`the
`
`decompression process and displays the corresponding output buffer.
`
`DETAILED DESCRIPTION OF THE INVENTION.
`
`20
`
`The combination of hardware and software algorithms
`
`makes possible the stereoscopic 3D-image information compression,
`
`which are received as two independent video signals but with the same
`
`time_code, corresponding to the left and right signals coming from a
`
`3Dvision® camera, by sending
`
`two simultaneous programs with
`
`25
`
`stereoscopic pair
`
`identifiers,
`
`thus promoting
`
`the coding-decoding
`
`process. Also, two interdependent video signals can be handled by
`
`obtaining their difference, which is stored as a "B" type frame with the
`
`image type identifier. As the coding process was left open in order to
`
`IPR2018-00534
`Sony EX1005 Page 13
`
`
`
`CA 02557534 2006-08-25
`8
`
`promote the technological development, it is only necessary to follow
`
`this decoding process, namely: apply variable-length decoding to the
`
`coded data where a substantial reduction is obtained, but a look-up
`
`table must be used to carry out decoding; apply an inverse scan
`
`5
`
`process; apply an inverse quantization process in which each data is
`
`multiplied by a scalar; apply the inverse cosine transform function;
`
`apply error correction or motion compensation stage and eventually
`
`obtain the decoded image.
`
`The novel characteristics of this invention in connection
`
`10
`
`with its structure and operation method will be better understood from
`
`the description of the accompanying figures, together with the attached
`
`specification, where similar numerals refer to similar parts and steps.
`
`Figure 1 represents the technology map to which the
`
`subject object of
`
`the present
`
`invention pertains.
`
`It shows a
`
`15
`
`stereoscopic 3D-image coding and decoding system and corresponding
`
`method. The images come from a stereoscopic camera (32),
`
`the
`
`information compiled in (31) and are displayed in any adequate system
`
`(30) or (33). The information is coded in (34) and then it can be
`
`transmitted to a system having an adequate previous decoding stage
`
`20
`
`such as (35), which may be a cable system (36), a satellite system
`
`(37), a high definition television system (38) or a stereoscopic vision
`
`system such as TDVision®'s 3DVisors® (39).
`
`Figure 2 shows a flowchart in which the steps of the
`
`process are outlined. The objective is to obtain three-dimensional
`
`25
`
`images from a digital video stream by making modifications to the
`
`current MPEG2 decoders, and changes to software (3) and hardware
`
`(4) in the decoding process (2): the decoder (1) must be compatible
`
`with MPEG2-4.
`
`IPR2018-00534
`Sony EX1005 Page 14
`
`
`
`CA 02557534 2006-08-25
`9
`
`Figure 3 outlines the structures that must be modified and
`
`the video_sequence of the data stream
`
`in order to
`
`identify the
`
`TDVision® technology image type at the bit level.
`
`Each of the stages of the decoding process is detailed
`
`5
`
`below (20):
`
`The coded data (1 0) are bytes with block information,
`
`macroblocks, fields, frames, and MPEG2 format video images.
`
`Variable_length_decoding
`
`(11)
`
`(VLC, Variable-length
`
`Decoder)
`
`is a compression algorithm
`
`in which the most frequent
`
`10
`
`patterns are replaced by shorter codes and those occurring less
`
`frequently are replaced by longer codes. The compressed version of
`
`this information occupies less space and can be transmitted faster by
`
`networks. However, it is not an easily editable format and requires
`
`decompression using a look-up table.
`
`15
`
`For example, the word BEETLE
`
`Letter
`8
`E
`L
`T
`
`ASCII Code
`01000010
`01100101
`0110 1100
`0111 0100
`
`VLC
`0000 0010 10
`11
`0001 01
`0100
`
`Therefore, the ASCII code for the word is:
`
`0100 0010 0110 0101 0110 0101 0111 01000 0110 1100 0110 0101
`
`in VLC: 0000 001 0 10 11 11 0100 00010 01 11.
`
`20
`
`A substantial decrease is noted, however, in order to go
`
`back from VLC to the word 'Beetle' a search in the look-up table is
`
`needed to decode the bit stream, this is made by exact comparison of
`
`the read bits.
`
`Inverse scan (12): The information must be grouped by
`
`25
`
`blocks, and by coding the information with the VLC a linear stream is
`
`IPR2018-00534
`Sony EX1005 Page 15
`
`
`
`CA 02557534 2006-08-25
`10
`
`obtained. The blocks are 8x8 data matrixes, so it is necessary to
`
`convert the linear information in a square 8x8 matrix. This is made in a
`
`descending zigzag manner, top to bottom and left to right in both
`
`sequence types, depending on whether it is a progressive image or an
`
`5
`
`interlaced image.
`
`Inverse Quantization (13): It consists simply in multiplying
`
`each data value by a factor. When codified, most of the data in the
`
`blocks are quantized to remove information that the human eye is not
`
`able to perceive, the quantization allows to obtain a greater MPEG2
`
`10
`
`stream conversion, and it is also required to perform the inverse
`
`process (Inverse quantization) in the decoding process.
`
`Inverse
`
`cosine
`
`transform
`
`(14)
`
`(I OCT,
`
`inverse_discrete_cosine_transform): The data handled within each
`
`block pertain to the frequency domain, this inverse cosine transform
`
`15
`
`allows to return to the samples of the space domain. Once the data in
`
`the IDCT have been transformed, pixels, colors and color corrections
`
`can be obtained.
`
`Motion compensation (15) allows to correct some errors
`
`generated before
`
`the decoding stage of MPEG
`
`format, motion
`
`20
`
`compensation takes as a reference a previous frame and calculates a
`
`motion vector relative to the pixels (it can calculate up to four vectors),
`
`and uses them to create a new image. This motion compensation is
`
`applied to the P and 8 type images, where the image position is
`
`located over a "t" time from the reference images. Additionally to the
`
`25
`
`motion compensation, the error correction is also applied, as it is not
`
`enough to predict the position of a particular pixel, but a change in its
`
`color can also exist. Thus, the decoded image is obtained (16).
`
`To decode a P or 8 type image, the reference image is
`
`IPR2018-00534
`Sony EX1005 Page 16
`
`
`
`CA 02557534 2006-08-25
`11
`
`taken, the motion vectors are algebraically added to calculate the next
`
`image, and finally the error correction data is applied, thus generating
`
`the decoded image successfully. Actually, in the video_sequence, two
`
`interdependent video signals exist, "R-L= delta, the delta difference is
`
`5
`
`that stored as a 8
`
`type stereoscopic pair frame with TDVision®
`
`identifier and which is constructed at the moment of decoding by
`
`differences from the image. This is, R-delta= L and L-delta= R, the left
`
`image is constructed from the difference with the right image, which in
`
`turn is constructed from the difference with the left image.
`
`10
`
`The previous process is outlined in such a way that the left
`
`or right signal is taken, both are stored in a temporary buffer, then the
`
`difference between the left and right signals is calculated, and then it is
`
`coded as a 8 type image stored in the video_sequence to be later
`
`decoded by differences from said image.
`
`15
`
`In the decoding process it can be deducted that the data
`
`inputted by the VLC stage are much smaller than the data outputted by
`
`the same stage.
`
`MPEG video sequence structure: This is the maximum
`
`structure used in the MPEG2 format and has the following format:
`
`20
`
`Video sequence (Video_Sequence)
`
`Sequence header (Sequence_Header)
`
`Sequence extension (Sequence_Extension)
`
`User Data (0) and Extension (Extension_and_User_Data
`
`(0))
`
`25
`
`Image group header (Group_of_Picture_Header)
`
`User Data (1) and Extension (Extension_and_User _Data
`
`( 1))
`
`Image header (Picture_Header)
`
`IPR2018-00534
`Sony EX1005 Page 17
`
`
`
`CA 02557534 2006-08-25
`12
`
`Coded image extension (Picture_Coding_Extension)
`
`User Data (2) and Extensions (Extension_and_User_Data
`
`(2))
`
`Image Data (Picture_Data)
`
`5
`
`Slice( Slice)
`
`Macroblock (Macroblock)
`
`Motion vectors (Motion_ Vectors)
`
`Coded Block Pattern (Coded_Biock_Pattern)
`
`Block (Block)
`
`10
`
`Final Sequence Code (Sequence_end_Code)
`
`These structures make up the video sequence. A video
`
`sequence is applied for MPEG format, in order to differentiate each
`
`version there must be a validation that immediately after the sequence
`
`header,
`
`the sequence extension is present; should the sequence
`
`15
`
`extension not follow the header, then the stream is in MPEG1 format.
`
`At
`
`the
`
`beginning
`
`of
`
`a
`
`video
`
`sequence,
`
`the
`
`sequence_header
`
`and
`
`sequence_extension
`
`appear
`
`in
`
`the
`
`video_sequence. The sequence_extension
`
`repetitions must be
`
`identical on the first try and the "s" repetitions of the sequence_header
`
`20
`
`vary little compared to the first occurrence, only the portion defining
`
`the quantization matrixes should change. Having sequences repetition
`
`allows a random access to the video stream, i.e., if the decoder wants
`
`to start playing at the middle of the video stream this may be done, as
`
`it only needs to find the sequence_header and sequence_extension
`
`25
`
`prior to that moment in order to decode the following images. This also
`
`happens for video streams that could not start from the beginning, such
`
`as a satellite decoder turned on after the transmission time.
`
`The full video signal coding-decoding process is comprised
`
`IPR2018-00534
`Sony EX1005 Page 18
`
`
`
`CA 02557534 2006-08-25
`13
`
`of the following steps:
`
`Digitizing the video signals, which can be done in NTSC,
`
`PAL or SECAM format.
`
`Storing the video signal in digital form
`
`5
`
`Transmitting the signals
`
`Recording the digital video stream in a physical media
`
`(DVD, VCD, MiniDV)
`
`Receiving the signals
`
`Playing the video stream
`
`Decoding the signal
`
`Displaying the signal
`
`10
`
`It is essential to double the memory to be handled by the
`
`adequate DSP and have the possibility of disposing of up to 8 output
`
`buffers, which allow the previous and simultaneous representation of a
`
`15
`
`stereoscopic image on a device such as TDVision®'s 30Visor®
`
`Actually, two channels must be initialized when calling the
`
`programming API of the DSP as, by example, the illustrative case of
`
`the Texas Instruments TMS320C62X DSP.
`
`MPEG2VDEC_create
`
`(canst
`
`IMPEG2VDEC_fxns*fxns,
`
`20
`
`canst MEPG2VDEC_Params* params).
`
`Where IMPEG2VDEC_fxns y MEPG2VDEC_Params are
`
`pointer structures defining the operation parameters for each video
`
`channel, e.g.:
`
`3Dlhandle=MPEG2VDEC_create (fxns3DLEFT,Params3DLEFT).
`
`25
`
`3DRhandle=MPEG2VDEC_create(fxns3DRIGHT,Params3DRIGHT.
`
`Thereby enabling two video channels to be decoded and
`
`obtaining
`
`two video handlers, one
`
`for the
`
`left-right stereoscopic
`
`IPR2018-00534
`Sony EX1005 Page 19
`
`
`
`CA 02557534 2006-08-25
`14
`
`channel.
`
`A double display output buffer is needed and by means of
`
`software, it will be defined which of the two buffers must display the
`
`output by calling the AP function:
`
`5
`
`Namely,
`
`MPEG2VDEC_APPL Y(3DRhandle,
`
`inputR1,
`
`inputR2, inputR3, 3doutright_pb, 3doutright_fb ).
`
`MPEG2VDEC_APPLY(3Dlhandle,
`
`inputl 1,
`
`inputl2,
`
`inputl3, 3doutleft_pb, 3doutleft_fb).
`
`This same procedure can be implemented for any DSP,
`
`10
`
`microprocessor or electronic device with similar functions.
`
`Where 3Dlhandle is the pointer to the handle returned by
`
`the DSP's
`
`create
`
`function,
`
`the
`
`input1
`
`parameter
`
`is
`
`the
`
`FUNC_DECODE_FRAME or FUNC_START_PARA address, input2 is
`
`the pointer to the external input buffer address, and input3 is the size
`
`15
`
`of the external input buffer size.
`
`3doutleft_pb is the address of the parameter buffer and
`
`3doutleft_fb is the beginning of the output buffer where the decoded
`
`image