(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2003/0112327 A1
`Jeong et al.
`(43) Pub. Date:
`Jun. 19, 2003
`
`US 2003O112327A1
`
`(54) CAMERA INFORMATION
`CODING/DECODING METHOD FOR
`SYNTHESIZING STEREOSCOPIC REAL
`VIDEO AND A COMPUTER GRAPHIC
`IMAGE
`(76) Inventors: Se Yoon Jeong, Taejon (KR); Byung
`Tae Chun, Taejon (KR); Kyu Seo Han,
`Taejon (KR); Ho Sub Yoon, Taejon
`(KR); Young Lae Bae, Taejon (KR)
`
`Correspondence Address:
`JACOBSON, PRICE, HOLMAN & STERN
`PROFESSIONAL LIMITED LIABILITY
`COMPANY
`400 Seventh Street, N.W.
`Washington, DC 20004 (US)
`
`(21) Appl. No.:
`(22) Filed:
`(30)
`
`10/121,583
`Apr. 15, 2002
`Foreign Application Priority Data
`
`Dec. 17, 2001 (KR)....................................... 2001-7992O
`
`Publication Classification
`
`(51) Int. Cl." ......................... H04N 13/02; H04N 5/225;
`HO4N 5/228
`(52) U.S. Cl. .......................... 348/47; 348/53; 348/207.1;
`348/208.12; 348/208.16; 348/164
`
`ABSTRACT
`(57)
`A camera information coding/decoding method for Synthe
`sizing Stereoscopic real Video and a computer graphic image
`is disclosed. The method Stores the camera information
`generated when picturing, which is not contained in the
`existing multimedia contents coding Standard, that is, the
`location motion information of the camera and the camera
`Viewpoint displacement information, together with contents
`as the camera motion information Stream. The camera
`information obtained through the camera position Sensor and
`the camera Viewpoint displacement Sensor when picturing
`image is coded and Stored together with the contents,
`thereby easily and precisely Synthesizing with the computer
`graphic image. In addition, Since the information may be
`easily transformed to the descriptor of the camera motion
`information of MPEG-7 format, there is another advantage
`of easily carrying out camera motion information indexing
`operation among Searching methods of the multimedia con
`tentS.
`
`
`
`
`
`
`
`
`
`51)
`
`530
`
`SYNC SIGNAL
`GENERATING
`
`540
`
`OpenGL
`RENDERER
`
`
`
`PROJECTION
`PANEL
`
`
`
`550
`
`
`
`St. VIEWPOINT
`peria VIEWPOINT
`
`O
`
`O
`
`IPR2018-01045
`Sony EX1012 Page 1
`
`

`

`Patent Application Publication Jun. 19, 2003 Sheet 1 of 7
`
`US 2003/0112327 A1
`
`F.G. 1A
`
`F.G. 1B
`
`
`
`IPR2018-01045
`Sony EX1012 Page 2
`
`

`

`Patent Application Publication Jun. 19, 2003 Sheet 2 of 7
`
`US 2003/0112327 A1
`
`F. G. 1 C
`
`
`
`IPR2018-01045
`Sony EX1012 Page 3
`
`

`

`Patent Application Publication Jun. 19, 2003 Sheet 3 of 7
`
`US 2003/0112327 A1
`
`F. G. 2A
`
`
`
`TILT UP
`
`PAN
`
`ROLL
`
`N- PAN LEFT
`TLT DOWN
`
`BOOMUP
`
`
`
`TRACK RIGHT
`
`DOLLY BACKWARD
`
`DOLLY FORWARD
`
`TRACK EFT
`
`BOOK DOWN
`
`IPR2018-01045
`Sony EX1012 Page 4
`
`

`

`Patent Application Publication Jun. 19, 2003. Sheet 4 of 7
`
`US 2003/0112327 A1
`
`FG 3
`
`
`
`CAMERA INFORMATION OF
`PRESENT FRAME (t)
`
`310
`
`
`
`
`
`F. G. 4
`
`CAMERA INFORMATION OF
`PREVIOUS FRAME (t–1)
`
`350
`
`410
`
`420
`
`
`
`
`
`DECODER
`
`CAMERADISPLACEMENT
`INFOROSETEENERAES
`
`CAMERA NFORMATION OF
`PRESENT FRAME (t)
`
`CAMERA. INFORMATION OF
`PREVIOUS FRAME (t–1)
`
`
`
`
`
`IPR2018-01045
`Sony EX1012 Page 5
`
`

`

`Patent Application Publication Jun. 19, 2003 Sheet 5 of 7
`
`US 2003/0112327 A1
`
`F.G. 5
`
`USER
`
`510
`
`
`
`
`
`
`
`530
`
`SYNC SGNAL
`GENERATING
`
`PROJECTION
`PANEL
`
`
`
`
`
`
`
`
`
`O
`
`LEFT VEWPOINT
`
`IPR2018-01045
`Sony EX1012 Page 6
`
`

`

`Patent Application Publication Jun. 19, 2003 Sheet 6 of 7
`
`US 2003/0112327 A1
`
`F.G. 6
`
`CameraLOCationInfo
`
`S600
`
`ReSet= TRUE 2
`
`
`
`Left ViewOOint
`Leyex Cameralocation info. DeltaVector 0.
`Leye7= CameraLocation info. DeltaVector 1,
`Leyey= CameraLocation info. DeltaVector 2)
`Right ViewPoint
`ReyeX LeyeX + DNormal. X,
`Reyey Leyey
`DNor Inay,
`ReyeZF LeyeZ + DNormal, Z;
`
`Left Viewpoint
`Leyex-PLeyex + CameraLOcation info.DeltaVector 0.
`Leyey-PLeyey + CameraLocation info. DeltaVector 1)
`Leyez-PLeyez + CameraLOcation info. DeltaVector 2):
`Right ViewPoint
`Reyex-PReyeX + CameraLOcationInfo.De tavector 0,
`Reyey-PReyey + at:a::RS:
`Reyez=PReyez + CameraLOcation info. DeltaVector 2
`
`RENEW PREVIOUS VALUE
`PLeyeX, PLeyey, PLeyey,
`PReyeX, PReyey, PReyez,
`
`
`
`
`
`
`
`COMPENSATION TRANSFORM TO
`COCRDNATE SYSTEM
`
`S640
`
`Left Viewpoint (Leyex, Leyey, Leye7)
`Right Viewpoint (ReyeX, Reyey, Reye7)
`
`
`
`S650
`
`IPR2018-01045
`Sony EX1012 Page 7
`
`

`

`Patent Application Publication Jun. 19, 2003 Sheet 7 of 7
`
`US 2003/0112327 A1
`
`F. G. 7
`
`
`
`CameraViewMovelnfo
`
`S700
`
`Reset= TRUE 2
`
`Center XF
`CameraViewlovenfo. TiltingAngle
`Centery=
`CameraViewMovelnfo. Panning Angle
`Center ZF
`CameraviewMove info.Zooming Angle
`
`ROTATE AROUND THE X-AXS BY A
`TILTING ANGLE, AND THE Y-AXIS
`BY A PANNING ANGLE
`PCenter--Center
`PUp--Up,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`PCenter X= CenterX
`PCenter X= CenterX
`PCenter X= CenterX
`PUOXFUpX, PUpy-Upy, PUD7-UOZ
`
`
`
`S730
`
`COORDINATE SYSTEM COMPENSATION
`TRANSFORM CALCULATE DNORMAL VECTOR
`
`740
`S
`
`S750
`
`Left Viewpoint
`CenterX, LCentery, LCeter Z
`Right Viewpoint
`RCenter X= LCenterxt Dnormal, X
`Roentery= LCentery+ Dnormal.y
`RCenter ZF LCenter Zt Dnormal.Z.
`
`IPR2018-01045
`Sony EX1012 Page 8
`
`

`

`US 2003/O112327 A1
`
`Jun. 19, 2003
`
`CAMERA INFORMATION CODING/DECODING
`METHOD FOR SYNTHESIZING STEREOSCOPIC
`REAL VIDEO AND A COMPUTER GRAPHIC
`IMAGE
`
`BACKGROUND OF THE INVENTION
`0001) 1. Field of the Invention
`0002 The present invention relates to a camera informa
`tion coding/decoding method for Synthesizing a Stereoscopic
`real Video and a computer graphic image, and more particu
`larly, to a camera information coding/decoding method for
`Synthesizing a Stereoscopic real Video and a computer
`graphic image, in which camera information required for
`Synthesizing the Stereoscopic real Video and the computer
`graphic image is coded upon picturing , and is Stored
`together with contents, thereby shortening a time of calcu
`lating the camera information Separately and allowing the
`recorded value to be used.
`0003 2. Background of the Related Art
`0004. The most general standard among methods of
`coding digital moving picture contents comprises MPEG-1
`format, MPEG-2 format, MPEG-4 format, and MPEG-7
`format.
`0005. The MPEG-1 format is a standard method of
`Storing data in a storage medium of a constant bit rate, and
`the MPEG-2 format is a standard for a digital broadcast of
`high quality, the formats being used in a digital Video disc
`(DVD) and HDTV, respectively.
`0006. The MPEG-4 format is a new concept of multime
`dia coding Standard in which an object-oriented idea is
`firstly introduced. The MPEG-4 format is divided into a
`Scene portion and an object portion. While the conventional
`Standards are divided into an audio portion and a video
`portion for protecting it, respectively, in View of a simple
`moving picture, the MPEG-4 format divides the moving
`picture more closely. Specifically, each of the audio and
`Video is divided by an object, and each object is coded
`through an optimized method. And then, constructional
`information of the Scene consisting of objects is separately
`coded, thereby increasing the efficiency.
`0007. The MPEG-4 format consists of a visual portion,
`an audio portion, and a System portion. From now on, it is
`expected that the multimedia contents coding Standards are
`applied with these concepts, and are continuously devel
`oped. In view of these points, the inventors of the present
`invention proposed a method of Storing the camera infor
`mation as a camera stream object. Moreover, the MPEG-7
`format is a Searching information coding Standard for a
`Search.
`0008. The information of the camera happened upon
`picturing is not contained in the multimedia coding Stan
`dards, as well as the MPEG formats. Even though the
`MPEG-7 format has the camera information, it is a method
`of estimating camera motion through an image processing
`technique of the Video image and Storing the estimated
`resultant. Since the above method contains only information
`adapted for Searching, there is a deficiency in Synthesizing
`the actual image with the computer graphics.
`0009. In order to acquire viewpoint information for syn
`thesizing the actual image with the computer graphics, it
`
`requires a camera location and a viewpoint upon picturing.
`Since the existing multimedia coding Standards do not
`contain Such information, the environment of the picturing
`time is reversely estimated to calculate Such information and
`then Synthesizing the image. The operation of estimating the
`camera information is very difficult, So it is hard to acquire
`the correct resultant.
`0010. At present, the contents manufacture of synthesiz
`ing the actual image with computer graphics is increased.
`For example, most of movies employ the computer graphics,
`and application of virtual information is significantly
`increased in Sportscasts. At that time, a memory header is
`used to acquire the camera information of the picturing time.
`On broadcasting a Sports game, a Score of interested game
`or information of a player is Synthesized with the actual
`image and is displayed. In addition, Since there is no a
`coding Standardizing method of the information of the
`memory header at present, it is difficult to Store the infor
`mation and apply it. AS the image which is recorded, if
`necessary, is edited and reused, if the pictured image is
`recorded together with the camera information of the pic
`tured time, it is possible to easily manufacture other contents
`by Synthesizing the actual image with a new computer
`graphics, if necessary.
`0011) Another disadvantage of the existing memory
`header is that Since the information of the camera location
`displacement can not measured, the location of the camera
`has to be remained Stationary. In case the camera location is
`displaced, it requires other Sensors for measuring the loca
`tion movement of the camera.
`
`SUMMARY OF THE INVENTION
`0012. Accordingly, the present invention is directed to a
`camera information coding/decoding method for Synthesiz
`ing Stereoscopic real Video and a computer graphic image
`that Substantially obviates one or more problems due to
`limitations and disadvantages of the related art.
`0013 An object of the present invention is to provide a
`camera information coding/decoding method for Synthesiz
`ing Stereoscopic real Video and a computer graphic image, in
`which camera information required for Synthesizing the
`Stereoscopic real Video and the computer graphic image is
`coded upon picturing, and is Stored together with contents,
`thereby shortening a time of calculating the camera infor
`mation Separately and allowing the recorded value to be
`used.
`0014) Specifically, since the camera information of the
`picturing time is not stored in the digital moving picture
`multimedia coding Standard, the Viewpoint information
`required for Synthesizing the actual image with the computer
`graphic image is separately estimated and calculated. There
`is a drawback that the operation of estimating the viewpoint
`information is very difficult, and it is hard to acquire the
`correct resultant. In order to solve the drawback, if the
`camera location information and the viewpoint information
`are coded and Stored when coding the moving picture, the
`Synthesizing operation of the computer graphics is easily
`implemented, and it is possible to acquire the correct result
`ant.
`0015. In addition, the present invention may be applied in
`the case, that the camera location is displaced, by adding the
`
`IPR2018-01045
`Sony EX1012 Page 9
`
`

`

`US 2003/O112327 A1
`
`Jun. 19, 2003
`
`camera location information by mounting a location Sensor
`on a memory header which is applied to a Stationary camera
`only. In View of manufacturing the contents Stereoscopically
`to increase the objective description when Synthesizing the
`real image with the computer graphics, there is provided a
`camera information coding/decoding method adapted for the
`Stereoscopically pictured image.
`0016 To achieve the object and other advantages,
`according to one aspect of the present invention, there is
`provided a camera information coding method for Synthe
`sizing a Stereoscopic real Video and a computer graphic
`image, the method comprising the Steps of detecting and
`inputting a camera information on a corresponding frame of
`an image being pictured at present, calculating each com
`ponent difference between the camera information on an
`inputted present (t) frame and a camera information on a just
`previous (t-1) frame to acquire a camera displacement
`information between the frames, coding the camera dis
`placement information between the frames to compress and
`Store pictured image contents and the coded camera infor
`mation; and updating the coded camera information on the
`corresponding frame as the camera information on the
`previous frame. The camera information comprises at least
`one of a camera location information (x, y, z) and a camera
`Viewpoint information. The camera viewpoint information
`comprises at least one of a tilting angle information of the
`camera, a panning angle information of the camera, a
`Zooming angle of the camera. The camera location infor
`mation comprises at least one of a tracking information of
`the camera, a dolling information of the camera, and a
`booming information of the camera.
`0.017. The coding step may employ a Huffman or LZW
`coding method, or an arbitrary entropy coding method.
`0.018. According to another aspect of the present inven
`tion, there is provided a camera information decoding
`method for Synthesizing a Stereoscopic real Video and a
`computer graphic image, the method comprising the Steps
`of if a coded camera information of a current frame at a
`camera information Stream is inputted, decoding the input
`ted camera to acquire a camera displacement information
`between the frames, adding the camera displacement infor
`mation and a camera information of a previous frame to
`calculate a location vector, Viewpoint vector, and upward
`vector information of the camera of the current frame;
`updating the calculated location vector, Viewpoint vector,
`and upward vector information of the camera of the current
`frame as the camera information on the previous frame, and
`Synthesizing the Stereoscopic real Video and the computer
`graphic image by use of the location vector, Viewpoint
`vector, and upward vector information of the camera of the
`current frame.
`0019. The step of calculating the location vector infor
`mation of the camera comprises the Steps of determining
`whether a bReset value of the camera location information
`(CameraLocationInfo) of the current frame is true, and if the
`bReset value is false, adding a DeltaVector of the Camer
`aLocationInfo to a location vector of the previous frame, to
`acquire a location vector of the camera; updating the loca
`tion vector of the camera as a previous value for a next
`calculation; carrying out coordinate System compensation
`transform of the location vector of the camera for compen
`Sating an origin location variation between origins upon
`
`implementing the computer graphic Synthesis, Supposing
`that the origin is the location of an initial location of the
`camera; and acquiring locations of left and right cameras
`which are required for Setting the Viewpoint of the current
`frame by carrying out the coordinate System compensation
`transform.
`0020. The method further comprises the steps of: if the
`bReset value of CameraLocationInfo of the current frame is
`true, using the CameraLocation information as the location
`of the left camera as, and calculating a normal vector
`(Dnormal) by carrying out the coordinate System compen
`sation transform of the location information of the camera to
`acquire a location vector of the camera as the location of the
`right camera; updating the location vector of the camera as
`a previous value for a next calculation; carrying out coor
`dinate System compensation transform of the location vector
`of the camera for compensating an origin location variation
`between origins upon implementing the computer graphic
`Synthesis, Supposing that the origin is the location of an
`initial location of the camera; and acquiring locations of left
`and right cameras which are required for Setting the View
`point of the current frame by carrying out the coordinate
`System compensation transform.
`0021. The normal vector (Dnormal) is parallel with a
`normal vector of a plane formed by a left viewpoint vector
`Center and an upward direction vector Up, and has a
`magnitude corresponding to a distance D between lenses of
`tWO cameras.
`0022. The method further comprises the step of: to con
`duct Zooming when Synthesizing the Stereoscopic real Video
`and the computer graphic image, Storing a camera constant
`information used when matching Spaces of the actual image
`and the computer graphic image, a distance information
`between two lenses used when the right camera information
`is acquired from the left camera information, and a lens
`information for determining a rotating amount of a Zoom
`controller upon changing from the minimum distance to the
`maximum distance of the lens.
`0023 The camera constant information comprises at least
`one of the number of Sensors in an X-axis direction of the
`camera, the number of pixels in an X-axis direction of the
`camera, a size per Sensor in an X-axis direction of the
`camera, a size per Sensor in a y-axis direction of the camera,
`a size per X-axis pixel, a Size per y-axis pixel, a center point
`of X-axis, a center point of y-axis, and Scale element infor
`mation.
`0024. According to still another aspect of the present
`invention, there is provided a recording medium for execut
`ing a camera information coding method for Synthesizing a
`Stereoscopic real Video and a computer graphic image, the
`recoding medium capable of being read by a digital proces
`Sor, and Storing a program of commands executed by the
`digital processor, the program being implemented by types,
`with the program comprising the Steps of detecting and
`inputting a camera information on a corresponding frame of
`an image being pictured at present, calculating each com
`ponent difference between the camera information on an
`inputted present (t) frame and a camera information on a just
`previous (t-1) frame to acquire a camera displacement
`information between the frames, coding the camera dis
`placement information between the frames to compress and
`Store pictured image contents and the coded camera infor
`
`IPR2018-01045
`Sony EX1012 Page 10
`
`

`

`US 2003/O112327 A1
`
`Jun. 19, 2003
`
`mation; and updating the coded camera information on the
`corresponding frame as the camera information on the
`previous frame.
`0.025 According to further still another aspect of the
`present invention, there is provided a recording medium for
`executing a camera information decoding method for Syn
`thesizing a Stereoscopic real Video and a computer graphic
`image, the recoding medium capable of being read by a
`digital processor, and Storing a program of commands
`executed by the digital processor, the program being imple
`mented by types, with the program comprising the Steps of:
`if a coded camera information of a current frame at a camera
`information Stream is inputted, decoding the inputted cam
`era to acquire a camera displacement information between
`the frames, adding the camera displacement information and
`a camera information of a previous frame to calculate a
`location vector, Viewpoint vector, and upward Vector infor
`mation of the camera of the current frame, updating the
`calculated location vector, Viewpoint vector, and upward
`vector information of the camera of the current frame as the
`camera information on the previous frame; and Synthesizing
`the Stereoscopic real Video and the computer graphic image
`by use of the location vector, Viewpoint vector, and upward
`vector information of the camera of the current frame.
`0026. It is to be understood that both the foregoing
`general description and the following detailed description of
`the present invention are exemplary and explanatory and are
`intended to provide further explanation of the invention as
`claimed.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`0027. The accompanying drawings, which are included
`to provide a further understanding of the invention and are
`incorporated in and constitute a part of this application,
`illustrate embodiment(s) of the invention and together with
`the description Serve to explain the principle of the inven
`tion. In the drawings:
`0028 FIG. 1 is a view of a stereoscopic camera with a
`memory header and a location Sensor mounted on a camera
`stand, in which FIG. 1a is a front view of the stereoscopic
`camera, FIG. 1b is a top plan view, and FIG. 1c is a
`perspective view;
`0029 FIG. 2 is a view explaining camera motion opera
`tion. FIG. 2a is a view explaining a viewpoint change
`motion of camera, and FIG. 2b is a view explaining a
`location change motion of camera;
`0030 FIG. 3 is a view illustrating the encoding process
`of the camera motion information of a current frame;
`0.031
`FIG. 4 is a view illustrating the decoding process
`of camera motion information of a current frame;
`0032 FIG. 5 is a view illustrating a PC-based stereo
`Scopic image display appliance;
`0.033
`FIG. 6 is a flowchart showing a process of calcu
`lating position vectors of a camera for a present left and right
`frame, and
`0034 FIG. 7 is a flowchart showing a process of calcu
`lating a viewpoint vector and upward direction vector of
`present left and right frames.
`
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENTS
`0035 A camera information coding/decoding method for
`Synthesizing Stereoscopic real Video and computer graphic
`image according to one preferred embodiment of the present
`invention will now be explained with reference to the
`accompanying drawings.
`0036 FIG. 1 is a view of a stereoscopic camera with a
`memory header and a location Sensor mounted on a camera
`stand, in which FIG. 1a is a front view of the stereoscopic
`camera, FIG. 1b is a top plan view, and FIG. 1c is a
`perspective view.
`0037. The stereoscopic camera shown in FIG. 1 gener
`ally employs two cameras, with the cameras being arranged
`in Such a manner that optical axes of these camera are
`Substantially parallel. A distance D between these cameras is
`about 5 to 10 centimeters corresponding to a distance
`between eyes of an ordinary perSon.
`0038. Upon taking a picture, a unit for measuring infor
`mation of camera motion in real time is called as a memory
`header. The memory header consists of a camera viewpoint
`motion Sensor 122 and a Zooming controller 121. The
`Zooming controller 121 comprises a Zooming Sensor.
`0039 The motion sensor 122 is a sensor for measuring
`tilting information and panning information, and comprises
`a tilting Sensor and a panning Sensor. Two Zooming control
`lerS 121 are employed in the Stereoscopic camera.
`0040. The motion morphology of the Stereoscopic cam
`era will be considered with reference to FIG. 2. FIG. 2 is a
`View explaining a camera motion operation of the Stereo
`scopic camera in FIG. 1, in which FIG. 2a is a view
`explaining a viewpoint change motion of the Stereoscopic
`camera, and FIG. 2b is a view explaining a location change
`motion of the Stereoscopic camera.
`0041. In the camera motion, there are two types: one is
`that the location of the camera is fixed, while a picturing
`direction, i.e., the Viewpoint is altered; and the other is that
`the viewpoint is maintained in parallel, while the location of
`the camera is moved.
`0042 FIG. 2a is a view illustrating a kind of the view
`point change motion of the camera. A term "panning” means
`that a cameraman rotates the camera Stand from Side to Side
`by use of a handle, thereby changing the camera Viewpoint
`left and right. A term "tilting” means that the cameraman
`rotates the camera Stand up and down by use of the handle,
`thereby changing the camera viewpoint upward and down
`ward. And, a term "Zooming” means that a Zoom lens is
`operated by manipulating the Zooming controller 121
`attached on the handle of the camera.
`0043. Since the motion information of the camera view
`point is represented by an angle, Sensors for obtaining the
`information are to measure the angle. In other words, the
`memory header may consist of a plurality of Sensors for
`measuring information of rotating angle in each direction.
`0044) Again explaining the function of each sensor in
`brief, the panning Sensor is to measure a horizontal rotating
`angle of the camera Stand, while the tilting Sensor is to
`measure a vertical rotating angle of the camera Stand. Two
`Sensors are mounted in the camera Stand 122, as shown in
`FIG. 1.
`
`IPR2018-01045
`Sony EX1012 Page 11
`
`

`

`US 2003/O112327 A1
`
`Jun. 19, 2003
`
`004.5 The Zooming sensor is a sensor provided in the
`Zooming controller 121, and is to measure an angle rotated
`of the Zoom lens for adjusting a focus distance of the lens.
`In case of the Stereoscopic camera, two Zooming controllers
`121 are employed, as shown in FIG. 1. Specifically, the
`Zooming controller and the Zooming Sensor of the memory
`header of the Stereoscopic camera differ from those of a
`memory header of a typical camera.
`0.046 A device capable of identically adjusting the lenses
`of two cameras must be employed. Therefore, it requires a
`device capable of Simultaneously regulating the Zooming by
`equal amount. Since measured values of two Zooming
`Sensors are identical to each other, only one value has to be
`Stored. The coincidence of these units can be realized by use
`of a digital motor Such as a stepping motor.
`0047. In order to synthesizing the pictured image and the
`computer graphic image, it requires transforming informa
`tion of an image plane and an actual three-dimensional
`Space. The operation of acquiring the information is called
`as camera calibration. To the end, it requires three-dimen
`Sional location information of the camera and information of
`the lens. In order to acquire the location information of the
`camera, a location Sensor 123 is provided.
`0.048. The location sensor 123 serves as a role of mea
`Suring relative coordinates on a reference point. Upon Syn
`thesizing, the relative coordinates are easily calculated as
`actual coordinates by transforming the reference point to a
`reference point of a Synthesized image.
`0049 FIG.2b is a view explaining the information of the
`camera location motion, in which a term “tracking means
`that the camera moves Side by Side, a term “booming” means
`that the camera moves up and down, and a term "dolling”
`means that the camera moves back and forth. This informa
`tion is acquired on the basis of a measured value of the
`location information change of the location Sensor 123 in
`FIG. 1. In view of the measured value of the location sensor
`123, if a value of the location sensor 123 is represented by
`three-dimensional value of x, y and Z-coordinates, the track
`ing may be represented by movement of left/right directions,
`i e., an X-axis direction, of the location information, the
`booming may be represented by movement of up/down
`directions, i.e., a y-axis direction, and the dolling may be
`represented by movement of back/forth directions, i.e., a
`Z-axis direction.
`0050 For reference, according to the present invention,
`the camera location Sensor 123 may be replaced with a
`common moving object three-dimensional location measur
`ing device may be the camera location Sensor 123.
`0051. The camera information has to be recorded every
`frame of video. In order to represent it, FrameCamerainfo
`descriptor is devised.
`0.052 FrameCamerainfo {
`0053 CameraViewMoveInfo;
`0054 CameraLocationInfo
`0055) }
`0056. The descriptor consists of CameraViewMoveInfo
`and CameraLocation Info.
`0057 CameraViewMove.Info {
`0.058 Bool bReset;
`0059 Float TiltingAngle;
`
`0060 Float PanningAngle;
`0061
`Flost ZooningAngle
`0062) }
`0063 CameraLocationInfo {
`0064.) Bool bReset;
`0065 Float Delta Vector.3
`0066) }
`0067. The camera location information and the angle
`information are required every frame, and it is preferable to
`record a difference value between the current frame and a
`previous frame, in order to increase the coding efficiency. In
`View of the above State, a process of calculating the camera
`location and Viewpoint at the current frame, i.e., a process of
`coding the camera motion information on the current frame
`will now be explained with reference to FIG. 3. FIG. 3 is
`a view illustrating the encoding process of the camera
`motion information on a current frame.
`0068 Camera information 310 of the current frame is
`inputted from the Sensor. At that time, the Sensor may be a
`motion Sensor Such as a panning Sensor, a tilting Sensor, a
`Zooming Sensor or the like.
`0069. The camera information inputted from the sensor
`comprises a camera location information (x, y, z) and the
`camera viewpoint information (tilting angle, panning angle,
`Zooming angle). It is noted that the horizontal rotation is
`represented by a rotation around the y-axis, while the
`Vertical rotation is represented by a rotation around the
`X-axis. Accordingly, the tilting angle corresponds to an X
`component, and the panning angle corresponds to a y
`component.
`0070 A difference between the camera information on
`the present (t) frame inputted from the Sensor and the camera
`information on the just previous (t-1) frame is calculated by
`a Subtractor 320 to acquire a camera displacement informa
`tion 330 between the frames. The acquired camera displace
`ment information between the frames is coded by a encoder
`340. At that time, an arbitrary coding method such as a
`Huffman coding method or a LZW coding method may be
`employed. However, the coding method is identical to that
`of a decoder 410 shown in FIG. 4.
`0071. If the coding is completed, the encoder 340 updates
`the coded frame camera information as the camera infor
`mation on the previous frame, So that if camera information
`on a new frame is inputted, the updated frame camera
`information is used as the camera information on the pre
`vious information. The coded camera information is Syn
`chronized with the Stereoscopic motion picture and is Stored.
`0072 Such camera displacement information is repre
`sented by the CameraViewMovenfo descriptor and the
`Cameral locationInfo descriptor. The represented informa
`tion is compressed and stored by the encoder 340 on
`recording.
`0073. In order to increase error tolerance of the camera
`information, it is preferable to restart the difference calcu
`lation of the camera information after certain time interval.
`In other words, the difference calculation is restarted every
`
`IPR2018-01045
`Sony EX1012 Page 12
`
`

`

`US 2003/O112327 A1
`
`Jun. 19, 2003
`
`a certain frame unit. In case of restarting the difference, a
`bReset value of the descriptor is set to true, and the other
`case is Set to false.
`0.074.
`In case of restarting the difference calculation, the
`CameraViewMovenfo descriptor is not recorded with not
`the angle information but X, y and Z values of the viewpoint
`vector, respectively.
`0075. The FrameCameranfo descriptor must be synchro
`nized with each frame of the video. To the end, the descriptor
`may be contained in the header of each frame, and then may
`be coded. Otherwise, the descriptor may be recoded in
`Separate Stream, and then may be coded. In case of inserting
`the descriptor in the header, Since the existing Video coding
`device has to be modified, it is preferable to record the
`descriptor in Separate Stream. The Separate camera informa
`tion Stream method is adapted for an object-oriented coding
`Scheme Such as MPEG-4.
`0.076
`FIG. 4 is a view illustrating a decoding process of
`camera information of a current frame. In the camera
`information Stream, the coded information value of the
`current frame is inputted in the decoder 410, and is decoded
`to provide a camera displacement information 420 between
`the frames. At that time, if the bReset value is false, if other
`words, if the operation of the camera is not reset, a value of
`a previous frame 450 is added to the coded camera displace
`ment information between the frames by an adder 430,
`thereby acquiring a current frame camera information 440,
`i.e., the present location information (x, y, z) and the view
`point information (tilting angle, panning angle, and Zooming
`angle) of the camera. For the nest calculation, the value is
`updated as the previous frame camera information.
`0077 FIG. 5 is a view illustrating a PC-based stereo
`Scopic image display appl