Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 1 of 31 PageID #: 48
`
`Exhibit 2
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 2 of 31 PageID #: 49
`
`US007292261B1
`
`(12)
`
`United States Patent
`Te0
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 7,292,261 B1
`Nov. 6, 2007
`
`(54) VIRTUAL REALITY CAMERA
`
`(76) Inventor: Patrick Teo, 7200 Bollinger Rd., #511,
`San Jose’ CA (Us) 95129
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 439 days.
`
`(21) Appl' NO‘: 09/378398
`
`(22) Film:
`
`Aug 20, 1999
`
`(51) Int. Cl.
`(2006-01)
`H04N 7/18
`(200601)
`G03B 35/00
`(52) US. Cl. ....................................... .. 348/36; 396/324
`(58) Field of Classi?cation Search .............. .. 345/660;
`348/36, 42, 333.12; 358/506; 378/71; 396/121,
`396/130, 322, 324; H04N 7/18
`See application ?le for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`8/1988 Robinson et al.
`4,763,280 A
`3/1989 Bunker et al.
`4,811,245 A
`6/1989 Zeno
`4,841,292 A
`6/1991 Cok
`5,022,085 A
`8/1992 Egawa .................. .. 48/333.12
`5,138,460 A *
`5,155,586 A 10/1992 Levy et a1.
`5,175,808 A 12/1992 Sayre
`5,185,808 A
`2/1993 Cok
`5,231,385 A
`7/1993 Gengler et al.
`5,251,022 A 10/1993 Kitamura
`5,262,867 A 11/1993 Kojima
`5,307,451 A
`4/1994 Clark
`5,325,449 A
`6/1994 Burt et a1.
`5,338,200 A
`8/1994 Olive
`5,396,583 A
`3/1995 Chen et a1.
`5,412,764 A
`5/1995 Tanaka
`5,420,533 A
`5/1995 Park
`5,424,773 A
`6/1995 Saito
`5,442,738 A
`8/1995 Chapman et al.
`5,444,478 A
`8/1995 Lelong et al.
`5,446,833 A
`8/1995 Miller et a1.
`
`5,488,674 A
`5,528,290 A
`5,544,283 A
`5,550,959 A
`
`1/1996 Burt et al.
`6/1996 Saund
`8/1996 Kaufman et a1‘
`8/1996 Freeman
`
`8/1996 Shirman et al'
`5,550,960 A
`5,561,756 A 10/1996 Miller et a1~
`5,574,836 A 11/1996 Broemmelsiek
`5,577,176 A 11/1996 Friedman et a1.
`5,579,456 A 11/1996 Cosman
`
`5,583,975 A 12/1996 Naka etal.
`5,586,231 A 12/1996 Florent et al.
`5,586,232 A 12/1996 YOShida
`5,586,246 A 12/1996 Nobori etal.
`5,590,250 A 12/1996 Lamping et al.
`
`(Continued)
`
`OTHER PUBLICATIONS
`
`“Round Shot Model Super 35”, SeitZ, 4 pp., 1997.
`
`(Continued)
`Primary Examiner4Gims Philippe
`(74) Attorney, Agent, or FirmiBerkeley LaW &
`Technology Group LLP
`
`(57)
`
`ABSTRACT
`
`A camera including a camera lens, acquisition circuitry
`receiving images via the camera lens, for acquiring a ?rst
`?eld of vieW When the camera lens is in a ?rst orientation
`and for acquiring a second ?eld of vieW When the camera
`lens is in a second orientation, and a view?nder displaying
`the second ?eld of vieW When the camera lens is in the
`second orientation and displaying at least a portion of the
`?rst ?eld of vieW at least partially composited With the
`second ?eld of vieW.
`
`60 Claims, 15 Drawing Sheets
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 3 of 31 PageID #: 50
`
`US 7,292,261 B1
`Page 2
`
`1/1997
`1/1997
`1/1997
`1/1997
`3/1997
`3/1997
`5/1997
`7/1997
`7/1997
`7/1997
`4/1998
`5/1998
`10/1999
`8/2000
`9/2000
`11/2000
`
`Greggain et al.
`O’Neill
`Sakai et al.
`Florent et al.
`Robertson
`Chen et al.
`Schindler
`Yano et al.
`Burt et al.
`Florent et al.
`Jain et al. ................... .. 348/42
`Chen
`Guedalia et al.
`Yano et al. ................. .. 348/36
`
`Miller ...... ..
`
`.. 348/36
`
`Inoue ....................... .. 396/121
`
`7/2001
`6,256,058 B1 *
`7/2001
`6,268,936 B1 *
`7/2001
`6,269,144 B1 *
`6,304,284 B1 * 10/2001
`6,657,667 B1 * 12/2003
`2002/0175924 A1* 11/2002
`
`Kang et al. ................. .. 348/36
`Truc et al.
`358/506
`Dube et al. ................. .. 378/71
`
`Dunton et al. .............. .. 348/36
`Anderson
`.. 348/36
`
`Yui et al. .................. .. 345/660
`
`OTHER PUBLICATIONS
`
`“Casio QV700 Digital Camera & DP-8000 Digital Photo Printer”,
`Joe Farace, 3 pp., Nov. 6, 1997.
`“Round Shot”, 4pp.
`“Casio QV-200, QV-700”, James Karney, PC Magazine, 2pp., Feb.
`10, 1998.
`“Round Shot Super 35 May 13, 1996”. Bob Erickson, 1 pg.
`“Casio Adds New Camera To Its Lineup”, Kelly Ryer, MacWeek,
`vol. 11, issue 38, 1 pg., Oct. 2, 1997.
`
`* cited by examiner
`
`US. PATENT DOCUMENTS
`
`5,594,676
`5,594,843
`5,594,844
`5,594,845
`5,608,850
`5,613,048
`5,630,037
`5,646,679
`5,649,032
`5,650,814
`5,745,126
`5,748,194
`5,963,213
`6,112,033
`6,118,595
`6,144,804
`
`>>>>>>>>>>>>>>>>
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 4 of 31 PageID #: 51
`
`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 1 0f 15
`
`US 7,292,261 B1
`
`LI60
`
`0
`I
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 5 of 31 PageID #: 52
`
`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 2 0f 15
`
`US 7,292,261 B1
`
`2 I O
`V INITIALIZE COMBINING MODULE
`
`220V
`
`‘'
`ACQUIRE FIRST FRAME
`i
`230
`M COMPOSITE OVERLAY ONTO ‘
`VIEWFINDER FIELD OF VIEW
`i
`240
`ACQUIRE NEXT FRAME
`\—/
`i
`250
`\J COMBINE NEXT FRAME
`
`260
`
`DONE? NO
`
`270
`\J FINALIZE COMBINING MODULE
`
`YES
`
`FIGURE 2
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 6 of 31 PageID #: 53
`
`U.S. Patent
`
`Nov. 6,2007
`
`Sheet 3 or 15
`
`US 7,292,261 B1
`
`m WEDGE
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 7 of 31 PageID #: 54
`
`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 4 0f 15
`
`US 7,292,261 B1
`
`FIGURE 4
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 8 of 31 PageID #: 55
`
`U.S. Patent
`
`Nov. 6,2007
`
`Sheet 5 or 15
`
`US 7,292,261 B1
`
`II‘ T
`
`O O x
`
`P | A XUE 0 CE; U
`
`
`
`, 8m Q8/\
`
`I‘ ‘I
`
`\
`\
`\
`
`
`
`I L \\v\/|\\Q$
`
`03v
`
`////
`
`m mEDmVE
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 9 of 31 PageID #: 56
`
`U.S. Patent
`
`NOV. 6,2007
`
`Sheet 6 6f 15
`
`US 7,292,261 B1
`
`JKL
`
`DEF
`
`FIGURE 6
`
`GHI
`
`ABC
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 10 of 31 PageID #: 57
`
`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 7 or 15
`
`US 7,292,261 B1
`
`710
`
`<——dx
`
`730
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 11 of 31 PageID #: 58
`
`U.S. Patent
`
`Nov. 6,2007
`
`Sheet 8 or 15
`
`US 7,292,261 B1
`
`1 1 FIGURE 8
`
`810
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 12 of 31 PageID #: 59
`
`U.S. Patent
`
`Nov. 6,2007
`
`Sheet 9 or 15
`
`US 7,292,261 B1
`
`one Q
`
`owo
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 13 of 31 PageID #: 60
`
`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 10 0f 15
`
`US 7,292,261 B1
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 14 of 31 PageID #: 61
`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 14 of 31 PageID #: 61
`
`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 11 of 15
`
`US 7,292,261 B1
`
`FIGURE11
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 15 of 31 PageID #: 62
`
`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 12 0f 15
`
`US 7,292,261 B1
`
`owwp
`
`OFNF
`
`NF MEDQE
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 16 of 31 PageID #: 63
`
`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 13 0f 15
`
`US 7,292,261 B1
`
`<2 MEDQE
`
`
`
`
`
`82\ F82 Fu? F82 F5: K82 fag
`
`m2 $30; .\\||
`
`mmmp
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 17 of 31 PageID #: 64
`
`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 14 0f 15
`
`US 7,292,261 B1
`
`031
`m3
`l
`
`wt:
`W
`
`
`
`
`
` a unusa- Z_ awn-“5m EwwEDU “Edi, #
`
`»
`
`3 EDGE
`
`no:
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 18 of 31 PageID #: 65
`
`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 15 0f 15
`
`US 7,292,261 B1
`
`5966
`
`mp wwSQE
`
`82k
`
`f own? fowmp
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 19 of 31 PageID #: 66
`
`US 7,292,261 B1
`
`1
`VIRTUAL REALITY CAMERA
`
`FIELD OF THE INVENTION
`
`The present invention relates to a camera that acquires
`?elds of vieW and combines them together.
`
`BACKGROUND OF THE INVENTION
`
`Panoramic images are used to provide immersive “sur
`round” vieWs of scenes, up to 3600 in extent. Digital
`panoramic images can be archived on Internet servers, for
`access by remote clients. Panoramic images are typically
`texture mapped into a suitable surface geometry, such as a
`cylindrical or a spherical geometry. Texture mapping gen
`erates a “Warped” look to the panorama, and usually con
`verts straight lines into boWed curves.
`Client vieWer softWare enables users to interactively vieW
`panoramic images by navigating through the panorama.
`Speci?cally, for a prescribed vieW WindoW on a client
`computer video display, client vieWer softWare converts a
`selected portion of the panoramic image at a selected level
`of magni?cation from cylindrical or other surface geometry
`to rectilinear geometry, Which is appropriate for typical
`perspective vieWing. The converted portion of the pan
`oramic image is displayed in the vieW WindoW.
`A user interface enables a user to dynamically change the
`selected portion of the panoramic image being vieWed.
`Rapid display of the dynamically changing selected portion
`gives the vieWer a sensation of moving through the pan
`orama, and an experience of being immersed Within the
`surrounding scene. Typically a user interface enables a user
`to select portions of the panoramic image for display by
`indicating shifts and changes in magni?cation of a current
`portion of the panoramic image, rather than by selecting a
`neW portion Without reference to the current portion. Typi
`cally the user interface provides for shifting the selected
`portion up, doWn, left, right, or other directions, and for
`reducing or enlarging the current magni?cation factor, by
`zooming in and out. Such a user interface can include, for
`example, keyboard buttons or mouse controls and move
`ments.
`Panoramic content creation is typically carried out by a
`photographer Who captures multiple photos of a scene, as he
`rotates his camera around in a circle. For typical camera
`lenses such as a 35 mm video camera lens, the photos
`acquired are characterized by a rectilinear geometry. That is,
`there exists a linear correspondence betWeen distances in the
`photo and distances in the scene. Each such photo represents
`a portion of the scene. The photos are digitized by a digital
`scanner or, in a case Where the photographer uses a digital
`camera, the photos are digitized by hardWare Within the
`camera.
`The digital photos are then doWnloaded to a computer,
`and a “stitching” application is run to combine the digital
`photos into a single panoramic image. An example of such
`a stitching application is assignee’s PhotoVista® softWare.
`The photographer can eliminate the combining step by
`capturing a large portion of a scene by using a Wide angle
`lens, or a hemispherical lens such as the Portal lens system
`of Be Here Corporation, or a parabolic lens such as the
`ParaShotTM attachment of CycloV1sion Technologies, Inc.,
`the latter tWo of Which capture a 3600 image in one expo
`sure.
`When a photographer creates a panoramic image by
`combining multiple photos, typically he must be careful to
`ensure that adjacent ?elds of vieW are appropriately aligned
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`as he rotates the camera. When rotating the camera in a
`horizontal plane, such alignment involves providing for
`horizontal overlap betWeen adjacent photos, and minimizing
`vertical displacements outside of the horizontal plane of
`rotation. Similarly, When rotating the camera in a vertical
`plane, such alignment involves providing for vertical over
`lap betWeen adjacent photos, and minimizing horizontal
`displacements outside of the vertical plane of rotation.
`In the ensuing discussion, for the sake of conciseness and
`simplicity of explanation, panoramas in a horizontal plane of
`rotation are described, it being understood that the present
`invention is not limited to panoramas in a horizontal plane
`of rotation.
`When rotating the camera freely in his hand, it is di?icult
`for a photographer to accurately align adjacent ?elds of
`vieW. A reliable Way for a photographer to align adjacent
`?elds of vieW is to mount his camera on a tripod, using a
`tripod bracket that has equi-spaced notches for rotating the
`camera in increments of a ?xed angle. An example of such
`a bracket is the Kaidan KiWiTM tripod head. Typically such
`tripod brackets include one or more spirit levels, so that a
`photographer can adjust the tripod to keep the axis of
`rotation of the camera vertically disposed.
`Use of a computer for combining acquired photos to
`produce a panoramic image has several disadvantages. One
`disadvantage is that the photographer needs to take a com
`puter With him in the ?eld. OtherWise, he cannot create and
`vieW the panoramic image While in the ?eld. There are
`problems associated With both of these situations.
`The ?rst option of taking a computer in the ?eld can be
`dif?cult, because often a photographer has to hike through
`un-trodden ground in order to ?nd “the perfect scene.”
`Carrying both a camera and a computer along With his food
`and gear can be burdensome on the photographer.
`Even When the photographer does take a computer With
`him in the ?eld, he may have to move the camera from its
`axis of rotation in order to doWnload the photos acquired
`thereby. As soon as a camera is moved from its axis of
`rotation, typically the only Way to re-generate a panoramic
`image is to recapture at a neW axis of rotation all of the
`individual photos included in the panorama. For example, if
`one of sixteen frames of a scene comes out bad, the
`photographer can correct the problem and recapture such
`frame as long as the camera has not been moved from its axis
`of rotation. HoWever, once the camera has been moved from
`its axis of rotation, a single frame cannot normally be
`accurately aligned Within previously acquired adjacent
`frames. Thus, even When using a notched tripod bracket, the
`photographer Will normally have to recapture all of the
`photos included in the panoramic image.
`The second option of not creating the panoramic image in
`the ?eld is problematic, because the photographer cannot see
`the quality of his panoramic image While in the ?eld. If he
`subsequently discovers problems With the panoramic image,
`it may very Well be hard, if not impossible, to return to the
`scene in the same favorable Weather and other conditions.
`
`SUMMARY OF THE INVENTION
`
`In a preferred embodiment, the present invention provides
`a camera that acquires multiple frames and combines them
`into a panoramic image Within the camera. The camera of
`the present invention eliminates the need for a photographer
`to use a computer for creating his panoramic image content.
`He simply acquires the photos to be included Within the
`panoramic image, and the camera produces the ?nal product,
`a panoramic image of the scene.
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 20 of 31 PageID #: 67
`
`US 7,292,261 B1
`
`3
`The camera of the present invention assists the photog
`rapher in accurately aligning adjacent ?elds of vieW for his
`photos. This is preferably achieved by compositing a desired
`overlap strip from a previously captured ?eld of vieW onto
`a current ?eld of vieW displayed Within a vieW?nder display
`of the camera. As the photographer rotates the camera, the
`current ?eld of vieW changes accordingly, but the desired
`overlap strip from the previously captured ?eld of vieW does
`not change. When the camera reaches an orientation that
`produces the desired overlap, the overlap strip Within the
`vieW?nder display is seamlessly “matched up” With the
`current ?eld of vieW being displayed. This serves as a cue to
`the photographer to capture the current ?eld of vieW.
`The camera of the present invention also preferably
`includes a playback mechanism enabling a photographer to
`interactively vieW the panoramic image, as it Would appear
`to a user vieWing the panoramic image With client vieWer
`softWare. The playback mechanism is responsive to vieW
`control hardWare Within the camera, that displays appropri
`ate portions of the panoramic image at appropriate levels of
`magni?cation as a photographer dynamically navigates his
`Way through the panoramic image.
`There is thus provided in accordance With a preferred
`embodiment of the present invention a camera including a
`camera lens, acquisition circuitry receiving images via the
`camera lens, for acquiring a ?rst ?eld of vieW When the
`camera lens is in a ?rst orientation and for acquiring a
`second ?eld of vieW When the camera lens is in a second
`orientation, and a vieW?nder displaying the second ?eld of
`vieW When the camera lens is in the second orientation and
`displaying at least a portion of the ?rst ?eld of vieW at least
`partially composited With the second ?eld of vieW.
`There is further provided in accordance With a preferred
`embodiment of the present invention a method for convert
`ing an image stored in a buffer as a ?rst plurality of color
`values at a plurality of pixel locations in a ?rst coordinate
`system, to a corresponding image stored in the buffer as a
`second plurality of color values at the plurality of pixel
`locations in a second coordinate system, the second coor
`dinate system being related to the ?rst coordinate system by
`a non-linear transformation, including the steps of comput
`ing color values at pixel locations along an axis Within the
`buffer, using the non-linear transformation, storing the com
`puted color values in the buffer at the pixel locations along
`the axis, and repeating the computing and storing steps for
`subsequent axes Within the buffer, parallel to the axis.
`There is further provided in accordance With a preferred
`embodiment of the present invention a camera comprising a
`camera lens, acquisition circuitry receiving images via the
`camera lens, for acquiring a ?rst frame When the camera lens
`is in a ?rst orientation and for acquiring a second frame
`When the camera lens is in a second orientation, and com
`bining circuitry for at least partially combining the ?rst
`frame and the second frame into a panoramic image.
`There is further provided in accordance With a preferred
`embodiment of the present invention a camera including a
`camera lens, a memory for storing data for a panoramic
`image, a display for displaying at least a portion of the
`panoramic image, and display control circuitry for selecting
`a portion of the panoramic image to display.
`There is further provided in accordance With a preferred
`embodiment of the present invention a method for combin
`ing a ?rst frame and a second frame, including the steps of
`determining horiZontal and vertical offsets for spatially
`aligning the ?rst and second frames, including the step of
`summing absolute values of color differences betWeen the
`?rst frame and the second frame at a multiplicity of pixel
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`locations, based on trial values for horiZontal and vertical
`offsets, further determining brightness and contrast param
`eters for chromatically aligning the ?rst and second frames,
`and generating a panoramic image, including the step of
`compositing a portion of the second frame onto a portion of
`the ?rst frame, based on the horiZontal and vertical offsets
`and based on the brightness and contrast parameters.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The present invention Will be more fully understood and
`appreciated from the folloWing detailed description, taken in
`conjunction With the draWings in Which:
`FIG. 1 is a simpli?ed illustration of a camera that pro
`duces panoramic images in accordance With a preferred
`embodiment of the present invention;
`FIG. 2 is a simpli?ed ?owchart for the overall frame
`acquisition and combining that takes place Within a camera,
`in accordance With a preferred embodiment of the present
`invention;
`FIG. 3 is a simpli?ed diagram illustrating the geometrical
`relationship betWeen tWo successive frames that are
`acquired by a camera, in accordance With a preferred
`embodiment of the present invention;
`FIG. 4 is a ?rst simpli?ed illustration of texture mapping
`from rectilinear coordinates to cylindrical coordinates, in
`accordance With a preferred embodiment of the present
`invention;
`FIG. 5 is a second simpli?ed illustration of texture map
`ping from rectilinear coordinates to cylindrical coordinates,
`in accordance With a preferred embodiment of the present
`invention;
`FIG. 6 is a simpli?ed illustration of tWo-pass processing
`for a rectilinear to cylindrical texture map transformation, in
`accordance With a preferred embodiment of the present
`invention;
`FIG. 7 is a simpli?ed illustration of horizontal motion
`estimation, in accordance With a preferred embodiment of
`the present invention;
`FIG. 8 is a simpli?ed illustration use of a histogram to
`estimate horiZontal offset betWeen successive frames, in
`accordance With a preferred embodiment of the present
`invention;
`FIG. 9 is a simpli?ed illustration of the application of
`color correction to each of tWo frames, in accordance With
`a preferred embodiment of the present invention;
`FIG. 10 is a simpli?ed illustration of a portion of a
`previous frame that is stored Within a panoramic image;
`FIG. 11 is a simpli?ed illustration of the stitching of a
`current frame With a previous frame, in accordance With a
`preferred embodiment of the present invention;
`FIG. 12 is a simpli?ed illustration of the stitching of a
`current frame together With a previous frame in a manner
`that avoids ghosting, in accordance With a preferred embodi
`ment of the present invention;
`FIGS. 13A and 13B are simpli?ed illustrations of the
`storing of data Within a panoramic image in vertical data
`strips of a ?xed Width, in accordance With a preferred
`embodiment of the present invention;
`FIG. 14 is a simpli?ed ?owchart of a stitch module
`operative in accordance With a preferred embodiment of the
`present invention; and
`FIG. 15 is a simpli?ed illustration of display of a portion
`of a panoramic image in rectilinear coordinates Within a
`vieW WindoW, in accordance With a preferred embodiment of
`the present invention.
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 21 of 31 PageID #: 68
`
`US 7,292,261 B1
`
`5
`DETAILED DESCRIPTION OF A PREFERRED
`EMBODIMENT
`
`The present invention concerns a camera that acquires
`multiple frames, each representing a ?eld of vieW, and
`combines them. Reference is noW made to FIG. 1, Which is
`a simpli?ed illustration of a camera that produces panoramic
`images in accordance With a preferred embodiment of the
`present invention. A camera 100 of the present invention
`typically includes standard hardWare such as a camera
`housing 110, a camera lens 120 and a camera vieW?nder
`display 130, along With additional hardWare speci?c to the
`present invention.
`In a preferred embodiment, camera 100 acquires succes
`sive frames, each representing a ?eld of vieW of a panoramic
`scene 140, designated by angles A and B in FIG. 1, as the
`camera rotates. Camera 100 aligns, blends and stitches the
`frames together to form a panoramic image of up to 360° in
`extent about any desired axis of rotation. In addition camera
`100 enables a user to previeW a playback of the panoramic
`image, using vieW?nder display 130 that forms part of the
`camera itself.
`A ?rst frame is acquired from a ?rst ?eld of vieW,
`subtending angle A. After the ?rst frame is acquired, camera
`100 is rotated to a second ?eld of vieW, subtending angle B.
`Camera 100 composites a strip 150 from the ?rst frame onto
`the second ?eld of vieW 160 seen on vieW?nder display 130.
`Thus vieW?nder display 130 simultaneously displays both
`strip 150 and the current ?eld of vieW 160, enabling a
`photographer to see the relative alignment betWeen the ?rst
`and second ?elds of vieW, as described in detail With
`reference to FIG. 2 hereinbeloW. When the ?rst and second
`?elds of vieW are brought into alignment on vieW?nder
`display 130, the photographer takes a picture and acquires a
`second frame, representing the second ?eld of vieW.
`The photographer then rotates camera 100 to acquire
`additional frames, representing additional ?elds of vieW (not
`shoWn). As each additional frame is acquired and camera
`100 is further rotated, camera 100 updates vieW?nder dis
`play 130 so as to display a strip from such additional frame
`composited onto a current ?eld of vieW. The photographer
`continues taking pictures and acquiring frames representing
`additional ?elds of vieW, until the frames span the entire
`panoramic scene 140, or a desired portion of panoramic
`scene 140.
`In accordance With a preferred embodiment of the present
`invention, and as described hereinbeloW With reference to
`FIGS. 4-14, camera 100 combines the acquired frames into
`a single panoramic image. VieWing the panoramic image
`interactively by a user as described hereinbeloW With refer
`ence to FIG. 15, provides an immersive experience, as if the
`user is present in panoramic scene 140.
`Reference is noW made to FIG. 2, Which is a simpli?ed
`?owchart for the overall frame acquisition and combining
`that takes place Within a camera, in accordance With a
`preferred embodiment of the present invention. At step 210
`a module for combining frames is initialized. At step 220 a
`?rst frame is acquired.
`At step 230 a loop begins, the ?rst step of Which is
`compositing an overlay from a previously acquired frame
`onto a current ?eld of vieW seen on a vieW?nder display. The
`overlay is preferably a vertical strip from the previous frame,
`as described hereinbeloW. At step 240 a next frame, repre
`senting the ?eld of vieW seen on the vieW?nder display, is
`acquired. At step 250 this neWly acquired next frame is
`combined into a panoramic image containing all previously
`acquired frames. At step 260 a determination is made
`
`5
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`Whether or not further additional frames are to be acquired.
`If there are additional frames to acquire, execution returns to
`step 230, at Which an overlay from the neWly acquired next
`frame is composited onto a current ?eld of vieW as seen on
`the vieW?nder display. If there are no additional frames to
`acquire, the combining module is ?nalized at step 270, at
`Which any residual data may be combined With the pan
`oramic image, as described hereinbeloW With reference to
`FIGS. 13A and 13B.
`It should be apparent to those skilled in the art that the
`present invention applies to frames acquired as a camera
`pans horizontally, and to frames acquired as a camera tilts
`vertically. In the former case, the panoramic image gener
`ated is that of a scene projected onto a surface of a cylinder
`having a vertical axis of revolution. In the latter case, the
`panoramic image generated is that of a scene projected onto
`a surface of a cylinder having a horizontal axis of revolution.
`For the sake of clarity the ensuing description relates to the
`former situation; namely, that of a camera acquiring succes
`sive frames as it rotates approximately Within a horizontal
`plane about a vertical axis of revolution.
`It should also be apparent to those skilled in the art that
`the panoramic images generated in accordance With the
`present invention need not span a full 360°. They can span
`any angle up to and including 360°. For the sake of clarity
`the ensuing description relates to a full 360° panoramic
`image.
`The present invention includes three stages: (i) acquisi
`tion, (ii) combining and (iii) playback. Each of these stages
`is described hereinbeloW.
`Acquisition Stage
`Preferably, the camera of the present invention includes a
`user interface for a photographer to indicate When the
`process for the combining of successive frames starts and
`stops. For example, the photographer can press a button on
`the camera housing to indicate When he begins acquiring
`frames that are to be combined into a panoramic image, and
`he can press the same button or another button to indicate
`When he ?nishes acquiring these frames. Multiple pan
`oramic images can be stored Within a single camera memory
`unit, and the panoramic images can each include different
`numbers of individual frames.
`Preferably, from the time a photographer begins acquiring
`frames that are to be combined into a panoramic image, the
`focal length of the camera is kept ?xed so that it does not
`change from one frame to the next, until the photographer
`acquires all of the frames of the panoramic image. Keeping
`the focal length ?xed can be accomplished by instructing the
`photographer not to change focus during acquisition of
`frames relating to a common panoramic image, or by a
`mechanism Which locks the focus automatically from When
`the photographer indicates the start of a combining process,
`until the photographer indicates the end of the combining
`process.
`Alternatively, the present invention can also be used to
`produce panoramic images by combining frames taken at
`varying focus.
`While acquiring frames, the camera of the present inven
`tion is typically held so that its lens is approximately
`horizontally disposed. The camera may include a spirit or
`other level to assist a photographer in positioning the
`camera. The rotation of the camera is typically approxi
`mately Within a horizontal plane about a vertical axis of
`revolution. Imagining an entire 360° panoramic scene to be
`laid out on the circumference of a vertically standing cyl
`inder, each frame captured by the camera corresponds to an
`
`

`

`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 22 of 31 PageID #: 69
`Case 2:20-cv-00234-JRG Document 1-2 Filed 07/13/20 Page 22 of 31 PageID #: 69
`
`US 7,292,261 B1
`
`7
`angular strip of such circumference. The camera of the
`present
`invention stitches the frames together so as to
`generate a single image of the entire cylindrical circumfer-
`ence.
`
`When acquiring frames for the purpose of combining
`them into a panoramic image, it is desirable that successive
`frames overlap.
`In order to accurately align successive
`frames it
`is desirable that (i) the overlap area between
`successive frames have a
`significant width,
`typically
`approximately 25% of the width of each frame, and (ii) the
`vertical displacement between successive camera positions
`be relatively small. For example, each frame can have a 30°
`field of view in the horizontal direction, and the camera can
`be rotated 221/2° per frame. For this example, 360/221/2:l6
`frames are used to produce a full 360° scene.
`There are several ways in which a photographer may
`rotate the camera so that successive frames are aligned as
`described hereinabove. One such way is to use a tripod head
`with notches. Each time a new frame is to be acquired, the
`photographer rotates the camera about the tripod head from
`one notch to the next.
`
`Alternatively, a portion of the previously acquired frame
`may be composited onto the current field of view seen on the
`viewfinder display. In this way the photographer can deter-
`mine when the camera is rotated to an orientation for which
`
`the current field of view overlaps the previously acquired
`frame by a prescribed amount, such as 25%. Continuing
`with the above example, supposing the camera is rotating
`counter-clockwise (left to right) within a panoramic scene,
`each time a frame is acquired, the rightmost strip of such
`frame is composited onto the current field of view seen on
`the viewfinder display as the camera rotates to a new angular
`orientation. When the camera is rotated so that the current
`
`field of view overlaps the field of view of the previously
`acquired frame, typically by 25% in the horizontal direction,
`the overlaid strip is brought into alignment, and “matches
`up” with the current field of view.
`Reference is now made to FIG. 3, which is a simplified
`diagram illustrating the geometrical relationship between
`two successive frames that are acquired by a camera, in
`accordance with a preferred embodiment of the present
`invention. The line ABC represents the location of a field of
`view when the camera is orientated along the direction of ray
`OB. The field of view corresponds to a vertically standing
`rectangle, perpendicular to the plane of the Figure, with top
`and bottom sides parallel to line ABC. The frame acquired
`subtends a field of view angle AOC.
`When the camera is rotated through an angle, a, to a new
`orientation along ray OF, the corresponding field of view is
`located at line DEFG, and the frame corresponding thereto
`subtends a field of view angle DOG. The focal length, f, of
`the camera is preferably fixed during capture of frames of a
`scene, and, as such, the field of view angles AOC and DOG
`are equal. For the same reason, the lengths ofAC and DG are
`equal. Moreover, since B is the mid-point of AC and F is the
`mid-point of DG, the lengths of AB, BC, DF an