Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 1 of 30 PageID #: 145
`
`Exhibit 8
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 2 of 30 PageID #: 146
`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 2 of 30 PageID #: 146
`
`USOO9961264B2
`
`(12) United States Patent
`Teo
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 9,961,264 B2
`*May 1, 2018
`
`(54) VIRTUAL REALITY CAMERA
`
`(56)
`
`References Cited
`
`(75)
`
`Inventor: Patrick Teo, San Jose, CA (US)
`
`US. PATENT DOCUMENTS
`
`'0
`
`.
`
`(73) Ass1gnee.
`
`( i‘ ) Notice:
`
`Iwnfllacstual Ventures I LLC, Bellevue,
`(
`)
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 400 days.
`This patent is subject to a terminal dis-
`claimer
`
`°
`
`(21) APP1~ N04 13/220579
`
`(
`
`22
`
`)
`
`Filed:
`
`Au . 29, 2011
`g
`
`(65)
`
`Prior Publication Data
`
`US 2012/0176515 A1
`
`Jul. 12, 2012
`
`4,763,280 A
`
`a
`
`a
`
`4,811,245 A
`4,841,292 A
`5,022,085 A
`5,138,460 A
`2313235533 2
`5,185,808 A
`5,231,385 A
`5,251,022 A
`5,262,867 A
`5,307,451 A
`5,325,449 A
`5,338,200 A
`5,396,583 A
`5,402,171 A
`3412764 A
`5,420,533 A
`
`8/1988 Robinson et al.
`
`21er
`
`3/1989 Bunker et a1.
`6/1989 Zeno
`6/1991 Cok
`8/1992 Egawa
`8/133; EGVY et 31.
`2/1993 Cok
`7/ 1993 (3531-1816:? et al~
`10/1993 Kltamura
`11/1993 Kojima
`4/1994 Clark
`6/1994 Burt et a1.
`8/1994 Olive
`3/1995 Chen et a1.
`3/1995 Tagami et a1.
`5/1995 Tamika
`5/1995 Park
`(Continued)
`
`Related US. Application Data
`
`OTHER PUBLICATIONS
`
`(63) Continuation of application No. 11/515,498, filed on
`Aug. 313 2006, now Pat. No. 850313223: which is a
`continuation of application No. 09/378,398, filed on
`A 0. 20 1999
`Pt.N .7292261.
`ug
`’
`3 now a
`O
`3
`3
`Int. Cl.
`H04N 7/00
`H04N 5/232
`H04N 5/262
`(52) US. Cl.
`CPC
`
`(51)
`
`(2011.01)
`(2006.01)
`(2006.01)
`
`H04N 5/23293 (2013 01), H04N 503238
`°°°°° (2013 01), H04N 5/2621 (32013 01), H04N
`5/2624’ (2613.01); H04N 5/2628 (2613.01)
`(58) Field of Classification Search
`CPC
`H04N 5/2323, H04N 5/272, H04N 9/07
`USPC °°°°°°°°
`348/3 6 39 239’ 284 2181
`1pc
`..........................
`,
`1,404}, 7/06 5/262
`See application file for complete search history.
`
`CQStom Panoramic Lab: “Round ShOt MOdel SUPer 35”a made by
`Seltz; 1997, 4 pp, http://wwwroundshot.com/rssup35.htm.
`(Continued)
`.
`..
`Primary Examiner — Glms Phlllppe
`(74) Attorney, Agent, or Firm — Perkins Coie LLP
`57
`ABSTRACT
`
`.
`.
`.
`.
`.
`.
`.
`)
`(
`A camera mclud1ng a camera lens, acqu1s1t10n Clrcu1try
`receiving images via the camera lens, for acquiring a first
`field of view when the camera lens is in a first orientation
`and for acquiring a second field of view when the camera
`lens is in a second orientation, and a viewfinder displaying
`the second field of v1ew when the camera lens 1s 1n the
`second orientation and displaying at least a portion of the
`first field of view at least partially composited with the
`second field of view.
`
`21 Claims, 15 Drawing Sheets
`
`1405
`
`BEGIN COMBINE FRAME
`d—v—Hmm- --
`
`
`
`
`
`
` HRS-f FRAME?
`HAW-f!
`
`1:55vzewrwoee minim 18913511987171?Bdrm?
`
`
`
`
`
`...._. w...
`
`coil/19m: o“'ve‘R‘LA‘?
`OVERLAY BUFFER
`
`
`
`1 490
`A \_.
`COMPUTE OVERLAY FROM PREVIOUS
`32:35:29 INTO OVERLfY aurfgridw
`
`
`\/[ WARP PREVIOUS BUFFER 1N PLACE
`
`i
`
`1430
`xv,
`
`PRW‘OUS—FAR-EDGE ” 0
`PREV!OUS_NEAR_EDGE = WIDTH
`
`..
`
`-
`
`
`1
`
`
`:20 CURRENT Tent-FER No
`
`
`
`,
`
`
`
`”FER IN PM
`
`1440
`
`1445
`
`1450
`
`2455
`
`woo
`
`1465
`
`ESUMA'IE DISPLACEMEN‘i' AND COLOR connecnon
`BETWEEN CURRENT BUFFER AND PREVIOUS BUFFER
`
`COLOR CORRECT PREVIOUS BUFFER AND
`
`CU RENT BUFFER
`
`
`stud-1 cunrefimiirrm wmi PREVIOUS—BUFFER
`USING CURRENT omemcsmsm _
`
`
`
`
`
`
`“SAVE PREVIOUS BU
`
`
`
`r- R R0
`
`ramming-tribes TO PREwOUSfiEARfiDGE
`WIIH PREVIOUS_DESPLACEMENT
`-_._——-.
`
` - END COMBINE 9124,1155} *4
`
`
`
`
`PREVIOUS_FAR_EDGE = CURRENLFAR EDGE
`PWOUSHNEARdEDG-E =1 ‘WiDTH
`PREVIOUS_DISPLACEMENT =2 CURRENTHDISF'LACEMENT
`SWAP PREVIOUS BUFFER AND CURRENT BUFFER
`
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 3 of 30 PageID #: 147
`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 3 of 30 PageID #: 147
`
`US 9,961,264 B2
`
`Page 2
`
`(56)
`
`References Cited
`
`US. PATENT DOCUMENTS
`
`6/1995
`
`Saito
`
`8/1995
`
`8/1995
`
`8/1995
`
`Chapman et a1.
`Lelong et al.
`Miller et a1.
`
`1/1996
`
`Burt et al.
`
`6/1996
`
`Saund
`
`8/1996
`
`Kaufman et a1.
`
`8/1996
`
`Freeman
`
`8/1996
`
`Shirman et al.
`
`10/1996
`
`Miller et al.
`
`11/1996
`
`Broemmelsiek
`
`11/1996
`
`Friedman et a1.
`
`11/1996
`
`Cosman
`
`12/1996
`
`Naka et a1.
`
`12/1996
`
`Florent et a1.
`
`12/1996
`
`Yoshida
`
`12/1996
`
`Nobori et al.
`
`12/1996
`
`1/1997
`
`1/1997
`
`Lamping et a1.
`Greggain et al.
`O’Neill
`
`1/1997
`
`Sakai et a1.
`
`1/1997
`
`Florent et a1.
`
`3/1997
`
`Robertson
`
`3/1997
`
`Chen et al.
`
`5/1997
`
`Schindler
`
`7/1997
`
`Yano et al.
`
`7/1997
`
`Burt et a1.
`
`7/1997
`
`Florent et a1.
`
`“Round Shot”, 4 pp, http://W7w.spectraweb.ch/-mltrad/rs35mm.
`htrn.
`
`Karney, James, “Casio QV-200, QV-700”, PC Magazine, Feb. 10,
`
`1998. 2 pp.
`
`1 p.
`Erickson, Bob, “Round Shot Super 35”, May 13, 1996,
`Ryer, Kelly, “Casio Adds New Camera to Its Lineup”, MacWeek,
`
`1 p.
`vol. 11, issue 38, Oct. 2, 1997,
`International Search Report, Application No. PCT/US98/13465;
`
`Applicant: Live Picture, Inc.; dated Oct. 19, 1998, 5 pp.
`United States Patent and Trademark Oflice, Restriction Require-
`ment, U.S. Appl. No. 09/378,398, dated Apr. 7, 2001, 8 pp.
`United States Patent and Trademark Oflice, Non-Final Oflice
`Action, U.S. Appl. No. 09/378,398, dated Jun. 4, 2003, 14 pp.
`United States Patent and Trademark Oflice, Final Oflice Action,
`U.S. Appl. No. 09/378,398, dated Nov. 20, 2003, 10 pp.
`United States Patent and Trademark Oflice, Advisory Action, U.S.
`Appl. No. 09/378,398, dated Feb. 5, 2004, 3 pp.
`United States Patent and Trademark Oflice, Examiner’s Answer to
`
`Appeal Brief, U.S. Appl. No. 09/378,398, dated Jun. 16, 2004, 18
`
`PP
`United States Patent and Trademark Oflice, BPAI Decision, U.S.
`Appl. No. 09/378,398, dated May 6, 2005, 9 pp.
`United States Patent and Trademark Oflice, Non-Final Oflice
`
`Action, U.S. Appl. No. 09/378,398, dated Sep. 7, 2005, 14 pp.
`United States Patent and Trademark Oflice, Final Office Action,
`
`U.S. Appl. No. 09/378,398, dated Jun. 16, 2006, 17 pp.
`United States Patent and Trademark Oflice, Non-Final Oflice
`
`>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
`
`5,424,773
`5,442,738
`5,444,478
`5,446,833
`5,488,674
`5,528,290
`5,544,283
`5,550,959
`5,550,960
`5,561,756
`5,574,836
`5,577,176
`5,579,456
`5,583,975
`5,586,231
`5,586,232
`5,586,246
`5,590,250
`5,594,676
`5,594,843
`5,594,844
`5,594,845
`5,608,850
`5,613,048
`5,630,037
`5,646,679
`5,649,032
`5,650,814
`5,721,585
`5,745,126
`5,748,194
`5,963,213
`6,104,840
`6,112,033
`6,118,595
`6,128,108
`6,141,034
`6,144,804
`6,192,393
`6,256,058
`6,268,936
`6,269,144
`6,304,284
`6,333,826
`6,657,667
`
`6,677,981
`7,148,909
`7,215,364
`7,292,261
`7,733,368
`7,920,161
`2007/0109398
`
`2/1998
`
`Keast et al.
`
`4/1998
`
`Jain et a1.
`
`5/1998
`
`Chen
`
`10/1999
`
`Guedalia et a1.
`
`8/2000
`
`8/2000
`
`Ejiri et al.
`Yano et a1.
`
`9/2000
`
`Miller
`
`10/2000
`
`Teo
`
`.....
`
`............... 382/284
`
`10/2000
`
`McCutchen
`
`11/2000
`
`Inoue
`
`2/2001
`
`Tarantino et a1.
`
`7/2001
`
`7/2001
`
`Kang et a1.
`Truc et a1.
`
`7/2001
`
`Dube et a1.
`
`10/2001
`
`Dunton et al.
`
`.
`
`................. 348/36
`
`12/2001
`
`Charles
`
`12/2003
`
`Anderson 000000
`
`H04N 5/23238
`
`348/333.11
`
`1/2004
`
`Mancuso et a1.
`
`............... 348/36
`
`12/2006
`
`Yui et al.
`
`5/2007
`
`Wachtel et a1.
`
`............ 348/218.1
`
`11/2007
`
`Teo
`
`6/2010
`
`Teo ................
`
`................. 348/36
`
`4/2011
`
`Niemi et a1.
`
`................. 348/36
`
`A1
`
`5/2007
`
`Teo
`
`OTHER PUBLICATIONS
`
`Action, U.S. Appl. No. 09/378,398, dated Mar. 14, 2007, 20 pp.
`United States Patent and Trademark Oflice, Notice of Allowance,
`U.S. Appl. No. 09/378,398, dated Jul. 2, 2007, 4 pp.
`United States Patent and Trademark Oflice, Non-Final Oflice
`Action, U.S. Appl. No. 11/515,498, dated Sep. 30, 2010, 12 pp.
`United States Patent and Trademark Oflice, Final Office Action,
`U.S. Appl. No. 11/515,498, dated Mar. 17, 2011, 5 pp.
`United States Patent and Trademark Oflice, Interview Summary,
`U.S. Appl. No. 11/515,498, dated May 3, 2011, 3 pp.
`United States Patent and Trademark Oflice, Notice of Allowance,
`
`U.S. Appl. No. 11/515,498, dated May 27, 2011, 5 pp.
`United States Patent and Trademark Oflice, Restriction Require-
`ment, U.S. Appl. No. 11/935,344, dated Dec. 2, 2008, 5 pp.
`United States Patent and Trademark Oflice, Non-Final Office
`Action, U.S. Appl. No. 11/935,344, dated Apr. 13, 2009, 8 pp.
`United States Patent and Trademark Oflice, Notice of Allowance,
`U.S. Appl. No. 11/935,344, dated Jan. 29, 2010, 7 pp.
`Complaint in the United States District Court for the District of
`Delaware, Case l:11-cv-00792-SLR, Intellectual Ventures I LLC
`
`and Intellectual Ventures II LLC v. Canon Inc, Canon USA. Inc.,
`
`Olympus Corporation, Olympus Corporation of the Americas,
`Olympus America Inc, and Olympus Imaging America Inc., Sep. 9,
`2011, 16 pages.
`Complaint in the United States District Court for the District of
`Delaware, Case No. 1:11-cv-01025-SLR, Intellectual Ventures I
`LLC and Intellectual Ventures II LLC v. Sendai Nikon Corporation,
`Nikon Imaging Japan Inc. , Nikon Americas Inc, and Nikon Inc., Oct.
`26, 2011, 9 pages.
`
`Farace, Joe, “Casio QV700 Digital Camera & DP-8000 Digital
`
`Photo Printer”, Nov. 6, 1997, 3 pp.
`
`* cited by examiner
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 4 of 30 PageID #: 148
`
`US. Patent
`
`May 1, 2018
`
`Sheet 1 0f 15
`
`US 9,961,264 B2
`
`
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 5 of 30 PageID #: 149
`
`US. Patent
`
`May 1, 2018
`
`Sheet 2 0f 15
`
`US 9,961,264 B2
`
`
`
`VIEWFINDER FIELD OF VIEW
`
`
`
`
`
`DONE? NO
`
`
`
`‘ YES
`
`270
`
`.
`..
`.
`.
`' FINALIZE COMBNINIGMODULE
`
`FlGURE 2
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 6 of 30 PageID #: 150
`
`US. Patent
`
`May 1, 2018
`
`Sheet 3 0f 15
`
`US 9,961,264 B2
`
`
`
`G
`
`FIGURE3
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 7 of 30 PageID #: 151
`
`US. Patent
`
`May 1, 2018
`
`Sheet 4 0f 15
`
`US 9,961,264 B2
`
`430
`
`
`
`
`
`.~
`
`1' oos(a)
`
`420
`
`9
`
`y
`
`.
`6 5 -
`4....................... X .......................p 0
`
`FIGURE 4
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 8 of 30 PageID #: 152
`
`US. Patent
`
`May 1, 2018
`
`Sheet 5 0f 15
`
`US 9,961,264 B2
`
`FIGURE5
`
`520
`
`510
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 9 of 30 PageID #: 153
`
`US. Patent
`
`May 1, 2018
`
`Sheet 6 0f 15
`
`US 9,961,264 B2
`
`
`
`FIGURE6
`
`
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 10 of 30 PageID #: 154
`
`US. Patent
`
`May 1, 2018
`
`Sheet 7 0f 15
`
`US 9,961,264 B2
`
`720
`
`FIGURE7
`
`
`
`710
`
`730
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 11 of 30 PageID #: 155
`
`US. Patent
`
`May 1, 2018
`
`Sheet 8 0f 15
`
`US 9,961,264 B2
`
`dx
`
`FIGURE8
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 12 of 30 PageID #: 156
`
`US. Patent
`
`May 1, 2018
`
`Sheet 9 0f 15
`
`US 9,961,264 B2
`
`O U
`
`)
`0‘
`
`FIGURE9
`
`910
`
`920
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 13 of 30 PageID #: 157
`
`US. Patent
`
`May 1, 2018
`
`Sheet 10 0f 15
`
`US 9,961,264 B2
`
`_m00ml2<mZiwDQ>m~E
`
`mwomsmfiuthgo
`
`
`
`mwomrafiwDOSme
`
`_
`
`OFmEDmVE
`
`In
`
`w.
`
`mDO_>mw_n_
`
`awntbm
`
` was,[K0.
`_on4/‘22
`
`____
`
`____.__._________~
`
`O<
`
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 14 of 30 PageID #: 158
`
`US. Patent
`
`May 1, 2018
`
`Sheet 11 0f 15
`
`US 9,961,264 B2
`
`FIGURE11
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 15 of 30 PageID #: 159
`
`US. Patent
`
`May 1, 2018
`
`Sheet 12 of 15
`
`US 9,961,264 B2
`
`FIGURE12
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 16 of 30 PageID #: 160
`
`US. Patent
`
`May 1, 2018
`
`Sheet 13 0f 15
`
`US 9,961,264 B2
`
`1320
`
`1315
`
`1310
`
`1305
`
`FIGURE13A
`
`v1360
`
`1345
`
`1340
`
`III
`IIII
`|i_'-:1I
`III-=1
`LII!
`
`
`
`13301335
`
`1325
`
`FIGURE138
`\
`
`g
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 17 of 30 PageID #: 161
`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 17 of 30 PageID #: 161
`
`US. Patent
`
`May 1, 2018
`
`Sheet 14 0f 15
`
`US 9,961,264 B2
`
`wov—
`
`ORE
`
`mm":
`
`
`
`zmnmbm>Sam>0
`
`
`
`
`
`
`0:1:@3quclaw—EDDE03“.>szw>0msmS—OO
`
`
`
`wDO_>maaEOwE>Smm>0gnaw/.00
`
`
`
`zwnfibm>33m>001.2.23qu
`
`
`
`
`
`ZmeEmm ZOFPOmNEOO
`
`
`mOAODDZ<prEm—DSamameg/Emu
`zmmmamwDOENEDZ<zwumbmEmcw—DU
`
`
`
`
`
`
`
`mOSaZ.awn—mamthEaDOa242,
`
` 0nwwowlm<ulw305mza_mOSa2.gunmanw30_>w~._n_maxi)
`
`
`
`E93um00w1m<wZImDOSww—n.
`
`m><m EmEmUSama
`E0400
`mwomjzfizdaoawaOpmwowtzEImDOSmE
`mmtammDOSm—zn—IE52&quthNEDU20:5
`
`
`
`DZ<gunmanmDO_>m_~_n_HOME—coo
`
`
`
`HZmEmDU02%:
`
`
`5.0a”.Ewan—3mmDOSwwE
`
`
`
`mun—“SmEmmmDU
`
`Zaxfiwou.4wSn..00ma
`
`
`
`
`
`$58image92Eta“39>szens/m
`
`
`
`EmfiwofimmfilmaoaamIE5
`
`hzwfiwofiamadzmaaaouEmEMOSamEISOEE
`
`
`
`mwomméahzmmmaoumwomdéumaosmfi
`..:53)nmwomuafizdaoafi
`
` wee/2&1
`
`mZE—ZOU02m
`
`mm:
`
`EMEDGE
`
`
`
`
`
`zwuubmEwaaDU0:1:m§<~Eawn—2E5;Q<O._
`
`
`
`mg»mmDZESwS
`
`
`
`mwmmnm9.03me0:4.
`
`
`
`
`
`0(04
`
`93.
`
`ON?P
`
`va_.
`
`021.
`
`Z_mvwm
`m§<zuwZES-OU
`
`
`
`
`amt/Emu“.5am
`
`0:1.
`
`
`
`
`
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 18 of 30 PageID #: 162
`
`US. Patent
`
`May 1, 2018
`
`Sheet 15 0f 15
`
`US 9,961,264 B2
`
`£9.010
`
`9$50.“.
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 19 of 30 PageID #: 163
`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 19 of 30 PageID #: 163
`
`US 9,961,264 B2
`
`1
`
`2
`
`VIRTUAL REALITY CAMERA
`
`CROSS-REFERENCES TO RELATED
`
`APPLICATIONS
`
`This application is a continuation application of US.
`patent application Ser. No. 11/515,498, filed on Aug. 31,
`2006, and entitled “VIRTUAL REALITY CAMERA,” now
`US. Pat. No. 8,031,223, which is a continuation application
`of US. patent application Ser. No. 09/378,398, filed on Aug.
`20, 1999, and entitled “VIRTUAL REALITY CAMERA,”
`now US. Pat. No. 7,292,261, both of which are hereby
`incorporated herein in their entireties by reference.
`
`FIELD OF THE INVENTION
`
`10
`
`15
`
`The present invention relates to a camera that acquires
`fields of view and combines them together.
`
`BACKGROUND OF THE INVENTION
`
`20
`
`Panoramic images are used to provide immersive “sur-
`round” views of scenes, up to 3600 in extent. Digital
`panoramic images can be archived on Internet servers, for
`access by remote clients. Panoramic images are typically 25
`texture mapped into a suitable surface geometry, such as a
`cylindrical or a spherical geometry. Texture mapping gen-
`erates a “warped” look to the panorama, and usually con-
`verts straight lines into bowed curves.
`Client viewer software enables users to interactively view 30
`panoramic images by navigating through the panorama.
`Specifically,
`for a prescribed view window on a client
`computer video display, client viewer software converts a
`selected portion of the panoramic image at a selected level
`of magnification from cylindrical or other surface geometry 35
`to rectilinear geometry, which is appropriate for typical
`perspective viewing. The converted portion of the pan-
`oramic image is displayed in the view window.
`A user interface enables a user to dynamically change the
`selected portion of the panoramic image being viewed.
`Rapid display of the dynamically changing selected portion
`gives the viewer a sensation of moving through the pan-
`orama, and an experience of being immersed within the
`surrounding scene. Typically a user interface enables a user
`to select portions of the panoramic image for display by
`indicating shifts and changes in magnification of a current
`portion of the panoramic image, rather than by selecting a
`new portion without reference to the current portion. Typi-
`cally the user interface provides for shifting the selected
`portion up, down,
`left, right, or other directions, and for
`reducing or enlarging the current magnification factor, by
`zooming in and out. Such a user interface can include, for
`example, keyboard buttons or mouse controls and move-
`ments.
`
`40
`
`45
`
`50
`
`Panoramic content creation is typically carried out by a
`photographer who captures multiple photos of a scene, as he
`rotates his camera around in a circle. For typical camera
`lenses such as a 35 mm video camera lens,
`the photos
`acquired are characterized by a rectilinear geometry. That is,
`there exists a linear correspondence between distances in the
`photo and distances in the scene. Each such photo represents
`a portion of the scene. The photos are digitized by a digital
`scanner or, in a case where the photographer uses a digital
`camera, the photos are digitized by hardware within the
`
`camera.
`
`The digital photos are then downloaded to a computer,
`and a “stitching” application is run to combine the digital
`
`55
`
`60
`
`65
`
`photos into a single panoramic image. An example of such
`a stitching application is assignee’s PhotoVista® software.
`The photographer can eliminate the combining step by
`capturing a large portion of a scene by using a wide angle
`lens, or a hemispherical lens such as the Portal lens system
`of Be Here Corporation, or a parabolic lens such as the
`ParaShotTM attachment of CycloVision Technologies, Inc.,
`the latter two of which capture a 3600 image in one expo-
`
`sure.
`
`When a photographer creates a panoramic image by
`combining multiple photos, typically he must be careful to
`ensure that adjacent fields of view are appropriately aligned
`as he rotates the camera. When rotating the camera in a
`horizontal plane, such alignment
`involves providing for
`horizontal overlap between adjacent photos, and minimizing
`vertical displacements outside of the horizontal plane of
`rotation. Similarly, when rotating the camera in a vertical
`plane, such alignment involves providing for vertical over-
`lap between adjacent photos, and minimizing horizontal
`displacements outside of the vertical plane of rotation.
`In the ensuing discussion, for the sake of conciseness and
`simplicity of explanation, panoramas in a horizontal plane of
`rotation are described, it being understood that the present
`invention is not limited to panoramas in a horizontal plane
`of rotation.
`
`When rotating the camera freely in his hand, it is difficult
`for a photographer to accurately align adjacent fields of
`view. A reliable way for a photographer to align adjacent
`fields of view is to mount his camera on a tripod, using a
`tripod bracket that has equi-spaced notches for rotating the
`camera in increments of a fixed angle. An example of such
`a bracket is the Kaidan KiWiTM tripod head. Typically such
`tripod brackets include one or more spirit levels, so that a
`photographer can adjust
`the tripod to keep the axis of
`rotation of the camera vertically disposed.
`Use of a computer for combining acquired photos to
`produce a panoramic image has several disadvantages. One
`disadvantage is that the photographer needs to take a com-
`puter with him in the field. Otherwise, he cannot create and
`view the panoramic image while in the field. There are
`problems associated with both of these situations.
`The first option of taking a computer in the field can be
`difficult, because often a photographer has to hike through
`un-trodden ground in order to find “the perfect scene.”
`Carrying both a camera and a computer along with his food
`and gear can be burdensome on the photographer.
`Even when the photographer does take a computer with
`him in the field, he may have to move the camera from its
`axis of rotation in order to download the photos acquired
`thereby. As soon as a camera is moved from its axis of
`rotation, typically the only way to re-generate a panoramic
`image is to recapture at a new axis of rotation all of the
`individual photos included in the panorama. For example, if
`one of sixteen frames of a scene comes out bad,
`the
`photographer can correct the problem and recapture such
`frame as long as the camera has not been moved from its axis
`of rotation. However, once the camera has been moved from
`its axis of rotation, a single frame cannot normally be
`accurately aligned within previously acquired adjacent
`frames. Thus, even when using a notched tripod bracket, the
`photographer will normally have to recapture all of the
`photos included in the panoramic image.
`The second option of not creating the panoramic image in
`the field is problematic, because the photographer cannot see
`the quality of his panoramic image while in the field. If he
`subsequently discovers problems with the panoramic image,
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 20 of 30 PageID #: 164
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 21 of 30 PageID #: 165
`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 21 of 30 PageID #: 165
`
`US 9,961,264 B2
`
`5
`
`6
`
`that avoids ghosting, in accordance with a preferred embodi-
`ment of the present invention;
`FIGS. 13A and 13B are simplified illustrations of the
`storing of data within a panoramic image in vertical data
`strips of a fixed width,
`in accordance with a preferred 5
`embodiment of the present invention;
`FIG. 14 is a simplified flowchart of a stitch module
`operative in accordance with a preferred embodiment of the
`present invention; and
`FIG. 15 is a simplified illustration of display of a portion
`of a panoramic image in rectilinear coordinates within a
`view window, in accordance with a preferred embodiment of
`the present invention.
`
`10
`
`DETAILED DESCRIPTION OF A PREFERRED
`
`EMBODIMENT
`
`The present invention concerns a camera that acquires
`multiple frames, each representing a field of view, and
`combines them. Reference is now made to FIG. 1, which is
`
`a simplified illustration of a camera that produces panoramic
`images in accordance with a preferred embodiment of the
`present invention. A camera 100 of the present invention
`typically includes standard hardware such as a camera
`housing 110, a camera lens 120 and a camera viewfinder
`display 130, along with additional hardware specific to the
`present invention.
`In a preferred embodiment, camera 100 acquires succes-
`sive frames, each representing a field of view of a panoramic
`scene 140, designated by angles A and B in FIG. 1, as the
`camera rotates. Camera 100 aligns, blends and stitches the
`frames together to form a panoramic image of up to 3600 in
`extent about any desired axis of rotation. In addition camera
`100 enables a user to preview a playback of the panoramic
`image, using viewfinder display 130 that forms part of the
`camera itself.
`
`A first frame is acquired from a first field of view,
`subtending angle A. After the first frame is acquired, camera
`100 is rotated to a second field of view, subtending angle B.
`Camera 100 composites a strip 150 from the first frame onto
`the second field of view 160 seen on viewfinder display 130.
`Thus viewfinder display 130 simultaneously displays both
`strip 150 and the current field of view 160, enabling a
`photographer to see the relative alignment between the first
`and second fields of view, as described in detail with
`reference to FIG. 2 hereinbelow. When the first and second
`
`fields of View are brought into alignment on viewfinder
`display 130, the photographer takes a picture and acquires a
`second frame, representing the second field of view.
`The photographer then rotates camera 100 to acquire
`additional frames, representing additional fields of view (not
`shown). As each additional frame is acquired and camera
`100 is further rotated, camera 100 updates viewfinder dis-
`play 130 so as to display a strip from such additional frame
`composited onto a current field of view. The photographer
`continues taking pictures and acquiring frames representing
`additional fields of view, until the frames span the entire
`panoramic scene 140, or a desired portion of panoramic
`scene 140.
`
`In accordance with a preferred embodiment of the present
`invention, and as described hereinbelow with reference to
`FIGS. 4-14, camera 100 combines the acquired frames into
`a single panoramic image. Viewing the panoramic image
`interactively by a user as described hereinbelow with refer-
`ence to FIG. 15, provides an immersive experience, as if the
`user is present in panoramic scene 140.
`
`Reference is now made to FIG. 2, which is a simplified
`flowchart for the overall frame acquisition and combining
`that takes place within a camera,
`in accordance with a
`preferred embodiment of the present invention. At step 210
`a module for combining frames is initialized. At step 220 a
`first frame is acquired.
`the first step of which is
`At step 230 a loop begins,
`compositing an overlay from a previously acquired frame
`onto a current field of view seen on a viewfinder display. The
`overlay is preferably a vertical strip from the previous frame,
`as described hereinbelow. At step 240 a next frame, repre-
`senting the field of view seen on the viewfinder display, is
`acquired. At step 250 this newly acquired next frame is
`combined into a panoramic image containing all previously
`acquired frames. At step 260 a determination is made
`whether or not further additional frames are to be acquired.
`If there are additional frames to acquire, execution returns to
`step 230, at which an overlay from the newly acquired next
`20 frame is composited onto a current field of view as seen on
`the viewfinder display. If there are no additional frames to
`acquire, the combining module is finalized at step 270, at
`which any residual data may be combined with the pan-
`oramic image, as described hereinbelow with reference to
`25 FIGS. 13A and 13B.
`
`15
`
`35
`
`It should be apparent to those skilled in the art that the
`present invention applies to frames acquired as a camera
`pans horizontally, and to frames acquired as a camera tilts
`vertically. In the former case, the panoramic image gener-
`30 ated is that of a scene projected onto a surface of a cylinder
`having a vertical axis of revolution. In the latter case, the
`panoramic image generated is that of a scene projected onto
`a surface of a cylinder having a horizontal axis of revolution.
`For the sake of clarity the ensuing description relates to the
`former situation; namely, that of a camera acquiring succes-
`sive frames as it rotates approximately within a horizontal
`plane about a vertical axis of revolution.
`It should also be apparent to those skilled in the art that
`the panoramic images generated in accordance with the
`40 present invention need not span a full 3600. They can span
`any angle up to and including 360°. For the sake of clarity
`the ensuing description relates to a full 3600 panoramic
`image.
`The present invention includes three stages: (i) acquisi-
`45 tion, (ii) combining and (iii) playback. Each of these stages
`is described hereinbelow.
`
`Acquisition Stagg
`Preferably, the camera of the present invention includes a
`user interface for a photographer to indicate when the
`50 process for the combining of successive frames starts and
`stops. For example, the photographer can press a button on
`the camera housing to indicate when he begins acquiring
`frames that are to be combined into a panoramic image, and
`he can press the same button or another button to indicate
`55 when he finishes acquiring these frames. Multiple pan-
`oramic images can be stored within a single camera memory
`unit, and the panoramic images can each include difierent
`numbers of individual frames.
`
`Preferably, from the time a photographer begins acquiring
`60 frames that are to be combined into a panoramic image, the
`focal length of the camera is kept fixed so that it does not
`change from one frame to the next, until the photographer
`acquires all of the frames of the panoramic image. Keeping
`the focal length fixed can be accomplished by instructing the
`65 photographer not
`to change focus during acquisition of
`frames relating to a common panoramic image, or by a
`mechanism which locks the focus automatically from when
`
`

`

`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 22 of 30 PageID #: 166
`Case 2:20-cv-00234-JRG Document 1-8 Filed 07/13/20 Page 22 of 30 PageID #: 166
`
`US 9,961,264 B2
`
`7
`
`8
`
`the photographer indicates the start of a combining process,
`until the photographer indicates the end of the combining
`
`pI’OCGSS.
`Alternatively, the present invention can also be used to
`produce panoramic images by combining frames taken at
`varying focus.
`While acquiring frames, the camera of the present inven-
`tion is typically held so that
`its lens is approximately
`horizontally disposed. The camera may include a spirit or
`other level
`to assist a photographer in positioning the
`camera. The rotation of the camera is typically approxi-
`mately within a horizontal plane about a vertical axis of
`revolution. Imagining an entire 360° panoramic scene to be
`laid out on the Circumference of a vertically standing cyl-
`inder, each frame captured by the camera corresponds to an
`angular strip of such circumference. The camera of the
`present
`invention stitches the frames together so as to
`generate a single image of the entire cylindrical circumfer-
`
`ence.
`
`When acquiring frames for the purpose of combining
`them into a panoramic image, it is desirable that successive
`frames overlap.
`In order to accurately align successive
`frames it
`is desirable that (i) the overlap area between
`successive frames have a
`significant width,
`typically
`approximately 25% of the width of each frame, and (ii) the
`vertical displacement between successive camera positions
`be relatively small. For example, each frame can have a 30°
`field of view in the horizontal direction, and the camera can
`be rotated 221/2° per frame. For this example, 360/221/2=16
`frames are used to produce a full 360° scene.
`There are several ways in which a photographer may
`rotate the camera so that successive frames are aligned as
`described hereinabove. One such way is to use a tripod head
`with notches. Each time a new frame is to be acquired, the
`photographer rotates the camera about the tripod head from
`one notch to the next.
`
`Alternatively, a portion of the previously acquired frame
`may be composited onto the current field of view seen on the
`viewfinder display. In this way the photographer can deter-
`mine when the camera is rotated to an orientation for which
`
`the current field of View overlaps the previously acquired
`frame by a prescribed amount, such as 25%. Continuing
`with the above example, supposing the camera is rotating
`counter-clockwise (left to right) within a panoramic scene,
`each time a frame is acquired, the rightmost strip of such
`frame is composited onto the current field of view seen on
`the viewfinder display as the camera rotates to a new angular
`orientation. When the camera is rotated so that the current
`
`field of view overlaps the field of view of the previously
`acquired frame, typically by 25% in the horizontal direction,
`the overlaid strip is brought into alignment, and “matches
`up” with the current field of view.
`Reference is now made to FIG. 3, which is a simplified
`diagram illustrating the geometrical relationship between
`two successive frames that are acquired by a camera, in
`accordance with a preferred embodiment of the present
`invention. The line ABC represents the location of a field of
`view when the camera is orientated along the direction of ray
`OB. The field of view corresponds to a vertically standing
`rectangle, perpendicular to the plane of the Figure, with top
`and bottom sides parallel to line ABC. The frame acquired
`subtends a field of view angle AOC.
`When the camera is rotated through an angle, a, to a new
`orientation along ray OF, the corresponding field of View is
`located at line DEFG, and the frame corresponding thereto
`subtends a field of view angle DOG. The focal length, f, of
`the camera is preferably fixed during capture of frames of a
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`scene, and, as such, the field of view angles AOC and DOG
`are equal. For the same reason, the lengths ofAC and DG are
`equal. Moreover, since B is the mid-point of AC and F is the
`mid-point of DG, the lengths of AB, BC, DF and FG are all
`equal. The overlap angle is angle DOE, which preferably
`corresponds roughly to 25% of angle DOG.
`In a preferred embodiment of the present invention, the
`rightmost strip of the frame at line ABC is compo sited at line
`DE onto the field of view seen on the viewfinder display.
`That is, the leftmost 25% of the field of view seen on the
`viewfinder display contains the rightmost 25% of the pre-
`viously acquired frame. The resultant overlay can be semi-
`transparent, typically having 50% opacity, so that the pho-
`tographer can see both the overlay and the current field of
`view composited together in the viewfinder display. Alter-
`natively, the overlay and the current field of view can be
`displayed using alternate pixels,
`in a checkerboard-like
`fashion, providing a screening type composite of the overlay
`on the current field of view, rather than a semi-transparent
`type composite.
`As the camera is rotated from the orientation along ray
`OB to the orientation along ray OF, the overlay portion DE
`is brought into alignment with the field of view seen on the
`viewfinder display. Visual alignment serves as the cue that
`informs the photographer how far to rotate the camera.
`In a preferred embodiment of the present invention the
`ca