Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 1 of 37 PageID #: 37
`
`
`
`
`
`
`
`Exhibit B
`
`U.S. Patent No. 6,741,250
`
`Method and System for Generation of Multiple Viewpoints into a Scene Viewed by Motionless
`Cameras and for Presentation of a View Path
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 2 of 37 PageID #: 38
`I 1111111111111111 11111 111111111111111 IIIII IIIII IIIII 11111 1111111111 11111111
`US006741250Bl
`
`(12) United States Patent
`Furlan et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 6,741,250 Bl
`May 25, 2004
`
`(54) METHOD AND SYSTEM FOR GENERATION
`OF MULTIPLE VIEWPOINTS INTO A SCENE
`VIEWED BY MOTIONLESS CAMERAS AND
`FOR PRESENTATION OF A VIEW PATH
`
`(75)
`
`Inventors: John L. W. Furlan, Belmont, CA (US);
`Edward C. Driscoll, Jr., Portola Valley,
`CA (US); Robert G. Hoffman,
`Fremont, CA (US); Derek Fluker, San
`Jose, CA (US); Karen L. Bechtel, San
`Jose, CA (US); Katerina L. Shiffer,
`Milpitas, CA (US); Venugopal
`Garimella, Sunnyvale, CA (US);
`Daniel B. Curtis, Newark, CA (US)
`
`(73) Assignee: Be Here Corporation, Fremont, CA
`(US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 393 days.
`
`(21) Appl. No.: 09/982,614
`
`(22) Filed:
`
`Oct. 17, 2001
`
`Related U.S. Application Data
`( 60) Provisional application No. 60/288,396, filed on May 2,
`2001, and provisional application No. 60/267,538, filed on
`Feb. 9, 2001.
`Int. CI.7 ................................................ G06T 15/00
`
`(51)
`
`(52) U.S. Cl. ....................................................... 345/427
`(58) Field of Search ................................. 345/419, 427,
`345/428, 473, 619, 581, 582
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`5,748,768 A * 5/1998 Sivers et al. ................ 382/130
`5,974,521 A * 10/1999 Akerib ........................ 712/11
`6,381,072 Bl * 4/2002 Burger ....................... 359/622
`6,590,606 Bl * 7/2003 Hiller et al.
`................ 348/203
`* cited by examiner
`
`Primary Examiner-Phu K. Nguyen
`(74) Attorney, Agent, or Firm-Bever, Hoffman & Harms,
`LLP; Edward S. Mao
`
`(57)
`
`ABSTRACT
`
`Methods and apparatus for defining a view path through at
`least one wide-angle video stream and for creation of a
`resultant unwarped video stream responsive to the view
`path. Aspects of the invention allow an operator to monitor
`the wide-angle video stream(s) and select which portion of
`the wide-angle frame to unwarp in real-time. Further, the
`operator can stop the playback of the wide-angle video
`stream and dwell within a particular wide-angle frame to
`create special effects such as camera tilt, pan and zoom
`operations in a stop action situation.
`
`56 Claims, 21 Drawing Sheets
`
`1105
`
`1101
`
`1117
`
`J
`
`1107
`
`1109
`
`1103
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 3 of 37 PageID #: 39
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 1 of 21
`
`US 6,741,250 Bl
`
`1 1 1
`
`101 "''
`
`105 9--J--.::__ -::
`
`Fig. 1
`(Prior Art)
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 4 of 37 PageID #: 40
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 2 of 21
`
`US 6,741,250 Bl
`
`205Q
`
`/
`
`/
`
`........
`
`.....
`
`205
`
`/
`
`/
`
`.....
`
`........
`
`101
`
`I
`2osidt/
`
`\
`
`\
`
`I
`
`I
`
`\
`
`\
`
`\
`
`\
`
`I
`
`I
`
`\
`
`\r~i°5
`
`I
`
`I
`
`@
`
`205
`
`........
`
`.....
`
`209
`Dedicated Video
`
`........
`
`.........
`207J
`
`/
`
`/
`
`/
`
`/
`
`205
`
`213
`
`Fig.2
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 5 of 37 PageID #: 41
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 3 of 21
`
`US 6,741,250 Bl
`
`303
`
`205
`
`205~1301
`
`I
`
`I 303
`
`I
`I
`
`I
`I
`~205
`I
`301
`
`303
`
`@
`
`I
`301~
`
`205~1
`
`101
`
`I
`
`I
`I
`I
`I
`I
`
`205
`
`303
`
`205
`
`209
`Dedicated Video
`
`211
`
`Fig. 3
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 6 of 37 PageID #: 42
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 4 of 21
`
`US 6,741,250 Bl
`
`405
`
`400 /
`
`Fig. 4A
`
`440! Fig. 4B
`
`465
`
`441
`
`463
`
`/ Fig. 4C
`
`460
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 7 of 37 PageID #: 43
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 5 of 21
`
`US 6,741,250 Bl
`
`501
`
`500
`
`205
`
`205
`101
`/\303
`U - - , / . . . .1 - - - - - - - - - - - -++ - - - - " ' i~~ - t ' , )< ' 205
`
`205
`
`Fig. 5
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 8 of 37 PageID #: 44
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 6 of 21
`
`US 6,741,250 Bl
`
`501
`
`503
`
`~601
`I
`I I
`
`I ~603
`I
`I
`I
`
`I
`I
`
`600
`
`,----'..,__-......-----.---~~
`
`205
`
`607
`
`205
`
`Fig. 6
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 9 of 37 PageID #: 45
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 7 of 21
`
`US 6,741,250 Bl
`
`T"""
`0
`I"--
`
`LO
`0
`C\I
`
`I
`0
`0
`I'--
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 10 of 37 PageID #: 46
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 8 of 21
`
`US 6,741,250 Bl
`
`801
`
`Initialize
`
`Determine Area of Interest
`
`803
`
`805
`
`Communicate
`807 Area of Interest to
`Video Processors
`
`809
`
`Receive Video Frame Containing
`Distorted Image at Video Processo
`
`811
`
`Transform Portion of Distorted
`Image representing the Area of
`Interest
`
`813
`
`Generate Undistorted Video Frame
`Using Transformed image
`
`815
`
`Select/Generate a Video Frame
`Responsive to User-Specified
`Viewpoint
`
`817
`
`Send Video Frame
`
`Fig. BA
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 11 of 37 PageID #: 47
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 9 of 21
`
`US 6,741,250 Bl
`
`851
`
`853
`
`Initialize
`
`Select Initial Camera & Area of
`Interest
`
`855
`
`869
`Update AOI for
`Different Camera
`
`857
`
`rame Containing
`Receive Video
`Distorted Image
`
`867
`Select Different
`Camera Video
`
`~--<.
`
`no
`
`861
`
`Transform Portion of Distorted
`Image representing the Area of
`Interest
`
`863
`
`Generate Undistorted Video Frame
`Using Transformed image
`
`865
`
`Present Video Frame
`
`Fig. BB
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 12 of 37 PageID #: 48
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 10 of 21
`
`US 6,741,250 Bl
`
`0)
`
`• -~ LL:
`
`,,i:-·
`
`J
`
`-
`
`-
`
`I
`I
`I
`I
`1
`
`('I)
`0
`O')
`
`' r
`
`-,
`
`\
`
`0
`0
`O')
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 13 of 37 PageID #: 49
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 11 of 21
`
`US 6,741,250 Bl
`
`~,.
`
`'I.
`
`" .,
`
`,._
`Q
`-~
`•
`LI:
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 14 of 37 PageID #: 50
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 12 of 21
`
`US 6,741,250 Bl
`
`I
`
`LC)
`0
`,-
`,-
`
`Cf)
`0
`,-
`,-
`
`,.._
`'-=:(
`,.._
`
`• -~ La:
`
`~
`0
`0
`,-
`,-
`
`,(cid:173)
`,-
`
`,(cid:173)
`,-
`
`,(cid:173)
`,-
`---~-------,-
`I"--
`,-
`,(cid:173)
`,-
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 15 of 37 PageID #: 51
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 13 of 21
`
`US 6,741,250 Bl
`
`1130 I
`
`1137
`
`Fig. 11B
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 16 of 37 PageID #: 52
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 14 of 21
`
`US 6,741,250 Bl
`
`lo...
`Cl)
`
`.c u +-'
`
`,.._______,____,'(
`,....
`,....
`.------'---- C\J
`
`....
`
`...
`
`lo...
`
`.9 C
`ctS 0
`lo...·-
`Cl) +-'
`a.~
`0 Cl)
`
`I(
`.....__,--____, 0)
`0
`C\J
`,....
`
`l
`
`.4
`
`.c +-' z lo...
`-
`.9 C
`CU CU 0
`§m~ a.-
`0.. 0 Cl)
`0
`..___ __ __. LO
`,....
`C\J
`,....
`
`,....
`,....
`C\I
`,....
`
`)
`
`~
`0
`0
`C\J
`,....
`
`• -~ LL:
`
`lo...
`Cl)
`.::£. "C
`Cl)
`lo...
`
`-
`
`Cl)
`
`0 8
`cu a:
`+-'
`"c, 0
`· - Cl)
`0
`"'C
`
`...____,+~--'~
`C\J
`,....
`cu
`g Q) ~
`-c E <
`,....
`·- cu >o
`----- ,....
`0
`C\I
`
`Cl)
`0)
`C
`Cl) (
`<C C
`I
`Cl)
`....J C')
`Cl)
`0
`"C
`s
`C\I
`,....
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 17 of 37 PageID #: 53
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 15 of 21
`
`US 6,741,250 Bl
`
`1301
`
`Initialize
`
`Record Wide-Angle Video
`
`Specify Video Segment From
`Recorded Wide-Angle Video
`
`1303
`
`1305
`
`1307
`
`1309
`
`Specify View Path Through the
`Wide-Angle Video Segment
`
`1311
`
`Present View Into Video Segment
`Respective to View Path
`
`Fig. 13
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 18 of 37 PageID #: 54
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 16 of 21
`
`US 6,741,250 Bl
`
`1401
`
`Initialize
`
`Select First Wide-Angle Frame
`
`1403
`
`1405
`
`1407
`
`Select Last Wide-Angle Frame
`(Defaults to Last Frame)
`
`1409
`
`End
`
`Fig. 14
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 19 of 37 PageID #: 55
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 17 of 21
`
`US 6,741,250 Bl
`
`1501
`
`150
`
`Initialize View Path
`
`For Each Frame in Video Segment 14-----
`
`1507
`
`Grow View Path
`
`150
`
`Package Segment
`
`1500/
`
`Fig. 15A
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 20 of 37 PageID #: 56
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 18 of 21
`
`US 6,741,250 Bl
`
`1521
`
`1523
`
`Acquire Pan, Tilt, and Zoom
`Parameters from Operator
`
`1527
`
`Incorporate PTZT Parameters in
`View Path
`
`1533
`
`Specify View
`Path within
`Frame
`
`Fig. 15B
`
`1s31
`
`1520/
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 21 of 37 PageID #: 57
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 19 of 21
`
`US 6,741,250 Bl
`
`1551
`
`155
`
`Initialize Frame View Path
`
`While lntraframe
`
`1555
`
`1557
`
`Acquire Tilt, Pan, Zoom and Time
`Parameters from Operator
`
`1559
`
`Incorporate TPZT Parameters in
`Frame View Path
`
`1561
`
`Add Frame View Path to Segment
`View Path
`
`1550/
`
`Fig. 15C
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 22 of 37 PageID #: 58
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 20 of 21
`
`US 6,741,250 Bl
`
`1601
`
`1603
`
`Initialize
`
`1605 Start Operator Monitor
`
`While Segment Active
`
`1607
`
`1609
`
`Extrapolate/Predict Position on
`View Path
`
`1611
`
`Generate a Viewing Vector into the
`Wide-Angle Frame
`
`1613
`
`Wait for Video Buffer
`
`1615
`
`Transform Portion of Wide-Angle
`Frame into a View
`
`1617
`
`Output Video Buffer
`
`1600/
`
`Fig. 16
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 23 of 37 PageID #: 59
`
`U.S. Patent
`
`May 25, 2004
`
`Sheet 21 of 21
`
`US 6,741,250 Bl
`
`1701
`
`1702
`
`Initialize
`L--------------------
`
`170
`
`Open View Path
`
`170
`
`Sync with View Path Display
`
`707
`- - - - - - i - - - - - - - - - ' - ,
`Track View Path Progress
`
`1711
`
`?
`
`yes
`
`1713
`
`Alter View Path
`
`1700/
`
`Fig. 17
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 24 of 37 PageID #: 60
`
`US 6,741,250 Bl
`
`1
`METHOD AND SYSTEM FOR GENERATION
`OF MULTIPLE VIEWPOINTS INTO A SCENE
`VIEWED BY MOTIONLESS CAMERAS AND
`FOR PRESENTATION OF A VIEW PATH
`
`CROSS REFERENCE TO RELATED PATENT
`APPLICATIONS
`
`5
`
`This patent application is related to the following
`copending, commonly assigned patent application:
`Method and System for Generation of Multiple View(cid:173)
`points into a Scene Viewed by Motionless Cameras by, John
`L. W. Furlan and Edward C. Driscoll Jr., filed on Oct. 17,
`2001, Ser. No. 09/982,451
`This application claims priority from U.S. Provisional 15
`Patent Application No. 60/267,538, filed Feb. 9, 2001 and
`entitled Method and System for Generation of Multiple
`Viewpoints into a Scene Viewed by Motionless Cameras.
`This application also claims priority from U.S. Provi(cid:173)
`sional Patent Application No. 60/288,396, filed May 2, 2001 20
`and entitled Method and System for Generation of Multiple
`Viewpoints into a Scene Viewed by Motionless Cameras and
`for Presentation of a View Path.
`
`BACKGROUND OF THE INVENTION
`
`2
`tarted image that the viewer desires to view thus providing
`the viewer with pan, tilt, and zoom (PTZ) operations that can
`be applied to the distorted image. Thus, a viewer can apply
`pan, tilt, and zoom onto the distorted image during the
`transformation so as to provide a view into the scene that is
`substantially the same view as that provided by a camera/
`lens that includes a remote controlled pan, tilt, and/or zoom
`capability (see for example, U.S. Pat. No. Reissue 36,207,
`entitled Omniview Motionless Camera Orientation System,
`10 by Zimmerman et al.).
`U.S. patent application Ser. No. 08/872,525, entitled
`Panoramic Camera provides an example of the use of a
`catadioptric lens to capture a distorted image containing the
`area-of-interest.
`It is also known in the art how to optimize the transfor(cid:173)
`mation process to provide high quality real-time video (see
`for-example, U.S. Pat. No. 5,796,426, entitled Wide-Angle
`Image Dewarping Method and Apparatus, by Gullichsen et
`al.).
`One disadvantage of the Eye Vision technology is that it
`uses remotely controlled moveable cameras to provide the
`pan, tilt and zoom functions required to keep the cameras on
`at least one area-of-interest of the scene that is to be viewed
`from the viewpoint of the camera. Because the camera must
`25 move to track the area-of-interest of the scene as the scene
`evolves over time, the camera must have the capability to
`perform the pan, tilt, and zoom operations sufficiently rap(cid:173)
`idly to keep the area-of-interest of the scene in the field of
`view of the camera/lens combination. This requirement
`30 increases the cost and size of the remotely controlled camera
`and dictates how close the camera/lens combination can be
`placed to the scene. In addition, as the camera/lens combi(cid:173)
`nation is placed further from the scene, the point of view
`captured by the camera (even with zoom) is that view from
`35 a more distant location than the view from a camera that is
`up-close to the scene.
`It would be advantageous to use stationary camera/lens
`combinations to capture the video images of the scene that
`40 do not require mechanical actuators to pan, tilt or zoom to
`track the area-of-interest of the scene. In addition, it would
`be advantageous to use a camera/lens combination that
`would allow placement of the camera/lens combination
`close to a rapidly moving area-of-interest.
`One of the problems in sports broadcasting or any other
`type of broadcasting where unexpected actions can occur
`(for example, in live news or event coverage) is that these
`unexpected actions often occur outside the field of view of
`any camera. Another difficulty is that a video segment or
`instant replay generally must be from a single camera. This
`limits the footage that is available for a commentator to
`discuss.
`Another problem with the prior art is that the commen(cid:173)
`tator cannot change the video segment that is available to
`55 him/her and is limited to discussing the view provided by the
`available video segment. This reduces the commentator's
`ability to provide spontaneous, interesting and creative
`commentary about the action.
`Thus, it would be advantageous to provide a system that
`60 reduces the amount of lost action and that allows a com(cid:173)
`mentator to specify and/or change the view into the action
`during the commentary of the video segment.
`
`1. Field of the Invention
`This invention relates to the field of video image process(cid:173)
`ing.
`2. Background
`It is known in the art how to use multiple video cameras
`to capture multiple viewpoints of a scene and to either select
`a desired video viewpoint of the scene or to recreate a
`desired video viewpoint from a model of the scene (the
`model created from information gathered by the captured
`multiple viewpoints of the scene) (see, for example, U.S.
`Pat. No. 5,745,126 entitled Machine synthesis of a virtual
`video camera/image of a scene from multiple video cameras/
`images of the scene in accordance with a particular perspec(cid:173)
`tive on the scene, an object in the scene, or an event in the
`scene, by Jain et al. and U.S. Pat. No. 5,850,352 entitled
`Immersive video, including video hypermosaicing to gen(cid:173)
`erate from multiple video views of a scene a three(cid:173)
`dimensional video mosaic from which diverse virtual video
`scene images are synthesized, including panoramic, scene
`interactive and stereoscopic images, by Moezzi et al.)
`However, these patents teach maintaining a computerized
`model of the scene and require very significant computer
`processing (even with respect to circa 2001 computers) to
`maintain the model and to generate real-time images from
`the model.
`In addition, U.S. patent application Ser. No. 09/659,621
`discloses technology for tracking an object in a warped
`video image.
`It is also known in the art how to use multiple video
`cameras to capture images from a plurality of viewpoints
`into the scene and to assemble some of these images to
`provide apparent motion of the viewers viewpoint around an
`area-of-interest in the scene. This technology was displayed
`during the telecast of the Superbowl XXXV played on
`January of 2001 in Tampa Fla. (Eye Vision TM video provided
`by CBS Sports and Core Digital Technologies).
`It is known in the art how to capture a distorted image of
`the scene (for example, by using a wide-angle lens or a
`fish-eye lens) and to transform an area-of-interest of the 65
`captured distorted image into a non-distorted view. This
`allows a viewer to specify the area-of-interest of the dis-
`
`45
`
`50
`
`SUMMARY OF THE INVENTION
`Aspects of the present invention provide for the specifi(cid:173)
`cation of a view path through one or more video segments.
`The view-path so defined then is used to determine which
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 25 of 37 PageID #: 61
`
`US 6,741,250 Bl
`
`4
`FIG. 15A illustrates a 'specify view path' process in
`accordance with a preferred embodiment.
`FIG. 15B illustrates a 'grow view path' process in accor(cid:173)
`dance with a preferred embodiment.
`FIG. 15C illustrates a 'specify view path through wide(cid:173)
`angle video frame' process in accordance with a preferred
`embodiment.
`FIG. 16 illustrates a video segment viewing process in
`accordance with a preferred embodiment.
`FIG. 17 illustrates a view path modification process in
`accordance with a preferred embodiment.
`
`DESCRIPTION OF THE PREFERRED
`EMBODIMENTS
`15 Notations and Nomenclature
`The following 'notations and nomenclature' are provided
`to assist in the understanding of the present invention and the
`preferred embodiments thereof.
`Procedure-A procedure is a self-consistent sequence of
`20 computerized steps that lead to a desired result. These steps
`are defined by one or more computer instructions. These
`steps can be performed by a computer executing the instruc(cid:173)
`tions that define the steps. Thus, the term "procedure" can
`refer (for example, but without limitation) to a sequence of
`25 instructions, a sequence of instructions organized within a
`programmed-procedure or programmed-function, or a
`sequence of instructions organized within programmed(cid:173)
`processes executing in one or more computers. Such a
`procedure can also be implemented directly in circuitry that
`performs the steps.
`Overview
`One aspect of the invention is an improvement to the
`methods, apparatus, and systems that are used to select a
`viewpoint of a video stream of an area-of-interest of a scene.
`35 Prior Art
`FIG. 1 illustrates a multiple moveable camera configura(cid:173)
`tion 100 having a defined scene 101 within which exists at
`least one area-of-interest 103 that is tracked by at least some
`of moveable cameras 105. Each of the moveable cameras
`40 105 has a field-of-view 107 that may or may not be the same
`field of view as that of another of the moveable cameras 105.
`As the area-of-interest 103 moves within the defined scene
`101, some of the moveable cameras 105 move (for example,
`they pan, tilt, and/or zoom) to keep the area-of-interest 103
`45 within the field-of-view 107 of each of the moveable cam(cid:173)
`eras 105 that follow the area-of-interest 103. One skilled in
`the art will understand that while the pan and tilt movements
`use actual movement of the camera, the zoom function is
`either a movement of a lens element, and/or an electronic
`zoom applied to the image received by the camera through
`a fixed lens. Because each of the moveable cameras 105 are
`in different locations with respect to the area-of-interest 103
`the moveable cameras 105 provide a plurality of different
`spatial viewpoints of the area-of-interest 103.
`The camera movement is performed by mechanical means
`(such as by a robotic actuator 109 shown with one of the
`moveable cameras 105). The movements of the moveable
`cameras 105 can be synchronized by an operator providing
`input to a computerized controller 111 ( or a tracking pro-
`60 gram in communication with the computerized controller
`111) that uses a network 113 to communicate the pan, tilt and
`zoom parameters to the robotic actuator 109 of each of the
`moveable cameras 105 to position at least some of the
`moveable cameras 105 to follow and capture a video image
`65 of the area-of-interest 103 as the area-of-interest 103 moves
`through the defined scene 101. By selecting a specific
`moveable camera, or by selecting frames from each of the
`
`3
`video frames in the video segments are used to generate a
`view. The video frames can include a wide-angle distorted
`image and the view path need not cause the display of the
`entire video frame. In addition, the view path can dwell in
`a particular video frame to provide pan, tilt, and zoom effects 5
`on portions of the video frame. Furthermore, the view-path
`can jump between video segments to allow special affects
`(for example, but not limited to) revolving the presented
`viewpoint around the area-of-interest.
`The foregoing and many other aspects of the present 10
`invention will no doubt become obvious to those of ordinary
`skill in the art after having read the following detailed
`description of the preferred embodiments that are illustrated
`in the various drawing figures.
`
`DESCRIPTION OF THE DRAWINGS
`
`30
`
`FIG. 1 illustrates a multiple moveable camera configura(cid:173)
`tion for capturing a view of interest known to the prior art.
`FIG. 2 illustrates a symmetrical stationary-camera con(cid:173)
`figuration for capturing a view of interest in accordance with
`a preferred embodiment.
`FIG. 3 illustrates an asymmetrical stationary-camera con(cid:173)
`figuration for capturing a view of interest in accordance with
`a preferred embodiment.
`FIG. 4A illustrates placement of a 180 degree partial
`annular image as collected by a digital imager in accordance
`with a preferred embodiment.
`FIG. 4B illustrates placement of a conical image as
`collected by a digital imager in accordance with a preferred
`embodiment.
`FIG. 4C illustrates placement of a truncated conical image
`as collected by a digital imager in accordance with a
`preferred embodiment.
`FIG. 5 illustrates a first camera configuration for a playing
`area used as the defined scene in accordance with a preferred
`embodiment.
`FIG. 6 illustrates a second camera configuration for a
`sports ring used as the defined scene in accordance with a
`preferred embodiment.
`FIG. 7 illustrates a third camera configuration for a sports
`ring as the defined scene in accordance with a preferred
`embodiment.
`FIG. SA illustrates a video creation process in accordance
`with a first preferred embodiment.
`FIG. SB illustrates a video creation process in accordance
`with a second preferred embodiment.
`FIG. 9 illustrates a wide-angle video frame in accordance 50
`with a preferred embodiment.
`FIG. 10 illustrates a perspective-corrected view frame
`selected from the wide-angle video frame of FIG. 9 in
`accordance with a preferred embodiment.
`FIG. llA illustrates a view path through a wide-angle 55
`video segment in accordance with a preferred embodiment.
`FIG. llB illustrates a view path through multiple wide(cid:173)
`angle video segments in accordance with a preferred
`embodiment.
`FIG. 12 illustrates an interactive view selection system in
`accordance with a preferred embodiment.
`FIG. 13 illustrates an overview of the operation of the
`view selection system illustrated in FIG. 12 in accordance
`with a preferred embodiment.
`FIG. 14 illustrates a 'select video segment' process in
`accordance with a preferred embodiment.
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 26 of 37 PageID #: 62
`
`US 6,741,250 Bl
`
`10
`
`35
`
`5
`moveable cameras 105, a video representation of the area(cid:173)
`of-interest 103 can be selected/created.
`Some of the disadvantages of the prior art are that
`mechanical actuator mechanisms are expensive, delicate,
`require sufficient unobstructed space around the camera so 5
`as to move the camera and lens, are large enough to obstruct
`a viewer's view into the defined scene 101 (if the camera is
`between the viewer and the area-of-interest 103), and are
`unable to follow the area-of-interest 103 if it moves suffi-
`ciently rapidly through the defined scene 101.
`In addition, it is very difficult to synchronize each of the
`moveable cameras and to align the images from the cameras.
`The mechanical aspects of the PTZ cameras result in small
`mismatches in the view received by each of the cameras.
`This mismatch requires that (for some special effects such as
`the viewpoint pan) the video processor (or computer) that 15
`receives the frames from each of the cameras must align and
`otherwise connect the frames. In addition, vibrations (such
`as environmental vibrations and/or vibrations of the opera(cid:173)
`tion of the PTZ camera) often cause the mechanical features
`to become out of alignment.
`Another disadvantage with PTZ cameras is that to track a
`rapidly moving area-of-interest 103 these cameras must be
`placed sufficiently far from the scene so that the area-of(cid:173)
`interest stays within the field of view of the PTZ camera as
`it moves. This placement of the camera far from the scene 25
`limits the ability to provide an "up-close" viewpoint of the
`area-of-interest even with zoom. For example, the difference
`in effect is that of watching an approaching locomotive from
`5 feet away from the track as compared to watching the
`locomotive from 500 feet from the track with lOOx binocu- 30
`lars. Furthermore, the noise and motion of a PTZ camera can
`be distracting when placed close to the scene.
`For clarity, FIG. 1 shows only one of the robotic actuators
`as being connected to the network 113. In general, the
`robotic actuators for each of the moveable cameras 105 are
`connected to the network 113 by wire or wireless means.
`Detailed Description
`One aspect of the invention is an improvement to the
`multiple moveable camera configuration 100 shown in FIG. 40
`1. This improvement uses a novel combination of existing
`technologies to address the problems previously discussed
`with respect to FIG. 1 (as well as other problems).
`FIG. 2 illustrates a symmetrical stationary-camera con(cid:173)
`figuration 200 as applied to the defined scene 101. In the 45
`symmetrical stationary-camera configuration 200, the area(cid:173)
`of-interest 103 is tracked by stationary cameras 205 each
`having a field-of-view 207. The field-of-view 207 is suffi(cid:173)
`ciently wide such that the entirety of the defined scene 101
`is within the field-of-view 207. Using this configuration, the 50
`area-of-interest 103 is always within the field-of-view 207 of
`each of the stationary cameras 205. Each of the stationary
`cameras 205 is connected to either a dedicated video pro(cid:173)
`cessor 209 or a shared video processor 211 that are in turn
`connected to a computerized controller 213 by the network 55
`113. For clarity, not all of the network connections nor are
`all of the camera-to-video processor connections shown in
`FIG. 2. In some embodiments, the dedicated video processor
`209 can be included inside the camera. The video stream can
`also be captured either by the dedicated video processor 209 60
`(internal or external to the camera) or the shared video
`processor 211 can be locally stored for later retrieval.
`The network 113 can be an optical fiber network, an
`Ethernet, a star network, a multiplexed network etc. so long
`as the needed bandwidth and distance requirements are met. 65
`At least some of the stationary cameras 205 use a dis(cid:173)
`torting lens (for example, a wide-angle lens or a catadioptric
`
`6
`lens) to capture a sufficiently wide field-of-view. The video
`frames containing such wide-angle video views make up a
`distorted video stream.
`Using technology similar to that disclosed by Zimmerman
`et al. in U.S. Pat. No. Reissue 36,207, a portion of each video
`frame of the distorted video stream captured by any par(cid:173)
`ticular camera of the stationary cameras 205 could be
`transformed in the dedicated video processor 209 or the
`shared video processor 211 to generate a non-distorted
`image of the portion of the distorted video stream frame that
`contains the area-of-interest 103. However, the techniques
`disclosed by Zimmerman are not fast enough to maintain an
`acceptable video frame rate in real-time unless expensive
`specialized dedicated hardware or extremely fast computers
`are used.
`In comparison to Zimmarman, techniques disclosed by
`Gullichsen et al. in U.S. Pat. No. 5,796,426 allow high-speed
`transformations while still maintaining high image quality.
`By using the techniques disclosed in Gullichsen et al., a high
`20 quality undistorted image of the area-of-interest 103 can be
`produced sufficiently quickly to support a video represen(cid:173)
`tation of the area-of-interest 103 in real-time.
`Thus, each of the stationary cameras 205 captures a
`distorted image containing the area-of-interest 103. The
`portion of the distorted image containing the area-of-interest
`103 (in one embodiment, as specified to the video processor
`associated with the camera by the computerized controller
`213) is then transformed by the video processor, at video
`frame rates, to generate a video frame containing a non(cid:173)
`distorted image of the area-of-interest 103. These video
`frames (that are equivalent to video frames generated by any
`of the moveable cameras 105) can then be processed using
`prior art techniques to provide a video representation of a
`user-specified viewpoint of the area-of-interest 103 or to
`provide special effects that utilize the availability of multiple
`viewpoint video streams of the area-of-interest 103 to create
`a video representation.
`One skilled in the art will also understand that the video
`processors 209, 211 can self determine the viewpoint by
`tracking particular portions of the area-of-interest 103 and
`communicate the self-determined viewpoint to the comput(cid:173)
`erized controller 213.
`The symmetrical stationary-camera configuration 200
`also provides the ability to provide "instant replay" capa(cid:173)
`bility where the "instant replay" is from a different view(cid:173)
`point into the defined scene 101 than the viewpoint provided
`by the live sequence. Because the entirety of the defined
`scene 101 is captured in every video frame (and in this
`instance, recorded at the dedicated video processor 209 or
`the shared video processor 211), views of areas of the
`defined scene 101 can be provided that were found to be of
`interest after the "live" representation of the initial area-of(cid:173)
`interest was provided. Thus, allowing "instant replay" capa(cid:173)
`bility from unexpected viewpoints into unexpected areas(cid:173)
`of-interest. For smooth, around-the-object tracking (without
`the use of a computer model), the cameras should be placed
`with a suitably small angular difference from the scene (such
`as 20-degrees or less).
`In addition, the area-of-interest 103 can be automatically
`tracked as it moves through the defined scene 101 by using
`a distorted image with techniques such as those disclosed in
`U.S. patent application Ser. No. 09/659,621. For example,
`the dedicated video processor 209 or the shared video
`processor 211 can also execute image tracking programs that
`determine when a portion of the area-of-interest 103 is
`facing the stationary camera attached to the image processor.
`
`

`

`Case 1:21-cv-00227-UNA Document 1-2 Filed 02/18/21 Page 27 of 37 PageID #: 63
`
`US 6,741,250 Bl
`
`25
`
`8
`FIG. 4Aillustrates an image placement configuration 400
`having a digital imager 401 within a camera upon which a
`partial annular image 403 is positioned by the camera and
`catadioptric lens optics. An unused imager portion 405 is the
`5 portion of the digital imager 401 outside the partial annular
`image 403. The characteristics and advantages of an annular
`image are described in U.S. patent application Ser. No.
`08/872,525.
`FIG. 4B illustrates an image placement configuration 440
`using a digital imager 441 within a camera upon which a
`conical image 443 is positioned by the camera and conical
`lens optics. An unused imager portion 445 is the portion of
`the digital imager 441 outside the conical image 443.
`FI