United States Patent (19)
`Ritchey
`
`|||||||||
`US00549.5576A
`11
`Patent Number:
`5,495,576
`(45) Date of Patent:
`* Feb. 27, 1996
`
`54)
`
`76)
`
`*
`
`21
`22)
`51
`52)
`58)
`
`56)
`
`PANORAMIC MAGE BASED WIRTUAL
`REALITYFTELEPRESENCE AUDIO-VISUAL
`SYSTEMAND METHOD
`
`Inventor: Kurtis J. Ritchey, 26374 Tonganoxie
`Rd., Leavenworth, Kans. 66048
`The portion of the term of this patent
`subsequent to Jul. 14, 2009, has been
`disclaimed.
`
`Notice:
`
`Appl. No.: 2,582
`Filed
`Jan. 11, 1993
`Int. Cl. ...r. G06T 15/10
`U.S. C. .........................
`395/125; 395/154
`Field of Search ..................................... 395/119, 125,
`395/130, 154, 50,902, 2.55, 2.6, 2.79,
`2.85; 34.5/139, 158, 184
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,868,682 9/1989 Shimizu et al. ........................ 358/335
`4,951,040 8/1990 McNeil et al. .......................... 340,729
`Primary Examiner-Mark K. Zimmerman
`Attorney, Agent, or Firm-Litman, McMahon, & Brown;
`
`Dennis Crawford
`ABSTRACT
`57)
`An improved panoramic image based virtual reality/telep
`resence audio-visual system and method includes panoramic
`three-dimensional input devices, a computer processor, and
`a panoramic audio-visual output device. In one embodiment
`of the system the input devices comprise a sensor assembly
`including a plurality of positionable radar, camera, and
`acoustical sensors for recording signatures of all sides of
`three-dimensional subjects simultaneously. The computer
`processor integrates the sensor signals, processes signals as
`a virtual model, updates the model based on participant
`interaction, and selects and distributes portions of the pro
`cessed virtual model for presentation on display units and
`audio speakers. The processor includes participant interac
`tive input devices for instantaneous interaction with the
`virtual model. The panoramic audio-visual output device
`includes a head-mounted display or a closed structure hav
`ing contiguous individual display units mounted in all
`viewable directions surrounding the participant. Conven
`tional, stereoscopic, autostereoscopic, and holographic dis
`play systems are provided to view the panoramic three
`dimensional image based model. Computer graphics,
`artificial intelligence, telecommunications, and vehicle con
`trol embodiments of the system are provided.
`26 Claims, 13 Drawing Sheets
`
`l
`PANORAMC
`PANORAMC
`ar
`VISUAL SIGNAL
`AUDIO SIGNAL
`2
`PROCESSENG
`PROCESSING
`3 - 1 - - - - -
`5:
`NPUT
`wn
`PAN.
`
`8
`
`SOURCE
`
`SC
`
`POSITION
`NG 8
`
`6So
`SiGNALS 228
`O SUB
`JECTS IN
`PAN.
`THE
`AUDO
`PROC, MODEL
`MEANS
`
`ION OF
`PAN.
`COMPUT.
`GENERAT
`EDMODE
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`FUSON OF SENSOR
`SIGNALS TO FORM
`MODEL SEGMENTS
`
`FUSON
`OF MOOEL
`SEGMENTS
`
`as
`
`-
`GRAPH
`CS
`VISUAL
`MASS
`SIMULATION - STOR.
`OF PAN.
`(VIS)
`
`ST
`Eis
`15
`ÉEc
`MEANS
`PAN.
`SIG.
`PROC.
`MEANS
`
`
`
`IMAGE PROC.
`SFORDS. 8
`DISPLAY
`
`PARTICIPANT
`CONTROL OF
`NTERACTIVE
`INPUT DEVICE
`
`NTER.
`SSEM
`22
`O
`SCM
`APAN.
`AN
`SEA
`gi CONVAUDVIS,
`PANORAMC
`ASSEM.
`EastEENS
`STER.
`AUTO
`MODEL
`SE
`(AUDIO 8
`3, Holol
`VISUAL)
`
`w
`
`t; 2
`
`
`
`RPX Exhibit 1003
`
`

`

`U.S. Patent
`
`Feb. 27, 1996
`
`Sheet 1 of 13
`
`5,495,576
`
`PANORAMC
`AUDIO SIGNAL
`PROCESSING
`
`PANORAMIC
`VISUAL SIGNAL
`PROCESSING
`
`1.
`re
`
`2
`
`V N
`
`3-D AUDIO
`3-D SHAPE6 3-D IMAGE
`PAN.
`SSE 7&SEM&E INPUT
`31 INPUT
`INPUT
`INPUT
`SOURCE
`- 5
`NTHSat
`NTH
`NTH
`FUSION OF SENSOR
`SIGNALS TO FORM
`MODEL SEGMENTS
`5
`
`s
`:
`
`ST
`PROC.
`MEANS
`/5
`2ND
`E.S
`/6
`
`2Gh
`
`POSITION
`ING 8
`of 53. 266
`6SSo
`SSSIs 228, 2O, SEGMENTS
`DEV.
`s
`
`2.
`
`AUDIO
`THE
`Eis MODEL
`5
`
`
`
`SIMULAF
`TIONS
`COM5.
`GENERAT
`ED MODEL
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`O
`
`GRAPH
`5 --SES
`A.K
`VISUAL
`5
`SIMULATION
`OF PAN.
`
`
`
`
`
`E.
`
`5
`
`s'
`
`DISPLAY
`
`PROC
`MEANS
`3,
`3RD ()
`PROC,
`MEANS
`/7
`4TH FRS
`
`MEANS
`18
`
`CONTROL OF
`INTERACTIVE
`sc span.
`INPUT DEVICE
`BS37 CONV. ASSES
`SEEastEREENS
`MODE
`AYgle
`(AUDIO 8
`M :
`VISUAL)
`
`es 3.
`
`RPX Exhibit 1003
`
`

`

`U.S. Patent
`
`Feb. 27, 1996
`
`Sheet 2 of 13
`
`5,495,576
`
`
`
`N
`
`-
`
`
`
`
`
`
`
`
`
`
`
`NW2/Z, f
`
`561 ROC.
`
`OVERLAPPNG
`FIELD-OF-REGARD COVERAGE
`SMC? 24
`RADAR
`SAMERez
`7 N
`1.
`HONY So-
`85 86-8
`say
`N NS N
`NiaS V VA
`VY AN
`Les AAYAAAs see AYAs a
`
`N
`
`
`
`
`
`UDIO
`AY39
`P
`
`Fig.5.
`
`PROC.
`
`RPX Exhibit 1003
`
`

`

`U.S. Patent
`US. Patent
`
`Feb. 27, 1996
`Feb. 27, 1996
`
`Sheet 3 of 13
`Sheet 3 of 13
`
`5,495,576
`5,495,576
`
`
`
`54, 5b, 5C
`5a) 513,55
`
`
`
`RPX Exhibit 1003
`
`RPX Exhibit 1003
`
`

`

`U.S. Patent
`US. Patent
`
`Feb. 27, 1996
`Feb. 27, 1996
`
`Sheet 4 of 13
`Sheet 4 of 13
`
`5,495,576
`5,495,576
`
`
`
`6.
`
`. . .
`.. . .
`
`Esme. 56b-8
`
`561; -b, ssh-4.
`scb-d
`
`n—fi-~
`
`_
`
`A 4‘
`lll1l_llll
`[ \
`
`RPX Exhibit 1003
`
`RPX Exhibit 1003
`
`

`

`U.S. Patent
`US. Patent
`
`Feb. 27, 1996
`Feb. 27, 1996
`
`Sheet S of 13
`Sheet 5 of 13
`
`5,495,576
`5,495,576
`
`
`
`
`
`RPX Exhibit 1003
`
`RPX Exhibit 1003
`
`

`

`U.S. Patent
`
`Feb. 27, 1996
`
`Sheet 6 of 13
`
`5,495,576
`
`IMAGE
`
`-
`
`4 M
`
`3-D
`COMPUTER
`GENERATED
`MODEL
`|-a-b 1 ty
`/ WIREFRAME
`
`AUDIO SIG.
`RECORDING
`
`
`
`
`
`RPX Exhibit 1003
`
`

`

`US. Patent
`
`Feb. 27, 1996
`
`Sheet 7 of 13
`
`5,495,576
`
`\Rm3m:>
`
`moqummo
`
`\mmgm;
`
`mOqummo
`
`Aim
`
`:
`
`omkzaoz
`o<mx
`
`><4ama
`
`>Jm2mmm<
`
`mead:—
`
`.
`
`>4.._n_m_n_
`
`>.._m_2mmm<
`
`M
`
`£9
`
`w>_._.0<mm:.z_
`
`sakmtw
`
`._.Dn_z_
`
`._.mOI
`
`fivimhm>m
`mugbmzoo
`
`FmOI
`
`KMHDQEOO
`
`@555
`
`3.38m>_5<mm:z_
`
`EMFm>m
`
`.552.
`
`_,69«mm
`
`
`
`¢mz<m=2>4m2wwm<><u_n_m_o
`
`
`mz<m_2oz_mmmoomn_44203Nmz<m=2
`cum qlllL
`
`
`
`momnom.Saz_
`
`o_2<moz<n_
`
`mMNrEoE
`
`Emkmrw
`
`0.2452,».
`
`<mm2<o
`
`2mkm>m
`
`nvm
`
`o__2<moz<n_
`
`Suhmrm
`
`05:4
`
`cum
`
`RPX Exhibit 1003
`
`RPX Exhibit 1003
`
`
`
`
`
`
`
`
`
`

`

`US. Patent
`
`Feb. 27, 1996
`
`Sheet 8 of 13
`
`5,495,576
`
`m
`
`Em.IsonI mz<m2
`
`>I_m_2mmm<>43me
` mz<m2
`
`oz_mmmoomn_J<Z@_m
`
`
`mzfizmumzom.sz_
`
`m
`
`mowaow«thin
`
`5=mm$
`
`.3,
`
`ogflmmz<a
`
`KMNELQE
`
`2mkm>m
`
`Pmo:
`
`cum
`
`$55.28It,
`o_5_<moz<n_
`
`<mM2<o
`
`Ewhm>m
`
`Suhmrm
`
`m>co<mMFz_
`
`smkm>m
`F252.
`
`F2¢z_
`
`m>_5<mEz_dl1£39@2596.
`
`Gin...
`
`o.2<moz<n_
`
`05:4
`
`Sukm>m
`
`RPX Exhibit 1003
`
`RPX Exhibit 1003
`
`
`
`
`
`
`
`

`

`U.S. Patent
`US. Patent
`
`Feb. 27, 1996
`Feb. 27, 1996
`
`Sheet 9 of 13
`Sheet 9 of 13
`
`5,495,576
`5,495,576
`
`
`
`24m._.\._.Z<ill93.oomn.A035ERor,mi._\u:«a.
`..ill._.zoow._a.0cm'me
`
`
`mp—KEMn—OI'-ZVON.fink..n=>_OU_\V2SKA
`\mwgfimw_>.'I.roOQLoom‘FmOIuh.O__o.
`
`
`
`
`
`
`EuEuww8.504%.mz<m_25Eodumwas;$20050:.m>mmmx=2.ooma29m:mumz<m2humm:
`
`
`
`oz;>w.>.m_>-.n5<Mmz<wzwzfimmooma.2206momnombiz.fa
`
`wnzmI'-Ikzoocom.00mm.240anAm}..85SQ.mu&P:m<o<mE.
`
`wn=mHZOU.0mmnoQO
`amAwoku.30:23.r4“n“eomdwom<I.wow—cum
`waokéflill38.33a-..,..mmIIImmi.
`.IIIHzoo_SE3Ba
`I.-ZQa?48mi
`
`Q
`
`
`
`
`
`
`
`
`
`
`
`
`
`IIIa.-.“BEAg“was.asanarzwmwvmnn:um.um2085IIIsvoméomémmm.mm7x1.”_.5
`
`
`
`
`
`..QOI"5.2002m...m>m,.mmaQO
`
`m.Smd
`
`mewIII928H24«mm\wmm.o%ma3mam
`$8-...6m-.sueIIIL3mm.wmwsageI.u.
`
`IIIE.“52N»kA
`ommEm3%m
`.mz<EE8..
`
`$.22:
`
`2amm
`
`RPX Exhibit 1003
`
`RPX Exhibit 1003
`
`
`
`
`

`

`U.S. Patent
`
`Feb. 27, 1996
`
`Sheet 10 of 13
`
`5,495,576
`
`3-D
`l,71
`3-D
`S-D
`INPUT SOURCE
`12
`SIGNAL
`MEANS
`PROCESSING MEANS DISPLAY MEANS
`- -
`- - -
`-
`e
`7WTH23NTH 3
`4OW7h
`HOST
`ATA coMER PR DiSEAY as 5ANTH
`14a,25d.
`PAN,
`NTH A7–SksPKNTH
`3-D IMAGE
`DATA BASE
`e fa,234)
`splay
`MASS STOR,
`HOST
`is
`DISPLA
`cois TER
`D'A'.
`|b,25b
`3-5Dio T "E"
`EAIAEASE
`KSPK. A 4-35a
`MASS STOR,
`82
`q.7/SP
`as
`Fit
`opT 255NS9.
`TMING
`3.
`- PROCESSING
`5s
`|CONB9 ER
`UNIT
`(CLOCK)
`File:F----\a ?o,78
`Fig.16.
`88
`O
`
`6
`
`4c -
`4c4
`fi
`32,33\
`1e/-SY 2
`PSEN22 ea
`ASSEMBLY 21-4 4-2
`"le 12212
`%
`
`390 A-8 2
`
`
`
`al
`
`Oa
`
`RPX Exhibit 1003
`
`

`

`U.S. Patent
`
`Feb. 27, 1996
`
`Sheet 11 of 13
`
`5,495,576
`
`
`
`
`
`
`
`
`
`
`
`INPUT SOURCE
`la
`7
`AUD.-VIS.
`3-D
`PANORAMC ASSEMBL
`INTER.
`NPUT
`DIGITIZER
`MEANS Y
`SYSTEM
`SYSTEM
`11, 12
`92
`43 To
`6
`LOCATION
`ENSSPEE/Encrypt.
`ESSE
`PANöRAMC
`3-D
`gER systEMS COMPRESS.
`9 )78
`
`i49. 20.
`
`O
`
`l,20
`VIEWER/\ ^
`se
`OPERATOR
`-2
`
`
`
`
`
`
`
`3-D
`PANORAMC
`AUDO
`SYSTEM
`
`DECODER/
`EXPANDER
`
`FROM
`LOCATION
`
`
`
`EECOMM.h.2Od
`UNIT A
`
`LOCATION A
`
`LOCATION B
`
`RPX Exhibit 1003
`
`

`

`U.S. Patent
`
`Feb. 27, 1996
`
`Sheet 12 of 13
`
`5,495,576
`
`fl. 2
`
`DISPLAY WIEWING
`MEANS
`SPACE
`
`t
`
`tent
`3 LADAR5. 2
`O
`SIDE CAM. f.
`MIC, or
`Sar Tib
`
`PROC
`150
`A
`AXE 23d
`
`rod DELAYSE
`A
`SPEAKER
`35. A
`SIDE
`
`70
`
`
`
`
`
`N
`
`&
`
`S. S
`
`Nyke VS
`
`RPX Exhibit 1003
`
`

`

`U.S. Patent
`
`Feb. 27, 1996
`
`Sheet 13 of 13
`
`5,495,576
`
`l2.
`
`SPHEBIACQYSRAGE
`DISPLAY ASSEMBLY MEANS
`
`O1a
`
`SRS
`TRANS-
`CEIVER
`
`
`
`OPERATOR
`255RAS
`
`
`
`G
`SIGNAL
`PROCESS.4
`UNIT
`
`
`
`
`
`
`
`
`
`
`
`46
`
`REMOTELY PLOTED
`VEHICLE
`
`3-D PAN.
`IMAGING,
`SHAPE 8
`AUDIOSYS.
`
`
`
`RPX Exhibit 1003
`
`

`

`1.
`PANORAMIC MAGE BASED WIRTUAL
`REALITYITELEPRESENCE AUDO-VISUAL
`SYSTEMAND METHOD
`
`BACKGROUND OF THE INVENTION
`
`O
`
`15
`
`25
`
`30
`
`35
`
`1. Field of the Invention
`This invention relates generally to panoramic display
`methods and more particularly to the sensor fusion of data
`from the panoramic arrangement of three-dimensional imag
`ing sensors and surface contour sensors to form virtual
`objects and scenes, the processing of the virtual objects and
`scenes based on a viewer operating interactive computer
`input devices to affect the manipulation of the virtual objects
`and scenes defined in the computer, and the display of the
`affected virtual objects and scenes on a panoramic display
`unit to the extent that the viewer perceives that the virtual
`objects and scenes completely surround the viewer.
`2. Description of the Related Art
`20
`My previous U.S. Pat. No. 5,130,794 describes a pan
`oramic image based virtual reality system that incorporates
`a multi-lens camera system with spherical field-of-view
`(FOV) coverage. As shown in FIG. 2, objective lenses of the
`'794 camera system face outward with adjacent or overlap
`ping FOV coverage. The imagery from the camera is surface
`mapped onto the interior of a three-dimensional(3-D) shape
`defined in a special effects processor of a computer. Alter
`natively, the input source is at least one computer graphics
`system that generates three-dimensional graphics of spheri
`cal FOV coverage. The viewer operates interactive input
`devices associated with the computer to manipulate the
`texture mapped virtual images. The virtual environment is
`instantaneously affected before the viewer and displayed on
`either a head-mounted display assembly or on contiguous
`display units positioned beneath, to the sides, and above the
`viewer.
`Limitations of the panoramic video camera system in 794
`are that the panoramic camera does not record a non
`40
`spherical field of view(FOV) and does not incorporate a
`non-contact shape sensor.
`An improvement over the existing system is proposed in
`my Disclosure Document No. 197612, specifically FIG. 15,
`filed with the U.S. Patent and Tradmark Office in February
`45
`1986, and in my recent paper entitled "Image Based Pan
`oramic Virtual Reality System", presented at the SPIE/IS&T
`Symposium on Electronic Imaging: Science & Technology
`92; Visualization, Holography, and Stereographics; Visual
`Data Interpretation, Paper No. 1168-02, on Feb. 9, 1992.
`In these documents a multi-lens camera system with
`positionable taking lenses is described. Taking lenses of the
`camera are faced inward or outward to record imagery of a
`subject in an continuous simultaneous manner. By combin
`ing panoramic visual field of view sensor data with associ
`ated shape sensor data a realistic panoramic image based
`three-dimensional computer generated model is rendered.
`Imagery from the camera is surface mapped onto the surface
`of a three-dimensional shape defined in a computer. The
`shape is input by a panoramic 3-D digitizer device. Audio
`data is input by a panoramic 3-D audio system. Audio
`attributes are assigned to subjects in the model. Shape,
`imagery, and audio sensors may be combined to form one
`sensor array. Sensors are positioned adjacent to one another
`to facilitate adjacent or overlapping coverage of a subject.
`Preferably corresponding panoramic shape, imagery, and
`audio signatures of a subject(s) are collected simultaneously.
`
`50
`
`55
`
`60
`
`65
`
`5,495,576
`
`2
`In this manner action of a 3-D subject is recorded from
`substantially all aspects at a single moment in time. The
`participant operates interactive input devices associated with
`the computer to manipulate the virtual object. In one
`example, the participant observes the model on a head
`mounted display system. In another example, the participant
`is surrounded by contiguous audio-visual display units. In
`the latter example, each display unit displays a segment of
`the model.
`It is therefore the objective of this invention to provide a
`more versatile image based panoramic virtual reality and
`telepresence system and method. Still another objective is to
`produce systems and methods for recording, formatting,
`processing, displaying, and interacting with data represent
`ing 3-D beings, objects, and scenes. More specifically, an
`objective of this invention is to provide a positionable
`multi-lens camera system for recording contiguous image
`segments of an object, being, adjacent surrounding scene, or
`any combination of these types of subjects; a signal pro
`cessing means comprising first computerized fusion pro
`cessing system for integrating the positional camera system
`with corresponding digitized shape and contour data, a
`second computerized fusion processing system for integrat
`ing first fused data with other fused data representing
`adjacent portions of a being, object, or scene comprising a
`panoramic computer generated model; where various 3-D
`digitizer systems may be incorporated for entering 3-D
`shape and contour data into a image processing computer; a
`third processing means to manipulate the geometry of sub
`jects comprising the virtual model; a forth processing means
`for sampling out given fields of regard of the virtual model
`for presentation and distribution to display units and audio
`speakers; where signal processing means includes an expert
`system for determining the actions of subjects of the com
`puter generated model; where the signal processing means
`includes image segment circuit means for distributing, pro
`cessing, and display of the model; where the system includes
`a 3-D graphics computer system for the generation, alter
`ation, and display images; and a system and method for
`image based recording of 3-D data which may be processed
`for display on various 3-D display systems to include head
`mounted display systems, and room display systems with
`stereographic, autostereoscopic, or holographic display sys
`teS.
`It is also an objective of this invention to provide inter
`active input devices operable by a viewer to cause the
`generation, alteration, display of 3-D images on said display
`assembly means; to provide associated 3-D audio systems;
`to provide alternative viewer interactive and feedback
`devices to operate the interactive input devices and associ
`ated processing means such that the resultant virtual envi
`ronment is simultaneously effected before the viewers eyes;
`to provide an associated telecommunications system; and to
`provide a system for incorporation with a host vehicle,
`teleoperated vehicle, or robot.
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a flowchart to which reference will be made in
`generally explaining the overall operation of the recording,
`processing, and audio-visual system 1 according to the
`present invention.
`FIG. 2 is a perspective view of a cameraman carrying a
`panoramic camcorder system of spherical coverage
`described in prior art.
`FIG. 3 is a greatly enlarged fragmentary sectional view of
`one of the camera arrangements for optically recording
`
`RPX Exhibit 1003
`
`

`

`5,495,576
`
`3
`image segments representing sides of a three-dimensional
`subject into a single frame according to the present inven
`tion.
`FIG. 4 is a perspective view of a sensor array for
`recording accoustical, visual, and shape data for input
`according to the present invention.
`FIG. 5 is a side sectional view of the sensor array shown
`in FIG. 4.
`FIG. 6 is a diagrammatic representation of an inward
`looking three-dimensional input source incorporating the
`sensor array shown in FIGS. 4 and 5.
`FIG. 7 is a diagrammatic representation of a inward and
`outward looking panoramic three-dimensional input source
`assembly incorporating the sensor array shown in FIGS. 6
`and 7.
`FIGS. 8A-8D are diagramatic representations of video
`frames of three-dimensional coverage of beings and objects
`to be modeled in 3-D in the present invention.
`FIGS. 9A-9B are diagramatic representations of video
`frames of three dimensional coverage of beings, objects, and
`background scenes, respectively, to be modeled in 3-D in the
`present invention.
`FIG. 10 is a diagramatic representation of a HDTV frame
`on which includes both foreground and background imagery
`necessary to model a virtual environment.
`FIG. 11 is a fragmentary diagrammatic view onto the top
`of the virtual world model in which recorded three-dimen
`sional beings, objects, and/or scenes are Incorporated
`according to the present invention.
`FIG. 12 is a fragmentary diagrammatic view onto the side
`of the virtual model shown in FIG. 11.
`FIG. 13 is a diagramatic illustration showing how imag
`ery is texture mapped onto a three-dimensional wireframe
`model to form a three-dimensional virtual model generated
`and processed for presentation by audio and video computer
`signal processing means of system 1.
`FIG. 14 is a perspective, partially diagramatic view show
`ing an image based virtual model generated for audio-visual
`presentation by the visual signal processing means of system
`1.
`FIG. 15 is a block diagram of an image formatting system
`for recording, processing, and display of an image of three
`dimensional coverage which embodies the present inven
`tion.
`FIG. 16 is a block diagram of a second embodiment of
`system.
`FIG. 17 is a block diagram of a third embodiment of the
`system.
`FIG. 18 is a block diagram illustrating the incorportation
`of a three-dimensional display system according to the
`present invention.
`FIG. 19 is a perspective, partially diagramatic view illus
`trating the three-dimensional viewing system described in
`FIG. 18.
`FIG. 20 is a block diagram of an embodiment of the
`present invention including a telecommunications system.
`FIG. 21 is a perspective, partially diagramatic view illus
`trating the telecommunications embodiment according to
`FIG. 20.
`FIG. 22 is block diagram illustrating an embodiment of
`the present invention wherein a host vehicle control system
`with a panoramic sensor, processing, and display system
`provides telepresence to a viewer/operator for control of the
`host vehicle.
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`FIG. 23 is a sectional view of a host vehicle incorporating
`the present invention shown in FIG. 22.
`FIG. 24 is a block diagram illustrating an embodiment of
`the present invention wherein a remote control system for a
`remotely piloted vehicle with a panoramic sensor system
`transmits a three-dimensional panoramic scene to a control
`station for processing and spherical coverage viewing in
`order to assist the controller in piloting the teleoperated
`vehicle.
`FIG. 25 is a perspective, partially diagrammatic view
`illustrating the remote control three-dimensional viewing
`system described in FIG. 24.
`
`LISTED PARTS IN DRAWINGS
`1: Improved panoramic image based virtual reality/tele
`presenc audio-visual system and method
`2: Panoramic 3-D input source means
`3: Panoramic 3-D signal processing means
`4: Panoramic audio-visual presentation means
`5: Suitable electrical interface means and assoiciated
`signal(general)
`(5a video signal means)
`(5b. digital shape signal means)
`(5c audio signal means)
`6: Panoramic 3-D camera system
`7: Panoammic 3-D digitizing system
`8: Panoramic 3-D audio recording system
`9: Host computer system
`10: Interactive input system
`11: Head-mounted display(HMD) system
`12: Large display assembly
`13: Subject
`(13a; being)
`(13b: object)
`(13c. scene)
`(-a: side a)
`(-b: side b)
`(-c: side c), etc.
`14: Computer generated virtual world model
`(14a: visual and shape model)
`(14b audio model)
`(-a: modeled being)
`(-b: modeled object)
`(-c: modeled Scene)
`15: First processing means; fusion processor to wed shape
`and image segments.
`16: Second processing means; fusion of model segments.
`17: Third processing means; host simulation computer for
`manipulating world model; geometry processor.
`18: Forth processing means; image processing for display
`and distribution.
`19: Computer graphics system
`20: VRT telecommunications system
`21: VRT vehicle control system
`22: Artificial intelligence system
`23: Audio processing system
`24: Participant; viewer/operator)
`(24a: first participant)
`(24b. second participant)
`25: Mass storage device
`(25a: visual and shape mass storage)
`
`RPX Exhibit 1003
`
`

`

`S
`(25b. audio data mass storage)
`26: Panoramic model segments
`(26a: visual and shape model segment)
`(26b; audio model segment)
`27: Sensor(s)
`28: Image sensor(s)
`29: Shape sensor(s)
`30: Audio(Accoustical) sensor(s)
`31: Conventional display unit(s)
`32: Stereographic display unit(s)
`33: Autostereoscopic display unit(s)
`34: Holographic display unit(s)
`35: Audio speaker(s)
`36: Sensor array
`37: Camera
`38: Radar
`39: Microphone
`40: Array housing
`41: Overlapping field of regard coverage of sensors.
`42: Edge of adjacent field of regard coverage of sensors.
`43: Rigid transparent support
`44: Array assembly
`45: Screw
`46: Support armature
`47: Panoramic optical assembly arrangement
`48: Objective lens
`49: Light sensitive surface of the camera
`50: Fiber-optic image conduit(bundle)
`51: Focusing lens
`52: Camera housing
`53: Charge Coupled Device(CCD)
`54: Sheathing of image conduit
`55: Shape data(wireframe) representing subject model
`56: Image data representing a subject
`(56a: being)
`(56b: object)
`(56c. scene)
`57: Audio data representing a subject
`(57a: being)
`(57b: object)
`(57c: scene)
`58: Viewing space
`59: Head position of participant
`60: Hand location of participant
`61: Sample frame of panoramic camera
`62: Transimitter; for transmitting an over-the-air stereo
`audio signal.
`63: Receiver; for recieiving an over-the-air stereo audio
`signal.
`64: Stereo audio headphones
`65: Structural supports of the large display assembly
`66: Graphics input system
`67: Videotape player
`68: Videodisc player
`69: Video analog-to-digital converter
`70: Display unit; generally; may include audio system.
`71: Displayed scene
`72: Image segment circuit means
`
`O
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`5,495,576
`
`6
`73: Image control unit (including chasis, processors, etc.);
`may include audio means.
`74: Polygonal surfaces of model 14a
`75: Head position of viewer
`76: Position sensing system sensor
`77: Position sensing system source
`78: Position sensing system electronics unit
`79: Audio signal to means 4
`80: Video signal to means 4
`81: Display unit viewing surface
`82: Position and orientation data and associated conductor
`from interactive input system 10
`83: Source conductor line
`84: Sensor conductor line
`85: Radar antenna
`86: Radar waveguide
`87: Radar transmitterdreciever
`88: Master clock
`89: Conventional signal routeriswitcher
`90: 3-D display unit; generally.
`91: 3-D display system embodiment of system 1.
`92: Encoder/compressor
`93: Encryptor
`94: Modem
`95: Decryptor
`96: Decoderlexpander
`97: Non-contact position and orientation sensor system
`(i.e. Radar or LADAR); may include camera system.
`98: Digital data network
`99: Telephone line
`100: Edge of projected image
`101: Floor of large assembly
`102: Host vehicle
`103: Host vehicle controls
`104: Host vehicle control surfaces and motors.
`105: Rear projection screen
`106: Entry/exit assemblies for assembly 12
`107: Structural support, framework, and fasteners for
`large assembly 12.
`108: Remotely piloted vehicle
`109: Tranceiver; for sending and recieving radio frequen
`cy(RF) over-the-air digital data.
`110: Over-the-air RF digital data link
`111: Participant support means
`112: Remote vehicle control system
`113: Remote vehicle control surfaces and motors
`114: Remote vehicle manipulators
`115: Timing signal conductor
`116: Model signal conductor
`117: Processing means for conventional TV
`118: Processing means for stereo display TV
`119: Processing means for autostereoscopic TV
`220: Processing means for holographic TV
`221: Processing and distribution system for image seg
`ment circuit means
`222: Audio-visual units of image segment circuit means
`223: Hemispherical scan of LADAR system; may include
`integral registered camera system.
`
`RPX Exhibit 1003
`
`

`

`5,495,576
`
`7
`224: Near field of view of LADAR system; may include
`integral registered camera system.
`225: VRT control station for remotely piloted vehicle
`226: Video compression and data system (including com
`munications buffer)
`227: Video decompression and data system (including
`communications buffer)
`228: Peripherial devices
`
`8
`14a. The third processing means is typically operated to
`perform interactive 3-D visual simulation and teleoperated
`applications. Signal processing means 3 also includes a
`fourth computer processing means 18 to sample out and
`transmit image scene 71 (FIG. 24) segments of the world
`model 14a to each respective display unit of the audio-visual
`assembly means 2. Means 2 includes processing means for
`interfacing with input sources 2, peripheral computer data
`entry and manipulation apparatus refered to as an interactive
`input system 10, and assembly 4. Signal processing means
`15, 16, 17, 18, and 23 (FIG. 17) include a central processing
`unit, terminal bus, communication ports, memory, and the
`like typical to a conventional computer(s). Operating system
`software, board level software, processing data, generated
`images and the like are stored in mass storage devices 25
`which may include disk drives, optical disk drives, and so
`forth. All signal processing means 15, 16, 17, 18, and 23 may
`be incorportated into a single computer 9 or a plurality of
`networked computers (9 to the nth) housed in a single or
`separate chassis. Additionally, means 3 may include a com
`puter graphics system 19, a telecommunications system 20,
`a vehicle control system 21, or artificial intelligence system
`22 to perform special processing functions. Special process
`ing systems 19, 20, 21, and 22 may be integral or networked
`to computer 9.
`Audio sensors 30 are faced inward about a subject or
`outward to record signals representing audio segments 26b
`of a surrounding subject. Preferably, the image, shape, and
`audio sensors 28, 29, and 30 (FIG. 5) respectively, are
`positioned adjacent to one another and record a continuous
`corresponding subject 13. The audio processing system 23
`receives recorded audio signals 5c from the panoramic 3-D
`audio input system 1. The audio signals 5c as assigned to
`modeled subject 14a comprise an accoustical world model
`14b. The audio model 14b is continuously updated by the
`computer 23 based on data recieved from the interactive
`input system 10. Computer 9 communicates changes to the
`world model 14 via digital data link interconnected to
`computer 23. Audio means 23 includes processing means
`and software means for the generation of 3-D audio output
`in response to changes and actions of subjects modeled in
`the computer generated model 14a. The output audio signals
`are transmitted to speakers positioned about the participant
`by way of the panoramic audio-visual assembly means 4.
`The preferred embodiment of the system 1 may generally
`comprise two alternative panoramic audio-visual assembly
`means 4: A headmounted display (HMD) assembly 11, or a
`large display assembly 12. The large display assembly 12
`may incorporate conventional 31, stereographic 32, auto
`stereographic 33, or holographic 34 display units. Specific
`processing means 18 compatible with a given display unit's
`31, 32, 33, or 34 format operate on the virtual model 14a.
`The processing means 18 then outputs a signal representing
`a model segment 26a to a predetermined display unit 31, 32,
`33 or 34. Display units 31, 32, 33, or 34 are placed
`contiguous to one another in a communicating relationship
`to the participant such that a continuous scene is presented
`to the participant. In this manner the same basic imagery,
`shape, and audio data is rendered into a model 14a that may
`be operated upon for presentation on conventional, stereo
`graphic, autostereoscopic, or holographic display Systems.
`The model 14 presented to the participant may be derived
`from prerecorded data stored in a mass storage device 25.
`Alternatively, live feeds from input sources 2 at a remote
`location are processed in near real time and the participant
`can interact with the remote location by using teleoperated
`devices. In these manners the viewer is immersed in a highly
`interactive and realistic computer simulation.
`
`10
`
`15
`
`25
`
`30
`
`40
`
`35
`
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENTS
`As required, detailed embodiments of the present inven
`tion are disclosed herein; however, it is to be understood that
`the disclosed embodiments are merely exemplary of the
`invention which may be embodied in various forms. There
`fore, specific structural and functional details disclosed
`herein are not to be interpreted as limiting, but merely as a
`basis for teaching one skilled in the art to variously employ
`the present invention in virtually any appropriately detailed
`Structure.
`For clarity of description, a preliminary summary of the
`major features of the recording, processing, and display
`portions of a preferred embodiment of the system is now
`provided, after which individual portions of the system will
`be described in detail.
`Referring to the drawings in more detail.
`As shown in FIG. 1 the reference 1 generally designates
`a system and method of rendering and interacting with a
`three-dimensional (3-D) computer generated model that
`comprises the virtual reality/telepresence system 1 presented
`to a participant 24. The system 1 generally includes a
`panoramic input source means 2, panoramic signal process
`ing means 2, and panoramic audio-visual presentation
`assembly means 2 connected generally by suitable electrical
`interface means 2. Electrical interface means 2, including
`the appropriate conductor, input/output port or jack inter
`connections, and associated signal, is indicated by lines and
`arrows in the drawings. Optional or alternative interface
`means and devices are indicated by dashed lines in the
`drawings. Input means 2, generally consists of a panoramic
`3-D camera system 6, a panoramic 3-D digitizing system 7,
`and a panoramic 3-D audio recording system 8. Input means
`45
`6, 7, and 8 include a plurality of respective sensors that are
`positioned to record geographic and geometric subjects. A
`subject 13 may comprise three-dimensional beings or things
`in the real world. The world model 14 comprises a model
`14a that includes shape and imagery, and an audio model
`14b that includes accoustical recordings. The audio model
`corresponds to the shape and imagery model. Preferably, all
`sides of the subject are recorded simultaneously by the input
`means 6, 7, and 8.
`Signal processing means 3 preferably includes a first
`computer processing means 15 for sensor fusion of the
`resulting imagery signal 5a and shape data signal 5b. The
`first processing means operates on the signals 5a and 5b to
`combine shape and surface data of corresponding segments
`of the 3-D subject. The resulting 3-D model segments 26a
`are portions of the computer generated world model 14a.
`Signal processing means 3 preferably also includes a second
`computer processing means 16 for fusion of imaging and
`shape data segments 26a derived by first apparatus 15 to
`form a continuous panoramic world model 14. Signal pro
`cessing means 3 also includes a third computer processing
`means 17 for manipulating the computer generated model
`
`50
`
`55
`
`60
`
`65
`
`RPX Exhibit 1003
`
`

`

`9
`Still referring to FIG. 1, in operation a panoramic sensor
`array comprising a plurality of shape, visual, and aural
`sensors are positioned to record a three-dimensional subject
`in a substantially continuous panoramic fashion. Each sen
`sor 27 outputs a corresponding signal specific to that sensors
`field of coverage. Signals representing visual and shape data
`are transmitted from input sources 6 and 7 to the signal
`processing means 3. A first computer processing means