United States Patent c191
`Redmond
`
`[54] METHODS AND APPARATUS FOR
`GENERATING AND PROCESSING
`SYNTHETIC AND ABSOLUTE REAL TIME
`ENVIRONMENTS
`
`[75]
`
`Inventor: Scott Redmond, Fresno, Calif.
`
`[73] Assignee: Redmond Productions, Inc., San
`Francisco, Calif.
`
`[21] Appl. No.: 483,547
`
`[22] Filed:
`Feb. 22, 1990
`Int. a.5 •••••••••••••••••••••••••••••••••••••••••••••• G06F 15/21
`[51]
`[52] U.S. a ...................................... 364/578; 364/400
`[58] Field of Search ............... 364/401, 419, 578, 400;
`434/38, 43-45, 55, 61, 69
`
`[56]
`
`References Cited
`U.S. PATENT DOCUMENTS
`4,001,949 1/1977 Francis ................................ 434/226
`4,281,994 8/1981 Dell et al. ............................. 434/49
`4,542,291 9/1985 Zimmerman ................... 250/551 X
`4,709,917 12/1987 Yang ..................................... 482/63
`4,751,642 6/1988 Silva et al. ...................... 364/410 X
`4,843,568 6/1989 Krueger et al. ................ 395/135 X
`4,948,371 8/1990 Hall ................................. 364/410 X
`5,184,319 2/1993 Kramer ............................... 364/806
`
`FOREIGN PATENT DOCUMENTS
`1267976 4/1990 Canada ........................... 364/413.01
`
`OTHER PUBLICATIONS
`Design News, May 7, 1979, Stengel, "Play Pebble Beach
`Indoors", p. 36-38.
`New Riverside University Dictionary, The Riverside Pub.
`Co., 1984.
`Swive13D product description, Paracomp. Inc., publi(cid:173)
`cation date unavailable.
`New York Times, Apr. 10, 1989, Pollack, "What is Real(cid:173)
`ity? Wear a Computer and See".
`
`I 111111111111111111111111111111111111111111111111111 lllli ll1111111111111111
`US005255211A
`5,255,211
`[111 Patent Number:
`[45] Date of Patent:
`Oct. 19, 1993
`
`PC Computing, Ditlea, "Another World: Inside Artific(cid:173)
`ial Reality", pp. 91-100, publication date unavailable.
`San Jose Mercury News, Jan. 30, 1990, Scheinin, "The
`Artificia_l Realist".
`Dataglove Model 2 product description/brochure and
`pricelist, VPL Research Inc., publication date unavail(cid:173)
`able.
`Body Electric product description/brochure, VPL Re(cid:173)
`search Inc., publication date unavailable.
`RB2 Virtual Reality System product description/broc(cid:173)
`hure and pricelist, VPL Research Inc., publication date
`unavailable.
`Earphone System product description, VPL Research
`Inc., publication date unavailable.
`Scientific American, Oct. 1987, Foley, "Interface for
`Advanced Computing", pp. 127-135.
`Success, Mar. 1990, Poe, "Manipulating Reality", p. 80.
`Technical Report Power Series, publication date unavail(cid:173)
`able.
`C. Blanchard et al. "Reality Built For Two: A Virtual
`Reality Tool", publication source and date unavailable.
`Primary Examiner-Roy N. Envall, Jr.
`Assistant Examiner-David Huntley
`Attorney, Agent, or Firm-Limbach & Limbach
`ABSTRACT
`[57]
`A system for generating and processing synthetic and
`absolute real time remote environments for interaction
`with a user and her biological senses is comprised of
`seven modules. These modules store, retrieve and pro(cid:173)
`cess data to generate an output which interfaces with
`the system user's biological senses. These modules also
`track user data to accurately place the user in the gener(cid:173)
`ated and processed model. Various embodiments of the
`system -user sensory interface are provided including
`visual and aural input devices and a three dimensional
`chamber having interactive tactile output via matrix(cid:173)
`addressed, electromechanically operated rods driving a
`flexible skin.
`
`5 Qaims, 16 Drawing Sheets
`
`14
`
`DIRECTVISION IIC IMAGE PUMP & \IJEWER
`SOUND PROOF PANEL
`
`16
`
`MOUNT SPAN
`
`IEADY
`ROOM
`DOOR 10
`
`USER
`
`GENERAL LAYOUT, NOMINAL 10' X 10' VERITY CHAMBER
`
`Netflix, Inc. - Ex. 1023, Page 000001
`
`IPR2021-01319 (Netflix, Inc. v. CA, Inc.)
`
`

`

`U.S. Patent
`
`Oct. 19, 1993
`
`Sheet 1 of 16
`
`5,255,211
`
`14
`
`DIRECTVISION IIC IMAGE PUMP & VIEWER
`SOUND PROOF PANEL
`
`MOUNT SPAN
`
`16
`
`READY
`ROOM
`DOOR 10
`
`USER
`
`GENERAL LAYOUT, NOMINAL 101 X 10' VERITY CHAMBER
`FIG. 1
`
`CYBERCYCLE JET STREAMS
`
`VERITY CHAIR
`
`REAR
`PROJECTOR
`
`22
`
`32
`FOLD-OUT CYBERCYCLE
`20
`
`34
`
`30
`
`B'X 10'
`REAR
`PROJ.
`SCREEN
`
`REFLEX
`TREADMILL
`28.
`
`DOOR
`
`24
`
`26
`PROGRAMMERS
`CONNECTION BLOCK
`
`FIG. 2
`
`Netflix, Inc. - Ex. 1023, Page 000002
`
`

`

`POWER SERIES SYSTEM DIAGRAM
`(EXCEPT 4D/210 SYSTEMS)
`SYNC BUS
`
`36
`
`___.__, 60
`
`CPU
`
`52
`
`CPU J----f FPU
`
`I I
`
`I I
`
`I CPU t--i FPU
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`G>
`
`~ ::c
`C') en
`en
`C m
`u3
`n1
`
`~
`
`38
`PROCESSOR BUS
`
`38
`
`PROCESSOR BUS
`
`1STLEVEL
`0-CACHE
`
`40
`48
`
`62
`
`1STLEVEL
`0-CACHE
`
`64
`
`42
`
`50
`
`I-CACHE
`
`44,
`
`WRITE
`BUFFER
`.54
`
`2NDLEVELI I
`
`D-CACHE
`
`46.
`
`I 11 B=~R I
`
`WRITE
`BUFFER
`... 56
`-
`12ND LEVEL -
`0-CACHE 1 1
`
`~
`~
`
`~
`~
`m
`~
`0
`~
`
`~ - - - - - - - - -
`
`SYNC BUS
`INTERFACE
`
`INTE~fACE " 66
`· SCVSI
`INTERFACE+ 7o
`ETHERNET ..
`INTERFACE --68
`MPUNKBUS
`INTERFACE
`
`I
`
`I I
`
`1 ECC
`CNTRL
`p
`I
`
`I
`
`FIG. 3
`
`(PRIOR ART)
`
`MPUNKBUS
`
`'-sa
`
`0 •
`r.lJ. •
`""d a
`~ '"1--
`
`0 n
`~ ....
`.. ....
`i (N
`
`\Cl
`
`00. =-fl)
`
`fl) -N = ...,
`
`.... e"I
`
`I
`
`UI
`. ~·
`UI
`UI
`
`..
`.. ~ I
`....
`....
`
`Netflix, Inc. - Ex. 1023, Page 000003
`
`

`

`U.S. Patent
`
`Oct. 19, 1993
`
`Sheet 3 of 16
`
`5,255,211
`
`MULTI-THREADED KERNAL ALLOWS OVERLAPPING
`COMPUTATION AND 1/0 PROCESSING
`
`72
`KERNAL
`
`USER1
`CPU0
`
`USER 1
`CPU0
`
`KERNAL
`
`IDLE
`
`USER1
`CPUO/CPU1/CPU2/CPU3
`
`CONTEXT
`IDLE SWITCH
`
`(PRIOR ART)
`FIG. 4
`
`Netflix, Inc. - Ex. 1023, Page 000004
`
`

`

`GEOMETRY 1§.
`SUBSYSTEMS
`
`IRIS GTX GRAPHICS ARCHITECTURE
`MPUNKBUS
`
`84
`
`GT GEOMETRY
`ACCELERATOR
`
`86
`88
`90
`92
`
`94
`
`FIFO
`// BUFFER
`
`~ GEOMETRY
`) ~+--····· 6~~iLLER
`
`GEOMETRY
`ENGINES
`
`SCAN
`CONVERSION
`SUBSYSTEMS
`
`18.
`
`POLYGON ID
`
`PROCESSORS
`
`96
`
`SKEW
`CALCULATORS
`
`EDGE
`PROCESSORS
`
`PIXEL BUS
`
`AH········-··--· EDGE
`PROCESSOR
`
`SPAN
`_,,./ PROCESSOR
`
`:,:__100
`
`FIG. SA
`
`(PRIOR ART)
`
`TOFIG.5B
`
`~ •
`00
`•
`
`~ a ti> = f"'t',
`
`0 µ.
`,_
`~ ,_
`!
`
`en
`
`,llo,
`
`[
`a ,_
`
`0\
`
`...
`UI
`~
`UI
`...
`UI
`~
`1-l
`1-l
`
`Netflix, Inc. - Ex. 1023, Page 000005
`
`

`

`FROM FIG. SA
`
`H
`
`RASTER
`SUBSYSTEM
`80
`
`H
`
`H
`
`H IE 1110
`IE 1102
`IE 1118
`IE 1126
`•
`•
`'
`'
`IVRAM I-
`IVRAM 1--
`IVRAM I-
`IVRAM 1--
`HIE I 104 HIE I 112 H
`IE I 120 HIE I 128
`•
`•
`'
`'
`IVRAM I-
`IVRAM 1--
`IVRAM I-
`IVRAM I-
`HIE I 106 HIE I 114 HIE I 122 HIE I 130
`•
`•
`'
`'
`IVRAM I--
`IVRAM I-
`IVRAM I-
`IVRAM I-
`IE I 108 y
`IE I 116 y
`IE I 124 y IE I 132
`y
`•
`•
`•
`'
`IVRAM I-
`IVRAM 1--
`IVRAM I-
`IVRAMI-
`
`134
`HIE --------- IMAGE
`'
`ENGINE
`IVRAM I-
`HIE
`•
`IVRAM I-
`HIE.
`•
`IVRAM 1--
`L-j
`IE
`•
`IVRAM I-
`
`136
`
`138
`
`140
`
`, ,
`,
`
`IMAGE
`DISPLAY
`SUBSYSTEM
`, ,
`82
`, ,
`, ,
`,
`MULTIMODE
`GRAPHICS
`PROCESSOR
`
`1142
`•
`12-B
`CM
`
`MGP
`•
`R
`G
`B
`A
`
`• !
`
`:
`
`'
`
`I MGP
`•
`R
`G
`B
`A
`
`'
`
`'
`
`I 144 I MGP
`•
`•
`R
`12-B
`G
`CM
`B
`A
`'
`
`I 145
`•
`12-B
`CM
`
`'
`
`'
`1148 I MGP
`•
`•
`R
`12-B
`G
`CM
`B
`A
`,L
`
`I MGP
`•
`R
`G
`B
`A
`,L
`
`'
`
`I 150
`
`•
`12-B
`CM
`
`,L
`
`:
`12-BIT
`COLOR MAP
`
`RED
`
`GREEN
`
`BLUE
`
`FIG. 58
`
`(PRIOR ART)
`
`AL >HA-ALPHA OUTPUT
`1280
`__/
`X
`1024
`
`--.......
`
`I
`\
`
`~ ·•
`00
`•
`1-d
`
`i = """
`
`0
`
`fl. ...
`:;c ...
`!
`
`r,J
`
`UI
`
`1:1"'' l
`e ...., ...
`°'
`
`...
`...
`
`(II
`
`~
`(II
`(II
`
`~ ... ...
`
`Netflix, Inc. - Ex. 1023, Page 000006
`
`

`

`156
`
`SENSOR
`
`162
`
`MODEL
`
`168
`
`XFORM
`
`174
`
`ZBUFF
`
`-152
`
`154
`
`SENSOR
`
`SENSOR
`
`• • •
`
`158
`
`160
`
`MODEL
`
`MODEL
`
`164
`
`166
`
`XFORM
`
`XFORM
`
`170
`
`172
`
`ZBUFF
`
`ZBUFF
`
`I
`
`176
`
`DISPLAY
`
`FIG. 6
`
`~ • 00
`
`•
`~
`~
`8
`"""
`
`i,,,l
`
`0
`f4.
`:;c
`i,,,l !
`
`ga
`~ a
`°' a.
`i,,,l °'
`.. ~
`.. ~
`'"""'
`'"""'
`
`UI
`UI
`
`UI
`
`Netflix, Inc. - Ex. 1023, Page 000007
`
`

`

`U.S. Patent
`
`Oct. 19, 1993
`
`Sheet 7 of 16
`
`5,255,211
`
`178
`
`180
`
`CAMERA 1
`
`CAMERA2
`
`182
`
`184
`
`CONTRAST
`
`CONTRAST
`
`188
`
`THIN
`
`196
`-
`SIMULATION
`
`186
`
`THIN
`
`190
`
`XYZVERT
`
`192
`
`XFORM
`
`194
`
`ZBUFF
`
`198
`
`DISPLAY
`
`CAMERA / MODEL
`FIG. 7
`
`Netflix, Inc. - Ex. 1023, Page 000008
`
`

`

`U.S. Patent
`
`Oct. 19, 1993
`
`Sheet 8 of 16
`
`5,255,211
`
`0
`0
`z
`
`z
`w w
`a:
`~
`
`co
`0
`N
`
`0
`z
`<
`
`0
`N
`
`~
`
`~
`~
`
`.
`
`'q"
`~ w
`N
`0.
`0.
`~
`5
`
`~ I-
`0
`0
`:c
`Cl) L:J
`
`'q"
`N
`
`0 b
`0
`:c
`Cl)
`>-
`I
`
`N
`N
`
`0 b
`0
`:c
`Cl)
`><
`
`_J
`
`w
`~
`>
`@)
`
`N
`N
`
`~ ;
`
`Cl.
`en
`0
`
`:c
`I-z
`~ CD
`--
`a:
`bl -
`•
`w C,
`IL
`~
`0
`(f)
`
`Netflix, Inc. - Ex. 1023, Page 000009
`
`

`

`220 6-AXIS SENSOR
`'
`
`6 219 BRAINWAVE CONTACT SENSOR
`
`224 WIDEANGLE LENS
`
`226 FOCUS DIOPTER
`
`1
`
`-1--
`
`222 LCD PLATE
`
`I
`
`c
`
`CABLES OUT & IN
`
`l
`t216 MICRO-STROBE
`
`PROGRAMMERS 3D VIEWMASK
`FIG. 9
`
`~ •
`00.
`•
`
`"'O = s-= .....
`
`0 µ.
`
`to-
`~
`to-
`~
`CH
`
`\C
`
`rJl [
`0 ...,
`to-°'
`
`UI
`
`UI
`UI
`
`.. ~
`.. ~
`
`t,o(cid:127) l
`t,o(cid:127) l
`
`Netflix, Inc. - Ex. 1023, Page 000010
`
`

`

`U.S. Patent
`
`Oct. 19, 1993
`
`Sheet 10 of 16
`
`5,255,211
`
`t-z ::>
`t-en
`c3
`~
`0
`a:
`m
`::iii::
`0
`~
`~
`z
`c:C
`LL
`f2'
`
`t-
`::>
`0
`~
`~
`
`a..
`::>
`0
`C
`z
`::>
`g
`...J
`~
`(!)
`c
`w
`z
`0
`:c
`~
`
`a: a: en
`
`C\I
`('I)
`C\I
`
`0 ,-
`•
`
`C, -LI.
`
`CX)
`
`~
`
`0
`C\I
`
`w
`~ ('I)
`LL
`0
`!z
`0
`a:
`LL
`
`(!)
`z
`en
`::>
`0
`:c
`t-z
`::>
`w
`i
`
`Netflix, Inc. - Ex. 1023, Page 000011
`
`

`

`U.S. Patent
`
`Oct. 19, 1993
`
`Sheet 11 of 16
`
`5,255,211
`
`250
`
`ERECTING'v--6----< ---"'- SPIRITPHONE, ONE SIDE
`PRISMS& - - (cid:173)
`LENS
`
`FIBER-OPTIC IMAGE GUIDES,
`ONE EACH EYE
`
`240
`
`242
`
`DUAL HIGH RESOLUTION
`STEREO-OFFSET COLOR MONITORS
`
`234
`ANTENNA
`
`236
`BACK-UP
`UMBILICAL
`CONNECTION
`
`USER LCD OR HI-RES. MONITOR FANNY PACK SYS.
`FIG. 11
`
`Netflix, Inc. - Ex. 1023, Page 000012
`
`

`

`234 SWEEP SEQUENCED
`RED l>SER DIODE
`
`236 GREEN L.D.
`
`240 MAGNETIC FIELD
`FOCUS RING
`
`.____1•
`
`238 BLUEL.D .
`I
`DIODE INER EYE-SCAN SYSTEM
`-BASIC MODEL
`FIG. 12A
`
`232
`
`I -.I
`
`240
`
`234
`
`236
`
`238
`
`244
`OPTICAL
`LENS
`
`242 PHOTON RASTER
`CONTROL FIELD
`FIG. 12B
`
`~248
`
`246
`FEEDER CABLE
`
`FIG. 12C
`
`~ •
`00 •
`"'O = ......
`('D = ......
`
`0
`
`!l ....
`~ ....
`!
`
`r .... N
`
`0 ...,
`....
`°'
`
`...
`UI
`~
`UI
`...
`UI
`
`~ ... ...
`
`Netflix, Inc. - Ex. 1023, Page 000013
`
`

`

`U.S. Patent
`
`Oct. 19, 1993
`
`Sheet 13 of 16
`
`5,255,211
`
`1280
`RES.+
`MONITOR
`
`POWER
`
`IMAGE
`SIGNAL
`
`250
`
`252 LIGHT SHIElD
`
`256
`
`DUAL C-MOUNT CAMERA LENSES, FIXED
`
`,__ HIGH-RES.
`SUPER-CONDUCTING,
`FIBER-OPTIC IMAGE GUIDE
`
`262
`
`ERECTING PRISMS AND EYEMASK MOUNT W. /
`FULL-FIELD-OF-VISION EYEPEICES.
`SYCHRONIZED SHUTTERS
`
`REMOTE SINGLE HI-RES MONITOR VISION DEVICE
`FIG. 13
`
`PROJECTING LENS
`264
`
`HOUSING_
`268
`
`266
`SEPERATE MICRO-SCREEN
`
`SIDE OF LEFT IMAGE GUIDE TERMINATION
`70
`ELASTIC BAND AROUND
`HEAD, SEINE CORD WIDTH
`MICRO-SCREEN OPTION
`FIG. 14
`
`Netflix, Inc. - Ex. 1023, Page 000014
`
`

`

`U.S. Patent
`
`Oct. 19, 1993
`
`Sheet 14 of 16
`
`5,255,211
`
`284
`
`USERS VIEW END
`FIG. 15B
`
`LEFT EYE
`SET-UP
`SAME FOR RIGHT EYE
`
`282 FLAT FIBER OPTIC RIBBON IMAGE GUIDE
`
`-
`280 IMAGE MULTI-PLEXER &
`FIELD GENERATOR PHOTON
`278 ENHANCER
`
`FEEDER CABLES
`
`272
`RED
`
`276
`BLUE
`
`274
`GREEN
`MICROCRT'S
`FIG. 15A
`
`DIRECTVISION 5,
`GENERAL LAY OUT
`TO USERS EYES
`
`Netflix, Inc. - Ex. 1023, Page 000015
`
`

`

`U.S. Patent
`
`Oct. 19, 1993
`
`Sheet 15 of 16
`
`5,255,211
`
`• • • • • •
`• • • • • • • • •
`• • • • • • • • •
`• • • • • •
`• • • • • • • • •
`• • • • • •
`• • • • • •
`• • • • • • • • •
`• . • • • • • • •
`• • • • • • • • •
`• • • • • •
`• • • • • •
`• • • • • •
`• • • • • • • • •
`• • • • • • • • •
`• • • • • •
`• • • • • •
`• • • • • • • • •
`• • • • • • • • •
`• • • • • •
`• • • • • •
`• • • • • • • • •
`• • • . • • • • •
`• • • • • • • • •
`• • • • • •
`• • • • • • • • .
`. • • • • •
`• • • • • •
`• . • • • • • • •
`• • • • • •
`m,
`
`C\I
`
`'
`
`"d"
`0)
`C\I
`
`CD "' :E
`
`Cl0'
`'
`
`C\I
`0)
`C\I
`
`-
`
`J:
`0
`
`~
`
`.
`
`co co
`
`C\I
`
`a: w
`0
`0
`0 w
`0
`
`~
`-
`
`-
`
`~ w
`a:
`0
`0 <
`
`.....
`co
`~
`
`-
`
`<
`~ 0
`
`-
`i C\I
`
`16!
`'
`
`'-
`
`m
`~ -
`
`Cl0' I'\
`
`~ co
`a:
`.....
`0
`0 <
`w -X LL
`•
`..J C,
`
`Cl.
`
`Netflix, Inc. - Ex. 1023, Page 000016
`
`

`

`U.S. Patent
`
`Oct. 19, 1993
`
`Sheet 16 of 16
`
`5,255,211
`
`a:
`~
`z
`0
`::E
`a: g
`
`0
`0
`
`~
`~
`0
`w
`~
`
`~
`~
`.,..
`w
`(!J
`~
`
`a..
`~
`~
`
`lO
`~
`z
`ca
`
`a..
`~
`~
`
`co
`
`~ z ca
`
`0 u,
`
`S-100 BUS
`
`~
`0
`::E
`w
`::E
`
`0
`~
`z
`ta
`
`::::>
`a..
`0
`
`~i 0W _:c
`
`5~
`a.,W
`~a..
`
`"' 0 a:
`~ m
`0.
`~
`~t-
`::c: ,-
`I-
`~ .
`~S2 w IL
`~ u,
`0
`"' z
`
`I-
`
`c(
`
`Netflix, Inc. - Ex. 1023, Page 000017
`
`

`

`METHODS AND APPARATUS FOR GENERATING
`AND PROCESSING SYNTHETIC AND ABSOLUTE
`REAL TIME ENVIRONMENTS
`
`TECHNICAL FIELD
`The present invention deals generally with computer(cid:173)
`generated and processed environments, and more spe(cid:173)
`cifically to systems for generating and processing syn(cid:173)
`thetic and absolute environments for three dimensional
`visual images, multidimensional aural images and tactile
`input to the system user's biological senses.
`
`BACKGROUND OF THE INVENTION
`Recent advances in hardware and software have 15
`permitted an increased awareness and development
`effort in the area of interactive "synthesized" environ(cid:173)
`ments. The goal of these development efforts is to pro(cid:173)
`vide a synthesized environment which is so realistic as
`to be virtually indistinguishable from the "absolute" or 20
`actual reality. The applications of these interactive syn(cid:173)
`thesized realities are diverse: flight simulation, precon(cid:173)
`struction architectural review and walkthrough, video(cid:173)
`game-like entertainment, molecular modeling and engi(cid:173)
`neering, microsurgery, communications and educa- 25
`tional applications.
`The NASA Ames Research Center has been working
`in synthetic environment development for many years.
`NASA's Virtual Interface Environment Workstation
`(VIEW) uses a head-mounted black and white LCD 30
`display for communicating generated three dimensional
`images to the system user.
`Another system developer is VPL Research. VPL
`Research has designed a DataGiove which employs
`fiberoptics to monitor flexion and extension of the user's 35
`fingers, and a Polhemus magnetic tracking device to
`position the user's hand in a three dimensional matrix.
`VPL Research provides a DataSuit which expands the
`technology used in the DataGiove to monitor move(cid:173)
`ment of the entire human body. VPL Research's Eye- 40
`Phone, or visual input device, consists of two color
`LCD monitors viewed through an optics system. In
`VPL Research's system, each monitor is driven by a
`separate computing path. A high resolution dot pattern
`is then superimposed over the transmitted image to 45
`improve perceived resolution. VPL Research provides
`a DataSuit which expands the technology used in the
`DataGlove to monitor movement of the entire human
`body.
`AutoDesk has recently announced its development so
`efforts in computer-generated synthetic reality systems.
`AutoDesk is primarily interested in architectural walk(cid:173)
`through applications of computer-generated synthetic
`environments, as a natural companion to AutoDesk's
`well established position in the CAD industry.
`TiNi Alloy Co. has announced a tactile array of tiny,
`electronically activated pins to simulate the sensation of
`touching shapes and forms.
`These existing systems fail to provide a totally inte(cid:173)
`grated sensory output, to capture photo realistic images, 60
`sound and tactile sensory stimuli in a single interactive
`time and space. The processing technology in these
`earlier systems permits only gross "computational"
`image representations, images which are not realistic
`because of computation-limited low polygon density. 65
`The system architectures are inherently inefficient, pre(cid:173)
`venting the provided system storage and processing
`capacity from operating at the computational speed
`
`1
`
`5,255,211
`
`2
`required to provide super-realistic visual, · sound and
`tactile imagery in an interactive mode.
`The present invention provides a fully integrated
`system with massive parallel processing capabilities for
`5 generating and processing super realistic visual images,
`multidimensional sound images and tactile output. The
`present system permits the incorporation of pre-existing
`(actual) realities, defined herein as "absolute" reality, in
`combination with the computer-generated "synthetic"
`10 environments. The present invention is able to accom(cid:173)
`plish these features by providing a system architecture
`which optimizes computer memory storage and re(cid:173)
`trieval and which enables the required computations
`with a minimum of compute server capacity.
`It is an object of this invention to provide a unique
`combination of system
`architecture,
`computer(cid:173)
`generated environmental output signals and computer
`output-human sensory input interfaces which provides
`the system user with a super realistic experience when
`interacting with the system.
`It is an object of this invention to generate synthetic
`and absolute environmental model output of super real(cid:173)
`istic visual images, multidimensional sound images and
`interactive tactile output.
`It is another object of this invention to provide data
`storage and retrieval architecture which employs image
`and data buffers to permit rapid access to stored data
`and unlimited capacity for increasing the speed of data
`retrieval and the data base volume.
`It is an object of this invention to provide system
`architecture which permits efficient use of data memory
`and retrieval means by rapid reconfiguration of stored
`and currently viewed images.
`It is an object of this invention to provide a system
`which can effectively combine "absolute" realities with
`"synthetic" realities to provide remote system operation
`capabilities, including remote robotics.
`It is an object of this invention to monitor user posi(cid:173)
`tion, movement and health telemetry data without in(cid:173)
`trusive sensors and cumbersome umbilical cords.
`It is an object of this invention to provide generated
`and processed environment modeling module output to
`the user's biological sensory interfaces through massive
`micro compute server parallel data processing tech(cid:173)
`niques.
`It is an object of this invention to employ an environ(cid:173)
`ment modeling modu~ in which three dimensional vi(cid:173)
`sual output is optically differentiated to obtain stereo(cid:173)
`graphic images using a single output mode.
`It is an object of this invention to provide an environ(cid:173)
`ment modeling module which incorporates as pre(cid:173)
`selected parameters the natural and known laws of
`physics in the generation of synthetic and absolute envi-
`55 ronments.
`These and further objects of the present invention
`will be apparent to those of ordinary skill in the art with
`reference to the following description and figures.
`
`· SUMMARY OF THE INVENTION
`A system for generating and processing synthetic and
`absolute real time environments comprises six separate,
`interconnected mod.ules. The data storage and retrieval
`module is designed for storing high resolution digital
`image, sound and user position coordinate data. The
`retrieval side of this module is designed to instanta(cid:173)
`neously retrieve the stored data. The data storage and
`retrieval module includes image files of natural objects;
`
`Netflix, Inc. - Ex. 1023, Page 000018
`
`

`

`5,255,211
`
`4
`3
`rear microscreens mounted one-quarter of an inch in
`digital sound files of natural sounds; image compression
`front of surface of the user's eyes from fiberoptically
`software which permits efficient data storage and re-
`routed high definition color monitors. In a fifth embodi-
`trieval; primitive image basic model files such as poly-
`ment, frame sequenced dense-pack fiberoptic image
`gons, fractals, ray tracing and subprimitive existing
`elements; image pre-design data processing array for 5 decompression onto rear micro-screens mounted one-
`the storage and retrieval of already generated synthetic
`quarter of an inch in front of surface of the eyes is ob-
`image data; and, interactive throughput image and
`tained by belt mounted HDTVs.
`sound processing which generates and processes images
`The user stimuli module, in a preferred embodiment,
`corresponding to the user's body parts in the user's field
`includes components for enhanced tactile input. A three
`of viewing the generated synthetic or absolute environ- 10 dimensional chamber is adapted to simulate gross envi-
`ronmental conditions such as smells, sonic rumbling,
`ment.
`The second system module is an umbilical-less sensor
`variable temperature and variable wind velocity. The
`chamber is equipped with a rectangular matrix of rods,
`which senses and transmits user position and movement
`data obtained by co-matrixed averaging of at least two
`which can be arranged with their longitudinal axis per-
`separate sensor modes. This sensor module detects and 15 pendicular to the chamber wall, or in a stand alone unit
`transmits user health telemetry data to the environmen-
`inside the chamber. The rods are electromechanically
`tal modeling module, data storage and retrieval module,
`manipulated to generate shapes and textures for interac-
`and the primary control module.
`tion with the system user. In another embodiment, the
`The third module is an environmental modeling mod-
`user is provided with boot shaving bladders which can
`ule for the real time generation and processing of syn- 20 be inflated and deflated to change the user's perception
`of the chamber surface. In the preferred embodiment,
`thetic and absolute environment output which uses
`the chamber is equipped with a variable resistance
`pre-selected commands and parameters to render syn-
`thetic images, sounds, forces and movements with real-
`treadmill.
`istic accuracy, including user-viewed interactive user
`BRIEF DESCRIPTION OF THE FIGURES
`body parts, through massive micro compute server 25
`parallel data processing techniques.
`FIG. 1 is an elevation view of one embodiment of the
`user stimuli module of the present invention.
`The fourth system module is the user stimuli module,
`FIG. 2 is a plan view of an embodiment of the user
`referring to the output/input interface between the
`stimuli module of the present invention containing addi(cid:173)
`environmental modeling means output and the user's
`tional stimuli apparatus.
`biological sensory input. This module includes appara- 30
`FIG. 3 is a block diagram of the major components of
`tus for three dimensional visual input to the user's eyes,
`apparatus for multi-dimensional sound input to the
`the high performance workstation used in the present
`invention, tabled PRIOR ART.
`user's ears and apparatus for tactile input to the user's
`hands, feet and other body parts.
`FIG. 4 is a block diagram of the input/output subsys(cid:173)
`The fifth module is the primary control module for 35
`tem of the kemal of the high performance workstation,
`overall system operation, including the interface be(cid:173)
`labeled PRIOR ART.
`FIGS. SA and SB are block diagrams of the architec(cid:173)
`tween the data storage and retrieval module, sensor
`module, environmental modeling module and user stim(cid:173)
`ture of the graphics subsystem of the high performance
`workstation, labeled PRIOR ART.
`uli module. The primary control means is configured to
`FIG. 6 is a block diagram of an interactive through(cid:173)
`control remote robotics.
`The final system module, in-system communications
`put image and sound processor according to the present
`means, permits user vocal communication to and from
`invention.
`FIG. 7 is a block diagram of an interactive through(cid:173)
`the primary control module.
`put image and sound processor utilizing video camera
`Prior to the user interactive session, the system user
`selects a synthetic or absolute reality environment 45
`data collection techniques.
`FIG. 8 is a block diagram of a raster-based sensor
`which is then communicated to the primary control
`module. This user selection presets some of the compu(cid:173)
`method could be used to determine user position and
`movement data whiclt can be integrated with the envi(cid:173)
`tational parameters required to generate the synthetic
`ronmental modeling module of the present invention.
`and absolute realities, and creates a data library subset
`FIG. 9 is a side elevation view of the first embodi(cid:173)
`which can be more easily accessed and processed. 50
`ment of the visual input device.
`Thereafter, the synthetic and absolute environment is
`FIG. 10 is a schematic view of the first embodiment
`computed and generated, allowing the system user to
`of the visual input device.
`interact with the generated environment through the
`FIG. 11 is a schematic view of the second embodi-
`in-system communications means, sensor module and
`user stimuli module.
`55 ment of the visual input device.
`There are several embodiments of the three dimen(cid:173)
`FIG. 12a is a plan elevation; FIG. 12b is a side eleva(cid:173)
`sional visual input device which is part of the user stim-
`tion, and FIG. 12c is a schematic view of the third
`uli module. In the first embodiment, visual images are
`embodiment of the visual input device.
`FIG. 13 is a schematic view of the fourth embodi(cid:173)
`input to the user's eyes via biosynchronized peripheral
`microstrobes in combination with dual vision LCD 60
`ment of the visual input device.
`FIG. 14 is a detailed side elevation view of an op(cid:173)
`color eye monitors disposed on a headset. A second
`tional addition to the fourth embodiment of the visual
`embodiment of the visual input device employs fiberop-
`tic stacked array dual color images from two high defi(cid:173)
`input device.
`FIG. 1Sa is a schematic view of the fifth embodiment
`nition color monitors. In a third embodiment, direct
`of the visual input device, and FIG. 1Sb is a detailed
`retina scan and other internal eye image placement is 65
`achieved by low power laser diode imaging packs and
`view of the user end of the ribbon cable.
`magnetic light aiming field. A fourth embodiment of the
`FIG. 16 is a block diagram for a pixel addressing
`visual input device uses projection of the image onto
`scheme.
`
`40
`
`Netflix, Inc. - Ex. 1023, Page 000019
`
`

`

`5,255,211
`
`s
`FIG. 17 is a block diagram for a two port pixel ad(cid:173)
`dressing scheme.
`
`6
`The present invention comprises six basic modules:
`data storage and retrieval module, user sensor module,
`environmental modeling module, user stimuli module,
`DETAILED DESCRIPTION OF THE
`primary control module and in-system communications
`INVENTION
`5 module. There are many different system models which
`practice the present invention through these six mod-
`The present invention has applications in resource
`ules, beginning with the low end model which is essen-
`conservation, e.g., solicitation by non-profit organiza-
`tially configured like a personal computer workstation,
`tions and foundations, research groups, scientific com-
`through the upper end auditorium-sized, synthesized
`munity or university related institutions for funding and
`advance support from potential contributors. For exam- 10 environment outputs. All of these models contain the
`six modules, but each model bas different capacity de-
`pie, demonstrations involving ozone depletion, the cf-
`fects of rainforest depletion on water and air quality
`pending upon the output demands of the application.
`control, reforestation concerns, etc. could be generated
`The claims appended hereto are intended to generically
`cover the system having six modules, without regard to
`for fundraising purposes. Additionally, endangered spe-
`cies and wildlife programs could be developed. The 15 the particular model into which these modules are in-
`technology could be applied for evaluation studies of
`corporated.
`animal migration and habit patterns without the danger
`In describing these six functionally-defined modules,
`associated with actual field studies.
`it is sometimes the case that the functional modules are
`The system may be applied to study industrial sys-
`not actually discrete hardware units. For example, the
`terns for the design of recycling, waste disposal manage- 20 data storage and retrieval module, the environmental
`ment and hazardous waste handling apparatus.
`modeling module and the primary control module share
`common hardware.
`Synthesized environments may be applied in the pro-
`FIG. 1 shows an elevation view of one embodiment
`cedure of adoption, permitting potential parents and
`of the user stimuli module, and FIG. 2 shows a plan
`children to preview the situation.
`25 view of a three dimensional chamber in which user
`The system has unlimited direct medical applications,
`stimuli components have been deployed. The purpose
`e.g., in the training of surgeons, nurses and other staff, in
`of these figures is to show the breadth of anticipated
`rehabilitation programs for the disabled, handicapped
`user stimuli modes which are generated by the instant
`and substance addicted patients.
`invention.
`Unlimited educational applications in geography, 30 Referring to FIG. 1, the user 10 prepares for the
`interactive session with the system in a ready room 12.
`mathematics, physics, health education and many other
`subject areas are possible.
`Depending upon the model or embodiment, the user 10
`There are law enforcement applications of this sys-
`may be required to don application-specific apparel,
`tern in solving crimes by reenactment of events, secur-
`including suit, gloves and boots, which transmits user
`ing witnesses and protecting their identities. The system 35 position and movement data to the system, and moni-
`tors user health telemetry data.
`can be used in the training mode as well.
`Artistic applications of the three dimensional synthe-
`The user's sensory interaction with the system actu-
`sized models to two dimensional surfaces is contem-
`ally occurs in a chamber 14. This chamber is steel
`plated.
`framed _and modular, coming in cubic sections of six feet
`Each of these diverse applications of the system is 40 square. The size of the chamber .14 is dictated by the
`facilitated by the creation of a set of data specific to the
`anticipated applications. The walls, ceiling and floor of
`desired application at a time prior to the actual user
`the chamber 14 are easily adapted to fit a variety of
`interactive session. In other words, before the user actu-
`situations. As seen in FIG. 2, the chamber 14 is ade-
`quately flexible to permit the installation of various
`ally interacts with the synthesized and processed envi-
`ronment, there is a selection by the user of the environ- 45 sensory interface devices. The chamber 14 is designed
`to accept and operate coherent and incoherent biologi-
`ment in which the interaction is to take place. So, e.g.,
`if the user wanted to create and interact with a "nature"
`cally safe electromagnetic spectrum radiation devices to
`determine U£Cr position and movement within the three
`environment such as a beach, the user would select that
`environment, and the system technicians would then
`dimensional matrix defined by the chamber.
`generate a series of data files which would contain the 50 A high definition color television monitor 16 some-
`information necessary to generate and process the im-
`times referred to as an image pump is suspended from
`ages, sounds, smells and tactile stimulations which cor-
`the ceiling of the chamber 14. Headpiece 18 is fiberopti-
`respond to the chosen beach environment. This initial
`cally linked to the monitor 16 to permit the user to
`pre-interactive data development session cuts down on
`receive three dimensional visual and multidimensional
`the number of computations which must be made at the 55 aural input from the system. Specific embodiments of
`time of the actual user interactive session, by presegre-
`the visual input device 18 are more fully described
`gation of the data required for a particular application
`below in a section entitled USER STIMULI MOD-
`from the larger system database.
`ULE.
`The present invention is able to provide a fully inte-
`Additional stimuli apparatus are seen in FIG. 2.
`grated synthetic output by a programming technique 60 These apparatus are deployed in chamber 14. Rear
`projection screen 20, and accompanying projector 22,
`known as cascade programming. According to this
`technique, each of the sensory outputs (sound, smell,
`are used either in place of, or in combination with the
`touch) is tied to the visual image data associated with
`visual input device 18 shown in FIG. 1 to provide visual
`that sound, smell and touch so that a number of visual
`input. The chamber 14 is equipped with a device 24 for
`images can be tied together to create a scene, while 65 system output to the user's tactile senses. In this embodi-
`simultaneously all of the associated sensory perceptions
`ment of