Virtual Reality for Palmtop Computers
`
`GEORGE W. FITZMAURICE, SHUMIN ZHAI, and MARK H. CHIGNELL
`University of Toronto
`
`In our
`computers.
`palmtop
`can be applied toward
`theories
`reahty
`how virtual
`We are exploring
`acts as a palmtop
`computer
`prototype,
`called the Chameleon,
`a small 4-inch hand-held monitor
`device and a response button
`with the capabihties
`of a Silicon graphics workstation.
`A 6D input
`are attached
`to tbe small monitor
`to detect user gestures
`and input
`selections
`for
`issuing
`commands. An experiment was conducted to evaluate
`our design and to see how well depth could
`be perceived
`in the small
`screen compared
`to a large 21-inch
`screen, and the extent
`to which
`of
`movement
`the small
`display
`(in a palmtop
`virtual
`reality
`condition)
`could improve
`depth
`perception, Results show that with very little
`training,
`perception
`of depth in the palmtop
`virtual
`reality
`condition
`is about as good as corresponding
`depth
`perception
`in a large (but
`static)
`display. Variations
`to the initial
`design are also discussed, along with issues to be explored
`in
`future
`research, Our
`research
`suggests
`that
`palmtop
`virtual
`reality may
`support
`effective
`navigation
`and search and retrieval,
`in rich and portable
`information
`spaces.
`
`Information
`H.3.3 [Information
`and Subject Descriptors:
`Categories
`and Retrieval]:
`Storage
`Search and Retrieval—retneuaz
`Presentation]:
`Interfaces
`and
`models;
`[Information
`H.5.2
`1.3.1 [Computer
`User
`Interf’aces-~npuf
`deotees ad
`Graphics]:
`styles;
`strategies;
`interaction
`Methodology
`Hardware
`Architecture—three-dwnensional
`1.3.6 [Computer
`cksplays;
`Graphics]:
`1.3.7 [Computer
`Three-Dimensional
`and Techniques—in
`teraction
`tech iuques;
`Graphics]:
`Graphics and Realism—t,Lrtual
`realtty
`
`General
`
`Terms: Design, Human Factors
`
`Additional
`
`Kcy Words and Phrases: 3D control
`
`and display,
`
`palmtop
`
`computers,
`
`virtual
`
`reality
`
`1. MOTIVATION
`
`of
`
`industry
`in the computer
`are emerging
`(palmtops)
`devices
`hand-held
`Portable
`compact
`input
`and output
`problems
`due to their
`from a variety
`but
`suffer
`screen
`is their
`relatively
`small,
`claustrophobic
`primary
`problem
`size. One
`they
`can
`size. Palmtop
`computers
`are apparently
`limited
`in the functionality
`offer
`based
`on their
`small
`screen
`size; however,
`even
`larger monitors
`may
`have
`“real
`estate”
`problems
`as large
`amounts
`of
`information
`are
`being
`browsed
`or manipulated.
`Many
`users
`are finding
`that
`their
`19-inch
`or
`larger
`computer
`monitors
`do not provide
`enough
`viewing
`space for some tasks.
`
`of Toronto,
`Science, University
`of Computer
`Department
`addresses: G. Fitzmaurice,
`Authors’
`CSRI, 6 King’s College Road, Toronto, Ontario, Canada, M5S 1A4; email: gf@ dgp.toronto.edu;
`S.
`Zhal and M. H. Chignell, Department
`of Industrial
`Engineering,
`University
`of Toronto, 4 Taddle
`Creek Road, Toronto, Ontario, Canada, M5S 1A4; email:
`{zhai; chignell}@ ie.utoronto.ca
`Permission
`to copy without
`fee all or part of this material
`is granted provided that
`the copies are
`not made or distributed
`for direct commercial
`advantage,
`the ACM copyright
`notice and the title
`of the publication
`and its date appear, and notice is given that
`copying is by permission
`of the
`Association
`for Computing Machinery.
`To copy otherwise,
`or to republish,
`requires
`a fee and\or
`specific permission
`C 1993 ACM 1046–8188/93/0700–0197
`
`$01,50
`
`ACM
`
`Transactions
`
`on Informatmn
`
`Systems,
`
`Vol.
`
`11, No.
`
`3, July
`
`1993,
`
`Pages
`
`197-218
`
`Mullen Industries, LLC - Ex. 1016
`Page 1
`
`

`

`198
`
`.
`
`George W Fkzmaurlce et al
`
`feed-
`in displaying
`the
`of
`bandwidth
`band-
`increasing
`for
`a means
`of navigating
`(VR)
`[Bishop
`and Fuchs
`
`role
`has a preeminent
`generally
`screen
`computer
`The
`the
`communication
`improving
`to the
`user.
`Thus,
`back
`issue. One strategy
`computer
`display
`is an important
`width
`is to use 3D displays.
`Providing
`the user with
`through
`a 3D display
`creates
`a type of virtual
`reality
`1992; Ellis
`1991; Kruger
`1991].
`an image
`panel with
`2D display
`of a flat
`A minimal
`3D display
`consists
`techniques
`a sense of depth
`using
`with
`rendered
`in 3D. Objects
`are drawn
`given
`one or more
`light
`sources.
`such as occlusion,
`perspective,
`and shading,
`scene
`depending
`on a user’s
`More
`sophisticated
`3D displays
`adjust
`their
`a fishtank
`VR system [Arthur
`physical
`position
`and viewpoint.
`For example,
`et al. 1993, Ware
`and Slipp
`1991]
`tracks
`a user’s
`head position
`and dynami-
`cally
`renders
`the
`appropriate
`scene given
`the
`current
`user
`viewpoint
`on a
`large,
`stationary
`computer
`monitor.
`Alternatively,
`there
`are a set of head-
`mounted
`display
`(HMD)
`systems
`[Sutherland
`1968] which
`position
`a small
`monitor
`directly
`in front
`of one or both
`eyes, similar
`to a pair
`of goggles. As
`the user moves
`his or her head,
`the displays
`are updated
`to reflect
`the new
`viewpoint.
`The
`specific
`variation
`of
`this
`design
`developed
`by Sutherland
`allows
`users
`to see both
`the
`computer-generated
`data
`and
`their
`physical
`surroundings
`simultaneously.
`how palmtop
`we are exploring
`computing,
`Given
`the
`trends
`in portable
`a 3D display.
`Our
`palmtop-based
`solution
`computers
`can be equipped
`with
`has the following
`four
`design
`goals.
`First,
`users
`interact
`with
`one or more
`hand-held
`mobile
`displays
`which
`are constantly
`aware
`their
`spatial
`posi-
`of
`tion
`and orientation
`within
`a 3D workspace.
`Second,
`the displays
`are high-
`fidelity
`color monitors
`capable
`of presenting
`text,
`graphics,
`video,
`and audio
`data. Third,
`the palmtop
`units
`have the ability
`to generate
`3D models
`and 3D
`scenes.
`Finally,
`our
`systems
`are
`designed
`with
`portability
`in mind
`(the
`display
`is coupled
`with
`the processing
`unit).
`These
`design
`goals will
`enable
`users
`to bring
`their
`3D information
`spaces with
`them or
`to access
`and
`interact
`with
`3D spaces
`situated
`in the physical
`environment
`[Fitzmaurice
`1993].
`section
`The following
`Next,
`reality
`research.
`Chameleon
`prototype.
`presented
`along with
`design
`are then
`explored
`
`the virtual-
`some of
`and characterizes
`reviews
`briefly
`and
`describe
`the
`interaction
`model
`we define
`our
`A performance
`experiment
`with
`the Chameleon
`is then
`lessons
`learned.
`Variations
`to the
`current
`prototype
`in some detail.
`
`2. BACKGROUND AND RELATED RESEARCH
`acquired
`and have
`Human
`beings
`are constantly
`reacting
`to a 3D world
`Researchers
`remarkable
`capability
`to comprehend
`3D spatial
`relationships.
`3D human
`and
`engineers
`are
`learning
`to take
`advantage
`of
`the
`natural
`information-processing
`capabilities
`for man-machine
`system interfaces.
`been
`Although
`a generally
`accepted
`definition
`of virtual
`reality
`has not
`achieved,
`some key components
`in constructing
`VR systems
`have been identi-
`and presence
`are three
`fied. Zeltzer
`[ 1992]
`states
`that
`autonomy,
`interaction,
`independent
`components
`in VR systems. Wickens
`[1992]
`identifies
`the use of
`
`a
`
`ACM
`
`TransactIons
`
`on Inf’ormat,on
`
`Systems,
`
`T701. 11, No
`
`3, July
`
`1993
`
`Mullen Industries, LLC - Ex. 1016
`Page 2
`
`

`

`Virtual Reality for Palmtop Computers
`
`.
`
`199
`
`dis-
`ego-referenced
`interaction,
`closed-loop
`animation,
`dynamic
`3D displays,
`sensory
`experience
`as five characteristics
`of VR technol-
`plays,
`and enhanced
`the
`interaction
`and
`presence
`components
`are
`usually
`ogy.
`Interestingly,
`independent
`elements,
`but
`this
`is not
`the
`case in the
`considered
`as two
`palmtop
`VR system described
`below (where
`input
`control
`and output
`display
`are integrated
`as a single
`unit).
`in the VR commu-
`have been pursued
`Two seemingly
`disparate
`approaches
`within
`the
`2D com-
`3D environments
`systems—simulating
`nity:
`outside-in
`visualizer,
`and cone trees
`[Card
`et al.
`puter
`screen
`(e.g.,
`rooms,
`information
`199 1]—and
`systems—virtual-reality
`environments
`in which
`users
`inside-out
`are
`completely
`immersed
`into
`an artificial
`3D space
`by wearing
`a head-
`mounted
`display
`and, potentially,
`a DataGlove.TM
`to the virtual
`is external
`The outside-in
`approach
`assumes
`that
`the user
`sometimes
`This
`approach,
`world,
`looking
`inward
`via
`a viewing
`window.
`and Slipp
`et al. 1993; Ware
`referred
`to as “fish
`tank
`environments”
`[Arthur
`1991],
`is practical
`and unintrusive
`to the user. Some systems
`include
`head-
`tracking
`devices
`and/or
`a stereo
`display.
`Designers
`of
`these
`systems must
`pay particular
`attention
`to providing
`the proper
`amount
`of depth
`cues and
`intuitive
`user
`interface
`designs
`to prevent
`the
`user
`from becoming
`disori-
`ented. This
`approach
`offers
`a less engaging
`experience
`and sense of presence
`than
`the alternative
`approach.
`However,
`many
`researchers
`have successfully
`pursued
`the
`use
`this
`outside-in
`style
`of 3D displays
`to help
`human
`of
`operators
`to better
`perceive
`the natural
`task
`environment
`such as those
`in
`teleoperation
`(e.g.,
`[Zhai
`and Milgram
`1991]). These kinds
`of displays
`can also
`facilitate
`a user’s
`comprehension
`of abstract
`concepts
`such
`as those
`in ab-
`stract
`hierarchical
`structures
`(e.g.,
`[Chignell
`et al.
`1992; Robertson
`et al.
`1991; Xiao
`and Milgram
`1992]).
`of
`point
`focal
`be the
`user
`the
`that
`The inside-out
`approach
`requires
`the environment,
`is then
`in the midst
`of
`virtual
`world
`(ego-centric).
`The user
`requires
`a great
`deal of extra
`hardware
`and surrounded
`by it. This
`approach
`but may be rewarded
`by a more
`engaging
`3D experience.
`Designers
`of
`these
`systems must
`constantly
`grapple
`with
`the tradeoff
`between
`system response
`time
`and
`the
`quality
`the
`rendered
`virtual
`environment.
`Providing
`high
`levels
`of detail
`of
`local
`objects
`along with
`coarse
`approximations
`of objects
`situated
`within
`a panoramic
`view will
`quickly
`tax many
`current
`hardware
`configurations.
`Nevertheless,
`this
`approach
`allows
`users
`to
`exploit
`their
`spatial
`and
`physical
`skills
`to navigate
`through
`the
`space,
`and
`it
`should
`become more
`viable
`as processing
`power
`and computer
`graphics
`technology
`improve.
`virtual
`immersive
`type of
`to this
`has been drawn
`deal of attention
`A great
`1992;
`and
`Levit
`[Bryson
`head-mounted
`displays
`reality
`which
`implies
`Sutherland
`1968]. Furness
`[1986]
`introduced
`the idea of a “super
`cockpit”
`by
`providing
`the pilot
`a combination
`of physical
`controls
`with
`a synthetic
`scene
`to improve
`a pilot’s
`spatial
`awareness.
`Aiming
`at providing
`a transparent
`man-machine
`interface
`to control
`an anthropomorphically
`designed multiple-
`degrees-of-freedom
`telerobot,
`Fisher
`et al.
`[1986]
`described
`a “telepresence”
`system which
`employs
`a head-mounted
`display
`and Dataglove.TM
`The
`ulti-
`
`the
`
`of
`
`ACM
`
`TransactIons
`
`on Information
`
`Systems,
`
`Vol.
`
`11, No,
`
`3, July
`
`1993.
`
`Mullen Industries, LLC - Ex. 1016
`Page 3
`
`

`

`feel
`
`at
`“present”
`operator
`[Tachi
`systems
`telepresence
`over
`traditional
`teleoperation
`
`the
`et
`
`200
`
`.
`
`George W, Fltzmaurlce et al
`
`the human
`advanced
`advantages
`
`system is to make
`this
`goal of
`mate
`Indeed,
`one of
`the most
`site.
`remote
`demonstrated
`significant
`al. 1990]
`in a number
`of
`tasks.
`a large
`the 3D effect. With
`also been used to enhance
`Large
`screens
`have
`stereoscopic
`screen
`and a force-feedback
`robot
`arm as an input
`device, Brooks
`et al.
`[1990]
`demonstrated
`a new tool
`for
`scientists
`to better
`comprehend
`chemical
`reaction
`processes.
`small
`a very
`with
`3D comprehension
`to achieve
`In this
`paper, we attempt
`such as stereop-
`cues in 3D display,
`used depth
`screen. Other
`than
`frequently
`sis,
`interposition,
`and perspective,
`the act of movement
`also provides
`a great
`deal of 3D sensation,
`as suggested
`in the motor
`theory
`of space perception
`[Gibson
`1950; Wallach
`and O’Connell
`1953]. As Gibson
`argued,
`“seeing”
`and
`“acting”
`are not separable.
`He also states
`that
`3D comprehension
`in humans
`develops
`as a result
`of
`the
`intimate
`interaction
`of vision
`and muscular
`movement
`during
`ceaseless
`exploration
`of one’s
`environment.
`The
`palmtop
`to
`virtual-reality
`concept
`we are
`proposing
`utilizes
`kinesthetic
`movement
`overcome
`disadvantages
`of a small
`screen
`size while
`integrating
`visual
`the
`and motor
`exploration
`of 3D displays.
`
`3.
`
`INTERACTION MODEL
`
`based
`spatially
`is an “ego-centric”
`prototype
`for our
`model
`interaction
`The
`one cell of
`the
`spreadsheet”
`in which
`a palmtop
`screen
`shows
`“virtual
`cubic
`at any
`given
`time
`(Figure
`1). Objects
`within
`each
`cell
`can be
`spreadsheet
`in 2D or
`in 3D perspective.
`Moving
`the palmtop
`left or right
`(x axis)
`rendered
`will
`change
`the viewing
`of
`the spreadsheet
`on a column
`basis while
`raising
`or
`lowering
`the device
`(y axis) will
`allow for
`row movement
`within
`the spread-
`sheet. Moving
`the
`palmtop
`closer
`or
`further
`away
`from the
`user
`(z axis)
`allows
`a sequence
`of spreadsheets
`to be viewed.
`This
`z axis may best be used
`to alter
`the temporal
`view of data;
`older
`levels
`are visible
`as the
`device
`is
`pushed
`away,
`and future
`levels
`are made visible
`as the device is pulled
`closer
`to the user.
`the notion
`is to introduce
`this model
`of
`modification
`yet powerful
`A simple
`for
`issuing
`viewing
`of adjacent
`cells
`or
`for
`localized
`the
`palmtop
`tilting
`of
`commands. We can pretend
`that
`the device
`is resting
`on a pin located
`in the
`center
`of
`the device. Tilting
`the device will
`cause the display
`to jump
`to one of
`the
`8 adjacent
`cells
`depending
`on which
`edge(s)
`are moved
`(Figure
`2). For
`example,
`the
`adjacent
`left
`cell would
`be displayed
`if
`the
`right
`edge of
`the
`device
`is tilted
`upward.
`com-
`issuing
`of
`way
`be a natural
`could
`action
`tilting
`Alternatively,
`this
`1991;
`[Hopkins
`of
`pie menus
`concept
`the
`to
`mands.
`This
`similar
`is
`choices
`are selected
`by a physical
`that menu
`except
`Kurtenbach
`et al. 1993]
`tilting
`action
`in 3D instead
`of a mouse
`or pen-directional
`marking
`in 2D. For
`example,
`to browse
`textual
`data,
`a palmtop
`application
`could
`be designed
`which
`uses the tilting
`mechanism
`as a means
`of
`issuing
`commands.
`Tilting
`the top part
`of
`the palmtop
`toward
`the user would
`indicate
`a “Scroll-Up
`by
`
`ACIVI TransactIons
`
`on Inf’ormatlon
`
`Systems,
`
`Vol
`
`11, No
`
`3. July
`
`1993
`
`Mullen Industries, LLC - Ex. 1016
`Page 4
`
`

`

`Virtual Reality for Palmtop Computers
`
`.
`
`201
`
`Y
`
`o ~ “
`
`“ PALMTOP
`
`—x
`
`/
`
`‘
`
`z
`
`PAST
`
`FUTURE
`
`Q
`
`e
`
`at any
`the spreadsbeet
`one cell of
`displays
`The hand unit
`cubic spreadsheet.
`Fig. 1. Virtual
`locations,
`the appropriate
`cell of
`the
`spatial
`given time. As a user moves the unit
`to different
`In this
`instance
`the z axis has been
`spreadsheet
`is shown in the display
`of
`the hand unit.
`associated with time; past events are further
`away, and future
`events are closer
`to the person
`holding
`the palmtop.
`
`(a)
`
`by making
`cells (a) can be achieved
`of adjacent
`viewing
`Localized
`Fig. 2.
`mechanism.
`The user pretends
`that
`the hand unit
`is resting
`on a pin. Tilting
`allow for
`the appropriate
`adjacent
`cell
`to be viewed.
`
`use of a tilting
`the unit
`(b) will
`
`a “Scroll-Down
`indicates
`upward
`the bottom part
`tilting
`while
`command
`line”
`Page”
`command
`and “Previous
`a “Next
`Page”
`Similarly,
`by line”
`command.
`with
`a right
`and left
`tilting
`action,
`respectively
`(Figure
`could
`be associated
`provides
`a means
`of browsing
`video
`data on a palmtop
`3a). A second example
`using
`the tilting
`mechanism.
`Tilting
`the hand
`unit
`toward
`the right
`would
`indicate
`a “forwardVideo”
`command
`while
`tilting
`toward
`the left would
`issue
`
`ACM TransactIons on Information Systems, Vol 11, No, 3, July
`
`1993.
`
`Mullen Industries, LLC - Ex. 1016
`Page 5
`
`

`

`202
`
`.
`
`George W. Fltzmaurlce et al,
`
`(a)
`
`(b)
`
`13g. 3, Ple menus.
`
`(a) text-browsing
`
`commands;
`
`(b) v]deo-browsing
`
`commands
`
`could
`occurs
`the tilt
`to which
`the degree
`Finally,
`tilts would
`be small
`units
`( 1
`the command:
`slight
`of
`greater
`tilts would
`indicate
`larger
`units
`(100 frames
`
`command.
`a “reverseVideo”
`levels
`indicate
`parametric
`frame
`per second) while
`per second)
`(Figure
`3b ).
`interaction
`spreadsheet”
`cubic
`“virtual
`based
`This
`“ego-centric”
`spatially
`a simple mental model
`(i.e.,
`the
`technique
`relies
`on having
`the user maintain
`spreadsheet
`or cube) and exploits
`both
`spatial memory
`and muscle memory.
`
`4. THE CHAMELEON PROTOTYPE
`
`how palmtop
`for exploring
`a framework
`is to provide
`The goal of our design
`gestural
`in-
`be used with
`graphics
`could
`computers
`with
`high-performance
`visualizing,
`exploring,
`and
`puts
`to provide
`portable
`VR environments
`for
`interacting
`with
`3D information
`spaces. The prototype
`framework,
`known
`as
`Chameleon,
`supports
`this
`type of
`investigation
`which
`is described
`below.
`Many
`of
`today’s
`palmtop
`computers
`do not have
`the
`capabilities
`of high-
`performance
`graphics
`workstations
`such
`as the Silicon
`Graphics
`worksta-
`tions. However,
`we have been able to simulate
`a palmtop
`of
`the future
`which
`would
`have these features
`as well
`as additional
`input
`capabilities.
`4-inch
`The Chameleon
`prototype
`configuration
`(see Figure
`4) uses a small
`for
`the
`fits
`color LCD monitor,
`which
`comfortably
`in one’s hand,
`as the basis
`palmtop
`of
`the
`future.
`A Silicon
`Graphics
`4D/310GTX
`Iris workstation
`is
`used as the main
`processing
`unit
`and graphics
`engine. Currently,
`the images
`on the workstation
`are fed into
`the small
`hand
`unit
`by using
`a video
`camera.
`Technologically
`more
`robust
`solutions
`will
`improve
`the
`video
`quality
`(e.g.,
`using
`an NTSC output
`channel
`on the SGI workstation).
`This
`configuration
`gives
`the illusion
`that
`the 4-inch monitor
`(i.e., palmtop)
`has the capabilities
`an SGI workstation,
`albeit with
`an inferior
`image
`size and resolution.
`to
`ability
`To facilitate
`input
`controls,
`the
`palmtop
`has
`been
`given
`the
`The
`monitor
`its position
`and orientation
`information
`within
`a 3D workspace.
`input
`Ascension
`BirdTM (by Ascension
`Technology)
`is a 6-degree-of-freedom
`device
`and is attached
`to the small monitor.
`This
`allows
`the system to detect
`user gestures
`in terms
`of positional
`data
`(x, y, and z dimensions)
`as well
`as
`orientation
`data of
`the palmtop
`(pitch,
`yaw, and roll).
`In addition,
`a mechani-
`cal push
`button
`has been
`attached
`to the top of
`the device
`to obtain
`input
`selections
`from the user
`(see Figure
`5). Both
`the bird
`and the response
`button
`
`of
`
`ACM
`
`TransactIons
`
`on Information
`
`Systems.
`
`VO1
`
`11, No
`
`3, July
`
`1993
`
`Mullen Industries, LLC - Ex. 1016
`Page 6
`
`

`

`Virtual Real! tyfor Palmtop Computers
`
`.
`
`203
`
`COMPUTER
`DISPLAY
`
`/R6DsouRcE
`
`Fig.4.
`
`Configuration
`
`of the Chamelwmprototype.
`
`MONITOR AND
`6D SENSOR
`
`unit
`Palmtop
`Fig.5.
`for providing
`position
`
`display,
`ofavideo
`consisting
`and orientation
`information
`
`response bottom (top) anda6D
`(x, y, z, pitch, yaw, and roll).
`
`input
`
`sensor
`
`ACM Transactions on Information Systems, VoLll,
`
`No 3,,~ulY 1993.
`
`Mullen Industries, LLC - Ex. 1016
`Page 7
`
`

`

`204
`
`.
`
`George W. Fltzmaurlce et al.
`
`simple
`
`gestures
`
`to navigate
`
`within
`
`a 3D workspace
`
`to make
`allow the user
`and to issue commands.
`the BirdTM
`of
`response
`dynamic
`the
`measure
`It
`is difficult
`to precisely
`developed
`[ 1992]
`have
`et al.
`device
`as well
`as other
`6D devices. Adelstein
`these
`devices.
`Informal
`controlled
`testbed
`and methodology
`for measuring
`data measurements
`for our optimized
`bird
`device driver
`have an approximate
`response
`delay
`of 50 ms providing
`translational
`accuracy
`on the order
`of 0.1
`inch
`and angular
`accuracy
`on the order
`of 0.5 degrees within
`a range
`of a
`3-foot
`cube. All
`of
`these measurements
`are
`sufficiently
`fast
`and
`reliable
`enough
`to support
`novel
`interaction
`techniques.
`(x and y axes) and
`To navigate
`through
`a 3D workspace,
`the translation
`back
`to the
`SGI
`fed
`zoom (z axis)
`data
`the
`palmtop
`constantly
`of
`are
`workstation
`which
`dynamically
`updates
`the 3D scene being
`relayed
`back
`to
`the palmtop
`unit. Objects
`appearing
`in the 3D scene are drawn
`in perspec-
`tive. The net effect
`is that
`the palmtop
`unit
`acts as a porthole
`window
`into
`3D workspace.
`For example,
`as a user
`translates
`the palmtop
`unit
`to the left,
`he or she moves
`toward
`the left wall
`of
`the 3D workspace.
`cubic
`in the virtual
`In addition,
`we have opted
`to make
`the cell boundaries
`not
`spreadsheet
`transparent
`instead
`of opaque
`(Figure
`6a). Users,
`therefore,
`but
`of
`only
`see the contents
`the cell
`they
`are occupying
`in the spreadsheet
`the
`also all
`the deeper
`cells
`(i.e.,
`those
`in the z axis;
`see Figure
`6b). Finally,
`prototype
`allows
`for perspective
`viewing
`instead
`of an orthogonal
`view from
`the current
`cell
`(Figure
`6c). This means
`that
`the user
`views
`cells adjacent
`to
`the current
`cell
`in a slight
`fisheye manner
`[Furnass
`1986; Robertson
`et al.
`1991]. Both
`the transparent
`cells and perspective
`viewing
`provide
`users with
`additional
`orientation
`information.
`However,
`these
`characteristics
`can
`be
`changed
`depending
`on the
`application
`or
`information
`space being
`browsed.
`For example,
`if a user
`is browsing
`a spreadsheet
`of numbers,
`it may be too
`confusing
`to have
`transparent
`cells
`since
`the numbers
`would
`overlap
`along
`the z axis.
`may
`spreadsheet
`cubic
`in the virtual
`the cells
`size of
`The granularity
`granularity.
`Coarser
`fine
`at a very
`cells
`are
`the
`be set. By
`default,
`granularities
`offer a gridding
`effect which
`is best suited
`for discrete
`instead
`continuous
`translations.
`deeper
`a user moving
`7a shows
`Figure
`effect,
`To illustrate
`the navigation
`7b and 7C show a before-and-after
`view of
`the
`into
`a 3D workspace.
`Figures
`of
`the palmtop
`screen
`as the user
`zooms
`into
`palmtop
`unit
`and the contents
`The
`3D space consists
`of a room containing
`the 3D space along
`the z-axis.
`cubes
`randomly
`positioned
`in the space. The cubes
`could
`represent
`a spatial
`database
`or a user’s
`filing
`system in which
`each cube symbolizes
`a file. Note
`that
`as the palmtop
`is moved
`from the start
`to the end position,
`the user sees
`a smooth
`zooming
`animation
`during
`the traversal
`of
`the path.
`and
`The system can also switch
`into “clutch”
`mode in which
`the translation
`(i.e.,
`zoom controls
`are registered
`only while
`the response
`button
`is depressed
`the clutch
`is engaged). When
`the button
`is released,
`the position
`is frozen,
`and
`the
`user
`is able
`to move
`the
`palmtop
`to a closer, more
`comfortable
`position
`or
`to place the device
`on the desktop.
`If
`the clutch
`is engaged
`again,
`
`a
`
`a
`
`also
`cell
`of
`
`ACM
`
`Transactions
`
`on Information
`
`Sy.tems,
`
`Vol
`
`11, No
`
`:3, July
`
`1993
`
`Mullen Industries, LLC - Ex. 1016
`Page 8
`
`

`

`Virtual Reality for Palmtop Computers
`
`.
`
`205
`
`9 (c)
`
`(b) transparent
`
`orthogonal,
`
`Fig. 6. Cell options
`and (c) transparent
`
`the virtual
`for
`perspective.
`
`cubic spreadsheet:
`
`(a) opaque,
`
`Fig. 7a, Palmtop
`workspace.
`
`umt
`
`in clutch mode. A user holds a button to register movement within
`
`the 3D
`
`adjusts
`the model
`resume movements
`A cross-hair
`cursor
`select objects within
`cross-hair
`and
`click
`button.
`This
`causes
`from the palmtop
`
`position
`
`to the current
`the 3D workspace
`from the last
`registered
`position.
`to
`to allow the user
`is fixed
`in the center
`of
`the screen
`the
`the virtual
`world. Users
`line up the target
`object over
`in
`on (or
`some
`situations
`double-click)
`the
`response
`at
`an imaginary
`ray
`centered
`the cross-hair
`to emanate
`unit
`toward
`objects
`in the virtual
`world.
`The first
`object
`
`of
`
`the palmtop
`
`to
`
`ACM
`
`Transactions
`
`on Information
`
`Systems,
`
`Vol.
`
`11, No.
`
`3, July
`
`1993.
`
`Mullen Industries, LLC - Ex. 1016
`Page 9
`
`

`

`206
`
`.
`
`George W Fitzmaur[ce et al
`
`,,
`
`“/’
`
`,<’
`
`Fig, 7C What
`
`the user sees on the palmtop
`
`after gesturing
`
`deeper
`
`into the 3D workspace.
`
`ACM
`
`TransactIons
`
`on Information
`
`Systems,
`
`Vol
`
`11, No
`
`3,
`
`,July
`
`1993
`
`Mullen Industries, LLC - Ex. 1016
`Page 10
`
`

`

`Virtual Reality for Palmtop Computers
`
`.
`
`207
`
`a
`the
`since
`
`of
`
`and can
`are important
`feedback
`and auditory
`is selected. Visual
`encountered
`For example,
`as the cross-hair
`improve
`a user’s
`accuracy
`in selecting
`objects.
`can change
`color.
`In addition,
`cursor
`is in range
`over a target,
`the target
`can be made to indicate
`that
`nonspeech
`audio
`sound (e.g., a low click
`tone)
`feedback
`is important
`user
`is over a target.
`This
`type
`of reinforcement
`can be small.
`the display
`resolution
`can be low and since targets
`the
`into
`can
`be built
`Simple
`gestures
`and
`application-specific
`controls
`of a file
`system.
`For example,
`if
`the workspace
`contains
`a 3D tree hierarchy
`system,
`then
`users would
`be able to select a node and gesture
`downward
`in a
`tugging
`motion
`to cause
`the
`tree
`to rotate
`along
`the
`x axis. Conversely,
`selecting
`a node
`and tugging
`upward
`would
`cause the tree
`to rotate
`in the
`opposite
`direction.
`information
`the orientation
`use of
`does not make
`The current
`prototype
`ahead into the 3D workspace.
`the 6D input
`device;
`users always
`look straight
`of
`true
`one-to-one
`mappings
`Future
`versions
`of
`the prototype
`may
`allow for
`the palmtop
`orientation
`to the 3D model. Note that
`this
`interface
`may not be
`optimal
`since
`the
`user would
`have
`to physically
`move
`into
`a new viewing
`perspective
`to get a corresponding
`view into
`the 3D workspace.
`For example,
`rotating
`the device
`by 90 degrees
`along
`the y-axis
`to get a side view means
`that
`the user would
`also have to move his or her head to the side in order
`to
`see the display
`on the palmtop. While
`the scenes generated
`for
`the palmtop
`display
`are drawn
`in 3D perspective,
`we are looking
`at providing
`additional
`interactive
`viewpoint
`control
`[McKenna
`1992]. Moreover,
`tracking
`the user’s
`head
`position
`along with
`the
`palmtop
`position
`may
`provide
`greater
`depth
`sensation.
`proto-
`rapid
`us to perform
`allows
`configuration
`prototype
`The Chameleon
`to a
`efforts
`are not
`tied
`since
`our
`development
`typing
`of new applications
`com-
`found
`on palmtop
`highly
`customized
`programming
`environment
`often
`we are
`the
`applications
`puters.
`That
`is, depending
`on the
`requirements
`of
`exploring,
`it
`is possible
`to use any computer
`as our base machine
`(e.g., Silicon
`Graphics
`or Sun workstation
`or a Macintosh
`computer)
`and any development
`environment
`best suited
`for
`the task.
`(due
`by cords
`tethered
`is currently
`Although
`our prototype
`palmtop
`device
`for
`environment
`a rich
`it provides
`to the video
`feed
`and 6D input
`device),
`in a technology
`and user
`interactions
`testing
`new situations,
`applications,
`configuration
`which we anticipate
`be available
`in a few years
`in a highly
`portable
`form.
`
`will
`
`5. EXPERIMENT
`
`The
`in an experiment.
`system was tested
`the Chameleon
`of
`The performance
`screen
`in the
`small
`could
`be perceived
`purpose
`was
`to see how well
`depth
`compared
`to the large
`screen
`and the extent
`to which movement
`of
`the small
`display
`(in a palmtop
`virtual-reality
`condition)
`could
`improve
`depth
`percep-
`tion.
`in the experiment
`The subjects
`Toronto
`community.
`Each
`subject
`
`from the University
`30 trials
`in
`each
`
`of
`of 3
`
`12 volunteers
`were
`participated
`in
`
`ACM
`
`TransactIons
`
`on Information
`
`Systems,
`
`Vol.
`
`11. No.
`
`3, July
`
`1993
`
`Mullen Industries, LLC - Ex. 1016
`Page 11
`
`

`

`208
`
`-
`
`George W. Fitzmaunce et al.
`
`it.
`
`a 3D
`of cubes within
`a field
`the subject was shown
`For each trial
`conditions.
`further
`cubes was
`space and was asked
`to select which
`of
`two
`highlighted
`consisted
`of a 2 l-inch
`away
`on the z-axis
`(i.e.,
`into
`the screen). One condition
`consisted
`of
`the
`4-inch
`monitor
`with
`a static
`display.
`The
`second
`condition
`the
`third
`condition,
`the
`television
`monitor,
`also with
`a static
`display.
`In
`4-inch
`television
`monitor
`was again
`used,
`but
`now the
`subject
`was able to
`move through
`the display
`using
`the palmtop
`virtual-reality
`gestures
`for zoom
`and translation
`that were described
`earlier
`in this
`paper.
`Figures
`8a through
`8C show the three
`conditions.
`In the first
`two conditions,
`the mouse was used
`to position
`the cursor within
`the display
`and make
`the selection.
`In the third
`condition,
`the selection
`was made
`by moving
`the monitor
`until
`a cursor was
`positioned
`over
`the box, and then
`a response
`button
`attached
`to the monitor
`(as
`shown
`in Figure
`5) was
`clicked.
`A limited
`amount
`training
`was
`of
`provided
`in each of
`the
`three
`conditions,
`with
`subjects
`practicing
`the
`task
`until
`they
`said
`they were
`comfortable
`with
`In
`no case did
`the
`initial
`training
`period
`last more
`than
`a few minutes.
`in each condition,
`The
`number
`of errors
`were
`tabulated
`for each subject
`making
`a total
`of 36 (3 x 12) data points.
`These data were then
`subjected
`to a
`one-way
`analysis
`of variance.
`There was a significant
`difference
`between
`the
`three
`conditions
`(F[2, 331 = 34.9,
`p < 0.001),
`with
`the
`large
`display
`(2.08
`errors
`per subject)
`and the palmtop
`condition
`(3 errors
`per subject)
`leading
`to
`fewer
`errors
`than
`the
`small
`(stationary
`ancl static)
`display
`(6.08
`errors
`per
`subject).
`Post hoc analysis
`showed
`no significant
`difference
`in the number
`of
`errors
`produced
`in the
`large-display
`condition
`versus
`the
`palmtop
`virtual-
`reality
`condition.
`Tbe median
`conditions.
`the three
`across
`Response
`times were also collected
`response
`times
`for each condition
`were
`calculated
`for each subject
`and were
`then
`subjected
`to analysis
`of variance.
`Response
`times
`varied
`significantly
`across
`the three
`conditions
`(F[2, 33] = 112.9, p < 0.0001), with
`the large-
`and
`small-display
`condition
`having
`a significantly
`shorter
`mean
`response
`time
`(2.75
`and 3.75 seconds,
`respectfully)
`than
`the palmtop
`virtual-reality
`condi-
`tion ( 10.67 seconds). This
`result
`is not
`too surprising
`since the third
`condition
`allowed
`the
`subjects
`to explore
`the
`environment.
`two
`cubes
`had
`similar
`If
`depth
`placement,
`then
`the subject
`could investigate
`or visit
`each cube.
`In the
`other
`two static
`scene conditions,
`the user must make
`a decision,
`perhaps
`an
`arbitrary
`one, without
`investigation.
`very
`show that, with
`The results
`of
`this
`experiment
`virtual-reality
`condition
`tion
`of depth
`in the
`palmtop
`in
`a 1arge
`(but
`static)
`corresponding
`depth
`perception
`(an average
`of 3–4 more
`performance
`was much
`poorer
`was used.
`when
`the static
`small
`display
`in develop-
`seems to be critical
`The ability
`to move and explore
`the display
`ing adequate
`depth
`perception
`on a small
`screen. While
`depth
`perception
`was
`relatively
`good in the
`palmtop
`virtual-reality
`condition,
`the
`response
`time
`data
`showed
`that
`selecting
`the box that was further
`away
`took much
`longer
`in the palmtop
`condition.
`After
`the
`experiment,
`the subjects
`indicated
`that
`this
`longer
`time was partially
`due to the difficulty
`in physically
`making
`the
`
`percep-
`training,
`little
`is about
`as good
`as
`display.
`However,
`errors
`per 30 trials)
`
`.4(3M Transactions
`
`on Information
`
`Systems,
`
`\701
`
`11. No
`
`:3, July
`
`1993
`
`Mullen Industries, LLC - Ex. 1016
`Page 12
`
`

`

`Virtual Reality for Palmtop Computers
`
`.
`
`209
`
`First
`Fig. 8a.
`mouse input.
`
`experiment
`
`condition
`
`consisting
`
`of a large 21-inch monitor,
`
`static display,
`
`and
`
`the
`of
`further
`palmtop
`
`about which
`that with
`with
`the
`
`selection,
`decision
`the perceptual
`in making
`than
`rather
`away
`on the z-axis. We anticipate
`further
`two boxes was
`would
`be able
`to make
`selections
`experience
`subjects
`virtual-reality
`system much more
`quickly.
`the
`that
`stated
`initially
`While
`running
`the
`experiments,
`a few subjects
`perceived
`they
`and that
`palmtop
`virtual-reality
`controls
`were
`not
`intuitive
`to the
`left
`the
`device
`them as being
`completely
`reversed.
`That
`is, moving
`the user
`to the left with
`should move the entire
`3D world
`or a target
`object
`that
`they
`had an object
`always
`in a fixed
`location.
`These
`comments
`revealed
`After
`further
`usage,
`they
`view instead
`of an ego-centric
`view for
`the controls.
`all were able to switch
`to the ego-centered
`model. We found
`that
`users having
`difficulty
`with
`understanding
`the ego-centered
`model
`could make
`the transi-
`tion more
`quickly
`if
`they
`physically
`rocked
`in their
`seat while
`keeping
`the
`palmtop
`at a constant
`distance
`from their
`bodies. For example,
`to zoom in and
`out of
`the 3D workspace
`the subject would
`rock
`forward
`and then
`backward.
`The physical
`act of moving
`the body
`corresponded
`directly
`with
`the controls
`for
`the palmtop
`and complemented
`the visual
`display
`to reinforce
`the notion
`of moving
`around
`in a virtual
`world.
`
`6. DESIGN VARIATIONS
`After
`conducting
`the experiment,
`design
`space
`the
`palmtop
`
`for
`
`to explore
`begun
`we have
`unit
`in
`terms
`of modifying
`
`the
`into
`deeper
`the movement
`
`ACM
`
`Transactions
`
`on Information
`
`Systems,
`
`Vol
`
`11, No
`
`3, July
`
`1993
`
`Mullen Industries, LLC - Ex. 1016
`Page 13
`
`

`

`210
`
`.
`
`George W Fitzmaurlce
`
`et al
`
`Sb. Second experiment
`Flg
`mouse input
`
`condition
`
`consisting
`
`of a small
`
`4-inch screen, static display,
`
`and
`
`controls,
`interaction
`applications.
`
`shape
`size and
`the
`varying
`other
`of
`the palmtop
`with
`Each
`of
`these
`are described
`
`3D workspace,
`the
`of
`computational
`devices,
`in some detail
`below.
`
`the
`considering
`and exploring
`
`Movement
`
`Controls
`
`of
`
`two mechanisms:
`in our prototype
`implemented
`but not yet
`We have designed
`some
`of
`the
`physical
`orientation
`glancing
`and mirror
`mode
`for
`alleviating
`to the left or right
`the current
`problems.
`To take
`a temporary
`glance
`either
`the
`device
`between
`15 and
`45
`palmtop
`position,
`the
`user would
`rotate
`degrees.
`The
`angles
`received
`by the
`6D input
`device would
`be exaggerated
`such
`that
`a 45 degree
`rotation
`corresponds
`to a temporary
`full
`90 degree
`for
`smooth
`viewing
`change
`in the desired
`clirection.
`This
`allows
`the display
`of
`the palmtop
`to be seen by the user without
`much,
`if any, adjustment
`by the
`user’s
`head. To compensate
`for
`small
`unintentional
`rotations,
`no change
`in
`viewing
`perspective
`will
`be registered
`between
`+ 15 and
`– 15 degrees.
`could
`To view the back
`side of objects within
`the 3D workspace,
`the user
`switch
`into mirror
`mode
`in which
`the palmtop
`acts as a mirror.
`Instead
`of
`users
`seeing
`deeper
`into
`the
`z-axis,
`they
`are now viewing
`objects
`between
`their
`eyes and the palmtop
`unit.
`That
`is,
`instead
`of
`the “eye” being
`placed
`on
`the back of
`their
`hand,
`it
`is temporarily
`placed
`on the front
`of
`their
`hand.
`To
`
`ACM
`
`Transactions
`
`on