INTERACTION TECHNIQUES FOR COMMON TASKS IN
`IMMERSIVE VIRTUAL ENVIRONMENTS
`DESIGN, EVALUATION, AND APPLICATION
`
`A Thesis
`Presented to
`The Academic Faculty
`
`by
`
`Douglas A. Bowman
`
`In Partial Fulfillment
`of the Requirements for the Degree
`Doctor of Philosophy in Computer Science
`
`Georgia Institute of Technology
`June 1999
`
`Mullen Industries, LLC - Ex. 1030
`Page 1
`
`

`

`INTERACTION TECHNIQUES FOR COMMON TASKS IN
`IMMERSIVE VIRTUAL ENVIRONMENTS
`DESIGN, EVALUATION, AND APPLICATION
`
`Approved:
`
`_______________________________
`Larry F. Hodges, Chairman
`
`_______________________________
`Gregory Abowd
`
`_______________________________
`Albert N. Badre
`
`_______________________________
`Elizabeth T. Davis
`
`_______________________________
`Jarek Rossignac
`
`Date Approved: __________________
`
`i i
`
`Mullen Industries, LLC - Ex. 1030
`Page 2
`
`

`

`ACKNOWLEDGMENTS
`
`The author would like to thank the following people for their help and advice:
`• His advisor, Dr. Larry F. Hodges
`• The members of the thesis committee: Dr. Gregory Abowd, Dr. Albert Badre, Dr.
`Elizabeth Davis, and Dr. Jarek Rossignac
`• David Koller for his work on the viewpoint motion control experiments
`• Dr. Jean Wineman, Brian Wills, and Don Allison for their work on the VR Gorilla
`Exhibit
`• Donald Johnson for his work on the testbed evaluations and the final version of the
`Virtual Habitat
`• Mark Mine, Richard Stoakley, Jeff Pierce, Andrew Forsberg, Ken Hinckley,
`Matthew Conway, Barry Peterson, Ivan Poupyrev, Ernst Kruijff, Joseph LaViola,
`and Ben Watson for various collaborations and discussions
`• The members of the 3D UI mailing list
`• Drew Kessler for help with the SVE toolkit
`• The Virtual Environments group at Georgia Tech
`• The numerous experimental subjects who volunteered their time
`• Dawn Bowman
`
`iii
`
`Mullen Industries, LLC - Ex. 1030
`Page 3
`
`

`

`TABLE OF CONTENTS
`
`Introduction ...................................................................................... 1
`1.1 Motivation ....................................................................................1
`1.2 Definitions....................................................................................4
`1.3 Problem Statement...........................................................................6
`1.4 Scope of the Research.......................................................................7
`1.5 Hypotheses...................................................................................8
`1.6 Contributions.................................................................................9
`1.7 Summary of This Work.....................................................................9
`1.8 Summary of Recommendations.......................................................... 10
`1.8.1 Generic VE Interaction Guidelines ............................................... 10
`1.8.2 Guidelines for the Design of Travel Techniques................................ 10
`1.8.3 Guidelines for the Design of Selection Techniques............................. 10
`1.8.4 Guidelines for the Design of Manipulation Techniques........................ 10
`
`Interaction in Virtual Environments................................................ 1 2
`2.1 Human-Computer Interaction Concepts ................................................ 12
`2.2 Three-Dimensional User Interfaces...................................................... 14
`2.3 Perceptual and Cognitive Psychology Concepts....................................... 15
`2.4 Evaluation of Immersive Virtual Environments........................................ 16
`
`Design and Evaluation Concepts................................................... 1 8
`3.1 Taxonomy and Categorization ........................................................... 18
`3.2 Guided Design ............................................................................. 20
`3.3 Performance Measures.................................................................... 21
`3.4 Range of Evaluation Methods............................................................ 21
`3.5 Testbed Evaluation ........................................................................ 22
`3.6 Models of Human Performance.......................................................... 23
`3.7 Application of Results..................................................................... 23
`3.8 Summary of Methodology................................................................ 24
`
`Travel ............................................................................................. 2 6
`4.1 Introduction and Definitions.............................................................. 26
`4.2 Related Work............................................................................... 26
`4.3 Original Evaluation Framework.......................................................... 27
`4.3.1 Categorization of Techniques ..................................................... 27
`4.3.2 Performance Measures............................................................. 29
`4.4 Initial Experiments......................................................................... 30
`4.4.1 Spatial Awareness Experiment.................................................... 31
`4.4.2 Absolute Motion Experiment...................................................... 32
`4.4.3 Relative Motion Experiment....................................................... 32
`4.5 Expanded Evaluation Framework ....................................................... 33
`4.5.1 Task Characteristics ................................................................ 34
`i v
`
`Mullen Industries, LLC - Ex. 1030
`Page 4
`
`

`

`4.5.2 Environment Characteristics....................................................... 34
`4.5.3 User Characteristics ................................................................ 35
`4.5.4 System Characteristics ............................................................. 36
`4.5.5 Information Gathering Experiment ............................................... 36
`4.5.5.1 Method........................................................................ 37
`4.5.5.2 Results ........................................................................ 41
`4.5.5.3 Discussion.................................................................... 42
`4.6 Alternate Evaluation Framework......................................................... 43
`4.6.1 Taxonomy ........................................................................... 43
`4.6.2 Guided Design ...................................................................... 44
`4.6.3 Spatial Orientation Experiment.................................................... 46
`4.6.3.1 Method........................................................................ 48
`4.6.3.2 Results ........................................................................ 50
`4.6.3.3 Discussion.................................................................... 54
`4.7 Travel Testbed ............................................................................. 56
`4.7.1 Method............................................................................... 56
`4.7.2 Results ............................................................................... 61
`4.8 Summary ................................................................................... 66
`
`Selection and Manipulation ........................................................... 6 7
`5.1 Introduction and Definitions.............................................................. 67
`5.2 Related Work............................................................................... 68
`5.2.1 Interaction Metaphors .............................................................. 68
`5.2.2 Evaluation of Techniques.......................................................... 69
`5.3 Initial Evaluation and Design............................................................. 70
`5.3.1 Techniques Considered ............................................................ 70
`5.3.2 User Study .......................................................................... 72
`5.3.3 HOMER Technique ................................................................ 73
`5.4 Formal Evaluation Framework........................................................... 75
`5.4.1 Categorization of Techniques ..................................................... 75
`5.4.2 Performance Measures............................................................. 77
`5.4.3 Outside Factors ..................................................................... 78
`5.4.3.1 Task Characteristics ......................................................... 78
`5.4.3.2 Environment Characteristics................................................ 78
`5.4.3.3 User Characteristics ......................................................... 79
`5.4.3.4 System Characteristics ...................................................... 79
`5.4.4 Guided Design ...................................................................... 79
`5.5 Selection/Manipulation Testbed.......................................................... 80
`5.5.1 Method............................................................................... 81
`5.5.2 Results ............................................................................... 82
`5.6 Summary ................................................................................... 87
`
`Interaction in a real-world VE application...................................... 8 8
`6.1 Integrating Techniques into an Application: Issues and Challenges ................. 88
`6.2 The Virtual Habitat ........................................................................ 89
`6.2.1 Original VR Gorilla Application .................................................. 89
`6.2.2 Application to Environmental Design Education................................ 90
`6.2.3 Interaction Requirements .......................................................... 91
`6.3 Interaction Design ......................................................................... 92
`v
`
`Mullen Industries, LLC - Ex. 1030
`Page 5
`
`

`

`6.3.1 Naïve Interaction Design........................................................... 93
`6.3.2 Intermediate Design Iteration...................................................... 95
`6.3.3 Final Interaction Design...........................................................100
`6.4 Final Usability Evaluation ...............................................................101
`
`Conclusions and Future Work ...................................................... 1 0 4
`7.1 VE Interaction Guidelines ...............................................................104
`7.1.1 Generic VE Interaction Guidelines ..............................................105
`7.1.2 Guidelines for the Design of Travel Techniques...............................105
`7.1.3 Guidelines for the Design of Selection Techniques............................107
`7.1.4 Guidelines for the Design of Manipulation Techniques.......................107
`7.2 Formal Design & Evaluation Frameworks ............................................108
`7.3 Focus on Applications and Usability...................................................109
`7.4 Future Work...............................................................................109
`7.4.1 Automatic Interaction Design and Performance Modeling....................109
`7.4.2 Cross-task Interaction Techniques...............................................110
`7.4.3 Comparison with Usability Engineering........................................111
`7.4.4 Interaction in Other Display Modalities .........................................112
`
`Appendix A: Standard User Questionnaire .................................. 1 1 3
`
`Appendix B: Comfort Ratings Form .............................................. 1 1 5
`
`Appendix C: Complete results of the travel testbed experiment .. 1 1 6
`
`Appendix D: Complete Results of the Selection/Manipulation
`Testbed Experiment ..................................................................... 1 1 8
`
`References ................................................................................... 1 2 3
`
`Vita............................................................................................... 1 3 2
`
`v i
`
`Mullen Industries, LLC - Ex. 1030
`Page 6
`
`

`

`LIST OF TABLES
`
`Table 4.1 Average Values of Overall Score for Each Tested Treatment Combination in the
`Information Gathering Experiment; Higher Scores are Better........................ 41
`Table 4.2 Number of Subjects Observed Using Common Strategies for Each Travel
`Technique................................................................................... 53
`Table 4.3 Mean Completion Times (seconds) for Naïve Search Task (Standard Deviation in
`Parentheses) ................................................................................ 62
`Table 4.4 Mean Completion Times (seconds) for Primed Search Task, with Targets not
`Within View from Start Location (Invisible) or In View from Start Location
`(Visible)..................................................................................... 64
`(*normalized times – seconds per 100 meters).................................................. 64
`Table 5.1 Mean Manipulation Time Results by Technique from Testbed Evaluation....... 84
`(* The linear mapping used in these cases was a one-to-one physical to virtual hand
`mapping).................................................................................... 84
`Table 5.2 Interaction Between Required Accuracy and Degrees of Freedom for
`Manipulation Time (seconds)............................................................. 85
`Table 6.1 Mean Usability Ratings for the Intermediate Virtual Habitat Interaction Design 99
`Table 6.2 Mean Usability Ratings (standard deviations in parentheses) for the Final Virtual
`Habitat Interaction Design ...............................................................102
`(* features changed since the previous iteration)...............................................102
`Table C.1 Results of Primed Search Task ......................................................116
`Table C.2 Demographic and Comfort Rating Summary ......................................117
`Table D.1 Speed Results for Selection Task....................................................118
`Table D.2 Speed Results for Manipulation Task...............................................119
`Table D.3 Demographic and Comfort Rating Summary ......................................122
`
`vii
`
`Mullen Industries, LLC - Ex. 1030
`Page 7
`
`

`

`LIST OF ILLUSTRATIONS
`
`Figure 2.1 General Taxonomy Format........................................................... 19
`Figure 2.2 Flowchart of Design and Evaluation Methodology................................ 25
`Figure 4.1 Taxonomy of Travel Techniques for Immersive Virtual Environments......... 28
`Figure 4.2 Environment for the Spatial Awareness Experiment .............................. 31
`Figure 4.3 The Relative Motion Experiment Environment..................................... 33
`Figure 4.4 Outside View of a Three-Dimensional Corridor ................................... 38
`Figure 4.5 Interior of a Corridor from the Information Gathering Experiment.............. 39
`Figure 4.6 Example Completed Corridor Map with Four Word/Location Pairs, One
`Unpaired Location, and Two Unpaired Words ........................................ 40
`Figure 4.7 Alternate Taxonomy for Travel Techniques with Detail on Position Indication
`Subtask...................................................................................... 44
`Figure 4.8 Route-planning Technique Using Virtual Map and Stylus........................ 45
`Figure 4.9 Inside View of a Corridor with a Target Object.................................... 46
`Figure 4.10 Views of Four Corridor Types used in the Spatial Orientation Experiment: Top
`left: 2D, right angles, top right: 2D, non-right angles, bottom left: 3D, right angles,
`bottom right: 3D, non-right angles....................................................... 48
`Figure 4.11 Mean Error in Various Treatment Combinations for Analysis 1 (left) and
`Analysis 2 (right) .......................................................................... 51
`Figure 4.12 Mean Time in Various Treatment Combinations for Analysis 1 (left), and
`Analysis 2 (right) .......................................................................... 52
`Figure 4.13 Example Obstacles from the Travel Testbed Experimental Environment...... 57
`Figure 4.14 Target Object from the Travel Testbed Experimental Environment Including
`Flag and Required Accuracy Radius..................................................... 58
`Figure 4.15 Virtual (top) and Physical (bottom) Views of the Map Dragging Travel
`Technique................................................................................... 60
`Figure 4.16 Interaction Between Task Type and Technique for Think Time on Search
`Tasks ........................................................................................ 65
`Figure 5.1 Mapping Function for the Go-Go Technique: Rr=Physical Hand Distance,
`Rv=Virtual Hand Distance. Reproduced from (Poupyrev et al, 1996).............. 71
`Figure 5.2 Stretch Go-Go Technique, with Gauge............................................. 71
`Figure 5.3 Time Sequence of the HOMER Technique ......................................... 74
`Figure 5.4 Taxonomy of Selection/Manipulation Techniques................................. 76
`Figure 5.5 Example Trial Setup in the Selection/Manipulation Testbed...................... 81
`Figure 5.6 Interaction of Selection Technique with Object Distance for Selection Time
`Measure ..................................................................................... 83
`Figure 5.7 Interaction of Selection Technique with Object Size for Selection Time Measure
`............................................................................................... 83
`Figure 5.8 Logarithmic Scale Graph of Interaction Between DOFs and Accuracy ......... 86
`Figure 6.1 The Virtual Reality Gorilla Exhibit .................................................. 89
`Figure 6.2 Embedded Audio and Text Information in the Virtual Habitat ................... 91
`viii
`
`Mullen Industries, LLC - Ex. 1030
`Page 8
`
`

`

`Figure 6.3 Virtual Menus in the CDS System................................................... 94
`Figure 6.4 Constrained Object Manipulation in CDS with Ray-Casting ..................... 95
`Figure 6.5 Physical Devices used in the Virtual Habitat Application ......................... 96
`Figure 6.6 User’s View of the Interface Tablet in the Virtual Habitat ........................ 97
`Figure 7.1 Example Taxonomy and Technique Components: If Performance Results for
`Techniques A, B, and C are Known, the Performance of Technique D can be
`Inferred.....................................................................................110
`Figure 7.2 Simplified Taxonomies Linked Together by Cross-Task Techniques..........111
`
`i x
`
`Mullen Industries, LLC - Ex. 1030
`Page 9
`
`

`

`SUMMARY
`
`Human-Computer Interaction (HCI)* in three dimensions is not well understood, and
`there are few 3D applications in common use. Moreover, the complications of 3D
`interaction are magnified in immersive virtual environment (VE) applications: characteristics
`such as inaccurate tracking and lack of access to traditional input devices cause the design
`of user interfaces (UIs) and interaction techniques (ITs) for immersive VEs to be extremely
`difficult. Despite these difficulties, we maintain that there are complex applications for
`which immersive VEs are desirable, so special attention needs to be paid to the design and
`implementation of ITs for these applications.
`A large percentage of interactions that take place in immersive VEs fall into a small
`number of general categories, which include travel (movement of the user's viewpoint from
`place to place), selection (indicating virtual objects within the environment), and
`manipulation (setting the position and/or orientation of virtual objects). Given techniques
`with good performance characteristics for these three interactions, a large number of
`complex and effective VE applications could be built. In this research we studied ITs for
`these three universal tasks in the context of a formal, systematic framework, including the
`design of novel ITs and empirical, comparative evaluations of techniques.
`This thesis presents several important results of the use of this methodology. First,
`we have developed new ITs perform well in a variety of application scenarios. Second, we
`have designed general testbeds for IT evaluation that may be reused for future performance
`comparisons. Third, we have obtained a large set of empirical results regarding the
`performance of ITs. These results led to general principles and guidelines (section 7.1) that
`can be applied to VE systems to improve performance. Finally, we validated these results
`by applying them to a real-world VE application, and showing that its usability was
`measurably improved as a direct result. The results presented in this thesis should be useful
`and important to anyone developing a VE system with even a moderate amount of
`interaction complexity.
`
`
`* For precise definitions of this and other key terms, see section 1.2.
`x
`
`Mullen Industries, LLC - Ex. 1030
`Page 10
`
`

`

`CHAPTER I
`
`INTRODUCTION
`
`1.1 Motivation
`Immersive virtual environments (VEs) made their debut in the late 1960s when Ivan
`Sutherland created the first system involving a tracked head-mounted display (HMD) and
`real-time three-dimensional computer graphics (Sutherland, 1968). The system was crude,
`and the amount of computing and rendering power was minuscule, compared to today’s
`technology, but all of the basic components that make up the virtual reality (VR) systems of
`the 1990s were present in Sutherland’s prototype.
`Since that time, there have been over thirty years of continuous research in the area of
`virtual environments. New hardware technology is continuously in development that
`allows us to render more complex 3D scenes at interactive frame rates. Graphics displays
`have seen tremendous improvement: we are able to display millions of different colors
`simultaneously on a very large screen at a refresh rate so fast that the human eye cannot
`perceive the flicker (Foley et al, 1990). There are many different tracking technologies
`available which provide 3D position and orientation data for multiple receivers
`simultaneously (Meyer and Applewhite, 1992). Technologies are being developed which
`provide input to other human sensory modalities besides vision. Haptic devices allow a VE
`user to seemingly “touch” virtual objects (Gomez, Burdea, and Langrana, 1995). Spatial
`sound creates the illusion of audio sources coming from certain locations in the 3D space
`(Durlach, 1991). There is even research into the use of olfactory input in virtual
`environments (Dinh et al, 1999).
`VE research has not focused entirely on hardware; software advances have also been
`made. Algorithms have been implemented and refined in the areas of model simplification,
`level of detail culling, geometry database management, texture mapping, lighting and
`shading, hidden surface elimination, and so on. All of these algorithms allow us to present
`a more complex and more realistic environment, while still maintaining real-time frame
`rates. Also, large software systems have been created expressly for the purpose of aiding
`the development of virtual environment applications (e.g. Kessler et al, 1998). These VE
`support systems can handle rendering, model maintenance, lighting, interfaces with
`trackers and other input devices, etc. This allows the developer to focus on the components
`which distinguish his VE application from others: the environment itself and the behavior
`of the application (e.g. response to button presses, animation, and interaction with virtual
`objects).
`What does the virtual environment community (primarily university researchers,
`small commercial ventures, and hobbyists) have to show for these thirty years of
`advancement in hardware and software specifically targeted at immersive VEs? Certainly,
`the degree of realism and complexity has increased, and making the virtual world more
`
`1
`
`Mullen Industries, LLC - Ex. 1030
`Page 11
`
`

`

`believable in this way may lead to a higher sense of immersion, or presence, for the user.
`But what applications have emerged into more common use outside of the laboratory?
`Surprisingly, our experience in the field indicates that there are very few VE applications in
`common use. To understand why, we should examine those applications that have become
`useful, and determine their common characteristics that allowed their success. Three such
`applications are architectural walkthrough, psychotherapy, and VE gaming (we discuss
`flight simulation and training, two other applications used for real work, below).
`Architectural walkthrough (Brooks, 1992) was perhaps the application which
`brought VE technology into the public eye more than any other. The basic idea is simple:
`the user can be immersed within a 3D model of an architectural space, and view it and
`move about it from a first-person perspective, as she would in an actual building. In this
`way, architects can verify the appropriateness and visual impact of their designs, engineers
`can study physical aspects of the space, and prospective clients can assess the current status
`of the project and suggest changes before a structure is even built. Why are VEs needed for
`this task, rather than simply viewing 3D models on a computer screen? One possible reason
`is that the user is immersed within the model, and can use her proprioceptive and
`kinesthetic senses to evaluate the space in a natural manner. Furthermore, this application
`requires only one additional component over those first proposed by Sutherland: some
`method of moving the user’s viewpoint about the space.
`Applications in the field of psychotherapy (Hodges et al, 1995, North, North, and
`Coble, 1996) have emerged rapidly since the early 1990s. One of the most well-known
`areas, which is beginning to see practical usage, is the treatment of various phobias. A
`common method of therapy for phobias is called graded exposure. The patient is placed in a
`situation in which the fear is triggered, but only slightly. He remains there with the
`therapist until he has mastered his fear in that situation, at which point a slightly more
`intense situation is presented. In this way, the patient gradually becomes able to deal with
`his fear. For example, to treat acrophobia, the fear of heights, the patient might be taken to
`a second floor balcony, then a fifth floor balcony, then the roof of a ten story building.
`This treatment has been shown to be effective, but also time-consuming, potentially
`embarrassing for the patient, and sometimes costly. The only requirement for exposure
`therapy is that the patient feel present in a situation which triggers his fear, which makes
`this application a natural one to try in a VE. The treatment can now take place in the
`therapist’s office, without the time, embarrassment, or cost associated with traditional
`exposure therapy. Unlike architectural walkthrough, VE exposure therapy does not even
`require a means for the user to move about. It is usually sufficient for the user to be able to
`sense the environment and to look around (using head tracking), so that the fear stimulus
`can be perceived.
`VE entertainment and game applications have also become popular in recent years.
`This has most often taken the form of location-based entertainment (LBE) through
`companies such as Virtuality(cid:212)
`, which involves a complete VE system installed in some
`permanent location, with users paying for each game. In any case, most of the games
`available for such systems can be characterized as first-person “shoot-em-up” games, in
`which the user moves through the virtual environment shooting his enemies. In many
`ways, the requirements of these games are similar to those for architectural walkthrough:
`real-time 3D graphics, head tracking, and some technique for moving through the
`environment. The only additional requirement is some sort of weapon that can be aimed
`and fired at the enemies in the game.
`What do these applications have in common? It seems that they all benefit from the
`enhanced sense of presence that an immersive virtual environment provides. “Being there”
`2
`
`Mullen Industries, LLC - Ex. 1030
`Page 12
`
`

`

`is what makes these systems more compelling or useful than the same 3D graphics
`rendered on a screen, with no head tracking. However, we also claim that each of these
`applications requires very little in terms of user interactivity. In applications such as
`exposure therapy, the user is mostly passive, simply looking around the space using
`standard head tracking. In the walkthrough and entertainment applications, the user may be
`more active (moving through the space, shooting, etc.), but the actions are very simple and
`repetitive. We would call this a high frequency but a low complexity of interaction.
`There are, however, a small number of applications being used for real work which
`have more complex characteristics of interaction. These include flight and vehicle
`simulation, which has been in use for many years, and training applications such as those
`used by NASA for simulation of astronaut “space walks.” Although these applications are
`more complex, the interaction is designed in a manner very specific to the system, and not
`in a way that could be extended to other types of applications. As Fred Brooks pointed out
`in his 1999 keynote address to the IEEE Virtual Reality conference, this is most often done
`by replicating the devices that the user would interact with in the real-world situation (e.g.
`the throttle and flight stick, or the spacesuit controls) and using those to drive the
`simulation. Because of this specificity to the application domain, we claim that there is little
`that we can learn in general about VE interaction from such systems.
`On the other hand, many more application areas have been proposed and researched
`for immersive VEs. The architectural community wants to take the walkthrough to the next
`step and be able to not only view, but also design artifacts in a VE (Bowman, 1996, Mine,
`1997). Prototype scientific visualization applications have been developed (Bryson and
`Levit, 1992, Taylor et al, 1993), in which scientists can interactively view complex
`simulations and structures, and also change the parameters of the simulation, move and
`regroup elements, and so on. Educational applications have been proposed (Dede,
`Salzman, and Loftin, 1996) that allow students to learn about certain concepts by engaging
`themselves in a virtual laboratory, and viewing the effects of changes first hand. The list
`goes on.
`However, we have not seen these applications in common use. It is our opinion that
`this is not because they are inappropriate for immersive VEs, but because their
`requirements for interaction are much more complex than the applications discussed
`previously. These systems require not only head tracking and a method of movement, but
`also the ability to select objects, to pick up, position, orient, and place objects, to change
`the system mode, to control the speed of a simulation, etc. One could argue that these
`applications are not in the mainstream due to the limitations of technology (input devices,
`trackers, displays, etc.), but researchers have been attacking the technology problem for
`thirty years. Our claim, on the other hand, is that because little research has been devoted to
`the user interface and interaction techniques for immersive VEs, the resulting prototype
`applications are not as usable as they need to be, and therefore do not see real-world usage.
`We must ask the question, “Given the current state of VE technology, is it possible for a
`virtual environment system to simultaneously be immersive, have complex interaction, and
`exhibit high levels of usability?”
`Why is it difficult to develop appropriate user interfaces and interaction techniques
`for immersive virtual environments? Shouldn’t interaction in VEs be completely natural,
`replicating the real world? Some have argued that this should be the case (Nielsen, 1993).
`Considering the applications we wish to develop for VEs, however, such natural
`interaction would be woefully inadequate. Instead, we want to extend the user’s physical,
`perceptual, and cognitive capabilities so that real work can be performed in a VE that could
`not be done easily in another setting. Therefore, we need new techniques for interaction.
`
`3
`
`Mullen Industries, LLC - Ex. 1030
`Page 13
`
`

`

`Why is the current state of the art not good enough? Interaction res