30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page i
`
`
`
`
`
`faces3D User Inte3D User Interr faces
`
`Supercell
`Exhibit 1009
`Page 1
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page ii
`
`Supercell
`Exhibit 1009
`Page 2
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page iii
`
`faces3D User Inte3D User Interr faces
`
`
`
`Theory and Practice
`
`Doug A. Bowman
`Doug A. Bowman
`
`ErEr nst Kruijfnst Kruijfff
`Joseph J. LaVJoseph J. LaViola, Jriola, Jr..
`
`
`
`Ivan PoupyrIvan Poupyrevev
`
`Boston • San Francisco • New York • Toronto • Montreal
`London • Munich • Paris • Madrid • Capetown
`Sydney • Tokyo • Singapore • Mexico City
`
`Supercell
`Exhibit 1009
`Page 3
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page iv
`
`Many of the designations used by manufacturers and sellers to distinguish their products are claimed as
`trademarks. Where those designations appear in this book, and Addison-Wesley was aware of a trade-
`mark claim, the designations have been printed with initial capital letters or in all capitals.
`The authors and publisher have taken care in the preparation of this book, but make no expressed or im-
`plied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed
`for incidental or consequential damages in connection with or arising out of the use of the information or
`programs contained herein.
`The publisher offers discounts on this book when ordered in quantity for bulk purchases and special sales.
`For more information, please contact:
`U.S. Corporate and Government Sales
`(800) 382-3419
`corpsales@pearsontechgroup.com
`For sales outside of the U.S., please contact:
`International Sales
`(317) 581-3793
`international@pearsontechgroup.com
`Visit Addison-Wesley on the Web: www.awprofessional.com
`Library of Congress Cataloging-in-Publication Data
`3D user interfaces : theory and practice / Doug A. Bowman . . .[et al.].
`p. cm.
`Includes bibliographical references and index.
`ISBN 0-201-75867-9 (hardbound : alk. paper)
`1. User interfaces (Computer systems) 2. Three-dimensional display
`systems.
`I. Bowman, Doug A., 1971–
`QA76.9.U83A14 2004
`005.4'37—dc22
`
`2004008403
`
`Copyright © 2005 by Pearson Education, Inc.
`All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or trans-
`mitted, in any form, or by any means, electronic, mechanical, photocopying, recording, or otherwise,
`without the prior consent of the publisher. Printed in the United States of America. Published simultane-
`ously in Canada.
`For information on obtaining permission for use of material from this work, please submit a written
`request to:
`Pearson Education, Inc.
`Rights and Contracts Department
`75 Arlington Street, Suite 300
`Boston, MA 02116
`Fax: (617) 848-7047
`ISBN 0-201-75867-9
`Text printed on recycled paper
`1 2 3 4 5 6 7 8 9 10—CRS—08070605040
`First printing, July 2004
`
`1234567890123456789012345678901234567890
`
`Supercell
`Exhibit 1009
`Page 4
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page v
`
`For Dawn, Drew, and Caroline, my true joys on this earth.
`Doug
`
`To my family, for their faith, affection, and support.
`Ernst
`
`To my family with love—they are my life.
`Joe
`
`To my parents for bringing me up.
`Ivan
`
`Supercell
`Exhibit 1009
`Page 5
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page vi
`
`Supercell
`Exhibit 1009
`Page 6
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page vii
`
`Contents
`Contents
`
`Foreword
`Preface
`
`xv
`xix
`
`1
`
`3
`
`FOUNDATIONS OF 3D USER INTERFACES
`Introduction to 3D User Interfaces
`1.1. What Are 3D User Interfaces?
`1.2. Why 3D User Interfaces?
`4
`1.3. Terminology
`6
`1.4. Application Areas
`1.5. Conclusion
`9
`
`8
`
`3
`
`3D User Interfaces: History and Roadmap
`2.1. History of 3D UIs
`11
`14
`2.2. Roadmap to 3D UIs
`2.2.1. Areas Informing the Design of 3D UIs
`2.2.2. 3D UI Subareas
`18
`2.2.3. Areas Impacted by 3D UIs
`2.3. Scope of This Book
`25
`2.4. Conclusion
`26
`
`22
`
`11
`
`15
`
`PART I
`Chapter 1
`
`Chapter 2
`
`PART II
`Chapter 3
`
`30
`
`HARDWARE TECHNOLOGIES FOR 3D USER INTERFACES
`3D User Interface Output Hardware
`29
`3.1. Introduction
`29
`3.1.1. Chapter Roadmap
`3.2. Visual Displays
`31
`3.2.1. Visual Display Characteristics
`3.2.2. Depth Cues
`34
`3.2.3. Visual Display Device Types
`
`31
`
`40
`
`27
`
`vii
`
`Supercell
`Exhibit 1009
`Page 7
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page viii
`
`viii
`
`Contents
`
`59
`
`64
`
`59
`3.3. Auditory Displays
`3.3.1. 3D Sound Localization Cues
`3.3.2. 3D Sound Generation
`62
`3.3.3. Sound System Configurations
`3.3.4. Audio in 3D Interfaces
`66
`3.4. Haptic Displays
`68
`68
`3.4.1. Haptic Cues
`3.4.2. Haptic Display Characteristics
`3.4.3. Haptic Display Types
`71
`77
`3.4.4. Haptic Displays in 3D Interfaces
`3.5. Design Guidelines: Choosing Output Devices for
`3D Interfaces
`77
`3.6. Conclusion
`83
`
`70
`
`87
`
`88
`
`91
`
`3D User Interface Input Hardware
`4.1. Introduction
`87
`4.1.1. Input Device Characteristics
`4.1.2. Chapter Roadmap
`89
`4.2. Desktop Input Devices
`90
`4.2.1. Keyboards
`91
`4.2.2. 2D Mice and Trackballs
`4.2.3. Pen-Based Tablets
`92
`4.2.4. Joysticks
`93
`4.2.5. Six-DOF Input Devices for the Desktop
`4.3. Tracking Devices
`96
`4.3.1. Motion Tracking
`4.3.2. Eye Tracking
`4.3.3. Data Gloves
`4.4. 3D Mice
`110
`4.4.1. Handheld 3D Mice
`4.4.2. User-Worn 3D Mice
`4.5. Special Purpose Input Devices
`4.6. Direct Human Input
`118
`4.6.1. Speech Input
`119
`4.6.2. Bioelectric Input
`4.6.3. Brain Input
`120
`122
`4.7. Home-Brewed Input Devices
`122
`4.7.1. Strategies for Building Input Devices
`4.7.2. Connecting the Home-Brewed Input Device to the
`Computer
`124
`
`95
`
`96
`
`105
`106
`
`111
`113
`
`114
`
`120
`
`Chapter 4
`
`1234567890123456789012345678901234567890
`
`Supercell
`Exhibit 1009
`Page 8
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page ix
`
`Contents
`
`PART III
`Chapter 5
`
`Chapter 6
`
`4.8. Choosing Input Devices for 3D Interfaces
`4.8.1. Important Considerations
`126
`4.8.2. Input Device Taxonomies
`128
`4.8.3. Empirical Evaluations
`132
`
`126
`
`ix
`
`135
`139
`
`3D INTERACTION TECHNIQUES
`Selection and Manipulation
`5.1. Introduction
`139
`140
`5.1.1. Chapter Roadmap
`140
`5.2. 3D Manipulation Tasks
`141
`5.2.1. Canonical Manipulation Tasks
`143
`5.2.2. Application-Specific Manipulation Tasks
`143
`5.3. Manipulation Techniques and Input Devices
`5.3.1. Control Dimensions and Integrated Control in
`3D Manipulation
`144
`145
`5.3.2. Force versus Position Control
`5.3.3. Device Placement and Form-Factor in
`3D Manipulation
`145
`147
`5.4. Interaction Techniques for 3D Manipulation
`5.4.1. Classifications of Manipulation Techniques
`5.4.2. Interacting by Pointing
`150
`5.4.3. Direct Manipulation: Virtual Hand Techniques
`5.4.4. World-in-Miniature
`162
`163
`5.4.5. Combining Techniques
`5.4.6. Nonisomorphic 3D Rotation
`5.4.7. Desktop 3D Manipulation
`5.5. Design Guidelines
`179
`
`147
`
`158
`
`168
`171
`
`184
`
`183
`Travel
`183
`6.1. Introduction
`6.1.1. Chapter Roadmap
`6.2. 3D Travel Tasks
`184
`6.2.1. Exploration
`185
`6.2.2. Search
`185
`186
`6.2.3. Maneuvering
`6.2.4. Additional Travel Task Characteristics
`6.3. Travel Techniques
`188
`188
`6.3.1. Technique Classifications
`6.3.2. Physical Locomotion Techniques
`6.3.3. Steering Techniques
`199
`
`192
`
`187
`
`Supercell
`Exhibit 1009
`Page 9
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page x
`
`x
`
`Contents
`
`206
`6.3.4. Route-Planning Techniques
`210
`6.3.5. Target-Based Techniques
`6.3.6. Manual Manipulation Techniques
`6.3.7. Travel-by-Scaling Techniques
`216
`217
`6.3.8. Viewpoint Orientation Techniques
`219
`6.3.9. Velocity Specification Techniques
`6.3.10. Integrated Camera Controls for Desktop 3D
`Environments
`220
`6.4. Design Guidelines
`222
`
`214
`
`Chapter 7
`
`Chapter 8
`
`227
`Wayfinding
`227
`7.1. Introduction
`229
`7.1.1. Chapter Roadmap
`229
`7.2. Theoretical Foundations
`231
`7.2.1. Wayfinding Tasks
`231
`7.2.2. Types of Spatial Knowledge
`7.2.3. Egocentric and Exocentric Reference Frames
`7.3. User-Centered Wayfinding Support
`234
`7.3.1. Field of View
`235
`7.3.2. Motion Cues
`235
`7.3.3. Multisensory Output
`7.3.4. Presence
`237
`237
`7.3.5. Search Strategies
`7.4. Environment-Centered Wayfinding Support
`7.4.1. Environment Design
`239
`7.4.2. Artificial Cues
`242
`7.5. Evaluating Wayfinding Aids
`7.6. Design Guidelines
`251
`7.7. Conclusions
`253
`
`236
`
`250
`
`232
`
`238
`
`255
`System Control
`255
`8.1. Introduction
`8.1.1. Human Factors of System Control
`8.1.2. Input Devices
`257
`8.1.3. System- and Application-Level Factors
`8.1.4. Chapter Roadmap
`258
`8.2. Classification
`259
`260
`8.3. Graphical Menus
`260
`8.3.1. Techniques
`8.3.2. Design and Implementation Issues
`8.3.3. Practical Application
`267
`
`257
`
`258
`
`265
`
`1234567890123456789012345678901234567890
`
`Supercell
`Exhibit 1009
`Page 10
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page xi
`
`Contents
`
`xi
`
`268
`8.4. Voice Commands
`268
`8.4.1. Techniques
`8.4.2. Design and Implementation Issues
`8.4.3. Practical Application
`269
`8.5. Gestural Commands
`270
`271
`8.5.1. Techniques
`8.5.2. Design and Implementation Issues
`8.5.3. Practical Application
`273
`8.6. Tools
`273
`274
`8.6.1. Techniques
`8.6.2. Design and Implementation Issues
`8.6.3. Practical Application
`277
`8.7. Multimodal System Control Techniques
`8.8. Design Guidelines
`280
`8.9. Case Study: Mixing System Control Methods
`8.9.1. The ProViT Application
`282
`8.9.2. System Control Design Approach for ProViT
`8.9.3. Mapping of Tasks to Devices
`283
`8.9.4. Placement of System Control
`284
`8.9.5. System Control Feedback
`8.10. Conclusions
`285
`
`268
`
`272
`
`276
`
`278
`
`282
`
`284
`
`Chapter 9
`
`288
`
`287
`Symbolic Input
`287
`9.1. Introduction
`9.1.1. Why Is Symbolic Input Important?
`9.1.2. Scenarios of Use
`288
`290
`9.1.3. Brief History of Symbolic Input
`9.1.4. Distinctive Features of Symbolic Input in 3D UIs
`9.1.5. Chapter Roadmap
`292
`9.2. Symbolic Input Tasks
`293
`293
`9.2.1. Alphanumeric Input
`9.2.2. Editing Alphanumeric Symbols
`9.2.3. Markup Input
`294
`294
`9.3. Symbolic Input Techniques
`9.3.1. Keyboard-Based Techniques
`9.3.2. Pen-Based Techniques
`300
`9.3.3. Gesture-Based Techniques
`9.3.4. Speech-Based Techniques
`9.4. Design Guidelines
`306
`9.5. Beyond Number and Text Entry
`
`293
`
`294
`
`303
`304
`
`310
`
`283
`
`291
`
`Supercell
`Exhibit 1009
`Page 11
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page xii
`
`Contents
`
`311
`
`DESIGNING AND DEVELOPING 3D USER INTERFACES
`Strategies for Designing and Developing 3D User
`Interfaces
`313
`313
`10.1. Introduction
`314
`10.1.1. Designing for Humans
`10.1.2. Inventing 3D User Interfaces
`10.1.3. Chapter Roadmap
`315
`10.2. Designing for Humans
`315
`10.2.1. Feedback in 3D User Interfaces
`10.2.2. Constraints
`322
`323
`10.2.3. Two-Handed Control
`10.2.4. Designing for Different User Groups
`10.2.5. Designing for User Comfort
`328
`10.3. Inventing 3D User Interfaces
`330
`10.3.1. Borrowing from the Real World
`10.3.2. Adapting from 2D User Interfaces
`10.3.3. Magic and Aesthetics
`340
`10.4. Design Guidelines
`345
`
`314
`
`315
`
`327
`
`331
`335
`
`349
`
`350
`
`351
`
`Evaluation of 3D User Interfaces
`11.1. Introduction
`349
`11.1.1. Purposes of Evaluation
`11.1.2. Terminology
`351
`11.1.3. Chapter Roadmap
`11.2. Background
`351
`11.2.1. Tools for Evaluation Design and Implementation
`11.2.2. Evaluation Methods Used for 3D Interfaces
`354
`11.3. Evaluation Metrics for 3D Interfaces
`357
`11.3.1. System Performance Metrics
`357
`11.3.2. Task Performance Metrics
`357
`11.3.3. User Preference Metrics
`358
`11.4. Distinctive Characteristics of 3D Interface Evaluation
`11.4.1. Physical Environment Issues
`360
`11.4.2. Evaluator Issues
`362
`11.4.3. User Issues
`363
`365
`11.4.4. Evaluation Type Issues
`366
`11.4.5. Miscellaneous Issues
`11.5. Classification of 3D Evaluation Methods
`11.6. Two Multimethod Approaches
`369
`11.6.1. Testbed Evaluation Approach
`
`352
`
`360
`
`367
`
`369
`
`xii
`
`PART IV
`Chapter 10
`
`Chapter 11
`
`1234567890123456789012345678901234567890
`
`Supercell
`Exhibit 1009
`Page 12
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page xiii
`
`Contents
`
`xiii
`
`11.6.2. Sequential Evaluation Approach
`11.6.3. Comparison of Approaches
`378
`11.7. Guidelines for 3D Interface Evaluation
`11.7.1. General Guidelines
`382
`11.7.2. Guidelines for Formal Experimentation
`
`382
`
`374
`
`383
`
`PART V
`Chapter 12
`
`385
`THE FUTURE OF 3D USER INTERFACES
`Beyond Virtual: 3D User Interfaces for the Real World
`12.1. Introduction
`387
`389
`12.1.1. What Is Augmented Reality?
`12.1.2. Bringing Virtual Interfaces into the Real World
`12.1.3. Chapter Roadmap
`391
`391
`12.2. AR Interfaces as 3D Data Browsers
`394
`12.3. 3D Augmented Reality Interfaces
`12.4. Augmented Surfaces and Tangible Interfaces
`12.5. Tangible AR Interfaces
`397
`398
`12.5.1. Design of Tangible AR
`12.5.2. Time-Multiplexed Interaction in Tangible AR
`12.5.3. Advantages and Disadvantages of Tangible AR
`12.6. Agents in AR
`403
`12.7. Transitional AR: VR Interfaces
`12.8. Conclusions
`405
`
`395
`
`404
`
`387
`
`390
`
`400
`402
`
`Chapter 13
`
`407
`The Future of 3D User Interfaces
`407
`13.1. Questions About 3D UI Technology
`410
`13.2. Questions About 3D Interaction Techniques
`13.3. Questions About 3D UI Design and Development
`13.4. Questions About 3D UI Evaluation
`415
`13.5. Million-Dollar Questions
`416
`
`412
`
`Appendix A Quick Reference Guide to 3D User Interface Mathematics
`A.1. Scalars
`420
`A.2. Vectors
`420
`A.3. Points
`421
`A.4. Matrices
`422
`A.5. Quaternions
`
`424
`
`419
`
`Bibliography
`Index
`457
`
`429
`
`Supercell
`Exhibit 1009
`Page 13
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page xiv
`
`Supercell
`Exhibit 1009
`Page 14
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page xv
`
`
`
`ForForeworewordd
`
`Three-dimensional user interfaces are finally receiving their due! Re-
`search in 3D interaction and 3D display began in the 1960s, pioneered by
`researchers like Ivan Sutherland, Bob Sproull, Fred Brooks, Andrew
`Ortony, and Richard Feldman. While many commercially successful 3D
`applications exist—computer-aided design and simulation, radiation
`therapy, drug discovery, surgical simulation, scientific and information
`visualization, entertainment—no author or group of authors has written
`a comprehensive and authoritative text on the subject, despite a continu-
`ing and rich set of research findings, prototype systems, and products.
`Why is that? Why is it that this book by Doug Bowman, Ernst Kruijff,
`Joe LaViola, and Ivan Poupyrev is the first thorough treatment of 3D UIs?
`Perhaps it was our digression during the last 20 years to the WIMP
`GUI. After all, the Windows, Icons, Menus, and Pointers GUI is used
`very widely by millions of users. Mac OS and Microsoft Windows users
`know it well, as do many UNIX users. Indeed, every user of the Web
`works with a GUI, and this year there are many hundreds of millions of
`them. Two-dimensional GUIs will be with us for a long time. After all, a
`lot of the workaday world with which we deal is flat—not just our Web
`pages but our documents, presentations, and spreadsheets too. Yes,
`some of these can be extended to 3D, but most of the time, 2D is just fine,
`thank you very much. Furthermore, pointing and selecting and typing
`are relatively fast and relatively error-free—they work, and they work well.
`
`xv
`
`Supercell
`Exhibit 1009
`Page 15
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page xvi
`
`xvi
`
`Foreword
`
`Perhaps it is that not as many people use 3D GUIs as use the 2D
`WIMP GUI, and so they are not thought to be as important. But the above
`list of 3D applications involves multibillion-dollar manufacturing indus-
`tries, such as aerospace and automotive, and equally large and even more
`important activities in the life-saving and life-giving pharmaceutical and
`health care industries.
`Perhaps it was that we needed the particular set of backgrounds that
`Doug, Joe, Ivan, and Ernst bring to the table. Doug comes out of the GVU
`Center at Georgia Tech, where he worked on 3D UIs with Larry Hodges
`and others and learned the value of careful user studies and experimen-
`tation, and he is now a member of an influential HCI group at Virginia
`Tech; Joe works at Brown with Andy van Dam, a long-time proponent of
`rich 3D interaction; Ivan comes from the HIT Lab at the University of
`Washington, where he worked with Tom Furness and Suzanne Weghorst,
`and now works with Jun Rekimoto at Sony CSL; and Ernst works with
`Martin Goebel in the VE Group at Fraunhofer IMK in Germany.
`Whatever the case, I am excited and pleased that this team has given
`us the benefit of their research and experience. As I reviewed the draft
`manuscript for this book, I jotted down some of the thoughts that came
`to my mind: comprehensive, encyclopedic, authoritative, taxonomic;
`grounded in the psychological, HCI, human factors, and computer
`graphics literature; grounded in the personal research experiences of the
`authors, their teachers, and their students.
`I myself have long preached the importance of integrating the study
`of the computer with the study of the human. Indeed, this is the key
`premise on which I built the GVU Center at Georgia Tech. This book
`certainly follows that admonition. There are numerous discussions of
`human issues as they relate to 3D navigation and interaction, drawing on
`references in psychology and human factors.
`This is indeed a book for both practitioners and researchers. The ex-
`tensive literature reviews, examples, and guidelines help us understand
`what to do now. Combined with the research agenda in Chapter 13, “The
`Future of 3D User Interfaces,” the material also helps us have a sense of
`what it is that we do not yet know.
`I particularly commend to the readers the Chapter 11 discussion of
`evaluating 3D UIs. We in the computer graphics community have tended
`to design devices and techniques and then “throw them over the wall” to
`the user community. This is not the route to success. Careful study of user
`needs coupled with evaluation as part of the ongoing design cycle is
`much more likely to lead to effective techniques. The authors, all of
`
`1234567890123456789012345678901234567890
`
`Supercell
`Exhibit 1009
`Page 16
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page xvii
`
`Foreword
`
`xvii
`
`whom have grappled with the difficult task of designing 3D interfaces,
`know from first-hand experience how crucial this is. Their section 11.4, on
`the distinctive characteristics of the 3D interface evaluation process, is a
`wonderful codification of that first-hand knowledge.
`Thanks to Doug and Ernst and Joe and Ivan!
`
`Jim Foley
`GVU Center
`College of Computing
`Georgia Tech
`March 2004
`
`Supercell
`Exhibit 1009
`Page 17
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page xviii
`
`Supercell
`Exhibit 1009
`Page 18
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page xix
`
`
`
`PrPrefaceeface
`
`An architect sits in her home office, putting the final touches on the de-
`sign of the new entrance to the city park. A three-dimensional virtual
`model of the park appears in front of her on the desk’s surface. She
`nudges a pathway slightly to the right to avoid a low-lying area, and
`then makes the model life-size so she can walk along the path to view the
`effect. “Those dark colors on the sign at the entrance are too foreboding,”
`she thinks, so she quickly changes the color palette to brighter primary
`colors. She looks up and notices that the clients are arriving for the final
`design review meeting. They are located in other offices around the city,
`but they can all view the 3D model and make suggested changes, as well
`as communicate with one another. “What’s the construction plan?” asks
`one of the clients. The architect starts an animation showing the progress
`of the project from start to finish. “That first step may not work,” says the
`client. “The excavation is much too close to the existing playground. Let
`me show you.” He looks out his window, which has a view of the park,
`and overlays the virtual construction plan on it. “You’re right,” says the
`architect, “let’s plan to move the playground slightly—that will be much
`cheaper than changing the construction site.” After viewing the effects of
`the change, all agree that this plan will work, and the meeting adjourns.
`This scenario and others like it illustrate the enormous potential of
`3D environments and applications. The technology to realize such a vision
`is available now, although it will certainly be improved. But the scenario
`
`xix
`
`Supercell
`Exhibit 1009
`Page 19
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page xx
`
`xx
`
`Preface
`
`also leaves out a great deal of information—information that is crucial to
`making this dream a reality. How did the architect load the park model,
`and how does she manipulate her view of it? What technique is used to
`change the pathway? How can multiple clients all manipulate the model
`at the same time? How do the participants appear to each other in the vir-
`tual space? How is the speed and playback of the animation controlled?
`How did the client instruct the system to merge the real and virtual
`scenes?
`These questions all relate to the design of the user interface (UI) and
`interaction techniques for this 3D application, an area that is usually given
`only a cursory treatment in futuristic films and books. The scenarios usu-
`ally either assume that all interaction between the user and the system
`will be “natural”—based on techniques like intuitive gestures and speech—
`or “automatic”—the system will be so intelligent that it will deduce the
`user’s intentions. But is this type of interaction realistic, or even desirable?
`This book addresses the critical area of 3D UI design—a field that
`seeks to answer detailed questions, like those above, that make the differ-
`ence between a 3D system that is usable and efficient and one that causes
`user frustration, errors, and even physical discomfort. We present practi-
`cal information for developers, the latest research results, easy-to-follow
`guidelines for the UI designer, and relevant application examples. While
`there are quite a few books devoted to UIs in general and to 2D UI design
`in particular, 3D UIs have received significantly less attention. The results
`of work in the field are scattered throughout numerous conference pro-
`ceedings, journal articles, single book chapters, and Web sites. This field
`deserves a reference and educational text that integrates the best prac-
`tices and state-of-the-art research, and that’s why this book was created.
`
`How This Book Came to Be
`
`The story of this book begins in April 1998, when Ivan Poupyrev and
`Doug Bowman were doctoral students at Hiroshima University and
`Georgia Tech respectively, working on 3D interaction techniques for ob-
`ject manipulation in virtual environments (VEs). We started a lively email
`discussion about the design and usability of these techniques and about
`3D UIs in general. Ivan, who was at the time a visiting research student at
`the University of Washington, suggested that the discussion would be
`even more profitable if other researchers in this new area could join in as
`
`1234567890123456789012345678901234567890
`
`Supercell
`Exhibit 1009
`Page 20
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page xxi
`
`Preface
`
`xxi
`
`well, and so the 3DUI mailing list was born. Since that time, over 100 re-
`searchers from around the globe have joined the list and participated in
`the discussion (to see an archive of all the list traffic or to join the list,
`check out http://www.3dui.org). Joe LaViola and Ernst Kruijff were two of
`the first people to join the list.
`In August of that same year, Doug forwarded to the list a call for tuto-
`rials for the upcoming IEEE Virtual Reality Conference. After some dis-
`cussion, Joe, Ivan, and Ernst agreed to join Doug to organize a tutorial on
`“The Art and Science of 3D Interaction.” The tutorial was a big hit at the
`conference in Houston, and the four of us continued to present courses
`on the topic at ACM Virtual Reality Software and Technology 1999, IEEE
`VR 2000, and ACM SIGGRAPH 2000 and 2001.
`After developing a huge amount of content for the notes supple-
`ments of these courses, we decided it would be silly not to compile and
`expand all of this information in book form. Furthermore, there was no
`way to include all the information available on 3D UIs in a one-day
`course. And that’s why you’re holding this book in your hands today—a
`book containing information on 3D UIs that can’t be found in any other
`single source.
`
`What’s in the Book
`
`The title of this book emphasizes that we have written it for both
`academics/researchers and practitioners/developers; both those inter-
`ested in basic research and those interested in applications. Most chapters
`of the book integrate both theory and practical information. We intend
`the book to be used both as a textbook (see suggestions below) and as a
`reference work.
`Theory-related content includes the following:
`
`• sections on the psychology and human factors of various 3D
`interaction tasks
`• information on different approaches for the evaluation of 3D UIs
`(Chapter 11)
`• results from empirical studies of 3D interaction techniques
`• a research agenda for 3D interaction (Chapter 13)
`• lists of recommended further reading at the end of most chapters
`• a comprehensive bibliography of important research articles
`
`Supercell
`Exhibit 1009
`Page 21
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page xxii
`
`xxii
`
`Preface
`
`Practice-related content includes the following:
`
`• principles for choosing appropriate input and output devices for
`3D systems (Chapters 3 and 4)
`• details and helpful tips for the implementation of common 3D in-
`teraction techniques
`• guidelines for the selection of interaction techniques for common
`3D tasks
`• case studies of 3D UIs in real-world applications
`
`The book is organized into five parts. Part I introduces the topic of 3D
`UIs. Part II discusses the input and output device technology used in the
`development of 3D UIs, with an emphasis on the impact of these devices
`on usability and performance. Part III presents a wide range of 3D inter-
`action techniques for the common tasks of navigation, selection and ma-
`nipulation, system control, and symbolic input. In Part IV, we discuss the
`design, development, and evaluation of complete 3D UI metaphors and
`applications. Finally, Part V considers the future, with chapters on 3D in-
`teraction in augmented reality applications and a research agenda for 3D
`UIs. The appendices include information on required mathematical back-
`ground and a bibliography of 3D UI references.
`Throughout the book, we offer several special features. First, most
`chapters contain numerous guidelines—practical and proven advice for
`the designer and developer. Guidelines are indicated in the text like this:
`
`Follow the guidelines in this book to help you design usable
`3D UIs.
`
`We also include implementation details for many of the most common
`and useful interaction techniques. We describe these algorithms using a
`combination of textual description and mathematical notation (to avoid a
`bias toward any particular development tool or programming style).
`
`How to Use the Book and Related Material
`
`If you are a 3D UI developer: Professional developers can use the book for
`inspiration and guidance in the design, implementation, and evaluation
`of applications with 3D UIs. In the design process, developers can consider
`
`1234567890123456789012345678901234567890
`
`Supercell
`Exhibit 1009
`Page 22
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page xxiii
`
`Preface
`
`xxiii
`
`overall UI metaphors from Part IV, choose specific interaction techniques
`from Part III, and match these with appropriate input and display de-
`vices from Part II. The design guidelines from all of these sections should
`help developers make rational, informed decisions. The implementation
`of the 3D UI can benefit from the textual and mathematical descriptions
`of interaction techniques we provide in Part III. Finally, developers can
`choose evaluation methods and assess the usability of their applications
`based on the information in Chapter 11.
`If you are a teacher: The book can also be used as a textbook in several
`different types of university-level courses. A graduate course on 3D UI
`design could use it as a primary textbook. A more generic virtual envi-
`ronments course could use Parts I, II, and III of this book as an introduc-
`tion to the basic technology and techniques used in VE interaction. An
`undergraduate HCI course could pull information from parts I and IV in
`a module on 3D interfaces and their differences from traditional UIs. Im-
`plementation of common techniques from Part III could enhance a course
`on interactive 3D graphics.
`If you are a researcher: This book can serve as a comprehensive refer-
`ence guide for researchers engaged in 3D UI design or evaluation, the in-
`vestigation of 3D applications, or the use of VEs or augmented reality.
`The research agenda in Chapter 13 also provides researchers and research
`students with a list of important questions to be addressed in the field. It
`could even be used as the starting point for a Ph.D. student looking for a
`topic related to 3D UIs.
`3D UI design is a fast-moving and evolving field. Therefore, we are
`committed to updating the material in this book. One way we will do this
`is through the book’s official Web site at http://www3dui.org. This site will
`contain information and links related to the latest 3D UI research and ap-
`plications, organized in the same manner as the book so you can easily
`find new information about the topics in a particular part or chapter. The
`site will also allow you to join the 3DUI mailing list. We also ask for your
`help in keeping the book up to date. Send us your comments, clarification
`questions, or links to additional information by visiting the web site
`above and using the online feedback form. Or email us directly at
`3dui@3dui.org. Your comments will help us update the Web site as well
`as future editions of this book.
`
`Supercell
`Exhibit 1009
`Page 23
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page xxiv
`
`xxiv
`
`Acknowledgments
`
`Preface
`
`This book would not have been possible without the hard work, support,
`and intelligence of a large group of people.
`First, we offer our gratitude to the reviewers who gave their time and
`energy in improving the quality of the book. Their comments and sug-
`gestions have made the book more complete, more readable, and more
`useful. Thanks to Ben Shneiderman, Harry Hersh, D. Jay Newman, Jeff
`Pierce, Dieter Schmalstieg, and Bob Zeleznik for providing this invalu-
`able service.
`Next, we would like to thank our editor at Addison-Wesley, Peter
`Gordon, for his invaluable advice and encouragement. The rest of the
`staff, including Bernie Gaffney, Amy Fleischer, Julie Nahil, Heather Mul-
`lane, and Curt Johnson has also been extremely helpful. Thanks also to
`Simone Payment and Carol Lallier for their competent and professional
`work during the production phase.
`All of us would like to personally thank our colleagues in the 3D UI
`community for their fruitful discussions and collaborations. They include
`Mark Mine, Robert Lindeman, Matthew Conway, Ken Hinckley, Shumin
`Zhai, Kiyoshi Kiyokawa, Chris Shaw, Mark Billinghurst, Rudy Darken,
`Pablo Figueroa, and Bernd Fröhlich.
`Portions of this material are based upon work supported by the Na-
`tional Science Foundation under Grants No. DUE-0127326 and IIS-
`0237412. Any opinions, findings and conclusions, or recommendations
`expressed in this material are those of the author(s) and do not necessar-
`ily reflect the views of the National Science Foundation (NSF).
`Doug Bowman: I would like to thank my wife, Dawn, for her unfail-
`ing love and support, as well as my extended family and friends, espe-
`cially those at Grace Covenant Presbyterian Church. Much gratitude is
`due to Joe, Ivan, and Ernst for seeing this project through and for all their
`years of friendship and collaboration. Thanks also go to my colleagues
`and students at Virginia Tech, including Chris North, Ron Kriz, Mehdi
`Setareh, Walid Thabet, Thomas Ollendick, David Cox. Debby Hix, Joe
`Gabbard, Chad Wingrave, Jian Chen, Nicholas Polys, Wendy Schafer,
`and Marcio Pinho. Past colleagues at Georgia Tech also deserve thanks.
`They include Larry Hodges, John Kelso, Drew Kessler, David Koller,
`Donald Johnson, Donald Allison, Brian Wills, Jean Wineman, Jay Bolter,
`Elizabeth Davis, Albert Badre, and Ben Watson.
`Ernst Kruijff: First of all, my thanks go to Doug, Ivan, and Joe for their
`great cooperation, help, and extensive discussions. Thanks also go to my
`
`1234567890123456789012345678901234567890
`
`Supercell
`Exhibit 1009
`Page 24
`
`

`

`30706 00 ppi-xxvi r2jm.ps 5/13/04 5:50 PM Page xxv
`
`Preface
`
`xxv
`
`parents, my brother, and my sister-in-law for their support, each in their
`own way. Furthermore, my thanks go to all of my colleagues at the VE
`group of IMK, especially Martin Goebel for being a great chief, Gerold
`Wesche, Andreas Simon, Gernot Goebbels, Stefan Conrad, Aeldrik Pan-
`der, and Steffi Beckhaus for their past and current cooperation and help
`during the making of this book and its instigators, the courses we (the au-
`thors) gave together. My thanks also go to my past colleagues at Bauhaus
`University, especially the members of igroup, in particular Holger Regen-
`brecht. Furthermore, thanks to all the students who he