Pressure-Sensitive Pen Interactions
`
`by
`
`Gonzalo Alberto Ramos
`
`A thesis submitted in conformity with the requirements
`for the degree of Doctor of Philosophy
`
`Graduate Department of Computer Science
`University of Toronto
`
`Copyright © by Gonzalo A. Ramos 2008
`
`APPLE 1008
`
`1
`
`

`

`Pressure-Sensitive Pen Interactions
`
`Gonzalo Alberto Ramos
`
`Doctor of Philosophy
`
`Graduate Department of Computer Science
`University of Toronto
`
`2008
`
`
`Abstract
`
`
`Pen-based computers bring the promise of tapping into people’s expressiveness with pen and
`
`paper and producing a platform that feels familiar while providing new functionalities only
`
`possible within an electronic medium. To this day, pen computers’ success is marginal be-
`
`cause their interfaces mainly replicate keyboard and mouse ones. Maximizing the potential of
`
`pen computers requires redesigning their interfaces so that they are sensitive to the pen’s input
`
`modalities and expressiveness. In particular, pressure is an important and expressive, yet un-
`
`derutilized, pen input modality.
`
`This dissertation advances our knowledge about pressure-aware, pen-based interactions and
`
`how people use these techniques. We systematically explore their design by first investigating
`
`how pressure can affect pen interactions. We propose novel techniques that take advantage of
`
`the pressure modality of a pen to control, link, and annotate digital video.
`
`We then study people’s performance using pressure to navigate through a set of elements and
`
`find that they can discriminate a minimum of six different pressure regions. We introduce the
`
`concept of Pressure Widgets and suggest visual and interaction properties for their design.
`
`
`
`ii
`
`2
`
`

`

`We later explore pressure’s use to enhance the adjustment of continuous parameters and pro-
`
`pose Zliding, a technique in which users vary pressure to adjust the scale of the parameter
`
`space, while sliding their pen to perform parameter manipulations. We study Zliding and find
`
`it a viable technique, which is capable of enabling arbitrarily precise parameter adjustments.
`
` We finally present a novel interaction technique defined by the concurrent variation in pres-
`
`sure applied while dragging a pen. We study these pressure marks and find that they are a
`
`compact, orientation-independent, full interaction phrase that can be 30% faster than a state-
`
`of-the-art selection-action interaction phrase.
`
`This dissertation also makes a number of key contributions throughout the design and study
`
`of novel interaction techniques:
`
`It identifies important design issues for the development of pressure-sensitive, pen-
`operated widgets and interactions,
`
`It provides design guidelines for interaction techniques and interface elements utilizing
`pressure-enabled input devices,
`
`It presents empirical data on people’s ability to control pressure, and
`
`It charts a visual design space of pressure-sensitive, pen-based interactions.
`
`iii
`
`(cid:131)
`
`(cid:131)
`
`(cid:131)
`
`(cid:131)
`
`
`
`3
`
`

`

`Acknowledgements
`
` I
`
` am deeply grateful to many people whom have been by my side all these years. Firstly, I am
`
`grateful to my committee members, a fantastic blend that honored me by lending their exper-
`
`tise, candid advice and support. Professor Ronald Baecker provided me with an invaluable
`
`pool of experience on seminal aspects of HCI and best practices of on User-Centered Design.
`
`I remain in awe of his enthusiasm for the field and for his ability to defuse, single-handed,
`
`tense situations with hotel officials while securing a whole parking lot to have an after-hours
`
`party. Professor Mark Chignell has inspired me to seek rigor in experiment design and analy-
`
`sis, and has been an incredible resource in areas of Human Factors. I regret not seeking out to
`
`his generous Kiwi mentorship more often. Professor Karan Signh’s nimble intellect and
`
`knowledge has helped me to find and fill gaps of my own. His talent to combine effortlessly
`
`art and science has been a source of inspiration that will stay always with me. He is a man
`
`truly outside of time, since all the clocks in his house give a different time. My advisor, Pro-
`
`fessor Ravin Balakrishnan has constantly challenged both my intellect and tolerance for cap-
`
`saicin and saturated oils: good scientific knowledge comes from constantly questioning what
`
`we think we know, and good beef is better deep-fried, extra spicy and extra crispy. Ravin’s
`
`mentorship has helped me to learn to sail the often-tempestuous waters of academia and al-
`
`lowed me to grown professionally. Thank you, Ravin.
`
`To my circle of UofT friends: You are family. To my family in Argentina: You are my roots
`
`and you travel with me wherever I go. My love for you is beyond measure. To Buenos Aires
`
`“La Reina de Plata”: Being a Porteño is awesome.
`
`Last, but not least, to Rebecca: you make all possible. I love you.
`
`
`
`iv
`
`4
`
`

`

`Table of Contents
`
`Abstract ....................................................................................................................................ii
`Acknowledgements .................................................................................................................iv
`Table of Contents.....................................................................................................................v
`List of Tables............................................................................................................................x
`List of Figures .........................................................................................................................xi
`Introduction .....................................................................................................................1
`1
`1.1
`Pressure as an Additional Input Modality ...................................................................3
`1.2
`Thesis Organization.....................................................................................................4
`1.3
`Research Methods........................................................................................................6
`1.4
`Contributions ...............................................................................................................7
`2 Background ......................................................................................................................9
`2.1
`The Perception of Pressure ..........................................................................................9
`2.1.1
`Cutaneous and Subcutaneous Mechanoreceptors..................................................9
`2.1.2
`Muscle and Skeletal Mechanoreceptors ..............................................................12
`2.1.3
`Psychophysics and the Study of Variations in Signal Intensity ..........................14
`2.1.4
`Implications for Design .......................................................................................16
`2.2
`Pressure Input in the GUI: Prior Art..........................................................................17
`3
`Pen Interactions Incorporating Pressure as an Additional Input Channel:
`
`a Case Study. ..................................................................................................................25
`3.1
`Video as Media..........................................................................................................25
`3.2
`Traditional Video/Film Practices...............................................................................26
`3.3
`Related Systems and Techniques...............................................................................29
`3.3.1
`Pen-based Interaction Systems and Techniques..................................................29
`3.3.2
`Video Annotation and Navigation Systems.........................................................33
`3.4
`Overview and Design Philosophy of LEAN .............................................................36
`3.5
`Gestures, Commands, and Scribbling........................................................................38
`3.6
`Pressure and Pressure Widgets ..................................................................................40
`3.7
`Video Control ............................................................................................................41
`3.7.1
`Position+Velocity Sliders ....................................................................................42
`
`
`
`v
`
`5
`
`

`

`3.7.2
`3.7.3
`3.8
`3.8.1
`3.9
`3.10
`4
`4.1
`4.2
`4.2.1
`4.2.2
`4.3
`4.4
`4.5
`4.6
`4.6.1
`4.6.2
`4.6.3
`4.6.4
`4.6.5
`4.7
`4.8
`4.9
`4.9.1
`4.9.2
`4.10
`4.10.1
`4.10.2
`4.10.3
`4.10.4
`4.10.5
`4.10.6
`
`Twist-Lens Sliders...............................................................................................44
`Video Segments...................................................................................................46
`Annotations and Links...............................................................................................47
`Link Navigation and Manipulation......................................................................50
`Discussion and User Feedback ..................................................................................51
`Summary....................................................................................................................54
`Pressure Widgets ...........................................................................................................55
`Introduction ...............................................................................................................55
`Experiment.................................................................................................................56
`Apparatus.............................................................................................................56
`Task and Stimuli..................................................................................................57
`Participants ................................................................................................................59
`Procedure and Design................................................................................................59
`Performance Measures...............................................................................................62
`Results .......................................................................................................................62
`Selection Techniques...........................................................................................63
`Discernable Number of Pressure Levels..............................................................66
`Effect of Visual Feedback ...................................................................................68
`Conformity with Fitts’ Law.................................................................................69
`Control at Different Pressure Levels ...................................................................70
`Summary....................................................................................................................71
`Implications for Design .............................................................................................73
`Factors for the Design of Pressure Widgets ..............................................................74
`Interference..........................................................................................................75
`Visuals Elements and Behavior...........................................................................76
`A Study on Visual Features and Interference............................................................78
`Apparatus.............................................................................................................81
`Participants ..........................................................................................................81
`Procedure and Design..........................................................................................81
`Performance Measures.........................................................................................82
`Results .................................................................................................................82
`Discussion............................................................................................................86
`vi
`
`
`
`6
`
`

`

`Summary....................................................................................................................87
`4.11
`5 Zooming and Sliding for High-Precision Parameter Manipulation .......................89
`5.1
`Related Work.............................................................................................................90
`5.2
`Motivation and Goals ................................................................................................95
`5.3
`The Zlider ..................................................................................................................96
`5.3.1
`Pressure Cursor....................................................................................................97
`5.3.2
`Integrated Zoom & Slide Control........................................................................98
`5.3.3
`Clutching the Zoom Level.................................................................................100
`5.3.4
`The Selection Mechanism .................................................................................101
`5.3.5
`Scrolling.............................................................................................................105
`5.4
`Alternatives for Decoupled Zoom Control..............................................................106
`5.4.1
`Force Button ......................................................................................................106
`5.4.2
`Discrete Keys.....................................................................................................107
`5.5
`Experiment...............................................................................................................108
`5.5.1
`Apparatus...........................................................................................................108
`5.5.2
`Participants ........................................................................................................110
`5.5.3
`Task and Stimuli................................................................................................110
`5.5.4
`Procedure and Design........................................................................................111
`5.6
`Results .....................................................................................................................113
`5.6.1
`Selection Time...................................................................................................113
`5.6.2
`Crossings ...........................................................................................................115
`5.6.3
`Qualitative Results: User Preferences ...............................................................117
`5.7
`Discussion................................................................................................................118
`5.7.1
`Other Designs: The Zliding Wheel....................................................................119
`5.7.2
`Other Technologies, Other Directions...............................................................121
`5.8
`Summary..................................................................................................................122
`6
`Pressure Marks ............................................................................................................124
`6.1
`Previous Work .........................................................................................................126
`6.2
`Pressure Marking.....................................................................................................129
`6.2.1
`Browsing Through Pressure Marks ...................................................................132
`6.2.2
`Pressure Marks’ Anatomy: Reduction and Parsing...........................................134
`6.3
`User Study #1 ..........................................................................................................138
`vii
`
`
`
`7
`
`

`

`Apparatus...........................................................................................................138
`6.3.1
`Participants ........................................................................................................138
`6.3.2
`Task and Stimuli................................................................................................139
`6.3.3
`Procedure and Design........................................................................................140
`6.3.4
`Results ...............................................................................................................141
`6.3.5
`Summary............................................................................................................145
`6.3.6
`User Study #2 ..........................................................................................................145
`6.4
`Apparatus and Participants ................................................................................146
`6.4.1
`Task and Stimuli................................................................................................146
`6.4.2
`Procedure and Design........................................................................................147
`6.4.3
`Results ...............................................................................................................148
`6.4.4
`On The Use of Pressure Marks................................................................................151
`6.5
`Pressure Marking Menus ...................................................................................152
`6.5.1
`Simple Pressure Marks ......................................................................................153
`6.5.2
`Pressure Tails.....................................................................................................155
`6.5.3
`Pressure Fanning................................................................................................155
`6.5.4
`Summary..................................................................................................................156
`6.6
`7 Conclusions and Future Research..............................................................................158
`7.1
`Design Issues & Guidelines.....................................................................................159
`7.1.1
`Interaction Duration...........................................................................................159
`7.1.2
`Visual Feedback ................................................................................................160
`7.1.3
`Pressure Taming: Noise.....................................................................................160
`7.1.4
`Pressure Taming: Control..................................................................................162
`7.1.5
`Avoiding Task’s Disruptions.............................................................................162
`7.1.6
`Performance Limits in Discrete Pressure Widgets ............................................164
`7.2
`Charting the Design Space of Pressure Widgets .....................................................164
`7.2.1
`An Atom-Inspired Description of Pressure Widgets.........................................166
`7.3
`Future Directions .....................................................................................................168
`7.3.1
`Revisiting Digital Video Interaction..................................................................168
`7.3.2
`Learning Experience..........................................................................................171
`7.3.3
`Pressure as a Measure of Intention ....................................................................172
`7.3.4
`A Concert of Expression Channels....................................................................172
`viii
`
`
`
`8
`
`

`

`Other Devices ....................................................................................................173
`7.3.5
`Place in the Interface Ecology ...........................................................................173
`7.3.6
`Final Remarks..........................................................................................................174
`7.4
`8 Bibliography.................................................................................................................175
`9 Appendix A: Pressure Widgets’ Study Survey Forms ............................................190
`10 Appendix B: Zlider’s Study Survey Form ..............................................................197
`11 Appendix C: Pressure Marks’ Study Survey Form ................................................201
`
`
`
`
`ix
`
`9
`
`

`

`List of Tables
`
`Table 2-1: Main muscular and skeletal proprioceptors ............................................................14
`
`Table 3-1: Gesture grid that shows the basic set of … the LEAN system. ...............................40
`
`Table 4-1: Distribution of outlier samples................................................................................63
`
`Table 4-2: Distribution of outlier samples................................................................................83
`
`Table 4-3: Average number of crossings (NC) per tracking level............................................84
`
`Table 5-1: Distribution of outlier data points across main experimental conditions..............113
`
`Table 7-1: Design matrix for the design of pressure widgets.................................................165
`
`
`
`
`
`x
`
`10
`
`

`

`List of Figures
`
`Figure 1-1: Basic expression channels for a pen. .......................................................................2
`
`Figure 2-1: Cross section of skin showing main cutaneous tactile mechanoreceptors.............11
`
`Figure 2-2: Muscle spindle .......................................................................................................12
`
`Figure 2-3: Golgi organ ............................................................................................................13
`
`Figure 2-4: Generic psychometric function..............................................................................15
`
`Figure 2-5: Reproduction of photographs of Herot and Weinzapfel’s system.........................18
`
`Figure 2-6: Main display for paint program – from (Buxton et al., 1985) ...............................18
`
`Figure 2-7: Spaceball used as a reference isometric device in Zhai's experiments..................19
`
`Figure 2-8: Experimental setup for measuring pressure perception….....................................19
`
`Figure 2-9: Pinch grip used on Bao & Silverman's study.........................................................20
`
`Figure 2-10: Block diagram of the Experimental setup used by Srinivasan and Chen. ...........21
`
`Figure 2-11: Device used by Lecuyer in one of his experiments .............................................21
`
`Figure 2-12: The Geozui3d interface........................................................................................22
`
`Figure 2-13: Examples of force sensing hardware ...................................................................23
`
`Figure 2-14: Examples of force sensing hardware ...................................................................23
`
`Figure 2-15: Pop-through mouse and an its three-state button.................................................24
`
`Figure 3-1: Animator Lynn Smith using the Genesys system..................................................29
`
`Figure 3-2: Screenshots of the Tivoli and the Flatland environment. ......................................30
`
`Figure 3-3: Screenshots of the Electronic Cocktail Napkin and Xlibris prototype ..................31
`
`Figure 3-4: Marking menu technique; Flowmenu widget; and a color toolglass.....................32
`
`Figure 3-5: CrossY system … & Hover widgets......................................................................33
`
`Figure 3-6: SILVER’s video editing interface & Photofinder’s interface................................35
`
`Figure 3-7: Videotater's interface consists of three main regions […].....................................36
`
`Figure 3-8: The LEAN system running on a Tablet PC ...........................................................38
`
`
`
`xi
`
`11
`
`

`

`Figure 3-9: Pressure Widgets....................................................................................................41
`
`Figure 3-10: A typical video player with a VCR-like media control widget ...........................42
`
`Figure 3-11: The PVslider widget and features........................................................................43
`
`Figure 3-12: This partial view of the TLSlider shows […] ......................................................45
`
`Figure 3-13: Twist Lens Slider.................................................................................................46
`
`Figure 3-14: A sequence demonstrating the contextual visualization of an annotation...........49
`
`Figure 3-15: Frames connected to a note are visualized as thumbnails […]............................50
`
`Figure 3-16: An example of a typical session with LEAN........................................................51
`
`Figure 4-1: Visual feedback conditions....................................................................................58
`
`Figure 4-2: Signatures for the selection methods .....................................................................59
`
`Figure 4-3: Subdivisions of pressure space used for the study.................................................61
`
`Figure 4-4: Effect of selection on Mean Time (left), and Error Rate (right). ...........................64
`
`Figure 4-5: Effect of selection on Number of Crossings throughout experimental blocks. .....65
`
`Figure 4-6: Effect of the number of pressure levels on Error Rate […] ..................................67
`
`Figure 4-7: Effect of the number of levelson Number of Crossings […].................................67
`
`Figure 4-8: Effect of the number of levelson Mean Time for the Full Visual condition..........68
`
`Figure 4-9: Linear regression of Index of Difficulty versus Mean Time […].........................70
`
`Figure 4-10: Effect of distance (amplitude) on pressure level on Number of Crossings .........71
`
`Figure 4-11: Example of visual elements on a traditional linear menu....................................77
`
`Figure 4-12: Concept designs for different pressure widgets...................................................78
`
`Figure 4-13: […] behavior of different discrete pressure widget designs................................80
`
`Figure 4-14: Average distance traveled by the pen's tip […] ...................................................83
`
`Figure 4-15: Average % error rate (+/- SE) per widget design. ...............................................84
`
`Figure 4-16: Average Navigation time (seconds +/- SE) per widget. ......................................85
`
`Figure 5-1: Drag and Pop interaction technique.......................................................................91
`
`Figure 5-2: […] window where its buttons are weighted according to their semantics...........92
`
`
`
`xii
`
`12
`
`

`

`Figure 5-3: The Alphaslider […] & the FineSlider widget(s) ..................................................92
`
`Figure 5-4: Diagram of the take-off technique.........................................................................93
`
`Figure 5-5: Precision handle.....................................................................................................93
`
`Figure 5-6: Igarashi and Hinckley's implementaion of speed dependent zooming..................94
`
`Figure 5-7: Zliding on the Zlider widget. .................................................................................97
`
`Figure 5-8: Pressure cursor.......................................................................................................98
`
`Figure 5-9: […] parabolic-sigmoid transfer function used to preprocess the pressure signal..99
`
`Figure 5-10: Effect of the signal’s stabilization and filtering techniques...............................100
`
`Figure 5-11: Clutching the zoom level […] ...........................................................................101
`
`Figure 5-12: Zlider’s state-transition diagram........................................................................103
`
`Figure 5-13: Air Clutching in action. .....................................................................................105
`
`Figure 5-14: Wacom CintiQ interactive display […].............................................................109
`
`Figure 5-15: Phidgets pressure transducer used in our experiment........................................109
`
`Figure 5-16: Elements in the experimental setup. ..................................................................111
`
`Figure 5-17: Average selection time per technique and width...............................................114
`
`Figure 5-18: Average selection time per block and scale-adjusting technique. .....................114
`
`Figure 5-19: Average crossings per distance*width...............................................................115
`
`Figure 5-20: Average crossings per target’s width and scale-adjusting technique. ...............116
`
`Figure 5-21: Average number of crossings per experimental block […]...............................116
`
`Figure 5-22: Zliding Wheels...................................................................................................120
`
`Figure 6-1: A pressure mark is used to select and copy […]..................................................124
`
`Figure 6-2: Flowmenu can integrate selection and direct manipulation […].........................127
`
`Figure 6-3: This image illustrates how a pigtail delimiter […]..............................................127
`
`Figure 6-4: Marquee menu’s four types of commands...........................................................128
`
`Figure 6-5: Profiles of the four proposed pressure signatures................................................131
`
`Figure 6-6: Browsing through pressure marks. ......................................................................133
`
`
`
`xiii
`
`13
`
`

`

`Figure 6-7: Pressure vs. traveled distance. .............................................................................135
`
`Figure 6-8: An example of experimental stimuli […]............................................................139
`
`Figure 6-9: Average errors per block for straight lines and lassos.........................................142
`
`Figure 6-10: Average errors per pressure signature for lines and lassos................................143
`
`Figure 6-11: Average Trial Time per block for lines and lassos. ...........................................143
`
`Figure 6-12: Average Trial Time per pressure signature for lines and lassos........................144
`
`Figure 6-13: Example of an experimental trial […] ...............................................................147
`
`Figure 6-14: Average trial time (+/- SE) per technique and selection type............................149
`
`Figure 6-15: Average trial time per block. Power regression lines are shown.......................150
`
`Figure 6-16: Average error per technique and selection type.................................................150
`
`Figure 6-17: Average error per block per technique. Power regression lines are shown.......151
`
`Figure 6-18: Pressure marking menu design. .........................................................................152
`
`Figure 6-19: Different simple pressure mark flicks................................................................154
`
`Figure 6-20: Ripple feedback. ................................................................................................154
`
`Figure 6-21: Pressure tails example........................................................................................155
`
`Figure 6-22: Example of pressure fanning. ............................................................................156
`
`Figure 6-23: A map annotation in a multi-user scenario […] ................................................157
`
`Figure 6-24: Example of compound pressure marks......