Ulllted States Patent
`
`[19]
`
`[11] Patent Number:
`
`5,617,114
`
`
`Bier et a1.
`[45] Date of Patent:
`Apr. 1, 1997
`
`1|||||||||l|||llllllllllllllllllIllll|||||l|||lllllllllllllllllllllilllllll
`USOOS617114A
`
`[54] USER INTERFACE HAVING
`CLICK-THROUGH TOOLS THAT CAN BE
`~
`COMPOSED WITH OTHER TOOLS
`Inventors: Eric A. Bier, Mountain View, Calif.;
`William A. S. Buxton, Toronto,
`Canada; Maureen C. Stone, Los Altos,
`Calif.
`
`[75]
`
`[73] Assignee: Xerox Corporation, Stamford, Conn.
`
`[21] Appl. No.: 449,584
`_
`Ffled:
`
`[22]
`
`May 24’ 1995
`Related US. Application Data
`
`[63] Continuation of Ser. No. 95,445, Jul. 21, 1993, abandoned.
`
`[51]
`Int. Cl.6 ....................................................... G09G 5/00
`
`[52]
`345/113; 345/146; 345/902
`[58] Field of Search ..................................... 345/146, 145,
`345/113, 115, 1197 120, 902; 395/156, 157;
`364/474.22
`
`[56]
`
`References Cited
`U.S PATENT DOCUMENTS
`'
`.. 340/709
`3/1933 Atkinson -
`Re- 32,632
`
`' gig/$29]
`Z’ggg’gg 1313:: 2111:1888"
`4:686:522
`8/1987 Hemandezuet'ali." 345/146
`4,743,441
`5/1988 Brzezinski _
`4,788,538
`11/1988 Klein et a1.
`............................. 340/747
`
`- 345/113
`4,827,253
`5/1939 Maltz -----
`£1338 3:581:3381
`1:383;
`32:23?
`
`__ 400/489
`4,917,516
`4/1990 Retter .........
`6/1990 Atkinson ,,,,,,,,,,,,
`,, 345/146
`4,931,783
`
`----- 345/113
`479821343
`“1991 HOUIVfiZ 9131-
`gflégzgié 12:83; 32:23:23: """""""""""" 340/706
`5Z250:929
`10/1993 Hoffman et 213.:...................... 345/146
`5,276,797
`1/1994 Bateman et a1.
`.
`5,283,560
`2/1994 Bartlett .................................... 345/902
`5,287,417
`2/1994 Eller et a1.
`.....
`382/41
`
`
` 8/1994 Perlin et al. ............ 395/139
`5,341,466
`....................... 345/156
`5,381,158
`1/1995 Takahara et a1.
`
`FOREIGN PATENT DOCUMENTS
`0533424
`3/1993
`Euro ean Pat. Off.
`.
`62465232
`7,1987
`1apa}:-
`3219324
`9/1991
`Japan‘
`OTHER PUBLICATIONS
`
`Macintosh “Macpaint”, 1983.
`Brown et a1., “Windows on tablets as a means of achieving
`virtual input devices”. In D. Diaper et al. (Eds). Human—
`Computer Interaction—Interact ’90. Amsterdam: Elsevier
`Science Publishers B.V. (North Holland), 675—681 (1990)
`[Reprint of article pp. 1—12 submitted].
`Hardock, Gary. “Design issues for line—driven text editing/
`Annotation Systems”. 11715 Graphics Interface 1991. 3—7
`Jun. 1991, Calgary (1991) Toronto, ON, CA, pp. 77—84.
`,
`_
`(List continued on next page.)
`
`Primmy Examiner—Richard Hierpe
`Assistant Examiner—Regina Liane
`Attamey, Agent, or Firm—Townsend and Townsend and
`Crew
`
`ABSTRACT
`[57]
`A user interface technique operates in the environment of a
`processor-controlled machine for executing a program that
`operates on a set of underlying data and displays a visible
`representation thereof. The system further provides a visual
`depiwon Of a 5“ 0f 10015“ The 10015 incmde CliCk'thmugh
`tools that can be combined with other tools (including other
`click—through tools) to provide composite tools. A click-
`through tool
`includes a generally transparent delineated
`active area that is movable so that it can be placed over a
`desired portion of the visible representation. When the user
`interacts with the visible representation through the active
`area,
`the action takes on an attribute of the particular
`click—through tool. A click-through tool can be superim-
`POSCd on “Other CliCk'mmugh 1001’ WhereuPon 3“ 3°90“
`taken through the two click-through tools onto the visible
`representation takes on attributes of both tools.
`
`28 Claims, 16 Drawing Sheets
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`32
`
` 23
`
`PROCESSOR
`
`15
`
`Y
`
` 25
`
`KEYBOARD 20
`
`MEMORY
`
`FILE STORAGE
`
`40
`
`30
`
`37
`
`(
`
`

`

`5,617,114
`Page 2
`
`OTHER PUBLICATIONS
`
`David Kurlander and Steven Feiner. Interactive Constraint—
`
`Canvas 3.0 User Guide: Chapter 1 (“The Fundamentals”), p.
`2; and Section 19 (Reference), pp. 14—18, 59—60, 69—70,
`90—91, 97—98, 100—102, 130, 150—151, 171—172, 173—174,
`181—182, 189—197, 300 (Jun. 15, 1991 (Deneba Software,
`Miami, FL)).
`“User Interface for Data Entry with Pen Device,” IBM
`Technical Disclosure Bulletin, vol. 37, No. 1, Jan. 1994, pp.
`33—34.
`
`“Menu Manager,” IBM Technical Disclosure Bulletin, v01.
`34, No. 12, May 1992, pp. 113—116.
`Hopkins, D., “Directional selection is Easy as Pie Menus,”
`Proceedings Fourth Computer Graphics Workshop, 8 Oct.
`1987, Cambridge, MA, p. 103.
`Kurtenbach, G., et al., “Issues In Combining Marking And
`Direct Manipulation Techniques,” Proceedings on the Sym-
`posium on User Interface Software and Technology, Nov.
`1991, South Carolina, US, pp. 137—144.
`Eric A. Bier and Maureen C. Stone. Snap—Dragging. In
`Proceedings of Siggraph ’86 (Dallas, August), Computer
`Graphics, vol. 20, No. 4, ACM, 1986, pp. 233—240.
`Eric A. Bier. Snap—Dragging: Interactive Geometric Design
`in Two and Three Dimensions. Xerox PARC Technical
`
`Report EDL—89—2 (Sep. 1989). Also available as Report No.
`UCB/CSD 88/416, Apr. 28, 1988, Computer Science Divi-
`sion, Department of Electrical Engineering and Computer
`Science, University of California, Berkeley, CA 94720.
`Eric A. Bier and Aaron Goodisman. Documents as User
`Interfaces. In R. Furuta (ed). EP90, Proceedings of the
`International Conference on Electronic Publishing, Docu—
`ment Manipulation and Typography, Cambridge University
`Press, 1990, pp. 249—262.
`Eric A. Bier. EmbeddedButtons: Documents as User Inter-
`
`faces. In Proceedings of the ACM SIGGRAPH Symposium
`on User Interface Software and Technology (Hilton Head,
`South Carolina, Nov.), ACM, 1991, pp. 45—53.
`Eric A. Bier and Steve Freeman. MMM: A User Interface
`
`Architecture for Shared Editors on a Single Screen. In
`Proceedings of the ACM SIGGRAPH Symposium on User
`Interface Software and Technology (UIST ’91 Hilton Head,
`South Carolina, Nov. 11—13), ACM, 1991, pp. 79—86.
`Eric A. Bier. EmbeddedButtons: Supporting Buttons in
`Documents. Xerox
`PARC Technical Report
`ISTL
`—ADoc—1992—10—01 (Oct. 1992). Also available in ACM
`Transactions on Information Systems, vol. 10, No. 4, Oct.
`1992, pp. 381—407.
`Paul M. English, Ethan S. Jacobson, Robert A. Mom's,
`Kimbo B. Mundy, Stephen D. Pelletier, Thomas A. Polluci,
`and H. David Scarbro. An Extensible, Object—Oriented
`System for Active Documents. In R. Furuta (ed). EP90,
`Proceedings of the International Conference on Electronic
`Publishing, Document Manipulation and Typography, Cam-
`bridge University Press, 1990, pp. 263—276.
`David Goldberg and Aaron Goodisman. Stylus User Inter-
`faces for Manipulating Text. In Proceedings of the ACM
`Symposium on User Interface Software and Technology
`(UIST ’91, Hilton Head, South Carolina, Nov.), ACM 1991,
`pp. 127—135.
`Don Hopkins. The Design and Implementation of Pie
`Menus. Dr. Dobb’s Journal. vol. 16, No. 12, Dec. 1991, pp.
`16—26.
`
`Based Search and Replace. In Proceedings of CHI ’92
`(Monterey, California, May 3—7, 1992), Human Factors in
`Computing Systems, ACM, New York, 1992, pp. 609—618.
`Gordon Kurtenbach and William Buxton. Issues in Com-
`
`bining Marking and Direct Manipulation Techniques. In
`Proceedings of the ACM Symposium on User Interface
`Software and Technology (UIST ’91 Hilton Head, South
`Carolina, Nov. 11—13), ACM, 1991, pp. 137—144.
`J. K. Ousterhout. Tcl: An Embeddable Command Language.
`In winter USENIX Conference Proceedings, 1990, pp.
`133—146.
`
`Ken Pier, Eric A. Bier, and Maureen C. Stone. An Introduc-
`tion to Gargoyle: An Interactive Illustration Tool. Xerox
`PARC Technical Report EDL—89—2 (Jan. 1989). Also avail—
`able in Proceedings of the Intl. Conf. on Electronic Publish-
`ing, Document Manipulation and Typography (Nice, France,
`Apr. 1988). Cambridge Univ. Press, (1988), pp. 223—238.
`Dean Rubine. Specifying Gestures by Example. In Proceed—
`ings of ACM SIGGRAPH ’91, Computer Graphics, vol. 25,
`No. 4, Jul. 1991, pp. 329—337.
`Daniel C. Swinehart, Polle T. Zellweger, Richard J. Beach,
`Robert B. Hagmann. A Structural View of the Cedar Pro-
`gramming Environment. Xerox PARC Technical Report
`CSL—86—1. Also available as ACM Transactions on Pro—
`
`gramming Languages and Systems, vol. 8, No. 4, 1986, pp.
`419—490.
`
`David Kurtlander and Eric A. Bier. Graphical Search and
`Replace. Computer Graphics, vol. 22, No. 4, Aug. 1988, pp.
`1 13—120.
`
`Myron W. Krueger, Thomas Gionfriddo, and Katrin Hin—
`richsen. Videoplace—An Artificial Reality. CHI ’85 Pro-
`ceedings, Apr. 1985, pp. 35—40.
`Alto User’s Handbook. Chapter 4 (Markup User’s Manual),
`Xerox Corporation Sep. 1979, pp. 85—95.
`
`Ken Perlin and David Fox. Pad An Alternative Approach to
`the Computer Interface. Proceedings of Siggraph ’93 (Ana-
`heim, California Aug. 1—6, 1993) Computer Graphics Pro-
`ceedings, Annual Conference Series, pp. 57—64.
`
`Manojit Sarkar and Marc H. Brown. Graphical Fisheye
`Views of Graphs. CHI ’92 Proceedings, May 3—7, 1992, pp.
`83—91.
`
`William Buxton and Brad A Myers. A Study in Two—Handed
`Input. CHI ’86 Proceedings, Apr. 1986, pp. 321—326.
`Joel F. Bartlett. Transparent Controls for Interactive Graph-
`ics. WRL Technical Note TN—30, Digital Equipment Cor—
`poration, Jul. 1992, pp. 1—9.
`Paul Kabbash, I. Scott MacKenzie and William Buxton.
`Human Performance Using Computer Input Devices in the
`Preferred and Non—Preferred Hands. Proceedings of Inter-
`Chi Conference (Human Factors in Computing Systems)
`(Amsterdam Apr. 1993) ACM, pp. 474—481.
`William Buxton. There’s More to Interaction Than Meets
`the Eye: Some Issues in Manual Input. Chapter 8: The
`Haptic Channel, pp. 366—375 (from Readings in Human—
`Computer Interaction; A Multidisciplinary Approach, Mor-
`gan Kaufman Publishers, Inc. 1987).
`
`

`

`US. Patent
`
`Apr. 1, 1997
`
`transparent overlay
`
`Sheet 1 of 16
`5,617,114
`1stwmdow 22 display device
`
`________
`I mMmem._ v
`‘°‘
`nmuamnuupdn-w- Illnnfllh
`lwmrmmmmwnwmv -
`
`I
`|
`
`2nd window
`
`23 display area
`
`
`
`12
`
`25
`
`PROCESSOR
`
`KEYBOARD
`
`15
`
`17
`
`20
`
`j
`
`10
`
`FIG.1
`
`83
`
`DISPLAY IMAGE
`
`80
`
`lMAGE DATA STRUCTURE
`FOR PROGRAMS
`
`
`
`
` IMAGE DATA STRUCTURE
`
`
`
`
`
`FOR OVERLAY
`
`77
`
`RENDERER _
`
`UNDERLYING DATA
`FOR OVERLAY
`
`70
`
`UNDERLYING DATA
`FOR PROGRAMS
`
`FIG. 2
`
`

`

`US. Patent
`
`Apr. 1, 1997
`
`Sheet 2 of 16
`
`5,617,114
`
`RAW DEVICE
`INPUT (OS)
`
`DRAG AND DROP
`OBJECT
`
`ANOTHER
`OVERLAY
`
`ANOTHER
`APPLICATION
`
`OVERLAY
`
`UNIVERSAL
`LANGUAGE
`
`85
`
`92
`
`WINDOW
`MANAGER
`
`93
`
`97
`
`TRANSLATOR #1
`
`TRANSLATOR #2
`
`APP. #1
`INPUT
`LANGUAGE
`
`APP. #2
`INPUT
`LANGUAGE
`
`APPLICATION
`PARSER #1
`
`PROCEDURE
`CALLS
`
`APPLICATION
`PARSER #2
`
`PROCEDURE
`CALLS
`
`PROCEDURE
`CALLS
`
`PROCEDURE
`CALLS
`
`87
`
`88
`
`APPLICATION #1
`
`APPLICATION #2
`
`FIG. 3
`
`

`

`US. Patent
`
`Apr. 1, 1997
`
`Sheet 3 of 16
`
`5,617,114
`
`§§Am §\\\\\\
`
`
`
`
`
`awaw
`§\\
`
`
`
`

`

`US. Patent
`
`Apr. 1, 1997
`
`Sheet 4 of 16
`
`5,617,114
`
`SUMMARY OF THE INVENTION
`NTlON
`- a user interface _) The present invention provides a user interface
`
`
`bold italic techniquethat allows a user to pe rm many common
`technique that allows a user to perform many common
`- .
`gnificantly enhancing
`tasks with fewer actions, thereby significantly enhancing
`
`productivity. The technique makes use of actions with
`productivity. The technique makes use of actions with
`which the user tends to be familiar, and therefore may be
`which the user tends to be familiar, and therefore may be
`learned rather quickly.
`learned rather quickly.
`
`
`FIG. 8
`
`
`
`

`

`US. Patent
`
`Apr. 1, 1997
`
`Sheet 5 of 16
`
`5,617,114
`
`O
`9O0O0‘.’0
`
`III.9QIIII‘.0.0%v
`
`Select Vertex
`
`FIG. 14
`
`

`

`US. Patent
`
`Apr. 1, 1997
`
`Sheet 6 of 16
`
`5,617,114
`
`
`
`

`

`US. Patent
`
`Apr. 1, 1997
`
`Sheet 7 of 16
`
`5,617,114
`
`"""" ‘1?
`
`U
`
`l
`
`“m"?
`
`
`
`-------a
`
`
`FIG. 19
`
`
`
`FIG. 20
`
`I—l—I
`
`I—|_I
`§l §K.
`
`I—l-I
`
`.
`
`

`

`US. Patent
`
`Apr. 1, 1997
`
`Sheet 8 of 16
`
`5,617,114
`
`
`
`llI
`
`
`1
`I
`Interfaces based on use and keyboard have become dominant in recent years.
`
`1
`Format: Except during typing, these interfaces make a very uneven use of the user's two
`general, th- strong (dominant) hand (i.e., the right hand for right-handed ;
`
`0 y
`. .
`takes of a ri h interaction with the mouse, while the weak (non-
`,
`
`I ominant) hand is re] gated to occasionally holding down a modifier key.
`II
`There is little opportunity in such interfaces to perform independent tasks
`simultaneously. Furthermore, the dominant hand spends time switching from one
`task to another more often than is necessary.
`
`:1
`r
`
`I '
`
`:
`
`
`
`

`

`US. Patent
`
`Apr. 1, 1997
`
`Sheet 9 of 16
`
`5,617,114
`
`FIG. 26
`
`I\
`
`defghijklmnopqrs
`ABCDEFGHIJKLMN
`
` abcdefghijklmno
`
`abcdefghijklmno
`ABCDEFGHIIKL
`
`
`
`ABCDEFGHIIKL
`
`
`
`
`afitd’dgfiijkfinnopgrst
`mmygmymm
`
`
`
`
`
`aficdigfiflijflmnopqrst
`wcmygymxm
`
`
`
`

`

`US. Patent
`
`Apr. 1, 1997
`
`Sheet 10 of 16
`
`5,617,114
`
`
`
`
`x
`
`y
`l
`I\
`,
`I
`_ ____
`Thosdocumem I
`II
`I;
`I
`.. ._ - I
`containstext
`I
`I
`andgraphics.
`I
`I
`
`I
`
`[- ------------- 1
`__..l
`I
`I
`l
`—I
`I
`I
`I
`
`,’\
`
`x
`
`\
`
`I
`
`’
`
`/
`I /|
`:
`'1‘
`I
`I
`I
`'l
`
`l
`r'
`‘
`|
`I
`I
`I
`I
`'
`
`IIL
`
`>
`
`FIG. 30
`
`
`
`FIG. 31
`
`~
`
`“I III
`
`A
`
`ND
`
`'
`
`——>
`
`“"III
`
`
`FIG. 32
`
`

`

`US. Patent
`
`Apr. 1, 1997
`
`Sheet 11 of 16
`
`5,617,114
`
`User Input Routine
`
`Move the Overlay
`that is connected
`to this device as a
`function of the
`device movement.
`
`
`
`Resize the overlay
`that is connected
`to this device as a
`function of the
`device movement.
`
`Yes
`
`Yes
`
`Receive
`
`
`input.
`
`
`user
`
`
`
`
`
`ls
`input from
`
`
`Overlay-moving
`
`
`device?
`
`
`
`
`
`
`
`
`
`
`
`ls
`input from
`Overlay-scaling
`
`device?
`
`
`
`
`is
`
`
`
`Move cursor as
`input from
`
`a function of the
`
`
`cursor-moving
`device?
`device movement.
`
`
`
`
`
`Pass event to the root
`
`application and perform the
`
`Event To Application
`
`Routine.
`
`
`
`
`
`FIG. 33
`
`

`

`US. Patent
`
`Apr. 1, 1997
`
`Sheet 12 of 16
`
`5,617,114
`
`OVERLAY #2
`
`OVERLAY #1
`
`
`
`
`
`APP #38
`
`APP #3A
`
`
`
`
`
`APP #38
`
`APP #3A
`
`FIG. 34C
`
`

`

`US. Patent
`
`Apr. 1, 1997
`
`Sheet 13 of 16
`
`5,617,114
`
`Event to Application Routine
`
`Application,
`
`A, receives
`an event, E.
`
`
`
`
`ls
`
`
`Is
`
`P
`event
`
`
`
`any program,
`,
`from a pointing
`device?
`selected?
`
`
`
`
`
`
`
`
`
`Determine which child, C,
`
`of a is an ancestor of or is
`equal to P. Translate <x,y>
`of E into the coordinate
`system of C. Pass event
`to C.
`
`
`
`Is
`
`any program,
`
`
`P, in gesture
`
`mode?
`
`
`
`
`
`
`
`A handles the event
`
`
`
`Determine the frontmost
`
`is C
`using the Translate
`
`child, C, of A that is behind
`
`
`an overlay?
`and Execute
`
`
`belowChild[A] and contains
`
`
`
`Routine.
`coordinates <x,y> (if any
`such 0 exists).
`
`
`
`
`Perlonn the Event to
`
` Does
`Overlay Routine.
`
`
`such a child 0
`exist?
`
`
`
`belowChild[A] (- c
`
`FIG. 35
`
`

`

`US. Patent
`
`Apr. 1, 1997
`
`Sheet 14 of 16
`
`5,617,114
`
`Event to Overlay
`Routine
`
`
`Overlay, 0,
`
`receives point-
`ing event, E,
`
`at <x,y>.
`
`
`
`
`Pass event to the tool,
`Is
`Determine the frontmost tool, T,
`
`
`
`T, that is the current
`0 in gesture
`of O that is behind belowTool[O]
`
`
`
`gesture-handling tool.
`mode?
`and contains coordinates <x,y>
`
`
`
`(if any such T exists). T looks up E in its
`
`event table to
`
`
`
`
`
`
`determine the action,
`A, to perform.
`
`Let P be the parent program
`of 0. Translate <x,y> of E
`
`
`
`and any coordinates <x,y> in
`
`Does such
`
`
`
`E's command list, L, that are
`tool T exist?
`
`
`
`marked for translation, into
`
`
`Is A the end
`
`
`the coordinate system of P.
`
`of a gesture?
`Pass event E to P. Perform
`
`
`
`the Event to Application
`
`Routine.
`
`
`Pass E to T for processing.
`
`T looks up E in its event
`
`table to determine the
`action, A, to perform.
`
`Turn off gesture
`mode for O.
`
`
`
`
`
`
`
`
`Turn on gesture mode for 0,
`making T be the current gesture-
`
`handling tool. Let the list of
`
`gesture data points be empty.
`
`
`
`Is A the
`
`beginning of
`
`a gesture?
`
`
`
`
`
`
`
`
`
`
`
`Process the event by pertormng the
`
`Event to Tool Routine. The Event to
`
`
`
`Add this data point to the list of
`Tool Routine may process the event
`
`
`
`gesture data points. T may draw
`further by recursively executing this
`
`
`
`immediate user feedback based
`flowchart, beginning at circle A above.
`
`
`
`on the partial gesture.
`
`
`FIG. 36
`
`

`

`US. Patent
`
`Apr. 1, 1997
`
`Sheet 15 of 16
`
`5,617,114
`
`_ Event to Tool
`'
`ROUtme
`
`Perform action A
`immediately. Perform
`
`feedback specified by A. any final user feedback.
`
`
`
`Tool, T,
`
`
`receives
`pointing
`
`
`
`
`event, E,
`
`
`at <x,y>.
`
`
`No
`
`
`
`
`ls T a
`click-through
`tool?
`
`
`Let A be a data structure
`representing the action that
`T performs in response to E
`as computed in the Event
`to Overlay Routine.
`
`Perform any initial user
`
` Yes
`
`Determine the frontmost
`
`application, Q, that is
`Is T
`
`
`behind T's overlay, O,
`application
`
`
`and is of a type that T
`specific?
`
`
`knows how to
`communicate with.
`
`
`
`
`
`
`
`
`
` From A and coordinates
`<x,y >, compose a
`
`
`
`Extract from E the list of
`command to deliver to Q,
`
`commands, L, that were added to
`and/or a set of calls to the
`
`
`E by any tools that E has been
`procedures of Q. Deliver
`
`
`passed to previously (it any).
`the commands and/or
`
`
`make the calls.
`
`
`
`
`Use the list of
`Compose a new list of
`commands that
`commands from A and L
`Routine.
`is part of A.
`using the Composition
`
`
`
`
`
`Recursively call the Event to Overlay
`Routine entering at circle A. This routine
`may return data (e.g., if T is a clipboard).
`Use this data and perform any parts of
`action A that T can perform.
`
`
`
`
`
`
`
`FIG. 37
`
`
`
`
`Perform any final
`user feedback.
`
`
`
`Replace the list L in E with the new list
`just computed. Set belowTool[O] <— T.
`
`
`
`
`

`

`US. Patent
`
`Apr. 1, 1997
`
`Sheet 16 0f 16
`
`5,617,114
`
`fillanalue —
`
`Command:
`
`
`
`
`List of programs
`waiting for this value
`
`The actual shape
`
`
`
`APP #3A
`
`
`
`APP #38
`
`FIG. 40
`
`

`

`5,617,114
`
`1
`USER INTERFACE HAVING
`CLICK-THROUGH TOOLS THAT CAN BE
`COMPOSED WITH OTHER TOOLS
`
`This is a Continuation of application Ser. No. 08/095, 5
`445, filed Jul. 21, 1993, now abandoned.
`
`2
`A display device, typically a visual display device such as
`a cathode ray tube (CRT) or a liquid crystal display (LCD),
`provides the user with information about the application and
`the underlying data, and allows the user to generate appro(cid:173)
`priate input signals and thus control the operation of the
`machine to produce the intended work product. The com-
`bination of input devices, display devices, and the nature of
`the information that the application provides the user may be
`thought of as the user interface to the application.
`Although it is in principle possible for every application
`program to be entirely self-sufficient, it is almost universally
`the case that the application program executes in conjunc(cid:173)
`tion with an operating system ("OS"). The OS is a program
`that schedules and controls the machine resources to provide
`15 an interface between the application programs and the
`machine hardware. The OS typically provides the basic
`housekeeping functions that all application programs are
`likely to require, such as maintaining a file system, sched(cid:173)
`uling the CPU, receiving input from input devices, commu-
`20 nicating with storage devices, sending data to display
`devices, and providing a generic mechanism according to
`which a user can manage files and cause various applications
`to execute. In the world of personal computers ("PCs") and
`workstations, operating systems are often associated with a
`25 particular type of hardware configuration, but this is not
`necessarily the case. Unix is an example of an OS that has
`been ported to run on many types of machine.
`One type of operating system that has come into increas(cid:173)
`ing use in recent years provides a graphical user interface
`30 ("GUI"). Apple Computer's Macintosh as, IBM's OS/2,
`and Microsoft's Windows (actually a GUI shell that runs on
`top of a character-based operating system known as DOS)
`are the best known GUIs in the PC realm. The Macintosh OS
`has to date been available only on Apple's own Macintosh
`35 PCs based on the Motorola 680xO family of microprocessors
`while OS/2 and Windows have only been available on
`so-called IBM-compatible PCs based on the Intel 80x86
`family of microprocessors. This trend is in the process of
`changing, with Microsoft's Windows NT having versions
`40 capable of running on more than one type of microprocessor.
`One relevant aspect of a GUI is that an open file for a
`given application is typically given a window, which is a
`movable and resizable region on the screen. The OS can
`45 have its own windows showing directory structures, with
`files and applications possibly being represented by icons
`(small graphical objects representing actions or items).
`There may be other windows that do not correspond to open
`files. An advantage of a GUI is that it provides a rather
`50 consistent user environment across applications. Some GUIs
`allow mUltiple applications to be open at the same time.
`Regardless of the type of as, the application program,
`with varying amounts of help from the as, typically pro(cid:173)
`vides the user with a visible representation (sometimes
`referred to as the "screen image" or the "display image") of
`the underlying data. The user acts on the visible represen(cid:173)
`tation, and the program translates these actions to operations
`on the underlying data. As used herein, the term "visible
`representation" will refer to the visual representation of the
`60 underlying data not only for application programs, but for all
`kinds of programs, including the as and various types of
`utility programs.
`For example, in a word-processor, the underlying data
`consists of text with associated information specifying how
`the document will look when it is printed out on a printer.
`The associated information relates to document layout such
`as paragraphs and columns, and to text attributes such as
`
`10
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`The following three commonly-owned copending appli(cid:173)
`cations, including this one, are being filed concurrently and
`the other two are incorporated by reference into this one:
`Eric A. Bier, William A. S. Buxton, entitled "USER
`INTERFACE HAVING MOVABLE SHEET WITH
`CLICK-THROUGH TOOLS" U.S. application Ser. No.
`08/095,598, now U.S. Pat. No. 5,581,670;
`William A. S. Buxton and Eric A. Bier, entitled "USER
`INTERFACE HAVING SIMULTANEOUSLY MOVABLE
`TOOLS AND CURSOR" U.S. application Ser. No. 08/095,
`591; and
`Eric A. Bier, William A. S. Buxton, and Maureen C. Stone
`entitled
`"USER
`INTERFACE HAVING CLICK(cid:173)
`THROUGH TOOLS THAT CAN BE COMPOSED WITH
`OTHER TOOLS".
`
`BACKGROUND OF THE INVENTION
`
`The present invention relates generally to processor(cid:173)
`controlled machines such as computers and more specifi(cid:173)
`cally to user interfaces for allowing a user to interact with
`the machine.
`A frequent use of a processor-controlled machine such as
`a computer is to communicate information to a user of the
`machine and accept information from the user, thereby
`allowing the user to perform a specified task. Depending on
`the task at hand, the user will often make use of a task(cid:173)
`specific application program such as a word processor
`(sometimes referred to as a text editor), a spreadsheet, a
`database, or a drawing program (sometimes referred to as a
`graphics editor). A reference to a specific type of program or
`editor is not intended to imply a stand-alone application
`program having only the particular functionality, since many
`programs have more than one type of functionality.
`A typical application program consists of a set of instruc(cid:173)
`tions (the "application") that are executed in response to
`input signals to create and modify associated data (some(cid:173)
`times referred to as the underlying data). In many instances,
`this associated data is stored on a disk as a data file
`(sometimes referred to as "the file"), and portions are read
`into memory during program execution. For at least some
`applications, the data represents a document that is to be
`viewed (e.g., printed or displayed on a screen), and the
`application allows a user to modify the document.
`In many instances, a user provides at least some of the
`input signals through one or more input devices, often a
`keyboard and a pointing device such as a mouse. By way of
`background, a mouse is a device that is moved over a work
`surface, typically next to the keyboard, and provides posi(cid:173)
`tion signals so as to cause a cursor on the screen to move in
`accordance with the mouse's movements. The cursor is a
`special symbol that is used by an interactive program as a
`pointer or attention-focusing device. The mouse contains
`one or more pushbutton switches ("buttons") to provide 65
`additional input signals, which may be interpreted as part of
`a cursor event.
`
`55
`
`

`

`5,617,114
`
`15
`
`4
`programs, shows the direction in which some of these efforts
`have led, and the way that improving one aspect of a user
`interface can degrade another.
`A common configuration for drawing programs has a
`fixed tool palette to one side of the drawing area and a menu
`bar above the drawing area. To change tools, the user moves
`the cursor to the palette, clicks on the icon for the desired
`tool, and moves the cursor back to the appropriate location
`in the drawing area. To effect a desired operation on a
`10 desired object, the user moves the cursor to the object, clicks
`the object to select the object, moves the cursor to the menu
`bar, depresses the mouse button to pull down the menu,
`drags to the desired menu item, and releases the mouse
`button. The user then moves the cursor to the drawing area,
`to another item in the menu bar, or to the tool palette. This
`is a lot of mouse movement for even the simplest actions.
`Tear-off menus and movable tool palettes allow the user
`to position what amount to permanently open menus and the
`tool palette near the area where drawing is actively occur-
`20 ring, and thereby reduce the length of mouse travel. Tear-off
`menus and movable palettes have made drawing more
`efficient in the sense of reducing the distances the user has
`to move the cursor, but have made it less efficient in another.
`They tend to take up a lot of the drawing area, especially
`25 near where the user is drawing. This can result in the user's
`constantly having to interrupt the drawing tasks to move the
`menus and palettes out of the way. This difficulty is com(cid:173)
`pounded by the fact that as programs have gotten more
`powerful (greater functionality), the menus have grown
`longer and take up even more area. Unfortunately, this
`example of the trade-offs encountered in trying to meet the
`above goals is far from rare.
`
`30
`
`3
`font, size, style, and color. Depending on the particular
`word-processor and the operating system, the screen image
`may be limited to the text content, or may show the
`document substantially as it will appear when printed
`(WYSIWYG-pronounced "wizzywig," an acronym for 5
`"what you see is what you get"). A program designed for a
`character-based OS such as DOS is likely to provide some(cid:173)
`thing approaching the former, one designed for a GUI is
`likely to provide something approaching the latter.
`A similar range of possible screen images will be found in
`other types of application programs. For example, in a
`drawing program, the underlying data will contain a descrip(cid:173)
`tion of each graphical object that will appear on the docu(cid:173)
`ment. The description includes what is needed to give the
`object its intended appearance, including shape, size, line
`color and thickness, fill color and pattern, relative position in
`the plane of the document, and stacking order (whether the
`object is in front of or behind other objects). The screen
`image may show only the outlines of the objects (wireframe
`view) or may be a full WYSIWYG view.
`Regardless of the type of application, the user manipu(cid:173)
`lates input devices with reference to the screen image in
`order to effect desired changes. This is typically done by
`placing a cursor at a particular position on the screen that
`corresponds to the displayed location of an object to be
`modified, and executing one or more user events such as
`keystrokes or mouse actions. Mouse actions include button
`depression, button release, mouse movement, clicks, and
`drags. A mouse click refers to the user depressing and
`releasing one of the buttons without moving the mouse, but
`the term is also used to refer to the act of depressing one of
`the buttons. A drag (or sometimes click-and-drag) refers to
`the user positioning the cursor with the mouse, depressing
`one of the buttons, moving the mouse to a new position with
`the button still down, and releasing the button at the new 35
`location. The effect of mouse button depressions, mouse
`button releases, clicks, and drags may be modified by
`holding down a keyboard key or a different mouse button (if
`present).
`For example, placing a cursor at a particular location in a
`word processor image may operate to insert typed text at that
`location. Dragging the cursor over a portion of the displayed
`text may select the text (shown on the screen as highlighted)
`so that the user can apply an operation (such as deleting,
`moving, or changing the font) to the selected text by some
`other mechanism. Depending on the application and the
`desired operation, the mechanism may be selecting the
`operation from a menu or entering a command from the
`keyboard.
`Similarly, in a drawing program, the cursor can be placed
`in a mode by clicking on a tool icon (e.g., rectangle tool, line
`tool, polygon tool) so that subsequent clicks and drags with
`the cursor result in the creation of graphical objects. Click(cid:173)
`ing on an existing object with a plain cursor may result in
`selecting the object so that an operation may be applied via
`some other mechanism. If a drag is initiated with the cursor
`on an object, the result of the drag may be to cause the object
`to move along with the cursor, or may be to cause the object
`to be resized, depending on the cursor location on the object. 60
`For users to be more productive, they should be provided
`with tools that are relatively easy to learn, easy to use, and
`powerful. These goals are sometimes easy to achieve indi(cid:173)
`vidually, but rarely in combination. Nevertheless, consider(cid:173)
`able efforts have been expended in attempts to design user 65
`interfaces that are more intuitive, efficient, and versatile. The
`example discussed below, taken from the realm of drawing
`
`40
`
`SUMMARY OF THE INVENTION
`
`The present invention provides a user interface technique
`that allows a user to perform many common tasks with fewer
`actions, thereby significantly enhancing productivity. The
`technique makes use of actions with which the user tends to
`be familiar, and therefore may be leamed rather quickly. The
`invention may be implemented in the context of a single
`program, or may be incorporated into the operating system
`. so as to be available across different programs, including the
`45 operating system.
`The invention operates in the environment of a processor(cid:173)
`controlled machine for executing a program that operates on
`a set of underlying data and displays a visible representation
`thereof. The system further provides a visual depiction of a
`50 set of controllers, such as tool palettes, property palettes,
`menus, switches, dialog boxes, and sliders. The controllers
`are collectively referred to as tools. The user normally
`specifies operations to the program by interacting with the
`tools and the visible representation (using a set of input
`55 devices), and views the results of those operations on a
`display device. The user typically positions a cursor at a
`desired location and takes an action, such as a mouse click,
`with the cursor.
`In brief, the invention is characterized by what are
`referred to as click-through tools that can be combined with
`other tools (including other click-through tools) to provide
`composite tools. The visual depiction of a click-through tool
`includes a generally transparent delineated active area that is
`movable so that it can be placed over a desired portion of the
`visible representation. When the user interacts with the
`visible representation through the active area, the action
`takes on an attribute of the particular click-through tool. In
`
`

`

`5,617,114
`
`5
`
`40
`
`5
`an embodiment of the invention that operates in the context
`of a drawing program, where the visible representation is a
`representation of a set of graphical objects, the click-through
`tools may include tools for creating objects, and tools for
`copying, modifying, and deleting existing objects.
`Once a click-through tool has been positioned relative to
`the visible representation, the result of an action through the
`tool will typically depend on the particular location within
`the active area that the action occurs. Put another way, the
`result of an action through a click-through tool will typically 10
`depend on the portion of the visible representation that is
`under the cursor when the action is taken. While a conven(cid:173