as) United States
`a2) Patent Application Publication 10) Pub. No.: US 2008/0316183 Al
`
`(43) Pub. Date: Dec. 25, 2008
`Westermanet al.
`
`US 20080316183A1
`
`(54) SWIPE GESTURES FOR TOUCH SCREEN
`KEYBOARDS
`
`(75)
`
`Inventors:
`
`Wayne Carl Westerman, San
`Francisco, CA (US); Henri
`Lamiraux, San Carlos, CA (US);
`Matthew Evan Dreisbach, Boulder
`Creek, CA (US)
`
`Correspondence Address:
`APPLE C/O MORRISON AND FOERSTER ,LLP
`LOS ANGELES
`555 WEST FIFTH STREET SUITE 3500
`
`LOS ANGELES, CA 90013-1024 (US)
`
`(73) Assignee:
`
`Apple Inc.
`
`(21) Appl. No.:
`
`11/766,929
`
`(22)
`
`Filed:
`
`Jun. 22, 2007
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`(2006.01)
`GO6F 3/041
`(52) US. CMe coecccccsscsssssssssssssvenesessessessssssssssniensees 345/173
`
`(57)
`
`ABSTRACT
`
`Systems, methods, and devices for interpreting manual swipe
`gestures as input in connection with touch-sensitive user
`interfaces that include virtual keyboardsare disclosed herein.
`Theseallow for a user entering text using the virtual keyboard
`to perform certain functions using swipes across the key area
`rather than tapping particular keys. For example, leftward,
`rightward, upward, and downward swipescan be assigned to
`inserting a space, backspacing, shifting (as for typing capital
`letters), and inserting a carriage return and/or new line. Vari-
`ous other mappingsare also described. The described tech-
`niques can be used in conjunction with a variety of devices,
`including handheld devices that include touch-screen inter-
`faces, such as desktop computers, tablet computers, notebook
`computers, handheld computers, personal digital assistants,
`media players, mobile telephones, and combinationsthereof.
`
`100
`
`a
`
`
`
`11:30 AM
`
`What's for
`dinner tonight?
`
`How about
`meeting at
`Whole Foods?
`
`116
`
`
`
`
`
`APPLE 1023
`
`APPLE 1023
`
`1
`
`

`

`Patent Application Publication
`
`Dec. 25, 2008 Sheet 1 of 14
`
`US 2008/0316183 Al
`
`0OT
`
`ecb
`
`
`
`é}ybiuo}souuip
`
`JO}SJEUAA
`
`
`
`£SpoosjouM
`
`jnogeMoH
`
`yeBureau
`
`CZEE
`
`FEOe
`
`
`
`é}4Biuo0}souuip
`
`JO}SJEUM
`
`
`
`&Spoo,sou
`
`jnoqeMoH
`
`yeBujeow
`
`ETE
`
`eS.0zIaES
`
`OLL
`
`dlOld
`
`VIDIA
`
`2
`
`
`

`

`Patent Application Publication
`
`Dec. 25, 2008 Sheet 2 of 14
`
`US 2008/0316183 Al
`
`
`
`é}y61u0}JauuIp
`
`JoySJeUM
`
`
`
`&SPO0S|OUAA
`
`jnogeMOH
`
`yeBuyeow
`
`
`
`é}yBiucyJauulp
`
`JO}SJEUM
`
`
`
`&SPO0s/OUAA
`
`jnogeMOH
`
`yeBusow
`
`cl
`
`GoOIA
`
`VeDid
`
`esoe
`
`3
`
`

`

`Patent Application Publication
`
`Dec. 25, 2008 Sheet 3 of 14
`
`US 2008/0316183 Al
`
`
`
`
`
`
`
`wilel[allelzILa
`
`
`
`é}yBluo}Jauuip
`
`Jo}SJeUM
`
`éSpoodsou
`
`ynoqeMOH
`
`yeBuyjeow
`
`OOT
`
`OLb
`
`9c
`
`
`
`é}ybiuo}JOuUIp
`
`JoySJEUM,
`
`
`
`&SPooysjOuAA
`
`jnoqeMOH
`
`yeBuneow
`
`€Old
`
`ICOIA
`
`4
`
`
`

`

`Patent Application Publication
`
`Dec. 25, 2008 Sheet 4 of 14
`
`US 2008/0316183 Al
`
`
`
`é1y61u0}souuIp
`
`JoyJeu
`
`
`
`SPOOFB[OUAA
`
`jnoqeMoH
`
`yeBujoow
`
`
`
`é1yBluo}JouuIp
`
`
`
`&ESPOOyS|OUAA
`
`jnoqeMoH
`
`yeBuljoow
`
`Joyseu
`
`EE
`
`Be
`
`dhDIA
`
`VROld
`
`5
`
`

`

`Patent Application Publication
`
`Dec. 25, 2008 Sheet 5 of 14
`
`US 2008/0316183 Al
`
`
`
`é}ybiuo}JEuUlp
`
`Joys,JeunA
`
`
`
`é1yBluo}JauUIp
`
`Jo}SJEUAA
`
`
`
`&ESPoOorsjOUMA
`
`jnogeMoH
`
`yeBuysow
`
`
`
`Spoors|OUAA
`
`jnoqeMoH
`
`yeBusou
`
`VSDIADtDIA
`
`Bel
`
`aLLe
`
`esEB
`
`TDx
`
`00T
`
`6
`
`
`

`

`Patent Application Publication
`
`US 2008/0316183 Al
`
`ISOIA
`
`Dec. 25, 2008 Sheet 6 of 14
`
`Joys7eUM\ dS‘Old
`
`
`
`é1yB6lu0}JuU!p
`
`
`
`&ESPOOySIOUAA
`
`jnoqeMoH
`
`yeBuljoow
`
`
`
`é}46iuo}J@EuUUIp
`
`
`
`JolsyeU\\
`
`
`
`ESPOOY[OU
`
`jnoqeMoH
`
`yeBuljoow
`
`espuss
`
`Nels
`
`De
`
`OLL
`
`7
`
`

`

`Patent Application Publication
`
`Dec. 25, 2008 Sheet 7 of 14
`
`US 2008/0316183 Al
`
`00T
`
`Bee
`
`
`
`é}4Blu0}JBUUIp
`
`Jo}sJeUM
`
`
`
`&SPOorsjOUAA
`
`jnoqeMoH
`
`yeBuysow
`
`eS=BES
`
`
`
`é}yBiuo}JauuUlp
`
`
`
`&SPOOyS|OUAA
`
`jnoqeMoH
`
`yeBusou
`
`Joyseu
`
`G9‘OIA
`
`V9OTH
`
`eaee
`
`Le
`
`8
`
`

`

`Patent Application Publication
`
`Dec. 25, 2008 Sheet 8 of 14
`
`US 2008/0316183 Al
`
`
`
`[bl[e|[H]le}S/Lsi[s][v1
`
`
`
`é}4b6Iu0}JauUlp
`
`JO}SEU
`
`
`
`ESPOOY[OU
`
`jnoqeMoH
`
`yeBunoow
`
`
`
`é1yB1uo}sauulp
`
`
`
`ESPOOYS|OUAA
`
`jnoqeMoH
`
`yeBunjoow
`
`Jo}SJeUM
`
`
`
`
`
`
`
`ESICIDIENNalc
`
`ee|_\Oe
`
`
`
`eeoe@|[n][y|[s|[4]lo]x][z]L¢)
`
`9DIld
`
`DIDIA
`
`9
`
`

`

`Patent Application Publication
`
`Dec. 25, 2008 Sheet 9 of 14
`
`US 2008/0316183 Al
`
`JEEWOWWAG
`
`Lo}Ls)Le)Lee)Ls)Le}elLzbe
`
`Jo}SJeUNA
`é}yBluo}JauUlpé}y6luo}JoUUIp
`
`
`
`JoysJeUM,
`ESPOOJSjOUAAéSpoodslOuM
`
`
`jnogeMOHjnoqeMOH
`yeBuyeowyeBureau
`
`8DIAZOU
`
`10
`
`10
`
`

`

`Patent Application Publication
`
`Dec. 25, 2008 Sheet 10 0f14
`
`US 2008/0316183 Al
`
`
`
`
`FEEDBACK
`POPUP
`GRAPH.
`
`
`
`
`
`
`fo 907
`
`KEYBOARD INPUT
`MANAISER
`
`
`
`
` TERT EVENTS
`
`
`
`
`FQ
`APPLICATIONS
`
`
`901
`
`(O 902
`
`PROCESSING
`
`TOUCH IMAGE
`
`ro 903
` FINGER PATH
`
`TRACKING
`EVENTS
`
`REYSOARD SLIDE
`RECOGNIZER
`
`
`
`
`
`KEYBOARD
`SWIPE
`EVENTS
`
`
`
`
`GANCEL TAP
`
`
`
`
`SiGNWAL
`
`
`REYBOARE
`
`
`KEY TAP
`JAP
`EVENTS
`
`RECOGNIZER
`
`904
`
`FIG. 9
`
`11
`
`11
`
`

`

`Patent Application Publication
`
`Dec. 25,2008 Sheet 11 of 14
`
`US 2008/0316183 Al
`
`
`
`
`ISSUE FINAL
`
`
`GET NEXT FINGER
`KEV TAF
`curr
`
`CHOICE
`1010~
`PATH EVENT
`
`
`
`IS PATH i
`
`
`TOUCH LIFTED
`_— 1002
`a
`
`
`18 PATH UNEW
`TOUGH?
`
`
`
`
`P1001
`
`
`CAPTURE
`FOUICRECAVA
`
`LOCATION & TIME
`
`1006
`
`771005
`
`
`BATH i
`
`PATHi
`TOUCKDOMN
`
`
`
`LOCATION &
`LacATION &
`
`
`
`FIWE
`
`
`TIME
`
`UPDATE KEY
`CHOICE UNDER
`
`
` MEASURE
`DISPLACEMENT
`
`
`(ORs, OEE TH
`
`§ DBT
`
`
`TOUCHING Tih
`
`> Min Sipe
`< Swipe Gesare
`
`
`Length
`
`Timeout
`
`
`
`
`1007
`
`1018
`
`
`ISSUE SWIPE
`
`
`RYGHT ACTION
`
`
`
`
`1013.
`YY
`
`
`
`
`Oy
`
`
`> | OTt
`?
`
`
`
`
`
`
`1012
`
`CANCEL KEY TAP
`
`1015
`
`ISSUE SWIPE
`
`UP ACTION
`
`1017
`
`¥
`
`ISSUE SWAFE
`LEFT ACTION
`
`PERNODECALLY
`UL
`PATH
`|
`TRL PATH i
`
`1020
`
`ISSUE SWIPE
`DOWN
`ACTION
`DOWNACTION
`
`FIG. 10
`
`12
`
`12
`
`

`

`Patent Application Publication
`
`Dec. 25,2008 Sheet 12 of 14
`
`US 2008/0316183 Al
`
`tangentof 45° = 1
`
`tangentof theta = Ay / Ax
`
`
`tangent of 60° = V3
`
`O= 45°
`
`@ = 60°
`
`Fig. 11B
`
`x7, y7
`
`Fig. 11C
`
`x2, y2
`
`13
`
`

`

`Patent Application Publication
`
`Dec. 25,2008 Sheet 13 of 14
`
`US 2008/0316183 Al
`
`Multi-touch
`Interface
`
`Display
`
`0
`Processor
`
`Memory/
`Program
`Storage
`
`Fig. 12
`
`14
`
`14
`
`

`

`Patent Application Publication
`
`Dec. 25,2008 Sheet 14 of 14
`
`US 2008/0316183 Al
`
`
`
`15
`
`15
`
`

`

`US 2008/0316183 Al
`
`Dec. 25, 2008
`
`SWIPE GESTURES FOR TOUCH SCREEN
`KEYBOARDS
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`
`
`[0001] This is related to the following U.S. patents and
`patent applications, each of which is hereby incorporated by
`reference in its entirety:
`titled “Method and
`[0002] US. Pat. No. 6,323,846,
`Apparatus for Integrating Manual Input,” issued Nov.
`27, 2001;
`[0003] U.S. patent application Ser. No. 10/840,862,
`titled “Multipoint Touchscreen,” issued May 6, 2004;
`[0004] U.S. Provisional Patent Application No. 60/804,
`361, titled “Touch Screen Liquid Crystal Display,”filed
`Jun. 9, 2006;
`[0005] U.S. Provisional Patent Application No. 60/883,
`979, titled “Touch Screen Liquid Crystal Display,”filed
`Jan. 8, 2007;
`[0006] U.S. patent application Ser. No. 11/367,749,
`titled “Multi-functional Hand-held Device,” filed Mar.
`3, 2006;
`[0007] U.S. patent application Ser. No. 11/228,737,
`titled “Activating Virtual Keys of a Touch-Screen Virtual
`Keyboard,”filed Sep. 16, 2005;
`[0008] U.S. patent application Ser. No. 11/228,700,
`titled “Operation of a Computer with a Touch Screen
`Interface,” filed Sep. 16, 2005;
`titled “System and
`[0009] U.S. Pat. No. 6,677,932,
`Method for Recognizing Touch Typing Under Limited
`Tactile Feedback Conditions,” issued Jan. 13, 2004; and
`[0010] US. Pat. No. 6,570,557, titled “Multi-touch Sys-
`tem and Method for Emulating Modifier Keys Via Fin-
`gertip Chords,” issued May 27, 2003.
`
`BACKGROUND
`
`[0011] The present invention relates generally to input sys-
`tems, methods, and devices, and moreparticularly, to sys-
`tems, methods, and devices for interpreting manual swipe
`gestures as input in connection with touch-screen keyboards.
`[0012] There currently exist various types of input devices
`for performing operations in electronic devices. The opera-
`tions, for example, may correspond to moving a cursor and
`making selections on a display screen. The operations may
`also include paging, scrolling, panning, zooming, etc. The
`input devices may include, for example, buttons, switches,
`keyboards, mice, trackballs, pointing sticks, joy sticks, touch
`surfaces (including touch pads and touch screens, etc.), and
`other types of input devices.
`[0013] Various types of touch surfaces and touch screens
`are describedthe related applications cross-referenced above.
`Touch screens may include a display, a touch panel, a con-
`troller, and a software driver. The touch panel may include a
`substantially transparent panel that incorporates touch-sens-
`ing circuitry. The touch panel can be positioned in front of a
`display screen or constructed integrally with a display screen
`so that the touch sensitive surface correspondsto all or a
`portion of the viewable area of the display screen. The touch
`panel can detect touch events and send corresponding signals
`to the controller. The controller can processthese signals and
`send the data to the computer system. The software driver can
`translate the touch events into computer events recognizable
`
`by the computer system. Other variations of this basic
`arrangementare also possible.
`[0014] The computer system can comprise a variety of
`different device types, such as a pocket computer, handheld
`computer, or wearable computer (such as on the wrist or arm,
`or attached to clothing, etc.). The host device may also com-
`prise devices such as personaldigital assistants (PDAs), por-
`table media players (such as audio players, video players,
`multimedia players, etc.), game consoles, smart phones, tele-
`phonesor other communicationsdevices, navigation devices,
`exercise monitors or other personaltraining devices, or other
`devices or combination of devices.
`[0015]
`In some embodiments, touch screens can include a
`plurality of sensing elements. Each sensing element in an
`array of sensing elements (e.g., a touch surface) can generate
`an output signal indicative of the electric field disturbance
`(for capacitance sensors), force (for pressure sensors), or
`optical coupling (for optical sensors) at a position on the
`touch surface correspondingto the sensor element. The array
`of pixel values can be considered as a touch, force, or prox-
`imity image. Generally, each of the sensing elements can
`work independently of the other sensing elements so as to
`produce substantially simultaneously occurring signals rep-
`resentative of different points on the touch screen 120 at a
`particular time.
`[0016] Recently, interest has developed in touch-sensitive
`input devices, such as touch screens, for hand-held or other
`small form factor devices. For example U.S. patent applica-
`tion Ser. No. 11/367,749, titled “Multi-functional Hand-held
`Device,” discloses a multi-functional hand-held device that
`integrates a variety of device functionalities into a single
`device having a hand-held form factor. In such applications,
`touch screens can be used for a variety of forms of input,
`including conventionalpointing and selection, more complex
`gesturing, and typing.
`[0017] Conventional touch-typing techniques may be dif-
`ficult to use on touch-screen based small form factor devices.
`
`As aresult, users often use “hunt and peck”typing techniques
`to input text into such devices. Various techniques for enhanc-
`ing such textual input are disclosed in U.S. patent application
`Ser. Nos. 11/367,749; 11/228,737; and 11/228,700 and U.S.
`Pat. Nos. 6,677,932 and 6,570,557 incorporated by reference
`herein. It is believed that textual input on virtual keyboards
`can be further enhanced.
`
`SUMMARY
`
`[0018] The present invention can relate, for example, to a
`methodofinterpreting swipe gesture input to a device having
`a touch-sensitive input. The touch-sensitive input can include
`a virtual keyboard area, in which taps of a touch object gen-
`erate text input. The method can include detecting a swipe
`gesture across the virtual keyboard, determining a direction
`of the swipe gesture, and performing a predetermined func-
`tion determinedbythe direction ofthe swipe gesture. A swipe
`gesture can include a touchdownof a touch object followed
`by a sliding motion of the touch object across the virtual
`keyboard.
`[0019] Detecting a swipe gesture can include acquiring
`touch image data from the touch-sensitive device, processing
`the image to generate one or more finger path events, deter-
`mining a displacementof the one or morefinger path events,
`and detecting a swipe gesture if the displacement exceeds a
`predeterminedthreshold. Ifthe displacementdoes not exceed
`the threshold, the input can be interpreted as a conventional
`
`16
`
`16
`
`

`

`US 2008/0316183 Al
`
`Dec. 25, 2008
`
`tap. The time of the motion associated with the input can also
`be compared to a maximum swipe gesture timeout threshold.
`If the timeout threshold is exceeded, the input can be inter-
`preted as a conventionaltap.
`[0020] Determining the direction of a swipe gesture can
`include comparing a magnitudeof a vertical displacement of
`the swipe gesture to a magnitude of a horizontal displacement
`of the swipe gesture. A swipe gesture can be interpreted as a
`vertical swipe gesture if the vertical displacementis greater
`than the horizontal displacement. Alternatively, a swipe ges-
`ture can be interpreted as a horizontal swipe gesture if the
`vertical displacementis less than the horizontal displacement.
`Alternatively, determining the direction of a swipe gesture
`can include comparinga ratio of the vertical displacement of
`the swipe gesture to the horizontal displacementof the swipe
`gesture to a predetermined threshold. A swipe gesture can be
`interpreted as a vertical swipe gesture if the ratio is greater
`thanthe threshold, while a swipe gesture can be interpreted as
`a horizontal swipe gesture if the ratio is less than the thresh-
`old.
`
`[0021] A vertical swipe gesture can be interpreted as an
`upward or downward swipe gesture depending on the sign
`(i.e., positive or negative) of the vertical displacement. A
`horizontal swipe gesture can be interpreted as a rightward or
`leftward swipe gesture depending on the sign (i.e., positive or
`negative) of the horizontal displacement.
`[0022] A variety of functions can be invoked using the
`swipe gestures. For example, the swipe gestures can be used
`for spacing, erasing, or punctuation insertion. Feedback can
`be presented to the user indicating the function performed.
`The functions performed by particular swipe gestures may
`also be customized bythe user.
`[0023]
`Particular swipe gestures that can be linked to par-
`ticular functions can include using a rightward swipe gesture
`to invoke a space, using a leftward swipe gesture to invoke a
`backspace, using a downward swipe gesture to invokea car-
`riage return and/or a new line, and using an upward swipe
`gesture to invoke a shift. Additionally, a second upward swipe
`gesture can be used to invoke a caps lock. The second upward
`swipe can either be performed sequentially to the first upward
`swipe or concurrently with the first upward swipe, i.e., as a
`multi-fingered swipe gesture. Other alternative mappings of
`swipe gestures to functions include using an upward swipe
`gesture to insert a punctuation character, such as a period or
`apostrophe.
`[0024] Multi-fingered swipe gestures can be used to invoke
`additional functionality. For example, a multi-fingeredleft-
`ward swipe can be used to invoke deletion of a word in a
`manner analogous to using a leftward swipe gesture as a
`backspace. Similarly, a multi-fingered rightward swipe can
`be used to insert a punctuation character and a space, such as
`a period and a spaceat the end of a sentence. Swipe gestures
`can also be used to invoke alternative keyboards containing
`punctuation, numbers, or special symbols of various types.
`[0025] The present invention can also relate to a computer
`system including a multi-touch interface that has been
`adapted and/or programmedto detect and process swipe ges-
`ture input in the various ways described above. The computer
`system can take the form of a desktop computer, a tablet
`computer, a notebook computer, a handheld computer, a per-
`sonal digital assistant, a media player, a mobile telephone,
`and combinations of one or more of these items. The multi-
`touch interface can include a touch screen.
`
`In some embodiments, the computer system can
`[0026]
`include a touch image processor that receives touch image
`data from the multi-touch interface and generates finger path
`tracking events. These finger path tracking events can be
`processed by a keyboard tap recognizer and a keyboard slide
`recognizer, which generate key tap and swipe events, respec-
`tively, in response to the touch input received. A keyboard
`input manager can receive the key tap events and the keyboard
`swipe events and generates text events for an application
`running on the computer system and feedback popup graph-
`ics to be displayed on a display of the computer system.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0027] The following drawings form part of the present
`specification and are included to further demonstrate certain
`aspects of the present invention. The invention may bebetter
`understood by reference to one or more of these drawings in
`combination with the detailed description of specific embodi-
`ments presented herein.
`[0028]
`FIGS. 1A-1B depict a front plan view of a user
`typing using an exemplary electronic device with touch
`screen display in accordance with an embodiment of the
`present invention.
`[0029]
`FIGS. 2A-2C depict a user invoking a backspace
`using a leftward swipe gesture in accordance with an embodi-
`mentof the present invention.
`[0030]
`FIG. 3 depicts a user typing in accordance with an
`embodimentof the present invention.
`[0031]
`FIGS. 4A-4C depict a user invoking a space using a
`rightward swipe gesture in accordance with an embodiment
`of the present invention.
`[0032]
`FIGS. 5A-5C depict a user invoking a carriage
`return/new line using a downward swipe gesture in accor-
`dance with an embodimentof the present invention.
`[0033]
`FIGS. 6A-6D depict a user invoking a shift using an
`upward swipe gesture in accordance with an embodiment of
`the present invention.
`[0034]
`FIG. 7 depicts a user invoking a caps lock using two
`sequential upward swipe gestures in accordance with an
`embodimentof the present invention.
`[0035]
`FIG. 8 depicts a user invoking the display of an
`alternative keyboard using a swipe gesture in accordance with
`an embodimentof the present invention.
`[0036]
`FIG. 9 depicts a block diagram of an exemplary
`swipe recognition system in accordance with various
`embodiments of the present invention.
`[0037]
`FIG. 10 depicts a flow chart of an exemplary swipe
`gesture detection technique in accordance with various
`embodiments of the present invention.
`[0038]
`FIGS. 11A-11C depict analternative methodofdis-
`tinguishing vertical swipes from horizontal swipes in accor-
`dance with various embodiments ofthe present invention.
`[0039]
`FIG. 12 depicts a simplified block diagram of a
`computer system implementing one or more embodiments of
`the present invention.
`[0040]
`FIG. 13 depicts various computer form factors that
`may be used in accordance with embodiments of the present
`invention.
`
`DETAILED DESCRIPTION
`
`[0041] Reference is now made to FIG. 1A, which depicts a
`front plan view of an exemplary electronic device 100 that
`implements a touch screen-based virtual keyboard. Elec-
`
`17
`
`17
`
`

`

`US 2008/0316183 Al
`
`Dec. 25, 2008
`
`tronic device 100 includesa display 110 that also incorporates
`a touch screen. The display 110 can be configuredto display
`a graphical user interface (GUI). The GUI may include
`graphical and textual elements representing the information
`and actions available to the user. For example, the touch
`screen may allow a user to move an input pointer or make
`selections on the GUI by simply pointing at the GUI on the
`display 110.
`[0042] As depicted in FIG. 1A, the GUI can be adapted to
`display a program application that requires text input. For
`example, a chat or messaging application is depicted. For
`such an application, the display can be divided into two basic
`areas. A first area 112 can be usedto display information for
`the user, in this case, the messagesthe user is sending, repre-
`sented by balloon 113a and the messageshe is receiving from
`the person he is communicating with, represented by balloon
`1135. First area 112 can also be used to show thetextthat the
`
`useris currently inputting in text field 114. First area 112 can
`also include a virtual “send”button 115, activation of which
`causes the messagesentered in text field 114 to be sent.
`[0043] A second area can be used to present to the user a
`virtual keyboard 116 that can be used to enter the text that
`appears in field 114 andis ultimately sent to the person the
`user is communicating with. Touching the touch screen at a
`“virtual key” 117 can cause the corresponding character to be
`generated in text field 114. The user can interact with the
`touch screen using a variety of touch objects, including, for
`example, a finger, stylus, pen, pencil, etc. Additionally, in
`some embodiments, multiple touch objects can be used
`simultaneously.
`[0044] Because of space limitations, the virtual keys may
`be substantially smaller than keys on a conventional key-
`board. To assist the user, “top hats” 118 can indicate to the
`user what key is being pressed. Additionally, not all charac-
`ters that would be found on a conventional keyboard may be
`presented. Such special characters can be input by invoking
`an alternative virtual keyboard. For example, a virtual key 119
`can be usedto invoke an alternative keyboard including num-
`bers and punctuation characters not present on the main vir-
`tual keyboard. Additional virtual keys for various functions
`may be provided. For example, a virtual shift key 120, a
`virtual space bar 121, a virtual carriage return or enter key
`122, and a virtual backspace key (not shown in FIG. 1A, 123
`in FIG. 1B) are provided in the illustrated embodiment.
`[0045]
`To provide more convenientuse ofcertain keys, for
`example, erasing and basic punctuation insertion, directional
`swipes (also referred to herein as “swipe gestures”) over the
`alphabetic keys can be usedas an alternative to striking cer-
`tain keys. Because the Space and Backspace keys are quite
`frequently used, they are logical candidates for the rightward
`and leftward swipes, respectively. Leftward and rightward
`swipes intuitively match the cursor travel caused by these
`symbols. Following this cursor movementanalogy, the Enter/
`Return may be invoked by a downward swipe, and a Shift/
`Caps may be invoked by an upward swipe. Alternatively, as
`the Enter and Shift functions maybe less frequently invoked,
`these may be substituted for other functions as well. One
`alternative mapping for the upward swipe, for example, could
`beto activate an alternative numeric and punctuation keypad
`or a particular punctuation symbol like an apostrophe or
`period.
`Interspersing the tedious task of typing using the
`[0046]
`small keys with swipes that do not require a specific point of
`origin can give users a nice break from the sensorimotor and
`
`cognitive load of targeting very tiny keys. Additionally, this
`can allow users keep their eyes focused and fingers on the
`central alphabetic layout, avoiding thumb contortions to hit
`space, backspace, shift and return keys that may be located in
`the bottom row or comers.
`
`[0047] An example of the usage of such a swipe gestures
`can be seen with respect to FIGS. 1A through 8. In FIG. 1A,
`the user is entering the text “ok” in response to a query
`received through a chat application. By tapping finger 124 on
`the area correspondingto the letter “o” on virtual keyboard
`116 the letter “o”is enteredin text field 114. A tophat 118 with
`a representation ofthe letter “o” is displayed providing feed-
`back to the user. As shown in FIG.1B, the user then attempts
`to tap the letter “k” using finger 124. However, the user
`inadvertently taps the letter “j” as indicated by top hat 125.
`Althoughthis typographical error could be corrected through
`the use of virtual backspace key 123, the backspace function
`can also be invoked by the rightward swipe illustrated in
`FIGS. 2A through 2C.
`[0048]
`Asillustrated in FIGS. 2A through 2C, the back-
`space function can be invokedwith a leftward swipeoffinger
`124. As described in greater detail below, the swipe can
`include a touchdownoffinger 124 in the virtual keyboard 116
`followed by a sliding leftward motion in contact with the
`touch-sensitive surface (FIG. 2B). After the leftward motion,
`the finger is lifted (FIG. 2C) and the backspace function is
`executed, removing the erroneously typedletter “.” Option-
`ally, a top hat 126 can be used to indicate to the user that the
`backspace function was performed. Other visual feedback,
`such as highlighting or flashing the key corresponding to the
`backspace gesture can also be used. The top hats can also have
`graphic arrowsor other effects indicating and reminding the
`user of the direction of swipe that can activate that key.
`[0049]
`Asillustrated in FIG. 3, the user can now enter the
`“k” of the “ok” by tapping the appropriate region of virtual
`keyboard 116. A top hat 127 indicates thattheletter “k” was
`pressed, as well as the appearance of the “k”character in text
`field 114. A space following the “ok” maybe entered using a
`rightward swipe as illustrated in FIGS. 4A through 4C. Like
`the leftward swipe, the rightward swipe comprises a touch-
`downoffinger 124 in virtual keyboard 116 (illustrated in FIG.
`4A). This touchdownis followed by a rightward swipe across
`the virtual keyboard area 116 (illustrated in FIG. 4B). After
`liftoff of the finger, a space appears in text field 114, and a top
`hat 128 (or other visual feedback as discussed above) can be
`used to indicate that the space was entered.
`[0050] Use of a downward swipeto enter a carriage return
`and new line is illustrated in FIGS. 5A through 5C. Again, the
`downward swipe includes a touchdownofthe finger 114 in
`virtual keyboard 116 (illustratedin FIG. 5A). This touchdown
`is followed by a downward motion of the finger across the
`virtual keyboard(illustrated in FIG. 5B). The insertion of the
`carriage return and newlineafterliftoff of the fingeris indi-
`catedto the user by the appearanceofthe newlinein text field
`114 and the appearance of top hat 129 indicating that the
`return was invoked(illustrated in FIG. 5C).
`[0051] Use of an upward swipe to invoke a shift for the
`entry of capital letters is illustrated in FIGS. 6A through 6D.
`Asillustrated in FIG. 6A, the swipe gesture starts with a
`touchdown offinger 124 in virtual keyboard area 116. The
`touchdownis followed by an upwardslide across the virtual
`keyboard area 116, as illustrated in FIG. 6B. As illustrated in
`FIG. 6C, liftoff of the finger following the upward slide
`invokesthe shift, as indicated by display of the top hat 130. A
`
`18
`
`18
`
`

`

`US 2008/0316183 Al
`
`Dec. 25, 2008
`
`subsequenttap ofa virtual key(e.g., the letter “c’’) results in
`a capital “C” being enteredin thetext field 114.
`two upward
`[0052]
`In a variation of this arrangement,
`swipes can invoke a Caps Lock,
`i.e., a shift that applies
`capitalization to a sequence of typedletters. The two upward
`swipes can be performed sequentially or can be performed
`simultaneously, i.e., as a two-fingered upward swipe. Such a
`scenario is illustrated in FIG. 7. Other numbers of swipes or
`numbers of fingers could also be used. Alternatively, an
`upwardstroke can be usedto invoke an alternative keyboard.
`As noted above, small virtual keyboards may not include
`numeric and punctuation characters, so an upward swipe as
`described above could cause a second virtual keyboard 131 to
`be displayed, the secondvirtual keyboard including numeric
`and punctuation characters or other special characters such as
`alternative alphabets, special symbols, etc.
`[0053] The examples above have focused primarily on the
`use of single-finger swipe gestures. However, other numbers
`of fingers could also be used to implement these functions.
`Additionally, multiple-finger swipes could be used to imple-
`ment additional functions. For example, a single-fingerleft-
`ward swipe could be used to invoke a backspace key, which
`would delete a previous character. A two-finger leftward
`swipe could be used to delete a previously entered word. A
`three-finger leftward swipe could be usedto delete a line. (In
`each case, the numberoffingers can vary.) Similarly, a single
`finger rightward swipe could be used to invoke a space, while
`atwo-finger rightward swipe could be usedto invoke a period
`and a space, ending a sentence. Alternatively, a two-finger
`rightward swipe could be used to accept a word completion
`suggestion offered by the application. Similarly, a three-fin-
`ger rightward swipe could be used to display a listing of
`possible word completions. (Again, the numberoffingers can
`vary.)
`[0054] Multi-finger swipes can also apply to swipes in the
`upward and downward directions. For example, a single-
`finger upward swipe could invoke a shift that applies only to
`the next entered character. A two-finger upward swipe could
`invoke a caps lock modeor an accented character mode(e.g.,
`for foreign languages). A three-finger upward swipe could
`invoke an accent mode or an alternative keypad mode(e.g.,
`punctuation, numeric, accented, etc.). Similarly, a single-
`finger downward swipe can invoke a newline, carriage return,
`or enter. A two-finger downward swipe can invoke a period
`(or other punctuation). A three-finger downward swipe can
`invoke a punctuation pad. (As above, the numberoffingers in
`each case can vary.)
`[0055] As will be appreciated, numerous other variations
`along these lines are also possible. Additionally, an interface
`can be provided to permit a user to customizethe activity that
`will be performed in response to various swipe gestures.
`[0056] Having described a user’s interaction with the sys-
`tem, the operation of such a system may be understood with
`reference to FIGS. 9 and 10. FIG. 9 depicts a block diagram of
`a system implementing various aspects of the present inven-
`tion. In FIG. 9, data structures and signals are represented
`with elliptical blocks. As would be understoodbyoneskilled
`in theart, these data structures and signals can take a variety
`offorms depending on a particular implementation.In FIG.9,
`processors for the data and signals are depicted with rectan-
`gular blocks. These processors can be implementedin hard-
`ware, software, firmware, or combinations thereof. Examples
`of data structures, signals, hardware, software, and firmware
`
`that can be used to implement the teachings of the present
`disclosure appear in the various references incorporated
`above.
`
`[0057] As shown in FIG. 9, a handheld device 100 can
`incorporate a touch screen 110. The touch screen detects
`touch events, which can be encoded in the form of touch
`image data 901. This touch image data is submitted to touch
`imageprocessor 902. Details regarding touch image acquisi-
`tion and processing are described in U.S. Pat. No. 6,323,846,
`which incorporated by reference. For present purposes, it is
`sufficient to understandthatthe processing ofthe touch image
`results in one or more finger path tracking events 903. These
`finger path tracking events can be submitted to two separate
`processes, keyboard tap recognizer 904 and keyboard slide
`recognizer 905.
`[0058] Keyboard tap recognizer 904 can serveto translate
`taps on the virtual keyboard surface in to key tap events 906.
`These key tap events are submitted to keyboard input man-
`ager 907. Keyboard input manager 907 interprets key tap
`events 906 and generates text events 908 that are sent to the
`applications, e.g.,
`the entry of letters into text fields as
`described above. Keyboard input manager907 also generates
`feedback popup graphics 909, e.g., the top hats showing
`whichletter has been tapped that were described above.
`[0059] Keyboard slide recognizer 905 can serve to recog-
`nize the sliding motionsthat distinguish keyboard taps from
`swipe gestures. If a swipeis detected, the tap that would have
`been recognized at the initial contact position of the swipe
`gesture must be cancelled. This can be accomplished by send-
`ing cancel tap signal 910 to keyboard tap recognizer 904.
`Keyboardslide recognizer 905 also generates keyboard swipe
`events 911 that can be submitted to keyboard input manager
`907. As in the case of key taps, keyboard input manager 907
`can generate text events 908 as well as pop up graphics 909
`that correspondto the detected swipes.
`[0060]
`FIG. 10 shows a combinedflow chart for an imple-
`mentation of keyboard tap recognizer 904 and keyboardslide
`recognizer 905. In block 1001, a finger path eventis retrieved.
`In block 1002 it is determined if the new path event corre-
`sponds to a new touch,e.g., a finger that has just appeared on
`the surface. If so, the touchdown location and time are cap-
`tured (block 1003), and a path data structure 1004 containing
`this informationis created. Ifthe finger path event is nota new
`touch, a preexisting p