as) United States
`a2) Patent Application Publication 10) Pub. No.: US 2008/0024459 Al
`(43) Pub. Date:
`Jan. 31, 2008
`Poupyrevet al.
`
`
`US 20080024459A1
`
`(54) APPARATUS AND METHOD FOR TOUCH
`SCREEN INTERACTION BASED ON
`TACTILE FEEDBACK AND PRESSURE
`MEASUREMENT
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`GO6F 3/041
`
`(2006.01)
`
`(75)
`
`Inventors:
`
`(52) US. Ch. cic cccccccceceteceesstteeeeetseees 345/173; 715/700
`
`Ivan Poupyrey, Tokyo (JP);
`Shigeaki Maruyama, Kanagawa
`JPce)
`
`ABSTRACT
`(57)
`:
`:
`:
`.
`An apparatus includes a display section with a touch screen,
`and the touch screen is adapted to display at least one
`graphical user interface object and detect a touch position on
`the touch screen. The apparatus has a haptic feedback
`generating unit attached to the touch screen and is adapted
`to generating haptic feedback. A pressure sensing unit is
`attached to the touch screen and adapted to detect pressure
`applied to the touch screen. A controller section is adapted
`to control and drive the display section. The graphical user
`interface object displayed on the touch screen has a plurality
`of logical states. The controller section determines a current
`logical state of the graphical user interface object and a form
`of the haptic feedback to be generated depending on the
`
`Jul. 31, 2006|IP) ooeee 2006-208047 detected touch position.
`
`Correspondence Address:
`REESEek MCCLELLAND MAIER &
`1940 DUKE.STREET
`ALEXANDRIA, VA 22314
`.
`:
`:
`(73) Assignee:
`Sony Corporation, Tokyo (IP)
`21)
`Appl. No.:
`11/831.703
`(21)
`Appl.
`No
`,
`(22)
`Filed:
`Jul. 31, 2007
`
`(30)
`
`Foreign Application Priority Data
`
`hardware aecreereareeeeereseeesekeeeeeees
`
`Aeeeecerereee
`
`[ | I { I I | j i I } I1 I I | I I I | | I { I ] I I | | I } ] I | I l | j ! i t
`
`1
`
`APPLE 1028
`
`
`
`eeeywee
`
`
`
`g
`
`
`Signal
`
`
`generation
`unit
`
`
`
`
`
`
`
`Tactile interface
`controller
`Display
`Visual display
`
`
`controller
`unit
`
`
`
`“3
`2D position
`
`Graphical User
`sensing unit
`2Position
`
`
`
`
`
`Interface
`controller
`
`
`
`
`controller
`
`
`Fressure
`
`Pressure ;
`
`
`sensing unit
`
`
`sensing unit
`
`
`
`controller
`
`
`Display/input
` Application/
`
`APPLE 1028
`
`1
`
`

`

`Patent Application Publication
`
`Jan. 31,2008 Sheet 1 of 11
`
`US 2008/0024459 Al
`
`
`
`0 D
`
`Application!
`
`Figure 1
`
`
`
`
`ata storage
`Signal
`
`Tactile waveshapes
`generation
`
`
`
`unit
`
`
`
`
`
`Tactile interface
`Display
`Visualdisplay
`controller
`
`controller
`iunit
`
`
`
`
`
`
`
`
`2D position
`
`;
`'
`‘
`2D position
`
`
`Graphical User
`sensing unit
`;
`controller
`sensing unit
`interface
`
`
`
`
`
`controller
`
`
`
`
`
`Fressure
`sensing unit
`
`
`
`contraller
`
`
`
`Pressure
`sensing unit
`
`[_
`
`hardware AeeeteeeeDe
`
`Display/input
`
`2
`
`

`

`Patent Application Publication
`
`Jan. 31,2008 Sheet 2 of 11
`
`US 2008/0024459 Al
`
`drag ——> lift off oY"
`
`L hold ———>lift off
`
`stop
`
`Figure 2
`
`3
`
`

`

`Patent Application Publication
`
`Jan. 31,2008 Sheet 3 of 11
`
`US 2008/0024459 Al
`
`203
`(BR
`
`203a 203b
`
`C)
`
`200
`
`Figure 3
`
`4
`
`

`

`Patent Application Publication
`
`Jan. 31,2008 Sheet 4 of 11
`
`US 2008/0024459 Al
`
`
`
`Tracking user
`finger/pen
`position on
`the screen
`
`
` Is finger/pen
`inside of the GUI
`element?
`
`
`
`
`
`
`Is pressing
`
`event recognized?
`
`
`
`
`Provide tactile feedback
`Provide
`for pressing event
`
`
`appropriate tactile
`
`feedback
`
`depending on
`
`position, applied
`
`pressure, the state
`
`
`
`of the GU
`Activate GUI element,
`
`elements and
`send appropriate events
`
`
`
`and perform other
`application
`
`
`appropriate actions
`
`
`
`Figure 4
`
`5
`
`5
`
`

`

`Patent Application Publication
`
`Jan. 31, 2008 Sheet 5 of 11
`
`US 2008/0024459 Al
`
`vor
`
`|#(9)B(q)LZe—t|Bprousautgl5ploysayyispoysenyjy&°)
`yemoeuonenjoeUOReNjoe:
`
`
`
`ggunbiy
`
`6
`
`
`

`

`Patent Application Publication
`
`Jan. 31,2008 Sheet 6 of 11
`
`US 2008/0024459 Al
`
`331
`
`$30
`
`
`userSlides fingerinto
`GUI element with
`pressure p > pt
`
`
`
`
`Tracking user
`fingerpen
`
`positionon
`the device
`
`
` Is finger/pen
`inside of the GUI
`element?
` Provide appropriate
`dependinfron position,
`theStatefftheGUI
`pplied
`pressure,
`
`tactile feedback
`
` P
`
`
`Activate GU! element
`send appropriate events
`
`and perform other
`appropriate actions
`
`
`
`
`
`Figure 6
`
`7
`
`

`

`Patent Application Publication
`
`Jan. 31, 2008 Sheet 7 of 11
`
`US 2008/0024459 Al
`
`cOS-
`
`00S:
`
`GOS
`
`
`+LM) take bl.sahad-2,
`5
`EPipaovitealarena
`i
`eas|laren,
`7
`5
`
`it?.RIOTCeteerweedbeaeearireeee
`
`adhe
`eerrnera S
`H
`
` Leia dhdel ofA)
`
`£aanits
`
`(P)
`
`(9)
`
`8
`
`
`
`
`
`

`

`Patent Application Publication
`
`Jan. 31,2008 Sheet 8 of 11
`
`US 2008/0024459 Al
`
`userslidesfingerinto
`GUI elementwith
`pressure po
`
`
`
`800
`
`803
`
`pressure
`
`Tracking user
`finger/pen
`position and
`pressure
`
`804
`
`mean
`
`
`
`
`
`
`ls finger/pen
`inside StithGUI
`
`
`
`
`element?
`
`
`
`
`Provide tactile feedback
`for acutation event
`
`890
`
` Is actuate event
`recognized?
`
`
`
`Pressure
`changed more than
`
`
`threshold yalue?
`
`| psp! [>d
`
`
`Activate GUI element
`
`
`send appropriate events
`
`and perform other
`
`appropriate actions
`
`pressure
`
`
`
`Provide appropriate
`tactile feedback.
`depending on position,
`
`applied pressure,
`the state of the GUI
`elements and application
`
`
`
`
`
`Figure 8
`
`9
`
`

`

`Patent Application Publication
`
`Jan. 31,2008 Sheet 9 of 11
`
`US 2008/0024459 Al
`
`userslides fingerinto
`GUI element
`
`900
`
`Remember
`position
`
`902
`
`903
`x8=x°
`yS=y°
`
`904
`
`Tracking user
`finger/pen
`position and
`pressure
`
`
`
`
`
`
` ~ Is finger/pen
`
`inside of the GUI
`element?
`
`recognized?
`
` Is actuate event
`
`
`Provide tactile feedback
`
`
`Position
`970
`for acutation event
`
`
`changed more than
`
`
`
`threshold value?
`
`
`
`Activate GUI element,
`
`send appropriate events
`
`
`
`922
`Remember
`and perform other
`
`
`
`appropriate actions
`
`current position
`
`
`
`Provide appropriate
`tactile feedback
`depending on position,
`
`applied pressure,
`
`
`the state of the GUI
`elements and application
`
`
`
`
`Figure 9
`
`10
`
`

`

`Patent Application Publication
`
`Jan. 31,2008 Sheet 10 of 11
`
`US 2008/0024459 Al
`
`(c)
`
`Figure 10
`
`11
`
`11
`
`

`

`Patent Application Publication
`
`Jan. 31,2008 Sheet 11 of 11
`
`US 2008/0024459 Al
`
`owe saw eweweerensw eee ewe eo =
`
`confirmation
`
`
`pressure
`
`
`4
`
`time
`
`Figure 11
`
`12
`
`

`

`US 2008/0024459 Al
`
`Jan. 31, 2008
`
`APPARATUS AND METHOD FOR TOUCH
`SCREEN INTERACTION BASED ON
`TACTILE FEEDBACK AND PRESSURE
`MEASUREMENT
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`[0001]
`[0002] The present invention relates to a method of a user
`interface utilizing a touch screen andtactile feedback, and an
`apparatus that employs such a user interface method.
`[0003]
`2. Discussion of the Related Art
`[0004]
`Japanese Patent Application Publication No. 2003-
`016502 discloses an example of such a user interface system
`for detecting a position of a user’s finger or a pointing device
`on the touch screen of a display device. In this user interface
`system, the tactile feedback are provided by vibrating the
`touch screen when the user touches one of graphical user
`interface objects displayed on the touch panel. A function-
`ality assigned to the selected graphical user interface object
`is actuated whenthe userreleases or detouchesthe finger or
`pointing device from the touch screen.
`[0005]
`Japanese Patent Application Publication No. 2005-
`190290 discloses another example of a user interface system
`capable of providing tactile feedbacks when a user touches
`on a touch screen. In this user interface system, the initial
`tactile feedback is provided when the user first touches the
`touch panel, and a different tactile feedback is provided
`whenthe touch position is moved to a region of the touch
`screen where a graphical user interface object is displayed.
`A function assigned to the selected graphical user interface
`object is actuated when the user detouches the finger or
`pointing device or presses for a longer period of time. The
`actuation of the selected graphical user interface object is
`notified to the user in a form of tactile feedback, color
`change of the graphical user interface object, sound or
`combination thereof.
`
`[0006] Minsky, M., “Manipulating simulated objects with
`real-world gestures using force and position sensitive
`screen”. Proceedings of SIGGRAPH’84. 1984: ACM:pp.
`195-203 discloses still another example of a user interface
`system in which a pressure sensoris added to a touch screen
`for detecting pressure applied to the touch screen, allowing
`more flexibility in the user interface operation.
`
`SUMMARY OF THE INVENTION
`
`It is desirable to providetactile notification when a
`[0007]
`user touches a user interface element on a touch screen
`
`without executing functionality of the user interface ele-
`ment. Furthermore, it is desirable to provide tactile notifi-
`cation to the user whenthe functionality of the user interface
`elementis executed.
`
`Furthermore, it is desirable to provide a method of
`[0008]
`user interfaceutilizing a touch screen display device capable
`of providing tactile feedback and measuring pressure
`applied to the touch screen, thereby allowing a user to have
`interactive operations similar to ones with physical opera-
`tion means, such as pressing buttons or keys. Further, it is
`also desirable to provide an apparatus that employs such a
`user interface method.
`
`invention is made in view of the
`[0009] The present
`forgoing issues described above.
`[0010]
`In an embodimentof the present invention, there is
`provided an apparatus including a display section with a
`
`touch screen. The touch screen is configured to display at
`least one graphical user interface object and detect a touch
`position on the touch screen. The touch position is inputted
`with a user’s finger or a pointing device. The apparatus
`includes: a haptic feedback generating unit attached to the
`touch screen and generating haptic feedback; a pressure
`sensing unit attached to the touch screen and detecting
`pressure applied to the touch screen; and a controller section
`configured to control and drive the display section. The
`graphical user interface object displayed on the touch screen
`has a plurality of logical states. The controller section
`determines a current
`logical state of the graphical user
`interface object using a history of detected touch positions
`and a history of detected pressure values. The controller
`section determines a form of the haptic feedback to be
`generated depending on (i) the detected touch position, (11)
`the detected pressure value and (1i1) the determined current
`logical state of the graphical user interface object.
`[0011]
`In another embodiment of the present invention,
`the haptic feedback generating unit may generate different
`tactile feedback for different logical states of the GUIobject.
`[0012]
`In another embodiment of the present invention,
`the logical states of the GUI object may include at least a
`selected state and an actuated state. The controller section
`
`may determine that the graphical user interface object is in
`the actuated state if a pressing event is recognized. The
`controller section may recognize the pressing event using a
`history of the detected pressure value. Alternatively, the
`controller section may determine that the GUI object is in
`the actuated state if:
`(1) the touch position is inside of the
`GUIobject; and (i1) the detected pressure is more than a
`preset actuation threshold value. In another example, the
`controller section may determine that the GUI object is in
`the actuated state if:
`(1) the touch position is inside of the
`GUI object; and (ii) a history of the detected pressure
`satisfies a preset actuation condition. In the present embodi-
`ment, the logical state of GUI object is allowed to change to
`the actuated state only after the selected state.
`[0013]
`In another embodiment of the present invention,
`the haptic feedback generating unit may include a single or
`plurality of piezoelectric elements. At
`least one of the
`piezoelectric elements may be used for generating the haptic
`feedback and detecting the pressure applied by the user.
`Alternatively, the at least one of the piezoelectric elements
`may generate the haptic feedback and detect the pressure in
`time sharing manner.
`[0014]
`In another embodiment of the present invention,
`the haptic feedback is controlled in either a frequency, an
`amplitude or both amplitude and frequency simultaneously.
`[0015]
`In another embodiment of the present invention,
`the haptic feedback generating unit may generate a continu-
`ous haptic feedback as long as the touch position is inside of
`the GUI object. Further, the continuoustactile feedback is
`changed in response to a change of the pressure applied to
`the touch screen. The change of the continuous tactile
`feedback dependson the current logical state of the graphi-
`cal user interface object.
`[0016]
`In another embodiment of the present invention,
`the haptic feedback generating unit may generate a single
`burst of the haptic feedback when the touch position crosses
`over a hotspot predefined within the GUI object. Alterna-
`tively, the haptic feedback generating unit may generate a
`
`13
`
`13
`
`

`

`US 2008/0024459 Al
`
`Jan. 31, 2008
`
`single burst of the tactile feedback when the touch position
`or the detected pressure changes more than a preset thresh-
`old value
`
`with physical operation means. Further, according to the
`present invention, an apparatus that employs such a user
`interface method is provided.
`
`In another embodiment of the present invention,
`[0017]
`the GUI object may be formed with a plurality of sub-
`elements, and the haptic feedback generating unit may
`generate different tactile feedbacks for different sub-ele-
`ments thereof.
`
`In another embodiment of the present invention,
`[0018]
`the controller section may determine that the GUIobjectis
`in the activated state by using a plurality of pressure thresh-
`olds.
`
`In another embodiment of the present invention,
`[0019]
`the controller section may differentiates a stronger push and
`a lighter push based on a noise level of a signal output from
`the touch screen or a circuitry thereof, the stronger push
`corresponding to the pressing event, the lighter push corre-
`sponding to sliding of the user’s finger or pointing device.
`[0020]
`In another embodiment of the present invention,
`the display section may generate visual feedback in corre-
`lation with the haptic feedback.
`[0021]
`In another embodimentof the present invention, a
`graphical user interface method for a touch screen is pro-
`vided. The method includes: displaying a graphical user
`interface object on the touch screen,
`the graphical user
`interface object having a plurality of logical states; detecting
`a touch position on the touch screen, at which a user’s finger
`or a pointing device is touching; detecting pressure applied
`on the touch screen whenthe touch position is detected; and
`generating haptic feedback in response to the touching, a
`form of the haptic feedback being determined depending on
`(i) the detected touch position,
`(11) the detected pressure
`value and (iii) a current logical state of the GUI object. The
`current logical state of the GUI object is determined by using
`a history of detected touch positions and a history of
`detected pressure values.
`[0022]
`In the embodiments of the present invention, the
`form ofthe haptic feedback is determined depending on the
`touch position, the pressure applied by the user and the
`current logical state of the graphical user interface object.
`Accordingly, various forms of the haptic feedback may be
`provided for different logical states of the graphical user
`interface object, making it easy for the user to know the
`current state of the graphical user interface object.
`
`ADVANTAGES OF THE INVENTION
`
`[0023] The present invention makesit possible to provide
`tactile notification when a user touches a user interface
`
`element on a touch screen without executing functionality of
`the user interface element, andtactile notification to the user
`when the functionality of the user interface element
`is
`executed.
`
`Furthermore, according to the present invention, a
`[0024]
`method of user interface utilizing a touch screen display
`device capable of providing tactile feedback and measuring
`pressure applied to the touch screen is provided. The method
`allows a user to have interactive operations similar to ones
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0025] Amore complete appreciation of the invention and
`many of the attendant advantages thereof will be readily
`obtained a the same becomesbetter understood by reference
`to the following detailed description when considered in
`connection with the accompanying drawings, wherein:
`[0026]
`FIG. 1 is a block diagram showing an example of
`an apparatus configuration according to an embodimentof
`the present invention;
`[0027]
`FIG. 2 is a schematic diagram showing an example
`of interaction with touch screens of prior art;
`[0028]
`FIG. 3 is an explanatory illustration of a user
`interface method according to an embodimentofthe present
`invention for a case where a user finger slides over a GUI
`object;
`FIG. 4 is a flow chart showing steps of a user
`[0029]
`interface method according to an embodimentofthe present
`invention;
`[0030]
`FIG. 5(a) is an explanatory illustration of a user
`interface method according to another embodiment of the
`present
`invention for a case where a pressing event
`is
`recognized;
`[0031]
`FIG. 5(6) is an explanatory illustration of a user
`interface method according to another embodiment of the
`present
`invention for a case where a pressing event
`is
`recognized;
`[0032]
`FIG. 5(c) is an explanatory illustration of a user
`interface method according to another embodiment of the
`present
`invention for a case where a pressing event
`is
`recognized;
`[0033]
`FIG. 6 is a flow chart showing steps of a user
`interface method according to another embodiment of the
`present invention;
`[0034]
`FIG. 7(a) is a schematic diagram showing an
`example of hotspot in a GUI object;
`[0035]
`FIG. 7(6)
`is a schematic diagram showing an
`example of hotspot in a GUI object;
`[0036]
`FIG. 7(c)
`is a schematic diagram showing an
`example of hotspot in a GUI object;
`[0037]
`FIG. 7(d)
`is a schematic diagram showing an
`example of hotspot in a GUI object;
`[0038]
`FIG. 8 is a flow chart showing steps of a user
`interface method according to still another embodiment of
`the present invention;
`[0039]
`FIG. 9 is a flow chart showing steps of a user
`interface method according to another embodiment of the
`present invention;
`[0040]
`FIG. 10(a) is an explanatory illustration of a user
`interface method according to an embodimentofthe present
`invention for a slider-type GUI object;
`[0041]
`FIG. 10(8) is an explanatory illustration of a user
`interface method according to an embodimentofthe present
`invention for a slider-type GUI object;
`[0042]
`FIG. 10(c) is an explanatory illustration of a user
`interface method according to an embodimentofthe present
`invention for a slider-type GUI object;
`[0043]
`FIG. 10(@) is an explanatory illustration of a user
`interface method according to an embodimentofthe present
`invention for a slider-type GUI object; and
`
`14
`
`14
`
`

`

`US 2008/0024459 Al
`
`Jan. 31, 2008
`
`FIG. 11 is a graph showing changes of pressure
`[0044]
`applied by a user with time during a pressing event of a user
`interface method according to another embodiment of the
`present invention.
`
`DESCRIPTION OF PREFERRED
`EMBODIMENTS
`
`[0045] Below embodiments of the present invention will
`be described with reference to the accompanyingfigures. In
`the following description, some terminology is used to
`describe certain characteristics of the present invention.
`[0046] The term “touch screen” is a transparent screen-
`type position sensing device capable of detecting a touch
`position on the screen surface, at which a user’s finger or any
`pointing device is touching.
`[0047] The term “logical states of a graphical user inter-
`face object” means distinct states of a graphical user inter-
`face object, by which different corresponding operations or
`processing are triggered. The logical states includes at least
`a selected state which indicates the graphical user interface
`object is selected by a user but none of the corresponding
`operation or operation is triggered, and an actuated state in
`which the corresponding operations or processing is per-
`formed.
`
`FIG. 1 shows an example of an apparatus to which
`[0048]
`a user interface method according to an embodimentof the
`present invention is applied. The apparatus 1 includes a
`display/input section 10, a controller section 20 and an
`application section 30.
`[0049] The display/input section 10 displays on a touch
`screen thereof imagesof buttons, keys, switches or any other
`Graphic User Interface (GUI) objects to prompt a user 2 to
`interact with the apparatus 1. The display/input section 10
`further detects a touch position of a user’s finger or a
`pointing device on the screen and pressure applied when the
`finger or pointing device touches the screen. The display/
`input section 10 further provides different types of tactile
`feedback in response to the user’s input operation.
`[0050]
`It should be noted, in this specification, that the
`word “tactile” and “haptic” indicate the same sensory effect
`and are used interchangeably.
`[0051] The control section 20 dynamically correlates: (i)
`the touch position on the screen or a GUIobject selected by
`the user’s input operation; (11) the pressure applied on the
`screen by the user’s input operation; and (ili) a current
`logical state of the selected GUI object; with the type of
`tactile feedback to be presented to the user 2.
`[0052] The application section 30 performs various opera-
`tions or functions in responseto the user’s input operation
`detected by the display/input section 10. The application
`section 30 may include various applications and software
`units or hardware.
`
`(1) Display/Input Section
`
`[0053] The display/input section 10 includes a haptic
`feedback generating unit 102, a visual display unit 103, a
`two-dimensional (2D) position sensing unit 104 and a pres-
`sure sensing unit 105.
`[0054] The visual display unit 103 presents visual infor-
`mation to the user 2. Such visual information may include
`various predefined GUI objects that user can interact with,
`such as images of buttons, sliders, drawing, scroll bars,
`hyper links and etc. The visual display unit 103 may be
`
`formed with any type of display as long as it can be used
`with the tactile feedback generating unit 102, the 2D posi-
`tion sensing unit 104 and the pressure sensing unit 105. For
`example, a Liquid Crystal Display (LCD), a Organic Light
`Emitting Diode (OLED) display or
`the like may be
`employed as the visual display unit 103.
`[0055] The haptic feedback generating unit 102 may be
`formed with piezoelectric bimorph actuators with single or
`multiple layer structure. Examples of such actuators for
`generating the tactile feedback are disclosed in Japanese
`Patent Application Publication No. 2006-48302. Alterna-
`tively, various types of mechanicalor electrical or electro-
`magnetic actuators/motors may be employed to generate the
`tactile feedback depending on a size/mass of the display
`and/or available power.
`[0056] The pressure sensing unit 104 allows to measure
`pressure applied to the touch screen by the user’s input
`operation. In the present embodiment, various types of
`pressure sensing units may be employed as long as such
`devices can measure the pressure of the user’s touch with a
`predetermined resolution and be incorporated in the display/
`input section 10 with other units 102-104. For example, a
`force sensitive circuit elements such as strain gauges or
`pressure sensitive resistors may be used to sense the force
`which the touch screen memberexerts on each support of the
`touch screen when finger pressure is applied to the member.
`[0057] Alternatively, the piezoelectric actuators may be
`used to measurethe pressure applied to the touch screen. For
`example, the piezoelectric actuators may be connected with
`a drivercircuit and a detector circuit so as to use someof the
`
`actuators for generating the tactile feedback and the others
`for measuring the pressure applied thereon, respectively. An
`example of such a pressure sensing unit formed with the
`piezoelectric actuators is disclosed in Japanese Patent Appli-
`cation Publication No. 2006-48302. Alternatively, the driv-
`ing of the actuators and measuring of the pressure may be
`performed time sharing manner. Morespecifically, a single
`actuator may be used with a switching circuit for measuring
`pressure and generating the feedback.
`[0058] The 2D position sensing unit 105 detects where the
`user is touching on the touch screen. Any type of a touch
`screen or touch panel technology may be used as the 2D
`position sensing unit 105 as long as the touch screen/panel
`can measure two-dimensionalposition of the user’s finger or
`pointing device. For example, a resistive touch screen, a
`capacitive touch screen, a surface acoustic wave touch
`screen, or the like may be used.
`
`(2) Controller Section
`
`[0059] The controller section 20 drives and controls sub-
`sections of the display/input section 10 in response to the
`user’s input operation detected by the display/input section
`10. The controller section 20 controls the display/input
`section 10 to change the tactile feedback depending on the
`position, pressure of the user’s touch on the screen and the
`current logical state of GUI object, attempting to simulate
`the interactive operations with physical interface objects.
`Accordingly,
`the apparatus of the present embodiment
`allows the user to easily and intuitively perform input
`operations even without the physical user interface objects.
`[0060] The control section 20 and the application section
`30 may be embodied with a computer (not shown in the
`figure), which may include a CPU, a memory, an external
`data storage, and an input/outputinterface. Various functions
`
`15
`
`15
`
`

`

`US 2008/0024459 Al
`
`Jan. 31, 2008
`
`performed by sub-sections of the control section 20 and the
`application section 30 may be realized by executing corre-
`sponding software installed in the computer, or adding
`dedicated circuitry or hardware to the computer. The appli-
`cation section 30 may include any application software or
`hardware that may be controlled in response to the user’s
`input operations detected by the display section 20.
`[0061] The software may be installed into the computer
`via a recording medium or a carrier signal. The software may
`also be installed by downloading from a server on a network
`or Internet through wired or wireless connection.
`[0062] The controller section 20 includes a signal gener-
`ating unit 106, a display controller 107, a two-dimensional
`(2D)position sensing unit controller 108, a pressure sensing
`unit controller 109, a data storage 110, a tactile interface
`controller 111, and a graphical user interface (GUI) control-
`ler 112.
`
`[0063] The signal generating unit 106 generates and pro-
`vides a signalto the tactile feedback generating unit 102 for
`driving tactile feedback generating elements or the piezo-
`electric actuators. The signal may be a voltage function of
`time, with amplitude, shape and period changed in response
`to the position and/or pressure of the user’s input operation
`on the screen of the display/input section 10. Examples of
`output signal are a square wave, sinusoidal and so on.In the
`present embodiment, the type of signal is not limited to the
`above-described examples. Other signals may also be
`employed providing that the signal can be used to generate
`and change the tactile feedback in response to the user’s
`input operation.
`[0064] The tactile feedback generating unit 102 receives
`the input signal and converts the input signal into force
`patterns that are transmitted to the user 2 via a mechanical
`assembly that combines the screen with thetactile feedback
`generating elements or the piezoelectric actuators. The user
`2 can feel the force patterns when the user 2 is touching the
`screen.
`
`Forthe user’s input operation, a pen-type pointing
`[0065]
`device may be used for selecting an image on the screen
`instead of the user’s own finger. The user input to the
`apparatus 1 may be detected using a) touch screen technol-
`ogy where the user 2 can directly touch the screen with their
`fingers, or b) pen input technology where the pen-type
`devices are used to report a position where the user 2 is
`touching the screen.
`[0066] The pressure sensing unit controller 109 deter-
`mines the value of pressure applied whenthe user is touch-
`ing on the screen. The position sensing unit controller 108
`determines the position where the user 2 is touching on the
`screen. The determined data is communicated to the GUI
`controller 112.
`
`[0067] When the user presses the screen and at the same
`time tactile feedback is provided to the user, the pressure
`signal will have a component from the tactile feedback
`signal. This signal may be filtered out because the exact
`shapeofthe signal is known. Alternatively, the pressure may
`be measured only at the point of time when nothetactile
`feedback is provided, i.e. when the tactile feedback wave-
`shape is at zero value.
`[0068] The GUI controller 112 determines which GUI
`object the user 2 is intending to interact with. Further,
`depending on a) the current state of the GUI object and b)
`pressure value applied to the GUI object, the GUI controller
`112 determines an appropriate change in the state of the GUI
`
`object. For example, if the GUI object is a graphical button,
`the GUI controller 112 can calculate is there was enough
`pressure applied on the graphical button on the screen to
`change the state of the button from “free” to “pressed” or
`“non-actuated” to “actuated”. After determining the state of
`the GUIobjects, the GUI controller 112 changesthe visual
`state of GUI object by sending commands to the display
`controller 107.
`
`[0069] Alternatively, sound or audio alarm may be gen-
`erated whenthe visualstate of the GUI object changed so as
`to inform the user 2 about the change in the state of GUI
`object
`[0070] The GUI controller 112 further sends the com-
`mandsto the tactile interface controller 111 that generates
`appropriate commandsfor driving the tactile signal genera-
`tion unit 102. The data for tactile feedback may be generated
`algorithmically by the signal generation unit 106 as well as
`stored as data 110a@ on the data storage 110. Any widely
`available data storage devices may be used as the data
`storage 110 including flash memory, ROM, hard drive as
`well as network storage. Any file systems can be used to
`organize data on the data storage 110.
`[0071]
`In another embodimentof the present invention, a
`history of the previous input and a history of the previous
`states of the GUI objects is used to determine thetactile
`feedback to the user 2.
`
`[0072] Before describing further details of embodiments
`of the present invention, a short description of touch-screen
`interaction operation of related art might be useful and
`provided below:
`[0073]
`FIG. 2 presents an example of a typical touch-
`screen interaction of related art. The interaction starts when
`the user 2 touches the screen (touch down event T1). The
`user 2 can then either drag a finger across the input space
`(drag or slide state T2) or hold it steady in one place (hold
`state T3). Further, the user 2 can lift the finger off the screen,
`which can happen either from inside of the GUI object (lift
`off in event T4) or from outside of the GUI object (lift off
`out event T5).
`[0074] Therefore, each GUI object can be in the following
`states: 1) neutral; 2) selected: that is when the user 2 selects
`the GUIobject by touching it, such as placing the finger or
`pen-type device inside of the GUI object; and 3) activated,
`that is when the user 2 indicates that the GUI object should
`execute an associated command, corresponding to pressing
`of a physical button. In additional to these states, the GUI
`object can also be inactive, meaning that it can not be
`actuated, but it may or may not respondto the user input.
`[0075]
`It should be noted that, in the related art technol-
`ogy, the user 2 can select a GUI object and then actuate it or
`return the GUI object into the neutral state, by moving the
`finger/pen outside of the GUI object and lifting the finger/
`pen. Such interaction methodofthe related art technologyis
`different from what the user typically would do with the
`physical button while the user 2 typically presses the physi-
`cal button to actuate.
`
`Further, in all these cases the user 2 exerts a certain
`[0076]
`amount of pressure on the screen. It is desirable to provide
`different levels of the tactile feedback with a GUI object
`depending on the pressure applied on the GUI object.
`Furthermore,
`it
`is desirable to provide the user interface
`mechanism capable of realizing more intuitive interaction
`method that is closer to the interaction with the physical
`controllers.
`
`16
`
`16
`
`

`

`US 2008/0024459 Al
`
`Jan. 31, 2008
`
`the
`In an embodiment of the present invention,
`[0077]
`