as) United States
`a2) Patent Application Publication co) Pub. No.: US 2012/0176401 Al
`
` Haywardet al. (43) Pub. Date: Jul. 12, 2012
`
`
`US 20120176401A1
`
`(54) GESTURE MAPPING FOR IMAGEFILTER
`INPUT PARAMETERS
`
`(75)
`
`Inventors:
`
`David Hayward, Los Altos, CA
`(US); Chendi Zhang, Mountain
`View, CA (US); Alexandre
`Naaman, Mountain View, CA (US);
`Richard R.Dellinger, San Jose,
`CA (US); Giridhar Murthy,
`Sunnyvale, CA (US)
`yale,
`
`(73) Assignee:
`
`Apple Inc., Cupertino, CA (US)
`
`(21) Appl. No.:
`
`13/052,994
`
`(22)
`
`.
`Filed:
`
`Mar.21, 2011
`
`Related U.S. Application Data
`
`(60) Provisional application No. 61/431,718, filed on Jan.
`11, 2011.
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`(2006.01)
`GO6F 3/041
`(2006.01)
`GO09G 5/00
`(2006.01)
`G06K 9/40
`(52) US. Ch oe 345/619; 382/260; 345/173
`(57)
`ABSTRACT
`Lo.
`.
`asdisclosure pertains to systems, methods, and computer
`rea
`e medium Jor mapping particular user interactions,
`e.g., gestures, to the input parameters of various image pro-
`cessing routines, e.g., image filters, in a way that provides a
`seamless, dynamic, and intuitive experience for both the user
`and the software developer. Such techniques may handle the
`processing of both “relative” gestures, i.c., those gestures
`having values dependent on how muchaninputto the device
`has changedrelative to a previous value of the input, and
`“absolute” gestures, i.e., those gestures having values depen-
`dent only bn the instant value of the inputto the device.
`Additionally, inputs to the device beyonduser-input gestures
`may be utilized as input parameters to one or more image
`processing routines. For example, the device’s orientation,
`acceleration, and/or position in three-dimensional space may
`be used as inputs to particular image processing routines.
`
`
`%z££££££££££££££££££££££££££££££££££££££££££££££££££££££££££££££££££££££
`
`¢“££££££££££
`
`ii
`
`iz
`
`APPLE 1016
`
`APPLE 1016
`
`1
`
`

`

`Patent Application Publication
`
`Jul. 12,2012 Sheet 1 of 21
`
`US 2012/0176401 Al
`
`
`
`2
`
`

`

`Patent Application Publication
`
`Jul. 12,2012 Sheet 2 of 21
`
`US 2012/0176401 Al
`
`Figure2
`
`
`
`3
`
`

`

`Patent Application Publication
`
`Jul. 12,2012 Sheet 3 of 21
`
`US 2012/0176401 Al
`
`Figure3
`
`4
`
`

`

`Patent Application Publication
`
`Jul. 12,2012 Sheet 4 of 21
`
`US 2012/0176401 Al
`
`Figure4
`
`5
`
`

`

`Patent Application Publication
`
`Jul. 12,2012 Sheet 5 of 21
`
`US 2012/0176401 Al
`
`208
`
`Figure5
`
`6
`
`

`

`Patent Application Publication
`
`Jul. 12,2012 Sheet 6 of 21
`
`US 2012/0176401 Al
`
`Figure6
`
`Hg*
`
`7
`
`

`

`Patent Application Publication
`
`Jul. 12,2012 Sheet 7 of 21
`
`US 2012/0176401 Al
`
`mh
`
`
`
`8
`
`

`

`Patent Application Publication
`
`Jul. 12,2012 Sheet 8 of 21
`
`US 2012/0176401 Al
`
`4
`
`A DER YG)|
`
`
`
`%
`
`Figure &
`
`9
`
`

`

`Patent Application Publication
`
`Jul. 12,2012 Sheet 9 of 21
`
`US 2012/0176401 Al
`
` i
`
`"
`
`C
`
`Figure $
`
`10
`
`10
`
`

`

`Patent Application Publication
`
`Jul. 12,2012 Sheet 10 of 21
`
`US 2012/0176401 Al
`
`Figure 10A @®
`
`
`
`
`
`
`11
`
`11
`
`

`

`Patent Application Publication
`
`Jul. 12, 2012 Sheet 11 of 21
`
`US 2012/0176401 Al
`
`EH
`
`SOR 25
`
`0 HAP
`
`
`
`Aeegates
`
`» 208
`
`Stee wd
`
`see. du ang anette + meee Bee wf tee fan eng
`
`Figure 71
`
`12
`
`12
`
`
`
`

`

`Patent Application Publication
`
`Jul. 12,2012 Sheet 12 of 21
`
`US 2012/0176401 Al
`
`ANG QRGTHER DEVICE INPUTS
`
`i
`[MANAGES HO
`_ RECEIVES SELECTION GF URGE PETER) TO BE aPRUED
`f- REGHTERS HIGR LEVEL EVENT) AY REVERE 6.6, GRUBER
`~ REBUY GEYER GRENTATION ACCELERATION, BOUTON
`
`~
`
`GESTURETRAOLATIONLAYER
`
`~ CODE PRINIDED BY DEVICE GPERATING TIER
`
`
`~ PERFORRAPPING SETWEER THE GESTURES REPORTED
`BY THE CUERT APPLICATION LAYER AND THE INF
`
`PARARETERS OF THE CURRERTLY SELECTED IMAGE FLIES)
`
`~ PERFORMS MARIPULATION OF UNDERLYING INAGh DATA 1)
`EMPOGE IMAGE FILTER CPFECT OR INAGE
`
`~ SERDS HUTERED INAGE DATA 10 DRHCE DIPLAY
`
`
`
`
`
`
`
`~ FALTERED PAGE DATA
`
`13
`
`~ NAGE HETER SELECTOR
`- EVENTS
`> BEALE INNES
`
`« {RAGE FILTER URPUT
`PARAMETERS
`
`Figure 12
`
`13
`
`

`

`Patent Application Publication
`
`Jul. 12,2012 Sheet 13 of 21
`
`US 2012/0176401 Al
`
`PAARL
`
`CUTER)THHDGESTURES
`
`~ TSPUP BARANETERS: SORE
`
`- ERASE FRLTERS
`THERMAL
`RRAY
`RORHAL
`
`ANQURY, ROTAHON ANDUNT
`
`- EXARPLE PATER:
`GH? HANBEL
`
`RALEIDOMGPE
`
`1SG2—, pyreswwrryuygesrimnrs
`
`INPUT POINT
` |. NPLPARAMETERS:
` | ~ ERABPLE FLIER:
`SRETER
`
`~ HIPGT PARAMETERS:
`OREMTATON
`
`INPUN POINT, DEVICE
`
`
`
`
`
`
`
`~ ESANPEE FILTER:
`PRRDR
`
`
`PRTAGE
`
`~ FRPOT PARARETERS: INPUT POINT, INPUT
`ANOUN
`
`_ EXAMPLE FILTERS:
`SQUEHE
`
`~ BPEPARAMETERS: INPT POINT
`
`~ ERAMPLE ALTO
`
`Figure 13
`
`14
`
`14
`
`

`

`Patent Application Publication
`
`Jul. 12,2012 Sheet 14 of 21
`
`US 2012/0176401 Al
`
`4400 _
`
`MRROR
`
`KALEDERCOPE 2. LIGHT THENNEL
`
`15
`
`15
`
`

`

`Patent Application Publication
`
`Jul. 12,2012 Sheet 15 of 21
`
`US 2012/0176401 Al
`
`TRERMAL
`
`SiRETCH
`
`RALEIDOSCOPE
`
`Figure 75
`
`16
`
`16
`
`

`

`Patent Application Publication
`
`Jul. 12,2012 Sheet 16 of 21
`
`US 2012/0176401 Al
`
`a ae
`
`“EWES
`
`INAGE GYERLAT PREVIEW GRID
`
`17
`
`17
`
`

`

`Patent Application Publication
`
`Jul. 12,2012 Sheet 17 of 21
`
`US 2012/0176401 Al
`
`1700
`
`SENSORS IN COMRUNICATION WITH THE DEVICE
`
`RECERE SELECTION OF IMAGE FILTER) TH BE
`APPLIED
`
`RECENE DEVICE INPUT DBTA FROM ORE GR NORE
`
`RECENVE WO MECISTER HIGH LEVEL EVENTSAT CENCE!
`
`
`(E.G, GESTERES)
`
`
`
`
`
`ASE DEVICE INPUT DATA ANDREGISTERED EVENT BATA F
`1) PASS APPROPRIATE INPUT PARAMETERS 10
`SELECTED THAGE FILTERS
`
`PERPORN TAGE HIETRRING
`RETURN NLTERED TAGE DATA TO DEVICE BRPLAY
`
`V7I2
`
`Figure 77
`
`18
`
`

`

`Patent Application Publication
`
`Jul. 12,2012 Sheet 18 of 21
`
`US 2012/0176401 Al
`
`1800
`
`1802
`
`APPLIED
`
`RECENE DEVICE INPUT QATA FROM ONE O8 BORE
`
`
` RECEIVE SELECHION OF IMAGE HUTERS) TO BE
`SENSOR. EN CONBUNIGATION WATH THE DEVICE
`E.G. GEMURES)
`
`
`
`RECEVE AND REGISTER HIGH LEVEL EVENTS aT pew
`
`SEDEVICE INPUT DATA AND REGISTERED EVENT DATA7
`|
`10 PASS APPROPRIATE INPUT PARAMETERS 10
`|
`SELECTED IMAGE FILTERS)
`
`
`
`
`
`SELECTED IMAGE FHLTER(S) AND CURRENT DEVICE
`INPUTDATAAND GESTURE INPUN
`
` | CALCULATEIMAGEOVERLAYPREVIEWGRID FOR
`DRPLAY
`
`RETURNIMAGE OVERLAY PREVIEW GRID TO DEVICE
`
`Figure 18
`
`19
`
`

`

`Patent Application Publication
`
`Jul. 12,2012 Sheet 19 of 21
`
`US 2012/0176401 Al
`
`BEGIN DEQ BAIA CAPYORE
`
`
`APPLIED
`
`RECEIVE SELECTION QF USAGE FILTERS) 70 BE
`
`|
`|
`
`
`
`RECEHE DEVICE INPUT DATA FROM ONE OR MORE
`SERSORS IN CORRUNICATION WITH THE DEVICE
`
`
`
`
`(E.G, GESTURED
`
`RECEIVE AND REGISTER HIGH LEVEL EVENTS aT BERCE |
`
`
`
`HIE DEVICE INPUT DATAANDREGISTERED EVENT Bata b-7
`|
`TQ PASS APPROPRIATE INPUT PARAMETERS TO
`
`
`:
`SELECTEDIMAGE FILTERS)
`
`
`
`PERFORM IRAGE FILTERING ON CURRENT INAGE
`
`FRANE GF CAPTURED VIREO DATA Tada,
`
`
`TMIORE METERED IMAGE
`
` 14h SPORE ONLY VIDE
`
`
`
`7) DAD BENOWY
`
`GENUAE TRACE TQ
`DERE HENORY
`
`
`RETURN FILTERED WAGE DATA ToDENCE DIPUY T Figure 19
`
`1316
`
`20
`
`20
`
`

`

`Patent Application Publication
`
`Jul. 12,2012 Sheet 20 of 21
`
`US 2012/0176401 Al
`
`
`
`2000
`
`
`
`
`
`BEGIN CAPTURING DATA FROM IMAGE SENSOR
`|
`
`RECENE DEVICE INPUT INDICATIVE GF THE SELECTION
`OF A FACE IN THE IMAGE SENSOR DATA
`
`
`
`
`
`
`2004
`A
`
`-
`
`2000
`
`2008
`
`au44
`
`RECEIVE SELECHON OF IMAGE FILTERS) 10 BE
`
` APPLIED
`
`
`
`
`RECEIVE DEVICE (INPUT DATA FROM ONE OR MORE
`SENSORS IN COMMUNICATION WHR THE DEVICE
`
`|
`
`(E6., GESTURES)
`
`:
` [RECEIVE AND REGISTERHIGH LEVEL EVENTS AT DEVE -——“
`|
`2042
`
`
`
`HSE DEVICE INPUT DATA AND REGITERED EVENT DATA)
`|
`10 PASS APPROPRIATE INPUT PARAMETERS TO 7
`
`
`
`|
`SELECTED IMAGE FILTER), REEPING THE IMAGE
`| FILTERS “POCIS” OW THE CURRENT POSITION OF THE
`
`
`|
`SELECTED FACE
`|
`
`
`
` RETURN FILTERED TRADE DATA TO DEVICE DISPLAY | wee
`
`
`PERFORM IMAGE FILTERING
`
`
`
`2016
`
`Figure 20
`
`21
`
`21
`
`

`

`Patent Application Publication
`
`Jul. 12, 2012 Sheet 21 of 21
`
`US 2012/0176401 Al
`
`Leaanbiy
`
`
`
`
`
`JORGESE0SH9SROULAGd
`
`
`
`LSRDNDSNOLDINRNOD|
`
`ghee”wieNcoiz
`
`
`
`oLize*TFPELE
`
`EBLE”
`
`
`
`YOURE)LYONDRYfWOSRE)
`AIBIGd
`
`PMEVR)iANSTB
`
`ag
`ost
`agen
`ea
`
`ca
`oaser
`g
`
`22
`
`22
`
`
`
`
`
`
`
`
`
`
`

`

`US 2012/0176401 Al
`
`Jul. 12, 2012
`
`GESTURE MAPPING FOR IMAGE FILTER
`INPUT PARAMETERS
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`[0001] This application claimspriority to U.S. Provisional
`Application Ser. No. 61/431,718, filed on Jan. 11, 2011,
`entitled, “Gesture Mapping for Image Filter Input Param-
`eters,’ which is hereby incorporated by reference in its
`entirety. This application is related to the commonly-assigned
`USS. patent application having Atty. Dkt. No. P10551US1
`(119-0215US), filed on Mar. 21, 2011, entitled, “Gesture-
`Based Configuration of Image Processing Techniques,”
`which is hereby incorporated by referencein its entirety.
`
`BACKGROUND
`
`[0002] The disclosed embodiments relate generally to por-
`table electronic devices, and moreparticularly, to portable
`electronic devices that display filtered images on a touch
`screen display.
`[0003] As portable electronic devices have become more
`compact, and the numberoffunctions able to be performed by
`a given device has steadily increased, it has becomea signifi-
`cant challenge to design a user interface that allows users to
`easily interact with such multifunctional devices. This chal-
`lenge is particularly significant for handheld portable elec-
`tronic devices, which have much smaller screens than typical
`desktop or laptop computers.
`[0004] As such, some portable electronic devices (e.g.,
`mobile telephones, sometimes called mobile phones, cell
`phones, cellular telephones, and the like) have employed
`touch-sensitive displays (also known as a “touch screens”)
`with a graphical user interface (GUI), one or more processors,
`memory and one or more modules, programs or sets of
`instructions stored in the memory for performing multiple
`functions. In some embodiments, the user interacts with the
`GUIprimarily through finger contacts and gestures on the
`touch-sensitive display. In some embodiments, the functions
`may include telephoning, video conferencing, e-mailing,
`instant messaging, blogging, digital photographing, digital
`video recording, web browsing, digital music playing, and/or
`digital video playing. Instructions for performing these func-
`tions may be included in a computer readable storage medium
`or other computer program product configured for execution
`by one or more processors.
`[0005] Touch-sensitive displays can provide portable elec-
`tronic devices with the ability to present transparent and
`intuitive user interfaces for viewing and navigating GUIs and
`multimedia content. Such interfaces can increase the effec-
`tiveness, efficiency and usersatisfaction with activities like
`digital photography on portable electronic devices. In par-
`ticular, portable electronic devices used for digital photogra-
`phy and digital video may provide the user with the ability
`perform various image processing techniques, suchasfilter-
`ing, focusing, exposing, optimizing, or otherwise adjusting
`captured images—ceitherin real time as the image framesare
`being captured by the portable electronic device’s image sen-
`soror after the image has beenstored in the device’s memory.
`[0006] As imageprocessing capabilities of portable elec-
`tronic devices continue to expand and become more complex,
`software developers of client applications for such portable
`electronic devices increasingly need to understand how the
`various inputs and states of the device should be translated
`
`into input parameters for the image processing routines. As a
`simple example, consider a single tap gesture at a particular
`coordinate (x, y) on a touch screen. One example oftranslat-
`ing the input pointlocation of (x, y) to an “auto focus” image
`processing routine would be to cause the coordinate (x, y) to
`as the center ofa rectangular box over which the image sensor
`will attempt to focus for the next captured image frame. With
`more complex image processing routines, however, such as
`graphically intensive imagefilters, the number and type of
`inputs, as well as logical considerations regarding the orien-
`tation of the device and other factors may become too com-
`plex for client software applications to readily be able to
`interpret and/or process correctly.
`[0007] Accordingly, there is a need for techniques to imple-
`ment a programmatic interface to map particular user inter-
`actions, e.g., gestures, to the input parameters of various
`image processing routines, e.g., imagefilters, in a way that
`provides a seamless, dynamic, and intuitive experience for
`both the user andthe client application software developer.
`
`SUMMARY
`
`[0008] As mentioned above, with more complex image
`processing routines being carried out on personal electronic
`devices, such as graphically intensive imagefilters, e.g.,
`imagedistortionfilters, the numberand type of inputs, as well
`as logical considerations regarding the orientation of the
`device and other factors may become too complex for client
`software applications to readily interpret and/or process cor-
`rectly.
`in one embodiment described herein,
`[0009] As such,
`imagefilters are categorized by their input parameters. For
`example, circularfilters, i.e., imagefilters with distortions or
`other effects centered over a particular circular-shaped region
`ofthe image, may need input parameters of“input center” and
`“radius.” Thus, when a client application wants to call a
`particular circular filter,
`it queries the filter for its input
`parameters and then passes the appropriate values retrieved
`from user input (e.g. gestures) and/or device input(e.g., ori-
`entation information) to a gesture translation layer, which
`mapsthe user and device input information to the actual input
`parameters expected by the image filter itself.
`In some
`embodiments, the user and device input will be mappedto a
`value that is limited to a predetermined range, wherein the
`predetermined range is based on the input parameter. There-
`fore, the client application doesn’t need to handle logical
`operations to be performedbythe gesture translation layer or
`know exactly what will be done with those values by the
`underlying imagefilter. It merely needs to know that a par-
`ticular filter’s input parameters are, e.g., “input center” and
`“radius,” and then pass the relevant information along to the
`gesture translation layer, which will in turn give the image
`filtering routines the values that are neededtofilter the image
`as indicated by the user, and as will be discussed in further
`detail below.
`
`Inanother embodiment, one or more processors in a
`[0010]
`personal electronic device may receive the selection of image
`filter(s) to be applied. Next, the device may receive input data
`from one or more sensors which may be disposed within or
`upon the device (e.g., image sensor, orientation sensor, accel-
`erometer, Global Positioning System (GPS), gyrometer).
`Next, the device may receive and register high level events
`(e.g., gestures) and thenuse the received device input data and
`registered event data to pass appropriate input parameters to
`the selected imagefilter(s). Finally, the selected image pro-
`
`23
`
`23
`
`

`

`US 2012/0176401 Al
`
`Jul. 12, 2012
`
`
`cessing routines may be applied, andafiltered image may be [0018] FIG.4 illustrates a user tap point on a touch screen
`
`returned to the device’s display and/or stored to a memory in
`of a camera device in “landscapeorientation,” in accordance
`with one embodiment.
`communication with the device. In some embodiments, the
`imagefilter is applied in near-real time, i.e., substantially
`[0019]
`FIG. 5 illustrates a mirroring imagefilter in land-
`immediately after the act of passing the appropriate input
`scape orientation based on a user tap point on a touch screen
`of a camera device, in accordance with one embodiment.
`parameters to the selected imagefilter occurs.
`[0020]
`FIG.6 illustrates a user tap point on a touch screen
`[0011]
`In yet another embodiment, rather than merely
`of a camera device, in accordance with another embodiment.
`returning a filtered image to the device’s display, the device
`[0021]
`FIG. 7 illustrates a mirroring imagefilter in land-
`may calculate an image overlay preview grid forthe selected
`scape orientation based on a user tap point on a touch screen
`image filter(s) and current device input data and gesture
`of a camera device, in accordance with one embodiment.
`inputs. Such an overlay may indicate to the user of the device
`[0022]
`FIG.8 illustrates a user tap point on a touch screen
`whatinputs are available (e.g., touch, rotation, resizing) and
`of a camera device in “portrait orientation,” in accordance
`whateffect the currentsettings ofthe device’s inputs will have
`with one embodiment.
`on the image whenthe selected imagefilter(s) are applied.
`[0012]
`In still other embodiments, the gesture mapping
`techniques described above maybe applied to video capture
`as well. Oncethe device has begun to capture videodata,e.g.,
`in the form ofa video stream of images, received device input
`data and registered event data may be used to pass appropriate
`input parameters to the selected imagefilter(s) and perform
`imagefiltering on the current image frame of the captured
`video data. As each imageframeofvideo data is captured, the
`process may optionally store each rendered image frame to
`memory or merely store the time-tagged sequence of user
`inputs (e.g. gestures) and/or device inputs (e.g., orientation)
`that occurred during the video capturing (hereinafter the
`“gesture track”) so that the gesture track may be applied to
`unfiltered video data at a later date and time, perhaps by a
`device having a more powerful processor.
`[0013]
`In yet another embodiment, face detection algo-
`rithms may be employed such that, once the “focus” of an
`image processing routine,
`i.e., the central location of the
`imagefilter’s effect, has been placed on an area of the image
`determined to be a humanface, the focus of the imagepro-
`cessing routine will remain over the detected face so long as
`the detected face remains in the captured image data. For
`example, if the user of a device employing this embodiment
`were to indicate via a single tap gesture that a particular
`subject’s face should be the centerpoint, 1-e. the “focus,” of a
`particular image processing routine, the subject’s face could
`remain the center point ofthe image processing routine as that
`person moved within the captured image data, without the
`need for the operator of the device to have to continuously
`click on the subject’s face to indicate a new position for the
`center point of the image processing routine.
`[0014] Gesture mapping techniques in accordance with the
`various embodiments described herein may be implemented
`directly by a device’s hardware and/or software, thus making
`these intuitive and fun image filtering techniques readily
`applicable to any number of electronic devices, such as
`mobile phones, personal data assistants (PDAs), portable
`music players, monitors, televisions, as well as laptop, desk-
`top, and tablet computer systems.
`
`FIG. 9 illustrates a mirroring imagefilter in portrait
`[0023]
`orientation based on a user tap point on a touch screen of a
`camera device, in accordance with one embodiment.
`[0024]
`FIGS. 10A-10Cillustrate the input ofa relative ges-
`ture, in accordance with one embodiment.
`[0025]
`FIG. 11 illustrates the translation of a gesture from
`touch screen space to image sensor space, in accordance with
`one embodiment.
`[0026]
`FIG. 12 illustrates a three-tiered programmatic ges-
`ture mapping architecture, in accordance with one embodi-
`ment.
`
`FIG. 13 illustrates variousfilter types and their cor-
`[0027]
`responding input parameters and exemplary imagefilters, in
`accordance with one embodiment.
`
`FIG. 14 illustrates nine various imagesfilters on the
`[0028]
`display of a personal electronic device, in accordance with
`one embodiment.
`[0029]
`FIG. 15 illustrates grid overlays for the nine image
`filters of FIG. 14, in accordance with one embodiment.
`[0030]
`FIG. 16 illustrates an image overlay preview grid for
`a twirl imagefilter, in accordance with one embodiment.
`[0031]
`FIG. 17 illustrates, in flowchart form, one embodi-
`mentof a process for performing gesture mapping for image
`filter input parameters.
`[0032]
`FIG. 18 illustrates, in flowchart form, one embodi-
`mentof a process for displaying a grid overlay indicative of
`gesture mapping for imagefilter input parameters.
`[0033]
`FIG. 19 illustrates, in flowchart form, one embodi-
`mentof a process for performing gesture mapping for image
`filter input parameters during video capture.
`[0034]
`FIG. 20 illustrates, in flowchart form, one embodi-
`mentof a process for performing facial detection in conjunc-
`tion with gesture mapping for imagefilter input parameters.
`[0035]
`FIG. 21 illustrates a simplified functional block dia-
`gram ofa device possessing a display, in accordance with one
`embodiment.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 illustrates a typical outdoor scene with a
`[0015]
`human subject, in accordance with one embodiment.
`[0016]
`FIG. 2 illustrates a typical outdoor scene with a
`human subject as viewed on a camera device’s preview
`screen, in accordance with one embodiment.
`[0017]
`FIG. 3 illustrates a user interacting with a camera
`device via a touch gesture, in accordance with one embodi-
`ment.
`
`24
`
`DETAILED DESCRIPTION
`
`[0036] This disclosure pertains to systems, methods, and
`computer readable medium for mappingparticularuser inter-
`actions, e.g., gestures, to the input parameters of various
`image processing routines, e.g., imagefilters, in a way that
`provides a seamless, dynamic, and intuitive experience for
`both the user and the software developer. Such techniques
`may handle the processing of both “relative” gestures, 1.e.,
`those gestures having values dependent on how much aninput
`to the device has changedrelative to a previous value of the
`input, and “absolute” gestures, i.e., those gestures having
`values dependentonly on the instant value of the inputto the
`device. Additionally, inputs to the device beyond user-input
`
`24
`
`

`

`US 2012/0176401 Al
`
`Jul. 12, 2012
`
`gestures may be utilized as input parameters to one or more
`image processing routines. For example, the device’s orien-
`tation, acceleration, and/or position in three-dimensional
`space may be used as inputs to particular image processing
`routines.
`
`[0037] The techniques disclosed herein are applicable to
`any numberofelectronic devices with optical sensors: suchas
`digital cameras, digital video cameras, mobile phones, per-
`sonal data assistants (PDAs), portable music players, moni-
`tors, televisions, and, of course, desktop, laptop, and tablet
`computer displays.
`[0038]
`In the interestof clarity, not all features of an actual
`implementation are described in this specification. It will of
`course be appreciated that in the development of any such
`actual
`implementation (as in any development project),
`numerousdecisions must be madeto achieve the developers’
`specific goals (e.g., compliance with system- and business-
`related constraints), and that these goals will vary from one
`implementation to another. It will be appreciated that such
`developmenteffort might be complex and time-consuming,
`but would nevertheless be a routine undertaking for those of
`ordinary skill having the benefit of this disclosure.
`[0039]
`In the following description, for purposes of expla-
`nation, numerous specific details are set forth in order to
`provide a thorough understanding of the inventive concept.
`Aspart ofthe description, somestructures and devices may be
`shownin block diagram form in order to avoid obscuring the
`invention. (In addition, references to numbers without sub-
`scripts are understoodto referenceall instances of subscripts
`corresponding to the referenced number.) Moreover, the lan-
`guage usedin this disclosure has beenprincipally selected for
`readability and instructional purposes, and may not have been
`selected to delineate or circumscribe the inventive subject
`matter, resort to the claims being necessary to determine such
`inventive subject matter. Reference in the specification to
`“one embodiment”or to “an embodiment” meansthat a par-
`ticular feature, structure, or characteristic described in con-
`nection with the embodiments is included in at least one
`embodimentofthe invention, and multiple referencesto “one
`embodiment”or “an embodiment” should not be understood
`as necessarily all referring to the same embodiment.
`[0040] Referring now to FIG.1, atypical outdoor scene 100
`with a human subject 102 is shown, in accordance with one
`embodiment. The scene 100 also includes the Sun 106 and a
`natural object, tree 104. Scene 100 will be used in the subse-
`quent figures as an exemplary sceneto illustrate the various
`image processing techniques described herein.
`[0041] Referring now to FIG.2, atypical outdoor scene 200
`with a humansubject 202 as viewed on a camera device 208’s
`preview screen 2101s shown, in accordance with one embodi-
`ment. The dashed ones 212 indicate the viewing angle of the
`camera (not shown) on the reverse side of camera device 208.
`Camera device 208 may also possess a second camera, such
`as front-facing camera 250. Other numbers and positions of
`cameras on camera device 208 are also possible. As men-
`tioned previously, although camera device 208 is shown here
`as a mobile phone, the teachings presented herein are equally
`applicable to any electronic device possessing a camera, such
`as, but not limited to: digital video cameras, personal data
`assistants (PDAs), portable music players, laptop/desktop/
`tablet computers, or conventional digital cameras. Each
`object in the scene 100 has a corresponding representation in
`the scene 200 as viewed on a camera device 208’s preview
`
`screen 210. For example, human subject 102 is represented as
`object 202, tree 104 is represented as object 204, and Sun 106
`is represented as object 206.
`[0042] Referring now to FIG.3, a user 300 interacting with
`acamera device 208 via an exemplary touch gesture is shown,
`in accordance with one embodiment. The preview screen 210
`of camera device 208 may be, for example, a touch screen.
`The touch-sensitive touch screen 210 provides an input inter-
`face and an outputinterface betweenthe device 208 anda user
`300. The touch screen 210 displays visual outputto the user.
`The visual output may include graphics, text, icons, pictures,
`video, and any combinationthereof.
`[0043] A touch screen such as touch screen 210 has a touch-
`sensitive surface, sensor or set of sensors that accepts input
`from the user based on haptic and/ortactile contact. The touch
`screen 210 detects contact (and any movementor breaking of
`the contact) on the touch screen 210 and converts the detected
`contact into interaction with user-interface objects (e.g., one
`or more soft keys, icons, web pages, images or portions of
`images) that are displayed on the touch screen. In an exem-
`plary embodiment, a point of contact between a touch screen
`210 andthe user correspondsto a finger of the user 300.
`[0044] The touch screen 210 may use LCD liquid crystal
`display) technology, or LPD (light emitting polymerdisplay)
`technology, although other display technologies may be used
`in other embodiments. The touch screen 210 may detect con-
`tact and any movement or breaking thereof using any of a
`plurality of touch sensing technologies now knownorlater
`developed, including but not limited to capacitive, resistive,
`infrared, and surface acoustic wave technologies, as well as
`other proximity sensorarrays or other elements for determin-
`ing one or more points of contact with a touch screen 210.
`[0045] A touch-sensitive display in some embodiments of
`the touch screen 210 may be analogous to the multi-touch
`sensitive tablets described in the following U.S. Pat. Nos.
`6,323,846 (Westermanet al.), 6,570,557 (Westermanetal,),
`and/or 6,677,932 (Westerman), and/or U.S. Patent Publica-
`tion 2002/0015024A1, each of which is hereby incorporated
`by reference. A touch-sensitive display in some embodiments
`of the touch screen 210 maybeas describedin the following
`applications: (1) U.S. patent application Ser. No, 11/381,313,
`“Multipoint Touch Surface Controller,”filed May 2, 2006; (2)
`USS. patent application Ser. No. 10/840,862, “Multipoint
`Touchscreen,”filed May 6, 2004; (3) U.S. patent application
`Ser. No, 10/903,964, “Gestures For Touch Sensitive Input
`Devices,” filed Jul. 30, 2004; (4) U.S. patent application Ser.
`No. 11/048,264, “Gestures For Touch Sensitive Input
`Devices,”filed Jan. 31, 2005; (5) U.S. patent application Ser.
`No. 11/038,590, “Mode-Based Graphical UserInterfaces For
`Touch Sensitive Input Devices,” filed Jan. 18, 2005; (6) U.S.
`patent application Ser. No. 11/228,758, “Virtual Input Device
`Placement On A Touch Screen UserInterface,”filed Sep. 16,
`2005;
`(7) U.S. patent application Ser. No. 11/228,700,
`“Operation OfA Computer With A Touch Screen Interface,”
`filed Sep. 16, 2005; (8) U.S. patent application Ser. No.
`11/228,737, “Activating Virtual Keys OfA Touch-Screen Vir-
`tual Keyboard,”filed Sep. 16, 2005; and (9) U.S. patent appli-
`cation Ser. No. 11/367,749, “Multi-Functional Hand-Held
`Device,” filed Mar. 3, 2006. All of these applications are
`incorporated by reference herein.
`[0046] The touch screen 210 may have a resolution in
`excess of 300 dots per inch (dpi). In an exemplary embodi-
`ment, the touch screen hasa resolution of approximately 325
`dpi. The user 300 may make contact with the touch screen 210
`
`25
`
`25
`
`

`

`US 2012/0176401 Al
`
`Jul. 12, 2012
`
`using any suitable object or appendage, such as a stylus, a
`finger, and so forth. In some embodiments, the user interface
`is designed to work primarily with finger-based contacts and
`gestures, which typically have larger areas of contact on the
`touch screen than stylus-based input. In some embodiments,
`the device translates the rough finger-based gesture input into
`a precise pointer/cursor coordinate position or commandfor
`performingthe actions desired by the user 300.
`[0047] As used herein, a gesture is a motion of the object/
`appendage making contact with the touch screen display sur-
`face. One or morefingers may be used to perform two-dimen-
`sional or three-dimensional operations on one or more
`graphical objects presented on preview screen 210, including
`but not limited to: magnifying, zooming, expanding, mini-
`mizing,resizing, rotating,sliding, opening,closing, focusing,
`flipping, reordering, activating, deactivating and any other
`operation that can be performed on a graphical object. In
`some embodiments, the gestures initiate operations that are
`related to the gesture in an intuitive manner. For example, a
`user can place an index finger and thumbonthe sides, edges
`or corners of a graphical object and perform a pinching or
`anti-pinching gesture by moving the index finger and thumb
`togetheror apart, respectively. The operationinitiated by such
`a gesture results in the dimensions of the graphical object
`changing. In some embodiments, a pinching gesture will
`causethe size ofthe graphical object to decrease in the dimen-
`sion being pinched. In some embodiments, a pinching gesture
`will cause the size of the graphical object to decrease propor-
`tionally in all dimensions. In some embodiments, an anti-
`pinching or de-pinching movementwill cause the size of the
`graphical object to increase in the dimension being anti-
`pinched.In other embodiments, an anti-pinching or de-pinch-
`ing movement will cause the size of a graphical object to
`increase in all dimensions (e.g., enlarging proportionally in
`the x and y dimensions).
`[0048] Referring now to FIG.4, a user tap point 402 on a
`touch screen 210 of a camera device 208 positionedin “land-
`scape”orientation is shown, in accordance with one embodi-
`ment. The location of tap point 402 is represented by an oval
`shaded with diagonal lines. As mentioned above, in some
`embodiments, the device translates finger-based tap points
`into a precise pointer/cursor coordinate position, represented
`in FIG. 4 as point 404 with coordinates x1 and y1. As shown
`in FIG. 4, the x-coordinates of the device’s display corre-
`spondto the shorter dimension ofthe display, and the y-co-
`ordinates correspond to the longer dimension ofthe display.
`Axis 400, whoselocation is defined by the y-coordinate, y1,
`of point 404 may be used in the application of a particular
`image distortion filter, for example a mirroring distortion
`effect, as will be described in further detail below.
`[0049]
`In some embodiments, the user-input gestures to
`device 208 may be usedto drive the setting of input param-
`eters of various image processing routines, such as image
`filters, e.g., image distortionfilters. The above functionality
`can be realized with an input parameter gesture mapping
`process. The process begins by detecting N contacts on the
`display surface 210. When N contacts are detected, informa-
`tion such as the location, duration, size, and rotation of each
`of the N contacts is collected by the device. The useris then
`allowed to adjust the input parameters by making or modify-
`ing a gesture at or near the point of contact. If m