Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 1 of 53
`Case 5:20-cv-01756—NC Document 1-1 Filed 03/11/20 Page 1 of 53
`
`
`
`
`
`
`
`
`EXHIBIT 1
`
`EXHIBIT 1
`
`

`

`US007293.231B1
`
`(12)
`
`United States Patent
`Gunn et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7.293,231 B1
`Nov. 6, 2007
`
`(54) DATA ENTRY FOR PERSONAL COMPUTING
`DEVICES
`
`4,689,768 A * 8/1987 Heard et al. ................ 71.5/533
`4,730,252 A
`3/1988 Bradshaw ................... 364,403
`
`(75) Inventors: Harold David Gunn, Vancouver (CA):
`John Chapman, Vancouver (CA)
`
`(Continued)
`FOREIGN PATENT DOCUMENTS
`
`(73) Assignee: British Columbia Ltd. (CA)
`
`(*) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 1279 days.
`
`(21) Appl. No.: 09/631,101
`(22) Filed:
`Aug. 1, 2000
`
`Related U.S. Application Data
`(63) Continuation of application No. PCT/CA0000285,
`filed on Mar. 15, 2000, which is a continuation-in-part
`of application No. 09/272,700, filed on Mar. 18, 1999.
`
`(51) Int. Cl.
`(2006.01)
`G06F 7700
`(52) U.S. Cl. ...................... 715/531; 715/534; 715/541;
`345/179; 382/187
`(58) Field of Classification Search ................ 715/541,
`715/505, 534,530,531; 365/189.01; 34.5/173,
`345/179, 156; 382/187; 707/531
`See application file for complete search history.
`References Cited
`U.S. PATENT DOCUMENTS
`
`(56)
`
`2, 1972 Post ........................... 715,534
`3,644,898 A
`7/1980 Montgomery ............... 400,486
`4,211,497 A
`s E. East al. 3.3.
`i.
`A
`9, 1984 E. al. 71.5/533
`4471.459 A
`4,499,553 A * 2/1985 Dickinson et al. .......... 71.5/533
`4,559,598 A 12, 1985 Goldwasser et al. ........ 364/419
`4,648,044 A * 3/1987 Hardy et al. .................. TO6/60
`
`
`
`EP
`
`1/1990
`O 352 377 A1
`Continued
`(Continued)
`OTHER PUBLICATIONS
`Bellman, T., & MacKenzie, I. S. (1998). A probabilistic character
`layout strategy for mobile text entry. Proceedings of Graphics
`Interface '98, pp. 168-176, Toronto: Canadian Information Process
`Ing Society.
`
`(Continued)
`Primary Examiner Doug Hutton
`Assistant Examiner James H. Blackwell
`(74) Attorney, Agent, or Firm Townsend and Townsend
`and Crew LLP
`
`(57)
`
`ABSTRACT
`
`In one aspect of the invention, the user can rapidly enter and
`search for text through a combination of entering one or
`more characters (a partial text entry) on a digitally displayed
`keyboard with a pointing device and using a search list to
`display a list of selectable completion candidates based on
`the partial text entry. The user can select one of the comple
`tion candidates in the search list to replace the partial text
`entry which the user is currently entering. Alternatively, the
`user can select a completion candidate to be the basis of a
`search for a refined list of completion candidates. The user
`can also deactivate the interactive search list, and immedi
`ately continue adding to or modifying the current partial text
`entry, and may re-invoke the search list to further search for
`completion candidates based on the modified partial text
`entry.
`
`116 Claims, 23 Drawing Sheets
`
`104:fleussriest
`to user selects the
`e
`E. E.
`
`EXAMPLE 100
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 2 of 53
`
`10; if heuser
`J
`gestures to a dead
`zone, the selected
`candidates de
`selectecadest Ny Monir pointing
`ofcadicates
`dew8
`remains displayed,
`
`1.
`18:Wia
`candicatesslected if
`the usergestures to
`alcher candicals,
`the salection
`changes.
`
`J.
`108; if usergestures
`ada acidae
`(without lifting), that
`candidassisted
`aid becofe8 the
`salition
`
`scleared 3rd the
`
`--> keyboards anabled,
`
`E.
`20ne, the selected
`candidates (e.
`selected and the list
`of candidates
`remains displayed
`
`16; if the user conties to
`keepa candidate selected for
`more thandsfied time imit.2,
`areflecstafcadidatests
`obtained using the selected
`candidata and displayedt the
`addie
`interativesearchlist
`
`ity
`E.
`lies
`selected, the selected
`cacialis Sadd to the
`extinplace of the partial
`text entry, the listis
`cleared and the
`keyboard is enabled.
`
`120; if the usergestureskoward
`caricidate in thersfied
`candidatest, that candidate is
`selected and the usercar use
`the reisstie same
`manner as with the previous
`cardidate list
`
`122:If the pointing device is led up
`without significal movement the
`interacSwes&chistiscleared, the
`keyboard is enabled, the searchs
`terminated and the most recently
`selectedcariidates added to the text
`in place of the partial textantry
`
`i24:fhese can:
`contine buildinger
`Todifying partial
`extent of
`tale after
`search on the partial
`textently or
`cance thesearch
`sing or
`rever to last
`candidatest
`rever to the last
`searchstring or
`aire
`
`)
`
`

`

`US 7.293,231 B1
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`5/1988 Hirosawa et al. ........... 364,900
`4,744,050 A
`9/1988 Miyao et al. .....
`... 364/419
`4,774,666 A
`10, 1988 Goldwasser et al. ........ 364/419
`RE32,773 E
`4,782.464 A * 11/1988 Gray et al. ..........
`... 715,533
`4,783,761. A * 11/1988 Gray et al. ...
`... 715,533
`4,786,765 A 11, 1988 Yamanami et al.
`178/1906
`4,807,181 A
`2, 1989 Duncan, IV et al. ........ 364,900
`4,847,766 A * 7/1989 McRae et al. ....... ... 715,532
`4,891,786 A
`1/1990 Goldwasser ...
`364,900
`4.969,097 A 1 1/1990 Levin .........
`364,419
`4,980,855 A * 12/1990 Kojima ....
`71.5/533
`5,040,113 A
`8, 1991 Mickunas ...
`364,419
`5,060,154 A * 10/1991 Duncan, IV
`71.5/533
`5,067,165 A 11/1991 Nishida ......
`382,195
`5,096,423. A
`3, 1992 Goldwasser
`434/118
`5,203,704 A
`4, 1993 McCloud.
`434,156
`5,220,649 A * 6/1993 Forcier ....
`715,541
`5,220,652 A
`6/1993 Rowley .........
`395/275
`5,261,112 A * 11/1993 Futatsugi et al. .
`... 715,533
`5,297,041 A
`3, 1994 Kushler et al. ........ 364,419.15
`5,327,161 A
`7/1994 Logan et al. ............... 345,157
`5,329,609 A * 7/1994 Sanada et al. ....
`... 704,251
`5,347.295 A * 9/1994 Agulnick et al. ........... 34.5/156
`5,392.447 A * 2/1995 Schlack et al. ...
`... 715,863
`5,487.616 A *
`1/1996 Ichbiah ..........
`... 400/489
`5,500,935 A
`3, 1996 Moran et al. ..
`... 395,156
`5,574.482 A 11/1996 Niemeier .......
`... 345,173
`5,594,640 A
`1/1997 Capps et al.
`715,532
`5,596,699 A
`1/1997 Driskell ...
`395,352
`5,621,641. A
`4, 1997 Freeman ..
`395,796
`5,623.406 A
`4/1997 Ichbiah ....
`395/753
`5,649.223 A 7/1997 Freeman ...,
`395/796
`5,657,397 A
`8, 1997 Bokser .......
`382,225
`5,666,139 A
`9, 1997 Thielens et al.
`345,173
`5,689,667 A 11/1997 Kurtenbach.
`395,352
`5,704,029 A * 12/1997 Wright, Jr. ..
`715,505
`5,724.457 A
`3, 1998 Fukushima ................. 382,311
`5,734,749 A
`3, 1998 Yamada et al. ............. 382,187
`5,745,116 A
`4, 1998 Pisutha-Arnond .
`... 345,358
`5,790,115 A
`8/1998 Pleyer et al. ............... 71.5/716
`5,805,158 A
`9, 1998 Bertram et al. ............. 71.5/764
`5,805,159 A
`9, 1998 Bertram et al. ......
`... 715,764
`5,805,167 A
`9/1998 van Cruyningen .......... 345.353
`5,805,911 A
`9, 1998 Miller ........................ 395/796
`5,812,697 A * 9/1998 Sakai et al. ................. 382/186
`5,818,437 A 10/1998 Grover et al.
`5,821.512 A * 10/1998 O'Hagan et al. ........... 235,383
`5,835,635 A * 11/1998 Nozaki et al. .......
`... 382,226
`5,838,302 A * 11/1998 Kuriyama et al.
`345,173
`5,845,300 A * 12/1998 Comer et al. ............... 715/508
`5,864,340 A *
`1/1999 Bertram et al. ............. 715/780
`5,881,169 A * 3/1999 Henry, Jr. ................... 382/187
`5,896,321 A * 4, 1999 Miller et al. ........... 365,189.01
`5,911,485 A
`6, 1999 Rossmann ................... 341 (22
`5,914,708 A * 6/1999 LaGrange et al. .......... 345,179
`5,926, 178 A
`7, 1999 Kurtenbach ................. 345,352
`5,943,039 A
`8, 1999 Anderson et all
`345/146
`5,959,629 A
`9/1999 Masui ........................ 345,347
`5,963,666 A * 10/1999 Fujisaki et al. ............. 382,187
`5,963,671. A 10/1999 Comerford et al. ......... 382/230
`5,974,558 A 10/1999 Cortopassi et al.
`T13,323
`5,977,948 A 1 1/1999 Nishibori .................... T15,841
`5,982,351 A 11/1999 White et al.
`715,810
`6,002,390 A 12/1999 Masui ........................ 345,173
`6,005,549 A 12/1999 Forest ........................ 345,157
`6,008,799 A 12/1999 Van Kleeck ...
`... 345,173
`6,026,233 A * 2/2000 Shulman et al. ............ 717,113
`6,037.942 A * 3/2000 Millington .................. 715.835
`6,084,576 A
`7/2000 Leu et al.
`6,088,649 A
`7/2000 Kadaba et al.
`6,094,197 A * 7/2000 Buxton et al. .............. 345,863
`
`
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 3 of 53
`
`6,097.841 A * 8/2000 Gunji et al. ................ 382,229
`6,098,086 A * 8/2000 Krueger et al. .
`71.5/535
`6,101.461 A * 8/2000 Ukigawa et al. .............. TO4/10
`6,111,985 A * 8/2000 Hullender et al. .......... 382,229
`6,144,378 A * 1 1/2000 Lee .........
`715/767
`6,154,758 A * 1 1/2000 Chiang ....................... 71.5/541
`6,167.411 A * 12/2000 Narayanaswamy ......... 71.5/541
`6,167.412 A * 12/2000 Simons
`708/105
`6,188,789 B1* 2/2001 Marianetti et al. .......... 382, 189
`6.256,030 B1* 7/2001 Berry et al. ................ 71.5/854
`6,262,719 B1* 7/2001 Bi et al. ...
`345,179
`6,275,612 B1
`8/2001 Imoto ......................... 382, 189
`6.286,064 B1* 9/2001 King et al. ................... 710,67
`6,292, 179 B1* 9/2001 Lee ............
`345,173
`6,295,372 B1* 9/2001 Hawkins et al. ............ 382,187
`6,307.549 B1
`10/2001 King et al.
`6,369,807 B1
`4/2002 Nakashima ................. 345,179
`6,377.965 B1 * 4/2002 Hachamovitch et al. .... 715/534
`6,405,060 B1* 6/2002 Schroeder et al. .......... 455/566
`6,442,295 B2 * 8/2002 Navoni et al. ..
`382,229
`6,487,424 B1 * 1 1/2002 Kraft et al. ..
`455,566
`6,502,114 B1* 12/2002 Forcier .......
`715,541
`6,621.939 B1* 9/2003 Negishi et al. .
`382,291
`6,654,733 B1 * 1 1/2003 Goodman et al. .
`... 706.52
`6,661,920 B1* 12/2003 Skinner .........
`382,187
`6,751,603 B1* 6/2004 Bauer et al. .
`... TO7.1
`6,801,190 B1 * 10/2004 Robinson et al.
`345,173
`6,888,141 B2* 5/2005 Carr ........
`250,338.1
`6,934,906 B1* 8/2005 Cheok .
`... 715,500.1
`6,970,513 B1* 11/2005 Puri et al. .
`375,240.25
`6,978.421 B1* 12/2005 Aida .......
`... 715,532
`7,003,446 B2 * 2/2006 Trower et al. ................. TO4/9
`7,224.409 B2
`5/2007 Chin et al.
`2001/0000962 A1* 5/2001 Rajan ......................... 345,302
`2001/0027468 A1* 10, 2001 Okura ..
`709/202
`2002fOO67377 A1
`6, 2002 McGovern ..
`345,816
`2003.0137605 A1* 7, 2003 Chin et al. ...
`348,569
`2005, 0198144 A1* 9, 2005 Kraenzel et al. ............ TO9,206
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`JP
`JP
`JP
`JP
`JP
`JP
`JP
`WO
`WO
`WO
`
`O 643 357 A2
`O 643 357 A3
`O 844 570 A2
`O 844 571 A2
`O 858 023 A2
`O 858 O23 A3
`O 844 570 A3
`O 844 571 A3
`101.05324. A
`101.05324. A
`10154033. A
`10154144 A
`11143614
`11167569 A
`10333818
`WO 96,09579
`WO 98,11480
`WO 99,28811
`
`3, 1995
`3, 1996
`5, 1998
`5, 1998
`8, 1998
`9, 1998
`8, 1999
`8, 1999
`4, 1998
`4, 1998
`6, 1998
`6, 1998
`5, 1999
`6, 1999
`4/2002
`3, 1996
`3, 1998
`6, 1999
`
`OTHER PUBLICATIONS
`Jones, P.E., "Virtual Keyboard with Scanning and Augmented by
`Prediction', 1998, Proc. 2nd Euro. Conf. Disability, Virtual Reality
`& Assoc. Tec., Skovde, Sweden, 1998, pp. 45-51.*
`MacKenzie et al., “Alphanumeric Entry on Pen-Based Computers'.
`1994. Intl. Journal of Human-Computer Studies, 41, pp. 775-792,
`<http://www.yorku.ca/mack/IJHCS.html>.*
`MacKenzie et al., “Text Entry Using SoftKeyboards', 1999, Behav
`ior & Information Tech., 18, pp. 235-244. <http://www.yorku.ca/
`mack/BIT3.html>.
`Mankopff et al..."Cirrin: A Word-level Unistroke Keyboard for Pen
`Input', 1998, ACM, pp. 213-214.*
`
`

`

`US 7.293,231 B1
`Page 3
`
`MacKenzie et al..."The Design and Evaluation of a High-Perfor
`mance Soft Keyboard”, 1999, ACM, pp. 25-31.*
`Vitoria et al., “A Comparison of Prediction Techniques to Enhance
`the Communication Rate'. Springer-Verlag, 2004, pp. 400-417.*
`Masui, T. 1999. “POBox: An Efficient Text Input Method for
`Handheld and Ubiquitous Computers'. In Proc. of 1st Intl. Symp. on
`Handheld and Ubiquitous Computing (Karlsruhe, Germany, Sep.
`27-29, 1999). H. Gellersen, Ed. Lecture Notes. In Computer Science,
`vol. 1707. Springer-Verlag, London, 289-300.*
`3Com,“PalmPilot Handbook”, 1997 3Com Corp., pp. 37-39.*
`Masui, T. 1998, “An Efficient Text Input Method for Pen-based
`Computers.” Apr. 18-23, 1998. In Proc. SIGCHI Conf. on Human
`Factors in Computing Systems (LC. Karat, A. Lund, J. Coutaz, and
`J. Karat, Eds. Conference on Human Factors in Computing Sys
`tems. ACM, pp. 328-335.*
`Toshiyuki Masui, Sony Computer Science Laboratory Inc.,
`Shinagawa, Tokyo, “An Efficient Text Input Method for Pen-based
`Computers'. Proceedings of the ACM Conference on Human Fac
`tors in Computing Systems, Apr. 1998, pp. 328-335.
`Toshiyuki Masui, Sony Computer Science Laboratories, Inc.,
`Shinagawa, Tokyo, “Integrating Pen Operations for Composition by
`Example”.
`Don Hopkins, “The Design and Implementation of Pie Menus'. Dr.
`Dobb's Journal, Dec. 1991 (http://art.net/~hopkins/Don/piemenus/
`ddi/piemenus.html).
`Jason I. Hong, “Java Pie Menus', Sep. 4, 1999 (jttp://www.cs.
`berkeley.edu/~asonh/download/software/piemenu/).
`Don Hopkins, “A Description of Pie Menus' (http://catalog.com/
`hopkins/piemenus/PieMenuDescription.html).
`Don Hopkins, “Natural Selection: The Evolution of Pie Menus',
`BayCHI, Oct. 13, 1998 (http://catalog.com/hopkins/piemenus/
`Natural Selection.html).
`Russell Nelson, “Pie Menu Window Manager'. Jun. 3, 1998 (http://
`www.crynwr.com/piewm/).
`Don Hopkins, “ActiveX Pie Menus' (http://catalog.com/hopkins/
`piemenus/ActiveXPieMenus.html).
`Don Hopkins, “Pie Menu References” (http://catalog.com/hopkins/
`piemenu-references.html).
`Don Hopkins, “Pie Menus' (http://art.net/~hopkins/Don/piemenus/
`index.html).
`Tom Nantais, et al., IEEE Transactions on Rehabilitation Engineer
`ing, "A Predictive Selection Technique for Single-Digit Typing With
`a Visual Keyboard', vol. 2, No. 3 Sep. 1994, pp. 130-135.
`“GtkPieMenu' 1999 orfelyus (2 pages).
`Operation Guide: 1997: 2 pages; Mercedes-Benz of North America,
`Inc.; Montvale, New Jersey, United States.
`
`Auto Pilot System—Operation Guide; 1998; 5 pages; Mercedes
`Benz AG; Germany.
`Office Action dated Jul. 17, 2007 issued in related U.S. Appl. No.
`10/399,560.
`Baeza-Yates, et al., “A New Approach to Text Searching.” Com
`munications of the ACM 35, 10 (Oct. 1992), 74-82.
`Bellman et al., “A Probabilistic Character Layout Strategy for
`Mobile Text Entry.” Department of Computer Science and Depart
`ment of Computing and Information Science, University of Guelph.
`Guelph, Ontario, Canada. 1999.
`Bohan et al., “A Psychophysical Comparison of Two Stylus-Driven
`Soft Keyboards.” Department of Psychology, Wichita State Uni
`versity. Wichita, Kansas. 1998.
`Darragh et al., “The Reactive Keyboard: A Predictive Typing Aid'
`IEEE Computer 23, 11 (Nov. 1990), 41-49.
`Venolia et al., “T-Cube: A Fast, Self-Disclosing Pen-Based Alpha
`bet.” In Proceedings of the ACM Conference on Human Factors in
`Computing Systems (CHI '93) (Apr. 1993), Addison-Wesley, pp.
`265-270.
`Wu et al., “AGREP—A Fast Approximate Pattern-Matching Tool.”
`Department of Computer Science, University, of Arizona. Tucson,
`Arizona. In Proceedings of USENIX Technical Conference (San
`Francisco, CA) (Jan. 1992), pp. 153-162.
`Office Action dated Jun. 7, 2007 issued in related U.S. Appl. No.
`11/133,770, filed May 19, 2005.
`Office Action dated May 18, 2007 issued in related U.S. Appl. No.
`11/134,759, filed May 19, 2005.
`Office Action dated Oct. 18, 2006 issued in related U.S. Appl. No.
`10/399,560, filed Apr. 18, 2003.
`International Preliminary Examination Report dated Sep. 12, 2003
`issued in related PCT Application No. PCT/CA01/01473 filed Oct.
`18, 2001.
`International Preliminary Examination Report dated Jun. 19, 2001
`issued in related PCT Application No. PCT/CA00/00285 filed Mar.
`15, 2000.
`European Examination Report dated Jun. 23, 2005 issued in related
`European Application No. 01981 991.1.
`European Examination Report dated Nov. 8, 2004 issued in related
`European Application No. 01981 991.1.
`European Examination Report dated Feb. 7, 2002 issued in related
`European Application No. 00910 460.5.
`* cited by examiner
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 4 of 53
`
`

`

`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 1 of 23
`
`US 7.293,231 B1
`
`
`
`Video
`display
`Circuit
`13
`
`Display Device
`15
`
`HIW Input
`Interface
`17
`
`Application
`27
`Data entry system
`26
`
`O'S
`
`Computer-readable medium
`16
`
`Processing Unit
`12
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 5 of 53
`
`FIG. 1
`
`

`

`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 2 of 23
`
`US 7.293,231 B1
`
`Data Entry System
`
`26
`
`Digital Keyboard System
`
`Candidate Prediction
`System
`
`Dictionary
`
`Logic for interactive
`Search List
`
`FIG. 2
`
`/ Spanito i+)
`
`Completion
`Candidate Field 22
`
`Weight
`Field 24
`
`Dictionary Data
`Structure
`
`
`
`
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 6 of 53
`
`

`

`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 7 of 53
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 7 of 53
`
`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 3 of 23
`
`US 7,293,231 B1
`
`
`
`

`

`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 4 of 23
`
`US 7.293,231 B1
`
`
`
`Interactive
`Search
`List
`30
`
`FIG. 5
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 8 of 53
`
`Digital
`Keyboard
`28
`
`

`

`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 5 of 23
`
`US 7.293,231 B1
`
`Potential completion candidates
`in the dictionary for apartial text
`entry made up of a single
`character
`
`
`
`Portion of
`the
`dictionary
`
`
`
`
`
`Potential Candidate 1
`Potential Candidate 2
`Potential Candidate 3
`Potential Candidate 4
`Potential Candidate 5
`Potential Candidate 6
`Potential Candidate 7
`Potential Candidate 8
`Potential Candidate 9
`Potential Candidate 10
`Potential Candidate 11
`Potential Candidate 12
`Potential Candidate 13
`Potential Candidate 14
`Potential Candidate 15
`Potential Candidate 16
`Potential Candidate 17
`Potential Candidate 18
`
`Potential Candidate 1
`Potential Candidate 5
`Potential Candidate 2
`Potential Candidate 16
`Potential Candidate 18
`
`partial text entry set to
`Potential Candidate 5
`
`)
`
`Displayed
`in the
`interactive
`Search List
`
`
`
`
`
`
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 9 of 53
`
`
`
`List of completion candidates
`obtained by the candidate
`prediction system from the
`above 18 potential candidates
`in the dictionary
`
`
`
`
`
`
`
`
`
`Potential completion candidates
`in the dictionary when the
`partial text entry equals
`Potential Candidate 5
`
`
`
`Potential Candidate 5
`Potential Candidate 6
`Potential Candidate 9
`Potential Candidate 10
`Potential Candidate 1
`Potential Candidate 12
`
`Potential Candidate 5
`Potential Candidate 10
`Potential Candidate 6
`Potential Candidate 9
`Potential Candidate 12
`
`List of completion candidates
`obtained by the candidate
`prediction system from the
`above 7 potential candidates in
`the dictionary
`
`
`
`FIG. 5A
`
`

`

`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 6 of 23
`
`US 7.293,231 B1
`
`
`
`Digital
`Keyboard
`
`Search string =
`
`he digital
`eyboard is
`displayed.
`
`Digital
`Ke
`d
`
`Search string = 'e'.
`
`112
`
`interactive
`Search List
`
`Interactive
`Search List
`
`neaCtWe
`Search List
`With refined
`Candidate
`list
`
`S/Search string =
`end.
`
`Digital
`Keyboard
`
`&
`
`
`
`Search string = empty.
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 10 of 53
`
`120; If the user
`gestures toward a
`candidate in the refined
`Candidate list, that
`Candidate is selected
`and the user can use
`the refined list in the
`Same manner as with
`the previous candidate
`list (at 112 above).
`
`Digital
`Keyboard
`
`Search string =
`'end'.
`
`FIG. 6
`
`EXAMPLE OO
`
`Digital
`Keyboard
`
`Search string = 'e'.
`
`124: The user can:
`Continue building or modifying
`partial text entry or
`initiate a further search on the
`partial text entry or
`cancel the search string or
`revert to last candidate list Or
`revert to the last search string or
`carry Out another operation
`
`

`

`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 7 of 23
`
`US 7.293,231 B1
`
`
`
`102. User Selects the
`key Containing'e' on
`the keyboard. The
`selection forms part
`of the partial text
`entry and is added to
`the search string.
`
`110: If the user
`gestures to a dead
`Zone, the Selected
`Candidate is de
`Selected and the list
`of Candidates
`remains displayed.
`
`118: With a
`candidate selected, if
`the user gestures to
`another candidate,
`the Selection
`changes.
`
`104: If the user does not
`release the keyboard
`Selection in less than time
`limit L1, the Candidate
`Prediction System obtains a
`candidate list based on the
`user's keyboard Selection.
`The Candidate list is
`displayed in an interactive
`Search list.
`
`Monitor pointing
`device
`
`
`
`108: if usergestures
`toward a candidate
`(without lifting), that
`Candidate is hi-lited
`and becomes the
`Selection.
`
`EXAMPLE 100
`
`106: If pointing
`device is lifted up
`without significant
`movement, the
`interactive Search list
`is cleared and the
`keyboard is enabled.
`
`110: If the user
`gestures to a dead
`ZOne, the selected
`candidate is de
`Selected and the list
`of candidates
`remains displayed.
`
`
`
`116: If the user Continues to
`keep a candidate Selected for
`more than defined time limit L2,
`a refined list of candidates is
`obtained using the selected
`candidate and displayed in the
`interactive Search list.
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 11 of 53
`
`120: if the user gestures toward
`a candidate in the refined
`candidate list, that candidate is
`Selected and the user can use
`the refined list in the same
`manner as with the previous
`Candidate list.
`
`114: if the userlifts up
`the pointing device in
`less than time limit L2
`with a Candidate
`Selected, the selected
`candidate is added to the
`textin place of the partial
`text entry, the list is
`cleared and the
`keyboard is enabled.
`
`
`
`122: if the pointing device is lifted up
`without significant movement, the
`interactive search list is cleared, the
`keyboard is enabled, the Search is
`terminated and the most recently
`Selected candidate is added to the text
`in place of the partial text entry.
`
`FIG. 6A
`
`
`
`124: The user Can:
`Continue building Or
`modifying partial
`text entry Or
`initiate a further
`search on the partial
`text entry or
`Cancel the Search
`string or
`revert to last
`Candidate list Or
`revert to the last
`search string or
`carry Out another
`operation
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 8 of 23
`
`US 7.293,231 B1
`
`
`
`200
`
`202
`
`Initialization of
`Data Entry
`System
`
`Display Initialized
`User Interface
`
`NO
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`206
`
`
`
`Terminate
`data entry?
`
`Yes
`
`End
`
`Wait for user input
`
`ls user input
`a character
`Selection?
`
`Yes
`
`Add Character to
`search string
`
`
`
`Obtain
`Candidate
`List
`
`208
`
`
`
`
`
`No
`
`Carry Out
`Operation
`Corresponding to
`user input
`(e.g. terminate
`Current search)
`
`220
`
`210
`
`212
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 12 of 53
`
`
`
`218
`
`Disable
`Keyboard
`
`

`

`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 9 of 23
`
`US 7.293,231 B1
`
`222
`
`Display Interactive
`Search
`List Containing
`Candidate List
`
`
`
`Wait for user input
`
`inputagesture
`
`
`
`
`
`ift
`
`262
`
`Clear Candidate
`List but preserve
`search string.
`
`254
`
`Display Interactive
`Search
`List Containing
`Candidate List
`
`gesture
`
`
`
`
`
`5.SEwith 3.
`candidate?
`
`Yes
`
`228
`
`230
`
`
`
`
`
`Wait for user input
`
`Select the
`candidate.
`Start timer T2
`
`
`
`input a gesture
`
`No
`
`258
`
`gesture
`
`260
`
`Yes
`
`
`
`
`
`is gesture
`associated with a
`Candidate?
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 13 of 53
`
`FIG. 8
`
`232
`
`NO
`
`Yes
`
`23
`4
`
`Stop Timer T2.
`Set search string
`F Selected
`candidate
`
`
`
`Obtain
`Candidate
`List
`
`
`
`
`
`
`
`
`
`248
`
`246
`
`
`
`
`
`is gesture
`asSociated with a
`dead ZOne?
`
`
`
`Change
`Candidate
`Selection from list.
`Restart timer T2
`
`No
`
`ls
`gesture associated
`with a different
`Candidate?
`
`242
`
`Deselect Currently
`Selected
`Candidate in list.
`Disable timer T2.
`
`
`
`Monitor user input
`
`
`
`250
`
`ser inpu
`received?
`
`Accept candidate
`and add to text.
`Clear search
`string.
`
`

`

`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 10 of 23
`
`US 7.293,231 B1
`
`212
`
`Retrieve reference to the first
`entry in the dictionary beginning
`with the search string and
`allocate it to the first entry in the
`Search Span
`
`Retrieve reference to the last
`entry in the dictionary beginning
`with the search string and
`allocate it to the last entry in the
`Search Span
`
`Search SpanS
`predetermined maximum
`number?
`
`
`
`YeS
`
`
`
`
`
`
`
`
`
`FIG. 9
`
`Search through the range of
`entries defined by the Search
`Span for the most Common
`entries based on the weight
`value assigned to each entry
`
`Retrieve all entries within
`Search Span from the
`dictionary and compile into a list
`of completion candidates
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 14 of 53
`
`Set the list of completion
`candidates to the dictionary
`entries retrieved having the
`highest weight values, up to the
`predetermined maximum
`number
`
`

`

`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 11 of 23
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 15 of 53
`
`
`
`t
`
`w
`
`w
`
`s------ ?'
`
`FIG. 10
`
`

`

`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 16 of 53
`/
`/20 Page 16 of53
`
`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 12 of 23
`
`US 7,293,231 B1
`
`
`
`IE
`
`
`
`
`
`

`

`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 17 of 53
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 17 of 53
`
`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 13 of 23
`
`US 7,293,231 B1
`
`28
`
`

`

`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 14 of 23
`
`US 7,293,231 B1
`
`Candidate 1
`Candidate 2
`
`Candidate N
`
`FIG. 14
`
`? 3O
`
`r
`
`re
`
`-
`
`Candidate 2
`
`Candidate 5
`
`Candidate 1
`
`Candidate 3
`
`Candidate 4
`
`Candidate 9
`Candidate 5
`Candidate 8
`
`Candidate 2
`Candidate
`Candidate 4
`
`Candidate 6
`Candidate 3
`Candidate 7
`
`FIG. 15
`r
`
`FIG. 16
`r
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 18 of 53
`
`Candidate 5
`
`Candidate 2
`Candidate 1
`Candidate 4
`
`Candidate 3
`
`Candidate 2
`
`Candidate 1
`Candidate 4
`
`Candidate 3
`
`F.G. 17
`
`Candidate N
`
`FIG. 18
`
`

`

`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 15 Of 23
`
`US 7.293,231 B1
`
`
`
`Search String
`Box40
`
`Digital
`Keyboard
`28
`
`Display Area
`for Interactive
`Search List
`30
`
`Toolbar 42
`
`FIG. 19
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 19 of 53
`
`

`

`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 20 of 53
`
`U.S. Patent
`
`Nov. 6 2007
`
`Sheet 16 of 23
`
`
`
`

`

`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 17 of 23
`
`US 7.293,231 B1
`
`200
`
`Terminat
`eminate
`data entry?
`
`
`
`
`
`
`
`
`
`
`
`Carry out
`Operation
`corresponding to
`user input
`(eg. terminate
`Current search)
`
`Yes
`
`End
`
`220
`
`Yes
`
`202
`
`204
`N O
`
`206
`
`208
`
`No
`
`Initialization of
`Data Entry
`System
`
`Display initialized
`User Interface
`
`Wait for user input
`
`
`
`
`
`
`
`
`
`Yes
`
`Add character to
`search string
`
`Yes
`
`Obtain
`Candidate
`List
`
`
`
`213A
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 21 of 53
`
`
`
`
`
`
`
`
`
`
`
`
`
`Display indicator
`of Candidate List
`Size. Display
`Indicator of total
`potential
`Candidates
`aSSOciated with
`the search string.
`
`
`
`Has pointing
`device been released
`ithin time limit L12
`
`
`
`
`
`Disable
`Keyboard
`
`
`
`

`

`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 18 of 23
`
`US 7.293,231 B1
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`254
`Display Interactive lift
`Search
`List Containing
`Candidate List.
`Display indicator
`of total potential
`Candidates for the
`Search string.
`
`Wait for user input
`
`input agesture
`
`No
`
`Display interactive
`Search List
`
`Wait for user input
`
`inputagesture
`
`gesture
`
`228
`
`262
`
`Clear Candidate
`List but preserve
`search string,
`
`is gesture
`alE. 3.
`a
`
`
`
`Yes
`Select the
`Candidate.
`Start timer T2
`
`248
`
`
`
`230
`
`Deselect currently
`Selected
`(-Yes
`candidate in list.
`Disable timer T2.
`
`is gesture
`aSSOciated with a
`dead ZOne?
`
`
`
`246
`
`232
`
`No-Monitor user input
`
`
`
`Change
`Candidate
`Selection from list.
`Restart timer T2
`
`N
`O
`
`N
`O
`
`258
`
`Yes
`
`234
`
`gesture
`
`is gesture
`aSSOciated with a
`Candidate?
`
`260
`
`Yes
`
`Stop Timer T2.
`Set search string
`Selected
`Candidate
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 22 of 53
`
`
`
`Obtain
`Candidate
`List
`
`250
`
`Ser inpu
`received?
`
`
`
`Yes
`
`
`
`252
`inputagesture
`
`236
`
`
`
`238
`
`ls
`gesture associated
`with a different
`Candidate?
`
`242
`
`gesture
`
`FIG.22
`
`
`
`
`
`lift
`
`240
`
`ACCept Candidate
`and add to text.
`Clear search
`String.
`
`

`

`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 19 of 23
`
`US 7.293,231 B1
`
`Initialization of
`Data Entry
`System
`
`Display initialized
`User Interface
`
`202
`
`204
`NO
`
`
`
`206
`
`200
`
`Terminat
`erate
`data entry?
`
`Yes
`
`End
`
`
`
`
`
`
`
`Carry Out
`operation
`corresponding to
`user input
`(e.g. terminate
`Current search)
`
`208
`
`No
`
`
`
`220
`
`Yes
`
`
`
`Yes
`
`
`
`213B
`
`Add character to
`search string
`
`
`
`Obtain
`Candidate
`List
`
`Display interactive
`Search List
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 23 of 53
`
`
`
`
`
`
`
`
`
`Has pointing
`device been released
`ithin time limit L12
`
`
`
`Disable
`Keyboard and
`display search
`mode indicator.
`
`FIG. 23
`
`

`

`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 20 of 23
`
`US 7.293,231 B1
`
`
`
`
`
`
`
`254
`
`lift
`
`Display Interactive
`Search
`List Containing
`Candidate List.
`
`256
`
`
`
`Wait for user input
`
`input a gesture
`
`No
`
`Wait for user input
`1"
`
`Clear Candidate
`List but preserve
`search string,
`
`262
`
`input a gesture
`
`gesture
`
`228
`
`
`
`ls gesture
`asSociated with a
`Candidate?
`
`
`
`Yes
`
`Select the
`candidate.
`Start timer T2
`
`248
`
`
`
`246
`
`230
`
`Deselect currently
`1-Yes
`Selected
`candidate in list.
`Disable timer T2.
`
`is gesture
`asSociated with a
`dead Zone?
`
`gesture
`
`258
`
`
`
`260
`
`is gesture
`associated with a
`Candidate?
`
`234
`
`Yes
`Stop Timer T2.
`Set search string
`F Selected
`Candidate
`
`Obtain
`Candidate
`List
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 24 of 53
`
`FIG. 24
`
`
`
`Change
`candidate
`Selection from list.
`Restart timer T2
`
`No
`
`ls
`gesture associated
`with a different
`Candidate?
`
`gesture
`
`242
`
`232
`
`No
`
`Monitor user input
`
`No
`
`
`
`236
`
`238
`
`240
`
`250
`
`serinpu
`received?
`
`
`
`Yes
`
`
`
`252
`inputagesture
`
`lift
`Accept candidate
`and add to text.
`Clear Search
`String. Display
`keyboard mode
`indicator.
`
`
`
`

`

`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 21 of 23
`
`US 7.293,231 B1
`
`202
`
`204
`No
`
`2O6
`
`
`
`
`
`
`
`
`
`208
`
`NO
`
`
`
`200
`
`
`
`Terminat
`elate
`data entry?
`
`Yes
`
`End
`
`Carry Out
`Operation
`Corresponding to
`user input
`(eg. terminate
`Current search)
`
`220
`
`Yes
`
`Initialization of
`Data Entry
`System
`
`Display Initialized
`User Interface
`
`Wait for user input
`
`
`
`
`
`Yes
`
`Add character to
`search string
`
`Obtain
`Candidate
`List from
`multiple
`dictionaries
`
`Display indicator
`of Candidate List
`
`Yes
`
`
`
`
`
`213C
`
`
`
`
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 25 of 53
`
`

`

`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet 22 of 23
`
`US 7.293,231 B1
`
`
`
`
`
`
`
`
`
`lift
`
`
`
`
`
`
`
`
`
`
`
`254
`
`Display Interactive
`Search
`List Containing
`Candidate List.
`
`Wait for user input
`
`262
`
`Clear Candidate
`List but preserve
`Search string.
`
`Wait for user input
`
`inputagesture
`
`gesture
`
`228
`
`is gesture
`asSociated with a
`Candidate?
`
`
`
`Yes
`
`Select the
`Candidate.
`Start timer T2
`
`248
`
`
`
`246
`
`230
`
`Deselect currently
`Selected
`candidate in list.
`Disable timer T2.
`
`1-Yes
`
`ls gesture
`asSociated with a
`dead ZOne?
`
`input a gesture
`
`NO
`
`258
`
`gesture
`
`260
`
`Yes
`
`is gesture
`aSSOciated With a
`Candidate?
`
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 26 of 53
`
`FIG. 26
`
`234
`
`Yes
`Stop Timer T2.
`Set search string
`F Selected
`Candidate
`
`Obtain
`Candidate
`List from
`multiple
`dictionaries
`
`232
`
`NO
`
`Monitor user input
`
`
`
`Change
`Candidate
`Selection from list,
`Restart timer T2
`
`NO
`
`
`
`
`
`
`
`ls
`gesture aSSOciated
`with a different
`Candidate?
`
`242
`
`input a gesture
`
`
`
`gesture
`
`lift
`
`
`
`240
`
`Accept candidate
`and add to text.
`Clear Search
`string.
`
`

`

`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 27 of 53
`Case 5:20-cv-01756-NC Document 1-1 Filed 03/11/20 Page 27 of 53
`
`U.S. Patent
`
`Nov. 6, 2007
`
`Sheet