Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 1 of 49
`Case 5:20-cv-01756—NC Document 1-2 Filed 03/11/20 Page 1 of 49
`
`
`
`
`
`
`
`
`EXHIBIT 2
`
`EXHIBIT 2
`
`

`

`USOO7681 124B2
`
`(12) United States Patent
`Gunn et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,681,124 B2
`Mar. 16, 2010
`
`(54)
`
`(75)
`
`DATA ENTRY FOR PERSONAL COMPUTING
`DEVICES
`
`Inventors: Harold David Gunn, Vancouver (CA);
`John Chapman, Vancouver (CA)
`
`(73)
`
`Assignee: 602531 British Columbia Ltd. (CA)
`
`(*)
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 1120 days.
`
`(21)
`
`Appl. No.: 11/134,810
`
`(22)
`
`Filed:
`
`May 19, 2005
`
`(65)
`
`Prior Publication Data
`US 2005/0210O2O A1
`Sep. 22, 2005
`
`(60)
`
`Related U.S. Application Data
`Division of application No. 09/631,101, filed on Aug.
`1, 2000, now Pat. No. 7.293,231, which is a continua
`tion of application No. PCT/CA00/00285, filed on
`Mar. 15, 2000, which is a continuation-in-part of appli
`cation No. 09/272,700, filed on Mar. 18, 1999, now
`abandoned.
`
`(51)
`
`Int. C.
`G06F 7700
`
`(2006.01)
`
`(52)
`
`(58)
`
`U.S. Cl. ....................... 715/256: 715/261; 715/268;
`715/760; 34.5/179; 382/187
`Field of Classification Search ................. 715/.530,
`715/531, 534,541, 255, 256, 261,268, 760;
`365/189.01; 34.5/156, 173, 179; 382/187
`See application file for complete search history.
`
`
`
`(56)
`
`References Cited
`
`EP
`
`U.S. PATENT DOCUMENTS
`3,644,898 A
`2, 1972 Post
`(Continued)
`FOREIGN PATENT DOCUMENTS
`O352377 A1
`1, 1990
`(Continued)
`OTHER PUBLICATIONS
`Bohlman, Eric, "KeyCache The Universal Input Accelerator, Ver
`sion 2.1.” copyright Sep. 29, 1994, OMS Development, documenta
`tion,
`downloaded from <"http://www.sac.sk/files.php?d=16
`&I=K'>, 12 pages.*
`
`(Continued)
`Primary Examiner Doug Hutton
`Assistant Examiner James H Blackwell
`(74) Attorney, Agent, or Firm Townsend and Townsend and
`Crew LLP
`
`ABSTRACT
`(57)
`In one aspect, the user can rapidly enter and search for text
`using a data entry system through a combination of entering
`one or more characters on a digitally displayed keyboard with
`a pointing device and using a search list to obtain a list of
`completion candidates. The user can activate the search list to
`obtain a list of completion candidates at any time while enter
`ing a partial text entry with the data entry system. When the
`search list is active, a list of completion candidates is dis
`played on a graphical user interface for the user to select from
`and the user can perform one of several actions. When the user
`deactivates the search list, the user can continue adding to or
`modifying the current partial text entry being entered, and
`may re-invoke the search list to further search for completion
`candidates based on the modified partial text entry.
`
`41 Claims, 23 Drawing Sheets
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 2 of 49
`
`statisbove)
`
`

`

`US 7,681,124 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`7/1980 Montgomery ............... 400,486
`4,211,497 A
`5, 1982 Damerau
`4,330,845. A
`8/1983 Hayashi et al. ............. 364,900
`4,396,992 A
`9, 1984 Dickinson et al.
`4,471.459 A
`2, 1985 Dickinson et al.
`4,499,553 A
`4,559,598 A 12, 1985 Goldwasser et al. ........ 364/419
`4,648,044 A
`3/1987 Hardy et al.
`4,689,768 A
`8, 1987 Heard et al.
`4,730,252 A
`3/1988 Bradshaw ................... 364,403
`4,744,050 A
`5, 1988 Hirosawa et al.
`... 364,900
`4,774,666 A
`9/1988 Miyao et al. ......
`... 364/419
`RE32,773 E
`10/1988 Goldwasser et al. ........ 364/419
`4,782.464 A 1 1/1988 Gray et al.
`4,783,761. A 1 1/1988 Gray et al.
`4,786,765 A 11, 1988 Yamanami et al. ....... 178/1906
`4,807,181 A
`2, 1989 Duncan, IV et al. ........ 364,900
`4,847,766 A
`7, 1989 McRae et al.
`4,891,786 A
`1/1990 Goldwasser ................ 364,900
`4.969,097 A 1 1/1990 Levin ......................... 364/419
`4,980,855 A 12/1990 Kojima
`5,040,113 A
`8, 1991 Mickunas ................... 364/419
`5,060,154. A 10/1991 Duncan, IV
`5,067,165 A 11, 1991 Nishida
`5,096,423. A
`3, 1992 Goldwasser ................ 434/118
`5,203,704 A
`4, 1993 McCloud ....
`434,156
`5,220,649 A
`6, 1993 Forcier ....
`715,541
`5,220,652 A
`6/1993 Rowley ...................... 395/275
`5,258,748 A 11/1993 Jones
`5,261,112 A 11/1993 Futatsugi et al.
`5,297,041 A
`3, 1994 Kushler et al. ......... 364,419.15
`5,305,205 A
`4, 1994 Weber et al.
`5,327,161 A
`7, 1994 Logan et al. ..............., 345,157
`5,329,609 A
`7, 1994 Sanada et al.
`5,347,295 A
`9/1994 Agulnicket al. ............ 345,156
`5,392.447 A
`2f1995 Schlack et al.
`5.487,616 A
`1/1996 Ichbiah ....................... 400,489
`5,500,935 A
`3, 1996 Moran et al. ................ 395,156
`5,559,942 A
`9/1996 Gough et al.
`5,574.482 A 1 1/1996 Niemeier .................... 345,173
`5,594,640 A
`1/1997 Capps et al.
`5,596,699 A
`1/1997 Driskell ...................... 395,352
`5,606,674. A
`2, 1997 Root
`5,621,641 A
`4, 1997 Freeman ..................... 395/796
`5,623.406 A
`4, 1997 Ichbiah ....................... 395/753
`5,649,223. A
`7/1997 Freeman ..................... 395/796
`5,657,397 A
`8, 1997 BokSer
`5,666,139 A
`9, 1997 Thielens et al.
`5,689,667 A 11/1997 Kurtenbach ................. 395,352
`5,704,029 A 12/1997 Wright, Jr. .................. 715,505
`5,724.457 A
`3, 1998 Fukushima
`5,734,749 A
`3, 1998 Yamada et al.
`5,745,116 A
`4, 1998 Pisutha-Arnond ........... 345,358
`5,790,115 A
`8/1998 Pleyer et al.
`5,805,158 A
`9, 1998 Bertram et al.
`5,805,159 A
`9, 1998 Bertram et al.
`5,805,167 A
`9/1998 Van Cruyningen .......... 345.353
`5,805,911 A
`9, 1998 Miller ........................ 395/796
`5,812,697 A
`9, 1998 Sakai et al.
`5,818,437 A 10/1998 Grover et al.
`5,821.512. A 10/1998 O'Hagan et al.
`5,835,635 A 1 1/1998 Nozaki et al.
`5,838,302 A 1 1/1998 Kuriyama et al.
`5,845,300 A 12/1998 Comer et al.
`5,864,340 A
`1/1999 Bertram et al.
`5,881,169 A
`3/1999 Henry, Jr.
`5,896,321 A
`4/1999 Miller et al. ........... 365,189.01
`5,911,485 A
`6, 1999 Rossman
`5,914,708 A
`6/1999 LaGrange et al. ........... 345,179
`5,926, 178 A
`7, 1999 Kurtenback ................. 345,352
`5,943,039 A
`8, 1999 Anderson et al. ........... 345/146
`5,953,541 A
`9/1999 King et al.
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 3 of 49
`
`9/1999 Masui ........................ 345,347
`5,959,629 A
`5,963,666. A 10/1999 Fujisaki et al. ..
`382,187
`5,963,671. A 10/1999 Comerford et al. .......... 382,230
`5.974.558 A 10/1999 Cortopassi et al.
`5,977,887 A 11/1999 Grimmett
`5,977,948 A 11, 1999 Nishbori
`5,982,351 A 11/1999 White et al.
`345,173
`6,002,390 A 12/1999 Masui
`345,157
`6,005,549 A 12/1999 Forest
`6,008,799 A 12/1999 Van Kleeck ................. 345,173
`6,011,554 A
`1/2000 King et al.
`6,026,233 A
`2/2000 Shulman et al.
`6,037.942 A
`3/2000 Millington
`6,084,576 A
`7/2000 Leu et al. .................... 345,168
`6,088,649 A
`7/2000 Kadaba et al.
`6,094,197 A
`7/2000 Buxton et al. ............... 345,863
`6,097,392 A
`8/2000 Leyerle
`6,097.841 A
`8/2000 Gunji et al.
`6,098,086 A
`8/2000 Krueger et al.
`6, 101.461 A
`8/2000 Ukigawa et al.
`6,111,985 A
`8, 2000 Hullender et al.
`6,144,378 A 11/2000 Lee
`6,154,758 A 1 1/2000 Chiang
`6,167.411 A 12/2000 Narayanaswamy
`6,167.412 A 12/2000 Simons
`6,188,789 B1
`2/2001 Marianetti, II et al. ...... 382, 189
`6.256,030 B1
`7/2001 Berry et al.
`6,262,719 B1
`7/2001 Bi et al. ...................... 345,179
`6,275,612 B1
`8, 2001 Imoto
`6,282,315 B1
`8/2001 Boyer
`6,286,064 B1
`9/2001 King et al.
`6,292, 179 B1
`9/2001 Lee ............................ 345,173
`6,295,372 B1
`9/2001 Hawkins et al. .
`382,187
`6,307,549 B1
`10/2001 King et al. ...
`345,352
`6,369,807 B1
`4/2002 Nakashima ....
`... 345,179
`6,377,965 B1
`4/2002 Hachamovitch et al. .... 715,534
`6,405,060 B1
`6/2002 Schroeder et al.
`6,442,295 B2
`8, 2002 Navoni et al.
`6,473,006 B1
`10/2002 Yu et al.
`6,487,424 B1
`1 1/2002 Kraft et al.
`6,502,114 B1
`12/2002 Forcier
`6,539,421 B1
`3/2003 Appelman et al.
`6,621,939 B1
`9/2003 Negishi et al.
`6,654,733 B1
`1 1/2003 Goodman et al. ............. 7O6/52
`6,661,920 B1
`12/2003 Skinner ...................... 382,187
`6,751,603 B1
`6/2004 Bauer et al.
`6,801,190 B1
`10/2004 Robinson et al.
`6,888,141 B2
`5, 2005 Carr
`6,934,906 B1
`8, 2005 Cheok
`6,970,513 B1 1 1/2005 Puri et al.
`6,978.421 B1
`12/2005 Aida
`7,003,446 B2
`2/2006 Trower et al.
`7,224.409 B2
`5/2007 Chin et al.
`7,293,231 B1
`1 1/2007 Gunn et al.
`7,322,023 B2
`1/2008 Shulman et al.
`2001/0000962 A1
`5/2001 Rajan
`2001/0027468 A1 10, 2001 Okura
`2002/0067377 A1
`6/2002 McGovern .................. 345,816
`2002fO087279 A1
`7, 2002 Hall
`2003.0137605 A1
`7, 2003 Chin et al.
`2005, 0198144 A1
`9, 2005 Kraenzel et al.
`2005/0210402 A1
`9, 2005 Gunn et al.
`2005/0223308 A1 10, 2005 Gunn et al.
`2007/0188472 A1
`8/2007 Ghassabian
`
`
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`EP
`EP
`EP
`EP
`EP
`
`O 643 357 A2
`O 643 357 A3
`O 844 570 A2
`O 844 571 A2
`O 858 023 A2
`O472444
`
`3, 1995
`3, 1996
`5, 1998
`5, 1998
`8, 1998
`8, 1998
`
`

`

`US 7,681,124 B2
`Page 3
`
`O 858 O23 A3
`EP
`O 844 570 A3
`EP
`O 844 571 A3
`EP
`O982676
`EP
`101.05324 A
`JP
`10154033. A
`JP
`10154144 A
`JP
`11143614
`JP
`11167569 A
`JP
`10333818
`JP
`WO96,09579
`WO
`WO98, 11480
`WO
`WO99,28811
`WO
`WOO195095
`WO
`WO WO 2004 107101
`
`9, 1998
`8, 1999
`8, 1999
`3, 2000
`4f1998
`6, 1998
`6, 1998
`5, 1999
`6, 1999
`4/2002
`3, 1996
`3, 1998
`6, 1999
`12/2001
`12, 2004
`
`OTHER PUBLICATIONS
`NCIP listing of Word Prediction & Writing Tools, circa 1997, down
`loaded from <"http://www2.edc.org/incip/LIBRARY/wp/Pubs.
`htm'>, 7 pages.*
`National Council on Disability Document Archive, software guide to
`alternative input and output programs, circa 1996, downloaded from
`<"http://www.dimenet.com/disnews/archive.php?mode=P
`&id=49'>, 12 pages.*
`Wivik 2 REP Software manual, circa Feb. 1998, Orentke Romich
`Company, downloaded from <"http://web.archive.org/web/
`2000 1204165600/wivik.com/html/downloads.htm>, 128 pages.*
`Softype Version 3.1 Screendumps generated from demo version of
`software, downloaded from <"http://orin.com/binaries/st31dmz.
`exe'>, 21 Figures.*
`Softype 3.1 Help File, circa 1997, downloaded from <"http://orin.
`com/binaries/st31dmz.exe">, 38 pages.*
`Toshiyuki Masui, Sony Computer Science Laboratory Inc.,
`Shinagawa, Tokyo, "An Efficient Text Input Method for Pen-based
`Computers'. Proceedings of the ACM Conference on Human Factors
`in Computing Systems, Apr. 1998, pp. 328-335.
`Toshiyuki Masui, Sony Computer Science Laboratories, Inc.,
`Shinagawa, Tokyo, “Integrating Pen Operations for Composition by
`Example”.
`Don Hopkins, “The Design and Implementation of Pie Menus'. Dr.
`Dobb's Journal, Dec. 1991 (http://art.net/~hopkins/Don/piemenus/
`ddi/piemenus.hml).
`Jason I. Hong, “Java Pie Menus'. Sep. 4, 1999 (ttp://www.cs.
`berkeley.edu/~asonht download/sotware/piemenu/).
`Don Hopkins, “A Description of Pie Menus' (http://catalog.com/
`hopkins/piemenus/PieMenuDescription.html).
`Don Hopkins, “Natural Selection: The Evolution of Pie Menus',
`BayCHI, Oct. 13, 1998 (http://catalog.com/hopkins/piemenus/Natu
`ral Selection.html).
`Russell Nelson, “Pie Menu Window Manager'. Jun. 3, 1998 (http://
`www.crynwr.com/piewm/).
`Don Hopkins, “ActiveX Pie Menus' (http://catalog.com/hopkins/
`piemenus/ActiveXPieMenus.html).
`Don Hopkins, “Pie Menu References” (http://catalog.com/hopkins/
`piemenu-references.html).
`Don Hopkins, “Pie Menus' (http://art.net/~hopkins/Don/piemenus/
`index.html).
`Tom Nantais, et al., IEEE Transactions on Rehabilitation Engineer
`ing, "A Predictive Selection Technique for Single-Digit Typing With
`a Visual Keybaord’. No. 3 Sep. 1994.
`“GtkPieMenu' 1999 orfelyus (2 pages).
`Operation Guide: 1997: 2 pages; Mercedes-Benz of North America,
`Inc.; Montvale, New Jersey, United States.
`Auto Pilot System—Operation Guide; 1998; 2 pages; Mercedes
`Benz AG; Germany.
`Comand Betriebsanleitung- Operation Guide; 1998; 3 pages;
`Mercedes-Benz AG; Germany.
`3Com. “PalmPilot TMHandbook” 1997, 3Com Corporation, Table of
`Contents and pp. 37-39.
`Advisory Action dated Sep. 16, 2005, issued in related U.S. Appl. No.
`09/631, 101, filed Aug. 1, 2000.
`
`Baeza-Yates, R.A. et al. "A New Approach to Text Searching (Pre
`liminary Version).” Oct. 1992, Communications of the ACM, vol. 35.
`No. 10, pp. 74-82.
`Bellman, T. et al. "A Probabilistic Character Layout Strategy for
`Mobile Text Entry.” Proceedings of Graphics Interface '98, 1998,
`Canadian Information Processing Society: Toronto, pp. 168-176.
`Bohan, M. et al. “A Psychophysical Comparison of Two Stylus
`Driven Soft Keyboards.” 1998, Department of Psychology, Wichita
`State University: Wichita, Kansas, six pages.
`Callan, J. etal. "An Empirical Comparison of Pie vs. Linear Menus.”
`May 15-19, 1988, Proceedings of the CHI '88 Conference on Human
`Factors in Computing Systems, Washington, D.C., pp. 95-100,
`Abstract.
`Darragh, J.J. et al. “The Reactive Keyboard: A Predictive Typing
`Aid.” Nov. 1990, IEEE Computer, vol. 23, No. 11 pp. 41-49.
`European Examination Report dated Feb. 7, 2002, issued in related
`European Application No. 00910 460.5.
`European Examination Report dated Nov. 8, 2004, issued in related
`European Application No. 01981 991.1.
`European Examination Report dated Jun. 23, 2005, issued in related
`European Application No. 01981 991.1.
`Garay-Vitoria, N. et al. "A Comparison of Prediction Techniques to
`Enhance the Communication Rate.” 2004, U14AII, LNCS, Springer
`Verlag, vol. 3196, pp. 400-417.
`International Preliminary Examination Report dated Jun. 19, 2001,
`issued in related PCT Application No. PCT/CA00/00285 filed Mar.
`15, 2000.
`International Preliminary Examination Report dated Sep. 12, 2003,
`issued in related PCT Application No. PCT/CA01/01473 filed Oct.
`18, 2001.
`Jones, P.E. "Virtual Keyboard with Scanning and Augmented by
`Prediction.” 1998, Proc. 2nd Euro. Conf. Disability Virtual Reality &
`Assoc. Tech. Skovde, Sweden, pp. 45-51.
`MacKenzie, I.S. et al. “Alphanumeric Entry on Pen-Based Comput
`ers.” 1994. International Journal of Human-Computer Studies vol.
`41, pp. 775-792, located at <http://www.yorku.ca/mack/IJHCS.
`html>, last visited on Nov.30, 2006, eighteen pages.
`MacKenzie, I.S. et al. “Text Entry Using Soft Keyboard.” 1999,
`Behaviour & Information Technology, vol. 18, pp. 235-244, located
`at <http://www.yorku.ca/mack/BIT3.html>, last visited on Nov.30,
`2006, fifteen pages.
`MacKenzie, I.S. etal. “The Design and Evaluation of a High-Perfor
`mance SoftKeyboard.” May 15-20, 1999, ACM, pp. 25-31.
`Mankoff, J. etal. “Cirrin: A World-Level Unistroke Keyboard for Pen
`Input.” 1998, ACM, pp. 213-214.
`Masui, T. “POBox: An Efficient TextInput Method for Handheld and
`Ubiquitous Computers.” Date Unknown, Proc. of Ist Intl. Symp. On
`Handheld and Ubiquitous Computing, Computer Science, vol. 1707.
`pp. 289-300.
`Notice of Allowability dated Aug. 8, 2007, issued in related U.S.
`Appl. No. 09/631, 101, filed Aug. 1, 2000.
`Office Action dated Jan. 20, 2004, issued in related U.S. Appl. No.
`09/631, 101, filed Aug. 1, 2000.
`Office Action dated Jan. 14, 2005, issued in related U.S. Appl. No.
`09/631, 101, filed Aug. 1, 2000.
`Office Action dated Jun. 14, 2005, issued in related U.S. Appl. No.
`09/631, 101, filed Aug. 1, 2000.
`Office Action dated Mar. 20, 2006, issued in related U.S. Appl. No.
`09/631, 101, filed Aug. 1, 2000.
`Office Action dated Jan. 5, 2007, issued in related U.S. Appl. No.
`09/631, 101, filed Aug. 1, 2000.
`Office Action dated Jun. 4, 2007, issued in related U.S. Appl. No.
`09/631, 101, filed Aug. 1, 2000.
`Office Action dated Oct. 18, 2006, issued in related U.S. Appl. No.
`10/399,560, filed Apr. 18, 2003.
`Office Action dated Jul. 17, 2007, issued in related U.S. Appl. No.
`10/399,560, filed Apr. 18, 2003.
`Office Action dated Jun. 7, 2007, issued in related U.S. Appl. No.
`1 1/133,770, filed May 19, 2005.
`Office Action dated May 18, 2007, issued in related U.S. Appl. No.
`1 1/134,759, filed May 19, 2005.
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 4 of 49
`
`

`

`US 7,681,124 B2
`Page 4
`
`Venolia, D. etal. “T-Cube: A Fast, Self-Disclosing Pen-Based Alpha-
`bet.” Apr. 24-28, 1994, Human Factors in Computing Systems, Bos-
`ton, MA, pp. 265-270.
`Wu, S. etal “AGREP—A Fast Approximate Pattern-Matching Tool.”
`Jan. 1992, Proceedings of USENIX Technical Conference, San Fran-
`cisco, CA, pp. 153-162.
`Extended European Search Report dated May 27, 2009, for European
`Patent Application No. 07022182.5.
`Office Action dated Apr. 15, 2008, issued for U.S. Appl. No.
`10/399,560.
`Office Action dated Apr. 3, 2008, issued for U.S. Appl. No.
`11/134,759.
`Office Action dated Apr. 30, 2008, issued for U.S. Appl. No.
`11/036,267.
`Office Action dated Feb. 4, 2009, issued for U.S. Appl. No.
`11/134,759.
`
`Office Action dated Jan. 26, 2009, issued for U.S. Appl. No.
`11/036,267.
`Office Action dated Jan. 7, 2009, issued for U.S. Appl. No.
`10/399,560.
`Office Action dated Jun. 10, 2009, issued for U.S. Appl. No.
`11/133,770.
`Office Action dated Mar. 4, 2008, issued for U.S. Appl. No.
`11/133,770.
`Office Action dated Nov. 24, 2008, issued for U.S. Appl. No.
`11/133,770.
`Partial European Search Report dated Mar. 4, 2009, for European
`Patent Application No. 07022182.5.
`Office Action dated Sep. 29, 2009 issued for U.S. Appl. No.
`10/399,560.
`* cited by examiner
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 5 of 49
`
`

`

`U.S. Patent
`
`Mar. 16, 2010
`
`Sheet 1 of 23
`
`US 7,681,124 B2
`
`
`
`Display Device
`15
`
`4.
`
`10
`
`Application
`27
`Data entry system
`26
`
`Computer-readable medium
`16
`
`Processing Unit
`12
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 6 of 49
`
`FIG. 1
`
`

`

`U.S. Patent
`
`Mar. 16, 2010
`
`Sheet 2 of 23
`
`US 7,681,124 B2
`
`Data Entry System
`
`26
`
`Digital Keyboard System
`
`Candidate Prediction
`System
`
`Dictionary
`
`Logic for interactive
`Searchist
`
`FIG. 2
`
`Span to it
`
`Completion
`Candidate Field 22
`
`Weight
`Field 24
`
`Dictionary Data
`Structure
`
`
`
`
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 7 of 49
`
`

`

`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 8 of 49
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 8 of 49
`
`US. Patent
`
`Mar. 16, 2010
`
`Sheet 3 of 23
`
`US 7,681,124 B2
`
`
`
`

`

`U.S. Patent
`
`Mar. 16, 2010
`
`Sheet 4 of 23
`
`US 7,681,124 B2
`
`14
`
`10
`
`Digital
`Keyboard
`
`28
`
`N
`
`
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 9 of 49
`
`Digital
`Keyboard
`28
`
`interactive
`Search
`List
`30
`
`FIG. 5
`
`

`

`U.S. Patent
`
`Mar. 16, 2010
`
`Sheet 5 of 23
`
`US 7,681,124 B2
`
`Potential completion candidates
`in the dictionary for apartial text
`entry made up of a single
`character
`
`Potential Candidate 1
`Potential Candidate 2
`Potential Candidate 3
`Potential Candidate 4
`Potential Candidate 5
`Potential Candidate 6
`Potential Candidate 7
`Potential Candidate 8
`Potential Candidate 9
`Potential Candidate 10
`Potential Candidate 11
`Potential Candidate 12
`Potential Candidate 13
`Potential Candidate 14
`Potential Candidate 15
`Potential Candidate 6
`Potential Candidate 17
`Potential Candidate 18
`
`Potential Candidate 1
`Potential Candidate 5
`Potential Candidate 2
`Potential Candidate 16
`Potential Candidate 18
`
`partial text entry set to
`Potential Candidate 5
`
`)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Portion of
`the
`dictionary
`
`Displayed
`in the
`interactive
`Search list
`
`Potential completion candidates
`in the dictionary when the
`partial text entry equals
`Potential Candidate 5
`
`
`
`Potential Candidate 5
`Potential Candidate 6
`Potential Candidate 9
`Potential Candidate 10
`Potential Candidate 11
`Potential Candidate 12
`
`Potential Candidate 5
`Potential Candidate 10
`Potential Candidate 6
`Potential Candidate 9
`Potential Candidate 12
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 10 of 49
`
`
`
`List of completion candidates
`obtained by the candidate
`prediction system from the
`above 18 potential candidates
`in the dictionary
`
`List of completion candidates
`obtained by the candidate
`prediction system from the
`above 7 potential candidates in
`the dictionary
`
`
`
`FIG. 5A
`
`

`

`U.S. Patent
`
`Mar. 16, 2010
`
`Sheet 6 of 23
`
`US 7,681,124 B2
`
`
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 11 of 49
`
`120 if the user
`gestures towarda
`candidate in the refined
`candidatest, that
`Candidate is Selected
`and the user can use
`he refined list in the
`Sarne?anner as with
`the previous candidate
`list (at 112 above).
`
`e
`
`124: The usercar.
`continue building or modifying
`partial text entry or
`) initiate a further search on the
`partial textentry or
`cance the search string of
`revert to last candidate list of
`revert to the last search string Of
`carry out another operation
`
`Search string empty.
`
`Digital
`Keyboard
`
`24
`
`Search string =
`end.
`
`FIG. 6
`
`

`

`U.S. Patent
`
`Mar. 16, 2010
`
`Sheet 7 of 23
`
`US 7,681,124 B2
`
`102: User selects the
`key containing'e' on
`the keyboard. The
`Selection forms part
`of the partial text
`
`the Search string.
`
`10: if the user
`gestures to a dead
`ZOne, the Selected
`Candidate is de
`Selected and the list
`of Candidates
`remains displayed.
`
`18: With a
`Candidate selected, if
`the user gestures to
`another candidate,
`the Selection
`changes.
`
`104; if the user does not
`release the keyboard
`Selection in less than time
`limit L1, the Candidate
`Prediction System obtains a
`Candidate list based on the
`
`displayed in an interactive
`search list.
`
`Monitor pointing
`
`108; if usergestures
`toward a candidate
`(without lifting), that
`candidate is hi-lited
`and becomes the
`Selection.
`
`EXAMPLE 100
`
`106: if pointing
`device is tifted up
`without significant
`movement, the
`interactive Search list
`is cleared and the
`keyboard is enabled.
`
`110: if the user
`gestures to a dead
`Zone, the Selected
`Candidate is de
`Selected and the list
`of candidates
`remains displayed.
`
`
`
`
`
`16: if the user continues to
`keep a candidate selected for
`more than defined time limit L2,
`a refined list of candidates is
`obtained using the selected
`Candidate and displayed in the
`interactive search list.
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 12 of 49
`
`120: if the user gestures toward
`a Candidate in the refined
`candidate list, that candidate is
`Selected and the user can use
`the refined list in the same
`manner as with the previous
`Candidate list.
`
`114: if the userlifts up
`the pointing device in
`less than time limit L2
`With a candidate
`Selected, the selected
`Candidate is added to the
`text in place of the partial
`text entry, the list is
`cleared and the
`keyboard is enabled.
`
`122: if the pointing device is lifted up
`without significant movement, the
`interactive Search list is cleared, the
`keyboard is enabled, the search is
`terminated and the most recently
`Selected candidate is added to the text
`in place of the partial text entry.
`
`FIG. 6A
`
`124. The usercan:
`continue building or
`modifying partial
`text entry or
`initiate a further
`search on the partial
`text entry or
`cancel the search
`string Or
`revert to last
`Candidate list Of
`revert to the last
`search string or
`carry out another
`Operation
`
`

`

`U.S. Patent
`
`Mar. 16, 2010
`
`Sheet 8 of 23
`
`US 7,681,124 B2
`
`200
`
`220
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`initiatization of
`Data Entry
`System
`
`O
`is user input
`a character
`election?
`
`Obtain
`Candidate
`List
`
`Terminate
`data entry?
`
`Carry out
`Operation
`Corresponding to
`user input
`(eg. terminate
`Cu?tent search)
`
`-No
`
`
`
`
`
`
`
`
`
`No
`
`210
`
`212
`
`214
`
`Has
`pointing device been
`eleased within time
`innit?
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 13 of 49
`
`

`

`U.S. Patent
`
`Mar. 16, 2010
`
`Sheet 9 of 23
`
`US 7,681,124 B2
`
`222
`
`
`
`Display interactive
`Search
`list containing
`Candidate st
`
`
`
`Wait for user input
`
`262
`
`Clear Candidate
`List but preserve
`search sting.
`
`254
`Display interactive
`Search
`List containing
`Candidate list
`
`Wait for user input
`
`
`
`input a gesture
`
`No
`
`
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 14 of 49
`
`
`
`asSociated with a
`Candidate?
`
`
`
`230
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Select the
`candidate.
`Startner 2
`
`Yes
`
`Stop Timer T2.
`Set search sting
`= selected
`candidate
`
`Obtain
`Candidate
`list
`
`248
`
`
`
`246
`
`Deselect currently
`Selected
`candidate in list.
`Disable timer T2.
`
`Yes
`
`is gesture
`asSociated with a
`dead zone
`
`
`
`Change
`Candidate
`Selection from list.
`Restart timer 2
`
`No
`
`
`
`s
`gesture associated
`with a different
`candidate?
`
`242
`
`FIG. 8
`
`Accept candidate
`and add to text.
`Clear Search
`string,
`
`
`
`

`

`U.S. Patent
`
`Mar. 16, 2010
`
`Sheet 10 of 23
`
`US 7,681,124 B2
`
`212
`
`Retrieve reference to the first
`entry in the dictionary beginning
`with the search string and
`allocate it to the first entry in the
`Search Span
`
`Retrieve reference to the last
`entry in the dictionary beginning
`with the search string and
`allocate it to the last entry in the
`Search Span
`
`Search SpanS
`predetermined maximum
`number?
`
`
`
`Yes
`
`
`
`
`
`
`
`FIG. 9
`
`Search through the range of
`entries defined by the Search
`Span for the most common
`entries based on the weight
`value assigned to each entry
`
`Retrieve all entries within
`Search Span from the
`dictionary and compile into a list
`of completion candidates
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 15 of 49
`
`Set the tist of completion
`candidates to the dictionary
`entries retrieved having the
`highest weight values, up to the
`predetermined maximum
`number
`
`

`

`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 16 of 49
`
`Ca
`rall pump we
`
`
`
`f
`t
`
`t
`
`Y
`
`A.
`s------
`ry as ala CE
`
`FIG. 10
`
`

`

`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 17 of 49
`20 Page 17 of 49
`
`US 7,681,124 B2
`
`
`
`-59“
`@flmmm
`
`
`
`

`

`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 18 of 49
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 18 of 49
`
`US. Patent
`
`Mar. 16, 2010
`
`Sheet 13 of 23
`
`US 7,681,124 B2
`
`28
`
`

`

`U.S. Patent
`
`Mar. 16, 2010
`
`Sheet 14 of 23
`
`US 7,681,124 B2
`
`30
`
`Candidate 1
`Candidate 2
`
`Candidate N
`
`FIG. 14
`
`30
`
`r
`
`Candidate 2
`
`Candidate 5
`
`Candidate 1
`
`Candidate 3
`
`Candidate 4
`
`Candidate 9
`Candidate 5
`Candidate 8
`
`Candidate 2
`Candidate
`Candidate 4
`
`Candidate 6
`Candidate 3
`Candidate 7
`
`FIG. 15
`30
`
`FG 16
`r
`
`
`
`
`
`
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 19 of 49
`
`Candidate 5
`
`Candidate 2
`Candidate 1
`Candidate 4
`
`Candidate 3
`
`Candidate 2
`
`Candidate 1
`idate 4
`Candidate
`
`Candidate 3
`
`F.G. 17
`
`Candidate N
`
`FIG. 18
`
`

`

`U.S. Patent
`
`Mar. 16, 2010
`
`Sheet 15 Of 23
`
`US 7,681,124 B2
`
`
`
`Search String
`Box 40
`
`Display Area
`for interactive
`Search tist
`30
`
`Toolbar 42
`
`F.G. 19
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 20 of 49
`
`

`

`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 21 of 49
`
`U.S. Patent
`
`Mar. 16, 2010
`
`Sheet 16 0123
`
`US 7,681,124 B2
`
`
`
` “anal."“Infill!
`
`
`
`'2E,Eama
`
`I“
`
`FIG. 20
`
`

`

`U.S. Patent
`
`Mar. 16, 2010
`
`Sheet 17 Of 23
`
`US 7,681,124 B2
`
`Initialization of
`Data Entry
`Systern
`
`Display Initialized
`User interface
`
`Wait for user input
`
`
`
`
`
`user inputa
`Character?
`
`202
`
`204
`No
`
`206
`
`208
`
`No
`
`
`
`
`
`
`
`Yes
`
`Add character to
`search string
`
`Terminate
`data en
`try?
`
`
`
`
`
`
`
`
`
`
`
`Carry Out
`Operation
`Corresponding to
`user input
`(eg. terminate
`Current search)
`
`200
`
`Yes
`
`220
`
`Yes
`
`Obtain
`Candidate
`List
`
`Display indicator
`of Candidate List
`Size. Display
`indicator of total
`potential
`Candidates
`asSociated with
`the search string.
`
`Has pointing
`device been released
`thin time limit 12
`
`
`
`
`
`
`
`213A
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 22 of 49
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Mar. 16, 2010
`
`Sheet 18 of 23
`
`US 7,681,124 B2
`
`Display interactive
`Search List
`
`
`
`Wait for user input
`
`inputagesture
`
`
`
`
`
`
`
`ift
`
`262
`
`Clear Candidate
`list but preserve
`Search String.
`
`228
`
`
`
`
`
`Sgesture
`sy a
`
`248
`
`
`
`
`
`
`
`Yes
`Select the
`Candidate.
`Start timer T2
`
`230
`
`Deselect currently
`Selected
`candidate in list.
`Disable time? T2.
`
`232
`
`No
`
`234
`
`Yes
`Stop Timer T2.
`Set search string
`a selected
`candidate
`
`246
`
`
`
`
`
`
`
`
`
`
`
`is gesture
`asSociated with a
`dead zone?
`
`No
`
`
`
`Change
`candidate
`Selection fromist.
`Restart timer T2
`
`ls
`gesture associated
`With a different
`andidate
`
`254
`
`Display interactive
`Search
`List containing
`Candidate List.
`Display indicator
`of total potential
`candidates for the
`search string.
`
`Wait for user input
`
`No
`
`
`
`
`
`input a gesture
`or if
`
`258
`
`gesture
`
`260
`
`Yes
`
`asSociated with a
`candidate?
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 23 of 49
`
`252
`inputagesture
`
`Gesture
`
`242
`
`FIG.22
`
`ift
`
`240
`
`Accept candidate
`and add to text.
`Clear Search
`string.
`
`
`
`

`

`U.S. Patent
`
`Mar. 16, 2010
`
`Sheet 19 of 23
`
`US 7,681,124 B2
`
`202
`
`204
`No
`
`206
`
`208
`
`No
`
`200
`
`Termi
`eminate
`data entry?
`
`Yes
`
`
`
`
`
`Carry out
`operation
`Corresponding to
`user input
`(e.g. terminate
`current search)
`
`220
`
`Yes
`
`initialization of
`Data Entry
`System
`
`Display initialized
`User Interface
`
`Wait for user input
`
`
`
`
`
`
`
`user inputa
`character?
`
`Yes
`
`Add character to
`search string
`
`Obtain
`Candidate
`List
`
`Display interactive
`Search List
`
`
`
`Yes
`
`
`
`213B
`
`
`
`
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 24 of 49
`
`
`
`
`
`Has pointing
`device been released
`ithin time limit L12
`
`Disable
`Keyboard and
`display search
`mode indicator.
`
`
`
`FIG. 23
`
`

`

`U.S. Patent
`
`Mar. 16, 2010
`
`Sheet 20 of 23
`
`US 7,681,124 B2
`
`
`
`t Wait for user input
`
`input a gesture
`or if
`
`associated with a
`candidate?
`
`
`
`Select the
`candidate.
`Start timer 2
`
`228
`
`230
`
`262
`
`Clear Candidate
`list but preserve
`search string.
`
`248
`
`
`
`246
`
`Deselect currently
`selected
`candidate in list.
`Disable timer T2.
`
`
`
`
`
`is gesture
`associated with a
`dead ZOne?
`
`
`
`
`
`
`
`
`
`Display interactive
`Search
`list containing
`Candidate list.
`
`
`
`Wait for user input
`
`
`
`
`
`inputagesture
`
`No
`
`232
`
`No->Monitor user input
`
`
`
`Change
`candidate
`Selection fromist.
`Restart timer T2
`
`N
`O
`
`
`
`gesture
`
`258
`
`Yes
`
`234
`
`Stop Timer T2.
`Set search string
`= selected
`candidate
`
`Obtain
`Candidate
`list
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 25 of 49
`
`FIG. 24
`
`Accept candidate
`and add to text.
`Clear search
`string. Display
`keyboard mode
`indicator.
`
`

`

`U.S. Patent
`
`Mar. 16, 2010
`
`Sheet 21 of 23
`
`US 7,681,124 B2
`
`202
`
`204
`No
`
`
`
`206
`
`208
`
`200
`
`Terminate
`data entry?
`
`Carry out
`operation
`corresponding to
`user input
`(eg. terminate
`current search)
`
`220
`
`Yes
`
`Initialization of
`Data Entry
`System
`
`(e)
`
`Display initialized
`User Interface
`
`Wait for user input
`
`
`
`
`
`Yes
`
`Add character to
`Search string
`
`Obtain
`Candidate
`List from
`multiple
`dictionaries
`
`Display indicator
`of Candidate List
`
`YeS
`
`
`
`
`
`213C
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 26 of 49
`
`
`
`
`
`device been
`eleased within time
`limit 12
`
`
`
`FIG. 25
`
`

`

`U.S. Patent
`
`Mar. 16, 2010
`
`Sheet 22 of 23
`
`US 7,681,124 B2
`
`
`
`
`
`l Wait for user input
`
`input a gesture
`
`262
`
`Clear Candidate
`list but preserve
`search string.
`
`254
`
`lift
`
`Display interactive
`Search
`list containing
`Candidate list.
`
`
`
`
`
`Wait for user input
`
`is user
`input a gesture
`Of
`
`No
`
`258
`
`gesture
`
`
`
`asSociated with a
`candidate?
`
`260
`
`Yes
`
`228
`
`Sgesture
`associated sh a
`candidate
`
`
`
`Yes
`
`Select the
`Candidate.
`Start timer 2
`
`
`
`
`
`
`
`
`
`
`
`
`
`No
`
`
`
`Yes
`Stop Timer T2.
`Set search string
`F Selected
`candidate
`
`
`
`
`
`248
`
`
`
`246
`
`Deselect currently
`Selected
`candidate in list
`Disable timer 2.
`
`
`
`
`
`is gesture
`associated with a
`dead zone?
`
`Change
`candidate
`selection from list.
`Restart timer T2
`
`No
`
`gesture associated
`with a different
`
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 27 of 49
`
`FIG. 26
`
`
`
`
`
`Accept candidate
`and add to text.
`Clear Search
`string.
`
`

`

`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 28 of 49
`Case 5:20-cv-01756-NC Document 1-2 Filed 03/11/20 Page 28 of 49
`
`US. Patent
`
`Mar. 16, 2010
`
`Sheet 23 of 23
`
`US 7,681,124 B2
`
`
`
`

`

`1.
`DATA ENTRY FOR PERSONAL COMPUTING
`DEVICES
`
`US 7,681,124 B2
`
`CROSS-REFERENCES TO REL