`Case 5:20-cv-01756—NC Document 1-3 Filed 03/11/20 Page 1 of 48
`
`
`
`
`
`
`
`
`EXHIBIT 3
`
`EXHIBIT 3
`
`
`
`(12) United States Patent
`Gunn et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,716,579 B2
`May 11, 2010
`
`USOO7716579B2
`
`(54) DATA ENTRY FOR PERSONAL COMPUTING
`DEVICES
`
`(75) Inventors: Harold David Gunn, Vancouver (CA);
`John Chapman, Vancouver (CA)
`(73) Assignee: 602531 British Columbia Ltd. (CA)
`
`(*) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 45 days.
`(21) Appl. No.: 11/133,770
`(22) Filed:
`May 19, 2005
`O
`O
`Prior Publication Data
`US 2005/02104O2A1
`Sep. 22, 2005
`
`(65)
`
`7/1980 Montgomery ............... 400,486
`4,211,497 A
`4,330,845. A * 5/1982 Damerau ..........
`... 715,507
`4,396,992 A
`8/1983 Hayashi et al. ...
`... 364,900
`3. A ck g
`Rising al - - - - - - - - - - - 71.5/533
`I-1-1
`CKSO Ca
`4,559,598. A 12/1985 Goldwasser et al. ........ 364/419
`4,648,044 A * 3/1987 Hardy et al. .................. TO6/60
`(Continued)
`FOREIGN PATENT DOCUMENTS
`O352377 A1
`1, 1990
`(Continued)
`OTHER PUBLICATIONS
`Softype Version 3.1 Screendumps generated from demo version of
`software, downloaded from <"http://onn.com/bmanes/st31dmz.
`exe'>, 21 Figures.*
`
`EP
`
`Continued
`(Continued)
`Related U.S. Application Data
`(60) Division of application No. 09/631,101, filed on Aug. CEC. E.Ackwell
`1, 2000, now Pat. No. 7.293,231, which is a continua
`tion of application No. PCT/CA00/00285, filed on
`(74) Attorney, Agent, or Firm Townsend and Townsend and
`Mar. 15, 2000, which is a continuation-in-part of appli-
`Crew LLP
`RN 09/272,700, filed on Mar. 18, 1999, now
`(57)
`
`ABSTRACT
`
`51) Int. Cl
`2006.O1
`(51) .oF i 7/00
`(
`.01)
`(52) U.S. Cl. ....................... 715.256; 5261.75268;
`715/760; 34.5/179; 382/187
`(58) Field of Classification Search ................. 715/.530,
`715/531,534,541, 255, 256, 261, 268; 365/189.01;
`345/156, 173, 179; 382/187
`See application file for complete search history.
`References Cited
`
`(56)
`
`U.S. PATENT DOCUMENTS
`3,644,898 A
`2, 1972 Post ........................... 715,534
`
`A method, system and computer-readable media for Support
`ing text entry on a personal computing device by activating
`automated searching to search for completion candidates
`which are based on a partial text entry received from a user.
`The completion candidates are displayed in a search list. The
`user may select a completion candidate from among the
`completion candidates in the search list to correspondingly
`modify the partial text entry, or the user may decline all of the
`completion candidates displayed in the search list and termi
`nate the automated searching. The system may further pro
`vide a digital keyboard for use in entering text.
`
`29 Claims, 23 Drawing Sheets
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 2 of 48
`
`EXAMPLEfs:0
`
`Searchsting'e, N- strings'.
`
`S. x
`
`se
`wifend
`
`sa Search sting=
`8.
`
`3S
`
`2:fuse
`88s ward
`er
`raise
`s
`atthewsarcase
`efered sists
`$3.8maine ash
`tlepreviouscandidas
`skattaabwe.
`
`
`
`igital
`
`s
`
`24
`
`salary
`khassacas:
`anime bargomaying
`parasaanyor
`as 8th searchom he
`ext
`Ea
`restasicallestof
`revett has search 8sig of
`ahreation
`carryot
`
`gia
`Keyboard
`
`Searcastings
`
`
`
`US 7,716,579 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`4,689,768 A * 8/1987 Heard et al. ................ 71.5/533
`4,730,252 A
`3, 1988 Bradshaw ...
`364,403
`4,744,050 A
`5, 1988 Hirosawa et al.
`... 364,900
`4,774,666 A
`9/1988 Miyao et al. ......... ... 364/419
`RE32,773 E
`10/1988 Goldwasser et al. ........ 364/419
`4,782.464 A 1 1/1988 Gray et al.
`4,783,761 A 1 1/1988 Gray et al. .................. 71.5/533
`4,786,765 A * 1 1/1988 Yamanami et al. ....... 178/1906
`4,807,181 A
`2, 1989 Duncan, IV et al. ........ 364,900
`4,847,766 A * 7/1989 McRae et al. .....
`... 15,532
`4,891,786 A
`1/1990 Goldwasser ...
`364,900
`4.969,097 A 11, 1990 Levin .........
`364,419
`4,980,855 A * 12/1990 Kojima .......
`71.5/533
`5,040,113 A
`8, 1991 Mickunas ................... 364/419
`5,060,154. A 10/1991 Duncan, IV
`5,067,165 A 1 1/1991 Nishida
`5,096,423. A
`3, 1992 Goldwasser ................ 434/118
`5,203,704 A
`4, 1993 McCloud ....
`434,156
`5,220,649 A
`6, 1993 Forcier ....
`715,541
`5,220,652 A
`6/1993 Rowley ...................... 395/275
`5,258,748. A 1 1/1993 Jones
`5,261,112 A * 1 1/1993 Futatsugi et al. ............ 71.5/533
`5,297,041 A
`3/1994 Kushler et al. ......... 364,419.15
`5,305,205 A
`4, 1994 Weber et al.
`5,327,161 A
`7/1994 Logan et al. ................ 345,157
`5,329,609 A
`7, 1994 Sanada et al.
`5,347,295 A
`9/1994 Agulnicket al. ............ 345,156
`5,392.447 A * 2/1995 Schlacket al. .
`... 715,863
`5.487,616 A
`1/1996 Ichbiah .......
`400,489
`5,500,935 A
`3, 1996 Moran et al. ...
`... 395,156
`5,559,942 A * 9/1996 Gough et al. ..
`... 715,802
`5,574.482 A 11/1996 Niemeier ...,
`, 345,173
`5,594,640 A *
`1/1997 Capps et al.
`715,532
`5,596,699 A
`1/1997 Driskell ......
`395,352
`5,606,674. A * 2/1997 Root .......
`715,769
`5,621,641 A
`4/1997 Freeman ..
`395,796
`5,623.406 A
`4, 1997 Ichbiah ....................... 395/753
`5,649,223. A
`7/1997 Freeman ..................... 395/796
`5,657,397 A * 8/1997 Bokser ....................... 382,225
`5,666,139 A
`9/1997 Thielens et al.
`5,689,667 A 11/1997 Kurtenbach ................. 395,352
`5,704,029 A 12/1997 Wright, Jr. .................. 715,505
`5,724.457 A
`3, 1998 Fukushima
`5,734,749 A
`3, 1998 Yamada et al.
`5,745,116 A
`4, 1998 Pisutha-Arnond ........... 345,358
`5,790,115 A
`8/1998 Pleyer et al.
`5,805,158. A
`9, 1998 Bertram et al.
`5,805,159 A
`9, 1998 Bertram et al.
`5,805,167 A
`9/1998 Van Cruyningen .......... 345.353
`5,805,911 A
`9, 1998 Miller ........................ 395/796
`5,812,697 A
`9, 1998 Sakai et al.
`5,818,437 A 10/1998 Grover et al.
`5,821,512 A 10/1998 O'Hagan et al. ............ 235,383
`5,835,635 A 11, 1998 Nozaki et al. ............... 382,226
`5,838,302 A 1 1/1998 Kuriyama et al.
`5,845,300 A * 12/1998 Comer et al. ............... 715,508
`5,864,340 A
`1/1999 Bertram et al.
`5,881,169 A
`3/1999 Henry, Jr.
`5,896,321. A
`4, 1999 Miller et al. ........... 365,189.01
`5,911,485. A
`6/1999 Rossman
`5,914,708 A
`6/1999 LaGrange et al. ........... 345,179
`5,926,178 A
`7/1999 Kurtenbach ................. 34.5/352
`5,943,039 A
`8, 1999 Anderson et al. ........... 345/146
`5,953,541 A
`9/1999 King et al.
`5,959,629 A
`9/1999 Masui ........................ 345,347
`5,963,666 A * 10/1999 Fujisaki et al. .............. 382/187
`5,963,671. A 10/1999 Comerford et al. .......... 382,230
`5.974.558 A * 10/1999 Cortopassi et al. .......... T13,323
`5,977,887. A 1 1/1999 Grimmett
`5,977,948 A * 1 1/1999 Nishibori .................... T15,841
`5,982,351 A * 1 1/1999 White et al. ................ 715,810
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 3 of 48
`
`345,179
`
`6,002,390 A 12/1999 Masui ........................ 345,173
`6,005,549 A 12/1999 Forest
`345,157
`6,008,799 A 12/1999 Van Kleeck ................. 345,173
`6,011,554 A
`1/2000 King et al.
`6,026,233 A
`2/2000 Shulman et al. ............. 717,113
`6,037.942 A * 3/2000 Millington .................. 715.835
`6,084,576 A
`7/2000 Leu et al. .................... 345,168
`6,088,649 A
`7/2000 Kadaba et al.
`6,094,197 A
`7/2000 Buxton et al. ............... 345,863
`6,097.392 A * 8/2000 Leyerle ...................... T15,863
`6,097.841 A
`8/2000 Gunji et al.
`6,098,086 A
`8/2000 Krueger et al.
`6, 101.461 A
`8/2000 Ukigawa et al.
`6,111,985 A
`8, 2000 Hullender et al.
`6,144,378 A 11/2000 Lee
`6,154,758 A 1 1/2000 Chiang
`6,167.411 A * 12/2000 Narayanaswamy .......... 71.5/541
`6,167.412 A * 12/2000 Simons ...
`708/105
`6,188,789 B1
`2/2001 Marianetti,
`382, 189
`6.256,030 B1
`7/2001 Berry et al.
`6,262,719 B1
`7/2001 Bi et al. ....
`6,275,612 B1
`8, 2001 Imoto
`6,282,315 B1* 8/2001 Boyer ......................... 382,177
`6,286,064 B1
`9/2001 King et al.
`6,292, 179 B1
`9/2001 Lee ............................ 345,173
`6,295,372 B1
`9/2001 Hawkins etal
`382,187
`6,307.549 B1
`10/2001 King et al. ...
`345,352
`6,369,807 B1
`4/2002 Nakashima ....
`... 345,179
`6,377,965 B1
`4/2002 Hachamovitch et al. .... 715,534
`6,405,060 B1* 6/2002 Schroeder et al. ........... 455/566
`6,442,295 B2 * 8/2002 Navoni et al. ............... 382,229
`6,473,006 B1
`10/2002 Yu et al.
`6,487,424 B1
`1 1/2002 Kraft et al.
`6,502,114 B1 12/2002 Forcier
`6,539,421 B1
`3/2003 Appelman et al. .......... TO9,206
`6,621.939 B1* 9/2003 Negishi et al. ..
`382,291
`6,654,733 B1
`1 1/2003 Goodman et al.
`... 706.52
`6,661,920 B1
`12/2003 Skinner ......
`382,187
`6,751,603 B1
`6/2004 Bauer et al. .................... 707/1
`6,801,190 B1
`10/2004 Robinson et al.
`6,888, 141 B2
`5/2005 Carr ........................ 250/338.1
`6,934,906 B1* 8/2005 Cheok .
`... 715,500.1
`6,970,513 B1 * 1 1/2005 Puri et al. .............. 375,240.25
`6,978.421 B1
`12/2005 Aida
`... 704/9
`7,003,446 B2 * 2/2006 Trower et al.
`7,224,409 B2 * 5/2007 Chin et al. .................. 348,732
`7,293,231 B1
`1 1/2007 Gunn et al.
`7,322,023 B2
`1/2008 Shulman et al.
`2001/0000962 A1* 5/2001 Rajan ......................... 345,302
`2001/0027468 A1* 10, 2001 Okura ..
`709/202
`2002/0067377 A1
`6/2002 McGovern .................. 345,816
`2002fO087279 A1
`7, 2002 Hall
`2003.0137605 A1
`7, 2003 Chin et al.
`2005, 0198144 A1* 9, 2005 Kraenzel et al. ............ TO9,206
`2005/0210020 A1
`9, 2005 Gunn et al.
`2005/0223308 A1 10, 2005 Gunn et al.
`2007/0188472 A1
`8/2007 Ghassabian
`
`
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`JP
`JP
`JP
`
`O 643 357 A2
`O 643 357 A3
`O 844 570 A2
`O 844 571 A2
`O 858 023 A2
`0472444 B1
`O 858 O23 A3
`O 844 570 A3
`O 844 571 A3
`O982676 A1
`101.05324. A
`10154033. A
`10154144. A
`
`3, 1995
`3, 1996
`5, 1998
`5, 1998
`8, 1998
`8/1998
`9, 1998
`8, 1999
`8, 1999
`3f2000
`4, 1998
`6/1998
`6/1998
`
`
`
`US 7,716,579 B2
`Page 3
`
`5, 1999
`11143614
`JP
`6, 1999
`11167569 A
`JP
`4/2002
`10333818
`JP
`3, 1996
`WO96,09579
`WO
`3, 1998
`WO98, 11480
`WO
`6, 1999
`WO99,28811
`WO
`WOO1,95095 A2 12/2001
`WO
`WO WO 2004/107101 A2 12/2004
`
`OTHER PUBLICATIONS
`Softype 3.1 Help File, circa 1997, downloaded from <"http://onn.
`com/blnanes/st31dmz.exe">, 38 pages.*
`Toshiyuki Masui, Sony Computer Science Laboratory Inc.,
`Shinagawa, Tokyo, "An Efficient Text Input Method for Pen-based
`Computers'. Proceedings of the ACM Conference on Human Factors
`in Computing Systems, Apr. 1998, pp. 328-335.
`Toshiyuki Masui, Sony Computer Science Laboratories, Inc.,
`Shinagawa, Tokyo, “Integrating Pen Operations for Composition by
`Example”.
`Don Hopkins, “The Design and Implementation of Pie Menus'. Dr.
`Dobb's Journal, Dec. 1991 (http://art.ned-hopkins/Don/piemenus/
`ddi/piemenus.hml).
`Jason I. Hong, “Java Pie Menus'. Sep. 4, 1999 (ttp://www.cs.
`berkeley.edu/~asonht download/sotware/piemenu/).
`Don Hopkins, “A Description of Pie Menus' (http://catalog.com/
`hopkins/piemenus/PieMenuDescription.html).
`Don Hopkins, “Natural Selection: The Evolution of Pie Menus',
`BayCHI, Oct. 13, 1998 (http://catalog.com/hopkins/piementis/Natu
`ral Selection.html).
`Russell Nelson, “Pie Menu Window Manager'. Jun. 3, 1998 (http://
`www.crynwr.com/piewm/).
`Don Hopkins, “ActiveX Pie Menus' (http://catalog.com/hopkins/
`piemenus/ActiveXPieMenus.html).
`Don Hopkins, “Pie Menu References” (http://catalog.com/hopkins/
`piemenu-references.html).
`Don Hopkins, “Pie Menus' (http://art.net/~hopkins/Don/piemenus/
`index.html).
`Tom Nantais, et al., IEEE Transactions on Rehabilitation Engineer
`ing, "A Predictive Selection Technique for Single-Digit Typing With
`a Visual Keybaord’. No. 3 Sep. 1994.
`“GtkPieMenu' 1999 orfelyus (2 pages).
`Operation Guide: 1997: 2 pages; Mercedes-Benz of North America,
`Inc.; Montvale, New Jersey, United States.
`Auto Pilot System—Operation Guide; 1998; 2 pages; Mercedes
`Benz AG; Germany.
`COMAND Betriebsanleitung- Operation Guide; 1998; 3 pages;
`Mercedes-Benz AG; Germany.
`3Com. “PalmPilot TMHandbook” 1997, 3Com Corporation, Table of
`Contents and pp. 37-39.
`Advisory Action dated Sep. 16, 2005, issued in related U.S. Appl. No.
`09/631, 101, filed Aug. 1, 2000.
`Baeza-Yates, R.A. et al. “A New Approach to Text Searching (Pre
`liminary Version).” Oct. 1992, Communications of the ACM, vol. 35.
`No. 10, pp. 74-82.
`Bellman, T. et al. “A Probabilistic Character Layout Strategy for
`Mobile Text Entry.” Proceedings of Graphics Interface '98, 1998,
`Canadian Information Processing Society: Toronto, pp. 168-176.
`Bohan, M. et al. “A Psychophysical Comparison of Two Stylus
`Driven Soft Keyboards.” 1998, Department of Psychology, Wichita
`State University: Wichita, Kansas, six pages.
`Callan, J. etal. "An Empirical Comparison of Pie vs. Linear Menus.”
`May 15-19, 1988, Proceedings of the CHI '88 Conference on Human
`Factors in Computing Systems, Washington, D.C., pp. 95-100,
`Abstract.
`Darragh, J.J. et al. “The Reactive Keyboard: A Predictive Typing
`Aid.” Nov. 1990, IEEE Computer, vol. 23, No. 11 pp. 41-49.
`European Examination Report dated Feb. 7, 2002, issued in related
`European Application No. 00910 460.5.
`Euroepan Examination Report dated Nov. 8, 2004, issued in related
`European Application No. 01981991.1.
`European Examination Report dated Jun. 23, 2005, issued in related
`European Application No. 01981991.1.
`
`Garay-Vitoria, N. et al. "A Comparison of Prediction Techniques to
`Enhance the Communication Rate.” 2004, U14AII, LNCS, Springer
`Verlag, vol. 3196, pp. 400-417.
`International Preliminary Examination Report dated Jun. 19, 2001,
`issued in related PCT Application No. PCT/CA00/00285 filed Mar.
`15, 2000.
`International Preliminary Examination Report dated Sep. 12, 2003,
`issued in related PCT Application No. PCT/CA01/01473 filed Oct.
`18, 2001.
`Jones, P.E. "Virtual Keyboard with Scanning and Augmented by
`Prediction.” 1998, Proc. 2nd Euro. Conf. Disability Virtual Reality &
`Assoc. Tech. Skovde, Sweden, pp. 45-51.
`MacKenzie, I.S. et al. “Alphanumeric Entry on Pen-Based Comput
`ers.” 1994. International Journal of Human-Computer Studies vol.
`41, pp. 775-792, located at <http://www.yorku.ca/mack/IJHCS.
`html>, last visited on Nov.30, 2006, eighteen pages.
`MacKenzie, I.S. et al. “Text Entry Using Soft Keyboard.” 1999,
`Behaviour & Information Technology, vol. 18, pp. 235-244, located
`at <http://www.yorku.ca/mack/BIT3.html>, last visited on Nov.30,
`2006, fifteen pages.
`MacKenzie, I.S. etal. “The Design and Evaluation of a High-Perfor
`mance SoftKeyboard.” May 15-20, 1999, ACM, pp. 25-31.
`Mankoff, J. etal. “Cirrin: A World-Level Unistroke Keyboard for Pen
`Input.” 1998, ACM, pp. 213-214.
`Masui, T. “POBox: An Efficient TextInput Method for Handheld and
`Ubiquitous Computers.” Date Unknown, Proc. of Ist Intl. Symp. On
`Handheld and Ubiquitous Computing, Computer Science, vol. 1707.
`pp. 289-300.
`Notice of Allowability dated Aug. 8, 2007, issued in related U.S.
`Appl. No. 09/631, 101, filed Aug. 1, 2000.
`Office Action dated Jan. 20, 2004, issued in related U.S. Appl. No.
`09/631, 101, filed Aug. 1, 2000.
`Office Action dated Jan. 14, 2005, issued in related U.S. Appl. No.
`09/631, 101, filed Aug. 1, 2000.
`Office Action dated Jun. 14, 2005, issued in related U.S. Appl. No.
`09/631, 101, filed Aug. 1, 2000.
`Office Action dated Mar. 20, 2006, issued in related U.S. Appl. No.
`09/631, 101, filed Aug. 1, 2000.
`Office Action dated Oct. 18, 2006, issued in related U.S. Appl. No.
`10/399,560, filed Apr. 18, 2003.
`Office Action dated Jan. 5, 2007, issued in related U.S. Appl. No.
`09/631, 101, filed Aug. 1, 2000.
`Office Action dated May 18, 2007, issued in related U.S. Appl. No.
`1 1/134,759, filed May 19, 2005.
`Office Action dated Jun. 4, 2007, issued in related U.S. Appl. No.
`09/631, 101, filed Aug. 1, 2000.
`Office Action dated Jul. 17, 2007, issued in related U.S. Appl. No.
`10/399,560, filed Apr. 18, 2003.
`Venolia, D. etal. “T-Cube: A Fast, Self-Disclosing Pen-Based Alpha
`bet.” Apr. 24-28, 1994, Human Factors in Computing Systems, Bos
`ton, MA, pp. 265-270.
`Wu, S. etal “AGREP A Fast Approximate Pattern-Matching Tool.”
`Jan. 1992, Proceedings of USENIX Technical Conference, San Fran
`cisco, CA, pp. 153-162.
`Office Action dated Jan. 26, 2009 for U.S. Appl. No. 11/036,267.
`Office Action dated Jan. 26, 2009 for U.S. Appl. No. 1 1/134,810.
`Office Action dated Feb. 4, 2009 for U.S. Appl. No. 1 1/134,759.
`Softype 3.1 Help File, circa 1997, downloaded from <http://orin.
`combinaries st31dmz.exe), 38 pages.
`Softype Version 3.1 Screendumps generated from demo version of
`software, downloaded from <http://orin.com/binaries/st31dmz.
`exe>, 21 Figures.
`Bohlman, Eric, “KeyCache The Universal Input Accelerator, Ver
`sion 2.1.” copyright Sep. 29, 1994, OMS Development, documenta
`tion, downloaded from <http://www.sac.sk/files.php?d=16&I=K>.
`12 pages.
`NCIP listing of WordPrediction & Writing Tools, circa 1997, down
`loaded from <http://www2.edc.org/incip/LIBRARY/wp/Pubs.htm>.
`7 pages.
`National Council on Disability Document Archive, software guide to
`alternative input and output programs, circa 1996, downloaded from
`<http://www.dimenet.com/disnews/archive.php?mode=P&id=49),
`12 pages.
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 4 of 48
`
`
`
`US 7,716,579 B2
`Page 4
`
`Wivik 2 REP Software manual, circa Feb. 1998, Orentke Romich
`Company,
`downloaded
`from
`<http://web.archive.org/web/
`2000 1204165600/wivik.com/html/downloads.htm>, 128 pages.
`Office Action dated Apr. 15, 2008 issued for U.S. Appl. No.
`10/399,560.
`Office Action dated Apr. 3, 2008 issued for U.S. Appl. No.
`11/134,759.
`Office Action dated Apr. 30, 2008 issued for U.S. Appl. No.
`11/036,267.
`
`Office Action dated Jan. 7, 2009 issued for U.S. Appl. No.
`10/399,560.
`Office Action dated Sep. 29, 2009 issued for U.S. Appl. No.
`10/399,560.
`Office Action dated Nov. 24, 2009 issued for U.S. Appl. No.
`11/036,267.
`Notice of Allowance dated Nov.30, 2009 issued for U.S. Appl. No.
`1 1/134,810.
`* cited by examiner
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 5 of 48
`
`
`
`U.S. Patent
`
`May 11, 2010
`
`Sheet 1 of 23
`
`US 7,716,579 B2
`
`
`
`Display Device
`15
`
`Processing Unit
`12
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 6 of 48
`
`Data entry system
`26
`
`Computer-readable medium
`16
`
`FIG. 1
`
`
`
`U.S. Patent
`
`May 11, 2010
`
`Sheet 2 of 23
`
`US 7,716,579 B2
`
`Data Entry System
`
`26
`
`Digital Keyboard System
`
`Candidate Prediction
`System
`
`Logic for interactive
`Search List
`
`Dictionary Data
`Structure
`
`
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 7 of 48
`
`Completion
`Candidate Field 22
`
`Weight
`Field 24
`
`
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 8 of 48
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 8 of 48
`
`US. Patent
`
`May 11,2010
`
`Sheet 3 of 23
`
`US 7,716,579 B2
`
`
`
`
`
`U.S. Patent
`
`May 11, 2010
`
`Sheet 4 of 23
`
`US 7,716,579 B2
`
`14
`
`10
`
`Digital
`Keyboard
`28
`
`
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 9 of 48
`
`Digital
`Keyboard
`28
`
`interactive
`Search
`List
`30
`
`FIG. 5
`
`
`
`U.S. Patent
`
`May 11, 2010
`
`Sheet 5 of 23
`
`US 7,716,579 B2
`
`Potential completion candidates
`in the dictionary for apartial text
`entry made up of a single
`character
`
`Potential Candidate 1
`Potential Candidate 2
`Potential Candidate 3
`Potential Candidate 4
`Potential Candidate 5
`Potential Candidate 6
`Potential Candidate 7
`Potential Candidate 8
`Potential Candidate 9
`Potential Candidate 10
`Potential Candidate 11
`Potential Candidate 12
`Potential Candidate 13
`Potential Candidate 14
`Potential Candidate 15
`Potential Candidate 16
`Potential Candidate 17
`Potential Candidate 18
`
`Potential Candidate 1
`Potential Candidate 5
`Potential Candidate 2
`Potential Candidate 16
`Potential Candidate 18
`
`Potential completion candidates
`in the dictionary when the
`partial text entry equals
`Potential Candidate 5
`
`Potential Candidate 5
`Potential Candidate 6
`Potential Candidate 9
`Potential Candidate 10
`Potential Candidate 11
`Potential Candidate 12
`
`Potential Candidate 5
`Potential Candidate 10
`Potential Candidate 6
`Potential Candidate 9
`Potential Candidate 12
`
`
`
`
`
`
`
`
`
`partial text entry set to
`Potential Candidate 5
`
`-
`
`)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Portion of
`the
`dictionary
`
`Displayed
`in the
`interactive
`Search list
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 10 of 48
`
`
`
`list of completion candidates
`obtained by the candidate
`prediction system from the
`above 18 potential candidates
`in the dictionary
`
`list of completion candidates
`obtained by the candidate
`prediction system from the
`above 7 potential candidates in
`the dictionary
`
`
`
`FIG. 5A
`
`
`
`U.S. Patent
`
`May 11, 2010
`
`Sheet 6 of 23
`
`US 7,716,579 B2
`
`
`
`gestures toward a
`candidate in the refined
`
`a
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 11 of 48
`
`candidate is selected
`and the user can use
`the refined list in the
`Sae rarer as with
`the previous candidate
`st (at 112 above).
`
`e
`
`24: The usercar
`continue building of modifying
`partial text entry of
`initiate a further search on the
`
`cance the search string of
`revert to last candidate list of
`revert to the last search string of
`carry out another operation
`
`
`
`U.S. Patent
`
`May 11, 2010
`
`Sheet 7 of 23
`
`US 7,716,579 B2
`
`102: User selects the
`key containing'e' on
`the keyboard. The
`selection forms part
`of the partial text
`entry and is added to
`the search string.
`
`110: if the user
`gestures to a dead
`Zone, the Selected
`candidate is de
`Selected and the list
`of candidates
`remains displayed.
`
`18: With a
`candidate Selected, if
`the uSergestures to
`another candidate,
`the Selection
`changes.
`
`104: if the user does not
`release the keyboard
`Selection in less than time
`Emit L1, the Candidate
`Prediction System obtains a
`Candidate list based on the
`user's keyboard selection.
`The Candidate list is
`displayed in an interactive
`Search list.
`
`Monitor pointing
`
`108: if usergestures
`toward a candidate
`(without lifting), that
`candidate is hi-lited
`and becomes the
`Selection.
`
`
`
`
`
`EXAMPLE 100
`
`
`
`106: if pointing
`device is lifted up
`without significant
`movement, the
`interactive search list
`is cleared and the
`keyboard is enabled.
`
`
`
`f10: if the user
`gestures to a dead
`Zone, the selected
`Candidate is de
`Selected and the list
`of candidates
`remains displayed.
`
`16: if the user continues to
`keep a candidate selected for
`more than defined time limit 2,
`arefined list of candidates is
`obtained using the selected
`candidate and displayed in the
`interactive search list.
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 12 of 48
`
`
`
`120: if the user gestures toward
`a candidate in the refined
`candidate list, that candidate is
`Selected and the user can use
`the refined fist in the same
`manner as with the previous
`Candidate list.
`
`114. If the user fifts up
`the pointing device in
`less than time limit 2
`with a candidate
`Selected, the selected
`candidate is added to the
`text in place of the partial
`textentry, the list is
`cleared and the
`keyboard is enabled.
`
`
`
`122: if the pointing device is lifted up
`without significant movement, the
`interactive Search list is cleared, the
`keyboard is enabled, the search is
`terminated and the most recently
`Selected candidate is added to the text
`in place of the partial text entry.
`
`FIG.6A
`
`initiate a further
`search on the partial
`text entry or
`Cance the Search
`string or
`?evert to last
`candidate list or
`revert to the last
`search string or
`carry out another
`operation
`
`
`
`U.S. Patent
`
`May 11, 2010
`
`Sheet 8 of 23
`
`US 7,716,579 B2
`
`200
`
`202
`
`initialization of
`Data Entry
`System
`
`Display initialized
`User interface
`
`-No
`
`
`
`206
`
`
`
`Terminate
`data entry?
`
`Wait for user input
`
`Yes
`Add character to
`Search string
`
`Obtain
`Candidate
`list
`
`Has
`pointing device been
`eleased within time
`limit 12
`
`
`
`
`
`No
`
`Carry out
`operation
`corresponding to
`user input
`(e.g. terminate
`Current search)
`
`220
`
`22
`
`214
`
`218
`
`Disable
`Keyboard
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 13 of 48
`
`
`
`
`
`U.S. Patent
`
`May 11, 2010
`
`Sheet 9 of 23
`
`US 7,716,579 B2
`
`Display interactive
`Search
`List containing
`Candidate list
`
`
`
`262
`
`Clear Candidate
`List but preserve
`search string.
`
`
`
`
`
`254
`
`Display interactive
`Search
`list containing
`Candidate is
`
`
`
`
`
`Deselect currently
`Selected
`candidate in fist
`Disable timer 2.
`
`26
`
`associated with a
`dead Zone
`
`Wait for user input
`
`
`
`candidate.
`Start iner2
`
`
`
`inputagesture
`
`No
`
`
`
`
`
`associated with a
`candidate
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 14 of 48
`
`FG. 8
`
`240
`
`
`
`t
`Accept candidate
`and add to text.
`Clear search
`string.
`
`
`
`U.S. Patent
`
`May 11, 2010
`
`Sheet 10 of 23
`
`US 7,716,579 B2
`
`212
`
`Retrieve reference to the first
`entry in the dictionary beginning
`with the search string and
`allocate it to the first entry in the
`Search Span
`
`Retrieve reference to the last
`entry in the dictionary beginning
`with the search string and
`allocate it to the last entry in the
`Search Span
`
`Search SpanS
`predetermined maximum
`number
`
`
`
`
`
`
`
`FIG. 9
`
`Search through the range of
`g
`d by
`Search
`pan for the most common
`entries based on the weight
`value assigned to each entry
`
`
`
`
`
`Retrieve all entries within
`Search Span from the
`dictionary and compile into a list
`of completion candidates
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 15 of 48
`
`Set the list of completion
`candidates to the dictionary
`entries retrieved having the
`highest weight values, up to the
`predetermined maximum
`number
`
`
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 16 of 48
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 16 of 48
`
`US. Patent
`
`May 11,2010
`
`Sheet 11 of 23
`
`US 7,716,579 B2
`
`28A
`
`
`
`
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 17 of 48
`
` I
`
`
`
`
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 18 of 48
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 18 of 48
`
`US. Patent
`
`May 11,2010
`
`Sheet 13 of 23
`
`US 7,716,579 B2
`
`28
`
`
`
`U.S. Patent
`
`May 11, 2010
`
`Sheet 14 of 23
`
`US 7,716,579 B2
`
`30
`
`Candidate 1
`Candidate 2
`
`Candidate N
`
`F.G. 14
`
`Candidate
`
`Candidate 3
`
`Candidate 4
`
`Candidate 9
`Candidate 5
`Candidate 8
`
`Candidate 2
`Candidate 1
`Candidate 4
`
`Candidate 6
`Candidate 3
`Candidate 7
`
`FIG. 15
`30
`
`
`
`FIG 16
`r
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 19 of 48
`
`Candidate 5
`
`Candidate 2
`Candidate
`Candidate 4
`
`Candidate 3
`
`Candidate 2
`
`Candidate
`date 4
`Candidate
`
`Candidate 3
`
`FIG. 17
`
`Candidate N
`
`FIG. 18
`
`
`
`U.S. Patent
`
`May 11, 2010
`
`Sheet 15 of 23
`
`US 7,716,579 B2
`
`
`
`10
`
`Display Area
`for interactive
`Search List
`30
`
`Toolbar 42
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 20 of 48
`
`F.G. 19
`
`
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 21 of 48
`
`‘voavmum7?vam
`Ifl°4fla.
`
`FIG. 20
`
`V
`
`ana”
`
`“anallal
`
`Hamill!"
`
`M
`
`
`
`
`
`U.S. Patent
`
`May 11, 2010
`
`Sheet 17 Of 23
`
`US 7,716,579 B2
`
`Begin
`
`initialization of
`Data Entry
`System
`
`(e)
`
`Display initialized
`User Interface
`
`202
`
`204
`No
`
`
`
`2O6
`
`200
`
`Terminate
`data entry?
`
`Yes
`
`End
`
`208
`
`No
`
`
`
`
`
`
`
`Carry out
`operation
`Corresponding to
`user input
`(eg. terminate
`current search)
`
`220
`
`Yes
`
`Wait for user input
`
`user inpufa
`character
`
`
`
`
`
`Yes
`
`Add character to
`
`Obtain
`Candidate
`list
`
`Display thdicator
`of Candidate list
`Size. Display
`indicator of total
`potential
`candidates
`associated with
`the search string.
`
`Has pointing
`device been released
`ithin time init 12
`
`
`
`
`
`
`
`
`
`
`
`213A
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 22 of 48
`
`
`
`
`
`
`
`
`
`
`
`U.S. Patent
`
`May 11, 2010
`
`Sheet 18 of 23
`
`US 7,716,579 B2
`
`Display Interactive
`Search List
`
`
`
`Wait for user input
`
`262
`
`
`
`
`
`ift
`
`254
`Display interactive
`Search
`list containing
`Candidate list.
`Display indicator
`of total potential
`candidates for the
`search string.
`
`
`
`248
`
`Deselect currently
`Selected
`candidate in list
`Disable timer T2.
`
`Wait for user input
`
`candidate.
`Start timer T2
`
`
`
`232
`
`
`
`234
`
`Yes
`Stop Timer T2.
`Set Search string
`= Selected
`candidate
`
`Candidate
`List
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 23 of 48
`
`FIG. 22
`
`Accept candidate
`and add to text.
`Clear search
`string.
`
`
`
`
`
`U.S. Patent
`
`May 11, 2010
`
`Sheet 19 of 23
`
`US 7,716,579 B2
`
`Begin
`
`initialization of
`Data Entry
`System
`
`Display initialized
`User interface
`
`Wait for user input
`
`
`
`user inpufa
`Character?
`
`
`
`
`
`Yes
`
`Add character to
`search string
`
`200
`
`202
`
`204
`NO
`
`
`
`206
`
`Terminate
`data entry
`try?
`
`208
`
`No
`
`
`
`Carry out
`Operation
`Corresponding to
`user input
`(eg. terminate
`current search)
`
`s
`
`220
`
`Y
`
`eS
`
`Yes
`
`213B
`
`
`
`Display interactive
`Search List
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 24 of 48
`
`
`
`
`
`Has pointing
`device been released
`ithin time limit L12
`
`
`
`Keyboard and
`display search
`mode indicator.
`
`FIG. 23
`
`
`
`U.S. Patent
`
`May 11, 2010
`
`Sheet 20 of 23
`
`US 7,716,579 B2
`
`
`
`Display interactive
`Search
`list containing
`Candidate tist.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`262
`
`Clear Candidate
`list but preserve
`search string.
`
`246
`
`Deselect currently
`selected
`candidate in list.
`Disable tinner T2.
`
`Yes
`
`Change
`candidate
`Selection fromist
`Restart timer T2
`
`No
`
`gesture
`
`associated with a
`candidate
`
`260
`
`
`
`
`
`Stop Timer T2.
`Set search sting
`
`Obtain
`Candidate
`List
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 25 of 48
`
`FIG. 24
`
`and add to text.
`Clear search
`string. Display
`keyboard mode
`indicator.
`
`
`
`U.S. Patent
`
`May 11, 2010
`
`Sheet 21 of 23
`
`US 7,716,579 B2
`
`Initialization of
`Data Entry
`System
`
`(e)
`
`Display initialized
`User interface
`
`202
`
`204
`NO
`
`
`
`206
`
`200
`
`Terminate
`data entry?
`
`Y
`eS
`
`Wait for user input
`
`
`
`
`
`
`
`
`
`208
`
`No
`
`Carry out
`operation
`Corresponding to
`user input
`(eg. terminate
`current search)
`
`220
`
`Yes
`
`user inpufa
`character?
`
`Yes
`
`Add character to
`search string
`
`
`
`
`
`
`
`Obtain
`Candidate
`tist from
`multiple
`dictionaries
`
`Display indicator
`of Candidate List
`
`Yes
`
`
`
`
`
`213C
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 26 of 48
`
`
`
`
`
`device been
`eleased within time
`limit 12
`
`
`
`FIG. 25
`
`
`
`U.S. Patent
`
`May 11, 2010
`
`Sheet 22 of 23
`
`US 7,716,579 B2
`
`262
`
`Clear Candidate
`List but preserve
`Search string.
`
`254
`
`f
`
`Display interactive
`Search
`list containing
`Candidate list.
`
`
`
`
`
`Deselect currently
`Selected
`candidate in list.
`Disable timer T2.
`
`Select the
`candidate.
`Start timer 2
`
`
`
`246
`
`
`
`
`
`is gesture
`associated with a
`dead ZOne?
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 27 of 48
`
`
`
`Accept candidate
`and add to text.
`Clear Search
`string.
`
`
`
`
`
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 28 of 48
`Case 5:20-cv-01756-NC Document 1-3 Filed 03/11/20 Page 28 of 48
`
`US. Patent
`
`May 11,2010
`
`Sheet 23 of 23
`
`US 7,716,579 B2
`
`
`
`
`
`1.
`DATA ENTRY FOR PERSONAL COMPUTING
`DEVICES
`
`US 7,716,579 B2
`
`CROSS-REFERENCES TO RELATED
`APPLICATIONS
`
`This application is a Divisional of application Ser. No.
`09/631,101, filed Aug. 1, 2000 now U.S. Pat. No. 7.293,231;
`which is a Continuation of PCT application PCT/CAO0/
`00285, filed Mar. 15, 2000; which is a Continuation-In-Part of
`application Ser. No. 09/272,700, filed Mar. 18, 1999 now
`abandoned.
`
`FIELD OF THE INVENTION
`
`The present invention relates generally to computer-as
`sisted data entry and more particularly to a method, system,
`and apparatus for computer-assisted text generation and entry
`using a pointing device with a personal computing device,
`and to computer-readable media having executable instruc
`tions for Supporting text generation and entry using a pointing
`device.
`
`BACKGROUND OF THE INVENTION
`
`10
`
`15
`
`35
`
`40
`
`25
`The wide-spread adoption of miniaturized personal com
`puting devices. Such as hand-held devices and personal digi
`tal assistants (PDAs), has led to an increasing use of devices
`to send and receive text and data. One example of th