Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 1 of 43
`Case 5:20-cv-01756—NC Document 1-4 Filed 03/11/20 Page 1 of 43
`
`
`
`
`
`
`
`
`EXHIBIT 4
`
`EXHIBIT 4
`
`

`

`(12) United States Patent
`Knaven
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,552,984 B2
`Oct. 8, 2013
`
`US008552984B2
`
`(54) METHOD, SYSTEM, APPARATUS AND
`COMPUTER-READABLE MEDIA FOR
`DIRECTING INPUT ASSOCATED WITH
`KEYBOARD-TYPE DEVICE
`(75) Inventor: Peter Knaven, North Vancouver (CA)
`(73) Assignee: 602531 British Columbia Ltd.,
`Vancouver, B.C. (CA)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 1360 days.
`
`(*) Notice:
`
`(21) Appl. No.: 11/036,267
`
`(22) Filed:
`
`Jan. 13, 2005
`
`(65)
`
`Prior Publication Data
`US 2006/O152496 A1
`Jul 13, 2006
`ul. 13,
`
`(2006.01)
`(2006.01)
`
`(51) Int. C.
`G06F 3/02
`G09G 5/00
`(52) U.S. Cl.
`USPC ............................. 345/168:345/2.1; 715/773
`(58) Field of Classification Search
`USPC ............................ 345/168–172, 2.1; 715/773
`See application file for complete search history.
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`. 400,486
`
`5/1988 Hirosawa et al. ............. 364,900
`4,744,050 A
`9/1988 Miyao et al. ...........
`... 364/419
`4,774,666 A
`10, 1988 Goldwasser et al. ......... 364/419
`RE32,773 E
`4,782.464 A 1 1/1988 Gray et al.
`4,783,761. A 1 1/1988 Gray et al.
`(Continued)
`FOREIGN PATENT DOCUMENTS
`O352377 A1
`1/1990
`O 643 357 A2
`3, 1995
`Continued
`(Continued)
`OTHER PUBLICATIONS
`
`EP
`EP
`
`Don Hopkins, “A Description of Pie Menus' (http://catalog.com/
`hopkins/piemenus/PieMenuIDescription.html).
`(Continued)
`
`Primary Examiner — Michael Pervan
`(74) Attorney, Agent, or Firm — Kilpatrick Townsend &
`Stockton LLP
`
`ABSTRACT
`(57)
`In one aspect of the present invention a computer-imple
`mented method is provided of processing input key events
`associated with user input received from a keyboard-type
`device, wherein the keyboard-type device selected from at
`least one of a keyboard and a keypad. In this aspect, input key
`events associated with a first process active within an operat
`ing system are received and monitored for a first predefined
`input key event associated with user selection of a first key of
`the keyboard-type device for at least a predetermined time
`period. In response to identifying the first predefined input
`key event, the input key events are redirected from the first
`process to a second process. The input key events are moni
`tored for a second predefined input key event associated with
`further redirection of the input key events. In response to
`identifying the second predefined input key event, the input
`key events are redirected to the first process.
`53 Claims, 21 Drawing Sheets
`
`.
`
`. 364/419
`
`38. Rhigomery .
`5, 1982 Damerau
`8/1983 Hayashi et al. ............... 364,900
`9, 1984 Dickinson et al.
`
`3/1987 Hardy et al.
`8, 1987 Heard et al.
`3, 1988 Bradshaw ..................... 364,403
`
`
`
`
`
`so A
`4,330,845. A
`4,396,992 A
`4,471.459 A
`2.85 A
`4,648,044 A
`4,689,768 A
`4,730,252 A
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 2 of 43
`
`
`
`
`
`38. 2S, . . . . . .
`
`
`
`
`
`? O
`
`10
`
`Input
`redirection
`equested/ No
`
`
`
`Repetition
`indication
`
`Repetitive
`Keys are
`navigational?
`
`Send registered
`msg(s) to IMD
`
`Serid registered
`msg(s) to Mid
`
`Pass original
`message to OS22
`
`
`
`
`
`
`
`
`
`
`
`
`
`Yes
`
`14
`
`Transform repetition
`into individual
`registered messages
`
`Seadregistered msgs
`with altered info to
`M
`
`Pass original
`message to OS 22
`
`6
`
`118
`
`

`

`US 8,552,984 B2
`Page 2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`I was
`
`. K.
`
`J. J.
`
`K-1 - W L v. will wiwu.
`
`4,786,765 A 1 1/1988 Yamanami et al.
`4,807,181 A
`2/1989 Duncan, IV et al. .......... 364,900
`4,847,766 A
`7, 1989 McRae et al.
`4,891,786 A
`1/1990 Goldwasser .................. 364,900
`4.969,097 A 1 1/1990 Levin ............................ 364/419
`4,980,855. A 12/1990 Kojima
`5,040,113 A
`8, 1991 Mickunas ..................... 364/419
`5,060,154. A 10/1991 Duncan, IV.
`5,067,165 A 11, 1991 Nishida
`5,096,423. A
`3/1992 Goldwasser .................. 434/118
`5,203,704 A
`4, 1993 McCloud ...................... 434,156
`5,218,536 A
`6/1993 McWherter
`5,220,649 A
`6/1993 Forcier .......................... 71.5/541
`5,220,652. A
`6/1993 Rowley ......................... 395/275
`5,258,748 A 11/1993 Jones
`5,261,112 A 11/1993 Futatsugi et al.
`5,297,041 A
`3, 1994 Kushler et al. ........... 364,419.15
`5,305,205 A
`4, 1994 Weber et al.
`5,327,161 A
`7/1994 Logan et al. .................. 345,157
`5,329,609 A
`7, 1994 Sanada et al.
`5,347,295 A
`9/1994 Agulnicket al.
`5,392.447 A
`2f1995 Schlack et al.
`5.487,616 A
`1/1996 Ichbiah ......................... 400,489
`E. A RE Man et al. .................. 395,156
`CO
`33: A
`3: Sustal
`345,173
`39 A
`2. SE et al.
`riskell ........................ 395,352
`-- w
`5,606,674. A
`2, 1997 Root
`5,621,641 A
`4, 1997 Freeman ....................... 395/796
`5,623.406 A
`4, 1997 Ichbiah ......................... 395/753
`5,629,733. A
`5/1997 Youman et al.
`5,649,223. A
`7, 1997 Freeman ....................... 395/796
`5,657,397 A
`8, 1997 BokSer
`5,666,139 A
`9, 1997 Thielens et al.
`5,689,667 A 11/1997 Kurtenbach .................. 395,352
`5,704,029 A 12/1997 Wright, Jr. .................... 715,505
`5,724.457 A
`3, 1998 Fukushima
`5,734,749 A
`3, 1998 Yamada et al.
`5,745,116 A
`4, 1998 Pisutha-Arnond ............ 345,358
`5,758.324 A
`5, 1998 Hartman et al.
`5,790,115 A
`8/1998 Pleyer et al.
`5,805,158. A
`9, 1998 Bertram et al.
`5,805,159 A
`9, 1998 Bertram et al.
`5,805,167 A
`9/1998 Van Cruyningen ........... 345 (353
`5,805,911 A
`9, 1998 Miller ........................... 395/796
`5,812,697 A
`9, 1998 Sakai et al.
`5,818,437 A * 10, 1998 Grover et al. ................. T15,811
`5,821,512 A 10/1998 O'Hagan et al.
`5,835,635 A 11/1998 Nozaki et al.
`5,838,302 A 11/1998 Kuriyama et al.
`5,845,300 A 12/1998 Comer et al.
`5,864,340 A
`1/1999 Bertram et al.
`5,881,169 A
`3/1999 Henry, Jr.
`5,896,321 A
`4/1999 Miller et al. ............. 365,189.01
`5,911,485 A
`6, 1999 ROSSmann
`5,914,708 A
`6/1999 LaGrange et al. ............ 345,179
`5,926,178 A
`7/1999 Kurtenbach .......
`$45,353
`5,943,039 A
`8, 1999 Anderson et al. ............. 345/146
`5,953,541 A
`9/1999 King et al.
`5,959,629 A
`9/1999 Masui ........................... 345/347
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 3 of 43
`
`5,963,671 A 10, 1999 Comerford et al. ........... 382,230
`5,974,558 A 10/1999 Cortopassi et al.
`
`5,982,351 A 1 1/1999 White et al.
`6,002,390 A 12/1999 Masui ........................... 345/173
`6,005,549 A 12/1999 Forest ........................... 345/157
`6,008,799 A 12, 1999 Van Kleeck .................. 345,173
`6,011,554 A
`1/2000 King et al.
`6,026,233 A
`2/2000 Shulman et al.
`6,037.942 A
`3/2000 Millington
`6,084,576 A
`7/2000 Leu et al. ...................... 345,168
`
`
`
`6,608,639 B2
`
`8, 2003 McGovern
`
`7/2000 Kabada et al.
`6,088,649 A
`7/2000 Buxton et al. ................ 345,863
`6,094,197 A
`8/2000 Leyerle
`6,097,392 A
`8/2000 Gunji et al.
`6,097.841 A
`8/2000 Krueger et al.
`6,098,086 A
`8/2000 Ukigawa et al.
`6,101.461 A
`8, 2000 Hullender et al.
`6,111,985 A
`6,144,378 A 1 1/2000 Lee
`6,148,104 A
`1 1/2000 Wang
`6,154,758. A 1 1/2000 Chiang
`6,167.411 A 12/2000 Narayanaswamy
`6,167.412 A 12/2000 Simons
`6,188,789 B1
`2/2001 Marianetti, II et al. ....... 382, 189
`6.256,030 B1,
`7/2001 Berry et al.
`6,262,719 B1
`7/2001 Bi et al. ........................ 345,179
`6,275,612 B1
`8/2001 Imoto
`6,282,315 B1
`8/2001 Boyer
`6.286,064 B1
`9/2001 King et al.
`. . . . . . . . . . . . . . . 345,173
`6,292, 179 B1
`9, 2001 Lee .
`. . . . . . . . .
`6,295,372 B1
`9, 2001 Hawkins et al.
`... 382,187
`6,307.549 B1
`10/2001 King et al. .................... 345,352
`6,369,807 B1
`4/2002 Nakashima ................... 345,179
`6,377,965 B1
`4/2002 Hachamovitch et al. ..... 715,534
`6.405,060 B1
`6/2002 Schroeder et al.
`6.424,983 B1
`7/2002 Schabes et al.
`6,442,295 B2
`8/2002 Navoni et al.
`6,473,006 B1 10/2002 Yu et al.
`6,487,424 B1
`1 1/2002 Kraft et al.
`6,502,114 B1
`12/2002 Forcier
`6,539,421 B1
`3/2003 Appelman et al.
`6,621.939 B1
`9/2003 Negishi
`6,646,572 B1 11/2003 Brand
`6,654,733 B1
`11/2003 Goodman et al. .............. 7O6/52
`6,661,920 B1
`12.2003 Skinner........................ 382,187
`6,734,881 B1
`5/2004 Will
`6,751,603 B1
`6/2004 Bauer et al.
`6,801,190 B1
`10/2004 Robinson et al.
`6,888,141 B2
`5, 2005 Carr
`6,934,906 B1
`8, 2005 Cheok
`6,970,513 B1
`11/2005 Puri et al.
`6,978.421 B1
`12/2005 Aida
`7,003,446 B2
`2/2006 Trower et al.
`7,224.409 B2
`5/2007 Chin et al.
`7,322,023 B2
`1/2008 Shulman
`SEA 858: SM
`2002fOO67377 A1
`6, 2002 McGovern
`-
`11 . .
`.
`.
`.
`v. V. .
`2002-0087279 A1
`72002 Hall
`2003.0137605 A. T/2003 Chinet al.
`2004/0021691 A1
`2/2004 Dostie et al. .................. 345,773
`2005, 0198144 A1
`9, 2005 Kraenzel et al.
`2005/0210402 A1
`9, 2005 Gunn et al.
`2005/0223308 A1 10, 2005 Gunn et al.
`2006/0265208 A1 11/2006 ASSadollahi
`2007/0188472 A1* 8, 2007 Ghassabian ................... 345/169
`
`. . . . . . . . . . . .
`
`345,816
`
`FOREIGN PATENT DOCUMENTS
`
`E.
`EP
`EP
`Ep
`
`EP
`EP
`
`JP
`p.
`p.
`JP
`JP
`WO
`WO
`WO
`
`8.3% A.
`O 844 571 A2
`O 472 444 B1
`O 858 023 A2
`
`E.
`5, 1998
`8/1998
`8, 1998
`
`O 844 570 A3
`O 844 571 A3
`
`8, 1999
`8, 1999
`
`10154033. A
`10154144. A
`11143614
`11167569 A
`10333818
`WO96,09579
`WO98, 11480
`WO99,28811
`
`6/1998
`6/1998
`5, 1999
`6, 1999
`4/2002
`3, 1996
`3, 1998
`6, 1999
`
`

`

`US 8,552,984 B2
`Page 3
`
`(56)
`
`References Cited
`
`FOREIGN PATENT DOCUMENTS
`
`WOO1,95095 A2 12/2001
`WO
`WOO233527 A2 * 4, 2002
`WO
`WO WO 2004/107101 A2 12/2004
`
`OTHER PUBLICATIONS
`
`Don Hopkins, “Natural Selection: The Evolution of Pie Menus',
`BayCHI, Oct. 13, 1998 (http://catalog.com/hopkins/piernenus/Natu
`ral Selection.html).
`Russell Nelson, “Pie Menu Window Manager'. Jun. 3, 1998 (http://
`www.crynwr.com/piewm/).
`Don Hopkins, “ActiveX Pie Menus' (http://catalog.com/hopkins/
`piemenus/ActiveXPieMenus.html).
`Don Hopkins, “Pie Menu References” (http://catalog.com/hopkins/
`piemenu-references.html).
`Don Hopkins, “Pie Menus' (http://art.net/~hopkins/Don/piemenus/
`index.html).
`Office Action dated Jan. 14, 2005, issued for U.S. Appl. No.
`09/631, 101.
`Office Action dated Jan. 20, 2004, issued for U.S. Appl. No.
`09/631, 101.
`Office Action dated Jun. 14, 2005, issued for U.S. Appl. No.
`09/631, 101.
`Office Action dated Mar. 20, 2006, issued for U.S. Appl. No.
`09/631, 101.
`Office Action dated Oct. 18, 2006, issued for U.S. Appl. No.
`10/399,560.
`Office Action dated Jan. 5, 2007, issued for U.S. Appl. No.
`09/631, 101.
`Office Action dated May 18, 2007, issued for U.S. Appl. No.
`11/134,759.
`Office Action dated Jun. 4, 2007, issued for U.S. Appl. No.
`09/631, 101.
`Office Action dated Jun. 7, 2007, issued for U.S. Appl. No.
`11/133,770.
`Office Action dated Jul. 17, 2007, issued for U.S. Appl. No.
`10/399,560.
`Office Action dated Mar. 4, 2008, issued for U.S. Appl. No.
`11/133,770.
`Office Action dated Apr. 15, 2008, issued for U.S. Appl. No.
`10/399,560.
`Office Action dated Apr. 3, 2008, issued for U.S. Appl. No.
`11/134,759.
`Office Action dated Nov. 24, 2008, issued for U.S. Appl. No.
`11/133,770.
`Office Action dated Jan. 7, 2009, issued for U.S. Appl. No.
`10/399,560.
`Office Action dated Jan. 26, 2009, for U.S. Appl. No. 1 1/134,810.
`Office Action dated Feb. 4, 2009, issued for U.S. Appl. No.
`11/134,759.
`Office Action dated Jun. 10, 2009, for U.S. Appl. No. 1 1/133,770.
`Office Action dated Apr. 30, 2008, issued for U.S. Appl. No.
`11/036,267.
`Office Action dated Jan. 26, 2009, issued for U.S. Appl. No.
`11/036,267.
`Advisory Action dated Sep. 16, 2005, issued for U.S. Appl. No.
`09/631, 101.
`Notice of Allowability dated Aug. 8, 2007, issued for U.S. Appl. No.
`09/631, 101.
`IPER dated Jun. 19, 2001, issued for International Application No.
`PCT/CAOO/OO285.
`IPER dated Dec. 9, 2003, issued for International Application No.
`PCT/CAO1/O 1473.
`Office Action dated Feb. 7, 2002, issued for E.P. Patent Application
`No. OO 910 460.5
`Office Action dated Jun. 23, 2005, issued for E.P. Patent Application
`No. 01981991.1.
`Office Action dated Aug. 11, 2004, issued for E.P. Patent Application
`No. 01981991.1.
`
`Partial European Search Report dated Mar. 4, 2009, for European
`Patent Application No. 07022182.5.
`Extended European Search Report dated May 27, 2009 for European
`Patent Application No. 07022182.5.
`3Com, “PalmPilot Handbook”, 1997 3Com Corp., pp. 37-39.
`Baeza-Yates, “A New Approach to Text Searching” (Preliminary Ver
`sion), Centre for the New O.E.D. & Data Structuring Group, Depart
`ment of Computer Science, University of Waterloo, pp. 168-175.
`Bellman, T., & MacKenzie, I.S. (1998), A probabilistic character
`layout strategy for mobile text entry. Proceedings of Graphics inter
`face '98, pp. 168-176, Toronto: Canadian Information Processing
`Society.
`Bohan et al., “A Psychophysical Comparison of Two Stylus-Driven
`Sofi Keyboards'. Department of Psychology, Wichita State Univer
`sity.
`Bohlman, Eric, “KeyCache The Universal Input Accelerator, Ver
`sion 2.1.” copyright Sep. 29, 1994, OMS Development, documenta
`tion, downloaded from <http://www.sac.sk/files.php?d=16&l=K>.
`12 pages.
`Callhan at al abstract: . . . compare pie menus to linear menus.
`Darragh et al., “The Reactive Keyboard: A Predictive Tiping Aid
`University of Calgary, Nov. 1990.41-49.
`Jones, P.E. "Virtual Keyboard with Scanning and Augmented by
`Prediction”, 1998, Proc. 2" Euro. Conf. Disability, Virtual Reality &
`Assoc. Tec. Skovde, Sweden, 1998, pp. 45-51.
`MacKenzie et al., “Alphanumeric Entry on Pen-Based Computers'.
`1994. Intl. Journal of Human-Computer Studies, 41, pp. 775-792,
`<http://www.yorku.ca/mack/IJHCS.html>.
`MacKenzie et al., “Text Entry. Using Sofi Keyboard, 1999, Behavior
`& Information Tech., 18, pp. 235-244.
`MacKenzie et al., “The Design and Evaluation of a High-Perfor
`mance SoftKeyboard”, 1999, ACM, pp. 25-31.
`Mankopff et al., “Cirrin: A Word-level Unistroke Keyboard for Pen
`Input', 1998, ACM, pp. 213-214.
`Masui, T. 1998, “An Efficient Text Input Method for Pen-based
`Computers.” Apr. 18-23, 1998. In Proc. SIGCHI Conf. on Human
`Factors in Computing Systems (LC. Karat, A. Lund, J. Coutaz, and J.
`Karat, Eds. Conference on Human Factors in Computing Systems.
`ACM, pp. 328-335.
`Masui, T. 1999. “POBox: An Efficient Text Input Method for
`Handheld and Ubiquitous Computers'. In Proc. of 1 Intl. Symp. on
`Computer Science, vol. 1707. Springer-Verlg, London, 289-300.
`National Council on Disability Document Archive, software guide to
`alternative input and output programs, circa 1996, downloaded from
`<http://www.dimenet.com/disnews/archive.php?mode=P&id=49),
`12 pages.
`NCIP listing of WordPrediction & Writing Tools, circa 1997, down
`loaded from <http://www2.edc.org/incip/LIBRARY/wp/Pubs.htm>.
`7 pages.
`Softype 3.1 Help File, circa 1997, downloaded from <http://orin.
`combinaries st31dmz.exe), 38 pages.
`Softype Version 3.1 Screendumps generated from demo version of
`software, downloaded from <http://orin.com/binaries/st31dmz.
`exe>, 21 Figures.
`Venolia"T-Cube. A Fast, Self-Disclosing Pen-Based Alphabet”, Bos
`ton, Massachusetts USAApr. 24-28, 1994 Human Factors in Com
`puting Systems, pp. 265-270.
`Vitoria et al., “A Comparison of Prediction Techniques to Enhance
`the Communication Rate'. Springer-Verlg, 2004, pp. 400-417.
`Wivik 2 REP Software manual, circa Feb. 1998, Orentke Romich
`Company, downloaded from <http://web.archive.or web/
`2000 1204165600/wivik.com/html/downloads.htm>, 128 pages.
`Wu and Manber, “AGREP A Fast Approximate Pattern-Matching
`Tool,(Preliminary Version), Department of Computer Science, Uni
`versity of Arizona.
`Darragh et al., “The Reactive keyboard, Cambridge Series on
`Human-Computer Interaction.” Cambridge Univ. Press, (C) 1992, 186
`pageS.
`Notice of Allowance dated Nov. 26, 2010 for U.S. Appl. No.
`1 1/871,900.
`Office Action dated Dec. 21, 2010 for U.S. Appl. No. 1 1/871,904.
`Office Action dated Aug. 18, 2011 for U.S. Appl. No. 1 1/871,900.
`Office Action dated Oct. 17, 2011 for U.S. Appl. No. 1 1/871,904.
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 4 of 43
`
`

`

`US 8,552,984 B2
`Page 4
`
`(56)
`
`References Cited
`
`OTHER PUBLICATIONS
`
`Notice of Allowance dated Nov. 30, 2010 for U.S. Appl. No.
`1 1/871,887.
`Office Action dated Jul. 18, 2012 for U.S. Appl. No. 1 1/871,900.
`Office Action dated Aug. 6, 2012 for U.S. Appl. No. 1 1/871,904.
`Office Action dated Sep. 29, 2009 issued for U.S. Appl. No.
`10/399,560.
`Notice of Allowance issued or U.S. Appl. No. 1 1/134,810 on Nov.30,
`2009.
`Office Action dated Feb. 25, 2010 for U.S. Appl. No. 1 1/134,759.
`Julie, T9 Keyboard Review, the gadgeteer available at www.the
`gadgeteer.com/1998/08/27/t9 keyboard review?. Aug. 27, 1998
`(accessed Feb. 23, 2010).
`Office Action dated Apr. 14, 2010 for U.S. Appl. No. 1 1/871,887.
`Office Action dated Mar. 31, 2010 for U.S. Appl. No. 1 1/871,900.
`Office Action dated Apr. 14, 2010 for U.S. Appl. No. 1 1/871,904.
`Jakobsson, Matti, "Autocompletion in Full Text Transaction entry: A
`Method for Humanized Input', Univ. of Vaasa, ACM, (C) 1986 pp.
`327-332.
`Notice of Allowance dated Dec. 28, 2009 for U.S. Appl. No.
`11/133,770.
`Notice of Allowance dated Nov. 26, 2010 issued for U.S. Appl. No.
`1 1/871,900.
`Notice of Allowance dated Nov.30, 2010 issued for U.S. Appl. No.
`1 1/871,887.
`Toshiyuki Masui, Sony Computer Science Laboratory Inc.,
`Shinagawa, Tokyo, "An Efficient Text Input Method for Pen-based
`
`Computers'. Proceedings of the ACM Conference on Human Factors
`in Computing Systems, Apr. 1998, pp. 328-335.
`Toshiyuki Masui, Sony Computer Science Laboratories, Inc.,
`Shinagawa, Tokyo, “Integrating Pen Operations for Composition by
`Example”.
`Don Hopkins, “The Design and Implementation of Pie Menus'. Dr.
`Dobb's Journal, Dec. 1991 (http://art.net/~hopkins/Don/piemenus/
`ddi/piemenus.hml).
`Jason I. Hong, “Java Pie Menus'. Sep. 4, 1999 (ttp://www.cs.
`berkeley.edu/~asonh/download/sotware/piemenu/).
`Tom Nantais, et al., IEEE Transactions on Rehabilitation Engineer
`ing, "A Predictive Selection Technique for Single-Digit Typing With
`a Visual Keybaord”, No. 3 Sep. 1994.
`“GtkPieMenu' 1999 orfelyus (2 pages).
`Pre-Brief Appeal Conference Decision dated Jan. 14, 2013 for U.S.
`Appl. No. 1 1/871,900.
`Advisory Action dated Dec. 6, 2012 for U.S. Appl. No. 1 1/871,900.
`Pre-Brief Appeal Conference Decision dated Apr. 19, 2012 for U.S.
`Appl. No. 1 1/871,904.
`Office Action dated Dec. 20, 2012 for U.S. Appl. No. 12/749,318.
`Office Action dated Jan. 18, 2013 for U.S. Appl. No. 12/531.296.
`Mercedes-Benz Navigation System—Operation Guide (C) 1997
`Mercedes-Benz of North America Inc. Technical Information Order
`No. S-2690-097 Printed in the United States.
`Mercedes-Benz AG Auto Pilot System—Guide (Translation
`attached).
`Mercedes-Benz COMAND Betriebsanleitung Navigation Guide
`(Translation attached).
`* cited by examiner
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 5 of 43
`
`

`

`U.S. Patent
`
`US 8,552,984 B2
`
`
`
`
`
`(Jossapoud pioAA) V SS33OICH
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 6 of 43
`
`I “?IH
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 2 of 21
`
`US 8,552,984 B2
`
`
`
`Processing Unit
`12
`
`Video
`display
`circuit
`13
`
`Display Device
`15
`
`H/W Input
`Interface
`14
`
`Application(s)
`
`input Management
`System
`20
`
`Computer readable medium
`16
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 7 of 43
`
`FG. 2
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 3 of 21
`
`US 8,552,984 B2
`
`100
`
`104
`
`Repetition
`indication ?
`
`10
`
`
`
`
`
`Input
`redirection
`sequested 2
`
`
`
`
`
`
`
`Repetitive
`Keys are
`navigational
`
`No
`
`O2
`
`106
`
`No
`
`
`
`Send registered
`msg(s) to IMD
`
`Send registered
`msg(s) to IMD
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 8 of 43
`
`Pass original
`message to OS 22
`
`End
`
`F.G. 3
`
`Yes
`
`114
`
`Transform repetition
`into individual
`registered messages
`
`Send registered msgs
`with altered info to
`
`Pass original
`message to OS 22
`
`116
`
`118
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 4 of 21
`
`US 8,552,984 B2
`
`
`
`No
`
`Alphanumeric
`Key Down msg?
`
`No
`
`
`
`Yes
`
`148
`
`Clear
`redirection
`timer if
`Key up msg.
`
`Redirection
`event
`
`
`
`150
`
`Set redirection
`flag,
`record
`redirection
`cancellation key
`
`Set
`redirection
`e
`
`Pass key message to
`PredictionLogic
`
`146
`
`132
`
`
`
`
`
`
`
`Redirection
`cancellation
`requested?
`
`Yes
`
`No
`
`138
`
`134
`
`Pass
`Pass key
`termination
`mcssage to
`PredictionLogic request to
`PredictionLogic
`
`136
`
`Clear redirection
`flag
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 9 of 43
`
`End
`
`FIG. 4
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 5 of 21
`
`US 8,552,984 B2
`
`wfaram
`
`
`
`Param
`
`50
`
`Reserved
`
`Scan code
`
`Wirtual key code
`
`Context code Extended-key flag Key repeat count
`Repetition indication flag
`Keyup indication flag
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 10 of 43
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 6 of 21
`
`US 8,552,984 B2
`
`
`
`Process A,
`Word Processor
`
`Process B,
`Spreadsheet
`
`Process C,
`Media player
`
`Process N,
`Calculator
`
`Input management
`director
`
`Process PL
`
`Process Math
`
`Process Bitmap
`
`Process Sound
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 11 of 43
`
`Process Multimedia
`
`38
`
`FIG. 8
`
`Monitoring module
`
`Applications
`
`Keyboard function expanders
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 7 of 21
`
`US 8,552,984 B2
`
`130
`
`132
`
`Redirection
`cancellation
`requested?
`
`Yes
`
`134A
`
`Send
`termination
`request to
`selected
`expander,
`wait for
`confirmation
`
`136
`
`Clear redirection
`flag
`
`
`
`Alphanumeric
`Key Down msg?
`
`
`
`
`
`142
`
`Clear
`redirection
`timer if
`Key up msg
`
`Redirection
`event
`
`
`
`
`
`
`
`150
`
`Set redirection
`flag, record
`redirection
`cancellation key
`
`Send key
`message to
`Selected
`expander
`
`144
`
`52
`
`
`
`Set
`redirection
`timer
`
`Select the
`desired expander
`(if any)
`
`Collect key
`messages for future
`selected expander
`
`146A
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 12 of 43
`
`End
`
`FIG. 9
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 8 of 21
`
`US 8,552,984 B2
`
`Showall registered expanders who are
`registered for this event
`
`162
`Expander - No
`Selected?
`
`es
`
`164
`
`Forward collected keyboard input, and editor
`window infomation to selected expander
`Set indication to send future keyboard input to
`selected expander
`
`
`
`
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 13 of 43
`
`FIG. 10
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 9 of 21
`
`US 8,552,984 B2
`
`Registrants
`(Expanders)
`Database query
`Prediction Engine
`Thesaurus
`Calculator
`
`Database storage/
`query
`Pattem matching
`OCR recognition
`Dither/Smoothing
`
`Information passed
`
`Typed text or
`recent text in clipboard
`
`Current bitmap or
`Recent entered bitmap in clipboard
`
`Database storage
`Recording start/stop
`OCR recognition start/stop
`Dither/Smoothing start/
`Stop
`
`Recording start/stop
`Song Selection
`Patten matching
`Speech to text recognition
`Pitch/equalizer function
`
`Typed text
`Snapshot of datastream
`
`Typed text
`Snapshot of datastream
`
`Sound Context
`
`
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 14 of 43
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 10 of 21
`
`US 8,552,984 B2
`
`
`
`Process A
`(Word processor)
`
`Process B
`(Spread sheet)
`
`Keyboard input
`
`N Monitoring module
`
`20
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 15 of 43
`
`FIG. 12
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 11 of 21
`
`US 8,552,984 B2
`
`
`
`Input
`redirection
`equested ?
`
`
`
`Repetition
`indication ?
`
`
`
`
`
`Yes
`
`
`
`
`
`
`
`
`
`Repetitive
`Keys are
`navigational ?
`
`Send
`registered
`msg(s) to
`MD
`
`
`
`
`
`Send
`registered
`RE) to
`
`
`
`
`
`
`
`
`
`
`
`First
`Repetition?
`
`
`
`
`
`
`
`Pass original
`se tO
`
`Send
`registered
`msg to
`
`F.G. 13
`
`Transform repetition
`into individual
`registered messages
`
`Send registered msgs
`with altered info to
`IMD
`
`16
`
`Pass original
`message to OS 22
`
`118
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 16 of 43
`
`
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 12 of 21
`
`US 8,552,984 B2
`
`
`
`
`
`
`
`Alpanumeric
`Key Down
`msg?
`
`
`
`
`
`
`
`No
`
`No
`
`146
`
`Pass key message
`to PredictionLogic
`
`30
`
`132
`
`Redirection
`cancellation
`requested?
`
`
`
`Repeat
`indication?
`
`Yes
`
`150
`
`
`
`No
`
`138
`
`Yes
`
`134
`
`Set redirection
`flag,
`Record
`redirection
`cancellation key
`
`Pass key
`message to
`PredictionLogic
`
`Pass
`termination
`request to
`PredictionLogic
`
`36
`
`Clear
`redirection
`flag
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 17 of 43
`
`End
`
`FIG. 14
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 13 of 21
`
`US 8,552,984 B2
`
`130
`
`
`
`
`
`
`
`Alpanumeric
`Key Down
`msg?
`
`132
`
`Yes
`
`141
`
`
`
`
`
`Redirection
`cancellation
`requested?
`
`
`
`
`
`
`
`No
`
`Repeat
`indication?
`
`No
`
`Yes
`
`150A
`
`146
`
`No
`
`138
`
`Yes
`
`134
`
`Pass key message
`to PredictionLogic
`
`Set redirection
`flag,
`Record
`redirection
`cancellation key,
`Grab system
`level focus
`
`
`
`Pass key
`message to
`PredictionLogic
`
`Pass
`termination
`request to
`PredictionLogic
`
`136A
`
`
`
`Clear
`redirection
`flag,
`Assign
`system level
`focus back to
`first process
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 18 of 43
`
`End
`
`F.G. 15
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 14 of 21
`
`US 8,552,984 B2
`
`
`
`NO
`
`Alphanumeric
`Key Down msg?
`
`No
`
`
`
`Yes
`
`148
`
`
`
`
`
`Redirection
`event
`
`150A
`
`Set redirection
`flag,
`Record
`redirection
`cancellation key,
`Grab system
`level focus
`
`Clear
`redirection
`timer if
`Key up msg
`
`Set
`redirection
`le
`
`Pass key message to
`PredictionLogic
`
`146
`
`132
`
`
`
`
`
`Redirection
`cancellation
`requested?
`
`Yes
`
`No
`
`38
`
`134
`
`Pass
`Pass key
`termination
`message to
`PredictionLogic request to
`PredictionLogic
`
`136A
`
`Clear redirection
`flag,
`Assign system
`level input focus
`back to first
`process
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 19 of 43
`
`End
`
`FIG. 16
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 15 of 21
`
`US 8,552,984 B2
`
`
`
`
`
`
`
`
`
`
`
`No
`
`Alphanumeric
`Key Down
`msg?
`
`Yes
`
`148
`
`Redirection
`timer event
`
`Set redirection
`flag
`
`Clear
`redirection
`timer if
`keyup msg
`
`Set
`redirection
`timer
`
`Pass key message to
`PredictionLogic
`
`Redirection
`cancellation
`key
`Selected?
`
`132A
`
`Yes
`
`Pass key
`message to
`PredictionLogic
`
`Pass
`termination
`request to
`PredictionLogic
`
`Clear
`redirection
`flag
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 20 of 43
`
`End
`
`F.G. 17
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 16 of 21
`
`US 8,552,984 B2
`
`GetKeyState for Control Key
`
`100A
`
`100B
`
`
`
`No
`
`
`
`No control key
`detected or
`Ctrl-C detected
`
`100
`
`
`
`Input
`redirection
`equested?
`
`104
`
`
`
`
`
`Yes
`
`
`
`102
`
`Send
`registered
`msg(s) to
`IMD
`
`
`
`103
`
`Pass
`original
`message to
`OS 22
`
`Repetition
`indication ?
`
`10
`
`Repetitive
`Keys are
`navigational
`
`Yes
`
`114
`
`
`
`111
`
`Transform repetition
`into individual
`registered messages
`
`16
`
`Send registered msgs
`No with altered info to
`IMD
`
`Pass original
`OS 22
`message to OS2
`
`18
`
`13 Yes
`
`Send
`Ctrl-C
`registered
`1S
`3.
`to MD
`
`registered
`msg(s) to
`IMD
`
`108
`
`Pass original
`message to OS
`22
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 21 of 43
`
`End
`
`FIG. 18
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 17 of 21
`
`US 8,552,984 B2
`
`139
`
`NO
`
`130
`
`Grabbed
`focus?
`
`Yes
`
`
`
`32
`
`
`
`Redirection
`cancellation
`requested?
`
`48
`
`Redirection
`event
`
`150
`
`Set redirection
`flag,
`record
`redirection
`cancellation key
`
`No
`
`138
`
`Yes
`
`134
`
`Pass key
`message to
`Prediction
`Logic
`
`Pass
`termination
`request to
`PredictionLogic
`
`Ctrl C
`message?
`
`
`
`
`
`
`
`
`
`
`
`Yes
`
`150A
`
`Set
`redirection
`flag,
`record
`redirection
`cancellation
`key
`
`
`
`Alphanumeric
`Key Down msg?
`
`142
`
`Yes
`
`Clear
`redirection
`timer if
`Key up msg
`
`144
`
`Set
`redirection
`timer
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 22 of 43
`
`Pass key message to
`PredictionLogic
`
`146
`
`End
`
`FIG. 19
`
`36
`
`Clear redirection
`flag
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 18 of 21
`
`US 8,552,984 B2
`
`Overal.,,
`Overail...E
`Overalls
`Overarching
`Overawed
`
`
`
`FIG. 20
`
`Keyboard 206
`
`200
`
`Touch
`Sensitive
`Screen
`210
`
`Interface 208
`
`Interface 212
`
`CPU 202
`
`Source
`
`PTES 34.1A
`
`Computer readable medium 28
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 23 of 43
`
`Interface (214)
`to PCD
`
`Interface to (216)
`external storage
`
`F.G. 21.
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 19 of 21
`
`US 8,552,984 B2
`
`appear...
`application...
`approach...
`appropriate,..
`
`200
`
`206
`
`FIG. 22
`
`
`
`Free,
`Friend.,,
`French,,,
`Front.,
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 24 of 43
`
`FIG. 23
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 20 of 21
`
`US 8,552,984 B2
`
`
`
`
`
`
`
`Input
`redirection
`equested
`
`No
`
`104
`
`Yes
`
`Repetition
`indication ?
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Repetitive
`Keys are
`navigational 2
`
`Send msg(s) to
`IMD
`
`Send msg(s) to
`IMD
`
`
`
`
`
`Send message to
`PCD
`
`
`
`
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 25 of 43
`
`FIG. 24
`
`110
`
`Yes
`
`14
`
`Transform repetition
`into individual
`messages
`
`Send msgs with
`altered info to IMD
`
`116
`
`
`
`118
`
`Send message to
`PCD
`
`

`

`U.S. Patent
`
`Oct. 8, 2013
`
`Sheet 21 of 21
`
`US 8,552,984 B2
`
`130A
`
`
`
`No
`
`Redirection
`flag set?
`
`Alphanumeric
`Key Down msg?
`
`Yes
`
`
`
`
`
`132
`
`
`
`
`
`
`
`Redirection
`cancellation
`requested?
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Redirection
`event
`
`Sct redirection
`flag,
`Record
`redirection
`cancellation key
`
`
`
`Clear
`redirection
`timer if
`Key up msg
`
`
`
`
`
`
`
`
`
`Set
`redirection
`timer
`
`Pass key
`message to
`PTES 34.1A
`
`Pass
`termination
`request to
`PTES 34.1A
`
`
`
`
`
`136
`
`Clear redirection
`flag
`
`Pass key message to
`PTES 34. A
`
`
`
`Case 5:20-cv-01756-NC Document 1-4 Filed 03/11/20 Page 26 of 43
`
`FIG. 25
`
`

`

`US 8,552,984 B2
`
`1.
`METHOD, SYSTEM, APPARATUS AND
`COMPUTER-READABLE MEDIA FOR
`DIRECTING INPUT ASSOCATED WITH
`KEYBOARD-TYPE DEVICE
`
`FIELD OF THE INVENTION
`
`5
`
`The present invention relates to a method, system, appara
`tus and computer-readable media for directing input associ
`ated with a keyboard-type device.
`
`10
`
`SUMMARY OF THE INVENTION
`
`2
`(i) generate a plurality of predictive text completion
`candidates in response to said first input signals and
`display said plurality of predictive text completion
`candidates on said display device; and
`(ii) communicate at least one of said predictive text
`completion candidates to a personal computing
`device remote from the keyboard-type device in
`response to user selection of the at least one of said
`predictive text completion candidates.
`Several other aspects and features of the present invention
`will become apparent to those ordinarily skilled in the art
`upon review of the following description of specific embodi
`ments of the invention in conjunction with the accompanying
`drawings.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`In the accompanying drawings which illustrate embodi
`ments of the invention,
`FIG. 1 is a block diagram of an input management system
`in use in association with a first process and a second process,
`according to a first embodiment of the invention;
`FIG. 2 is a block diagram of a personal computing device
`loaded with a data entry system, according to a first embodi
`ment of the invention;
`FIGS. 3 and 4 are flow diagrams illustrating the operation
`of an input management system in accordance with the first
`embodiment of the present invention;
`FIG. 5 is a block diagram of a data structure for a keyboard
`message used in connection with the first embodiment;
`FIG. 6 is a block diagram illustrating operation of a key
`board when keyboard input is redirected to a second process,
`according to the first embodiment of the invention;
`FIG. 7 is a block diagram illustrating another form of
`operation of a keyboard when keyboard input is redirected to
`a second process, according to the first embodiment of the
`invention;
`FIG. 8 is a block diagram of an input management system
`in use in association with a first process and a second process,
`according to another em