(12) United States Patent
`Geilhuf e et al.
`
`I 1111111111111111 11111 lllll 111111111111111 111111111111111 IIIIII IIII 11111111
`US006584439Bl
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 6,584,439 Bl
`Jun.24,2003
`
`(54) METHOD AND APPARATUS FOR
`CONTROLLING VOICE CONTROLLED
`DEVICES
`
`(75)
`
`Inventors: Michael Geilhufe, Palo Alto, CA (US);
`David MacMillan, Woodside, CA (US);
`Avraham Bare), Doar na shimshon
`(IL); Amos Brown, Givat smmuhel
`(IL); Karin Lissette Bootsma, San
`Jose, CA (US); Lawrence Kent Gaddy,
`San Jose, CA (US); Phillip Paul Pyo,
`San Jose, CA (US)
`
`(73) Assignee: Winbond Electronics Corporation,
`Hsinchu (TW)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by O days.
`
`(21) Appl. No.: 09/316,643
`May 21, 1999
`
`(22) Filed:
`
`Int. CI.7 ................................................ GlOL 15/22
`(51)
`(52) U.S. Cl. ........................................ 704/270; 704/275
`(58) Field of Search ................................. 704/246, 270,
`704/272, 273, 275; 379/88.01, 88.02, 88.03,
`88.04
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`6/1985 Vander Molen
`4,520,576 A
`12/1985 Daniels et al.
`4,556,944 A
`2/1987 Clowes et al.
`4,644,107 A
`3/1988 Dubus
`4,731,811 A
`4/1988 Borth et al.
`4,737,976 A
`4,797,924 A * 1/1989 Schnars et al.
`8/1989 Fujisaki
`4,853,953 A
`8/1989 Rose
`4,857,030 A
`11/1989 van der Lely
`4,882,685 A
`5/1990 Kaneuchi et al.
`4,928,302 A
`10/1990 Marni et al.
`4,961,212 A
`4/1991 Schmuckal et al.
`5,007,081 A
`5/1991 Rohani et al.
`5,020,107 A
`(List continued on next page.)
`
`............. 704/275
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`EP
`WO
`WO
`WO
`WO
`WO
`
`0 238 695 Al
`0 618 710 Al
`0 618 710 A2
`0 700 031 Al
`0 739 121 A2
`0 730 261 A2
`0 810 502 Al
`0 854 417 A2
`0 867 860 A2
`0 911 808 Al
`WO 98 09265 A
`WO 98 55992
`WO 99/05671
`WO 99/08084
`WO 99/14928
`
`3/1986
`3/1994
`3/1994
`8/1995
`4/1996
`9/1996
`12/1997
`1/1998
`3/1998
`4/1999
`3/1998
`12/1998
`2/1999
`2/1999
`3/1999
`
`OTHER PUBLICATIONS
`
`Business Wire, Unisys announcing major enhancements to
`natural Language Assistant, Business Wire, Apr. 15, 1997.
`PR Newswire, PureSpeech Demonstrates Virtual Voice
`Assistant for PC Telephony Using Intel's MMX Technology,
`PR Newswire, Sep. 17, 1996.
`PR Newswire, PureSpeech Raises $4.3 Million in Venture
`Capital Financing, PR Newswire, May 7, 1996.
`Business Wire, Harris & Harris Group Increases Investment
`in PureSpeech, Business Wire, May 7, 1996.
`
`(List continued on next page.)
`
`Primary Examiner-Marsha D. Banks-Harold
`Assistant Examiner-Martin Lerner
`(74) Attorney, Agent, or Firm-Blakely, Sokoloff, Taylor &
`Zafman LLP
`
`(57)
`
`ABSTRACT
`
`Voice controlled devices with speech recognition have user
`assignable appliance names and default appliance names to
`address and control the voice controlled devices. Methods of
`controling voice controlled devices include addressing a
`voice controlled device by name and providing a command.
`
`37 Claims, 21 Drawing Sheets
`
`Microfiche Appendix Included
`(1 Microfiche, 72 Pages)
`
`601
`
`!·Pieooesay!.,.~"P!ease:
`!he new ~ repoatttie
`,
`:
`""'1ame>"
`
`:
`
`ea~• _:
`
`[""·";:'~;';;~~;;;~;;,;;;: \
`
`! re:t~:~n~ ~::!~~~=~If :
`
`[ "Sony,ldKln1understand.;
`NameChangecanceled"
`'
`eOORetumtoStart.
`
`-!
`600
`Samples
`areOK
`:
`/
`P1ay"My ~ Return
`to Start
`'
`
`<name> !
`
`Page 1 of 43
`
`GOOGLE EXHIBIT 1019
`
`

`

`US 6,584,439 Bl
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`5,042,063 A
`5,054,082 A
`5,117,460 A
`5,165,095 A
`5,222,121 A
`5,369,685 A
`5,406,618 A
`5,452,340 A
`5,457,769 A
`5,566,272 A
`5,602,963 A
`5,652,789 A
`5,685,000 A
`5,752,232 A
`5,774,841 A
`5,777,571 A
`5,794,204 A
`5,799,065 A
`5,805,672 A
`5,842,168 A
`5,867,574 A
`5,874,939 A
`5,878,394 A
`5,884,265 A
`5,895,447 A
`5,895,448 A
`5,924,070 A *
`6,052,666 A *
`6,119,088 A *
`6,397,186 Bl *
`
`8/1991
`10/1991
`5/1992
`11/1992
`6/1993
`11/1994
`4/1995
`9/1995
`10/1995
`10/1996
`2/1997
`7/1997
`* 11/1997
`5/1998
`6/1998
`7/1998
`8/1998
`8/1998
`9/1998
`11/1998
`2/1999
`2/1999
`3/1999
`3/1999
`4/1999
`4/1999
`7/1999
`4/2000
`9/2000
`5/2002
`
`Sakanishi et al.
`Smith et al.
`Berry et al.
`Borcherding
`Shimada
`Kero
`Knuth et al.
`Engelbeck et al.
`Valley
`Brems et al.
`Bissonnette et al.
`Miner et al.
`Cox, Jr .......................... 704/9
`Basore et al.
`Salazar et al.
`Chuang
`Miyazawa et al.
`Junqua et al.
`Barkat et al.
`Miyazawa et al.
`Eryilmaz
`Galvin
`Muhling
`Squitteri et al.
`Ittycheriah et al.
`Vysotsky et al.
`. . . . . . . . . 704/27 5
`Ittycheriah et al.
`Diehl et al. ................. 704/275
`Ciluffo ....................... 704/275
`................. 704/274
`Bush et al.
`
`OIBER PUBLICATIONS
`
`Apple Website, Dragon Systems Brings Speech Recognition
`Software to the Macintosh, Apple Computer, Inc.
`Network Briefing, Datacoms: General Magic Goes Back to
`Its Roots with New Handheld, Network Briefing, Jun. 23,
`1998.
`Audiotex Update, Unisys and Parity Create Spoken Lan(cid:173)
`guage Environment for IVR,Audiotex Update,Apr. 1, 1998.
`Network Briefing, Finance: Microsoft Invests in General
`Magic, Gets Hands on Serengeti, Network Briefing, Mar. 5,
`1998.
`Work-Group Computing Report, Voice Processing: General
`Magic and Encanto Netowrks Demonstrate Voice-Based
`E-Commerce Application, Dec. 15, 1997.
`Tele media News & Views, A Flood of Universal Mailboxes,
`Telemedia News & Views, Aug. 1, 1998.
`Business Wire, General Magic Names James McCormick
`Chief Operating Officer, Business Wire, Mar. 23, 1999.
`Business Wire, General Magic Announces Fourth Quarter
`and Year-End 1998 Results, Business Wire, Mar. 10, 1999.
`PR Newswire, NetSage Showcases Social Intelligence
`Server for Relationship Management, Pr Newswire, Feb. 11,
`1999.
`PR Newswire, NetSage Unveils Social Intelligence Server
`that Personalizes Online and Automated Voice Experiences,
`Feb. 8, 1999.
`Business Wire, Wireless Knowledge and General Magic
`Plan to Add Voice-Enabling Capabilities to Revolv, Feb. 8,
`1999.
`Computergram International, General Magic Spins Off
`Datarover Handheld Division, Computergram International,
`Oct. 30, 1998.
`
`Business Wire, General Magic Announces MagicTalk, the
`First Intelligent Voice User Interface Platform, Business
`Wire, Apr. 9, 1998.
`Computergram International, Microsoft Invests in General
`Magic, Gets Hands on Serengeti, Computergram Interna(cid:173)
`tional, Mar. 5, 1998.
`Business Wire, General Magic Announces Serengeti Field
`Test Completion and Continued Momentum Toward Next
`Milestone, Business Wire, Sep. 23, 1997.
`Network Computing, Enter The Odyssey, Network Comput(cid:173)
`ing, Jul. 15, 1997.
`Network Computing, Enter the Odyssey: Telescript mobile
`agent system, Network Computing, Jul. 1, 1997.
`PR Newswire, Xybernaut Announces New Wearable Com(cid:173)
`puter at International Conference, PR Newswire, Jun. 2,
`1998.
`PR Newswire, Voice Diary Ltd. Presents at TechVentures:
`Santa Clara, PR Newswire, May 28, 1998.
`PR Newswire, Sony Digital Products President Finalizes
`Deal to Manufacture Xybernaut's® Mobile Assistant IV®
`Wearable Computer, PR Newswire, May 13, 1998.
`Business Wire, General Magic Announces Reseller Channel
`Strategy for New Portico Virtual Assistant, Business Wire,
`May 5, 1998.
`Business Wire, Top Telecommunications Companies to Trial
`Portico Service for General Magic; Extended Trials Begin in
`Next 30 Days, Business Wire, May 5, 1998.
`Business Wire, General Magic and Encanto Networks Dem(cid:173)
`onstrate Voice-Based E-Commerce Application, Business
`Wire, Dec. 10, 1997.
`Business Wire, General Magic gives first look at future of
`mobile business communications, Business Wire, Apr. 28,
`1997.
`Business Wire, Handwriting and Speech Recognition
`Announced for Memos Open Operating Environment, Busi(cid:173)
`ness Wire, Sep. 24, 1996.
`PR Newswire, J. Robin Rohlicek Joins Purespeech as Vice
`President of Engineering, PR Newswire, Nov. 28, 1995.
`Telephone IP News, General Magic Begins Web Sign-Up
`for Virtual Assistant Service, Telephone IP News, May,
`1999.
`Audiotex Update, General Magic Acquires Netalk, Audiotex
`Update, Jan., 1999.
`Audiotex Update, "Portico" Enables Mobile Users to Access
`Info Via Phone, Audiotex Update, Oct. 1, 1998.
`Audiotex Update, ISD Gets Exclusive Rights to Embed
`Conversa Technology, Audiotex Update, Sep. 1, 1998.
`Telephone IP News, General Magic Signs PC Quote to
`Deliver Stock Market Data, Telephone IP News, Aug. 1,
`1998.
`Network Briefing, Finance: General Magic Bags Another
`$35M, Network Briefing, Jun. 26, 1998.
`Business Wire, General Magic Announces NeTalk Acquisi(cid:173)
`tion; Patent-Pending Technology to Further Advancement
`of magic Talk Platform, Business Wire, Nov. 24, 1998.
`PR Newswire, Xybernaut Corporation (NASDAQ) and JAE
`Launch New Wearable Flat Panel Display at COMDEX in
`Las Vegas, PR Newswire, Nov. 19, 1998.
`Business Wire, General Magic Announces Exclusive Agree(cid:173)
`ment with Intuit, Business Wire, Nov. 9, 1998.
`Business Wire, General Magic's DataRover Division to
`Become Independent Enterprise; General Magic will Focus
`Efforts on Expansion of the Portico Service and Develop(cid:173)
`ment of magicTalk, Business Wire, Oct. 28, 1 998.
`
`Page 2 of 43
`
`

`

`US 6,584,439 Bl
`Page 3
`
`PR Newswire, NetSage Merges with FunArts; Nass and
`Reeves of Stanford University Join with Combined Com(cid:173)
`pany to Lead Social Interface Market, PR Newswire, Oct.
`20, 1998.
`Business Wire, General Magic Launches Major Marketing
`Campaign for National Debut of Portico Service, Business
`Wire, Sep. 21, 1998.
`Business Wire, Saatchi & Saatchi Performs Magic; Business
`Wire, Jul. 6, 1998.
`Business Wire, General Magic Secures $35 Million in
`Added Financing, Business Wire, Jun. 25, 1998.
`Bernstein, Could this Be Magic?, Telephony, Apr. 6, 1998,
`vol. 234, No. 14, 1 page.
`Barchard, Natural Language Modernizes the Call Center,
`Telemarketing and Call Center Solutions, vol. 16, No. 3,
`9/97, 6 pages.
`Stevens, Are You Ready for the Desktop of the Future?,
`Datamation, vol. 40, No. 11, Jun. 1, 1994, 4 pages.
`Meisel, Straight Talk about Speech Recognition [ voice user
`interfaces], Business Communications Review, vol. 26, No.
`8, Aug., 1996, 5 pages.
`Business Wire, Generral Magic Secures Additional Financ(cid:173)
`ing; Company Obtains $20 Million to Further Voice-En(cid:173)
`abled Services, Predicasts, Mar. 31, 1991.
`PR Newswire, The Nation's First Free Voicemail Service
`Set for April Launch, PR Newswire, Mar. 31, 1991.
`Business Wire, Telephone Speech Recognition Technology
`will Impact E-commerce, Business Wire, Feb. 22, 1999.
`Business Wire, General Magic Selects Speech-Activated
`Technology from SpeechWorks; Business Wire, Jan. 19,
`1999.
`PR Newswire, Voice Control Systems' Award-Winning
`Speechware(TM) Successfully Deployed in Key Worldwide
`Accounts, PR Newswire, Jan. 14, 1999.
`Business Wire, General Magic to Demonstrate Portico-En(cid:173)
`abled Auto PC at CES; Business Wire, Jan. 7, 1999.
`Business Wire, General Magic Announces Web Sign-up for
`the Portico Virtual Assistant Service, Business Wire, Dec.
`28, 1998.
`Business Wire, General Magic Awarded New Innovator
`Award; magicTalk Honored for Excellence in Integrated
`Voice/Data Technology, Business Wire, Dec. 21, 1998.
`Business Wire, InTouch Systems Announces Inflection 3.0;
`Business Wire, Dec. 21, 1998.
`Business Wire, Telephone Speech Recognition May Grow
`Faster Than the Internet; New Summit Conference draws
`Technology Leaders, Business Wire, Dec. 19, 1998.
`
`Balentine, et al., Debouncing the Speech Button: A Sliding
`Capture Window Device for Synchronizing Turn-Taking,
`International Journal of Speech Technology 2, May 28,
`1996, _pp. 7-19 (1997), Kluwer Academic Publishers, The
`Netherlands.
`Balentine, et al., Goal-Orientation and Adaptivity in a
`Spoken Human Interface, Journal of the American Voice 1/0
`Society, San Jose, California, Feb., 1992.
`Balentine, A Multimedia Interface: Speech, Sound, Sight
`and Touch, AVIOS '94 Proceedings, American Voice Input/
`Output Society, San Jose, California, Sep., 1994.
`Zhu, et al., An Automated NC Programming system in
`integrated CAD/CAM, ICAS, Congress, 18th, Beijing,
`China, Sep. 20-25, 1992, Proceedings, vol. 2 (A93-14151
`03-01 ), American Institute of Aeronautics and Astronautics,
`Inc., Washington.
`Rolwing, Phones Heading Backward to Future, Arizona
`Republic, Feb. 5, 1999, newspaper article, Arizona.
`PR Newswire, NetSage Merges with FunArts: Nass and
`Reeves of Stanford University Join with Combined Com(cid:173)
`pany to Lead Social Interface Market, San Francisco, Cali(cid:173)
`fornia, Oct. 20, 1998.
`Automatic ID News, New Mobile Computing Company
`offers Pen-Based Terminal, 1/99, Automatic I.D. News, vol.
`15, No. 1, United States.
`Newsbytes News Network, Telecom Trials of Portico Voice
`User Interface, Newsbytes, news Network, May 6, 1998,
`United States.
`RCR Radio Communications Report, General magic unveils
`voice platform, RCR Radio Communications Report, vol.
`17, No. 15, Apr. 13, 1998, United States.
`Computergram International, General Magic Demonstrates
`New Voice Agents, Computergram International, Apr. 13,
`1999.
`Computergram International, General Magic Bags Another
`$35M, Computergram International, Jun. 26, 1998.
`Computergram International, General Magic Goes Back to
`Its Roots with New Handheld, Computergram International,
`Jun. 23, 1998.
`Military & Aerospace Electronics, Wearable Computers
`Could Revolutionize Military Maintenance and Inspection,
`Military & Aerospace Electronics, Nov., 1995, 3 pages.
`Gohring, Hello! Welcome to Portico ... , Telephony, vol.
`234, No. 20, May 18, 1995, 1 page.
`
`* cited by examiner
`
`Page 3 of 43
`
`

`

`U.S. Patent
`US. Patent
`
`Jun. 24, 2003
`Jun.24,2003
`
`Sheet 1 0f 21
`Sheet 1 of 21
`
`US 6,584,439 B1
`US 6,584,439 Bl
`
`C
`
`<(
`,-
`.
`C, -LL
`
`C
`N
`0
`
`t,
`N
`
`0 -
`
`LL..
`N
`0
`
`:r:
`N
`0
`
`(Il
`
`;; -u
`
`<
`0
`
`w
`N
`0
`
`(D
`0
`
`u -0
`
`~
`
`100
`
`0
`0
`
`u
`N
`0
`
`(Il
`N
`0
`
`0 u
`~ u
`
`w
`0
`
`< N
`
`0
`
`Page 4 of 43
`
`Page 4 of 43
`
`

`

`U.S. Patent
`
`Jun.24,2003
`
`Sheet 2 of 21
`
`US 6,584,439 Bl
`
`0
`0
`
`Lu
`N
`0
`
`0
`N
`
`(,0
`.-
`
`TELCO
`SWITCH
`
`GO
`.-
`
`COMPUTER
`
`§~
`
`'<I'"
`
`N
`
`Lu z
`0
`:I:
`a..
`Lu
`.....J
`Lu
`I-
`
`ca
`,...
`C, -u.
`
`•
`
`Page 5 of 43
`
`

`

`U.S. Patent
`US. Patent
`
`Jun. 24, 2003
`Jun.24,2003
`
`Sheet 3 0f 21
`Sheet 3 of 21
`
`US 6,584,439 B1
`US 6,584,439 Bl
`
`C\J
`
`FIG.2
`-LL
`
`N
`0
`N
`
`""'" 0
`N
`
`ti)
`0::
`ti)
`
`u
`~
`<
`
`u u
`a..
`<(
`
`N
`
`lO
`0
`N
`
`~
`N
`0
`
`:::;;
`N
`0
`
`102M
`
`c=::i
`0
`
`102J
`209
`
`
`N
`0
`N
`
`"")
`N
`0
`
`co
`0
`N
`
`u
`~
`<
`
`r--
`207
`0
`N
`
`en
`0
`N
`
`N
`202
`0
`N
`
`....J
`N
`0
`
`202
`
`u u a...
`<
`
`ti)
`0::
`ti)
`
`u <
`
`lO
`206
`0
`N
`
`""'" 0
`204
`N
`
`N
`0
`N
`
`Page 6 0f 43
`
`Page 6 of 43
`
`

`

`'° ~
`~
`
`i,-
`
`,I;;..
`~
`1J.
`O'I
`rJ'J.
`e
`
`'"""'
`N
`0 ....,
`~ ....
`'JJ. =(cid:173)~
`
`,J;..
`
`~
`
`8
`0
`N
`~,J;..
`N
`
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`IG. 3
`
`Disploy
`Option
`
`312
`
`{
`
`KEYPAD
`OPTION
`
`?
`
`311
`
`102
`
`:-----(]=
`
`310
`
`Host M Memory I
`
`Microcontroller
`
`J2§
`
`~-----
`
`ROM
`
`I '--------------,
`
`~
`
`SRAM
`
`-
`
`,,......
`r-------------L-----7
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`FL.ASH
`
`~
`
`,-.
`
`CONT
`ADDR
`
`i-----
`=-
`
`~
`
`'
`318
`
`J
`
`/
`
`I
`I
`I
`I
`I
`'
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`I
`I
`
`I
`
`..,
`315
`
`I
`r"" -"""--
`I
`I
`
`JOCS
`BMCS
`EMCS
`
`.301
`
`RESET
`Vee
`l
`
`;
`_______ ....
`
`~
`
`~
`~
`
`T
`VfC
`
`.4 ~
`-~
`
`316 -----
`
`POWER SUPPLY
`
`314--
`
`I
`
`•
`•
`
`EC!
`
`207B
`
`APPLIANCE PERIPHERALS
`
`206~
`
`MWDOUT
`MWRDY
`MWCS
`MWROST
`MWDIN
`MWCLK
`
`Yss ...
`
`~ CFSl
`CFSO
`CCLK
`COIN
`CDOUT
`
`ADDR 0-15
`D 0-15
`
`X2
`
`XI
`
`~
`
`~ t---
`.--
`,._
`>--
`207A ~
`
`EC! ~ Codel
`
`~
`
`308
`
`I
`
`105
`
`303
`
`'u· r o,d Speoke< ~ Codec
`317)
`
`L-------------~
`I
`I
`Amplifier
`,
`: ic
`Mic Preamp t+-
`----~--.I~~
`'
`202
`
`I ,Spkr
`.---
`
`304
`
`~
`
`305
`
`306
`
`,,-.....,
`
`Page 7 of 43
`
`

`

`'° ~
`~
`
`i--
`
`,I;;..
`~
`1J.
`O'I
`rJ'J.
`e
`
`'"""
`N
`0 ....,
`Ul
`~ ....
`'JJ. =(cid:173)~
`
`~
`
`8
`
`0
`N
`~,J;..
`N
`
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`301
`
`406\
`
`403
`
`r-J
`
`!_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ __ !
`,
`I
`--,
`
`RAM
`DSPM
`
`S~stem
`
`AM
`
`ROM
`
`1
`I
`
`Jl
`
`•
`
`~ 402
`
`J
`
`,.
`
`J
`
`Core Bus
`
`418-
`
`416
`
`DSPM
`
`Processing Units
`
`,,
`
`Al
`
`Core
`CR16A
`
`415/
`
`YDr
`
`' '
`
`'
`
`FIG. 4
`
`, l
`
`Jl
`
`l
`
`Controller
`MICROWIRE Configuration Generator
`
`Clock
`
`Reset and
`
`Master
`
`Interface
`MICROWIAE
`
`PWM
`
`Jl
`
`J.
`
`ICU
`
`I f +
`I
`I
`Interface
`Codec
`
`•
`434
`
`I
`
`,,
`Jl 432
`
`Jl 428 ,,4~
`
`,, 426 . (
`
`l
`
`r
`J ~ 422 H424
`
`•Ir
`
`i
`
`Peripheral Bus
`
`419)
`
`WATCHDOG
`
`436
`
`405100 ::ff -14087 -,--4017 -• -
`
`,r
`
`'
`
`Adaptor
`
`Bus
`
`"
`
`,l
`
`', ' .
`
`j l J l •
`
`' .. , ,.
`
`I l
`
`Ports
`1/0
`
`,,
`,l 420
`
`(
`
`r
`
`.L 404
`
`'
`
`••
`Control er
`Memo~
`Bus and
`
`Page 8 of 43
`
`

`

`U.S. Patent
`US. Patent
`
`Jun.24,2003
`Jun. 24, 2003
`
`Sheet 6 of 21
`Sheet 6 0f 21
`
`US 6,584,439 Bl
`US 6,584,439 B1
`
`= = ~
`.,. -0.c
`0.c <
`~
`... =
`
`~
`e.)
`
`Q e e Q u
`
`I
`)-(
`
`~
`"0
`
`.......
`•.-i:
`c,:
`
`0 u
`= 0
`u ·-"""""" ~
`~ <
`
`_..
`1""11 /
`
`,..
`
`_..
`"Ill
`
`...
`...
`
`~ -c,:
`rfisnnoc>
`
`,.Q
`c,:
`CJ
`
`"""""" =
`0 >
`
`' "
`
`I
`
`LO
`C,
`
`m.0.“—
`-LL
`
`350nowafiafiw
`5532mm<~%55:56-HD>
`
`~ >
`
`•
`
`/
`
`/
`
`(
`
`(\J
`
`,...
`Nwm
`
`I.()
`
`t--
`,...
`I{)
`
`.._,.
`(\J
`LO
`
`,...
`I{)
`LO
`
`(\J
`I.()
`LO
`
`0
`0
`I.()
`
`0 ,...
`09m
`
`I.()
`
`Now
`
`Page 9 0f 43
`
`Page 9 of 43
`
`

`

`'° ~
`~
`
`i--
`
`,I;;..
`~
`1J.
`O'I
`rJ'J.
`e
`
`'"""'
`N
`0 ....,
`-..J
`~ ....
`'JJ. =(cid:173)~
`
`~
`
`8
`0
`N
`~,J;..
`N
`
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`FIG. 6A
`
`601
`
`Return
`
`start
`
`to
`
`"Cancel access"
`
`to Start
`
`: _ messa.9!! .:
`:
`:
`help
`: playing
`:
`:
`
`I
`
`I
`
`Stop ~ Return
`600
`
`to Start
`i Play the Main 4
`Return
`r····----··---: r600
`
`:
`:
`
`: message
`: menu help
`
`600
`
`"Nevermind"
`User says
`
`start
`
`to
`
`Return
`00
`
`6
`
`start
`
`to
`
`Return
`
`factory
`Play ~600
`
`1
`
`L ____ _J
`I
`I
`I
`I
`I
`I assigned
`I
`I Play each ~
`r ----,
`L ______ I
`I
`I
`I
`I
`I
`, programmed
`:
`1
`,---
`
`<name>
`
`user
`
`<name>
`
`your names"
`
`"Tell me
`<name>
`
`"Tell me
`<name>
`
`name"
`your
`
`-7
`
`-
`
`"Help"
`
`start
`
`to
`
`Return
`
`600
`
`Fig. 6C
`
`See
`
`there?"
`
`'What is out
`<silence>
`
`to Start
`Return
`600
`
`to Start
`Return
`600
`
`word
`
`Out-of-vocabulary
`
`programmable)
`seconds(N is
`Silence of N
`
`<name>
`
`<silence>
`
`<name (optional)>
`
`,.. I Touch-to-talk
`
`pressed
`
`I
`
`Start
`
`600
`
`Page 10 of 43
`
`

`

`~
`
`8
`
`0
`N
`~,J;..
`N
`
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`---------
`
`<name>
`
`now
`
`to Start
`~ Return
`! /600
`
`name is
`Play "My
`are OK.
`Samples
`
`'
`r-------------.
`
`r--~-"~:_; --------------------
`
`"<name>"
`
`; ►j
`:
`'
`,
`:------------1
`
`repeat the
`"Please
`
`601
`
`; ___ -~~"'!: ___ ;
`
`"<name>"
`'
`' "Please say
`:-----------~1 ------
`
`the new
`
`,
`
`your name"
`
`"Change
`
`'° ~
`~
`
`i,-
`
`,I;;..
`~
`1J.
`O'I
`
`e
`
`r.,J.
`
`'"""
`N
`0 ...,
`~ ....
`'JJ. =(cid:173)~
`
`00
`
`FIG. 68
`
`:
`'"'• ---
`
`I
`
`·-------------------------
`Name change canceled."
`"Sorry, I didn't understand.
`still having problems play
`
`and Retum to Start.
`
`I
`I
`I
`I
`
`I
`
`: repeat up to 3 more times, If :
`:
`: No, samples are not OK -
`
`I
`
`'
`
`Page 11 of 43
`
`

`

`U.S. Patent
`
`Jun.24,2003
`
`Sheet 9 of 21
`
`US 6,584,439 Bl
`
`600
`
`<silence>
`"What is out
`there?"
`
`,--------------
`
`Wait random
`delay ofup to 2
`seconds, then
`emit beep, all the
`while listening
`for other devices'
`beep
`
`Beepwash~d-~~1------
`
`600
`No beep heard
`f "Youcancallme :~
`from any
`other device
`-------1•,
`► I
`I
`I
`I
`I------------- I
`
`<name>"
`
`I
`I
`I
`I
`
`Return to start
`
`any other device
`,-------------
`Wait random
`delay of up to 4
`seconds, then
`emit beep, all the
`while listening
`for other devices'
`beep
`
`Beep was heard from 1------
`
`any other device
`
`I ---------------
`
`No beep heard
`from any
`other device
`
`111,, 1
`
`600
`,-- -- ------ -- --,
`! "You can call me :~
`-------tp••
`
`<name>"
`
`I
`
`I
`
`Return to start
`
`I
`I
`I
`
`r • • - • • • • • • • • • • I
`', Wait random
`', No beep heard
`600
`: delayofupto8:
`fromany
`:--------------:
`f
`~
`seconds, then
`• other device
`: "You can call me
`:
`1
`: emit beep, all the •-------t►•:
`<name>" ~ Return to start
`: while listening
`:
`,
`: for other devices'
`:
`:
`beep
`---------------
`
`Beepwas he~d-~~~ 1 ··--- 1
`
`any other device
`r-------------
`Waitrandom
`delay ofup to 16
`seconds, then
`emit beep, all the
`while listening
`for other devices'
`beep
`
`No beep heard
`from any
`I
`I
`other device
`: "You can call me
`-------t►••
`<name>"
`
`I
`I
`I
`I
`
`,- ---- --___ . --.J:600
`
`I
`I
`:
`
`I
`
`Return to start
`
`Beep was heard from
`any other device
`
`FIG. 6C
`
`Page 12 of 43
`
`

`

`'° ~
`~
`
`i,-
`
`,I;;..
`~
`(It
`_,.a-...
`rJ"J.
`e
`
`'"""'
`N
`
`0 ...,
`'"""' 0
`~ ....
`rF.J. =(cid:173)~
`
`~
`
`~
`
`8
`
`0
`N
`~,J;..
`N
`
`~ = .....
`~ .....
`~
`•
`r:JJ.
`d •
`
`FIG. 6D
`
`to Start
`Return
`
`600
`
`to Start
`Return
`
`1
`
`I
`
`. ,_ -------.1
`' Generate DTMF
`,--------~
`600
`
`I
`
`to Start
`Return
`
`600
`
`,-------------L(
`
`I-------------·
`' "Cancelled".
`:
`'
`: Call or Dial
`:
`: Cancel the
`:
`
`and say
`
`seconds
`over 2
`
`Silence of
`
`ressed
`
`Touch-to-talk
`
`"<name>"
`
`"Hang Up"
`
`"Cancel"
`
`"Nevermind" or
`
`600
`
`, ___ ------------------_,
`,
`,
`
`is hung up.
`
`.
`
`600
`
`to Start
`Return
`600
`
`silence
`
`: _____ to call" _____ ,
`: name you want ~ "<voicetag>"
`: "Please say the
`:
`.-----------------
`
`I
`
`I
`
`'
`:
`.----------------------
`·---------------------~
`,
`,
`'
`r---------------------,
`
`"Dialing <digits>"
`
`"Calling <digits>" or
`
`--------------------_,
`,
`
`<volcetag>"
`or "Dialing
`
`:
`
`"Calling <voicetag>"
`
`silence
`
`"<voicetag>"
`
`by silence
`
`"<digits>" followed
`
`"Answer"
`
`"Hang Up"
`
`I
`
`.
`
`601
`
`t-l ----1•.if-:~~1~:~:iifi~Ji ~i:S~~:-~ t~e~~~t
`•f----;_;,:~!i..;~;,-----~ ~·:.!~,
`I
`
`The last call Is redialed and the response •--4
`
`I-•-•--•-----•-----------------------•
`,
`:
`Ill:
`:
`: "Redialing <digits>" or "Redialing <voicetag>
`:
`:
`,------------------------------------,
`: __________ played __________ :
`
`response "Nothing to redial" Is given.
`la given. If there was no earlier call, the
`
`I
`
`"Redial"
`
`Page 13 of 43
`
`

`

`~
`
`'° ~
`,I;;.. ts
`1J.
`O'I
`rJ'J.
`e
`
`i,-
`
`'"""'
`N
`
`'"""' 0 ....,
`'"""'
`~ .....
`'JJ. =(cid:173)~
`
`~
`
`~
`
`8
`0
`N
`~,J;..
`N
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`FIG. 6E
`
`to Start
`Return
`
`600
`
`to Start
`Return
`
`600
`
`See Fig. 8
`
`"Delete name"
`
`See Fig. 7
`
`"Store name"
`
`---------------------------------------·
`
`to Start
`~: Online command reverses the effect of the ~ Return
`! The microphone Is un-muted_(tumed on). The
`~---------------------------------------,
`
`Mute command.
`
`600
`
`: r
`
`I ----. -------•-•-•••--•--•-•••----------'
`
`to Start
`Return
`
`I
`I
`I
`I
`:
`'
`:
`
`t:
`: The microphone is muted. Note that since this
`i---------------------------------------~, 600
`
`party will hear the mute command.
`
`is a voice-activated command, the remote
`
`I
`I
`I
`I
`:
`
`1
`
`"Online"
`
`1
`
`"Mute"
`
`601
`
`Page 14 of 43
`
`

`

`U.S. Patent
`
`Jun.24,2003
`
`Sheet 12 of 21
`
`US 6,584,439 Bl
`
`Start
`
`600
`
`~-----------------~
`,,
`'•
`:, Optional prompt: If timeout
`:,
`NR
`•: occurs after 'Store', prompt with,:
`_____ _, :•
`'Please say the name
`11
`.._ _ __,,...._..,
`,:
`you want to store'
`::
`\'&"&'&"&."&'&'So"&':.'&-::.-:.-:.-:.-:.-:.-:.-::,,;
`
`◄--·-······-·············· ... ·· _______ _
`NR
`"Sorry, I didn't
`-----H understand. Please say
`.._ ____ _,
`the name again"
`
`NR
`
`"That sounds like
`<name>. Please say
`the name differently"
`
`NR
`
`NR
`
`No match
`
`"The names
`did not match
`Please start over"
`
`'Please say the
`number for <voicetag>"
`
`NR
`- - - - - - " ' "Please repeat 1------11~
`the number·
`
`<digits>
`
`"The number for <voicetag>
`is <digits>. Is this correct?"
`
`No
`
`>----------............................................................................. ..
`
`On second try, if still no, Start Over
`
`'The number for <voicetag>
`has been stored'
`
`Store number
`
`Return
`to
`start
`
`FIG. 7
`
`Page 15 of 43
`
`

`

`U.S. Patent
`
`Jun.24,2003
`
`Sheet 13 of 21
`
`US 6,584,439 Bl
`
`Start
`
`600
`
`1•=••••=•=••a•2sszr.
`I
`I
`I
`11
`:iocan after ·0e1ete·. prompt with
`:: Optional prompt If timeout
`.~-----------------(cid:173)
`········~·---
`NR
`
`~
`•1
`~
`,,
`
`NR
`
`·oelete name·
`
`,:
`•1
`
`"Please say the name you
`want to delete'
`
`"Sorry, I didn't
`understand. Please say
`the name again'
`
`~---------4 <voicetag>
`
`NR
`
`"That name iS
`not in the phone book
`Please start over"
`
`"Are you sure you
`want to delete <voieetag>?
`
`No
`
`600
`
`"<voicetag>
`deleted"
`
`Delete voicetag
`and number
`
`Return
`to
`start
`
`FIG. 8
`
`Page 16 of 43
`
`

`

`U.S. Patent
`
`Jun.24,2003
`
`Sheet 14 of 21
`
`US 6,584,439 Bl
`
`F:[ <question>?]
`j)
`I
`(3 seconds max)
`
`F:(Pleose answer Yes or No.)
`
`\ r
`\
`
`"Garbo~
`
`! F:(Remember to wait for the
`
`tone bere speaking
`
`r
`J
`spoke
`"--. too soon
`
`Time's Up
`
`"Yes"
`"No"
`Goodrec
`
`901
`
`Resolve
`
`/
`
`Failure
`
`Success
`
`Rejection ~ r - '
`l
`l
`
`.--------,
`Failure
`
`,--------,
`Success
`
`FIG. 9A
`
`Page 17 of 43
`
`

`

`U.S. Patent
`
`Jun.24,2003
`
`Sheet 15 of 21
`
`US 6,584,439 Bl
`
`901
`
`F:[Was that a Yes?]
`J)
`I
`
`(3 seconds max) !
`
`'
`
`"Garbage"
`
`Time's Up
`
`"Yes"
`"No"
`Goodrec
`
`r
`
`spoke
`too soon
`
`/
`
`"Yes"
`"No"
`Bodrec
`
`1st & 2nd
`responses match
`
`no match
`
`Success
`
`Failure
`
`FIG. 98
`
`Page 18 of 43
`
`

`

`'° ~
`~
`
`i,-
`
`,I;;..
`~
`1J.
`O'I
`rJ'J.
`e
`
`'"""'
`N
`0 ....,
`'"""' O'I
`~ ....
`'JJ. =(cid:173)~
`
`~
`
`~
`
`8
`
`0
`N
`~,J;..
`N
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`FIG. 10A
`
`Noise Trap D
`
`SUCCESS
`
`B
`
`silence
`
`C_ Win Timeout
`
`Wait for Event
`
`spoke-too-soon
`
`STS Counter
`
`Start Pre_Sil Timer
`
`RecognizerOn
`
`A
`
`Start C_Win Timer
`Play Prompt/Tone
`
`START
`
`3STS
`first
`
`C
`Play I Loop
`
`Buzzer
`
`STS
`
`subsequent
`
`Stop Recognition
`
`Page 19 of 43
`
`

`

`U.S. Patent
`
`Jun.24,2003
`
`Sheet 17 of 21
`
`US 6,584,439 Bl
`
`START
`
`A
`
`C
`
`Wait for Event
`
`C_ Win Timeout
`
`Check Disposition
`
`speech
`onset
`
`speech onset
`
`STS
`
`D
`
`Extend
`Window
`
`No
`speech
`
`No
`STS
`
`Stop
`Rec.
`
`Stop
`Rec.
`
`B
`
`SUCCESS
`
`E
`
`STS+Silence
`
`All Silence
`
`F
`
`FIG. 108
`
`Page 20 of 43
`
`

`

`U.S. Patent
`
`Jun.24,2003
`
`Sheet 18 of 21
`
`US 6,584,439 Bl
`
`START
`
`Save Pre_Sil
`Start Speech Timer
`
`A
`
`C Win Timeout
`
`Wait for Event
`
`C
`
`Extend
`Window
`
`speech
`timer
`decay
`
`speech
`offset
`
`speech
`offset
`
`Stop
`Rec.
`
`Stop Recognition
`Calculate Confidence
`
`D
`
`Noise Trap
`
`SUCCESS
`
`B
`
`FIG. 10C
`
`Page 21 of 43
`
`

`

`U.S. Patent
`
`Jun.24,2003
`
`Sheet 19 of 21
`
`US 6,584,439 Bl
`
`START
`
`success
`CapbJre & Recognition
`Confidence ~ - - -_ , First-Try Capture
`
`A
`
`good
`
`rejection
`
`failure
`
`Interpret First-Try &
`Change Prompt
`
`noise
`
`STS + silence
`
`silence
`
`lnteivene
`Prompt
`
`Repeat
`Prompt
`
`Solicit
`Prompt
`
`success
`
`Second-Try Capture
`
`failure
`
`B
`
`SUCCESS
`
`C
`
`D
`
`FAILURE
`
`FIG. 11
`
`Page 22 of 43
`
`

`

`'° ~
`~
`
`i,-
`
`,I;;..
`~
`1J.
`O'I
`rJ'J.
`e
`
`~ ....
`'JJ. =(cid:173)~
`
`'"""'
`N
`0 ....,
`0
`N
`
`~
`
`~
`
`8
`
`0
`N
`~,J;..
`N
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`.. •,. .... • .... ,
`
`• •,." ...... •
`
`.. •
`
`.,. .... •
`
`• ...... "',. ..
`
`...
`
`•
`
`.. •
`
`-"'"'.,. .. •
`
`.. -
`
`L
`
`Success
`
`No Selection
`
`Selected
`Menu Item
`
`Item Selected
`Recognized
`
`Selected
`This Item
`
`Yes
`
`Yes
`
`[this item)
`Yes/No
`
`No
`
`FIG. 12
`
`Yes
`
`No
`
`C
`
`(no speech)
`
`timeout
`
`Loop
`Menu
`
`Yes
`
`►<
`
`word
`
`Menu Item
`
`NEXT
`
`Response
`Analyze
`
`Query and Selection Logic
`
`B
`
`I
`
`Ii~♦ o...,...,...,+
`
`"'"Y
`
`A I
`
`)
`
`START
`
`Page 23 of 43
`
`

`

`U.S. Patent
`
`Jun.24,2003
`
`Sheet 21 of 21
`
`US 6,584,439 Bl
`
`8
`~ ,-
`
`z
`C\J
`0 ,-
`
`■
`
`C, -LL
`
`~~_[ ____ ]
`
`Page 24 of 43
`
`

`

`1
`METHOD AND APPARATUS FOR
`CONTROLLING VOICE CONTROLLED
`DEVICES
`
`US 6,584,439 Bl
`
`2
`BACKGROUND OF THE INVENTION
`
`10
`
`20
`
`Graphical user interfaces (GUis) for computers are well
`known. GUis provide an intuitive and consistent manner for
`5 human interaction with computers. Generally, once a person
`learns how to use a particular GUI, they can operate any
`computer or device which operates using the same or similar
`GUI. Examples of popular GUis are MAC OS by Apple, and
`MS Windows by Microsoft. GUis are now being ported to
`other devices. For example, the MS Windows GUI has been
`ported from computers to palm tops, personal organizers,
`and other devices so that there is a common GUI amongst a
`number of differing devices. However, as the name implies,
`GUis require at least some sort of visual or graphical display
`15 and an input device such as a keyboard, mouse, touch pad or
`touch screen. The displays and the input devices tend to
`utilize space in an device, require additional components
`and increase the costs of an device. Thus, it is desirable to
`eliminate the display and input devices from devices to save
`costs.
`Recently, voice user interfaces (VUis) have been intro(cid:173)
`duced that utilize speech recognition methods to control a
`device. However, these prior art VUis have a number of
`shortcomings that prohibit them from being universally
`25 utilized in all devices. Prior art VUis are usually difficult to
`use._ Prior art VUis usually require some sort of display
`device such as an LCD, or require a manual input device
`such as keypads or buttons, or require both a display and a
`manual input device. Additionally, prior art VUis usually are
`30 proprietary and restricted in use to a single make or model
`of hardware device, or a single type of software application.
`They usually are not widely available, unlike computer
`operating systems, and accordingly software programmers
`can not write applications that operate with the VUI in a
`35 variety of device types. Commands associated with prior art
`VUis are usually customized for that single type of device
`or software application. Prior art VUis usually have addi(cid:173)
`tional limitations in supporting multiple users such as how
`to handle personalization and security. Furthermore, prior art
`40 VUis require that a user know of the existence of the device
`in advance. Prior art VUis have not provided ways of
`determining the presence of devices. Additionally, prior art
`VUis usually require a user to read instruction manuals or
`screen displayed commands to become trained in their use.
`45 Prior art VUis usually do not include audible methods for a
`user to learn commands. Furthermore, a user may be
`required to learn how to use multiple prior art VUis when
`utilizing multiple voice controlled devices due to a lack of
`standardization.
`Generally, devices controlled by VUis continue to require
`some sort of manual control of functions. With some manual
`control required, a manual input device such as a button,
`keypad or a set of buttons or keypads is provided. To assure
`proper manual entry, a display device such as an LCD, LED,
`55 or other graphics display device may be provided. For
`example, many voice activated telephones require that tele(cid:173)
`phone numbers be stored manually. In this case a numeric
`keypad is usually provided for manual entry. An LCD is
`usually included to assure proper manual entry and to
`60 display the status of the device. A speech synthesis or voice
`feedback system may be absent from these devices. The
`addition of buttons and display devices increases the manu(cid:173)
`facturing cost of devices. It is desirable to be able to
`eliminate all manual input and display from devices in order
`to decrease costs. Furthermore, it is more convenient to
`remotely control devices without requiring specific buttons
`or displays.
`
`CROSS REFERENCE TO RELATED
`APPLICATION
`This application is also related to U.S. patent application
`Ser. No. 09/316,332, filed by inventors GEILHUFE et al,
`Attorney Docket No. 042236.P052, entitled "METHOD
`AND APPARATUS FOR STANDARD VOICE USER
`INTERFACE AND VOICE CONTROLLED DEVICES"
`and to be assigned to Information Storage Devices, Inc. the
`disclo~ure of which is hereby incorporated by reference,
`verbatim and with the same effect as though it were fully and
`completely set forth herein.
`This application is also related to U.S. patent application
`Ser. No. 09/316,604, filed by inventors GEILHUFE et al,
`Attorney Docket No. 042236.P052, entitled "METHOD
`AND APPARATUS FOR ENHANCING ACTIVATION OF
`VOICE CONTROLLED DEVICES" and to be assigned to
`Information Storage Devices, Inc. the disclosure of which is
`hereby incorporated by reference, verbatim and with the
`same effect as though it were fully and completely set forth
`herein.
`This application is also related to U.S. patent application
`Ser. No. 09/316,334, filed by inv