(12) United States Patent
`Bush et al.
`
`I 1111111111111111 11111 lllll 111111111111111 11111 11111 11111 1111111111 11111111
`US006397186Bl
`US 6,397,186 Bl
`May 28, 2002
`
`(10) Patent No.:
`(45) Date of Patent:
`
`(54) HANDS-FREE, VOICE-OPERATED REMOTE
`CONTROL TRANSMITTER
`
`(75)
`
`Inventors: William Stuart Bush, Manhattan
`Beach; Carlos Ferdinand Roura,
`Lomita, both of CA (US)
`
`(73) Assignee: Ambush Interactive, Inc., Manhattan
`Beach, CA (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by O days.
`
`(21) Appl. No.: 09/469,707
`
`(22) Filed:
`
`Dec. 22, 1999
`
`Int. Cl.7 ................................................ GlOL 21/00
`(51)
`(52) U.S. Cl. ........................................ 704/274; 704/272
`(58) Field of Search ................................. 704/200, 275,
`704/272, 231, 233, 251, 252, 255, 250
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`3/1982 Faierstain
`4,322,718 A
`4,725,827 A * 2/1988 Gallegos, Jr. et al.
`...... 340/696
`4,771,283 A * 9/1988 Imoto .................... 340/825.71
`9/1988 Rumbolt et al.
`4,774,511 A
`2/1990 Nishio
`4,905,279 A
`8/1992 Heep
`5,142,398 A
`5,199,080 A * 3/1993 Kimura et al. .............. 381/110
`7/1993 Kimura
`5,226,090 A
`9/1993 Kimura et al.
`5,247,580 A
`11/1993 Kimura
`5,267,323 A
`5,444,673 A * 8/1995 Mathurin ..................... 368/63
`9/1995 Thompson
`5,452,274 A
`11/1995 Chung et al.
`5,471,557 A
`11/1997 Pietraszak et al.
`5,691,710 A
`6/1998 Houser et al.
`5,774,859 A
`7/1998 Chuang
`5,777,571 A
`8/1998 Mozer et al.
`5,790,754 A
`12/1998 Sako
`5,852,804 A
`3/1999 Muhling
`5,878,394 A
`
`OTHER PUBLICATIONS
`
`"24AA64/24LC64 64K I2 Cim CMOS Serial EEPROM"
`published by Microchip Technology Inc. 1999, Document
`No. DS21189C, pp. 1-12.
`"CMOS Low Voltage 2Q SPST Switches ADG701/
`ADG702" published by Analog Devices, Inc. 1998, pp. 1-8.
`"CMOS Low Voltage 4Q SPDT SwitchADG719" published
`by Analog Devices, Inc. 1998, pp. 1-8.
`
`(List continued on next page.)
`
`Primary Examiner-Richemond Dorvil
`(74) Attorney, Agent, or Firm-Katten Muchin Zavis
`Rosenman
`
`(57)
`
`ABSTRACT
`
`A wireless, programmable, sound-activated and voice(cid:173)
`operated remote control transmitter can be used to add
`hands-free speech control operation to a plurality of
`remotely controlled appliances manufactured by various
`manufacturers, each of which is normally controlled with
`one or more signals from an associated remote control
`transmitter. The system may be pre-programmed with a
`universal library of codes for controlling various appliance
`categories and appliances produced by various manufactur(cid:173)
`ers within each category. The system may also be pro(cid:173)
`grammed using the controlled appliances' remote control
`transmitters and one or more operators' spoken commands.
`Once programming is complete, there is no need for the
`operator to manually operate the system, allowing true
`hands-free voice control of the remotely controlled products.
`Voice commands are organized into a plurality of linked
`recognition vocabulary sets, each representing a subset of
`the complete voice command vocabulary available. These
`subsets are structured in a fashion that is intuitive to the user
`because the structure is consistent with controlled appliance
`operation. As such, the system allows a user to easily
`navigate via voice commands between recognition sets to
`attain access to the intended voice commands.
`
`(List continued on next page.)
`
`55 Claims, 27 Drawing Sheets
`
`30
`
`50
`
`20
`
`MICROPHONE
`
`AUDIO
`SWITCHING
`CIRCUIT
`
`SPEECH
`RECOGNITION
`CIRCUIT
`
`70
`
`80
`
`INFRARED
`RECEIVER
`
`71
`
`r--3'
`
`1Q: SPEECH RECOGNIZING TRANSMITTER
`
`Page 1 of 59
`
`GOOGLE EXHIBIT 1001
`
`

`

`US 6,397,186 Bl
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`5,959,751 A
`5,983,186 A
`6,012,029 A *
`6,052,666 A *
`6,070,140 A *
`6,119,088 A
`
`9/1999
`11/1999
`1/2000
`4/2000
`5/2000
`9/2000
`
`Darbee et al.
`Miyazawa et al.
`Cirino et al.
`............... 704/275
`Diel et al.
`.................. 704/275
`Tran ........................... 704/275
`Ciluffo
`
`OIBER PUBLICATIONS
`
`"DMlO00 Component Directional Microphone" published
`by Lucent Technologies, pp. 1-2.
`
`"MACH 4 CPLD Family High Performance EE CMOS
`Programmable Logic," Publication 17466 ,published by
`Lattice Semiconductor Corporation, May 1999 pp. 1-62.
`
`"Preliminary Product Specification Z86L81/86/98 28-Pin
`Low-Voltage Infrared Microcontroller", Document No.
`DS000701-IRX1298, published by Zilog, Inc. 1999, pp.
`1-55.
`"RSC-300/364 DataBook" published by Sensory, Inc., Aug.
`1999, pp. 1-50.
`• Control",
`• Synthesis
`"RSC-300/364 Recognition
`datasheet part No. 80-0111-6, published by Sensory,
`Inc.1999, pp. 1-8.
`"Build a Voice-Activated Tape Recorder Switch", by Marc
`Spiwak, Popular Electronics magazine , Jan. 1996, pp.
`35-36, 79.
`"Voice Direct™ Speech Recognition IC", datasheet part No.
`80---0022-5, published by Sensory Inc., Sep. 8, 1998, pp.
`1-4.
`* cited by examiner
`
`Page 2 of 59
`
`

`

`'° "'---l
`~
`O'I
`rJ'J.
`e
`
`i,-
`~
`O'I
`~
`i,-
`
`-..J
`N
`
`'"""' 0 ....,
`~ .....
`'JJ. =(cid:173)~
`
`N
`0
`0
`N
`~CIO
`N
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`~
`I
`
`POWER SUPPLY CIRCUIT
`
`, " ~,
`
`I
`I
`
`10: SPEECH RECOGNIZING TRANSMITTER
`
`~
`
`71
`l
`RECEIVER
`INFRARED
`~
`
`.H
`
`-
`
`A l
`
`.il.h
`
`H.
`
`~
`
`DIODE
`
`LIGHT-EMITTING
`
`INFRARED
`rso
`
`CIRCUIT
`
`SIGNAL +
`
`CONTROLLING
`
`INFRARED
`
`[70
`
`MEMORY
`
`52
`l
`
`H,
`
`72-
`..
`... WRITE
`READ/
`, "
`I INDICATORS I
`A~l + u
`
`I KEYPAD r-
`I SPEAKER
`
`51 __.,
`
`53......,
`
`,,
`
`40
`l
`
`Jl
`
`H,.
`
`H.
`
`MICROPHONE r+ SWITCHING r+ ACTIVATION + RECOGNITION +
`
`CIRCUIT
`
`CIRCUIT
`
`SOUND
`
`AUDIO
`
`CIRCUIT
`
`SPEECH
`
`I
`
`.....
`...
`
`FIG. 1
`
`r20
`
`[30
`
`Page 3 of 59
`
`

`

`'° "'-..,l
`~
`O'I
`rJ'J.
`e
`
`i,-
`~
`O'I
`~
`i,(cid:173)
`
`-..J
`N
`0 ....,
`N
`~ ....
`'JJ. =(cid:173)~
`
`N
`0
`0
`N
`~CIO
`N
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`__ ,J
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`7
`
`64
`
`_
`
`---------I I
`
`c
`60
`
`80
`
`DIODE
`
`LIGHT-EMITTING
`
`INFRARED
`
`53
`
`71
`
`RECEIVER
`INFRARED
`
`51 1SPEAKER
`
`KEYPAD
`
`CIRCUIT
`
`CONTROLLING
`
`INFRARED SIGNAL
`
`MEMORY
`
`WRITE
`READ/
`
`72
`
`52 ........ INDICATORS
`
`54
`
`MEMORY
`
`WRITE
`READ/
`
`------·
`
`68
`
`GENERAL PURPOSE INPUT/OUTPUT
`
`69
`
`n.---.flMEMORY
`
`WRITE
`READ/
`
`SOURCE
`
`OSCILLATOR l+i--l CLOCK
`
`58
`. I ! L---------------
`~ . : ~i PREAMP H AID r ~ CONTROLLER 1 :
`I I
`I , -----.. ..---~ 51
`,
`:.,-••
`I .--------------~~E.::~~=:~6..N~T~~N-~~:~: _____ _
`--
`_____ _[_so _
`
`40
`
`CIRCUIT
`
`ACTIVATION
`
`SOUND
`
`UNIT
`
`PROCESSING
`
`SPEECH
`
`AGC
`
`59
`
`(56
`
`FIG. 2a
`
`~
`
`CIRCUIT
`
`AUDIO SWITCHING
`
`30
`
`MICROPHONE I ►I
`
`20
`
`Page 4 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 3 of 27
`
`US 6,397,186 Bl
`
`I
`
`68
`I GENERAL PURPOSE INPUT/OUTPUT
`
`I
`
`r -
`
`FIG. 2b
`
`I
`I
`I
`
`:--~;;E~~ -R~~;;N~;;;N-~I;~~;--•vso
`·-------------·
`•---------------•
`L--------
`-- --------•
`Lo
`-
`-
`;;-RA;;; S;;;l 7c;'°NT~ll~IRCUIT 7
`Slm.1
`------------------,
`Pl0 v1s
`
`MUX 0
`
`MUX 1
`
`. ..
`• • •
`------------------(cid:173)
`i----------~----------------------
`
`MUX n-1
`
`L
`
`1
`I
`I
`I
`I
`I
`I
`I
`
`.
`IP~2
`IP01,--111.-;:;;;--,---t-t
`P00
`
`I
`I
`I
`I
`I
`I
`I
`L-
`
`PIN INPUTS
`
`OSCILLATOR
`
`CLOCK
`SOURCE
`
`73
`..,_ ____ _,
`
`READ/
`WRITE
`1
`MEMORY
`
`72
`
`80
`
`IR
`LED
`
`74
`
`IR
`DRIVER
`
`INDICATOR
`
`52d
`
`INFRARED
`RECEIVER
`
`71
`
`Page 5 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 4 of 27
`
`US 6,397,186 Bl
`
`FIG. 2c
`
`30
`
`AUDIO SWITCHING
`CIRCUIT
`
`.-1
`
`I
`I
`I
`I
`I
`
`50
`
`SPEECH RECOGNITION
`CIRCUIT
`
`MODE PIN
`
`56
`
`PREAMP
`
`AID
`
`59
`
`AGC
`
`_____ £0
`
`SOUND ACTIVATION CIRCUIT
`
`INPUT/OUTPUT
`
`47Kn
`
`20
`
`,..... -
`
`1 Kn
`
`Page 6 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 5 of 27
`
`US 6,397,186 Bl
`
`12
`
`11
`
`,---__10
`
`19
`
`35
`
`FIG. 3
`
`51
`
`53_
`
`71
`
`Page 7 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 6 of 27
`
`US 6,397,186 Bl
`
`10--....__
`
`\
`16
`
`FIG.4a
`
`51 ----~--
`
`22
`
`23
`
`95
`
`80
`
`81
`
`84
`
`.__ 80
`
`81
`
`FIG.4c
`
`96·
`
`',---.,';;t 12
`:::::::==~~100
`
`----w~:•
`
`-~~~102 1~2 /--118
`I
`,
`
`116 1
`120
`
`110
`
`--~88
`
`FIG.4b
`
`Page 8 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 7 of 27
`
`US 6,397,186 Bl
`
`FIG. 5
`
`S508
`
`LISTEN FOR VOICE
`COMMANDS
`
`DETECT SOUND
`
`5503
`
`S504
`
`S505
`
`ACTIVATE SPEECH
`RECOGNTION CIRCUIT
`
`SWITCH MICROPHONE
`OUTPUT TO SPEECH
`RECOGNITION CIRCUIT
`
`ENTER
`RECOGNITION MODE
`
`---
`
`5507
`
`SET TIME OUT COUNTER
`& ERROR COUNTER= 0
`
`GENERATE PATTERN
`
`ACTIVATE INDICATOR
`
`PERFORM PATTERN
`RECOGNITION
`
`y
`
`SET TIME OUT COUNTER
`= 0 & INCREMENT
`ERROR COUNTER
`
`ACTIVATE INDICATOR
`
`S516
`
`N
`
`SELECT APPLIANCE
`CATEGORY
`S521 i.-----------,
`TRANSMIT INFRARED
`SIGNAL TO APPLIANCE
`
`Page 9 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 8 of 27
`
`US 6,397,186 Bl
`
`FIG. 6
`
`S621
`
`S622
`
`S623
`
`S624
`
`INACTIVATE / DISABLE
`UNNEEDED CIRCUITS
`
`ENTER SOUND
`ACTIVATION MODE
`
`SWITCH MICROPHONE
`OUTPUT TO SOUND
`ACTIVATION CIRCUIT
`
`LISTEN FOR SOUND
`
`N
`
`REMAIN IN
`SOUND ACTIVATION
`MODE
`
`S626
`
`Page 10 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 9 of 27
`
`US 6,397,186 Bl
`
`FIG. 7
`
`PATTERN
`QUEUING MODE
`F 14 - - - - - - -<
`
`CONTINUOUS
`LISTENING MODE
`> - - - - - - - . r D
`
`RESPONSE
`PROMPTING
`MODE
`
`Page 11 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 10 of 27
`
`US 6,397,186 Bl
`
`FIG. 8
`
`S632
`
`ACTIVATE
`PROMPTING INDICATOR
`
`LISTEN FOR VOICE
`COMMANDS
`
`S630
`
`S631
`
`ENTER RESPONSE
`PROMPTING
`RECOGNITION MODE
`
`SET TIME OUT
`COUNTER= 0, ERROR
`COUNTER = 0 & START
`TIME OUT COUNTER
`
`GENERATE PATTERN
`
`S637
`
`PERFORM PATTERN
`RECOGNITION
`
`SELECT APPLIANCE
`CATEGORY
`
`TRANSMIT INFRARED
`SIGNAL TO APPLIANCE
`
`y
`
`INCREMENT ERROR
`COUNTER
`
`ACTIVATE INDICATOR
`
`S640
`
`N
`
`Page 12 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 11 of 27
`
`US 6,397,186 Bl
`
`FIG. 9
`
`5660
`
`ENTER
`PATTERN QUEUING
`RECOGNITION MODE
`
`SET = 0 & START TIME
`OUT COUNTER
`
`SET PATTERN
`COUNTER = 0 AND
`QUEUE COUNTER = 0
`
`5661
`
`5662
`
`ACTIVATE
`PROMPTING INDICATOR
`
`LISTEN FOR VOICE
`COMMANDS
`
`GENERATE PATTERN
`
`5666
`
`QUEUE PATTERN
`IN MEMORY
`
`y
`
`PERFORM
`PATTERN
`RECOGNITION
`
`5672
`
`N
`
`QUEUE INSTRUCTION
`IN MEMORY
`
`INCREMENT QUEUE
`COUNTER BY 1
`
`5675
`
`5676
`
`DECREMENT PATTERN
`COUNTER BY 1
`
`14--.--J
`
`INCREMENT PATTERN
`COUNTER BY 1
`
`SUBMIT NEXT PATTERN
`FOR RECOGNITION
`
`5678
`
`Page 13 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 12 of 27
`
`US 6,397,186 Bl
`
`FIG. 10
`
`RETRIEVE INSTRUCTION
`FROM MEMORY
`
`SEND SIGNAL TO
`INFRARED SIGNAL
`CONTROLLING CIRCUIT
`
`DECREMENT QUEUE
`COUNTER BY 1
`
`5682
`
`5683
`
`5690
`
`- - - -L - - -
`I
`
`DETERMINE
`ASSOCIATED REMOTE
`CONTROL FUNCTION
`
`S691
`
`I
`I
`I
`I
`S692 I
`I
`I
`I
`I
`5693 I
`I
`I
`I
`S694 I
`I
`I
`I
`S695 I
`I
`I
`L ______ _J
`
`IDENTIFY MEMORY
`LOCATION OF
`CONTROL CODE
`
`GENERA TE SIGNAL FOR
`APPLIANCE
`
`SUBMIT SIGNAL TO
`INFRARED LED
`
`TRANSMIT INFRARED
`SIGNAL TO APPLIANCE
`
`Page 14 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 13 of 27
`
`US 6,397,186 Bl
`
`FIG. 11
`
`5700
`
`RETRIEVE INSTRUCTION
`FROM MEMORY
`
`- - - - - - - - - - , . - - - - - - - -......
`
`SEND SIGNAL TO
`INFRARED SIGNAL
`CONTROLLING CIRCUIT
`
`IDENTIFY TARGET
`APPLIANCE
`DEVICE CATEGORY
`
`y
`
`EXECUTE TIME DELAY
`
`5702
`
`N
`
`,___ _____ D
`
`-,
`
`N
`
`DETERMINE
`ASSOCIATED REMOTE
`CONTROL FUNCTION
`
`IDENTIFY MEMORY
`LOCATION OF
`CONTROL CODE
`
`GENERATE SIGNAL FOR
`APPLIANCE
`
`SUBMIT SIGNAL TO
`INFRARED LED
`
`TRANSMIT INFRARED
`SIGNAL TO APPLIANCE
`
`5713
`
`5714
`
`S715
`
`5716
`
`5717
`
`r
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`L
`
`Page 15 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 14 of 27
`
`US 6,397,186 Bl
`
`FIG. 12a
`
`S202
`
`DISCONNECT
`POWER SOURCE
`
`S203
`
`EXECUTE RESET
`& START TIME OUT
`COUNTER
`
`DEPRESS
`SETUP BUTTON
`
`r A~IT -;-TT-;;;
`
`SELECTION
`
`-
`
`-
`
`1rs2os
`
`V
`
`vi
`I
`I
`vi
`
`DEPRESS
`CODE BUTTON
`
`DEPRESS LEARN
`SIGNAL BUTTON
`
`DEPRESS NAME
`COMMAND BUTTON
`
`DEPRESS
`MACRO BUTTON
`
`N
`
`N
`
`Page 16 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 15 of 27
`
`US 6,397,186 Bl
`
`FIG. 12b
`
`M
`
`-
`
`-
`-
`....Ls~ -
`-
`-
`-
`-
`APPLIANCE CATEGORY SELECTION ROUTINE
`
`r -
`
`S221
`
`S228
`
`5229
`
`S232
`
`5233
`
`GENERATE PATTERN
`
`PERFORM PATTERN
`RECOGNITION
`
`IDENTIFY APPLIANCE
`CATEGORY NAME
`RECOGNIZED
`
`SEND INSTRUCTION TO
`IDENTIFY APPLIANCE
`CATEGORY TYPE
`
`IDENTIFY ASSOCIATED
`MEMORY SECTION
`
`ENTER RESPONSE
`PROMPTING
`RECOGNITION MODE
`
`ACTIVATE APPLIANCE
`CATEGORY NAME
`RECOGNITION SET
`
`SET= 0 & START
`RECOGNITION
`TIME OUT COUNTER
`
`ACTIVATE
`PROMPTING INDICATOR
`
`LISTEN FOR VOICE
`COMMAND
`
`S222
`
`S223
`
`S224
`
`S225
`
`N
`
`I
`I
`I
`L _____ _ __ J
`
`y
`
`Page 17 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 16 of 27
`
`US 6,397,186 Bl
`
`FIG. 13a
`
`GENERATE PATTERN
`
`PERFORM PATTERN
`RECOGNITION
`
`IDENTIFY APPLIANCE
`CATEGORY NAME
`RECOGNIZED
`
`SEND INSTRUCTION TO
`IDENTIFY APPLIANCE
`CATEGORY TYPE
`
`IDENTIFY ASSOCIATED
`MEMORY SECTION
`
`S240
`
`S241
`
`S242
`
`S243
`
`S244
`
`S245
`
`S246
`
`SEND INSTRUCTION TO
`ACTIVATE CODE
`SELECTION PROGRAM
`
`SET= 0 & START
`PROGRAMMING
`TIME OUT COUNTER
`
`ENTER RESPONSE
`PROMPTING
`RECOGNITION MODE
`
`ACTIVATE APPLIANCE
`CATEGORY NAME
`RECOGNITION SET
`
`SET= 0 & START
`RECOGNITION
`TIME OUT COUNTER
`
`ACTIVATE
`PROMPTING INDICATOR
`
`LISTEN FOR VOICE
`COMMAND
`
`S249
`
`S250
`
`S253
`
`S254
`
`SET= 0 & START
`PROGRAMMING
`TIME OUT COUNTER
`
`ENTER
`PATTERN QUEUING
`RECOGNITION MODE
`
`ACTIVATE DIGITS
`RECOGNITION SET
`
`S257
`
`S258
`
`Page 18 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 17 of 27
`
`US 6,397,186 Bl
`
`FIG. 13b
`
`S260
`
`S261
`
`SET= 0 & START
`RECOGNITION
`TIME OUT COUNTER
`
`SET PATTERN
`COUNTER= 0 AND
`QUEUE COUNTER = 0
`
`ACTIVATE
`PROMPTING INDICATOR
`
`LISTEN FOR VOICE
`COMMANDS
`
`GENERATE PATTERN
`
`S266
`
`QUEUE PATTERN
`IN MEMORY
`
`y
`
`PERFORM
`PATTERN
`RECOGNITION
`
`S270
`
`N
`
`QUEUE INSTRUCTION
`IN MEMORY
`
`INCREMENT QUEUE
`COUNTER BY 1
`
`S273
`
`S274
`
`DECREMENT PATTERN
`COUNTER BY 1
`
`INCREMENT PATTERN
`COUNTER BY 1
`
`SUBMIT NEXT PATTERN
`FOR RECOGNITION
`
`S276
`
`Page 19 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 18 of 27
`
`US 6,397,186 Bl
`
`FIG. 13c
`
`5281
`
`IDENTIFY APPLIANCE
`CODE RECOGNIZED
`
`RETRIEVE INSTRUCTION
`FROM MEMORY
`
`SEND INSTRUCTION TO
`INFRARED SIGNAL
`CONTROLLING CIRCUIT
`
`r ___ .c:::_ __ _
`5285
`I
`5286
`
`1
`
`DETERMINE IDENTIFIER
`ASSOCIATED WITH
`INSTRUCTION
`
`DECREMENT QUEUE
`COUNTER BY 1
`
`STORE IDENTIFIER
`IN MEMORY
`
`5287
`
`5283
`
`5284
`
`_ _J
`
`N
`
`CODE NOT ACCEPTED
`
`ASSIGN APPLIANCE
`CODE TO APPLIANCE
`CATEGORY
`
`EXIT
`PROGRAMMING MODE
`
`5293
`
`5294
`
`r -:90-
`1
`I
`I
`I
`I
`
`5291
`
`ACTIVATE INDICATOR
`
`5292
`ALLOW TIME OUT
`OF PROGRAMMING
`COUNTER
`
`~---_-_-_-_-
`
`....._ _____ D
`
`Page 20 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 19 of 27
`
`US 6,397,186 Bl
`
`FIG. 14a
`
`S300
`
`S301
`
`S302
`
`S303
`
`S304
`
`S305
`
`S306
`
`SEND INSTRUCTION TO
`ACTIVATE SIGNAL
`LEARNING PROGRAM
`
`SET= 0 & START
`PROGRAMMING
`TIME OUT COUNTER
`
`ENTER RESPONSE
`PROMPTING
`RECOGNITION MODE
`
`ACTIVATE APPLIANCE
`CATEGORY NAME
`RECOGNITION SET
`
`SET= 0 & START
`RECOGNITION
`TIME OUT COUNTER
`
`ACTIVATE
`PROMPTING INDICATOR
`
`LISTEN FOR VOICE
`COMMAND
`
`S309
`
`S310
`
`S313
`
`S314
`
`S315
`
`GENERATE PATTERN
`
`PERFORM PATTERN
`RECOGNITION
`
`IDENTIFY APPLIANCE
`CATEGORY NAME
`RECOGNIZED
`
`SEND INSTRUCTION TO
`IDENTIFY APPLIANCE
`CATEGORY TYPE
`
`IDENTIFY ASSOCIATED
`MEMORY SECTION
`
`SEND INSTRUCTION TO
`IDENTIFY SIGNAL
`LEARNING CODE
`
`SET= 0 & START
`PROGRAMMING
`TIME OUT COUNTER
`
`Page 21 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 20 of 27
`
`US 6,397,186 Bl
`
`FIG. 14b
`
`S320
`
`S321
`
`S322
`
`S323
`
`ENTER RECOGNITION
`MODE & ACTIVATE
`RECOGNITION SET
`
`SET= 0 & START
`RECOGNITION TIME OUT
`COUNTER
`
`ACTIVATE INDICATOR
`
`LISTEN FOR VOICE
`COMMANDS
`
`GENERATE PATTERN
`
`PERFORM PATTERN
`RECOGNITION
`
`S327
`
`N
`
`y
`
`S329
`
`IDENTIFY ASSOCIATED
`INSTRUCTION
`
`SEND INSTRUCTION
`TO IDENTIFY
`FUNCTION TYPE
`
`IDENTIFY MEMORY
`LOCATION TO STORE
`LEARNED SIGNAL
`
`ACTIVATE INDICATOR
`TO PROMPT USER TO
`TRANSMIT SIGNAL
`
`RECEIVE SIGNAL,
`FORMAT INTO CODE &
`STORE IN MEMORY
`
`5332
`
`5333
`
`S334
`
`5335
`
`ACTIVATE INDICATOR
`TO INDICATE RESULTS
`OF LEARNING ATTEMPT
`
`Page 22 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 21 of 27
`
`US 6,397,186 Bl
`
`FIG. 15
`
`S340
`
`ACTIVATE VOICE
`TRAINING PROGRAM
`
`SET= 0 & START
`TRAINING PROGRAM
`TIME OUT COUNTER
`
`ACTIVATE
`PROMPTING INDICATOR
`
`S341
`
`S342
`
`LISTEN FOR VOICE
`COMMAND
`
`GENERATE PATTERN
`
`GENERATE
`REFERENCE PATTERN
`
`N
`
`S354
`
`y
`
`PROMPT USER TO
`CHANGE VOICE
`COMMAND NAME
`
`STORE REFERENCE
`PATTERN IN MEMORY
`
`S348
`
`STORE PATTERN
`IN MEMORY
`
`PROMPT USER TO
`REPEAT VOICE
`COMMAND
`
`S349
`
`ALLOW TIME OUT OF
`TRAINING PROGRAM
`COUNTER
`
`Page 23 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 22 of 27
`
`US 6,397,186 Bl
`
`FIG. 16a
`
`S360
`
`S361
`
`ACTIVATE MACRO
`PROGRAMMING MODE
`
`EXECUTE APPLIANCE
`CATEGORY SELECTION
`ROUTINE
`
`ENTER
`SPEAKER DEPENDENT
`RECOGNITION MODE
`
`ACTIVATE SPEAKER
`DEPENDENT
`RECOGNITION SET
`
`SET= 0 & START
`RECOGNITION
`TIME OUT COUNTER
`
`ACTIVATE
`PROMPTING INDICATOR
`
`LISTEN FOR VOICE
`COMMAND
`
`S364
`
`S365
`
`S366
`
`S367
`
`GENERATE PATTERN
`
`PERFORM PATTERN
`RECOGNITION
`
`ACTIVATE INDICATOR
`
`IDENTIFY ASSOCIATED
`MEMORY LOCATION TO
`STORE INSTRUCTIONS
`
`ENTER DEFAULT
`RECOGNITION MODE
`
`S371
`
`y
`
`S373
`
`S374
`
`S375
`
`Page 24 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 23 of 27
`
`US 6,397,186 Bl
`
`5380
`
`5382
`
`5383
`
`S384
`
`FIG. 16b
`
`ENTER
`RECOGNITION MODE
`
`ACTIVATE
`RECOGNITION SET
`
`SET= 0 & START
`RECOGNITION TIME OUT
`COUNTER
`
`ACTIVATE
`PROMPTING INDICATOR
`
`LISTEN FOR VOICE
`COMMANDS
`
`GENERATE PATTERN
`
`5388
`
`PERFORM PATTERN
`RECOGNITION
`
`ACTIVATE INDICATOR
`
`STORE ASSOCIATED
`INSTRUCTION
`IN MEMORY
`
`STORE END MARKER
`INSTRUCTION
`IN MEMORY
`
`S393
`
`STORE INSTRUCTION
`FOR TIME DELAY IN
`MEMORY
`
`EXIT MACRO
`PROGRAMMING MODE
`
`Page 25 of 59
`
`

`

`U.S. Patent
`
`May 28, 2002
`
`Sheet 24 of 27
`
`US 6,397,186 Bl
`
`FIG. 17
`
`/51
`
`SPST SWITCH
`
`93
`
`1 Kn
`
`·--1
`
`I
`I
`I
`I
`I
`
`----o Vee
`92
`
`91
`
`1
`
`POWER DISTRIBUTION PLANE
`
`94
`
`V cc CONNECTIONS
`
`55
`
`SPEECH RECOGNITION
`CIRCUIT IC
`
`I(cid:173)
`::,
`Q.
`!5
`68 s ::::,
`
`Q. z
`
`Page 26 of 59
`
`

`

`'° "'-..,l
`~
`O'I
`rJ'J.
`e
`
`i,-
`~
`O'I
`~
`i,(cid:173)
`
`-..J
`N
`0 ....,
`Ul
`N
`~ .....
`'JJ. =(cid:173)~
`
`N
`0
`0
`N
`~CIO
`N
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`-?
`
`j~
`
`DIODE
`
`LIGHT-EMITTING
`
`INFRARED
`,so
`
`CONTROLLING f+
`
`SIGNAL
`
`.~, ..
`
`J ..
`
`CIRCUIT
`
`INFRARED
`
`r10
`
`l-+ SWITCHING ~ ACTIVATION f+ RECOGNITION f+
`
`CIRCUIT
`
`SPEECH
`
`... ...
`
`50
`
`FIG. 18a
`
`I SPEAKER I
`1a
`
`,,.
`
`J ..
`
`CIRCUIT
`
`SOUND
`
`30
`l
`CIRCUIT
`
`...
`
`AUDIO
`
`...
`
`~l
`
`MICROPHONE
`
`20
`
`71
`l
`RECEIVER
`INFRARED
`~
`
`J II,
`
`~
`I
`
`POWER SUPPLY CIRCUIT
`
`1 Ir
`
`1 Ir
`
`r 10: SPEECH RECOGNIZING TRANSMITTER
`I
`I
`
`~
`
`12-~
`MEMORY
`...
`... WRITE
`READ/
`
`,...._
`
`1 Ir
`
`52
`I
`
`J~
`
`51-'
`r KEYPAD ~
`53-
`
`I
`
`I INDICATORS I
`
`J ... l • 1,
`
`Page 27 of 59
`
`

`

`'° "'-..,l
`~
`O'I
`rJ'J.
`e
`
`i,-
`~
`O'I
`~
`i,(cid:173)
`
`~ ....
`'JJ. =(cid:173)~
`
`-..J
`N
`0 ....,
`O'I
`N
`
`N
`0
`0
`N
`~CIO
`N
`'-<
`~
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`~
`
`71
`l
`RECEIVER
`INFRARED
`~
`
`.~
`
`LIGHT-EMITTING
`
`DIODE .~
`
`INFRARED
`,so
`
`I INDICATORS
`HJ l + ~ .
`.~
`CONTROLLING ....
`
`MICROPHONE B + ACTIVATION + RECOGNITION
`
`CIRCUIT
`
`SPEECH
`
`CIRCUIT
`
`SOUND
`
`.~
`
`20b
`l
`
`Jl
`
`• l
`
`.AJ~
`
`CIRCUIT
`
`SIGNAL
`
`INFRARED
`
`....
`
`~
`
`POWER SUPPLY CIRCUIT
`
`, . ~ .
`
`r 10: SPEECH RECOGNIZING TRANSMITTER
`I
`
`___jf
`
`MEMORY
`
`....
`~ WRITE
`READ/
`
`72-
`
`-
`
`~,
`
`-
`
`52
`l
`
`H
`
`I KEYPAD ~
`I SPEAKER
`~ .
`
`40
`l
`
`I
`
`51-
`
`53-
`
`,10
`
`~ ....
`
`50
`
`MICROPHONE A
`
`20
`
`.
`
`FIG. 18b
`
`Page 28 of 59
`
`

`

`'° "'-..,l
`~
`O'I
`rJ'J.
`e
`
`i,-
`~
`O'I
`~
`i,(cid:173)
`
`~ ....
`'JJ. =(cid:173)~
`
`-..J
`N
`0 ....,
`-..J
`N
`
`N
`0
`0
`N
`00
`N
`'-<
`~
`~
`
`~
`
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`71
`l
`RECEIVER
`INFRARED
`~
`
`H,
`
`....
`LIGHT-EMITTING
`
`DIODE
`
`~
`
`INFRARED
`rso
`
`----
`
`~ r
`
`a~L
`
`I INDICATORS I
`....
`CIRCUIT
`
`~~L • , ..
`
`52
`l
`
`.a
`
`10: SPEECH RECOGNIZING TRANSMITTER
`
`r ~
`
`90
`
`POWER SUPPLY CIRCUIT
`
`I
`
`H
`
`~,
`
`I
`
`l
`I
`
`MEMORY
`..
`... WRITE
`READ/
`
`72-
`
`51-'
`
`I KEYPAD ~
`
`CONTROLLING
`
`SIGNAL
`
`INFRARED
`
`110
`
`f+ RECOGNITION ~
`
`CIRCUIT
`
`53-
`
`I SPEAKER
`
`~,
`
`40
`.... 7
`
`CIRCUIT
`
`ACTIVATION
`
`MICROPHONE B ~ SWITCHING ~
`
`30
`l
`CIRCUIT
`
`a
`
`20b
`l
`
`~ ..
`
`SPEECH
`
`.I
`
`~
`
`..
`
`FIG. 18c
`
`SOUND
`
`AUDIO
`
`+
`
`MICROPHONE A
`
`...
`
`20
`
`Page 29 of 59
`
`

`

`US 6,397,186 Bl
`
`1
`HANDS-FREE, VOICE-OPERATED REMOTE
`CONTROL TRANSMITTER
`BACKGROUND OF THE INVENTION
`1. Field of Invention
`This invention relates to devices for remotely controlling
`electronic equipment, and more particularly, to a wireless,
`user-programmable, voice-activated and voice-operated
`remote control system for controlling appliances.
`2. Description of Prior Art
`Historically, appliances, for example, electronic
`appliances, such as, televisions, VCRs, digital satellite
`systems, audio systems, and related accessories, have been
`remotely controlled by hand-held transmitters used to gen(cid:173)
`erate signals to receivers incorporated into the electronics of 15
`the remotely controlled appliances. Signals for such appli(cid:173)
`ances correspond to control commands, such as channel
`selection/tuning, power on/off, audio volume adjustment,
`and muting controls, typically generated by the user by
`depressing buttons on a remote control transmitter keypad. 20
`The basic composition and operation of such remote control
`systems are well known in the art.
`For convenience, such remote control transmitters are
`generally designed to be sufficiently small to be hand-held
`during operation. There are several disadvantages in using
`such hand-held transmitters. For example, the small size and
`mobility often contribute to misplacement or loss of the
`transmitter. Also, for device operators with restricted physi-
`cal mobility or sight limitations, hand-held remote controls
`may not provide sufficient access to the command controls
`of the remotely controlled appliances. Also, if an operator's
`hands are engaged in an activity, an interruption in the
`activity may be required to operate the hand-held remote
`control, causing inconvenience to the operator and poten(cid:173)
`tially having an adverse effect on productivity.
`Additional problems occur as a consumer acquires vari(cid:173)
`ous remotely controlled appliances, where each is typically
`supplied with a separate remote control transmitter. As the
`number of separate remote control transmitters increases,
`locating, distinguishing, and locating the appropriate trans(cid:173)
`mitters becomes increasingly difficult. In response to this
`problem, universal remote control transmitters, pre(cid:173)
`programmed by the manufacturers with control commands,
`typically coded infrared signals, have been developed to
`operate many different remotely controlled electronic 45
`appliances, for example, as disclosed in U.S. Pat. No.
`4,774,511. In addition to universal remote control
`transmitters, learning remote control transmitters have been
`developed which receive the control command signals,
`typically infrared codes, from the remote control transmit- so
`ters provided with remotely controlled appliances and gen(cid:173)
`erally store the frequencies and pulses into memory, with the
`signals becoming associated with buttons located on the
`keypad of the learning remote control transmitter. After
`programming is complete, depressing the keypad buttons ss
`prompts the learning remote control transmitter to
`re-transmit the codes stored within its memory. This allows
`the user to consolidate the control of several remotely
`controllable appliances into a single hand-held remote con(cid:173)
`trol transmitter. An example of such a system is disclosed in 60
`U.S. Pat. No. 5,142,398. For additional convenience to the
`user, the learning capability has been combined with uni(cid:173)
`versal remote control transmitters which are pre(cid:173)
`programmed by the manufacturers with control commands
`necessary to operate many different remotely controlled 65
`electronic appliances, for example as disclosed in U.S. Pat.
`No. 5,691,710.
`
`2
`All of the above-mentioned systems require the user to
`establish physical contact, typically in the form of manually
`depressing keypad buttons, to transmit a control command
`to the remotely controlled appliance. As such, the above-
`s mentioned systems due to their small size are often mis(cid:173)
`placed causing frustration to the user.
`As such, voice-operated remote control systems have
`recently been developed which operate in response to user(cid:173)
`spoken commands instead of commands selected by manu-
`10 ally depressing keypad buttons. U.S. Pat. Nos. 5,199,080;
`5,247,580; 5,267,323 and 5,226,090 disclose various
`embodiments of voice-operated remote control systems
`which employ voice control commands instead of control
`commands entered through buttons on a keypad.
`Unfortunately, such systems are not truly hands-free, requir(cid:173)
`ing manual intervention by the user during use. In particular,
`such remote control systems as disclosed in the above(cid:173)
`mentioned patents, are all based upon the use of a "talk
`switch"; which must be manually depressed to enter a voice
`command when the transmission of a remote control signal
`is desired.
`U.S. Pat. No. 5,226,090 further discloses a voice-operated
`remote control system which contains a detector for detect(cid:173)
`ing whether a voice command is received or not. A detected
`25 signal from the detector is applied to a power supply circuit.
`When there is no voice command received as detected by the
`detector, the power supply control circuit cuts off electric
`energy supplied to a speech recognition circuit. The voice
`command detector circuit is implemented by a detecting
`30 program stored in the ROM in a controller. The detecting
`program determines whether the talk switch has been
`depressed. The transmitter operates depending on whether
`the talk switch has been depressed. If the talk switch has
`been depressed, the transmitter is enabled to remote control
`35 signals. Once the talk switch is released, the transmitter is
`kept in a low power consumption mode, waiting for voice
`commands to be applied. As indicated above, the means for
`generating and transmitting a remote control signal based on
`the recognized spoken voice command is not hands-free,
`40 requiring the manual intervention of pressing a talk switch
`to accomplish these functions.
`Various other systems are known which use speech rec(cid:173)
`ognition to control appliances. For example, U.S. Pat. No.
`5,852,804 discloses a system for controlling several external
`appliances connected to a speech recognition system.
`However, the system disclosed in the '804 patent requires
`physical interconnections between the control system and
`the appliance which makes it difficult for a user to add
`additional appliances or change controlled appliances.
`U.S. Pat. No. 5,878,394 discloses a system which includes
`connections to a remote control network for transmitting
`infrared codes and a graphical user interface on a personal
`computer (PC). In addition to the inconvenience of estab(cid:173)
`lishing the remote control network, a user must have access
`to a PC and desire to make the PC available for such control
`functions. The system disclosed in the '394 patent has only
`limited utility for consumers who have access to a PC or can
`afford the expense of a PC and thus excludes a significant
`portion of consumers who may otherwise desire to add
`speech recognition control to their electronic appliances if
`an inexpensive alternative is available.
`U.S. Pat. No. 5,774,859 discloses a subscriber informa(cid:173)
`tion system with a speech interface, similar to the system
`disclosed in U.S. Pat. No. 5,878,394 the system in the '859
`patent is based upon a PC-class processor. Operation of this
`system is dependent upon receiving information from an
`
`Page 30 of 59
`
`

`

`US 6,397,186 Bl
`
`4
`ent if an appliance has the ability to process a series of
`remote control signals arriving within a predetermined time
`of each other differently than if the codes arrive at intervals
`outside of this predetermined time. For example, many
`televisions are capable of receiving the remote control signal
`transmissions for the digits one, two and three in rapid
`succession and responding by switching to channel 123
`instead of changing to channel 1, then to channel 2 and then
`to channel 3. As such, a user must be able to quickly submit
`10 a voice command following the recognition of a prior voice
`command, and then the recognition of the latter voice
`command, the submission of a remote control signal and the
`detection of the signal by the appliance must occur within
`the time limit if the appliance is to interpret the commands
`15 as a related series of commands instead of individual com-
`mands.
`Thus, there is a need for a stand-alone, wireless remote
`control that can be positioned in a location remote to both
`the user and the controlled appliances, which can be acti(cid:173)
`vated and operated entirely via voice commands, conve(cid:173)
`niently compatible with commercially available remote con(cid:173)
`trolled appliances, and can be operated by a plurality of
`users and can be programmed by a user to operate in a
`customized manner.
`
`5
`
`3
`information distribution center or head-end installation and
`therefore lacks the advantages of a stand-alone device.
`Unfortunately, the remote control is not capable of indepen(cid:173)
`dently completing the speech recognition process and trans(cid:173)
`mitting infrared signals indicated by such recognition results
`to controlled appliances. Furthermore, this system also
`requires a PC-class processor.
`Inconvenience is another problem that occurs as a user
`intends to perform multiple control functions involving one
`or more controlled appliances and must recall and enter a
`series of voice commands to a voice-operated remote control
`system. U.S. Pat. No. 5,852,804 discloses a system, where
`one speech command may be defined which is capable of
`manipulation of more than one connected appliance.
`However, this system requires that each controlled appliance
`be physically connected to the system and that the controlled
`appliance be manually manipulated during setup for each
`function to be executed upon recognition of the speech
`command. Also, if a single command is to be used to execute
`a series of controlled appliance functions, its capability is 20
`limited unless time delays of variable lengths may be
`inserted within the series of control functions to accommo(cid:173)
`date the initialization periods for certain controlled appli(cid:173)
`ances. For example, if a voice command sends signals to a
`television to power it on and turn to a specified channel, a 25
`period of time may elapse between the receipt of the power
`on command until the television warms up and is ready to
`receive additional commands. Such initialization periods
`vary from appliance to appliance and therefore means is
`needed to allow a user to adjust the time delay as necessary. 30
`There are various other problems with known systems
`which utilize voice recognition to control appliances. For
`example, one problem occurs as the list of voice commands
`grows. Generally, each time a speech recognition product
`detects a spoken word, it compares the word to those in a 35
`recognition vocabulary. In general, as a recognition vocabu(cid:173)
`lary increases in size, recognition time increases and accu(cid:173)
`racy decreases. The more words in a recognition vocabulary,
`the more likely the speech recognition circuit will make a
`mistake and accuracy generally diminishes. Without the 40
`capability to utilize multiple vocabulary recognition sets, the
`commands available for selection must all be available at the
`same time, meaning they must all reside in the same
`recognition vocabulary set. Thus, the user may be inconve(cid:173)
`nienced by the limited command availability of a small
`recognition vocabulary or the diminishing speech recogni(cid:173)
`tion accuracy and longer recognition times resulting as the
`single recognition vocabulary is increased in size. Another
`disadvantage of a single recognition vocabulary is that the
`recognition of voice command always yields the same
`response. In a multiple recognition vocabulary set scheme,
`the same voice command may yield different results upon
`recognition if placed within different vocabulary recognition
`sets. A still further disadvantage of a single recognition
`vocabulary is that the recognition vocabulary words must be 55
`phonetica