(12) United States Patent
`Matulich et al.
`
`I 1111111111111111 11111 111111111111111 1111111111 1111111111 1111111111 11111111
`US006188986Bl
`US 6,188,986 Bl
`*Feb.13,2001
`
`(10) Patent No.:
`(45) Date of Patent:
`
`(54) VOICE ACTIVATED SWITCH METHOD AND
`APPARATUS
`
`(75)
`
`Inventors: Richard Matulich; Allan Ligi, both of
`Poway, CA (US)
`
`(73) Assignee: VOS Systems, Inc., San Diego, CA
`(US)
`
`( *) Notice:
`
`This patent issued on a continued pros(cid:173)
`ecution application filed under 37 CFR
`1.53( d), and is subject to the twenty year
`patent term provisions of 35 U.S.C.
`154(a)(2).
`
`Under 35 U.S.C. 154(b), the term of this
`patent shall be extended for 0 days.
`
`(21) Appl. No.: 09/133,724
`Aug. 13, 1998
`(22) Filed:
`
`Related U.S. Application Data
`
`( 63) Continuation-in-part of application No. 09/002,436, filed on
`Jan. 2, 1998, now abandoned.
`Int. Cl.7 ..................................................... GlOL 15/00
`(51)
`(52) U.S. Cl. ............................................. 704/275; 704/275
`(58) Field of Search ..................................... 704/233, 225,
`704/270-275; 381/107-110; 375/260
`References Cited
`U.S. PATENT DOCUMENTS
`
`(56)
`
`H891
`3,555,192 *
`3,818,481
`4,052,568
`4,119,797
`4,433,435 *
`4,829,576
`4,843,627
`4,912,766
`5,086,385 *
`5,199,080
`5,351,272 *
`5,430,826
`5,488,273
`5,493,618
`
`2/1991
`1/1971
`6/1974
`* 10/1977
`10/1978
`2/1984
`5/1989
`6/1989
`3/1990
`2/1992
`3/1993
`9/1994
`7/1995
`1/1996
`2/1996
`
`Hashimoto.
`Hymer ................................. 381/110
`Dorfman et al. .
`Jankowski ... .... ... ... ... ... ... .... .. 704/233
`Wollert.
`David ................................... 381/110
`Porter.
`Stebbins.
`Forse .
`Launey ................................. 704/275
`Kimura et al..
`Abraham .............................. 375/260
`Webster et al. .
`Chang.
`Stevens et al. .
`
`7/1997 Andreshak et al. .
`5,644,061
`8/1998 Mozer et al. .
`5,790,754
`FOREIGN PATENT DOCUMENTS
`
`28 40 132 Al
`297 13 054
`U1
`297 18 636
`U1
`
`3/1980 (DE) .
`
`12/1997 (DE) .
`
`3/1998 (DE) .
`OTHER PUBLICATIONS
`Sensory, Inc., RSC-164 Data Sheet, pp. 1-8, 1996.*
`Lereboullet, Voice Recognition Processors (DATABASE),
`pp. 1-12, Nov. 1996.*
`Sensory, Inc., Voice Direct™, Speech Recognition IC, pp.
`1-4, 1996.*
`Beranek, Leo, Acoustics, McGraw-Hill Book Company,
`Inc., pp. 338, 407, 414, 415 and 419, 1954.*
`Printed Excerpts of Images Company's Web Page [http://
`www.imagesco.com/]-Speech
`Recognition
`Circuit;
`Speaker Dependent/Speaker Independent; Programming the
`HM2007; Printed Dec. 4, 1997.
`Printed Excerpts of Voice Connexion, Inc.'s Web Page
`[http://www.voicecnx.com/]-Intro Voice Pro-Module;
`Printed Dec. 4, 1997.
`Printed Excerpts of Sensory, Inc.'s. Web Page [http://ww(cid:173)
`w.sensoryinc.com]-RSC-164 Datasheet; Printed Nov. 19,
`1997.
`* cited by examiner
`Primary Examiner-David D. Knepper
`(74) Attorney, Agent, or Firm-Brown, Martin, Haller &
`McClain
`(57)
`
`ABSTRACT
`
`A voice activated device for producing control signals in
`response to speech is self-contained and requires no addi(cid:173)
`tional software or hardware. The device may be incorporated
`into a housing that replaces a wall switch that is connected
`to an AC circuit. An alternate housing is portable and
`includes a jack that plugs into and lies flush against a
`standard AC utility outlet, and at least one plug for accepting
`an AC jack of any electronic product or appliance. The
`device acts as a control interface between utility power and
`connected electrical devices by connecting or disconnecting
`power to the electrical devices based on speech commands.
`
`25 Claims, 11 Drawing Sheets
`
`MICRO(cid:173)
`
`CONTROLLER ~---___, c8J'JJk~0
`
`MEMORY
`
`14
`
`22
`
`ANALOG
`DC POWER
`SUPPLY
`
`26
`
`Page 1 of 21
`
`GOOGLE EXHIBIT 1016
`
`

`

`i,-
`~
`a-...
`~
`\0
`~
`~
`i,-
`_,.a-...
`rJ'J.
`e
`
`'"""'
`'"""'
`'"""' 0 ....,
`~ ....
`'JJ. =-~
`
`'"""'
`0
`0
`N
`'"""' ~~
`?'
`~
`"'!"j
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`I ~ I ~ :
`:
`I
`r-------------------------7
`
`/8 ~ ,,,,---10 ~ .,,--12
`
`!AUTOMATIC I
`
`CONTROL
`
`GAIN
`
`FIG. I
`
`24
`
`SOURCE
`
`AC
`
`SUP PLY
`POWER
`AC TO DC
`
`26
`
`CONTROL
`OUTPUT ~16
`
`44
`
`32
`
`DETECT
`
`AC
`
`SUPPLY
`
`DC POWER
`DIGITAL
`
`SUPPLY
`
`DC POWER
`
`ANALOG
`
`34
`
`INTERFACE
`
`USER
`
`I
`COMMAND 1,4
`
`MEMORY
`
`SPEECH
`
`CONTROLLER
`
`MICRO(cid:173)
`
`~L ________ -1-___________ ----1 _j
`
`42
`
`40
`
`38
`
`36
`
`I
`I
`I
`_,,.----l
`I
`
`20
`
`48
`
`I
`
`30
`
`28
`
`22
`
`I
`
`MEMORY
`PROGRAM
`
`4
`
`MIC
`
`2
`
`Page 2 of 21
`
`

`

`U.S. Patent
`
`Feb.13,2001
`
`Sheet 2 of 11
`
`US 6,188,986 Bl
`
`LO
`<..9
`LL
`
`(\J
`
`C\J
`
`~ -
`(.9
`L
`
`-L
`
`lD J_
`
`0
`(\J
`(\J
`
`(\J
`l9
`LL
`
`0
`C\J
`
`co
`<.O
`
`co
`(\J
`
`C\J
`
`<.O
`0
`N
`
`lD l
`
`0
`(\J
`(\J
`
`g___...
`
`C\J
`
`s;t-
`0
`(\J
`
`O'.)
`0
`N
`
`(\J
`ill
`
`0
`I"-
`
`<.O
`<.O
`
`Page 3 of 21
`
`

`

`U.S. Patent
`
`Feb.13,2001
`
`Sheet 3 of 11
`
`US 6,188,986 Bl
`
`START
`
`100
`
`102
`
`CONFIGURE
`110
`
`104
`
`TURN LED
`OFF
`
`106
`
`SETUP
`FIRST WORD
`VALUES
`
`108
`
`LISTEN FOR
`SILENCE
`PERIOD
`
`WA IT FOR
`UTTERANCE
`AND RECORD
`
`116
`
`CHECK
`DURATION
`OF WORD
`
`RECOGNIZE
`THE WORD
`
`122
`
`WAIT FOR
`UTTERANCE
`ANO RECORD
`
`SETUP
`SECOND WORD
`VALUES
`
`126
`
`INITIALIZE
`SILENCE
`LEVELS
`
`128
`
`LISTEN FOR
`SILENCE
`PERIOD
`
`CHECK
`DURATION
`OF WORD
`
`RECOGNIZE
`THE WORD
`
`FIG. 3A
`
`Page 4 of 21
`
`

`

`U.S. Patent
`
`Feb.13,2001
`
`Sheet 4 of 11
`
`US 6,188,986 Bl
`
`©
`
`166
`
`158
`
`SET DIM
`LEVEL
`TO LOW
`
`SET DIM
`LEVEL TO
`MEDIUM
`
`SET DIM
`LEVEL
`TO OFF
`
`SET DIM
`LEVEL
`TO ON
`
`FIG. 3C
`
`FIG. 3B
`
`Page 5 of 21
`
`

`

`i,-
`~
`a-...
`~
`\0
`~
`~
`i,(cid:173)
`_,.a-...
`rJ'l
`e
`
`'"""'
`'"""'
`0 ....,
`Ul
`~ ....
`'JJ. =(cid:173)~
`
`'"""'
`0
`0
`N
`'"""' ~~
`?'
`~
`"'!"j
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`----__ , _ _j
`I
`I
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`--1----
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`L
`
`: I AUTOMATIC I ~ ~ :
`-7
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`--
`
`--
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`64 ~--
`
`CONTROL
`
`GAIN
`
`CONTROLLER
`
`MICRO(cid:173)
`
`620
`
`630 I
`
`628
`
`622
`
`MEMORY
`PROGRAM
`
`I 616
`
`. §34
`
`. 614
`
`OUT PUT
`
`INTERFACE
`
`USER
`
`MEMORY
`COMMAND
`SPEECH
`
`CONTROL ' 646
`
`I / '
`
`CONTROLLER
`AND DIM
`TOUCH
`
`' 644.
`
`SOURCE
`
`I AC
`
`624
`
`~ CT
`
`632
`
`AC TO DC ~
`
`SUPPLY
`POWER
`
`FIG.6
`
`62~
`
`•
`DIGITAL I
`
`SUPPLY
`
`DC POWER
`
`f
`
`SUPPLY
`DC POWER
`ANALOG
`
`MIC
`
`604
`
`602
`
`Page 6 of 21
`
`

`

`U.S. Patent
`
`Feb.13,2001
`
`Sheet 6 of 11
`
`US 6,188,986 Bl
`
`START
`
`700
`
`INITIAL
`POWER TO
`WALL SWITCH
`
`RED LED ON
`TO INDICATE
`POWER
`
`BOTH RED
`AND GREEN
`LEDS ON
`
`LISTEN FOR
`A WORD
`
`702
`
`704
`
`706
`
`708
`
`712
`
`GO TO
`>N~O~---.tDEFAULT1------1~
`MODE
`
`NO
`
`D
`
`YES
`
`FIG. 7A
`
`Page 7 of 21
`
`

`

`U.S. Patent
`
`Feb.13,2001
`
`Sheet 7 of 11
`
`US 6,188,986 Bl
`
`DEFAULT
`MODE
`
`722
`
`738
`
`II LIGHTS
`ON/OFF"
`MODE
`
`TURN GREEN
`LED ON A ND
`WA,I T FOR WORD
`
`TURN RED AND
`GREEN LEDS ON
`AND WAIT
`FOR WORD
`
`730
`
`TURN LED
`OFF
`
`YES
`
`TURN LED
`OFF
`
`TOGGLE
`THE
`POWER
`
`732
`
`734
`
`FIG.7B
`
`TURN GREEN
`LED ON
`AND WAIT
`FOR WORD
`
`746
`
`TURN LED
`OFF
`
`748
`
`752
`
`TURN LED
`OFF
`
`TURN LED OFF
`AND TOGGLE ' - - - - -~
`POWER
`
`FIG. 7C
`
`Page 8 of 21
`
`

`

`U.S. Patent
`
`Feb.13, 2001
`
`Sheet 8 of 11
`
`US 6,188,986 Bl
`
`1 COMPUTER 1
`1 LIGHTS 1
`I ON/OFF I MODE
`
`758
`
`1 INTELA SWITCH 1
`I ON/OFF' MODE
`
`788
`
`TURN RED AND
`GREEN LEDS ON
`AND WAIT
`FOR WORD
`
`764
`
`766
`
`DID
`USER SAY
`·coMPUTER 1
`') .
`
`TURN GREEN
`LED ON AND
`WAIT FOR WORD
`
`TURN LED
`OFF
`
`768
`
`TURN RED AND
`GREEN LEDS ON
`AND WAIT
`FOR WORD
`
`794
`
`TURN GREEN
`LED ON AND
`WAIT FOR WORD
`
`796
`
`TURN LED
`OFF
`
`770
`
`772
`
`800
`
`802
`
`TURN LED OFF
`AND TOGGLE ~ - - - - - -a i
`POWER
`
`TURN LED OFF
`AND TOGGLE ~ - - - - - - i~
`POWER
`
`FIG. 70
`
`FIG. 7 E
`
`Page 9 of 21
`
`

`

`U.S. Patent
`
`Feb.13,2001
`
`Sheet 9 of 11
`
`US 6,188,986 Bl
`
`SPEAKER
`DEPENDENT
`MODE
`
`806
`
`808
`
`810
`
`FLASH RED AND
`GREEN LEDS ONCE
`TO
`INDICATE
`WAITING FOR
`FIRST WORD
`
`TURN ON GREEN
`L ED TO I N D I CA TE
`UNIT IS
`LISTENING
`
`AVERAGE
`CURRENT WORD
`PATTERN WITH
`PREVIOUS WORD
`PATTERN
`
`SAVE WORD
`INTO Fl RST
`POSITION IN
`EEPROM
`
`L
`
`820
`
`822
`
`REMEMBER
`YES
`>---ai1 HOW USER .__ ..
`SAID WORD
`
`FIG. 7F
`
`Page 10 of 21
`
`

`

`U.S. Patent
`
`Feb.13,2001
`
`Sheet 10 of 11
`
`US 6,188,986 Bl
`
`FLASH RED AND
`GREEN LEDS TWICE
`TO INDICATE
`WAITING FOR
`SECOND WORD
`
`TURN ON GREEN
`LED TO INDICATE
`UNIT IS
`LISTENING
`
`834
`
`836
`
`840
`
`.NO
`
`NO
`
`YES
`
`'FLASH RED
`LED ONCE
`
`842
`
`AVERAGE
`CURRENT WORD
`PATTERN WITH
`PREVIOUS WORD
`PATTERN
`
`NO
`
`SAVE WORD
`INTO SECOND
`IN
`POSITION
`EEPROM
`
`850
`
`J
`
`852
`
`REMEMBER
`~ -1~ HOW USER J--_.,
`SAID WORD
`
`FIG. 7G
`
`Page 11 of 21
`
`

`

`U.S. Patent
`
`Feb.13,2001
`
`Sheet 11 of 11
`
`US 6,188,986 Bl
`
`YES
`
`858
`
`NO
`
`TURN GREEN
`LED ON AND
`WAIT FOR WORD
`
`862
`
`YES
`
`TURN LED
`OFF
`
`864
`
`YES
`
`868
`
`FIG. 7H
`
`Page 12 of 21
`
`

`

`US 6,188,986 Bl
`
`1
`VOICE ACTIVATED SWITCH METHOD AND
`APPARATUS
`
`This is a continuation-in-part of application Ser. No.
`09/002,436, filed Jan. 2, 1998, now abandoned.
`
`FIELD OF THE INVENTION
`
`This invention relates generally to voice activated devices
`for producing control signals, and more specifically to a
`voice activated switch for producing control signals to
`switch on or switch off AC electrical devices.
`
`BACKGROUND OF THE INVENTION
`
`2
`must have the remote control unit with him or her, and each
`target appliance must be adapted to receive IR signals.
`Similar to any developing technology, speech recognition
`poses many hurdles, including designing the most effective
`5 user interface, and increasing response accuracy. A non(cid:173)
`friendly user interface is likely to frustrate the user when
`non-responsiveness of the device is the only indication of a
`recognition error. Another difficulty involves extemporane(cid:173)
`ous conversations and sounds that may falsely trigger a
`10 device response. Speech recognition devices have attempted
`to overcome this problem by allowing a very limited number
`of speech commands such as "ON" and "OFF." However,
`these devices must be programmed with the voices of the
`speakers that will use the device, and do not anticipate noisy
`15 environments in which the device is required to distinguish
`between the speaker and other noises. Also, the limited
`vocabulary allows the utilization of one device per room,
`unless the speaker desires to turn on all appliances at the
`same time.
`The current technology for the remote control of elec(cid:173)
`tronic consumer products fails to provide a hands-free,
`economical, compact, and easy-to-use device. Additionally,
`available designs do not offer solutions for inaccuracy due to
`false response, user frustration, and ambient noise interfer-
`25 ence. These problems and deficiencies are clearly felt in the
`art and are solved by the present invention in the manner
`described below.
`
`SUMMARY OF THE INVENTION
`
`The use of speech recognition technology is becoming a
`viable means to control one's environment. As the sophis(cid:173)
`tication of speech-activated technology increases and the
`cost of the associated hardware and software decreases, the
`use of speech-controlled devices will be commonplace.
`Applications for speech recognition technology are numer- 20
`ous and include the control of appliances, consumer
`electronics, toys, and tools. Products and services employing
`speech recognition are developing rapidly and are continu(cid:173)
`ously applied to new markets.
`The use of speech recognition is ideal wherever the hands
`and/or the eyes are busy. Speech commands are a quick,
`hands-free way to control electrical devices. The dangers
`associated with walking into a dark room, or the inconve(cid:173)
`niences of interrupting tasks in order to turn on appliances
`or lights, are alleviated by the utilization of speech recog- 30
`nition technology.
`Speech recognition technology has been in development
`for more than 25 years resulting in a variety of hardware and
`software tools for personal computers. In a typical
`application, a speech recognition circuit board and compat(cid:173)
`ible software programs are inserted into a computer.
`These add on programs, which operate continuously in
`the background of the computer's operating system, are
`designed to accept spoken words and either execute the 40
`spoken command or convert the words into text. The dis(cid:173)
`advantage in using this approach to control individual appli(cid:173)
`ances is the necessity of one or more computers. Also, it is
`unlikely that manufacturers will add full blown computer
`systems to control appliances such as washing machines or
`electronic products such as stereos. Computer controlled
`systems that utilize speech recognition have been employed
`to control the appliances and electronics throughout a house
`or building, however, these systems are expensive,
`complicated, and require custom installation.
`Remotely controlling an electrical appliance is currently
`possible using devices employing a variety of technologies.
`Products using acoustic signals are available on the market
`to control electrical appliances. These devices recognize
`specific sounds such as claps, and respond by toggling 55
`power switches. One drawback of utilizing an acoustic
`device is that it does not provide "hands-free" control. Also,
`the user must remember an acoustic code, such as a
`sequence of claps, for each appliance.
`Another way to control an appliance is by the utilization
`of a remote control. Remote control units utilizing speech
`recognition have been designed for electronic products such
`as VCRs. The speaker talks into a control unit while depress(cid:173)
`ing a switch, and the speech commands are recognized and
`transmitted to the VCR using infra-red signals.Although this 65
`system offers a means for the remote control of electronics,
`it does not offer a hands-free solution. Additionally, the user
`
`35
`
`It is an advantage of the present invention to provide a
`compact, stand alone, speech recognition circuit to control a
`variety of electrical devices, including consumer electronic
`products or appliances without the need for a host computer.
`It is another advantage to provide an easy-to-use device
`that is programmable to recognize a variety of command
`words so that more that one device can be utilized within one
`room.
`It is yet another advantage of the present invention to
`provide a low-cost replacement for a standard wall switch
`and switch box for speech control of electrical devices
`connected to the wall switch circuit.
`It is still another advantage to provide a portable speech
`recognition interface between a standard AC outlet and an
`45 electrical device.
`A further advantage of the invention is to provide a speech
`recognition device that incorporates user interfaces for con(cid:173)
`firming acceptance of speech commands thereby increasing
`recognition accuracy while reducing the necessity for train(cid:173)
`ing the user.
`In a preferred embodiment of the present invention, a
`stand-alone, programmable speech recognition device acts
`as a control interface between a 120 V or 230 V AC switch
`and a connected electrical appliance or light. In a preferred
`embodiment ("wall switch embodiment"), the voice acti(cid:173)
`vated control circuitry is designed to fit into a switch box
`shell that can be installed in place of a standard wall switch.
`In an alternate preferred embodiment ("outlet
`60 embodiment"), the voice activated control circuitry is
`encased in a portable, palm-sized shell that can be plugged
`into a standard outlet.
`In a wall switch embodiment, the speech recognition
`circuitry of the invention is contained on a circuit board
`having dimensions to fit within a standard wall switch box.
`The circuit board has connections for user interfaces includ(cid:173)
`ing input leads for a microphone for accepting a voice
`
`50
`
`Page 13 of 21
`
`

`

`US 6,188,986 Bl
`
`5
`
`10
`
`25
`
`3
`command, a manual switch controller for accepting manual
`operation of the switch, and at least one light-emitting diode
`("LED").
`The manual switch controller provides a manual means
`for operating the switch, and operates in cooperation with
`the speech recognition circuitry. A variety of technologies
`can be utilized for the manual switch including rocker
`switches, actuator-type controls, pushbuttons and touch
`plate technology. The preferred embodiment utilizes capaci(cid:173)
`tance touch plate technology that is known in the art.
`The speech recognition device operates in a continuous
`listening mode which allows it to actively listen for sounds
`at all times. Ideally, the device is located in a position that
`is exposed and not hidden behind an object such as a piece
`of furniture. An exposed location allows a built-in micro(cid:173)
`phone to pick up un-muffled sounds and speech in proximity
`to the device thereby increasing the response accuracy. The
`preferred wall switch embodiment is typically placed in a
`convenient location within a room and positioned at
`approximately four feet (122 cm) from a floor.
`Thus, the microphone will be at an optimal level to accept
`a speaker's commands, particularly in circumstances in
`which the speaker is seated. Obviously, where an AC outlet
`or a light is controlled by more than one wall switch, a
`microphone of at least one of the voice activated wall
`switches is more likely to be in proximity to the user
`(speaker).
`The outlet embodiment of the present invention is
`plugged into an AC outlet. The outlet embodiment has at
`least one plug for accepting the cord jack of an electrical
`device and may necessarily be plugged into an outlet behind
`an object that obstructs the line from the user to the device.
`Therefore, this embodiment may include a separate attach(cid:173)
`able microphone that is placed in a location most likely to
`maintain an unobstructed line between the microphone and
`the user. The use of a separate microphone allows the
`microphone to be placed in a convenient location that is in
`close proximity to the user. This is particularly useful where
`the environment is noisy, or where the user is disabled or has 40
`low mobility.
`In other embodiments of the outlet and wall switch
`devices, the microphone circuit includes a receiver for
`receiving transmitted radio frequency signals from a sepa(cid:173)
`rate remote microphone. These embodiments are desirable
`for users who cannot effectively trigger the speech recog(cid:173)
`nition because they are not in proximity to the device. For
`example, a user who is seated in a position outside the range
`of the microphone will be unable to control the device. An
`RF receiver will provide remote speech control of the speech
`recognition device.
`The voice-activated device is continuously listening for
`an acceptable speech command as long as power from the
`utility main is available. Thus the device is constantly
`processing background noises and establishing an ambient
`noise level. The ambient noise level is an average decibel
`level of the sounds in the frequency range of speech that are
`detected by the device. For example, a background noise
`level of a 50 decibel air conditioning unit causes the device
`to establish an ambient noise level of 50 decibels. Detected 60
`sounds below that level are ignored, and in order for the
`device to act upon a command word, the user must speak
`above that decibel level. Establishing an ambient noise level
`enables the device to be used in noisy environments.
`Upon receiving a signal in the frequency range of speech
`that is louder than the ambient level, the device determines
`whether the signal is a valid command word. A valid
`
`4
`command word is a member of either a set of pre(cid:173)
`programmed speaker independent words, or a set of user
`programmed speaker dependent words. These sets of com(cid:173)
`mand words correspond to two modes of operation known in
`the art as "speaker independent" and "speaker dependent"
`operation. The user has a choice of the mode of operation
`upon resetting the device. In the preferred wall switch
`embodiment of the invention, reset is activated by pressing
`the touch plate a specified number of times. Reset of the
`outlet embodiment occurs when the device is initially
`plugged into an outlet.
`The first mode of operation is a speaker independent
`mode. In this mode the device can be used by various
`speakers and does not have to be trained to recognize
`15 individual voices. Therefore, the device is pre-programmed
`to respond to a large variety of speech patterns, inflections,
`and enunciations of the target command word. This mode of
`operation usually has a lower number of valid command
`words than a speaker dependent systems that require more
`20 memory to store the various speech patterns. In the preferred
`embodiments, speaker independent command words include
`a name of an electrical device such as "LIGHTS" followed
`by action command words such as "ON" or "OFF" or
`"DIM."
`A speaker dependent mode of operation recognizes only
`one speaker, or a limited number of speakers at a time. The
`speaker dependent mode is activated by resetting and "pro(cid:173)
`gramming" the device. After detecting a reset condition, the
`device listens for a request to select the speaker dependent
`30 mode, and the user follows instructions to program the
`command words. In a preferred embodiment the user is
`prompted by the device through use of a user interface which
`includes prompts from an indicator such as an LED, or
`speech instructions from the device itself, or both. The
`35 device, operating in a speaker dependent mode of operation,
`achieves a high accuracy of word recognition. The disad(cid:173)
`vantage to using this mode of operation is that the system
`response accuracy is limited to the user who programmed
`the valid command set.
`The device limits user frustration by signaling an accep-
`tance of a valid command word through a user interface that
`includes an indicator such as an LED, or a speaker for
`communicating speech prompts. The feedback of the user
`interface permits the user to adjust his or her command word
`45 enunciations and inflections which results in a higher
`response accuracy. Once the device recognizes and indicates
`acceptance of a valid command word, the user responds with
`an action command word such as "ON." If the action
`command word is within the set of valid command words,
`50 the device will respond by performing the desired action.
`For example, in the preferred wall switch embodiment, the
`device responds to the action command word by connecting
`power or disconnecting power to an electrical circuit that is
`connected to the wall switch. For applications where the
`55 action command word is meant to dim or brighten lights, the
`device responds by connecting AC at a reduced or increased
`voltage. In an alternate mode of operation, the action com(cid:173)
`mand word in not used, and the command word such as
`"LIGHTS" is repeated to toggle the lights on or off.
`In a another embodiment of the present invention, the
`device incorporates current carrier modulation techniques as
`disclosed in U.S. Pat. No 3,818,481 of Dorfman, which
`patent is incorporated herein by this reference. Using this
`technology, the device recognizes a variety of electrical
`65 product command words, where only one command word is
`valid for the attached product. Other valid command words
`are transmitted over the utility main to a second device
`
`Page 14 of 21
`
`

`

`US 6,188,986 Bl
`
`5
`directly connected to the utility main or plugged into an AC
`outlet. The second device demodulates the command word
`and makes a determination of whether the command word is
`contained within its set of valid command words, and
`whether the command word corresponds to its attached
`product.
`The present invention provides a compact, continuously
`listening speech recognition circuit that may be incorporated
`into a variety of designs including wall switches and por(cid:173)
`table outlet devices. A voice activated wall switch or wall
`outlet provides an improved method for controlling electri(cid:173)
`cal devices. Limitations of the prior art, including the need
`for complex computer-controlled systems, user frustration,
`use in noisy environments, and limited speech command
`sets, are overcome by the present invention to increase
`response accuracy and device utility.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`Understanding of the present invention will be facilitated
`by consideration of the following detailed description of
`preferred embodiments of the present invention taken in
`conjunction with the accompanying drawings, in which like
`numerals refer to like parts, and in which:
`FIG. 1 is a block diagram of a speech controlled device;
`FIG. 2 is a perspective drawing of a portable speech
`activated device;
`FIG. 3a is a flowchart of a minimal functionality of a
`programming code for a preferred embodiment of the speech
`control device;
`FIG. 3b is a continuation of the flowchart of FIG. 3a for
`an on/off application;
`FIG. 3c is a continuation of the flowchart of FIG. 3a for
`a dimmer application;
`FIG. 4 is a front view of a wall switch of a preferred
`embodiment;
`FIG. 5 is a side view of a wall switch of a preferred
`embodiment;
`FIG. 6 is a block diagram of a speech activated wall
`switch;
`FIG. 7a is a flowchart of a wall switch of a preferred
`embodiment;
`FIG. 7b is a continuation of the flowchart of FIG. 7a for
`a default mode of a preferred embodiment;
`FIG. 7c is a continuation of the flowchart of FIG. 7a for
`a first user independent mode;
`FIG. 7d is a continuation of the flowchart of FIG. 7a for
`a second user independent mode;
`FIG. 7e is a continuation of the flowchart of FIG. 7a for
`a third user independent mode;
`FIG. 7/ is a continuation of the flowchart of FIG. 7a for
`a speaker dependent mode;
`FIG. 7g is a continuation of the flowchart of FIG. 7/; and
`FIG. 7h is a continuation of the flowcharts of FIGS. 7/ and
`7g.
`
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENTS
`
`"Speech recognition" refers to the ability of a device to
`recognize what words have been spoken and to take specific
`actions according to those words recognized. FIG. 1 is a
`block diagram of a preferred embodiment of the electrical
`components of a speech recognition device. A microphone 2
`accepts and converts speech and other sounds into electrical
`
`5
`
`20
`
`6
`audio signals. The electrical audio signals at the output of the
`microphone 1 are amplified by an input amplifier 4 and fed
`into a bandpass filter 6. The band pass filter 6 of the
`preferred embodiment is designed to filter signals outside of
`the frequency range of approximately 580 Hz to 4.2 kHz,
`which represents the typical frequency range of speech.
`The filtered audio signal is introduced into an automatic
`gain control circuit 8. The phrase "automatic gain control"
`usually refers to a feedback loop that accepts a varying input
`10 signal and uses feedback to maintain a constant output
`signal. The automatic gain control circuit 8 of the preferred
`embodiment operates in a different manner by supplying a
`continuous ambient level signal to the microcontroller 20
`over a pre-determined time window. The microcontroller 20
`15 maintains the ambient level during the time window by
`sending feedback signals 36, 38 to the automatic gain
`control circuit 8. The ambient level is used as a starting level
`for recognizing speech. Any background noise received by
`the device during the time window that is below the ambient
`level is ignored. After the time window has expired, the
`device establishes a new ambient level signal.
`Establishing an ambient level is desirable feature in noisy
`environments because the user need only speak above the
`ambient level to trigger the device. Additionally, the ambient
`25 level increases the device accuracy because the device will
`not falsely trigger in response to a stray or constant back(cid:173)
`ground noise. In the preferred embodiment, a time window
`has a duration in the range of 5 seconds to 1 minute, where
`approximately 5 seconds is an ideal duration. Obviously, the
`30 particular pre-determined time window may vary with the
`type of environment, and is not meant to be a limiting factor.
`For example, in a particularly quiet environment, the time
`window may be of a longer duration because changing
`background noises are not expected.
`In the preferred embodiment, the automatic gain control
`loop includes the automatic gain control circuitry 8, the
`microcontroller 20, an amplifier 10, and a multiplying buffer
`12. The output of the automatic gain control circuitry 8 is fed
`into an amplifier 10. The output of the amplifier 10 is fed
`40 into the microcontroller 20 and a multiplying buffer 12.
`Finally, the output of the multiplying buffer is also fed into
`the microcontroller. Thus, the "ambient level" that is sent to
`the microcontroller 20 consists of both a zero level 40 at the
`output of the amplifier 10, and a multiplied level 42 at the
`45 output of the multiplying buffer. The microcontroller 20 of
`the preferred embodiment of FIG. 1 is a general purpose
`microcontroller manufactured by Sensory, lnc. TM which is
`configurable for a variety of applications including speech
`recognition.
`The microcontroller of the preferred embodiment requires
`the zero level 40 and the multiplied level 42 to produce
`feedback signals 36, 38. However, other embodiments of the
`invention using different microcontrollers may have differ(cid:173)
`ing input requirements to maintain the ambient level that is
`55 established using feedback signals 36, 38. Also, the auto(cid:173)
`matic gain control circuitry 48 may be deleted from the
`circuit of FIG. 1 for embodiments where the microcontroller
`includes equivalent circuitry of the automatic gain control 8,
`amplifier 10, and multiply buffer 12. For such embodiments,
`60 the output of the band pass filter 6 is directly accepted by the
`microcontroller 20. As additional functions become avail(cid:173)
`able on the microcontroller 20, other circuit functions such
`as program memory band pass filter and input amplifier may
`eliminate the need for separate circuits to provide these
`65 functions.
`The microcontroller circuitry includes the microcontroller
`20 and a number of memory modules The memory modules
`
`35
`
`50
`
`Page 15 of 21
`
`

`

`US 6,188,986 Bl
`
`15
`
`20
`
`7
`of the preferred embodiment include the program memory
`22 and speech command memory 14 which are shown
`external to the microcontroller 20, but which may be internal
`to the microcontrollers of other embodiments of the inven(cid:173)
`tion. The program memory 22 is a Read Only Memory 5
`(ROM) module which stores the programming code of the
`microcontroller 20. The programming code establishes the
`sequence of events that are followed by the device to
`produce a control signal 44 in response to valid speech
`commands. The speech command memory 1 4 of the pre(cid:173)
`ferred embodiment employs a Random Access Memory 10
`(RAM) module which stores the speaker dependent speech
`commands. The speaker independent speech commands are
`stored in a separate memory ROM module (not shown)
`which may be internal to the microcontroller. The term
`"memory module" does not necessarily refer to separate
`circuit elements. For example, all ROM data may be stored
`in the same circuit element, but at different address block
`locations.
`Power circuitry of the preferred embodiment which sup-
`plies analog and digital operating voltages to the device
`circuitry includes an AC source circuit 24, an AC to DC
`power supply circuit 26, an analog DC power supply circuit
`28, and a digital DC power supply circuit 30. Standard utility
`AC is supplied to the AC source circuit 24 by means of a 25
`standard AC jack that is plugged in to a standard AC outlet
`72, as shown in FIG. 2. The device may be adapted to be
`compatible with a 120 V or 230 V AC standard. The AC to
`DC power supply circuit 26 convert the standard utility AC
`to DC voltages which are fed into the analog DC power 30
`supply circuit 28 and the digital DC power supply circuit 30.
`The analog DC power supply circuit 28 supplies power to
`the input amplifier 4 and the microcontroller 20. The Digital
`DC power supply supplies digital voltages to the microcon(cid:173)
`troller 20.
`Standard utility AC is also supplied to an AC detect circuit
`32 which is connected to the microcontroller 20 and the
`output control circuit 16. Upon recognition of a valid speech
`command, the microcontroller sends a control signal 44 to
`the output control circuit 16. The control signal 44 enables
`or disables a connection of the standard utility AC into the
`output control circuit 16.