United States Patent [19]
`Tomitsuka et al.
`
`I 1111111111111111 11111 111111111111111 IIIII IIIII IIIII IIIII 111111111111111111
`US005566271A
`5,566,271
`[11] Patent Number:
`[45] Date of Patent:
`Oct. 15, 1996
`
`[54] CONTROL APPARATUS FOR ELECTRONIC
`EQUIPMENT
`
`5,086,385
`5,lll,409
`
`2/1992 Launey et al. .......................... 364/188
`5/1992 Gasper et al ........................... 395/152
`
`[75]
`
`Inventors: Hidemi Tomitsuka; Asako Tamura,
`both of Kanagawa; Yasuhiro Chigusa;
`Shiro Omori, both of Tokyo, all of
`Japan
`
`[73] Assignee: Sony Corporation, Tokyo, Japan
`
`[21] Appl. No.: 228,923
`
`[22] Filed:
`
`Apr. 18, 1994
`
`FOREIGN PATENT DOCUMENTS
`
`075026
`313976
`369430
`3918298
`2220290
`
`4/1982
`10/1988
`11/1989
`12/1990
`6/1989
`
`European Pat. Off ..
`European Pat. Off ..
`European Pat. Off ..
`Germany.
`United Kingdom .
`
`OTIIBR PUBLICATIONS
`
`Patent Abstract of Japan, vol. 12, No. 385, Oct. 14, 1988
`(63- 129725).
`
`Related U.S. Application Data
`
`[63] Continuation of Ser. No. 815,157, Dec. 31, 1991, aban(cid:173)
`doned.
`
`[30]
`
`Foreign Application Priority Data
`
`Primary Examiner-Allen R. MacDonald
`Assistant Examiner-Richemond Dorvil
`Attorney, Agent, or Firm-Jay H. Maioli
`
`[57]
`
`ABSTRACT
`
`[JP]
`
`Japan .................................... 3-013758
`
`Jan. 12, 1991
`Int. Cl.6
`........................................................ GlOL 3/00
`[51]
`[52] U.S. Cl . ........................ 395/2.84; 395/2.55; 395/2.67
`[58] Field of Search ................................. 395/2, 12, 2.79,
`395/2.84, 2.55, 2.67, 152; 381/48; 364/419.2;
`345/122
`
`[56]
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,104,625
`4,305,131
`4,333,152
`4,445,187
`4,569,026
`4,736,447
`4,969,194
`
`8/1978 Bristow et al ............................ 381/48
`12/1981 Best ............................................ 395/2
`6/1982 Best ............................................ 395/2
`4/1984 Best ............................................ 395/2
`2/1986 Best ............................................ 395/2
`4/1988 Korinsky ................................... 382/69
`11/1990 Ezawa et al. ............................... 395/2
`
`An instruction for operation mode control of a VTR 40 and
`information on the video recording reservation is voice
`inputted. The voice input is recognized by a voice recogni(cid:173)
`tion circuit 13 and is fed to a control circuit 15. The control
`circuit 15 controls the VTR 40 in response to the instruction
`or information of the voice input and causes an animation
`character generating circuit 16 to generate a video image of
`an animation character AC for displaying it on the screen of
`a CRT display 30. A message from the animation character
`AC is voice synthesized in a voice synthesizing circuit 19
`and the synthesized voice is outputted from a speaker 20.
`
`The electronic equipment can be operated as if the user were
`talking with the animation character, so that a natural
`man-machine interface can be realized.
`
`10 Claims, 17 Drawing Sheets
`
`JO
`
`/3
`
`VOICE
`RECOGNITION
`CIRCUIT
`
`SWITCH
`STATE
`DETECTING
`UNIT
`
`14
`VOICE
`SYNTHESIZING
`CIRCUIT
`
`20
`
`/9
`
`15
`
`16
`
`CONTROL
`CIRCUIT
`
`ANIMATION
`CHARACTER
`GENERATING
`CIRCUIT
`
`VTR
`CONTROLLER
`
`/8
`
`17
`
`SUPERIMPOSER
`
`30
`
`VTR
`
`40
`
`Page 1 of 28
`
`GOOGLE EXHIBIT 1014
`
`

`

`~
`'1
`N
`,...
`=",
`=",
`U1
`,...
`U1
`
`....,1
`~
`~
`
`~
`
`I 0
`......
`rD
`=- rD
`
`00
`
`40
`
`VTR
`
`~
`
`I
`
`0'.
`\0
`\0
`~
`
`...,tll
`~
`r'"
`I")
`0
`
`(0 = '"""'
`~ = '"""'
`•
`rJ'J.
`~ •
`
`17
`
`30
`
`j
`
`SUPERIMPOSER L
`
`CIRCUIT
`
`GENERATING
`CHARACTER
`
`HcoN~~~LLERk
`
`----'
`
`I
`
`SYNTHESIZING ✓
`
`CIRCUIT
`
`VOICE
`
`14/
`
`FIG.1
`
`/8
`
`I
`
`I
`
`I
`
`I
`
`/9
`
`20
`
`STATE F1 CONTROL I
`VOICm H ANIMATION ~··
`l
`
`CIRCUIT
`
`DETECTING
`
`UNIT
`
`SWITCH
`
`RECOGNITION
`
`CIRCUIT
`
`(
`16
`
`(
`/5
`
`13
`
`10
`
`Page 2 of 28
`
`

`

`U.S. Patent
`
`Oct. 15, 1996
`
`Sheet 2 of 17
`
`5,566,271
`
`-
`
`(/)
`;
`
`cnZ
`-0
`a:o
`WW r---
`;1:z
`oo::
`c.. ::>
`I-
`
`za:o
`oww
`-1- ...J
`I-0-1
`<( <( <(
`:Ea:u
`-<(
`z :r: en
`<tu-
`
`C\J
`Cf)
`l
`
`I-
`:::>
`a.
`z
`-
`a:
`w
`Cf)
`:::>
`
`lO
`zz
`(/) ' - 00
`-1-
`I-<(
`<(>
`:Ea:
`a:w
`-en
`LLw
`Za:
`OLL
`(.) 0
`
`,--.
`
`,___
`
`,-..--.. \D
`(/)
`
`-
`
`W...J
`c,w
`zO
`<(Z .----
`J: <(
`uO
`
`>
`-
`
`z
`C!lO
`z-
`o-1-
`WO<(
`oa::E
`-Oa:
`>Oo
`Wu.
`a: z - I'----'\!'"
`
`(/)
`
`~
`
`N
`•
`
`0 -IL
`
`Z:EC!l
`ow
`-1-z
`I-en-
`.___.,. <(>-Z
`a:cnz w
`:::::>
`a. u.. a:
`oo
`
`~f'()
`(/)
`
`Page 3 of 28
`
`

`

`U.S. Patent
`
`Oct. 15, 1996
`
`Sheet 3 of 17
`
`5,566,271
`
`SC
`I
`
`AC
`
`WHAT CAN I HELP YOU ?
`
`BG
`
`SP
`
`FIG.3
`
`Page 4 of 28
`
`

`

`U.S. Patent
`
`Oct. 15, 1996
`
`Sheet 4 of 17
`
`5,566,271
`
`NJ.~
`(\J
`
`...I
`eta:
`zo
`- (./)
`Q(.J)
`.... UJ
`<to
`a:o
`Wa:
`0.. C.
`0
`/ ' ..
`
`lD
`(\J
`(
`
`~
`
`wW
`:::) 0
`(!}<
`ou.
`...1a:
`<W
`ZI-
`-
`<Z
`/"'>.
`
`(\J
`(\J~
`
`<;;j-
`C\J
`{
`
`NJ--.
`
`- 'i
`
`~-----------------------7
`I
`I
`I
`I
`I
`I (\J
`I
`I
`I
`I
`I "V
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`."/
`I
`I
`z
`I
`UJ
`I
`o
`I
`0
`<
`I
`w!::o I
`u.
`oza:
`I
`a:
`6 (!} ct I
`I
`UJ
`>001
`I-
`oco
`I
`z
`-
`I
`I
`w
`a:
`1
`I
`L_ ------- ---- __ ___ __ _____ ..J
`
`q
`•
`
`LL
`
`I "
`I -
`
`I
`
`A
`
`,
`
`A
`
`CZ(!}> v
`a:a:za: I" -
`<w-o
`C1-a:~
`Zt--Ow
`<<I-~ K
`I- 0.. (./)
`(./)
`
`L...-.\,,.
`r-v'
`
`a:
`UJ
`;:E-l
`w ....1
`1-0
`(./) a:
`>I-
`(.J)Z
`0
`0
`
`,
`
`\0
`C\J~
`
`i.n ____
`
`-
`
`-lt-o-
`a::::)
`1-0
`za:
`o-oO
`
`Page 5 of 28
`
`

`

`U.S. Patent
`
`Oct. 15, 1996
`
`Sheet 5 of 17
`
`5,566,271
`
`VOICE INPUT ,__3 I
`
`/
`
`/
`
`/
`
`/
`
`32
`)
`
`I
`NATURAL
`LANGUAGE INPUT
`PROCESSING
`PRODUCTION
`SYSTEM
`
`/
`
`/
`
`/
`
`/
`
`SENTENCE
`NORMALIZING ~PSI
`PRODUCTION
`SYSTEM
`
`DIVIDING
`PRODUCTION
`SYSTEM
`
`-----PS2
`
`\
`
`\
`
`\
`
`\
`
`r------,
`r...L-----, l
`I
`I
`r..1-------,
`_J
`I _f,
`l MEANING
`r
`'3 3
`FRAME
`I
`L_ - - - -~-__J
`
`LANGUACE
`PRODUCTION
`SYSTEM
`
`------p 53
`
`MEANING
`PRODUCTION
`SYSTEM
`
`f---. P S4
`
`\
`
`\
`
`\
`
`\
`
`\
`\
`
`\
`
`\
`
`- -
`
`-34
`
`\
`
`\
`
`- - - r - - - - - - - - .
`INTERPOLATION
`PRODUCTION ~ P 5 5
`SYSTEM
`
`CUSTOM
`LEARNING
`PRODUCTION
`SYSTEM
`
`~p
`56
`
`INFERENCE
`DIALOGUE
`PRODUCTION
`SYSTEM
`
`I
`
`\
`.------__.________\
`\
`VIDEO
`RECORDING
`RESERVING
`SCHEDULER
`
`)
`35
`
`\
`
`\
`\
`
`CUSTOM
`DATA BASE
`~
`HO B
`DIALOGUE
`PROCESSING :---p57
`SYSTEM
`
`\
`\
`\
`\
`
`\
`
`\
`
`\
`
`FIG.S
`
`MESSAGE
`SYSTEM
`\. MSS
`
`Page 6 of 28
`
`

`

`U.S. Patent
`
`Oct. 15, 1996
`
`Sheet 6 of 17
`
`5,566,271
`
`START
`
`CALL ANIMATION
`CHARACTER
`
`SI I
`
`ACTIVATE ANIMATION
`CHARACTER
`
`S/2
`
`NO
`
`S/4
`
`IDLING
`ANIMATION
`PROCESSING
`
`5/5
`
`INPUT
`
`S/7
`
`RESERVING
`PROCESSING
`
`S/9
`DISPLAY
`PROCESSING
`
`521
`CHANGE
`PROCESSING
`
`S23
`
`CANCEL
`PROCESSING
`
`S25
`ANIMATION
`CHARACTER
`OPERATION
`
`S27
`
`VTR
`OPERATION
`
`YES
`
`END
`
`FIG.&
`
`Page 7 of 28
`
`

`

`U.S. Patent
`
`Oct. 15, 1996
`
`Sheet 7 of 17
`
`5,566,271
`
`RESERVING
`PROCESSING
`
`WAITING FOR
`INFORMATION INPUT
`
`S 50
`
`ELEMENT ARY
`INFORMATION INPUT
`
`5 5 I
`
`ELEMENTARY
`INFORMATION
`INPUT PROCESSING
`
`5 52
`
`YES
`
`S54
`
`CONFIRMATION
`
`INPUT INSTRUCTION
`OR INFORMATION
`
`S55
`
`S56
`
`YES
`
`YES
`
`S66
`
`INQUIRY OF
`LACKING
`INFORMATION
`
`S67
`
`DUPLICATION
`CHECK
`
`END
`
`S60
`
`S63
`
`INQUIRY OF
`CONTENT TO BE
`CHANGE
`
`STOP
`RESERVATION
`
`S65
`
`ELEMENTARY
`INFORMATION
`
`S69
`
`FIG.7
`
`Page 8 of 28
`
`

`

`U.S. Patent
`
`Oct. 15, 1996
`
`Sheet 8 of 17
`
`5,566,271
`
`,--SC
`
`PH
`
`RESERVATION \
`
`AC
`
`PLEASE, MAKE A RESERVATION
`
`BG
`
`AC
`
`FIG.a
`
`/SC
`
`PH
`
`..----,::.....-_____._____._-h
`
`RESERVATION
`
`TUESDAY, NOV. 7TH
`
`3:00 a.m. TO 4:00 p.m.
`
`CHANNEL 6
`
`IS IT OK 7
`
`FIG.9
`
`BG
`
`SP
`
`PR
`
`SP
`
`Page 9 of 28
`
`

`

`~
`,..
`0'.
`0'.
`Ul
`,..
`Ul
`
`~
`
`---l
`~
`
`s,
`rJ1 =-a
`
`1,0
`
`="
`1,0
`1,0
`~
`
`~ ya
`r"l r+-
`0
`
`"""" ~ a
`
`~
`""d
`•
`00
`d •
`
`INFORMATION INPUT
`
`TO ELEMENTARY
`
`PROCESSING
`
`FIG.10
`
`END
`
`JUST PREVIOUS
`
`INQUIRY
`
`S76
`
`RESERVATION
`
`STOP
`
`INFORMATION INPUT WRITE INTO
`
`TO ELEMENTARY
`
`FRAME
`
`PROCESSING
`
`YES
`
`YES
`
`S75
`
`S78
`
`NO
`
`S73
`
`NO
`
`OF ERROR
`INQUIRY
`
`ITEM
`
`YES
`
`S79
`
`YES
`
`I f__r----S 71
`
`ANALYSIS
`MEANING
`
`INFORMATION INPUT
`
`ELEMENTARY
`
`PROCESSING
`
`Page 10 of 28
`
`

`

`U.S. Patent
`
`Oct. 15, 1996
`
`Sheet 10 of 17
`
`5,566,271
`
`LACKING INFORMATION
`PROCESSING
`
`S33
`
`S34
`
`S35
`
`NO
`INQUIRY OF DATE
`
`INFORMATION ON DATE
`S36
`
`S37
`
`NO
`INQUIRY ON CHANNEL
`
`CHANNEL INFORMATION
`
`S38
`
`S46
`
`YES
`
`CUSTOM
`DISPLAY AND
`CONFIRMATION
`
`S47
`
`YES
`
`YES
`
`S39
`
`YES
`
`S4/
`
`S45 S44,---__ ~
`
`INQUIRY OF STARTING TIME
`
`STARTING TIME INFORMATION
`
`INQUIRY OF
`ENDING TIME
`
`S43
`
`ENDING TIME
`INFORMATION
`INFROMATION
`ON RECORDING
`PERIOD OF TIME
`
`CONFIRMATION
`
`S49 FIG.11
`
`YES
`
`Page 11 of 28
`
`

`

`U.S. Patent
`
`Oct. 15, 1996
`
`Sheet 11 of 17
`
`5,566,271
`
`DUPLICATION
`CHECK
`
`CONFIRMATION
`
`$ 8 /
`
`YES
`
`S82
`
`NO
`
`S84
`
`S92
`
`RESERVATION
`COMPLETED
`
`DISPLAY OF DUPLICATED
`RESERVATION AND SELECTION
`OF CHANGE/CANCEL
`
`CHANGE/CANCEL
`
`S85
`S86
`NO
`
`S93
`
`S 8 7 ,-----___...____,
`STOP
`RESERVATION
`
`END
`
`INQUIRY OF CONTENT
`TO BE CHANGED
`
`S88
`
`ELEMENTARY
`INFORMATION
`
`S89
`
`UNDERSTAND
`MEANING
`
`S9/
`
`WRITING
`INTO SLOT
`
`YES
`
`S94
`
`INQUIRY OF
`ERROR ITEM
`
`FIG.12
`
`Page 12 of 28
`
`

`

`U.S. Patent
`
`Oct. 15, 1996
`
`Sheet 12 of 17
`
`5,566,271
`
`OISPLA Y PROCESSING
`
`INSTRUCTION/
`DISPLAY
`CHANGE INSTRUCTION/
`CANCEL INSTRUCTION
`
`S/00
`
`S/0I
`NO
`
`S/03
`
`RESERVATION
`
`IS DISPLAYED
`
`SI04
`
`S/02
`
`ERROR MESSAGE
`
`END
`
`YES i - - - - - - - - - - - - - - - - - - - -~
`S/05
`IS DISPLAY SWITCHED ?
`S/06
`
`YES/NO/RETURN/
`CHANGE INSTRUCTION/
`CANCEL INSTRUCTION
`
`S 115
`
`S//6
`
`YES
`
`NEXT
`INFORMATION
`IS DISPLA YEO
`
`PREVIOUS
`INFORMATION
`IS DISPLAYED
`
`CHANGE
`PROCESSING
`
`SI20 5121
`
`.__ ____ __,_ YES
`
`WHICH IS PERFORMED
`CHANGE OR CANCEL ?
`
`S113
`SI 14
`
`YES/NO/RETURN/ CHANGE
`INSTRUCTION/ CANCEL INSTRUCTION
`
`CANCEL
`PROCESSING
`
`TO STEP S125
`FIG.13
`
`Page 13 of 28
`
`

`

`U.S. Patent
`
`Oct. 15, 1996
`
`Sheet 13 of 17
`
`5,566,271
`
`FROM STEP S114
`
`S/25
`YES
`
`YES
`
`NO
`
`S131
`
`CHANGE
`PROCESSING
`
`S132
`
`CANCEL
`PROCESSING
`
`YES
`SELECTION OF
`CHANGE/CANCEL
`
`SI 2 8
`
`CHANGE INSTRUCTION/
`CANCEL INSTRUCTION/
`RETURN
`
`S 129
`
`NO
`
`END
`
`FIG.14
`
`Page 14 of 28
`
`

`

`U.S. Patent
`
`Oct. 15, 1996
`
`Sheet 14 of 17
`
`5,566,271
`
`CHANGE
`PROCESSING
`
`SELECTION OF
`CHANGE RESERVATION
`
`S/40
`
`FIRST TO FOURTH
`
`S/4 /
`
`S/42
`
`NO
`
`S150
`
`ERROR
`MESSAGE
`
`INQUIRY OF
`CONTENT TO BE
`CHANGED
`
`S /43
`
`ELEMENTARY
`INFORMATION
`
`S /44
`
`CONFIRMATION
`
`S/45
`
`YES/NO/ ELEMENTARY
`INFORMATION
`
`S /46
`
`S147
`YES
`
`YES
`
`S151
`
`NO
`
`INQUIRY OF CONTENT
`TO BE CHANGE
`
`DUPLICATION
`CHECK
`
`ENO
`
`FIG.15
`
`Page 15 of 28
`
`

`

`U.S. Patent
`
`Oct. 15, 1996
`
`Sheet 15 of 17
`
`5,566,271
`
`CANCEL
`PROCESSING
`
`SELECTION OF
`CANCELLATION
`OF RESERVATION
`INFORMATION
`
`S/60
`
`FIRST TO
`FOURTH ITEMS
`
`S/6 I
`
`S/62
`
`NO
`
`S/68
`
`ERROR
`MESSAGE
`
`CONFIRMATION
`
`YES/NO
`
`S/63
`
`S/64
`
`S/65
`
`NO
`
`S/66
`)
`
`COMPLETE
`CANCEL
`
`END
`
`S/67
`
`STOP
`CANCEL
`
`END
`
`FIG.16
`
`Page 16 of 28
`
`

`

`U.S. Patent
`
`Oct. 15, 1996
`
`Sheet 16 of 17
`
`5,566,271
`
`VTR OPERATION
`PROCESSING
`
`CHECK VTR
`STATE
`
`S/7I
`
`RETRIEVE OPERATION
`CORRESPONDING TO
`COMMAND FROM MATRIX
`
`S/72
`
`S173
`
`NO
`
`S/76
`
`YES
`
`S/74
`
`ERROR
`MESSAGE
`
`EXECUTE
`CORRESPONDING
`OPERATION
`
`MESSAGE
`
`S175
`
`END
`
`FIG.17
`
`Page 17 of 28
`
`

`

`U.S. Patent
`
`Oct. 15, 1996
`
`Sheet 17 of 17
`
`5,566,271
`
`YES
`
`S/86
`
`SELECT IDLING
`ANIMATION FOR
`TOP LEVEL
`
`IDLING ANIMATION
`PROCESSING
`
`S/82
`
`NO
`
`SELECT IDLING
`ANIMATION FOR
`PROCESSING MODE
`
`S/83
`
`SELECT RANDOM
`NUMBER GENERATING
`PATTERN
`
`S184
`
`NO
`
`S185
`
`YES
`
`IDLING ANIMATION
`OPERATION
`
`END
`
`FIG.18
`
`Page 18 of 28
`
`

`

`5,566,271
`
`1
`CONTROL APPARATUS FOR ELECTRONIC
`EQUIPMENT
`
`This is a continuation of application Ser. No. 07/815,157
`filed Dec. 31, 1991 now abandoned
`BACKGROUND OF THE INVENTION
`
`2
`equipment for designating the operation mode thereof, com(cid:173)
`prising: voice inputting means having an acoustic-electric
`transducer for voice inputting instructions and various items
`of information to designate the operation mode and for
`5 outputting an electric voice signal; voice recognition means
`for processing an output signal from the voice inputting
`means to recognize the instruction and various items of
`information; animation character generating means for out(cid:173)
`putting a video signal of an animation character who is a
`10 message speaker; video image display means for display the
`video signal from the animation character generating means;
`voice synthesizing means for synthesizing the voice signal
`of the message in response to a message signal input; voice
`outputting means for outputting the voice signal from the
`15 voice synthesizing means in voice; and control means
`responsive to at least the output signal from the voice
`recognition means for outputting an operation mode desig(cid:173)
`nation and control signal for designating the operation mode
`of said electronic equipment, an action control signal for
`20 controlling the action of the animation character and a
`message signal instructing a message voice which is desired
`to be synthesized in the voice synthesizing means.
`In accordance with the present invention, electronic
`equipment is controlled in response to a voice input of
`natural language. At this time, a voice message correspond(cid:173)
`ing to a message signal is outputted from voice output means
`and an animation character displayed on video display
`means is moved in synchronization with the message voice.
`Accordingly, it sounds as if the character spoke the message.
`Thus, the user feels as if the user were talking with the
`character.
`In a control apparatus for electronic equipment of the
`present invention, instructions and Various items of the
`information for designating the operation modes are inputted
`in voice and are converted into electrical signals and then
`processed to recognize the instructions and the various items
`of information. The operation mode of the electronic equip(cid:173)
`ment is designated in response to the recognized voice
`inputs. An animation character which will be a speaker of a
`40 message is displayed on video display means and the action
`of the character is controlled. A message voice is synthesized
`in response to the message signal output. The electronic
`equipment can be controlled in response to the voice input
`of natural language so that everybody can easily control the
`45 electronic equipment. A natural man-machine interface
`which makes the users feel as if they were talking with the
`animation character and enables the users to easily operate
`the equipment can be realized.
`
`35
`
`1. Field of the Invention
`The present invention relates to a control apparatus and in
`particular to a control apparatus for electronic equipment for
`making a reservation for video recording or for directly
`controlling the electronic equipment such as a video cassette
`recorder.
`2. Description of the Prior Art
`Various recent home appliances such as video cassette
`recorders have a multiple of functions. Various switches for
`setting various operation modes are provided on the main
`body of the appliance. The operation of those switches is
`complicated. Even if the switches are few in number, the
`operation process is complicated. Accordingly, few users
`can freely use timer function for making a reservation of the
`video recording of the VTR.
`Various approaches have been proposed to make the
`operation of such the appliances easier. For example, the bar
`code reader is used to simplify the input operation. Instruc- 25
`tions and various items of information are inputted in natural
`language, that is, in human voice from a microphone. The
`approach relying on the bar code reader has a low degree of
`freedom, since bar code labels which indicate various items
`of information of a program to be reserved should be 30
`provided. The application range of this approach is limited.
`Therefore, it is considered that the approach relying on the
`natural language is favorable.
`In a voice input operated apparatus in which operation
`instructions for designating the operation modes of the
`information recording/reproducing apparatus such as a VCR
`or VTR can be inputted in human voice, a voice input device
`has been proposed which is capable of stably controlling the
`VCR by generating control commands to the VCR with
`reference to the status data of the information recording/
`reproducing apparatus.
`A control data input device has been proposed which is
`capable of controlling an object by analyzing the natural
`language which is formed of a combination of words rep(cid:173)
`resenting a plurality of control instructions to provide the
`object to be controlled with a control instruction.
`Although these techniques are able to set and control the
`operation mode of the appliances such as a VTR, there is
`much room for improvement in response to an input. Only 50
`numerals and characters representative of the content of the
`reservation are displayed as the data train on a display panel.
`Preliminary knowledge is necessary to deal with the case in
`which correction is made in the course of input or the
`complete reservation cannot be made. Every user cannot 55
`easily operate the VCR. Many users find it difficult to accept
`data comprising only a numeral train. An improvement in
`man-machine interface has been demanded.
`The present invention has been made under the above
`mentioned circumstances. It is an object to provide a control
`apparatus of electronic equipment which make it possible
`for everybody to easily operate the appliance and which
`provide a natural man-machine interface.
`SUMMARY OF THE INVENTION
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a block diagram showing the schematic structure
`of an embodiment of a control apparatus of the present
`invention;
`FIG. 2 is a flow chart showing a main operation of the
`control apparatus of the present invention;
`FIG. 3 is a schematic view showing a CRT screen on
`which an animation character and a balloon in an initial state
`60 are displayed;
`FIG. 4 is a block diagram showing a structure of a voice
`recognition circuit;
`FIG. 5 is a block diagram explaining the function of a
`65 control circuit;
`FIG. 6 is a flow chart showing the detail of the processing
`at each step in the flow chart of FIG. 2;
`
`In order to accomplish the above mentioned object, the
`present invention provides a control apparatus for electronic
`
`Page 19 of 28
`
`

`

`5,566,271
`
`5
`
`3
`FIG. 7 is a flow chart showing the detail of the reservation
`processing in the flow chart of FIG. 6;
`FIG. 8 is an illustration showing the CRT screen on which
`a reservation input request is displayed;
`FIG. 9 is an illustration showing the CRT screen on which
`reserved information confirmation is displayed;
`FIG. 10 is flow chart showing the detail of an elementary
`information input and an elementary information input pro(cid:173)
`cessing in the flow chart of FIG. 7;
`FIG. 11 is a flow chart showing the detail of a lacking
`information processing in the flow chart of FIG. 7;
`FIG. 12 is a flow chart showing the detail of a duplication
`check processing in the flow chart of FIG. 7;
`FIG. 13 is a flow chart showing the detail of a display
`processing in the flow chart in FIG. 6;
`FIG. 14 is a flow chart following the flow chart of FIG. 13;
`FIG. 15 is a flow chart showing the detail of a change
`processing in the flow chart of FIG. 6;
`FIG. 16 is a flow chart showing the detail of a cancel
`processing in the flow chart of FIG. 6;
`FIG. 17 is a flow chart showing the detail of a VTR
`operation processing in the flow chart of FIG. 6; and
`FIG. 18 is a flow chart showing the detail of an idling 25
`animation processing in the flow chart of FIG. 6.
`
`4
`mode of the VTR 40. A video signal from the character
`generator 16 is fed to a superimposer 17 in which it is
`superimposed upon the video signal from the VTR 40 and
`then the superimposed signal is fed to a cathode ray tube
`(CRT) display 30. A voice signal from the voice synthesizing
`circuit 19 is fed to a speaker 20 in which it is converted into
`a sound. The speaker 20 and the CRT display 30 may be
`formed into an integral television receiver circuit.
`The control circuit 15 outputs at least an operation mode
`10 selection control signal which controls the selection of the
`operation mode of the VTR 40 in response to an output
`signal from the voice recognition circuit 13, an action
`control signal for controlling the action of the animation
`character AC of the animation character generating circuit
`15 16, a message signal for instructing a message voice which
`is desired to be synthesized in the voice synthesizing circuit
`19 and comprises a CPU such as a microprocessor. The
`control circuit 15 selects and outputs one control instruction
`which is appropriate for the current operation state of the
`20 VTR 40 from a plurality of control instructions in response
`to an instruction for the selection control of the operation
`mode recognized by the voice recognition circuit 13. The
`selection processing and the operation mode of the VTR 40
`will be described hereafter in detail. The control circuit 15
`outputs a message signal instructing a message which is
`optimal as the content of a response in response to an voice
`input content (and further in response to the current state)
`and feeds the output message signal to the animation char(cid:173)
`acter generating circuit 16 and the voice generating circuit
`30 19.
`In FIG. 1, the transmitter unit of the handset 10 converts
`a voice pronounced by a man such as an operator operating
`the VTR 40, that is, a user) into an electrical voice signal.
`The voice pronounced by the operator includes pronounced
`instruction words, such as "playback", "stop", "record",
`"pause", "slow", "fast-feeding", "rewind" and pronounced
`words such the day of the week (Sunday to Monday), every
`day, every week, channel (channels 1 to 12), starting/ending
`time (morning, afternoon, 0 to 12 o'clock) for making a
`40 reservation of video recording. In the present embodiment,
`the other words "Hey", "O.K.", "No" in addition to the
`above mentioned words can be inputted as voices.
`The press-to-talk switch 12 for designating the voice input
`state disposed in the handset 10 instructs the division of
`words pronounced by the operator by being turned on or off
`by the operator. In other words, the press-to-talk switch 12
`is provided to divide an input voice signal of a sentence
`comprising plural continuous discrete words into units to be
`processed by voice recognition in the voice recognizing
`circuit 13. An output from the press-to-talk switch 12 is fed
`to the switch state detecting unit 14 which is provided in
`parallel with the voice recognition circuit 13. The switch
`state detecting unit 14 generates a state instruction signal
`instructing the current on/off state in response to an output
`signal from a press-to-talk switch 12. The state instruction
`signal assumes states "O" and "l" when the press-to-talk
`switch 12 is in the inoperative and operative states, respec(cid:173)
`tively. Accordingly, if voice recognition is desired to be
`conducted, the press-to-talk switch 12 is turned on and is
`turned off after completion of the voice input. The voice
`recognition in processing units is thus performed in response
`to the corresponding state instruction signal. Accordingly, it
`will suffice for the voice recognition circuit 13 not to analyze
`whether or not the input is completed from the turning off
`(state "0") representative for the completion of the voice
`word input when continuous word inputting is performed by
`voice. In other words, the voice recognition circuit 13 can
`
`35
`
`DESCRIPTION OF THE PREFERRED
`EMBODIMENTS
`
`Now, the preferred embodiments of the present invention
`will be described with reference to the drawings.
`A control apparatus for electronic equipment of the
`present embodiment is a control apparatus for electronic
`equipment for controlling the selection of the operation
`mode of electronic equipment such a video tape recorder
`(VTR) 40 as shown in FIG. 1 . In the present embodiment,
`the VTR 40 is used as the electronic equipment. The present
`embodiment will be described with reference to controls
`such as the selection control of various operation modes of
`the VTR 40 such as recording, playback, fast feeding and
`rewinding and recording reservation.
`In the apparatus of the present embodiment of FIG. 1, a
`handset 10 which is similar to a telephone handset is 45
`provided as the voice input means for inputting instructions
`for selection control of operation modes and various items of
`information by voices. A transmitter unit 11 of the handset
`10 is provided with an acoustic-electrical conversion ele(cid:173)
`ment for converting an input voice into an electrical signal 50
`and outputting it. A press-to-talk switch 12 for dividing
`inputs by switching the voice input mode is provided in the
`vicinity of the transmitter unit 11. An output signal from the
`transmitter unit 11 of the handset 10 is fed to a voice
`recognition circuit 13 in which it is signal processed for 55
`recognizing the instruction and various items of information.
`An output signal from the switch 12 is fed to a switch state
`detecting unit 14 for detecting the on/off state of the switch.
`The voice recognition circuit 13 and the switch state detect(cid:173)
`ing unit 14 are connected with a control circuit 15. With the 60
`control circuit 15 are connected an animation character
`generating circuit 16 for outputting a video signal of an
`animation character which will become a message speaker,
`a voice synthesizing circuit 19 for synthesizing the voice
`signal of the message in response to a message signal input 65
`by using a technique of ruled voice synthesizing, and a VTR
`controller 18 for controlling the selection of the operation
`
`Page 20 of 28
`
`

`

`5,566,271
`
`10
`
`15
`
`20
`
`5
`determine clear starting and completion time when the voice
`is recognized. The range in which the voice can be recog(cid:173)
`nized can be easily determined by a software so that it will
`need not to perform an unnecessary voice recognition pro(cid:173)
`cessing for the noises outside this range. Since control of the
`handset 10 is not performed, noises on switching of voice
`input (on cutting off of voices) will not be inputted.
`A sense of reluctance or incompatibility to speak to a
`machine can be mitigated by feeding a voice or speech
`output signal from the voice synthesizing circuit 19 to the
`receiver unit of the handset 10 and performing input and
`output of voice via the handset 10 which is similar in shape
`to the telephone handset and malfunction upon use at a noisy
`place can be prevented. Responded voice can be shielded
`from co-receiving listeners by feeding the responded voice
`signal to only the receiver unit of the handset 10 without
`feeding responded voice to the speaker 20 in the TV set or
`stereo receiver. Video recording reservation presetting
`operation and VTR operation can be performed without
`interfering with the other listeners.
`A flow chart of main operations in the apparatus of the
`present embodiment is shown in FIG. 2. In FIG. 2, an
`animation character is called in the initial state after the
`power to the apparatus has been turned on by carrying out
`the voice input or the other input operation at step Sl by an
`operator (user), the animation character AC as shown in
`FIG. 3 is displayed on a screen SC of the CRT display 30 and
`the apparatus is brought into a stand-by state of a voice input
`by the operator. That is, the animation character AC and a
`balloon SP for character displaying the content spoken by
`the animation character are superimposed upon the video
`signal from the VTR 40, a background video image and
`displayed. A message that is the same as the message in the
`balloon SP (for example, a message "What can I help you
`with?" representing the stand-by state) is synthesized in the
`voice synthesizing circuit 19 simultaneously with the dis(cid:173)
`play of the message on the screen SC and the voice (for
`example "What can I help you with?") is generated from the
`speaker 20. A calling processing at the step Sl is specifically
`exemplarily shown. When the name of the animation char- 40
`acter AC (for example, Ivy) is called or a call "Hey" is made,
`or the power of the VTR 40 is turned on, the animation
`character AC is displayed together with a message "Yes,
`here is Ivy." More natural atmosphere of dialogue is
`obtained by subsequently shifting to the state of FIG. 3.
`The operator (user) voice inputs an instruction and vari(cid:173)
`ous items of information for the selection control of the
`operation mode at step S2. If the instruction is an instruction
`to activate the system for running the VTR 40, the program
`step proceeds to step S3, the apparatus is brought into a
`phase to directly control the operation mode of the VTR 40.
`If the instruction is to reserve the video recording, the
`apparatus is brought into a mode of the video reserving
`operation of the VTR 40. If the instruction is to confirm the
`reservation, the program step will proceed to a step S5 in
`which the apparatus is brought into a mode to confirm the
`reserved content. In the mode to confirm the reservation, the
`program step will proceed to the step 6 subsequently of the
`step 5 in which processing to change or cancel the reserva(cid:173)
`tion is made. In the modes to reserve the video recording and
`to confirm the reservation, more sophisticated conversation
`between the operator and the animation character AC is
`carried out as will be described. After completion of pro(cid:173)
`cessing at steps S3, S4 and S5 (S6), the apparatus is returned
`into the stand-by state of a voice input.
`In an example of display in FIG. 3, the animation char(cid:173)
`acter AC is displayed in the left and lower area on the screen
`
`6
`SC of the CRT display 30. If the animation character AC is
`displayed in the center of the screen SC, it will become a
`visual obstacle to a video image (background video image
`BG) displayed on the screen. An animation character AC
`5 displayed in the lower and left of the screen will not become
`an obstacle for the image on the screen. A character is
`displayed in the lower area of the screen SC, the legs of the
`character AC will not be levitated so that it will give a stable
`feeling. If the character AC is displayed on the lower and left
`side of the screen SC, a sense of incompatibility is less from
`the standpoint of a human being. The balloon SP to display
`a message in the screen SC is displayed in the lower and
`right areas of the screen SC if the characters of the message
`are an array in a horizontal direction from the left to right
`and in a vertical direction from the upper to lower side,
`respectively. This will scarcely obstruct the image (back(cid:173)
`ground video image BG) on the screen SC.
`An animation action control signal is fed to the animation
`character generating circuit 16 from the control circuit 15 so
`that the action of the animation character AC is controlled.
`The animation character generating circuit 16 which
`receives the action control signal from the control circuit 15
`outputs to the superimposer 17 a video signal which causes
`the animation character AC as shown in FIG. 3 which is a
`25 speaker talking with the operator to move his or her mouth
`in response to the voice output of the message. The anima(cid:173)
`tion signal includes a signal of a balloon SP for character
`displaying the message. The superimposer 17 superimposes
`the video image of the animation character AC upon the
`30 reproduced video image from VTR 40 and feeds it to the
`CRT display 30. The animation character AC displayed on
`the screen of the CRT display 30 is made a personified
`expressive character having familiarity. This causes the
`operator (user) to have a feeling as if he or she were talking
`with the animation character (or electronic equipment).
`The voice recognition circuit 13 in FIG. 1 recognizes the
`above mentioned instructions and various items of informa(cid:173)
`tion by processing the supplied voice signal and may be
`variable in structure. A voice recognition circuit having a
`structure as shown in FIG. 4 will be described.
`In FIG. 4, an output signal (input voice signal) from the
`transmitter unit 11 of the handset 10 is supplied to the input
`terminal 21 and the input voice signal is fed to an operational
`processor 23 via an analog interface 22. After the analog
`interface 22 changes the input voice level to a given level
`depending upon a control data supplied from a system
`controller 24 and then converts the voice signal