PATENT EXAI'INER
`TECIINOL0GV {xNrER 3600
`
`(Primary Examine0
`
`f.l" q
`
`@--16..el
`
`v
`
`rI
`
`1,llh,k,.
`(D"t.)
`
`P44',
`
`tt:,
`
`|
`
`:
`
`\!
`-
`l-I,
`f .
`r
`l\
`n[.4:: ti
`,#a is
`
`r:E-ZB
`NA: =lq*-.
`- -.,,,,
`|
`, a9/524a
`'',
`E-
`'
`:2:-lJi:j::::i::
`I'6i'-i'i':;;;;'';
`i'-'.i.ril.i.':...;
`i
`f,
`jr;:. i,j:jj..j i.;l
`i
`&
`
`I !i
`
`
`i
`XJ;F
`r'rF-
`i
`,,
`I
`
`i.,i;,,t..,. -i ,;.;;;,,,.1,,
`t{ i":i'ii.-'i.i--.'
`
`'J-..
`
`APPLTcANT(s)
`
`t-r TERMTNAL
`I lotsctArMER
`
`i---l Thc .:rm of this paternt
`subserir,arfli to ___ _---
`iias bEi,i: Cisclaimed.
`-r
`ihe term of this oatent shall
`xtend beyond ihe expiration
`,-'.S Patent. No.
`
`ict
`
`(date)
`/
`
`--.1 The terminal _ _months of
`rhis oatent have been disclaimed.
`
`lwenrurruc:
`
`-\
`
`'ti
`
`u
`
`PATENT NUMBER
`il4202-1*,*
`
`..,..::i:
`
`6742921
`
`Appiication
`
`DATE
`
`-TPATENT
`suacuoss lil-
`, t&tu\Wq
`
`PTo-2040
`1?J99
`
`ING CLASSIFICATION
`
`(oNE SUBCTLASS PER BLOCK)
`
`/RAwTNGS
`Figs. Drwg.r/
`
`n
`
`/
`cLAiUts ALLowED
`/ rotalclaims
`,rt Print Fig.,
`Print Claim for O.G.
`/t ;
`I
`+
`,/d# NOTICE OF ALLOWANCE MAILED
`t)46/
`T7f6-"_
`t
`tssuE FEE . \
`
`Anrount Due
`
`ffi
`
`ISSUE BATCH NUMBER
`
`3
`
`Date Paid
`
`JAq{E
`
`Sections 122,fi1and368.!
`
`t
`
`l',heinfor;.::l,ondisclosedher€inmayberestricted. UnauthorizeddisclosuremaybeprohibitedbytheUnitedStatesCcdeTitle35
`Possassion outside the U.S. Patent & Trademark Office is reslricted to authorized employees and contractois only
`
`;orm PTO-436A
`(Rev. 6/99)
`
`t, ,[
`['.t
`
`FTLED wtrH: I orsr 1cnrl flFtcHE I co-noH'r
`F th$hr $n t\rr t * b r
`
`(Atlached in pocket on right inside flap)
`
`; ,'.,. -.Ti'. Ti,
`:
`;j{
`"7i::.
`I
`_. ;rt .
`1rL i:
`
`: -
`
`(FACE)
`
`tg '\
`
`GOOGLE EXHIBIT 1006
`
`Page 1 of 314
`
`

`

`/1
`i-'l
`
`tNtflALS&Zr-:-i--.:"
`
`Daie Received
`(lncl. C. of \.)
`ori
`Da'e Mailed
`
`Fiiffiiilirilililffififfihilffiffil'o*
`
`I
`
`'"'gi?s'zfg95"o
`
`ilqtlul)ltluuluulllllllrlll
`
`09524095
`
`!-
`
`1t i"pplication ._
`i/l/\
`
`papers.
`
`4.
`
`5.
`
`6i
`
`,,1o'' 6'
`
`CONTENTS
`Daie Received
`(lncl. C. ot M.)
`ot
`Date Mailed
`
`42.
`
`43.
`
`44.
`
`45.
`
`46.
`
`47.
`
`48.
`
`49.
`
`rz- t t _ocr
`
`50.
`
`EJ4ln sl
`
`Q*t6 ---ot
`L'l-e::-ci
`tl ._"t.,6- O I
`6'tl'or
`
`' q-q4\
`10-5-ol
`{alrof '1
`r-
`1-15-g')-
`r&'tL:spu
`{;r-r 3.-tdi*t1}*
`
`- t0 -0:.
`\-lo 'oa
`tt- to -or-
`5--? 'O Z
`
`-
`
`52.
`
`53.
`
`54.
`
`55.
`
`56.
`
`57.
`
`58.
`
`59.
`
`60.
`
`61.
`
`62.
`
`63"
`
`64.
`
`65.
`
`66.
`
`67.
`
`68.
`
`69.
`
`70.
`
`71.
`
`A ?- zq.
`75.
`
`76.
`
`77.
`
`78.
`
`79.
`
`80.
`
`81.
`
`82.
`
`{LEFT OUTSIDE)
`
`fbs
`
`,^rn q=;ngZffi
`37. --
`
`38.
`
`39.
`
`40
`
`i,ti
`
`r1{
`
`1 I
`
`Page 2 of 314
`
`

`

`--.**"f-.
`
`P'AiIENI APFLICATION SERIAL NO.
`
`U.S. DEPARTMENT OF COMMERCE
`PA3ENT AND TRADEMARK OFFICE
`FEE RECORD SIMET
`
`$/e9/4000 $Dfiuls 000000e0 500384 095e4095
`0t F0rl0t
`690'00 Cll
`0? FC:103 630.00 CH
`
`PTO-1556
`(st87)
`
`'U.S. GPO: 1999.a59-08219144
`
`Page 3 of 314
`
`

`

`Page I of1
`
`.t
`
`UNrrEo STATES PrrEm llo ThIDEMARK OFFtcE
`\rA.sHrNGiloN, O.C. 20 e3l
`www.uspto.go/
`
`ffimffi ililtililtilililil ||ililffi ||il ililt ilil ilffi iltl
`Bib Data Sheet
`
`Chr,istine Halverson, San Jose, CA;
`Luc Julia, Menlo Park, CA;
`Dimitris Voutsas, Thessaloniki, GREECE;
`fden J. Cheyer, Palo Alto, CA;
`II
`- GPNTINUING DATA *************************
`/ THIS APPL|CATION lS A Ctp OF 09t225,198 01/p5/1999
`/ wHlcH cLAtMS BENEFTT OF 60t124,718 03/17l1999
`a wHlcH CLAIMS BENEFTT OF 60t124,719 03/1711999
`' wHtcH cLAtMS BENEFTT OF 60t124J20 03t17t1999
`
`REQUIRED, FOREIGN FILING LICENSE
`
`oreign Priority claimed E y", E no
`USC 119 (a-d) condit[ons E y", E no fl M",.0",
`Allowance
`
`and
`
`Stephens Goleman & Hughes LLP
`
`FILING FEE
`RECEIVED
`1529
`
`ES: Authority has been given in Paper
`to charge/credit DEPOSIT ACCOUNT
`
`El t.1z Fees ( Processing Ext. of
`
`E t.ta Fees ( lssue
`
`fi le ://C :\APPS\PreExam\correspondence\
`
`I _A.xml
`
`9/8t00
`
`Page 4 of 314
`
`

`

`rLT MPT
`
`CoM MtsstoNER FoR PATENTS
`UNITED STATES PATENT ANDTRADEMARK OFTICE
`\AsHlNG'IoN. O.C. 2o231
`www.uspto,go/
`
`GONFIRMATION NO.6294
`
`CLASS
`709
`
`GROUP ART UNIT
`2155
`
`ATTORNEY
`DOCKET NO.
`
`' snrt pogz
`
`ililil||rillllllllffi lllilllillllilllillllillilllilllilll illlll
`Bib Data Sheet
`
`SERIAL NUMBER .
`09/524,095
`
`FILING DATE
`03/1 3/2000
`
`RULE
`
`\PPLICAr{Ts-_-
`Christine Halverson, San Jose, CA;
`Luc Julia, Menlo Park, CA;
`Dimitris Voutsas, Thessaloniki, GREECE;
`Aeil J. Cheyer', Palo Alto, CA;
`** coNTINUING DATA *************************
`THtS APPLICATION lS A CIP OF 091225,198 01/05/1999
`wHlcH oLAIMS BENEFIT OF 601124,718 03/17l1999
`AND CLAIMS BENEFIT OF 601124,719 0311711999
`AND CLAIMS BENEFIT'OF 601124,720 0311711999
`r- FoREIGN APPLlcATloNs ********************
`tr nroulneD, FoRElcN FILING LlcENsE GMNTED** .MALL ENTtry **
`"
`r- 0511212000 ', __
`fi yu, Erno
`:oreign Priority claimed'
`SHEETS
`E r", [J no El M",.fl",
`DRAWING
`7
`
`15 USC 1 19 (a-d) conditions
`Tet
`r'erified and
`
`Allowancq
`
`I |
`
`STATE OR
`COUNTRY
`CA
`
`r ,
`
`TOTAL
`CLAIMS
`55
`
`NDEPENDENI
`CLAIMS
`3
`
`!
`
`ADDRESS
`THOMASON, MOSER & PATTERSON, LLP
`595 SHREWSBURY AVENUE
`SUITE 1OO
`SHREWSBURY,NJ 07702,
`IITLE
`rlavigating network-based etectronic information using spoken natural language input with multimodal error
`'eedback
`
`I
`FILING FEE lfees:Authority has been.given in Paper
`'iEb'errieo luo.
`to nharge/credit DEPOSIT ACCOUNT
`2141 lno.
`for following:
`
`--
`
`E nn Fees
`E t.to Fees ( Filing )
`J l.n Fees ( Processing Ext' of
`time )
`fl t.te Fees ( lssue )
`lE otn"t
`fl creoit
`
`Page 5 of 314
`
`

`

`i
`ll
`I
`I
`I
`I
`
`I
`
`I I I j
`
`.
`
`l
`
`:
`i
`
`Page I ofl
`
`L;oMMtsstoNER FoR pATENrs
`UNTTED STATEs pATErq ANoTRADeuanr orrrci
`v\AsHtNGroN, O.C. eO23l
`www.usptogo/
`
`CONFIRMATION NO.6294
`
`ATTORNEY
`DOCKET NO.
`sRtl P037
`
`Srares
`
`* BI BDATASHEET*
`
`Bib Data Sheet
`
`SERIAL NUMBER
`09/524,095
`
`Christine Halverson, San Jose, CA;
`Luc Julia, Menlo Park, CA;
`Dimitris Voutsas, Thessaloniki, GREECE:
`Aden J. Cheyer, Palo Alto, CA;
`
`GONTINUING DATA *************************
`THIS AppLtcATtoN ls A ctp oF 09t225,198 01/05/1999
`wHlcH cLAtMS BENEFTT OF 60t124,718 ogtlltlggg
`AND CLAIMS BENEFTT OF 60t124,719 o3/17l1999
`AND CLAf MS BENEFTT OF 60t124:,V20 O3t17t1ggg
`
`FOREIGN AppLlcATloNs ********************
`
`&t,,1ii[fi' F.REIGN Fll-lNG LlcENsE GRANTED..
`Priority claimed fl y", E no
`USC 119 (a-d) conditions E v". E no E M"t.ft",
`
`ancl
`
`Allorvance
`Examinefs Signature
`
`lnitials
`
`sMALL ENrlry **
`
`N, MOSER & PATTERSON, LLP
`SHREWSBURY AVENUE
`tTE 100
`REWSBURY,NJ 07702
`
`ing network-based electronic information using spoken natural language input with multimodal error
`
`FILING FEE
`RECEIVED
`2141
`
`Authority has been given,in paper
`to charge/credit DEPOS|T ACCOUNT
`for following:
`
`E t.tO Fees ( Fifing )
`1.17 Fees ( Processing Ext. of
`
`E l.te Fees ( tssue )
`
`Page 6 of 314
`
`

`

`Page I of1
`
`UoMMISSIoNER FoR PATENTS
`UNITED STATEs pATEr{T AND TRAoevmr Orrrci
`yAsHrNcloN, D.e. ZeA3l
`www.usplogo|/
`
`CONFIRMATION NO.6294
`
`GROUP ART UNIT
`2158
`
`ATTORNEY
`DOCKET NO.
`sRl1P037
`
`@urureB
`
`| |til||til||ililItiltililltililtiltilIiltilililillff ililililil]
`Bib Dhta Sheet
`
`l
`
`SER{AL NUMBER
`094524,09s
`
`i
`
`FILING DATE
`03/1 3/2000
`
`RULE
`
`Luc Julia, Menlo Park, GA;
`
`TH|S AppLlCATtoN ts A ctp oF 09t225,198 01/05
`WHICH CLATMS BENEFTT OF 60t124,718 Ogt17tt6
`AND CLATMS BENEFTT OF 60t124,719 o3t17t1ggg
`AND CLATMS BENEFTT OF 60t124,720 O3t17t1ggg
`
`999
`
`FO EIGN APPLICATIONS
`
`FtLtNc LtcENsE GRANTED..
`5$Bl)[ff,FoREtcN
`Priority claimect E ,". $ no
`U9 119 (a-d) conditions tr
`
`SMALL ENIry **
`
`I\"'
`
`KINS COIE LLP
`O JEFFERSON DRIVE
`Nf_o PARK ,CA 94025_1114
`
`ing network-based electronic information using spoken natural language input with multimodal error
`
`ll
`iii ,
`ti}."i
`FiNG FEE
`ECEIVED
`'2141
`
`: Authority has been given in paper
`to charge/credit DEPOSIT ACCOUNT
`for following:
`
`fl 1.t6 Fees
`fl t,tz Fees ( processing Ext. of
`f-tLl 1.18 Fees ( lssue
`
`Page 7 of 314
`
`

`

`Page I of I
`
`ilufl ffiilIil tilililil ||ilil flil ||ililtfl tilt iltiltilil il ||tl
`Bib Data Sheet
`
`SERIAL NUMBER
`09/524,095
`
`FILING DATE
`03t13t2000
`RULE
`
`Urrteo Srares PATENi AND ThADEMARK OFFtcE
`VlAsHrNGloN, D.C. eO23t
`www.uspto.go/
`
`CONFIRMATION NO.6294
`
`CLASS
`709
`
`GROUP ART UNIT
`2758
`
`ATTORNEY
`DOGKET NO.
`sRt1P037
`
`RffLlrrAN l5
`Christine Halverson, San Jose, CA;
`Luc Julia, Menlo Park, CA;
`Dimitris Voutsas, Thessaloniki, GREECE;
`Aden J. Cheyer, Palo Alto, CA;
`* GoNTINUING DATA *************************
`THIS AppLtCATtoN ts A ctp,oF 09t225.198 o1/05/1999
`wHlcH cLAtMS BENEFTT OF 60t124,718 03t17t1999
`wHlcH cLAtMS BENEFTT OF 60t124J19 03/17l1999
`wHlcH cLAtMS BENEFTT OF 60fi24J20 03t17t1999
`'* FOREIGN AppLlcATloNS ********************
`
`** SMALL ENTITY *"
`
`STATE OR
`COUNTRY
`CA
`
`SHEETS
`DRAWNG
`7
`
`TOTAL
`CLAIMS
`55
`
`NDEPENDENl
`CLAIMS
`3
`
`F REQUIRED, FOREIGN FILTNG LICENSE
`SRANTED ** 05t12t2000
`:oreign Priority claimed E y., E no
`15 USc 119 (a-d) conditions I u", fl no E M", ,ft",
`net
`y'erified and
`\cknowledged ffi
`?#'ffili'*'* C. Doq*t rr{5cn.(,
`IARLT.N FtELDS.Hp g* el.
`).O. BOX+aJrcg€- 3rgcl
`#,tl JOSE re/rgs trt ?ole' 16np4, FL
`rITLE
`{avigating network-based electronic information using spoken natural language input with multimodal error
`eedback
`
`Allowance
`
`t
`
`Es1,
`
`gful_ S/31
`
`FILING FEE
`RECEIVED
`2141
`
`=.EES:Authority has been given in paper
`Ito. _
`to charge/credit DEPOSTT ACCOUNT
`\o. _
`for following:
`
`E ntr""t
`E r.to Fees ( Fiting )
`,?J," Fees ( Processins Ex. of
`E t.te Fees ( tssue )
`E otn",
`E creoit
`
`Page 8 of 314
`
`

`

`Page I oi'l
`
`u*Fo t*t t***
`
`*-u* oo*ou
`
`*',='ro-- r-r,*o=
`UNTTED STATES PlreNr TUoTUDEMARK OFFtcE
`*"'-"-"';"ft1;u3353J
`
`IIu'l i
`
`l
`
`I
`I
`
`I
`
`1
`
`j
`
`l {
`
`.'"
`ti
`l,
`lli
`i:
`
`FILING DATE
`03t13t2000
`RULE
`
`Christine Halvei:son, San Jose, CA;
`Luc Julia, Menlo Park, CA;
`Dimitris Voutsas, Thessaloniki, GREECE;
`Aden J. Cheyer,:Palo Alto, CA ;
`
`CONTINUING DATA
`TH|S APPLICATTON tS A Crp OF 09t22s,198 01/05/1999
`wHlcH oLAIMS BENEFTT OF 60t124,718 03/17l1999
`wHlcH CLAIM$ BENEFTT OF 60t124,719 03117/1999
`wHlcH cLAtMS BENEFTT OF 60t124,720 03h7t1999
`'* FOREIGN APPLICATIC|NS ************r*******
`
`REQUIRED, FOREIGI.I FILING LICENSE
`LANTED "* 051121200A
`oreign Priority claimed fll uuu E[ no
`lRt T-l
`15 IJSC 119 (a-d) conditions n
`3. v:u ? ne trl ft'lt)l 3fls1
`nei
`
`** SMALL ENTIry *"
`
`NSi
`IS, WARD,'EN.IMANUEL, SMITH & CUTLER.
`6239
`L 33601-3239
`
`ating network-based electronic information using
`
`' .
`
`Y w\'t4r r,..di'i ; r.r.:..i;. I rtt-t t,
`
`t\,
`a{4C r
`
`FILING FEE
`RECEIVED
`2141
`
`:Authority has been given in Paper
`to charge/credit DEPOSIT ACCOUNT
`frrr following:
`
`I Attr*"t
`T-llJ 1.16 Fees Fillng )
`Q l.n Fees ( Processing Ext
`me)
`E t.tg pees
`
`lssue )
`
`E Credit
`
`f,rl e ://C :\APP S\PreExam\correspondence\ 1 A.xml
`
`i21t{-ttt:)r,
`
`Page 9 of 314
`
`

`

`3-lg'co
`IN rrrn uNrrno srntrs p,rtnNr.tNn tnrnnul.nr brncn
`
`, "
`
`CERTIFICATE OF E)(PRESS MAILING
`
`This hansmittal and the documeirts and/or fees itemized heneon and
`attrached hereto have been deposited as "Express Mail post Oflice to
`Addrcsee" in accordance with 37 CFR $1.10 with Mailing Iabel
`NumuerEL357581014US.
`
`Attorney Docket No.: SRI1P037
`
`First Named Inventor:
`
`HALVERSON, Christine ffi
`
`UTILITY PATENT APPLICATION TRANSMITTAL (37 CTR. $ 1.53(b)
`(Continuation, Divisional or Continuation-in-part application)
`
`Assistant Commissioner for Patents
`Box Patent Application
`Washington, DC 20231
`
`I Duplicate for
`fee processing
`
`Sir: This is a request for filing a patent application under 37 CFR. $ 1.53O) in the name of inventors:
`Christine Halverson
`
`FOT,fr NAVIGATING NETWORK.BASED ELECTRONIC III"FORMATION USING SPOKEN
`NA,T'URAL LANGUAGE INPUT WITH MULTIMODAL ERROR FEEDBACK
`i iI::i
`..;
`
`i"i,,i This application is a lJ Continuation I lOivisional
`X Continuation-in-part
`i" li
`of prior Application No.: 09/225,198, from which priority under 35 U.S.C. $ 120 is claimed.
`
`xX
`
`33 Pages of Specification, Claims and Abstact
`07 Sheets of Drawings
`Combined Declaration and Power ofAttomey
`Newly executed (original or copy)
`Copy from a prior application (37 CFR 1.63(d) for a continuation or divisional). The entire
`disclosure of the prior'application from which a copy of the declaration is herein supplied is
`considered as being part of the disclosure of the accompanying application and is hereby
`incorporated by reference therein.
`t-]
`LJ Deletion of inventors Signed stateme,nt attached deleting inventor(s) named in the prior
`application, see 37 CFR 1.63(d)(2) and 1.330).
`
`Accompanyine Application Parts :
`
`Assignment and Assignment Recordation Cover Sheet (recording fee of $40.00 enclosed)
`Power of Attomey
`37 CFR 3.73(b) Statement by Assignee
`Information Disclosure Statement with Form PTO-1449 X Copies of IDS Citations
`Page I of3
`
`X
`
`(Rwised l2l97,Pat App Trans 53(b) ContDivClp)
`
`Page 10 of 314
`
`

`

`frefminary Amendmu
`
`Jrtretv clains numbered after highest on., il claim in prior
`
`Return Receipt Postcard
`Small Entity Statonent(O X Statement filed in prior application. Status still proper and
`
`I
`application.)
`
`XX
`
`desired.I
`
`Claim For Foreign Priority
`
`Application No.
`
`filed on
`
`Priority of _
`3s u.s.c. $ 119.
`I rrr. certified copy has been filed in prior application u.s. Application No.
`f *. certified copy will follow.
`Extension of Time for Prior Pending Application
`
`is claimed under
`
`LJ A Petition for Extension of Time is being concurrently filed in the prior pending
`;;* application. A copy of the Petition for Extension of Time is attached.
`
`Arni6ndments
`i!i :i
`
`|
`I
`
`"L
`i'11
`
`l,i:
`
`:_
`,;
`
`i,:
`,:{
`
`Amend the specification by inserting before the first line the sentence: "This is a
`l-l Continuation-in-part l.l niusional
`l_l Continuation
`application of copending prior
`LJ Application No._
`l-J lnternational Application
`designated the United States,
`the disclosure of which is incorporated herein by reference.',
`
`filed on
`
`which
`
`filed on
`
`cancel in this application original claims
`of the prior application
`before calculating the filing fee. (At least one original independent claim must be retained.)
`
`Fee Calculation (37 CFR $ 1.16)
`(Col. l)
`(col.2)
`NO. FILED NO. EXTRA
`
`in the amount of $
`
`is enclosed.
`
`SMALL ENTITY OR
`RATE FEE
`$34s $34s.
`x09= $
`x39: $
`$130: $
`Total S
`
`OR
`OR
`OR
`OR
`OR
`
`LARGE ENTITY
`RATE FEE
`$690 $
`xl8: $
`x78= $
`$260 = $
`Total $
`
`BASIC FEE
`TOTAL CLAIMS
`-20 =
`INDEP CLAIMS
`-03 =
`[ ] Multiple Dependent Claim Presented
`* If the difference in Col. 1 is less
`than zero, enter "0" in Col. 2.
`l-l ctect No.
`X *. Commissioner is authorized to charge any fees beyond the amount enclosed which may be
`required, or to credit any ovqpayment, to Deposit Account No. 50-0384 (Order No. SRIlp037 ).
`Page 2 of3
`(Revised 12/97,Pat App Trans 53(b) ContDivCIP)
`
`Page 11 of 314
`
`

`

`General Authorization for Petition for Extension of Time (37 CFR $1.136)
`tr7
`l
`Applicants hereby make and generally authorize any Petitions for Extensions of Time as may be
`needed for any subsequent filings. The Commissioner is also authorized to charge any extension fees under
`37 CFR $1.17 asmaybeneededtoDepositAccountNo.50-0384(OrderNo. SRI1P037).
`
`X Please send correspondence to the following address:
`HICKMAN STEPH_ENS COLEMAN & HUGHES, LLP
`P. O. Box 52037
`Palo Alto, California 94303-0746
`(408) ss8-ees0
`
`Customer No.::
`
`ti;Dat6:
`
`ix
`
`March 13.2000
`
`D
`
`RegistuationNo. 42,762
`
`::l:
`
`l, t-
`
`(Revised 12197,Pat App Trans 53(b) ContDivCIP)
`
`Page 3 of3
`
`Page 12 of 314
`
`

`

`N.q.vrcA.rrNc Nnrwonx-B.q.srn Er,ncrnoxrc InronnnATIoN Usrxc Spornx
`NlrunLr, LlNcuacn Ixpur wrrH MulrnuooLl Ennon Fnnonlcr
`
`BacrcnouNu or rnn Ixvrnrrox
`
`This is a Continuation In Part of co-pending U.S. Patent Application No.
`
`09J225,198, filed January 5,1999, Provisional U.S. Patent Application No.
`
`601124,718, filed March 17,1999, Provisional U.S. Patent Application No.
`
`601124,720, filed March 17,1999, and Provisional U.S. Patent Application No.
`
`601124,719, filed March 17,1999, from which applications priority is claimed and
`
`these application are incorporated herein by reference.
`
`The present invention relates generally to the navigation of electronic data by
`
`means of spoken natural language requests, and to feedback mechanisms and methods
`
`for resolving the errors and ambiguities that may be associated with such requests.
`
`As global electronic connectivity continues to grow, and the universe of
`electronic data potentially available to users continues to expand, there is a growing
`
`l5
`
`need for information navigation technology that allows relatively narve users to
`
`navigate and access desired data by means of natural language input. In many of the
`most important markets - including the home entertainment arena, as well as mobile
`computing -- spoken natural language input is highly desirable, if not ideal. As just
`one example, the proliferation of high-bandwidth communications infrastructure for
`the home entertainment market (cable, satellite, broadband) enables delivery of
`
`movies-on-demand and other interactive multimedia content to the consumer's home
`
`television set. For users to take full advantage of this content stream ultimately
`
`requires interactive navigation of content databases in a manner that is too complex
`
`25
`
`for user-friendly selection by means of a traditional remote-control clicker. Allowing
`
`spoken natural language requests as the input modality for rapidly searching and
`
`accessing desired content is an important objective for a successful consumer
`
`entertainment product in a context offering a dizzytngrange of database content
`
`choices. As further examples, this same need to drive navigation of (and transaction
`
`with) relatively complex data warehouses using spoken natural language requests
`
`applies equally to surfing the InterneVlVeb or other networks for general information,
`
`multimedia content. or e-commerce transactions.
`
`Page 13 of 314
`
`

`

`ln general, the existing navigational systems for browsing electronic databases
`
`and data warehouses (search engines, menus, etc.), have been designed without
`navigation via spoken natural language as a specific goal. So today's world is fullof
`existing electronic data navigation systems that do not assume browsing via natural
`spoken commands, but rather assume text and mouse-click inputs (or in the case of
`TV remote controls, even less). Simply recognizing voice commands within an
`
`extremely limited vocabulary and grammar -- the spoken equivalent of button/click
`input (e.g., speaking "channel 5" selects TV channel 5) -- is really not sufficient by
`itself to satisfy the objectives described above. In order to deliver a true "win" for
`users, the voice-driven front-end must accept spoken natural language input in a
`
`manner that is intuitive to users. For example, the front-end should not require
`
`learning a highly specialized command language or format. More fundamentally, the
`
`front-end must allow users to speak directly in terms of what the user ultimately wants
`-- 0.g., "I'd like to see a Western film directed by Clint Eastwood" -- as opposed to
`speaking in terms of arbitrary navigation structures (e.g., hierarchical layers of menus,
`
`commands, etc.) that are essentially artifacts reflecting constraints of the pre-existing
`
`text/click navigation system. At the same time, the front-end must recognize and
`accommodate the reality that a sfieam of naive spoken natural.language input will,
`
`over time, typically present a variety of errors and/or arnbiguities: e.g.,
`
`garbled/unrecognized words (did the user say "Eastwood" or "Easter"?) and under-
`constrained requests ("Show me the Clint Eastwood movie";. An approach is needed
`for handling and resolving such errors and ambiguities in a rapid, user-friendly, non-
`
`frustrating mahner.
`
`l0
`
`15
`
`, ''!i
`
`; IT
`
`What is peeded is a methodology and apparatus for rapidly constructing a
`
`25
`
`voice-driven front-end atop an existing, non-voice data navigation system, whereby
`
`users can interact by means of intuitive natural language input not strictly conforming
`
`to the step-by-step browsing architecture of the existing navigation system, and
`
`wherein any errors or ambiguities in user input are rapidly and conveniently resolved.
`
`The solution to this need should be compatible with the constrainls of a multi-user,
`
`30
`
`distributed environment such as the Internet/Web or a proprietary high-bandwidth
`
`content delivery network; a solution contemplating one-at-a-time user interactions at a
`
`single location is insufficient, for example.
`
`a
`
`Page 14 of 314
`
`

`

`I
`
`v^t
`
`rv lr\\-
`\yw
`W
`
`Sumvrnnv or rnn InvnNrroN
`
`-\
`/
`
`'l
`
`present invention addresses the above needs byproviding a system,
`
`method,
`
`article of manufacture for navigating network-based electronic data
`
`sources m
`
`to spoken NL input requests. When a spoken natural language
`
`mput request r
`
`ived from a user, it is interpreted, such as by using a speech
`
`recogrutron
`
`natural language
`
`the spoken natural
`
`to extract speech data from acoustic voice signals, and using a
`to linguistically parse the speech data. The interpretation of
`request can be performed on a computing device locally
`
`l0
`
`with the user or remote
`
`thereupon used to
`
`the desired information
`
`from the user. The resulting interpretation of the request is
`ically construct an operational navigation query to retrieve
`one or more electronic network data sources. which is
`
`then transmitted to a client
`
`ice of the user. If the network data source is a
`
`database, the navigation query
`
`constructed in the format of a database query
`
`l5
`
`language.
`
`Typically, errors or ambigui
`request, such that the system cannot i
`
`emerge in the interpretation of the spoken NL
`
`iate a complete, valid navigational
`
`template. This is to be expected
`
`ly, and one prefened aspect of the
`
`invention is the abilitv to handle such
`
`and ambiguities in relatively graceful and
`
`20
`
`user-friendly manner. Instead of simply rej
`
`ing such input and defaulting to
`
`traditional input modes or simply asking the
`
`to try again, apreferred embodiment
`
`ofthe present invention seeks to converge rap
`
`toward instantiation of a valid
`
`navigational template by soliciting additional
`
`ion from the user as necessary,
`
`either before or after a navigation of the data
`means of menu selection or other input modalities i
`natural language. This clarifying, multi-modal d
`
`yia multimodal input, i.e., by
`
`and in addition to spoken
`
`advantage of whatever
`
`partial navigational information has been gleaned from t
`
`itial interpretation of the
`
`user's spoken NL request. This clarification process contin
`
`until the svstem
`
`converges toward an adequately instantiated navigational
`30 used to navigate the network-based data and retrieve the user's
`The retrieved information is transmitted across the network and
`
`which is in turn
`
`to the user
`
`on a suitable client display device.
`
`a
`,.1
`It
`'It
`f "
`\*li'
`
`Page 15 of 314
`
`

`

`In a further aspect of the present invention, the construction of the navigation
`
`query includes extracting an input template for an online scripted interface to the data
`
`source and using the input template to construct the navigation query. The extraction
`
`of the input template can include dynamically scraping the online scripted interface.
`
`:..i
`
`i'\
`
`I ili
`
`:l
`
`i r!1
`rl
`
`Page 16 of 314
`
`

`

`Bnmp DnscnrprroN oF rHn Dru,wrxcs
`
`The invention, together with further advantages thereof, may best
`understood by reference to the following description taken in conjunction with
`
`be
`
`the
`
`accompanying drawings in which:
`
`Figure la illushates a system providing a spoken natural language interface
`for network-based information navigation, in accordance with an embodiment of the
`
`present invention with server-side processing of requests;
`
`Figure lb illustrates another system providing a spoken natural language
`interface for network-based infonnation navigation, in accordance with an
`l0 embodiment of the present invention with client-side processing of requests;
`
`Figure 2 illustrates a system providing a spoken natural language interface for
`network-based information navigation, in accordance with an embodiment of the
`present invention for a mobile computing scenario;
`
`.
`
`Figure 3 illustrates the functional logic components of a request processing
`l5 module in accordance with an embodiment of the present invention;
`
`"J
`. .';
`til
`
`r iii
`
`i. "ii
`
`lil
`
`i*$
`
`Figure 4 illustrates a process utilizing spoken natural language for navigating
`
`an electronic database in accordance with one embodiment of the present invention;
`
`Figure 5 illushates a process for constructing a navigational query for
`accessing an online data source via an interactive, scripted (e.g., CGI) form; and
`
`Figure b iilustrates an embodiment of the present invention utilizing a
`community of distributed, collaborating electronic agents.
`
`-5-
`II /',
`l;
`!\ti
`
`Page 17 of 314
`
`

`

`Dnrnr,nn DnscnrprroN oF rnr IxvnnttoN
`
`L. System Architecture
`
`a. Server-End Processine of Spoken Input
`
`Figure la is an illustration of a data navigation system driven by spoken
`natural language input, in accordance with one embodiment of the present invention.
`As shown, a user's voice input data is captured by a voice input device 102, such as a
`
`microphone. Preferably voice input device 102 includes a button or the like that can
`be pressed or held-down to activate a listening mode, so that the slatem need not
`continually pay attention to, or be confused by, irrelevant background noise. In one
`prefened embodiment well-suited for the home entertainment setting, voice input
`device 102 is a portable remote conhol device with an integrated microphone, and the
`voice data is transmitted from device 102 preferably via infrared (or other wireless)
`link to communications box 104 (e.g., a set-top box or a similar communications
`
`device that is capable of retransmitting the raw voice data and/or processing the voice
`data) local to the user's environment and coupled to communications network 106.
`The voice data is then transmitted across network 106 to a remote server or servers
`108. The voice datamay preferably be transmitted in compressed digitized form, or
`alternatively --particularly where bandwidth constraints are significant- in analog
`format (e.g., via frequency modulated transmission), in the latter case being digitized
`
`upon arrival at remote server 108.
`
`l0
`
`l5
`
`At remote server 108, the voice data is processed by request processing logic
`300 in order to understand the user's request and construct an appropriate query or
`
`request for navigation of remote data source 110, in accordance with the interpretation
`process exemplified in Figure 4 and Figure 5 and discussed in greater detail below.
`For purposes of executing this process, request processing logic 300 comprises
`functional modules including speech recognition engine 310, natural language (NL)
`
`25
`
`parser 320, query construction logic 330, and query refinement logic 340, as shown in
`Figure 3. Data source 110 may comprise database(s), Intemet/web site(s), or other
`electronic information repositories, and preferably resides on a central server or
`servers -- which may or may not be the same as server 108, depending on the storage
`
`I;
`
`-6-
`
`Page 18 of 314
`
`

`

`and bandwidth needs of the application and the resources available to the practitioner.
`Data source 110 may include multimedia content, such as movies or other digital
`video and audio content, other various forms of entertainment data, or other electronic
`information. The contents of data source 110 are navigated -- i.e., the contents are
`accessed and searched, for retrieval of the particular information desired by the user --
`using the processes of Figures 4 and 5 as described in greater detail below.
`
`Once the desired information has been rekieved from data source 110, it is
`electronically transmitted via network 106 to the user for viewing on client display
`device ll2.In a preferred embodiment well-suited for the home entertainment setting,
`display device 112 is a television monitor or similar audiovisual entertainment device,
`typically in stationary position for comfortable viewing by users. In addition, in such
`preferred embodiment, display device 112 is coupled to or integrated with a
`communications box (which is preferably the same as communications box 104, but
`may also be a separate unit) for receiving and decoding/formatting the desired
`elechonic information that is received across communications network 106.
`
`l0
`
`l5
`
`i
`
`Network 106 is a two-way electronic communications network and may be
`embodied in electronic communication infrastructure including coanial (cable
`television) lines, DSL, fiber-optic cable, haditional copper wire (twisted pair), or any
`other tlpe of hardwired connection. Network 106 may also include a wireless
`connection such as a satellite-based connection, cellular connection, or other type of
`wireless connection. Network 106 may be part of the Internet and may support
`TCP/P communications, or may be embodied in a proprietary network, or in any
`other electronic communications network infrastructure, whether packet-switched or
`connection-oriehted. A design consideration is that network 106 preferably provide
`suitable bandwidth depending upon the nature of the content anticipated for the
`desired application.
`' b. Client-End Processine of Spoken Input
`
`Figure lb is an illustration of a data navigation system driven by spoken
`natural language input, in accordance with a second embodiment of the present
`invention. Again, a user's voice input data is captured by a voice input device 102,
`such as a microphone. h the embodiment shown in Figure lb, the voice data is
`
`-7-
`
`Yfi
`
`!" "t
`'':
`
`i. it
`
`,Y3.';
`
`i-n
`
`it rl
`
`20
`
`25
`
`Page 19 of 314
`
`

`

`transmitted from device202 to requests processing logic 300, hosted on a local speech
`processor, for processing and interpretation. In the preferred embodiment illushated
`in Figure 1b, the local speech processor is conveniently integrated as part of
`communications box 104, although implementation in a physically separate (but
`communicatively coupled) unit is also possible as will be readily apparent to those of
`skill in the art. The voice data is processed by the components of request processing
`logic 300 in order to understand the user's request and construct an appropriate query
`or request for navigation of remote data source 110, in accordance with the
`interpretation process exemplified in Figures 4 and 5 as discussed in greater detail
`below.
`
`The resulting navigational query is then transmitted electronically across
`network 106 to data source ll0, which preferably resides on a central server or
`servers 108. As in Figure la, data source 110 may comprise database(s), Intemet/web
`site(s), or other electronic information repositories, and preferably may include
`multimedia content, such as movies or other digital video and audio content, other
`various forms of entertainment data, or other electronic infomration. The contents of
`data source 110 are then navigated -- i.e., the contents are accessed and searched, for
`retrieval of the particular information desired by the user -' preferably using the
`process of Figures 4 and 5 as described in greater detail below. Once the desired
`information has been retrieved from data source 110, it is electronicallytransmitted
`via network 106 to the user for viewing on client display device 112.
`
`In ond embodiment in accordance with Figure lb and well-suited for the home
`entertainment setting, voice input device 102 is a portable remote control device with
`an integrated microphone, and the voice data is transmitted from device 102
`preferably via infrared (or other wireless) link to the local speech processor. The
`local speech processor is coupled to communications network 106, and also
`preferably to client display device 112 (especially for purposes of query refinement
`transmissions, as discussed below in connection with Figure 4, step 412), and,
`preferably may be integrated within or coupled to communications box 104. kl
`addition, especially for purposes of a home entertainment application, display device
`112 is preferably a television monitor or similar audiovisual entertainment device,
`tlpically in stationary position for comfortable viewing by users. In addition, in such
`
`10
`
`15
`
`20
`
`25
`
`i'n
`
`i:
`
`'8
`7t
`/ ,J'z
`
`tI! I
`
`Page 20 of 314
`
`

`

`preferred embodiment, display device 112 is coupled to a communications box (which
`is preferably the same as communications box 104, but may also be a physically
`separate unit) for receiving and decoding/formatting the desired electronic
`information that is received across communications network 106.
`
`Design considerations favoring senrer-side processing and interpretation of
`spoken input requests, as exemplified in Figure la, include minimizing the need to
`distribute costly computational hardware and software to all client users in order to
`perform speech and language processing. Design considerations favoring client-side
`processing, as exemplified in Figure lb, include minimizing the quantity of data sent
`upstream across the network from each client, as the speech recognition is performed
`
`before transmission across the network and only the query data and/or request needs
`
`to be sent, thus reducing the upstream bandwidth requirements.
`
`c. Mobile Client Embodiment
`
`A mobile computing embodiment of the present i