`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`SNAP EXHIBIT 1028
`Snap Inc. v. Palo Alto Research Center Inc.
`IPR2021-00987
`
`Page 1 of 13
`
`

`

`
`
`
`
`
`
`
`
`Patent Application Publication Sep. 21,2006 Sheet 1 of 7
`
`
`
`US 2006/0212408 Al
`
`
`
`
`
`OLLJaNlaguoNeoddy
`
`
`
`
`sabegnuajuoy
`
`oiweukg
`
`asudia}uy
`
`SLE
`
`
`
`SWUA}SAS
`Aoebo7
`
`9EL
`
`
`
`
`
`
`
`JASMOIGSSB[BJIAATWM
`
`
`
`
`
`BLetet-JQMA|[OO]sedojaneq»[fieJaneeeresBS
`JOSMOlGGa\\"IWLHBS=D,
`
`ChE221
`
`
`
`
`
`J8SMOIGBd10/ATWXA
`
`ootA
`
`
`
`
`
`
`
`
`
`Page 2 of 13
`
`S0PL8}u|
`
`29sn
`
`uonoY
`
`sjoalqg92t
`
`
`
`Page 2 of 13
`
`
`
`
`
`
`

`

`Patent Application Publication Sep. 21,2006 Sheet 2 of 7
`
`US 2006/0212408 Al
`
`
`
`
`
`NOKyE]U0d|JIM
`
`»AMOYs
`
`10}JuBMNOAOp‘yjuoU
`
`xéMOU1SJ8pso
`
`fenoadgdelg
`
`”
`
`¢Old
`
`Ss
`
`ayealgpue
`
`Ble
`
`Ole
`
`‘Buiem12D©}
`
`1”Buypremuoge9
`
`
`
`1081/00Jeu}
`
`Sl
`
`0c¢
`
`
`
`saaiasjeuonppe58hJamsuy
`
`‘Bumemed2ONSI
`gx9egodaysued|yousi1SON‘8L
`,oAeyUssasdas-soIas
`ppegoquaysues)TSqsuoneinjei6u05,N/
`
`JO}JIEMSead,aiqeeayBURA
`
`
`
`
`Jnosuogeinjesbueg,sno196}senauoydayay
`
`
`SSAIPpeS|104/_anaprod\/Jequinusnokexe“sepiosnodajajduoo,ZZojulsseuppe®,wihpea!\0}“*
`
`
`
`
`
`
`JNO~~
`
`aigiieaesiad1las“aseaig""yoayo*qkab
`yoBgoguaysued]\olpaauam,/PIASSaippy
`Ano10}Wemsid—_“oN
`
`
`4.09)Boras—_
`
`
`
`0«G9nokuGSul]/PUORIPPeUB26S01YANJeysued)JOaAowWpasunokoqjuemnosjunooseBuysixe
`
`yemsid‘ssaippe.pseuam‘senbas
`CHEGOHIOISUEA|ONS3ASSOPPYMN~-e-
`
`quaunounoXyeuawtmohieaul\4noksseoo:d01,/aulepoeo1,
`eoyuesINOJO}|auoydunos3260}
`
`Siu)p3aunod09,Nsrenoisa
`
`dauesaseaid“saquinu
`
`ONS@AaUrTeAoWbayNMOLY0}PSouOM,S8AaoINageuoygMeNbay
`
`0}JUBMNOKOG,aBOINasJOadAyjeuueaaeynodog,
`
`
`SNSEMEPIOTSOSIGENEAY©wosleargpuequaunsogWxweg
`
`*mauednjasof,aurppybey\ssasppyled,7~—7%
`
`
`S11S0fuawnoogWxyaie4jengald
`
`ONS@Adnmoyjojbey@””Saguayuas
`
`,}99.U09Jey)S!Coe
`
`ayinbu)0}JUEMNOA,
`onseaburigbey
`SiJ]iqsnodynoge
`
`BolNageucygMeNbey
`MOIJO)0}JUEMNOA,
`
`ysanbaiunoduodn
`
`S$}sIAesauoYd
`
`MOUJUBMNOA,
`
`1€}0809JEU}
`€10}BWOTIaMA,,
`91\asuodsayona
`
`Jeuagenio,sy
`SadIAlagDES
`al2LOdB010,
`LLf/ONSIJaMSUY
`
`
`ajdwisaziuBooesues
`
`j2uodad]0ASiu],
`jduucdaydes
`
`
`‘BAME}UBSaIdaBOISe‘paydacce
`uaeq
`
`aaeyuednod*suoyd
`sadOe$Afuo10}1$q
`
`JNOAJoyajqejeaeSI
`
`
`
`SPYJapsINOA,,
`
`ONS@ASaoIagppYy
`
`Japso0)JUeMNOAOG,
`
`
`
`»CBUIPIEAUO[2D
`
`—en
`
`0}poauam‘adINaS}auoydJojunosoe
`
`ONS@,AeurTppybey
`
`sa,
`
`SA
`
`Ole
`
`ONS9A
`
`UIeWONSAaUppybay
`
`jeuoMIppeuepeauno,,
`
`
`
`3081109JEU)$1Bul]
`
`él
`
`ole
`
`L0¢
`
`ONS8AIEROTSA
`
`Jap100}JUeMNOA,
`
`
`
`4198100JU)SI1SQ
`
`bLo
`
`vl
`
`Yo}2WON
`
`Page 3 of 13
`
`saquau’/auoydsnojetp40\A
`“~/,jtsou
`ONS@AIENQWaSpPYy
`Japs00}JUeMNOA,
`
`
`
`S30IAaS|BUCHIPpe
`
`Page 3 of 13
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`
`
`
`
`
`
`
`Patent Application Publication Sep. 21,2006 Sheet 3 of 7
`
`
`
`US 2006/0212408 Al
`
`
`
`* eyrepraad\[eseaigyoayo+ByeqJO}Auent)
`
`
`
` JNO40}eM9c€\ootsid‘sapsINOA
`
`
`oNNayayduio05oY[‘ suoneinjesBu09,
`
`
`dadBES
`
`1quewns0qyORgOOJeysuel|,tTWx/e}eqYdje4\owersnorendeH
`.,Buranent’WeMy
`
`‘woeJoyysenbay\!
`
`anan
`
`82E
`
`€‘9/4
`
`'0}paauam‘SoIA9S}\auoud10yunoase
`
`
` a“..N7iO!SSBIPPEsejeq
`
`
`
`bee>m~\a
`.meuednjesOla
`SSOIPPY}ED
`
`
`
`SSeppYplleA
`
`oog4
`
`
`
`Ooze
`
`
`
`
`
` ye50)poouom‘senbas_Inokssavoid01,/NiPaoDZee=/~~”
`
`auoydnodaseard| gaquinu
`
`
`
`jeipsoAes 7“/»éMOU\Nsequuinuauoyd\/JnoA
`
`
`
`
`ysanbayssauppy
`
`ONS®A048
`
`JapioOFUeMNOA,
`
`veutSt
`
`.J98109
`
`
`
`
`
`ada}uonoesuel|
`
`asuodseyONSed
`
`OLL
`
`NLPIEA
`
`NOILVOMddV
`
`YsAusS
`
` »ZUONSESUBL|INCA
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Soe
`
`S}eUNA
`
`[EHOgad!0AAW
`
`DS0}SUGDEN,
`
`
`
`[ELOdBINASBD1AIBS
`
`OLe
`
`Page 4 of 13
`
`Page 4 of 13
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`aa|SUaUODJOsee oorAOrrStr
`PaIWwoNeANG“a/GenjRAS|AWNINGAMOUyap“MEdaygS03aoIjayA."WAI103SAR]
`
`
`
`JIMUd1YMs$a20JdaredayauazegIU!PUR3/GnO43INOAayejOS![1MWaysAsABMS
`
`
`
`
`
`
`
`
`
`
`
`$$a201daredasayyayeniulpuk3]qGNO.QANOAa7e/O5}|JIMLUGISASpayeUOJne
`AND‘IIQUNIEAS$ILIINDAMOUYaA“IIIAUASJeday||g03aWAIay.Ny10)SARMY
`
`
`
`
`
`WabyOjsaysuesyLULA‘182Buiwuosuy
`
`
`
`"|ssaadasvajdusiuedsulasueysisse104.livsAemyy
`
`
`
`
`
`“adAsaSYdWoudpue33e4NIIeYUMNOAsplAosd[IMYY
`
`
`
`
`
`wnqsnuawYyonos4yanunuos
`
`
`
`
`
`“‘Burseaydysiuedsyum
`
`vSIA
`
`
`
`sAPMyI
`
`
`
`Siz10/puespayodsy
`
`
`
`Sptu‘S61u‘Za
`
`
`
`
`
`
`
`
`Patent Application Publication Sep. 21,2006 Sheet 4 of 7
`
`
`
`US 2006/0212408 Al
`
`
`
`
`
`
`
`
`
`
`
`
`
`“SAI95JdwoddPUeayesNdIeYUMNOAaplAgid
`
`suolbay
`
`OLP
`
`Clb
`
`bly
`
`9OLP
`
`Ocr
`
`Of
`
`
`
`cer
`
`bey
`
`LOP
`
`Page 5 of 13
`
`Page 5 of 13
`
`
`

`

`Patent Application Publication Sep. 21,2006 Sheet 5 of 7
`
`US 2006/0212408 Al
`
`500
`
`xO
`
`
`
`
`
`<import module="prompts” location="prompts.xml"/>
`
`
`
`
`
`
`
`
`
`<L-HHRREARHHARAAERARHRRRRRRAR AEH EE HHH EH aE EH BH BHR FR ER >
`<l— State Name: Greeting (PG 10 of the SRD
`->
`[<i-EnteringFrom:
`PO__
`<i- FERAA RARE ARERR HARRAH ARR REAR AR RRER AERA ERR RRR EHH EE >
`PT
`
`
`
`
`
`
`<router id="router2" xsi:type="ivr:Router'>
`
`<I-- In case Genesys lookup fails, we need to transfer to agent —>
`
`
`
`<ivalue>FALSE</rvalue>
`~
`
`
`
`
`
`
`
`<next-state refid="TransferloAgent’/>
`
`<default=route refid="/Greeting" />
`
`
`
`
`
`
` </criteria xsi:type="ivr-OperationCriteria" variable="app.region_name" op="eq">
`
`
`</condition-prompt>
`<condition-prompt>
`</criteria xsi:type="ivr:OperationCriteria” variable="app.region_name" op="ne">
`
`501
`
`510
`
`514
`
`512
`
`</grammar dimiKe =""]>
`
`
`
`
`
`PNW
`
`FIG. 5A
`
`Page 6 of 13
`
`
`
`
`
`Page 6 of 13
`
`

`

`Patent Application Publication Sep. 21,2006 Sheet 6 of 7
`
`US 2006/0212408 Al
`
`
`<criteria xsi:type="tvr:OperationCriteria" variable="gi
`
`<rvalue>1</rvalue>
`
` <router id="router2" xsi:type="iw:Router">
`
`
`
`
`
`
`
`
`
`
`
`<next-state refid="SpanishState"/>
`
`<cniteria xsi:type="Ivr:OperationCriteria" variable="noinputcount" op="eq">
`<rvalue>1</rvalue>
`
`<next-state refid="MainMenu"/>
`
`<criteria xsi:type="ivr:OperationCriteria" variable="noamtchcount" op="eq'>
`<rvalue>1</rvalue>
`
`<next-state refid="MainMenu"/>
`
`FIG. 5B
`
`534
`
`
`
`
`
`
`
`
`
`
`
`Page 7 of 13
`
`Page 7 of 13
`
`

`

`
`
`
`
`
`
`
`Patent Application Publication Sep. 21,2006 Sheet 7 of 7
`
`
`
`US 2006/0212408 Al
`
`
`“a 600
`
`601
`
`
`
`603
`
`
`
`605
`
`
`
`
`
`
`
`
`
`
`
`
`
`Specify a multimodaluserinterface
`
`
`
`
`
`
`
`
`for providing a communication
`
`
`
`
`mode between an application server
`
`
`and a user.
`
`
`
`
`
`
`
`
`
`
`
`Specify a businessrule for state
`
`
`
`
`transitions through a business service.
`
`
`
`
`
`
`
`
`
`Specify action objects for integration
`
`
`
`with business enterprises.
`
`
`
`
`
`
`FIG. 6
`
`Page 8 of 13
`
`Page 8 of 13
`
`

`

`
`
`US 2006/0212408 Al
`
`
`
`Sep. 21, 2006
`
`
`
`FRAMEWORK AND LANGUAGE FOR
`
`
`
`
`DEVELOPMENT OF MULTIMODAL
`
`
`APPLICATIONS
`
`
`BACKGROUND OF THE INVENTION
`
`
`
`
`
`
`
`
`
`1. Field of the Invention
`[0001]
`
`
`
`
`
`
`
`[0002] The present inventionrelatesto the specification of
`
`
`
`
`
`
`business interactions performed between a business cus-
`
`
`
`
`
`
`
`tomer and an interactive machine. In particular, the present
`
`
`
`
`
`
`
`invention provides a method and apparatus that provides a
`
`
`
`
`
`
`
`
`framework and language for defining and providing a com-
`
`
`
`
`
`
`puterized interactive response between multimodal users
`
`
`
`
`
`
`
`and a business enterprise based on defined businessrules.
`
`
`
`
`
`[0003]
`2. Description of the Related Art
`
`
`
`
`
`
`
`[0004]
`Interactive Voice Response (IVR) applications are
`
`
`
`
`
`
`
`often used to perform a business transaction with a caller
`
`
`
`
`
`
`
`over a telephonic connection without the need of the imme-
`
`
`
`
`
`
`
`
`
`diate presence of a business agent. In the past, IVRs have
`
`
`
`
`
`
`
`been developed using tools, programming languages, and
`
`
`
`
`
`
`Integrated Development Environments (IDEs) that have
`
`
`
`
`
`
`
`been provided by vendors and business enterprises to a
`
`
`
`
`
`
`telecommunications company which operates the IVR.
`
`
`
`
`
`
`
`
`These IDEs, tools, and languages generally provide a capa-
`
`
`
`
`
`
`
`
`
`bility to develop and create three main aspects of a VRU
`
`
`
`
`
`
`
`(Voice Response Unit) application, namely a voice user
`
`
`
`
`
`
`
`interface, business logic, and backend integration with a
`
`
`
`
`
`
`
`business enterprise. The voice user interface provides a
`
`
`
`
`
`
`mode of communication between a customer (user) and an
`
`
`
`
`
`
`
`IVR application and provides a structured flow through a
`
`
`
`
`
`
`business service to complete a business transaction. Busi-
`
`
`
`
`
`
`
`
`
`
`ness logic generally comprises a set of states and a set of
`
`
`
`
`
`
`
`
`
`rules for making transitions between states in reaction to
`
`
`
`
`
`
`customerinput. Backend integration enables information to
`flow back and forth between customer and business enter-
`
`
`
`
`
`
`
`
`
`
`prises.
`
`
`
`
`
`
`
`[0005] With the development of new technologies, such as
`
`
`
`
`
`
`
`
`
`the Internet and mobile phones having video displays, there
`
`
`
`
`
`
`
`come new possible modes of interaction between business
`
`
`
`
`
`
`and customer. A new generation of IVRs or equivalent
`
`
`
`
`
`
`
`interactive applications will need to address these new
`
`
`
`
`
`
`
`technologies and incorporate the new modesofinteraction.
`
`
`
`
`
`
`
`
`
`Several issues arise when tools for IVR development are
`
`
`
`
`
`
`
`proprietary to the vendor.First of all, such IVR applications
`
`
`
`
`
`
`
`
`are generally platform-dependent andare not portable from
`
`
`
`
`
`
`
`one platform to another. Secondly, these IVR applications
`
`
`
`
`
`
`
`
`are generally not designed to implement business logic and
`
`
`
`
`
`
`
`
`enterprise code with web applications and other recent
`
`
`
`
`
`
`
`technologies. Thirdly,
`these IVR applications cannot,
`in
`
`
`
`
`
`
`general, be implemented as multimodal applicationsinto the
`
`
`
`
`
`IVR. Multimodal applications represent a convergence of
`
`
`
`
`
`
`content—1.e., video, audio,
`text,
`images—with various
`
`
`
`
`
`
`
`
`modes of user interface interaction (web page, phone, etc.).
`
`
`
`
`
`
`
`
`Typically, multimodalinterfaces provide for user input using
`
`
`
`
`
`
`
`speech, a keyboard, keypad, mouse and/or stylus. Output is
`
`
`
`
`
`
`
`
`typically in the form of synthesized speech, audio, plain text,
`
`
`
`
`
`motion video and/or graphics, etc.
`
`
`
`
`
`
`
`[0006] Prior approaches to TVR development use one
`
`
`
`
`
`
`
`
`framework for creating the view components (which gen-
`
`
`
`
`
`
`
`
`
`erate dialog to interact with customers) and another for
`
`
`
`
`
`
`
`developing the business logic components (state manage-
`
`
`
`
`
`
`
`
`ment rules for providing the business service). Thus, a
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`language is used creating the components that
`different
`
`
`
`
`
`
`provide state management
`than for developing business
`
`
`
`
`
`
`
`
`logic. Often, view logic and business logic are tightly
`
`
`
`
`
`
`
`coupled andthere is no clear separation of the two within the
`
`
`
`
`
`framework. Also,
`applications
`created
`using
`prior
`
`
`
`
`
`
`approaches are typically single-mode applications, so that
`
`
`
`
`
`
`they are either [VR-only or web-only applications.
`
`
`
`
`
`
`
`
`[0007] Recently, there has been an effort to adopt a stan-
`
`
`
`
`
`
`
`dard programming language for voice applications. Voice
`
`
`
`
`
`
`Extensible Markup Language, which is also referred to as
`
`
`
`
`
`VoiceXML or VXML,is a standard established by the World
`
`
`
`
`
`
`
`
`Wide Web Consortium (W3C) standards body. The current
`
`
`
`
`
`
`generation of VXML, VXML 2.0, provides a standard
`
`
`
`
`
`
`
`
`languagethat facilitates the interactions between human and
`
`
`
`
`
`
`
`
`machine that
`traditionally have been provided by voice
`
`
`
`
`response applications, such as IVRs.
`
`
`
`
`
`[0008] VXML describes a human-machine interaction
`
`
`
`
`
`
`
`provided by voice response systems, which includes output
`
`
`
`
`
`
`
`of synthesized speech (text-to-speech), output of audiofiles,
`
`
`
`
`
`
`
`
`recognition of spoken input, recognition of DTMF input,
`
`
`
`
`
`
`
`
`
`recording of spoken input, control of dialog flow, and
`
`
`
`
`
`
`
`telephony features such as call
`transfer and disconnect.
`
`
`
`
`
`
`
`VXMLprovides meansfor collecting character and/or spo-
`
`
`
`
`
`
`
`
`ken input, assigning the input results to document-defined
`
`
`
`
`
`
`
`
`the
`request variables, and making decisions that affect
`
`
`
`
`
`
`
`interpretation of documents written in the language. A
`
`
`
`
`
`
`
`document may be linked to other documents through Uni-
`
`
`
`
`versal Resource Identifiers (URIs).
`
`
`
`
`
`
`
`[0009] VXMLpartially solves the portability problems of
`
`
`
`
`
`
`vendor-based IVR development by providing standards for
`
`
`
`
`
`
`
`
`basic IVR functions. VXMLseparates user interaction code
`
`
`
`
`
`
`
`
`
`
`(in VXML)from service logic (e.g. CGI scripts). But while
`
`
`
`
`
`
`VoiceXMLstrives to accommodate the requirements of a
`
`
`
`
`
`
`
`majority of voice response services, services with stringent
`
`
`
`
`
`
`requirements may best be served by dedicated applications
`
`
`
`
`
`
`
`
`
`that employ a finer level of control. Also, VXML is not
`
`
`
`
`
`
`intended for intensive computation, database operations, or
`
`
`
`
`
`
`
`legacy system operations. These are assumed to be handled
`
`
`
`
`
`
`
`
`by resources outside the documentinterpreter, e.g. a docu-
`
`
`
`
`
`
`
`
`mentserver. General service logic, state management, dialog
`
`
`
`
`
`
`
`generation, and dialog sequencing are assumed to reside
`
`
`
`
`
`
`
`outside the document
`interpreter. VXML 2.0 does not
`
`
`
`
`
`
`
`address issues of IVR development such as the creation of
`
`
`
`
`
`
`
`
`services that provide business logic, the creation of services
`
`
`
`
`
`
`
`
`that provide backend integration, and the dynamic creation
`
`
`
`
`of dialog specification at runtime.
`
`
`
`
`
`
`
`
`[0010] There is a need for a single framework that pro-
`
`
`
`
`
`
`vides a standard method of creating platform independent
`
`
`
`
`
`
`
`
`
`
`services that provide business logic for the IVR and for other
`
`
`
`
`
`
`
`enterprise applications, e.g., web applications. Also, there is
`
`
`
`
`
`
`
`
`a need for a standard method of defining business rules
`
`
`
`
`
`
`
`
`within services that can be shared, used, and interpreted by
`
`
`
`
`
`
`
`any modeofuser interface, be it speech (VXML), keyboard
`
`
`
`
`
`
`
`
`(HTML), or keypad (WML), etc. Also, there is a need fora
`
`
`
`
`
`
`
`
`
`standard method of defining view logic that can be used and
`
`
`
`
`
`
`
`interpreted by any modeofuserinterface, a standard method
`
`
`
`
`
`
`
`
`
`of accessing and using enterprise data to create services that
`
`
`
`
`
`
`
`
`provide enterprise business rules and logic, and a single
`
`
`
`
`
`
`
`methodology, language and environment that integrates the
`
`
`
`
`
`above requirements into one framework.
`
`Page 9 of 13
`
`Page 9 of 13
`
`

`

`
`
`US 2006/0212408 Al
`
`
`
`Sep. 21, 2006
`
`
`
`
`
`
`
`
`
`detailed description thereof that follows may be better
`
`
`
`
`
`
`
`understood andin order that the contributions they represent
`
`
`
`
`
`
`
`
`
`to the art may be appreciated. There are, of course, addi-
`tional features of the invention that will be described here-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`inafter and which will form the subject of the claims
`
`
`appendedhereto.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`
`
`
`
`
`
`
`
`
`[0016] For a detailed understanding of the present inven-
`
`
`
`
`
`
`tion, references should be made to the following detailed
`
`
`
`
`description of an exemplary embodiment, taken in conjunc-
`
`
`
`
`
`
`
`tion with the accompanying drawings, in which like ele-
`
`
`
`
`
`
`ments have been given like numerals.
`
`
`
`
`
`
`[0017] FIG. 1 illustrates an apparatus suitable for imple-
`
`
`
`
`
`menting an example of the present invention;
`
`
`
`
`
`
`[0018] FIG. 2 illustrates a state diagram representation of
`
`
`
`
`
`
`
`
`an exemplary multimodal application that can be imple-
`
`
`
`
`
`
`mented using an example of the present invention;
`
`
`
`
`
`
`[0019]
`FIG.3 illustrates an Application Server and vari-
`
`
`
`
`
`ous states of a business service in an example ofthe present
`
`invention;
`
`
`
`
`
`
`
`[0020] FIG. 4 illustrates an exemplary interface for a
`
`
`
`
`
`typical software developmentinterface of the example of the
`
`
`present invention;
`
`
`
`
`
`
`[0021]
`FIG.5 illustrates examples of software code cor-
`
`
`
`
`
`
`responding to the entries in the software developmentinter-
`
`
`
`face of FIG. 4; and
`
`
`
`
`
`
`[0022] FIG. 6 illustrates a flowchart by which the present
`
`
`
`
`invention provides a multimodal application in an example
`of the invention.
`
`
`
`SUMMARY OF THE INVENTION
`
`
`
`
`
`
`
`
`
`
`
`[0011] The present invention provides a method and appa-
`
`
`
`
`
`
`
`ratus that provide a framework for specifying a multimodal
`
`
`
`
`application in a communication network. A framework is
`
`
`
`
`
`
`
`provided that defines a metalanguage that enables a pro-
`
`
`
`
`
`
`grammerto specify an interactive application. The program-
`
`
`
`
`
`
`
`
`mer can specify a multimodal user interface for user inputto
`
`
`
`
`
`
`
`the interactive application. The programmer can specify
`
`
`
`
`
`
`
`
`
`business rules that act on a user input. The programmer can
`
`
`
`
`
`
`
`
`also specify an interface between the application and a
`
`
`
`
`
`
`
`business enterprise. The present invention enables a pro-
`
`
`
`
`
`
`
`grammer to specify a multimodal user interface of the
`
`
`
`
`
`
`
`
`multimodal application that provides view logic for provid-
`
`
`
`
`
`
`
`
`ing communication modes such as, a voice response unit, a
`
`
`
`
`
`
`
`
`textual web interface, or a video display. The response
`
`
`
`
`
`
`
`communication mode to the user can be automatically
`
`
`
`
`
`
`
`determined by the application and can be different from the
`
`
`
`input communication mode.
`
`
`
`
`
`
`
`
`[0012] The business rules comprise business logic and
`
`
`
`
`
`
`
`generally enable transitions between states of a business
`
`
`
`
`
`
`
`
`service in response to user input. The programmer also
`
`
`
`
`
`
`
`specifies how the application interacts with a business
`
`
`
`
`
`
`
`
`enterprise system or database. Also, user input can be stored
`
`
`
`
`
`
`
`in a database associated with a business enterprise. The
`
`
`
`
`
`
`
`programmerspecifies a response to the user input in accor-
`
`
`
`
`
`
`
`
`
`dance with the business rules to provide the multimodal
`
`application.
`
`
`
`
`
`
`
`In one aspect of the present invention a computer-
`[0013]
`
`
`
`
`
`
`
`
`ized method and apparatus are provided for providing an
`
`
`
`
`
`
`application in a communication network. The method and
`
`
`
`
`
`
`
`apparatus provides for receiving a first programmer input
`
`
`
`
`
`
`
`specifying a user interface for a user communication with
`
`
`
`
`
`
`
`the application, receiving a second programmerinput speci-
`
`
`
`
`
`
`
`
`fying a business rule in the application that acts on a user
`
`
`
`
`
`
`
`
`
`input from the user interface and receiving a third program-
`
`
`
`
`
`
`
`mer input specifying an interaction between the application
`
`
`
`
`
`
`
`
`and an enterprise system. The user interface further com-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`In view of the above, the present invention through
`[0023]
`prises a view logic for a mutimodal communication mode.
`
`
`
`
`
`
`
`
`
`
`
`
`one or more of its various aspects and/or embodiments is
`The multimodal communication mode further comprises at
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`presented to provide one or more advantages, such as those
`least one of the set consisting of a web browser and a cell
`noted below.
`
`
`
`
`
`
`
`
`phone. A metalanguage is provided to indicate a code
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`segment to specify a view, action or routing to a newstate
`VWUDFprovides a single integrated frameworkthat
`[0024]
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`in the application. The business rule providesat least one of
`covers all aspects of a customeror user-oriented application.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the set consisting of a transition betweenstates of a business
`Only one framework is used for developing the view com-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`service, and a transfer of information between the user and
`ponents, the business logic components andthe data/system
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`a database. The methodfurther provides for specifyinga first
`integration components of the application. The VWDFpro-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`communication mode for the user input and specifying a
`vides a single language and a single framework for defining
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`second communication modefora transmitting a response to
`the business logic, view logic and the data access logic for
`the user.
`
`
`
`
`
`
`
`
`
`multimodal applications. With VWDF, there is clear sepa-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`ration of the view logic components and business logic
`[0014]
`In another aspect of the invention a set of applica-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`components, while keeping the two within the same frame-
`tion program interfaces are provided embodied on a com-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`work.It also performs computation, database operation and
`puter readable medium for execution on a computer in
`
`
`
`
`
`
`
`
`
`
`
`
`legacy systems operations for business rule interpretation
`conjunction with an application program in a communica-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`within the single framework. Since the same language is
`tion network comprising a first interface that receivesafirst
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`used for developing any mode of user interface,
`is a
`it
`programmer input specifying a user interface for a user
`
`
`
`
`
`
`
`
`
`multimodal application.
`communication with the application, a second interface that
`
`
`
`
`
`
`
`
`
`
`
`
`receives a second programmerinput specifying a business
`[0025] Use of the VWDF improves the System Develop-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`rule in the application that acts on a user input from the user
`ment Life Cycle by providing an enabling concurrent code
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`interface anda third interface that receives a third program-
`development and a direct traceability of application code
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`mer input specifying an interaction between the application
`with the user specification and system requirement. Devel-
`
`
`
`
`
`
`
`
`
`and an enterprise system.
`opers can work concurrently on different parts of the appli-
`
`
`
`
`
`
`
`
`
`
`
`
`
`[0015] Examples of certain features of the invention have
`cation without concern for being out of sync. Developers can
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`been summarized here rather broadly in order that
`assemble the states together at a later time, or they can even
`the
`
`
`
`
`
`
`
`DETAILED DESCRIPTION OF THE
`
`
`INVENTION
`
`
`
`
`Page 10 of 13
`
`
`
`
`
`
`
`
`
`
`
`
`Page 10 of 13
`
`

`

`
`
`US 2006/0212408 Al
`
`
`
`Sep. 21, 2006
`
`
`
`
`
`
`
`
`
`
`
`
`
`test
`the states running on each others machine by just
`
`
`
`
`
`
`
`
`
`pointing their routers to the machine of another. For
`
`
`
`
`
`
`
`
`example, Developer A located in Chicago can use the state
`
`
`
`
`
`
`defined components of Developer B located at St. Louis by
`
`
`
`
`
`
`merely using the URI of the component of Developer B.
`
`
`
`
`
`
`[0026] FIG. 1 illustrates a conceptual hardware imple-
`
`
`
`
`
`
`
`
`mentation 100 for providing the present
`invention. The
`
`
`
`
`
`
`present example of the invention is presented as a Voice Web
`
`
`
`
`
`
`Development Framework (VWDF) that provides an open
`
`
`
`
`
`
`
`platform, single integrated environment and language for
`
`
`
`
`
`
`
`defining service logic, state management, dialog generation,
`
`
`
`
`
`
`
`
`and dialog sequencing for IVR, Web, or any multimodal
`
`
`
`
`
`
`
`application. FIG. 1 depicts the VWDF and shows how the
`
`
`
`
`
`
`
`different components of the application can be separated
`
`
`
`
`
`
`
`within a single framework. An Application Server 110
`
`
`
`
`
`
`
`comprises a Finite State Machine 112 for implementing
`
`
`
`
`
`
`
`
`business logic and a View Manager 114 for providing
`
`
`
`
`
`
`business services to a customer. A Framework Authoring
`
`
`
`
`
`
`Tool 120 provides an interactive development environment
`
`
`
`
`
`
`
`
`through which a developer can supply logical code objects
`
`
`
`
`
`
`for implementation on the Application Server 110.
`
`
`
`
`
`
`
`[0027] The Framework Authoring Tool comprises an
`
`
`
`
`
`
`
`
`
`interface to View Logic 120 for specifying user interface
`
`
`
`
`
`
`
`
`
`logic, Router Logic 122 for providing business rules and
`
`
`
`
`
`
`
`logic for transitioning to different states of an application,
`
`
`
`
`
`
`
`
`
`and Action Objects 124 which provide access to and inte-
`
`
`
`
`
`
`
`
`gration with backend systems(i.e. database, legacy systems
`
`
`
`
`
`
`
`and business enterprises). VWDF provides a single meta-
`
`
`
`
`
`
`
`
`
`
`language for defining the states that contain the view logic
`
`
`
`
`
`
`
`
`
`(the components that provide interaction with the user), the
`
`
`
`
`
`
`
`
`business logic (the component that contains the business
`
`
`
`
`
`
`
`
`
`
`rules for the application) and the systems and data access
`
`
`
`
`
`
`
`
`logic (the component that provides integration with enter-
`
`
`
`
`
`
`
`prise legacy systems, e.g., database, customer management
`
`
`
`
`
`
`
`
`systems or ordering systems). The defined states are com-
`bined into a Finite State Machine which interacts with Data
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`116 and Enterprise Legacy Systems 118 to store and produce
`
`
`
`
`
`
`
`data usable in a customer interaction, i.e. billing informa-
`
`
`
`
`
`
`
`
`tion, address information, etc. The View Manager 114
`
`
`
`
`
`
`
`
`interacts with the Finite State Machine and Enterprise
`
`
`
`
`
`
`
`
`Legacy Systems 118 and provides a mode for user interac-
`
`
`
`
`
`
`
`
`
`tion. The VWDFcreates a system that interacts with a user
`
`
`
`
`
`
`
`
`in a communication mode appropriate to the user. For
`
`
`
`
`
`
`
`
`example, HTTP for web users and WAPfor cell phoneusers.
`
`
`
`
`
`
`
`
`[0028] Dynamic page content is provided from the Appli-
`
`
`
`
`
`
`
`
`cation Serverto a user over a multimodalinterface 140 using
`
`
`
`
`
`
`
`
`one of several possible modes. For example, a Voice
`
`
`
`
`
`
`
`
`Browser 132 enables a voice interaction using VXMLcode,
`
`
`
`
`
`
`
`
`
`a Web Browser 134 enables web interaction through Hyper-
`
`
`
`
`
`
`
`Text Markup Language (HTML) code, or a Wireless
`
`
`
`
`
`
`
`Browser 136 enables an interaction using Wireless Markup
`
`
`
`
`
`
`Language (WML) code. Browsers may be accessed in a
`
`
`
`
`
`
`
`single mode or in a combination of modes. A user can
`
`
`
`
`
`
`
`
`interact with the Application Server using any available
`
`
`
`
`
`
`
`
`interface mode (cell phone, web, legacy telephone (plain
`
`
`
`
`
`
`
`ordinary telephone service—POTS)). The number of modes
`
`
`
`
`
`
`
`shownin the present invention is for illustrative purposes,
`and the numberof interface modes is not limited to those
`
`
`
`
`
`
`
`
`modeslisted herein.
`
`
`
`
`
`
`
`
`
`[0029] FIG. 2 illustrates a state diagram representation
`
`
`
`
`
`
`
`200 of an exemplary voice response application that can be
`
`
`
`
`
`
`
`implemented using the present invention. VWDF uses a
`
`
`
`
`
`
`
`
`state model to represent a business application. The states
`
`
`
`
`
`
`
`
`maybe defined in any language such as XML. The under-
`
`
`
`
`
`
`
`lying language is hidden from the programmerspecifying
`
`
`
`
`
`
`
`
`the states. The programmeruses the metalanguage provided
`
`
`
`
`
`
`
`
`instead of any particular vender or platform specific lan-
`
`
`
`
`
`
`
`
`
`guage. Each defined state is converted into real time objects
`
`
`
`
`
`
`
`
`
`on the Application Server 110. The resultant Finite State
`
`
`
`
`
`
`Machine model of the application is accessible through a
`
`
`
`
`
`
`
`mode of the multimodal user interface. Business logic
`
`
`
`
`
`
`
`
`
`determines the flow of the user through the state diagram
`
`
`
`
`
`
`
`
`200 in response to user input. A user may “enter”the state
`
`
`
`
`
`
`
`diagram at 201 andtransition to one of several accessible
`
`
`
`
`
`
`
`
`
`
`states based on the user response to a prompt by the
`
`
`
`
`
`
`
`Application Server. The example of FIG. 2 enables a
`
`
`
`
`
`
`
`customer to obtain a phone service from a telephone com-
`
`
`
`
`
`
`
`
`pany. States accessible from state 201 enable a customerto
`
`
`
`
`
`
`
`
`
`obtain new phoneservice 210, to obtain an addition line 212,
`
`
`
`
`
`
`
`
`
`to order DSL (Digital Subscriber Line) service 214, to order
`
`
`
`
`
`
`
`
`additional services like Call Waiting, Call Forwarding,etc.
`
`
`
`
`
`
`
`
`216, to inquire abouta bill 218, and to follow up on a request
`
`
`
`
`
`
`
`220. Arepromptstate is activatedif there is no user response
`within a set amount of time 222. If there is no match between
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`a user response and available state selections, the user is
`
`
`
`
`
`
`
`
`
`
`returned to entry state 201. The state transition that
`is
`
`
`
`
`
`
`performed depends on the value of the user input.
`
`
`
`
`
`
`
`
`[0030] FIG. 3 illustrates an Application Server 110 and
`
`
`
`
`
`
`
`
`various states of a business service. Each state is imple-
`
`
`
`
`
`
`
`
`mented on the Application Server 300 and comprises View
`
`
`
`
`
`
`
`
`Logic 303 implementing a user interface, Router Logic 305
`
`
`
`
`
`
`
`implementing rules for transitioning through business ser-
`
`
`
`
`
`
`
`
`vice, and system integration logic (Action) 301 providing
`
`
`
`
`
`
`
`access to a business enterprise. State 310 is a greeting for a
`
`
`
`
`
`
`
`customer and requests a customer to choose an option for
`
`
`
`
`
`
`
`proceeding. States perform a variety of functions from
`
`
`
`
`
`
`
`
`prompting a user for input and obtains a user response to
`
`
`
`
`
`
`obtaining accessing backend enterprise information. For
`
`
`
`
`
`
`
`
`
`
`example, state 320 prompts the user with the phrase “You
`
`
`
`
`
`
`
`
`
`
`want to order
`, is that correct?” and the user replies
`
`
`
`
`
`
`
`
`
`
`with either a “Yes” or a “No”. State 322 and state 324 gather
`
`
`
`
`
`
`
`
`
`information from a user. State 322 prompts the user for a
`
`
`
`
`
`
`telephone number, and receives a telephone number in
`
`
`
`
`
`
`
`
`
`response. State 324 prompts the user for an address and
`
`
`
`
`
`
`
`
`receives an address in response. State 326 obtains data from
`
`
`
`
`
`
`
`
`a database. Such information may indicate, for example,
`
`
`
`
`
`
`
`whether DSL service is available for a given telephone
`
`
`
`
`
`
`numberor address. State 328 provides a confirmation to a
`
`
`
`
`
`
`
`
`user and places the user call in a queue.
`
`
`
`
`
`
`[0031] FIG. 4 illustrates an exemplary metalanguage
`
`
`
`
`
`
`
`
`
`interface 400 of the present example of the invention,
`
`
`
`
`
`VWDFfor a software development interface. A title 401
`
`
`
`
`
`
`
`
`(“Greeting’’) for the state is displayed. The screen indicates
`
`
`
`
`
`
`
`
`
`a Default Routing state 440 (“Transfer to Agent’) and any
`
`
`
`
`
`
`
`
`
`
`prior states 445 from whichthe currentstate is accessed(e.g.
`
`
`
`
`
`
`
`
`“Incoming Call”). In the example, the “Greeting” state is
`
`
`
`
`
`
`
`
`activated by an incoming call. Sections 410 and 420 pr