in a step 906. The p . t facilitator responds to this service r . t and at a later
`
`time, the client agent receives the results of the request in a step 908, operations of
`
`Figure 9 being complete in a done step 910.
`)
`FIGURE 10 depicts operations involved in a client agent responding to a
`
`5
`
`service request in accordance with a preferred embodiment of the present invention.
`
`Once started in a step 1000, the client agent receives the service request in a step
`
`1002. In a next step 1004, the client agent parses the received request from ICL. The
`
`client agent then determines if the service is available in a step 1006. If it is not, the
`
`client agent returns a status report to that effect in a step 1008. If the service is
`
`10
`
`available, control is passed to a step 1010 where the client performs the requested
`
`service. Note that in completing step 1010 the client may form complex goal
`
`expressions, requesting results for these solvables from the facilitator agent. For
`
`example, a fax agent might fax a document to a certain person only after requesting
`
`and receiving a fax number for that person. Subsequently, the client agent either
`
`15
`
`returns the results of the service and/or a status report in a step 1012. The operations
`
`of Figure 10 are complete in a done step 1014.
`
`FIGURE 11 depicts operations involved in a facilitator agent response to a
`
`service request in accordance with a preferred embodiment of the present invention.
`
`The start of such operations in step 1100 leads to the reception of a goal request in a
`
`20
`
`step 1102 by the facilitator. This request is then parsed and interpreted by the
`
`facilitator in a step 1104. The facilitator then proceeds to construct a goal satisfaction
`
`plan in a next step 1106. In steps 1108 and 1110, respectively, the facilitator
`
`determines the required sub-goals and then selects agents suitable for performing the
`
`required sub-goals. The facilitator then transmits the sub-goal requests to the selected
`
`25
`
`agents in a step 1112 and receives the results of these transmitted requests in a step
`
`1114. It should be noted that the actual implementation of steps 1112 and 1114 are
`
`dependent upon the specific goal satisfaction plan. For instance, certain sub-goals
`
`may be sent to separate agents in parallel, while transmission of other sub-goals may
`
`be postponed until receipt of particular answers. Further, certain requests may
`
`30
`
`generate multiple responses that generate additional sub-goals. Once the responses
`
`have been received, the facilitator determines whether the original requested goal has
`
`been completed in a step 1118. If the original requested goal has not been completed,
`
`Attornev Docket No: SRI I PO 1604 77)/BRC/EW J
`
`Pa!!e 28 of 59
`
`Page 127 of 778
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 4256
`
`

`

`the facilitator recurs~ repeats the operations 1106 through • . Once the original
`
`requested goal is completed, the facilitator returns the results to the requesting agent
`
`1118 and the operations are done at 1120.
`
`A further preferred embodiment of the present invention incorporates
`
`5
`
`transparent delegation, which means that a requesting agent can generate a request,
`
`and a facilitator can manage the satisfaction of that request, without the requester
`
`needing to have any knowledge of the identities or locations of the satisfying agents.
`
`In some cases, such as when the request is a data query, the requesting agent may also
`
`be oblivious to the number of agents involved in satisfying a request. Transparent
`
`10
`
`delegation is possible because agents' capabilities (solvables) are treated as an abstract
`
`description of a service, rather than as an entry point into a library or body of code.
`
`A further preferred embodiment of the present invention incorporates
`
`facilitator handling of compound goals, preferably involving three types of
`
`processing: delegation, optimization and interpretation.
`
`15
`
`Delegation processing preferably supports facilitator determination of which
`
`specific agents will execute a compound goa1 and how such a compound goal's sub(cid:173)
`
`goals will be combined and the sub-goal results routed. Delegation involves selective
`
`application of global and local constraint and advice parameters onto the specific sub(cid:173)
`
`goals. Delegation results in a goal that is unambiguous as to its meaning and as to the
`
`~
`
`20
`
`agents that will participate in satisfying it.
`
`Optimization processing of the completed goal preferably includes the
`
`facilitator using sub-goal parallelization where appropriate. Optimization results in a
`
`goal whose interpretation will require as few exchanges as possible, between the
`
`facilitator and the satisfying agents, and can exploit parallel efforts of the satisfying
`
`25
`
`agents, wherever this does not affect the goal's meaning.
`
`Interpretation processing of the optimized goal. Completing the addressing of
`
`a goal involves the selection of one or more agents to handle each of its sub-goals
`
`(that is, each sub-goal for which this selection has not been specified by the
`
`requester). In doing this, the facilitator uses its knowledge of the capabilities of its
`
`30
`
`client agents (and possibly of other facilitators, in a multi-facilitator system). It may
`
`also use strategies or advice specified by the requester, as explained below. The
`
`Attornev Dockel No: SRIIP016(3477)/BRCIEWJ
`
`Pae:e 29 of 59
`
`Page 128 of 778
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 4257
`
`

`

`interpretation of a .nvolves the coordination of requests . e satisfying agents,
`
`and assembling their responses into a coherent whole, for return to the requester.
`
`A further preferred embodiment of present invention extends facilitation so the
`
`facilitator can employ strategies and advice given by the requesting agent, resulting in
`
`5
`
`a variety of interaction patterns that may be instantiated in the satisfaction of a
`
`request.
`
`A further preferred embodiment of present invention handles the distribution
`
`of both data update requests and requests for installation of triggers, preferably using
`
`some of the same strategies that are employed in the delegation of service requests.
`
`10
`
`Note that the reliance on facilitation is not absolute; that is, there is no hard
`
`requirement that requests and services be matched up by the facilitator, or that
`
`interagent communications go through the facilitator. There is preferably support in
`
`the agent library for explicit addressing of requests. However, a preferred
`
`embodiment of the present invention encourages employment the paradigm of agent
`
`15
`
`communities, minimizing their development effort, by taking advantage of the
`
`facilitator's provision of transparent delegation and handling of compound goals.
`
`A facilitator is preferably viewed as a coordinator, not a controller, of
`
`cooperative task completion. A facilitator preferably never initiates an activity. A
`
`facilitator preferably responds to requests to manage the satisfaction of some goal, the
`
`20
`
`update of some data repository, or the installation of a trigger by the appropriate agent
`
`or agents. All agents can preferably take advantage of the facilitator's expertise in
`
`delegation, and its up-to-date knowledge about the current membership of a dynamic
`
`community. The facilitator's coordination services often allows the developer to
`
`lessen the complexity of individual agents, resulting in a more manageable software
`
`25
`
`development process, and enabling the creation of lightweight agents.
`
`Maintaining Data Repositories
`
`The agent library supports the creation, maintenance, and use of databases, in
`
`the form of data solvables. Creation of a data solvable requires only that it be
`
`declared. Querying a data solvable, as with access to any solvable, is done using
`
`30
`
`oaa_Solve.
`
`Attornev Docket No: SRIIPOI6(3477l!BRC/EWJ
`
`Pae:e 30 of 59
`
`Page 129 of 778
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 4258
`
`

`

`A data solva.s conceptually similar to a relation in .ational database.
`
`The facts associated with each solvable are maintained by the agent library, which
`
`also handles incoming messages containing queries of data solvables. The default
`
`behavior of an agent library in managing these facts may preferably be refined, using
`
`5
`
`parameters specified with the solvable's declaration. For example, the parameter
`
`single_value preferably indicates that the solvable should only contain a single fact at
`
`any given point in time. The parameter unique_values preferably indicates that no
`
`duplicate values should be stored.
`
`Other parameters preferably allow data solvables use of the concepts of
`
`10
`
`ownership and persistence. For implementing shared repositories, it is often
`
`preferable to maintain a record of which agent created each fact of a data solvable
`
`with the creating agent being preferably considered the fact's owner. In many
`
`applications, it is preferable to remove an agent's facts when that agent goes offline
`
`(for instance, when the agent is no longer participating in the agent community,
`
`15
`
`whether by deliberate termination or by malfunction). When a data solvable is
`
`declared to be non-persistent, its facts are automatically maintained in this way,
`
`whereas a persistent data solvable preferably retains its facts until they are explicitly
`
`removed.
`
`A further preferred embodiment of present invention supports an agent library
`
`20
`
`through procedures by which agents can update (add, remove, and replace) facts
`
`belonging to data solvables, either locally or on other agents, given that they have
`
`preferably the required permissions. These procedures may preferably be refined
`
`using many of the same parameters that apply to service requests. For example, the
`
`address parameter preferably specifies one or more-particular agents to which the
`
`25
`
`update request applies. In its absence, just as with service requests, the update request
`
`preferably goes to all agents providing the relevant data solvable. This default
`
`behavior can be used to maintain coordinated "mirror" copies of a data set within
`
`multiple agents, and can be useful in support of distributed, collaborative activities.
`
`Similarly, the feedback parameters, described in connection with oaa_Solve,
`
`30
`
`are preferably available for use with data maintenance requests.
`
`n =
`
`lf
`
`ltd
`:;;p, w
`
`$
`
`1:::":
`
`Fi
`""?'
`;f!
`;!d
`~n
`<;:;;:
`
`Attornev Docket No: SRIIP016(3477)/BRCfEWJ
`
`Paee 31 of 59
`
`Page 130 of 778
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 4259
`
`

`

`A further p.ed embodiment of present invention slorts ability to
`
`provide data solvables not just to client agents, but also to facilitator agents. Data
`
`solvables can preferably created, maintained and used by a facilitator. The facilitator
`
`preferably can, at the request of a client of the facilitator, create, maintain and share
`
`5
`
`the use of data solvables with all the facilitator's clients. This can be useful with
`
`relatively stable collections of agents, where the facilitator's workload is predictable.
`
`Using a Blackboard Style of Communication
`
`In a further preferred embodiment of present invention, when a data solvable
`
`10
`
`is publicly readable and writable, it acts essentially as a global data repository and can
`
`be used cooperatively by a group of agents. In combination with the use of triggers,
`
`this allows the agents to organize their efforts around a "blackboard" style of
`
`communication.
`
`As an example, the "DCG-NL" agent (one of several existing natural language
`
`15
`
`processing agents), provides natural language processing services for a variety of its
`
`peer agents, expects those other agents to record, on the facilitator, the vocabulary to
`
`which they are prepared to respond, with an indication of each word's part of speech,
`
`and of the logical form (ICL sub-goal) that should result from the use of that word. In
`
`a further preferred embodiment of present invention, the NL agent, preferably when it
`
`20
`
`comes online, preferably installs a data solvable for each basic part of speech on its
`
`facilitator. For instance, one such solvable would be:
`
`solvable(noun(Meaning, Syntax), [], [])
`
`Note that the empty lists for the solvable's permissions and parameters are acceptable
`
`here, since the default permissions and parameters provide appropriate functionality.
`
`25
`
`A further preferred embodiment of present invention incorporating an Office
`
`Assistant system as discussed herein or similar to the discussion here supports several
`
`agents making use of these or similar services. For instance, the database agent uses
`
`the following call, to library procedure oaa_AddData, to post the noun 'boss', and to
`
`indicate that the "meaning" of boss is the concept 'manager':
`
`30
`
`oaa_AddData( noun(manager, atom(boss)), [ address(parent)])
`
`Auornev Docket No: SRII PO 16(3477)/BRCIEWJ
`
`Pa!!:e 32 of 59
`
`Page 131 of 778
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 4260
`
`

`

`Autonomous Mon.g with Triggers
`
`A further preferred embodiment of present invention includes support for
`
`triggers, providing a general mechanism for requesting some action be taken when a
`
`set of conditions is met. Each agent can preferably install triggers either locally, for
`
`5
`
`itself, or remotely, on its faci_litator or peer agents. There are preferably at least four
`
`types of triggers: communication, data, task, and time. In addition· to a type, each
`
`trigger preferably specifies at least a condition and an action, both preferably
`
`expressed in /CL. The condition indicates under what circumstances the trigger should
`
`fire, and the action indicates what should happen when it fires. In addition, each
`
`10
`
`trigger can be set to fire either an unlimited number of times, or a specified number of
`
`times, which can be any positive integer.
`
`Triggers can be used in a variety of ways within _preferred embodiments of the
`
`present invention. For example, triggers can be used for monitoring external sensors
`
`in the execution environment, tracking the progress of complex tasks, or coordinating
`
`15
`
`communications between agents that are essential for the synchronization of related
`
`tasks. The installation of a trigger within an agent can be thought of as a
`
`representation of that agent's commitment to carry out the specified action, whenever
`
`the specified condition holds true.
`
`Communication triggers preferably allow any incoming or outgoing event
`
`20
`
`(message) to be monitored. For instance, a simple communication trigger may say
`
`something like: "Whenever a solution to a goal is returned from the facilitator, send
`
`the result to the presentation manager to be displayed to the user."
`
`Data triggers preferably monitor the state of a data repository (which can be
`
`maintained on a facilitator or a client agent). Data triggers' conditions may be tested
`
`25
`
`upon the addition, removal, or replacement of a fact belonging to a data solvable. An
`
`example data trigger is: "When 15 users are simultaneously logged on to a machine,
`
`send an alert message to the system administrator."
`
`Task triggers preferably contain conditions that are tested after the processing
`
`of each incoming event and whenever a timeout occurs in the event polling. These
`
`30
`
`conditions may specify any goal executable by the local /CL interpreter, and most
`
`often are used to test when some solvable becomes satisfiable. Task triggers are
`
`Attornev Docket No: SRIIPOI6(3477VBRC/EWJ
`
`Pa!!e 33 of 59
`
`g
`k!
`.~
`Vi
`\0
`
`Page 132 of 778
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 4261
`
`

`

`useful in checking fo.k-specific internal conditions. Althou.n many cases such
`
`conditions are captured by solvables, in other cases they may not be. For example, a
`
`mail agent might watch for new incoming mail, or an airline database agent may
`
`monitor which flights will arrive later than scheduled. An example task trigger is:
`
`5
`
`"When mail arrives for me about security, notify me immediately."
`
`Time triggers preferably monitor time conditions. For instance, an alarm
`
`trigger can be set to fire at a single fixed point in time (e.g., "On December 23rd at
`
`3pm"), or on a recurring basis (e.g., "Every three minutes from now until noon").
`
`Triggers are preferably implemented as data solvables, declared implicitly for
`
`10
`
`every agent. When requesting that a trigger be installed, an agent may use many of the
`
`same parameters that apply to service and data maintenance requests.
`
`A further preferred embodiment of present invention incorporates semantic
`
`support, in contrast with most programming methodologies, of the agent on which the
`
`trigger is installed only having to know how to evaluate the conditional part of the
`
`15
`
`trigger, not the consequence. When the trigger fires, the action is delegated to the
`
`facilitator for execution.· Whereas many commercial mail programs allow rules of the
`
`form "When mail arrives about XXX, [forward it, delete it, archive it]", the possible
`
`actions are hard-coded and the user must select from a fixed set.
`
`U'1
`
`A further preferred embodiment of present invention, the consequence of a
`
`20
`
`trigger may be any compound goal executable by the dynamic community of agents.
`
`Since new agents preferably define both functionality and vocabulary, when an
`
`unanticipated agent (for example, a fax agent) joins the community, no modifications
`
`to existing code is required for a user to make use of it - "When mail arrives, fax it to
`
`Bill Smith."
`
`25
`
`The Agent Library
`
`In a preferred embodiment of present invention, the agent library provides the
`
`infrastructure for constructing an agent-based system. The essential elements of
`
`protocol (involving the details of the messages that encapsulate a service request and
`
`30
`
`its response) are preferably made transparent to simplify the programming
`
`applications. This enables the developer to focus functionality, rather than message
`
`Attornev Docket No: SRIIPOI6(3477VBRCIEWJ
`
`Pa!!e 34 of 59
`
`Page 133 of 778
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 4262
`
`

`

`construction details .omrnunication details. For example, .quest a service of
`
`another agent, an agent preferably calls the library procedure oaa_Solve. This call
`
`results in a message to a facilitator, which will exchange messages with one or more
`
`service providers, and then send a message containing the desired results to the
`
`5
`
`requesting agent. These results are returned via one of the arguments of oaa_Solve.
`
`None of the messages involved in this scenario is explicitly constructed by the agent
`
`developer. Note that this describes the synchronous use of oaa_Solve.
`
`In another preferred embodiment of present invention, an agent library
`
`provides both intraagent and interagent infrastructure; that is, mechanisms supporting
`
`10
`
`the internal structure of individual agents, on the one hand, and mechanisms of
`
`cooperative interoperation between agents, on the other. Note that most of the
`
`infrastructure cuts across this boundary with many of the same mechanisms
`
`supporting both agent internals and agent interactions in an integrated fashion. For
`
`example, services provided by an agent preferably can be accessed by that agent
`
`15
`
`through the same procedure (oaa_Solre) that it would employ to request a service of
`
`another agent (the only difference being in the address parameter accompanying the
`
`request). This helps the developer to reuse code and avoid redundant entry points into
`
`the same functionality.
`
`Both of the preferred characteristics described above (transparent construction
`
`20
`
`of messages and integration of intraagent with interagent mechanisms) apply to most
`
`other library functionality as well, including but not limited to data management and
`
`temporal control mechanisms.
`
`Source Code Appendix
`
`Source code for version 2.0 of theOAA software product is included as an
`
`25
`
`appendix hereto, and is incorporated herein by reference. The code includes an agent
`
`library, which provides infrastructure for constructing an agent-based system. The
`
`library's several families of procedures provide the functionalities discussed above, as
`
`well as others that have not been discussed here but that will be sufficiently clear to
`
`the interested practitioner. For example, declarations of an agent's solvables, and their
`
`30
`
`registration with a facilitator, are managed using procedures such as oaa_Declare,
`
`oaa_Undeclare, and oaa_Redeclare. Updates to data solvables can be accomplished
`
`with a family of procedures including oaa_AddData, oaa_RemoveData, and
`
`Anornev Docket No: SRIIP016(3477)/BRC/EWJ
`
`Paee 35 of 59
`
`Page 134 of 778
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 4263
`
`

`

`oaa_ReplaceData. S.arly, triggers are maintained using pr.res such as
`
`oaa_AddTrigger, oaa_RemoveTrigger, and oaa_ReplaceTrigger. The provided
`
`source code also includes source code for an OAA Facilitator Agent.
`
`The source code appendix is offered solely as a means of further helping
`
`5
`
`practitioners to construct a preferred embodiment of the invention. By no means is
`
`the source code intended to limit the scope of the present invention.
`
`Illustrative Applications
`
`To further illustrate the technology of the preferred embodiment, we will next
`
`present and discuss two sample applications of the present inventions.
`
`10 Unified Messaging
`
`A further preferred embodiment of present invention incorporates a Unified
`
`.Q
`
`Messaging application extending the Automated Office application presented
`
`previously herein with an emphasis on ubiquitous access and dynamic presentation of
`
`the information and services supported by the agent community. The agents used in
`
`15
`
`this application are depicted in Figure 12.
`
`A hypothetical example of realistic dialog using a preferred embodiment of
`
`the present invention can provide insight into how systems may preferably be built
`
`using the present invention. In this scenario, the user, with only a telephone as an
`
`interface, is planning a trip to Boston where he will soon give a presentation.
`
`\!d
`
`20
`
`Capitalized sentences are phrases spoken by the user into the telephone and processed
`
`by a phone agent 452.
`
`Responses, unless otherwise indicated, are spoken by the system using text-to(cid:173)
`
`speech generation agent 454.
`
`25
`
`1.1 Welcome to SRI International. Please enter your
`user ID and password.
`<User enters touchtone ID and password>
`
`Good to see you again Adam Cheyer.
`
`I am
`
`listening to you.
`
`30
`
`Every user interface agent 408, including the telephone agent 452, should
`
`know the identify of its user. This information is used in resolving anaphoric
`
`Attornev Docket No: SRI I PO 16(34 77)/BRC/EWJ
`
`Page 36 of 59
`
`Page 135 of 778
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 4264
`
`

`

`:.A
`;;;:
`
`rll
`fiJ
`
`~fi
`'iofi
`l '
`.r-
`\g
`t'fi
`·:.r.::
`
`references such as ··and "I", and allows multiple user inte.s operated by the
`
`same user to work together.
`
`5
`
`1.2 WHAT IS TODAY'S SCHEDULE?
`Here is today's schedule for Adam Cheyer:
`At lOam for 1 hour, meeting with Dave.
`At 3prn for 1 hour, presentation about software
`agents.
`
`End of schedule.
`
`If the user is operating both a graphical user interface and a telephone, as
`
`10
`
`described in conjunction with the Automated Office application, the result of this
`
`spoken request is to display a calendar window containing the current schedule. In
`
`this case, with no graphical display available, the GEN_NL agent 1202 is tasked to
`
`produce a spoken response that can be played over the phone. GEN_NL shares the
`
`same dynamic vocabulary and phrasal rules as the natural language parser DCG_NL
`
`15
`
`426, and contains strategies for producing responses to queries using either simple or
`
`list-based multimedia utterances.
`
`1.3 FIND FRIDAY'S WEATHER IN BOSTON.
`The weather in Boston for Friday is as follows:
`Sunny in the morning. Partly cloudy in the
`afternoon with a 20
`percent chance of thunderstorms late. Highs in the
`mid 70s.
`
`In addition to data accessible from legacy applications, content may be
`
`retrieved by web-reading agents which provide wrappers around useful websites.
`
`1.4 FIND ALL NEW MAIL MESSAGES.
`There are 2 messages available.
`Message 1, from Mark Tierny, entitled 11 0AA meeting."
`1.5 NEXT MESSAGE
`Message 2, from Jennifer Schwefler, entitled
`II Presentation Summary. II
`1. 6
`PLAY IT.
`This message is a multipart MIME-encoded message.
`There are two parts.
`Part 1.
`(Voicernail message, not text-to speech):
`Thanks for taking part as a speaker in our
`conference.
`The schedule will be posted soon on our homepage.
`1.7 NEXT PART
`Part 2. (read using text-to-speech):
`The presentation horne page is http://www ....
`1.8 PRINT MESSAGE
`Command executed.
`
`20
`
`25
`
`30
`
`35
`
`40
`
`Auornev Docket No: SRIIP0160477)/BRCIEWJ
`
`Pa2e 37 of 59
`
`Page 136 of 778
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 4265
`
`

`

`Mail messalre no longer just simple text docume.ut often consist of
`
`multiple subparts containing audio files, pictures, webpages, attachments and so forth.
`
`When a user asks to play a complex email message over the telephone, many different
`
`agents may be implicated in the translation process, which would be quite different
`
`5
`
`given the request "print it." The challenge is to develop a system which will enable
`
`agents to cooperate in an extensible, flexible manner that alleviates explicit coding of
`
`agent interactions for every possible input/output combination.
`
`In a preferred embodiment of the present invention, each agent concentrates
`
`only on what it can do and on what it knows, and leaves other work to be delegated to
`
`10
`
`the agent community. For instance, a printer agent 1204, defining the solvable
`
`print(Object,Parameters), can be defined by the following pseudo-code, which
`
`basically says, "If someone can get me a document, in either POSTSCRIPT or text
`
`form, I can print it.".
`
`15
`
`print(Object, Parameters) {
`I If Object is reference to "it", find an appropriate
`document
`if (Object = "ref(it) ")
`oaa_Solve(resolve_reference(the, document, Params,
`20 Object) , [] ) ;
`1 Given a reference to some document, ask for the
`document in POSTSCRIPT
`if (Object= "id(Pointer)")
`oaa_Solve(resolve_id_as(id(Pointer), postscript,
`[], Object),[]);
`1 If Object is of type text or POSTSCRIPT, we can
`print it.
`if ((Object is of type Text) or (Object is of type
`Postscript) )
`do_print (Object);
`
`25
`
`30
`
`}
`
`In the above example, since an email message is the salient document, the
`
`mail agent 442 will receive a request to produce the message as POSTSCRIPT.
`
`Whereas the mail agent 442 may know how to save a text message as POSTSCRIPT,
`
`35
`
`it will not know what to do with a webpage or voicemail message. For these parts of
`
`the message, it will simply send oaa_Solve requests to see if another agent knows
`
`how to accomplish the task.
`
`Attornev Docket No: SRIIP016(3477)!BRCIEWJ
`
`Page 38 of 59
`
`Page 137 of 778
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 4266
`
`

`

`Until now, .ser has been using only a telephone a . r interface. Now, he
`
`moves to his desktop, starts a web browser 436, and accesses the URL referenced by
`
`5
`
`the mail message.
`1.9 RECORD MESSAGE
`Recording voice message. Start speaking now.
`1.10 THIS IS THE UPDATED WEB PAGE CONTAINING THE
`PRESENTATION SCHEDULE.
`Message one recorded.
`1.11 IF THIS WEB PAGE CHANGES, GET IT TO ME WITH NOTE
`10 ONE.
`
`Trigger added as requested.
`
`i:j
`
`rl!
`
`~-1;
`
`In this example, a local agent 436 which interfaces with the web browser can
`
`return the current page as a solution to the request "oaa_Solve(resolve_reference(this,
`
`web_page, [],Ref),[])", sent by the NL agent 426. A trigger is installed on a web
`
`15
`
`agent 436 to monitor changes to the page, and when the page is updated, the notify
`
`agent 446 can find the user and transmit the webpage and voicemail message using
`
`the most appropriate media transfer mechanism.
`
`This example based on the Unified Messaging application is intended to show
`
`how concepts in accordance with the present invention can be used to produce a
`
`20
`
`simple yet extensible solution to a multi-agent problem that would be difficult to
`
`implement using a more rigid framework. The application supports adaptable
`
`presentation for queries across dynamically changing, complex information; shared
`
`context and reference resolution among applications; and flexible translation of
`
`multimedia data. In the next section, we will present an application which highlights
`
`25
`
`the use of parallel competition and cooperation among agents during multi-modal
`
`fusion.
`
`Multimodal Map
`
`A further preferred embodiment of present invention incorporates the
`
`Multimodal Map application. This application demonstrates natural ways of
`
`30
`
`communicating with a community of agents, providing an interactive interface on
`
`which the user may draw, write or speak. In a travel-planning domain illustrated by
`
`Figure 13, available information includes hotel, restaurant, and tourist-site data
`
`retrieved by distributed software agents from commercial Internet sites. Some
`
`preferred types of user interactions and multimodal issues handled by the application
`
`Attornev Docket No: SRIIPOJ6(3477VBRC/EWJ
`
`Pae:e 39 of 59
`
`Page 138 of 778
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 4267
`
`

`

`are illustrated by a b.cenario featuring working examples •
`
`from the current
`
`system.
`
`Sara is planning a business trip to San Francisco, but would like to schedule
`
`some activities for the weekend while she is there. She turns on her laptop PC,
`
`2.3
`
`executes a map application, and selects San Francisco.
`2.1
`[Speaking] Where is downtown?
`Map scrolls to appropriate area.
`[Speaking and drawing region] Show me all hotels
`2.2
`near here.
`Icons representing hotels appear.
`[Writes on a hotel] Info?
`A textual description (price, attributes, etc.)
`appears.
`2.4
`[Speaking] I only want hotels with a pool.
`Some hotels disappear.
`[Draws a crosscut on a hotel that is too close to a
`2.5
`highway]
`Hotel disappears
`[Speaking and circling] Show me a photo of this
`
`2.7
`
`2.8
`
`2.9
`
`2.10
`
`5
`
`10
`
`15
`
`25
`
`30
`
`w
`
`= -::~
`
`~
`
`2.6
`20 hotel.
`Photo appears.
`[Points to another hotel]
`Photo appears.
`[Speaking] Price of the other hotel?
`Price appears for previous hotel.
`[Speaking and drawing an arrow] Scroll down.
`Display adjusted.
`[Speaking and drawing an arrow toward a hotel]
`What is the distance from this hotel to Fisherman's
`Wharf?
`Distance displayed.
`[Pointing to another place and speaking] And the
`2.11
`distance to here?
`Distance displayed.
`
`35
`
`Sara decides she could use some human advice. She picks up the phone, calls
`
`Bob, her travel agent, and writes Start collaboration to synchronize his display with
`
`hers. At this point, both are presented with identical maps, and the input and actions
`
`of one will be remotely seen by the other.
`
`40
`
`3.1
`
`[Sara speaks and circles two hotels]
`Bob, I'm trying to choose between these two hotels.
`Any opinions?
`3.2
`[Bob draws an arrow, speaks, and points]
`Well, this area is really nice to visit. You can
`45 walk there from
`
`Auornev Docket No: SRilPOI6(3477)/BRCIEWJ
`
`Pa!!:e 40 of 59
`
`Page 139 of 778
`
`Petitioner Microsoft Corporation - Ex. 1008, p. 4268
`
`

`

`this ho • .
`Map scrolls to indicated area. Hotel selected.
`[Sara speaks] Do you think I should visit Alcatraz?
`[Bob speaks] Map, show video of Alcatraz.
`Video appears.
`[Bob speaks] Yes, Alcatraz is a lot of fun.
`
`3.3
`3.4
`
`3.5
`
`5
`
`A further preferred embodiment of present invention generates the most
`
`appropriate interpretation for the incoming streams of multiinodal input. Besides
`
`providing a user interface to a dynamic set of distributed agents, the application is
`
`10
`
`preferably built using an agent framework. The present invention also contemplates
`
`aiding the coordinate competition and cooperation among information sources, which
`
`in tum works in parallel to resolve the ambiguities arising at every level of the
`
`interpretation process: low-level processing of the data stream, anaphora resolution,
`
`cross-modality influences and addressee.
`
`15
`
`Low-level processing of the data stream: Pen input may be preferably
`
`interpreted as a gesture (e.g., 2.5: cross-out) by one algorithm, or as handwriting by a
`
`separate recognition process (e.g., 2.3: "info?"). Multiple hypotheses may preferably
`
`be returned by a modality recognition component.
`
`0
`
`20
`