`
`
`
`
`
`
`
`
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`____________________
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`____________________
`
`GOOGLE LLC
`Petitioner
`
`v.
`
`IPA TECHNOLOGIES INC.
`Patent Owner
`
`____________________
`
`Patent No. 6,757,718
`____________________
`
`DECLARATION OF DR. DAN R. OLSEN JR.
`
`
`
`
`
`GOOGLE EXHIBIT 1002
`
`Page 1 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`
`TABLE OF CONTENTS
`
`TABLE OF CONTENTS ........................................................................................... i
`I.
`INTRODUCTION .......................................................................................... 1
`II.
`BACKGROUND AND QUALIFICATIONS ................................................ 1
`III.
`SUMMARY OF OPINIONS .......................................................................... 4
`IV. PERSON OF ORDINARY SKILL IN THE ART ......................................... 7
`V.
`TECHNICAL BACKGROUND .................................................................... 8
`A. Natural language processing ................................................................ 8
`B. Multimodal input ................................................................................ 11
`C.
`Databases ............................................................................................ 13
`D. Distributed computing ........................................................................ 16
`E. Mobile computing .............................................................................. 21
`VI. OVERVIEW OF THE ’718 PATENT ......................................................... 22
`VII. CLAIM CONSTRUCTION ......................................................................... 27
`A.
`“navigation query” ............................................................................. 28
`B.
`“code segment [that] . . . ” .................................................................. 28
`C.
`“. . . logic[,] operable to . . . ” ............................................................. 30
`VIII. OVERVIEW OF THE PRIOR ART ............................................................ 31
`A.
`Cheyer................................................................................................. 31
`B.
`Shwartz ............................................................................................... 34
`C.
`Johnson ............................................................................................... 37
`D.
`Thrift ................................................................................................... 39
`E.
`Dureau ................................................................................................ 41
`F.
`Simmers .............................................................................................. 43
`
`
`
`
`
`i
`
`
`
`Page 2 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`
`
`IX. THE PRIOR ART DISCLOSES OR SUGGESTS ALL OF THE
`FEATURES OF CLAIMS 1-27 OF THE ’718 PATENT ............................ 43
`A.
`Cheyer, Shwartz, and Thrift Disclose or Suggest the Features of
`Claims 1-4, 6, 8-10, 12, 13, 15, 17-19, 21, 22, 24, 26, 27 ................. 43
`1.
`Claim 1 ..................................................................................... 44
`2.
`Claims 2 and 3 .......................................................................... 74
`3.
`Claim 4 ..................................................................................... 77
`4.
`Claim 6 ..................................................................................... 82
`5.
`Claims 8, 9 ............................................................................... 83
`6.
`Claim 10 ................................................................................... 85
`7.
`Claim 12 ................................................................................... 92
`8.
`Claim 13 ................................................................................... 92
`9.
`Claim 15 ................................................................................... 94
`10. Claims 17, 18 ........................................................................... 95
`11. Claim 19 ................................................................................... 95
`12. Claim 21 ................................................................................. 102
`13. Claim 22 ................................................................................. 102
`14. Claim 24 ................................................................................. 104
`15. Claims 26, 27 ......................................................................... 105
`Cheyer, Shwartz, Thrift, and Dureau Disclose or Suggest the
`Features of Claims 2, 11, and 20 ...................................................... 105
`1.
`Claim 2 ................................................................................... 105
`2.
`Claim 11 ................................................................................. 110
`3.
`Claim 20 ................................................................................. 111
`
`B.
`
`
`
`
`
`ii
`
`
`
`Page 3 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`
`C.
`
`Cheyer, Shwartz, Thrift, and Johnson Disclose or Suggest the
`Features of Claims 4, 13, and 22 ...................................................... 111
`4.
`Claim 4 ................................................................................... 111
`5.
`Claim 13 ................................................................................. 119
`6.
`Claim 22 ................................................................................. 119
`Cheyer, Shwartz, Thrift, and Simmers Disclose or Suggest the
`Features of Claims 5, 7, 14, 16, 23, and 25 ...................................... 120
`1.
`Claims 5, 7 ............................................................................. 120
`2.
`Claims 14, 16, 23, 25 ............................................................. 126
`CONCLUSION ........................................................................................... 127
`
`D.
`
`iii
`
`
`
`
`
`X.
`
`
`
`
`
`
`
`Page 4 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`
`I, Dr. Dan R. Olsen Jr., declare as follows:
`
`I.
`
`INTRODUCTION
`1.
`I have been retained by Google LLC (“Petitioner”) as an independent
`
`expert consultant in this proceeding before the United States Patent and Trademark
`
`Office (“PTO”) regarding U.S. Patent No. 6,757,718 (“the ’718 patent”) (Ex.
`
`1001). I have been asked to consider whether certain references disclose or
`
`suggest the features recited in claims 1-27 (“the challenged claims”) of the ’718
`
`patent. My opinions are set forth below.
`
`2.
`
`I am being compensated at my rate of $500 per hour for the time I
`
`spend on this matter. My compensation is in no way contingent on the nature of
`
`my findings, the presentation of my findings in testimony, or the outcome of this or
`
`any other proceeding. I have no other interest in this proceeding.
`
`II. BACKGROUND AND QUALIFICATIONS
`3.
`I have more than 35 years of experience in computer science and
`
`human-computer interaction (HCI). I hold a doctorate in Computing and
`
`Information from the University of Pennsylvania. For 3 ½ years I was an Assistant
`
`Professor of Computer Science at Arizona State University. I then served for 30
`
`years on the faculty of Brigham Young University, retiring as a full professor in
`
`2015. During that time at BYU, I also served as the chair of the Department of
`
`
`
`
`
`1
`
`
`
`Page 5 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`Computer Science. I took leave from BYU in 1996 to become the founding
`
`director of the Human Computer Interaction Institute in the School of Computer
`
`Science at Carnegie Mellon University. I returned to BYU in 1998. I am currently
`
`the CEO of a software startup in educational technology (SparxTeq, Inc).
`
`4.
`
`During the course of my academic career, I authored over 70 papers in
`
`the field of computer science. The topics on which I have published papers
`
`include: User Interface Management Systems; Interaction over the Internet;
`
`Syntactic representations of user interfaces; Multi-user interaction across networks;
`
`Induction of interaction behavior from pictures; Novel interaction techniques using
`
`laser pointers; Structure of speech-based interaction and integration of speech with
`
`other forms of interaction; Interactive machine learning; Interactive robotics; and
`
`Interactive television.
`
`5.
`
`I have extensive experience with graphical user interfaces that are
`
`driven by communications-based technologies. Out of my last 70+ published
`
`papers, 14 have involved development of custom network protocols to allow
`
`devices to interact and access information. In addition, there are 6 papers that
`
`explicitly address speech interaction and the integration of other interactive
`
`modalities with speech.
`
`
`
`
`
`2
`
`
`
`Page 6 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`I currently hold 4 patents in human-computer interaction. I have
`
`6.
`
`authored 3 textbooks on the techniques of software design for human-computer
`
`interaction.
`
`7.
`
`I have had extensive involvement in professional societies, such as the
`
`Association for Computing Machinery (ACM), the premier society in computing.
`
`I have served in many offices of ACM’s Special Interest Group on Computer
`
`Human Interaction (SIGCHI) and currently serve as its treasurer. I have been
`
`conference chair of CHI, which is the premier conference in Computer Human
`
`Interaction. I was the founding editor of ACM’s Transactions on Computer
`
`Human Interaction. I was a co-founder and active leader for the conference on
`
`User Interface Software and Technology (UIST) for the past 29 years. I have also
`
`served at the governor’s request on the Utah Science, Technology and Research
`
`(USTAR) board, which oversees and funds state economic development efforts in
`
`technology.
`
`8.
`
`I twice received best paper awards in intelligent user interfaces. In
`
`2004, I was appointed to the CHI Academy for international excellence in
`
`Computer Human Interaction research. In 2007, I was recognized as one of
`
`ACM’s Fellows for research in computer science and in 2012 received the CHI
`
`
`
`
`
`3
`
`
`
`Page 7 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`Lifetime Research Award, which is the highest award in Computer Human
`
`Interaction.
`
`9.
`
`I understand that a copy of my curriculum vitae, which includes a
`
`more detailed summary of my background, experience, and publications, is
`
`provided as Ex. 1003.
`
`III. SUMMARY OF OPINIONS1
`10. The opinions contained in this Declaration are based on the
`
`documents I reviewed, my professional judgment, as well as my education,
`
`experience, and knowledge regarding graphical user interfaces.
`
`11.
`
`In forming my opinions expressed in this Declaration, I reviewed the
`
`’718 patent (Ex. 1001); the prosecution file history for the ’718 patent (Ex. 1004);
`
`U.S. Patent Nos. 6,742,021 (Ex. 1005) and 6,851,115 (Ex. 1007), which I
`
`understand are in the chain of applications from which the ’718 patent claims
`
`priority, and their respective prosecution histories (Exs. 1006, 1008); U.S.
`
`Provisional Application Nos. 60/124,718 (Ex. 1009), 60/124,719 (Ex. 1010), and
`
`
`1 My citations to non-patent publications are to the original page numbers of the
`
`publication, and my citations to U.S. Patents are to the column:line number or
`
`paragraph number of the patents or published patent applications, as applicable.
`
`
`
`
`
`4
`
`
`
`Page 8 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`60/124,720 (Ex. 1011), to which I understand the ’718 patent claims priority;
`
`Cheyer et al., “Multimodal Maps: An Agent-based Approach,” published in Proc.
`
`of the International Conference on Cooperative Multimodal Communication
`
`(CMC/95), Eindhoven, The Netherlands, May 1995(“Cheyer”) (Ex. 1012); U.S.
`
`Patent No. 5,197,005 to Shwartz et al. (“Shwartz”) (Ex. 1013); U.S. Patent No.
`
`5,748,974 to Johnson (“Johnson”) (Ex. 1014); U.S. Patent No. 6,188,985 to Thrift
`
`et al. (“Thrift”) (Ex. 1015); U.S. Patent No. 6,345,389 to Dureau (“Dureau”) (Ex.
`
`1016); U.S. Patent No. 5,841,431 to Simmers (“Simmers”) (Ex. 1017); U.S. Patent
`
`No. 6,035,197 to Haberman et al. (“Haberman”) (Ex. 1018); Coen, M. H.,
`
`“Building Brains
`
`for Rooms: Designing Distributed Software Agents,”
`
`AAAI’97/IAAI’97 Proceedings of the Fourteenth National Conference on
`
`Artificial Intelligence and Ninth Conference on Innovative Applications of
`
`Artificial Intelligence (1997) (“Coen”) (Ex. 1020); Hodjat et al., “An adaptive
`
`agent oriented software architecture,” in Lee et al. (eds.) PRICAI’98: Topics in
`
`Artificial Intelligence, Lecture Notes in Computer Science (Lecture Notes in
`
`Artificial Intelligence), vol 1531, Springer, Berlin, Heidelberg (1998) (“Hodjat”)
`
`(Ex. 1021); U.S. Patent No. 5,584,024 to Shwartz (“Shwartz-024”) (Ex. 1022);
`
`Cheyer et al., “MVIEWS: Multimodal Tools for the Video Analyst,” in
`
`Proceedings of the 1998 International Conference on Intelligent User Interfaces
`
`
`
`
`
`5
`
`
`
`Page 9 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`(IUI98), San Francisco, California (Jan. 1998) (Ex. 1023); Kehler et al., “On
`
`Representing Salience and Reference
`
`in Multimodal Human-Computer
`
`Interaction,” in Proceedings of AAAI 1998 workshop on Representations for Multi-
`
`Modal Human-Computer Interaction, Madison, Wisconsin (1998) (Ex. 1024);
`
`Cohen et al., “An Open Agent Architecture,” in Proceedings AAAI Spring
`
`Symposium, Stanford, California (March 1994) (“Cohen”) (Ex. 1025); Martin et
`
`al., “Information brokering in an agent architecture,” in Proceedings of the Second
`
`International Conference on the Practical Application of Intelligent Agents and
`
`Multi-Agent Technology, Blackpool, Lancashire, UK (Apr. 1997) (“Martin”) (Ex.
`
`1026); Wyard et al., “Spoken language systems – beyond prompt and response,”
`
`BT Technol. J. vol. 14 no. 1 (Jan. 1996) (“Wyard”) (Ex. 1027); Excerpts from
`
`Knaster, Presenting Magic Cap, A Guide to General Magic’s Revolutionary
`
`Communicator Software, 1994 (Ex. 1028); Moran et al., “Multimodal User
`
`Interfaces in the Open Agent Architecture,” Proc. of the 2nd International
`
`Conference on Intelligent User Interfaces (IUI ’97), Orlando, Florida (1997) (Ex.
`
`1029); Konstan, J. A., “State Problems in Programming Human-Controlled
`
`Devices,” IEEE Transactions on Consumer Electronics, vol. 40, no. 4 (Nov. 1994)
`
`(“Konstan”) (Ex. 1033); and any other materials I refer to in this Declaration in
`
`support of my opinions.
`
`
`
`
`
`6
`
`
`
`Page 10 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`12. My opinions have also been guided by my appreciation of how a
`
`person of ordinary skill in the art would have understood the claims and the
`
`specification of the ’718 patent at the time of the alleged invention, which I have
`
`been asked to initially consider as the 1999 time frame, including and up to the
`
`March 17, 1999 date which the ’718 patent claims as priority date. My opinions
`
`reflect how one of ordinary skill in the art would have understood the ’718 patent,
`
`the prior art to the patent, and the state of the art at the time of the alleged
`
`invention.
`
`13. As I discuss in detail below, it is my opinion that certain references
`
`disclose or suggest all the features recited in claims 1-27 of the ’718 patent.
`
`IV. PERSON OF ORDINARY SKILL IN THE ART
`14. Based on my knowledge and experience, I understand what a person
`
`of ordinary skill in the art would have known at the time of the alleged invention.
`
`My opinions herein are, where appropriate, based on my understandings as to a
`
`person of ordinary skill in the art at that time. In my opinion, based on the
`
`materials and information I have reviewed, and based on my experience in the
`
`technical areas relevant to the ’718 patent, a person of ordinary skill in the art at
`
`the time of the alleged invention of the ’718 patent would have had at least a
`
`Bachelor’s degree in computer science, electrical engineering, or a similar
`
`
`
`
`
`7
`
`
`
`Page 11 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`discipline, and one to two years of work experience in user interfaces for computer
`
`systems (including speech-based interfaces), networked computer systems, or a
`
`related area. More education can substitute for practical experience and vice versa.
`
`I apply this understanding in my analysis herein.
`
`15. My analysis of the ’718 patent and my opinions in this declaration are
`
`from the perspective of one of ordinary skill in the art, as I have defined it above,
`
`during the relevant time frame, which I have been asked to assume is the March
`
`17, 1999 timeframe (the filing date of Provisional Application Nos. 60/124,718,
`
`60/124,719, and 60/124,720, from which the ’718 patent claims priority (Ex. 1001,
`
`Cover)). During this time frame, I possessed at least the qualifications of a person
`
`of ordinary skill in the art, as defined above.
`
`V. TECHNICAL BACKGROUND
`16.
`In this section, I discuss the state of the art with respect to certain
`
`technologies relevant to the subject matter of the ’718 patent. In particular, during
`
`the time preceding March 1999, a person of ordinary skill in the art would have
`
`been aware of various developments in the areas of natural language processing,
`
`distributed computing, databases, multimodal input, and mobile computing, as I
`
`discuss below.
`
`A. Natural language processing
`
`
`
`
`
`8
`
`
`
`Page 12 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`17. A person of ordinary skill in the art would have been aware of
`
`developments in the area of natural language processing systems prior to March
`
`1999. For example, it was well known that users could interact with computers
`
`using natural language inputs, such as sentences in English (or another human
`
`language), e.g., as described in a paper by Wyard et al. from 1996 entitled “Spoken
`
`language systems – beyond prompt and response” (“Wyard”). (Ex. 1027, 187.)
`
`Enabling such natural language inputs was often desirable, as it allowed users to
`
`express their requirements or desires more directly and efficiently. (Id.)
`
`18.
`
`In the mid-to-late 1990s, natural language input was frequently
`
`provided by way of spoken input. Wyard describes “a typical spoken language
`
`system architecture” as including a speech recognition component and a meaning
`
`extraction component. (Id., 188, FIG. 1 (reproduced below).)
`
`
`
`
`
`
`
`9
`
`
`
`Page 13 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`
`
`
`(Ex. 1027, FIG. 1 (showing speech recognition and meaning extraction
`
`components that process natural language speech input.)
`
`19. Consistent with Wyard’s disclosure, a person of ordinary skill would
`
`have known that the role of a speech recognition component (speech recognizer)
`
`was “to convert an input speech utterance to a string of words,” and the role of a
`
`meaning extraction component was “to extract as much of the meaning as is
`
`necessary for the application from the recogniser output and encode it into a
`
`suitable meaning representation.” (Id., 188.)
`
`10
`
`
`
`
`
`Page 14 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`20. A person of ordinary skill would have known that speech (voice) was
`
`one way for a user to provide natural language input, but another known way was
`
`for the user to provide language input via text, e.g., using a keyboard. It was also
`
`known prior to March 1999 that a user could use an electronic pen/stylus to
`
`provide input, e.g., by writing characters that were processed by a character
`
`recognition algorithm so that the user could enter words or sentences. For
`
`example, a paper by Moran et al. entitled “Multimodal User Interfaces in the Open
`
`Agent Architecture” describes input from a user via electronic pen, e.g., in
`
`conjunction with a handwriting recognizer. (Ex. 1029, 63.) The main difference
`
`between handling speech-based and text-based natural language input was that
`
`speech input had to be processed first by a speech recognizer in order to detect and
`
`identify speech utterances, whereas a speech recognizer would not have been
`
`necessary in the context of natural language text input provided via a keyboard.
`
`For handwritten text input, e.g., inputted using an electronic pen, a person of
`
`ordinary skill would have known how to implement a handwriting recognizer as
`
`discussed above.
`
`B. Multimodal input
`21. As discussed above, a person of ordinary skill would have known
`
`before March 1999 about the existence of various input modalities, including
`
`
`
`
`
`11
`
`
`
`Page 15 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`speech (voice), keyboard, pen/stylus, and also others such as mouse, trackball,
`
`touchpad, etc. Such a person would also have been aware of the existence and
`
`benefits of multimodal systems, which enabled a user to provide input via multiple
`
`input modalities. For example, Wyard describes a multimodal natural language
`
`system for providing a user with information regarding various products. (Ex.
`
`1027, 190.) Wyard describes that the user can provide natural language spoken
`
`input and also click on links using a mouse and provide text as input. (Id., 189
`
`(“systems such as the BT Business Catalogue access system . . . are multimodal
`
`and require a screen and a means of inputting text and mouse clicks and outputting
`
`text and graphics.”), 191 (disclosing that a user speaks “Which ones come in
`
`grey?” and later “clicks on the link next to the picture of [a particular phone that is
`
`displayed]”); see also id., 190 (“a film access system, in which users will be able to
`
`select films and videos using continuous speech and button pushes on a remote
`
`control handset”).) Thus, a person of ordinary skill would have been aware of
`
`“multimodal systems which aim to combine spoken language with other
`
`modalities, such as typed text and mouse clicks, in order to achieve the most user-
`
`friendly interface possible.” (Id., 204.)
`
`22. As another example, a paper by Coen from 1997 entitled “Building
`
`Brains for Rooms: Designing Distributed Software Agents” (“Coen”) describes an
`
`
`
`
`
`12
`
`
`
`Page 16 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`information retrieval system with which users can interact using pointing and
`
`natural language speech input. (Ex. 1020, 975.) Coen discloses techniques for
`
`resolving what the user means when he/she provides the natural language spoken
`
`input “What’s the weather here?” while pointing somewhere. (Id.) Coen refers to
`
`this process as “multimodal resolution.” (Id.)
`
`23. Thus, a person of ordinary skill in the art would have known how to
`
`implement multimodal systems (systems that enable input via multiple input
`
`modalities) in an effective, user-friendly manner prior to March 1999.
`
`C. Databases
`24. A person of ordinary skill would have known before March 1999 that
`
`a fundamental component of an information retrieval system was a database, and
`
`that database queries could be used to retrieve information from a database. For
`
`example, Wyard describes a natural language based system that includes a database
`
`query as a key processing component, in order to “retrieve the information
`
`specified by the output of the meaning extraction component.” (Ex. 1027, 188.) It
`
`was known to generate a database query after first processing natural language
`
`speech input with a speech recognition component and a meaning extraction
`
`component (or processing natural language text input with a meaning extraction
`
`component), as shown in the following flow diagram in Wyard:
`
`
`
`
`
`13
`
`
`
`Page 17 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`
`(Ex. 1027, FIG. 1.)
`
`25. Wyard explains the database query as follows:
`
`
`
`When the [dialogue manager] has prepared the query, it will be
`passed to the database query component. The database query
`component’s purpose is to convert the query from the [dialogue
`manager] into one or more queries which can be used to find
`the required information from within the database. Having
`established the queries, the database query component then
`extracts the actual information from the database.
`
`
`
`
`
`14
`
`
`
`Page 18 of 131
`
`

`

`(Id., 201.)
`
`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`
`26. A person of ordinary skill would have been aware that a database
`
`query could be implemented in, e.g., structured query language (SQL), which was
`
`a well-known programming language (and one of the most prevalent and
`
`commonly used languages) for working with databases. (Id., 202 (“the database
`
`querying module provides a means of separating the actual database query (in
`
`SQL, for example) from the internal representation in the [dialogue manager]”).
`
`For example, as disclosed in U.S. Patent No. 5,584,024 to Shwartz (“Shwartz-
`
`024”) (Ex. 1022), a person of ordinary skill would have known how to use an SQL
`
`statement called a SELECT statement to retrieve a set of records from database
`
`tables. (See Ex. 1022, FIGS. 2C, 3A, 1:56-2:26 (disclosing examples of SQL
`
`SELECT statements).) A SELECT statement was a fundamental aspect of SQL,
`
`and similar statements were used in other database programming languages.
`
`27. As one example, Shwartz-024 discloses that “to produce from a
`
`database a list of customer names and phones for New York customers sorted by
`
`zip code, the following SQL statement could be used: . . . SELECT NAME,
`
`PHONE FROM CUSTOMERS WHERE STATE = `NY` ORDER BY
`
`ZIP_CODE.” (Id., 1:59-66.) “In this example, the SELECT command defines
`
`which fields to use, the WHERE command defines a condition by which database
`
`
`
`
`
`15
`
`
`
`Page 19 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`records are selected, and ORDER BY keywords define how the output should be
`
`sorted.” (Id., 2:1-4.) “The FROM keyword defines in which tables the fields are
`
`located.” (Id., 2:4-5.)
`
`28. A person of ordinary skill would have known that databases were in
`
`widespread usage across a variety of contexts long before March 1999. For
`
`example, the World Wide Web (“the Web”), which was created in the early 1990s,
`
`involved web servers that provide users with access to remote databases. The Web
`
`was in widespread usage by March 1999, and a person of ordinary skill would have
`
`known how to program computers to access information from the Web. A person
`
`of ordinary skill would have known how to implement databases available via the
`
`Web to be accessible via database queries.
`
`D. Distributed computing
`29. A person of ordinary skill in the art would have been familiar with
`
`distributed systems for various computing contexts. Such a person would have
`
`known that various networked entities could communicate with one another and
`
`take respective actions to accomplish goals. For example, Coen describes “a
`
`distributed software agent system that controls the behavior of [a] laboratory’s
`
`Intelligent Room.” (Ex. 1020, 971 (at Abstract).) Coen discloses that a “system of
`
`
`
`
`
`16
`
`
`
`Page 20 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`software agents . . . known collectively as the Scatterbrain” control various aspects
`
`of a room. (Id., 971.) Coen explains:
`
`The Scatterbrain consists of approximately 20 distinct,
`intercommunicating software agents that run on ten different
`networked workstations. These agents’ primary task is to link
`various components of the room (e.g., tracking cameras, speech
`recognition systems) and to connect them to internal and
`external stores of information (e.g., a person locator, the World
`Wide Web). Although an individual agent may in fact perform
`a good deal of computation, we will focus our interest on the
`ways in which agents get connected and share information
`rather than how they internally manipulate their own data. And
`while the Intelligent Room is a fascinating project in itself, we
`will treat it here mainly as a test-bed to learn more about how
`software agents can interact with other computational and real
`entities.
`
`(Id.)
`
`30. Coen discloses that “[p]eople can interact with [a system in the room
`
`called Storm] using pointing and speech.” (Id., 975.) For example, when the user
`
`provides a natural language spoken input “Computer, what is the weather here?”
`
`the room “displays a weather forecast for San Juan.” (Id.) Coen discloses various
`
`agents, such as a SpeechIn Agent (for interfacing with speech recognition
`
`systems), Tracking Agent (for updating another agent in real-time), Weather Agent
`
`17
`
`
`
`
`Page 21 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`(for obtaining forecasts and satellite maps for particular places), and Display Agent
`
`(for displaying content at a location in the room where people can see it). (Id.,
`
`974.) “All the Scatterbrain agents then work together in parallel with different
`
`inputs and data being processed simultaneously in different places.” (Id.)
`
`31. A person of ordinary skill would have known how to implement
`
`agents in a layered, hierarchical configuration. For example, Coen discloses that
`
`“[layered] on top of the Scatterbrain, we created higher-level agents that rely on
`
`the Scatterbrain’s underlying behaviors.” (Id.)
`
`32. A paper by Hodjat et al. from November 1998 entitled “An Adaptive
`
`Agent Oriented Software Architecture” (“Hodjat”) describes an agent as an
`
`“autonomous individual the internals of which are not known and that conforms to
`
`a certain standard of communications and/or social laws with regard to other
`
`agents.” (Ex. 1021, 33.) Hodjat discloses “an agent-oriented methodology, which
`
`can be universally applied to any software design.” (Id., 34.) A person of ordinary
`
`skill would have known, based on Hodjat, how to configure an agent-based
`
`architecture so that “new agents supply other agents with information about their
`
`capabilities and needs.” (Id., 35.) Like Coen, Hodjat describes a cooperative
`
`collection of agents that coordinate with one another, including in a hierarchical
`
`manner, to accomplish a set of requests:
`
`
`
`
`
`18
`
`
`
`Page 22 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`
`The software as a whole should be thought of as a society,
`striving to accomplish a set of requests. The input requests are
`therefore propagated, processed by agent modules that may in
`turn create requests to other agents. Again, it is up to the
`designers to break down the system, as they feel suitable.
`Hierarchies of agents are possible and agents can be designed to
`be responsible for the minutest processes in the system.
`
`(Id., 37.)
`
`33. Hodjat explains that a known technique for implementing distributed
`
`systems with cooperative agents was to use the then-existing Open Agent
`
`Architecture:
`
`[Cheyer et al 96] use the Open Agent Architecture (OAA) . . .
`as a basis for their design. In this approach, based on a
`“federation architecture” . . . , the software is comprised of a
`hierarchy of facilitators and agents. The facilitators are
`responsible for the coordination of the agents under them so
`that any agent wanting to communicate with any other agent in
`the system must go through a hierarchy of facilitators (starting
`from the one directly responsible for it). Each agent, upon
`introduction to the system, provides the facilitator above it with
`information on its capabilities . . . .
`
`(Id., 40.)
`
`
`
`
`
`19
`
`
`
`Page 23 of 131
`
`

`

`Declaration of Dr. Dan R. Olsen Jr.
`U.S. Patent No. 6,757,718
`
`
`34. A person of ordinary skill would have been familiar with agent-based
`
`architectures like the Open Agent Architecture and would have known how to use
`
`it to implement distributed systems in various contexts, including speech-based
`
`information retrieval. The Open Agent Architecture was described in published
`
`documents at least as early as 1994, when Cohen et al. described in a paper