UNITED STATES PATENT AND TRADEMARK OFFICE
`_________________
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`_________________
`
`APPLE INC.
`Petitioner
`
`v.
`
`ZENTIAN LIMITED,
`Patent Owner
`_________________
`
`Inter Partes Review Case No. IPR2023-00036
`U.S. Patent No. 10,839,789
`
`DECLARATION OF CHRISTOPHER SCHMANDT
`IN SUPPORT OF PETITION FOR INTER PARTES REVIEW OF
`U.S. PATENT NO. 10,839,789
`
`Amazon / Zentian Limited
`Exhibit 1003
`Page 1
`
`

`

`Declaration of Chris Schmandt
`Patent No. 10,839,789
`
`I.
`
`TABLE OF CONTENTS
`INTRODUCTION AND QUALIFICATIONS ............................................. 14
`A.
`Educational Background and Professional Experience ............... 14
`II. METHODOLOGY; MATERIALS CONSIDERED ..................................... 17
`III. OVERVIEW AND LEGAL STANDARDS ................................................... 20
`A.
`Person of Ordinary Skill in the Art ............................................... 20
`B.
`Obviousness ..................................................................................... 21
`C.
`Analogous Art .................................................................................. 27
`D.
`Claim Construction ......................................................................... 27
`IV. LEVEL OF A PERSON OF ORDINARY SKILL ........................................ 28
`V. OVERVIEW OF THE TECHNOLOGY ....................................................... 29
`A.
`Speech Recognition ......................................................................... 29
`B.
`Feature Vectors ............................................................................... 37
`C.
`Acoustic Models ............................................................................... 46
`D.
`Hidden Markov Models .................................................................. 47
`E.
`Distance Calculations ...................................................................... 56
`F.
`Gaussian Distribution and Probability ......................................... 59
`G.
`Speech Recognition System Hardware .......................................... 62
`H.
`Pipelining ......................................................................................... 70
`I.
`Interrupts ......................................................................................... 74
`J.
`Prior Art Speech Recognition Systems ......................................... 76
`VI. OVERVIEW OF THE ’789 PATENT .......................................................... 77
`VII. OVERVIEW OF THE PRIOR ART ............................................................. 78
`A.
`Overview of Smyth .......................................................................... 78
`B.
`Overview of Mozer .......................................................................... 78
`C.
`Overview of Nguyen ........................................................................ 79
`D.
`Overview of Houser ......................................................................... 80
`E.
`Overview of Vensko ........................................................................ 80
`F.
`Overview of Comeau ....................................................................... 80
`
`Amazon / Zentian Limited
`Exhibit 1003
`Page 2
`
`

`

`Declaration of Chris Schmandt
`Patent No. 10,839,789
`G. Overview of Boike ........................................................................... 81
`H. Overview of Whittaker ................................................................... 81
`I.
`Overview of Ichikawa ..................................................................... 82
`J.
`Overview of Seshadri ...................................................................... 82
`K. Overview of Steinbiss ...................................................................... 83
`PROPOSED GROUNDS OF UNPATENTABILITY ...................... 83
`VIII.
`IX. OPINIONS REGARDING GROUND 1: CLAIMS 1, 10-11, 14, AND 17
`ARE OBVIOUS OVER SMYTH AND MOZER .................................................. 84
`A.
`Independent Claim 1 ....................................................................... 84
`1.
`Claim 1[Pre]: “An acoustic coprocessor for processing
`data associated with an audio signal, comprising:” ................ 84
`Claim 1(a): “a first interface for receiving at least one
`feature vector, wherein the feature vector is determined
`from the audio signal” ........................................................... 103
`Claim 1(b): “an acoustic model memory for storing an
`acoustic model defining a plurality of acoustic states;” ........ 110
`Claim 1(c): “a calculating apparatus for calculating
`distances indicating a similarity between the at least one
`feature vector and respective acoustic states of the
`acoustic model read from the acoustic memory; and” .......... 120
`Claim 1(d): “a second interface for sending at least one
`distance calculated by the calculating apparatus;” ................ 126
`Claim 1(e): “wherein the calculating apparatus and the
`acoustic model memory are fabricated on a single
`integrated circuit.” ................................................................. 131
`Independent Claim 10 ................................................................... 136
`1.
`Claim 10[Pre]: “An acoustic coprocessor, comprising:” ...... 136
`2.
`Claim 10(a):” a first interface for receiving at least one
`feature vector;” ...................................................................... 136
`Claim 10(b): “a calculating apparatus for calculating
`distances indicating a similarity between the at least one
`feature vector and at least one acoustic state of an
`acoustic model read from an acoustic model memory;” ....... 136
`
`B.
`
`2.
`
`3.
`
`4.
`
`5.
`
`6.
`
`3.
`
`Amazon / Zentian Limited
`Exhibit 1003
`Page 3
`
`

`

`4.
`
`C.
`
`D.
`
`Declaration of Chris Schmandt
`Patent No. 10,839,789
`Claim 10(c): “a second interface for sending at least one
`distance calculated by the calculating apparatus.” ................ 136
`Dependent Claim 11 ...................................................................... 136
`1.
`Claim 11: “The acoustic coprocessor of claim 10,
`wherein the calculating apparatus and the acoustic model
`memory are fabricated on a single integrated circuit.” ......... 136
`Dependent Claim 14 ...................................................................... 137
`1.
`Claim 14:” The acoustic coprocessor of claim 10,
`wherein the second interface for sending distances is a
`serial bus.” ............................................................................. 137
`Dependent Claim 17 ...................................................................... 138
`1.
`Claim 17: “The acoustic coprocessor of claim 10,
`wherein the acoustic coprocessor is capable of, in
`response to receiving a feature vector, autonomously
`calculating the distances for each acoustic state of the
`acoustic model.” .................................................................... 138
`X. OPINIONS REGARDING GROUND 2: CLAIMS 2, 4, 6-9, 12, 16, 18, AND
`20-21 ARE OBVIOUS OVER SMYTH, MOZER, AND NGUYEN .................. 139
`A.
`Dependent Claim 2 ........................................................................ 139
`1.
`Claim 2: “The acoustic coprocessor of claim 1, wherein
`the acoustic model contains Gaussian parameters and the
`distances are probabilities” ................................................... 139
`Dependent Claim 4 ........................................................................ 141
`1.
`Claim 4: “The acoustic coprocessor of claim 2, wherein
`the second interface for sending distances is a serial bus.” .. 141
`Dependent Claim 6 ........................................................................ 141
`1.
`Claim 6: “The acoustic coprocessor of claim 4, wherein
`the acoustic coprocessor, in response to receiving a
`feature vector, autonomously calculates the distance for
`an associated acoustic state of the acoustic model.” ............. 141
`Dependent Claim 7 ........................................................................ 142
`1.
`Claim 7: “The acoustic coprocessor of claim 6, further
`comprising a result memory, wherein the distance for an
`associated acoustic state of the acoustic model is stored
`in the result memory.” ........................................................... 142
`
`C.
`
`E.
`
`B.
`
`D.
`
`Amazon / Zentian Limited
`Exhibit 1003
`Page 4
`
`

`

`E.
`
`F.
`
`G.
`
`H.
`
`2.
`
`3.
`
`Declaration of Chris Schmandt
`Patent No. 10,839,789
`Dependent Claim 8 ........................................................................ 143
`Claim 8: “The acoustic coprocessor of claim 7, wherein
`1.
`the result memory is configured so that distances
`calculated from a first feature vector are sent over the
`second interface for sending while at least one distance
`calculated from a second feature vector is being stored in
`the one or more result memories.” ........................................ 143
`Dependent Claim 9 ........................................................................ 145
`1.
`Claim 9(Pre): “A speech recognition system comprising
`the acoustic coprocessor of claim 7, the speech
`recognition system further comprising at least one CPU,
`wherein:” ............................................................................... 145
`Claim 9(a): “the CPU executes software to perform or
`initiate a speech recognition word search using the
`distances calculated by the acoustic coprocessor in
`response to receiving the first feature vector, and” ............... 146
`Claim 9(b): “a time period for the speech recognition
`word search overlaps a time period for the acoustic
`coprocessor to calculate the distance in response to
`receiving the second feature vector.” .................................... 149
`Dependent Claim 12 ...................................................................... 156
`1.
`Claim 12: “The acoustic coprocessor of claim 10,
`wherein the acoustic model contains Gaussian parameters
`and the distances are probabilities.” ...................................... 156
`Dependent Claim 16 ...................................................................... 156
`1.
`Claim 16 [Pre]: “A speech recognition system
`comprising the acoustic coprocessor of claim 10, the
`speech recognition system further comprising at least one
`CPU, wherein:” ..................................................................... 156
`Claim 16(a): “the CPU executes software to perform or
`initiate a speech recognition word search using one or
`more distances calculated by the acoustic coprocessor in
`response to receiving a first feature vector, and” .................. 156
`Claim 16(b): “a time period for the speech recognition
`word search overlaps with a time period for the acoustic
`
`2.
`
`3.
`
`Amazon / Zentian Limited
`Exhibit 1003
`Page 5
`
`

`

`I.
`
`J.
`
`K.
`
`Declaration of Chris Schmandt
`Patent No. 10,839,789
`coprocessor to calculate one or more distances in
`response to receiving a second feature vector.” .................... 156
`Dependent Claim 18 ...................................................................... 157
`1.
`Claim 18: “The acoustic coprocessor of claim 17, further
`comprising one or more result memories configured to
`store the distances for each acoustic state of the acoustic
`model.” .................................................................................. 157
`Dependent Claim 20 ...................................................................... 157
`1.
`Claim 20: “The acoustic coprocessor of claim 18,
`wherein the calculating apparatus is capable of
`computing distances using acoustic models representing
`Gaussian probability distributions.” ...................................... 157
`Dependent Claim 21 ...................................................................... 157
`1.
`Claim 21: “The acoustic coprocessor of claim 20,
`wherein the acoustic model uses a neural network to
`calculate at least one of the distances.” ................................. 157
`XI. OPINIONS REGARDING GROUND 3: CLAIM 13 IS OBVIOUS OVER
`SMYTH, MOZER, AND HOUSER .................................................................... 159
`A.
`Dependent Claim 13 ...................................................................... 159
`1.
`Claim 13:” The acoustic coprocessor of claim 10,
`wherein the acoustic model memory is a FLASH
`memory.” .............................................................................. 159
`XII. OPINIONS REGARDING GROUND 4: CLAIMS 22-24 ARE OBVIOUS
`OVER SMYTH, MOZER, NGUYEN, AND VENSKO ...................................... 160
`A.
`Dependent Claim 22 ...................................................................... 160
`1.
`Claim 22: “The acoustic coprocessor of claim 21,
`wherein the acoustic coprocessor is capable of generating
`an interrupt in response to a completion of calculation of
`a set of distances.” ................................................................. 160
`Dependent Claim 23 ...................................................................... 161
`1.
`Claim 23: “The acoustic coprocessor of claim 22,
`wherein the acoustic model memory includes a plurality
`of storage locations within a static random access
`memory (SRAM).” ............................................................... 161
`Dependent Claim 24 ...................................................................... 163
`
`B.
`
`C.
`
`Amazon / Zentian Limited
`Exhibit 1003
`Page 6
`
`

`

`1.
`
`2.
`
`3.
`
`Declaration of Chris Schmandt
`Patent No. 10,839,789
`Claim 24: “The acoustic coprocessor of claim 23,
`wherein the SRAM and the calculating apparatus are
`fabricated on a single integrated circuit.” ............................. 163
`OPINIONS REGARDING GROUND 5: CLAIMS 25-28 AND 37
`XIII.
`ARE OBVIOUS OVER SMYTH, MOZER, NGUYEN, VENSKO, AND
`COMEAU 164
`A.
`Dependent Claim 25 ...................................................................... 164
`1.
`Claim 25: “The acoustic coprocessor of claim 22, further
`comprising a status register, wherein the acoustic
`coprocessor is capable of setting one or more bits in the
`status register to values that indicate the completion of
`calculation of the set of distances.” ....................................... 164
`Dependent Claim 26 ...................................................................... 168
`1.
`Claim 26: “The acoustic coprocessor of claim 25,
`wherein the acoustic coprocessor further comprises a
`direct memory access (DMA) controller capable of
`copying distances from the one or more result memories
`to a buffer memory.” ............................................................. 168
`Dependent Claim 27 ...................................................................... 170
`1.
`Claim 27: “The acoustic coprocessor of claim 26,
`wherein the DMA controller is capable of copying
`distances in a plurality of sequential bursts.” ........................ 170
`Dependent Claim 28 ...................................................................... 175
`1.
`Claim 28[Pre]: “A speech recognition system comprising
`the acoustic coprocessor of claim 26, wherein:” .................. 175
`Claim 28(a): “the speech recognition system further
`comprises at least one processor configured to execute
`software to perform or initiate a speech recognition word
`search using one or more distances calculated by the
`acoustic coprocessor in response to receiving a first
`feature vector, and” ............................................................... 175
`Claim 28(b): “a time period for the speech recognition
`word search overlaps with a time period for the acoustic
`coprocessor to calculate one or more distances in
`response to receiving a second feature vector.” .................... 175
`Dependent Claim 37 ...................................................................... 175
`
`C.
`
`B.
`
`D.
`
`E.
`
`Amazon / Zentian Limited
`Exhibit 1003
`Page 7
`
`

`

`1.
`
`Declaration of Chris Schmandt
`Patent No. 10,839,789
`Claim 37: “A speech recognition system comprising the
`acoustic coprocessor of claim 27, the speech recognition
`system further comprising at least one microphone.” ........... 175
`XIV. OPINIONS REGARDING GROUND 6: CLAIM 29 IS OBVIOUS OVER
`SMYTH, MOZER, NGUYEN, VENSKO, COMEAU, AND BOIKE ................ 176
`A.
`Dependent Claim 29 ...................................................................... 176
`1.
`Claim 29: “The speech recognition system of claim 28,
`wherein the acoustic coprocessor and the at least one
`processor are fabricated on a single integrated circuit.” ....... 176
`XV. OPINIONS REGARDING GROUND 7: CLAIM 35 IS OBVIOUS OVER
`SMYTH, MOZER, NGUYEN, VENSKO, COMEAU, AND WHITTAKER ..... 181
`A.
`Dependent Claim 35 ...................................................................... 181
`1.
`Claim 35: “The acoustic coprocessor of claim 26,
`wherein the acoustic model includes parameters
`represented with integer data.” ............................................. 181
`XVI. OPINIONS REGARDING GROUND 8: CLAIMS 38-39 ARE OBVIOUS
`OVER SMYTH, MOZER, NGUYEN, VENSKO, COMEAU, AND ICHIKAWA
`183
`A.
`
`Dependent Claim 38 ...................................................................... 183
`1.
`Claim 38: “A speech recognition system comprising the
`acoustic coprocessor of claim 27, the speech recognition
`system further comprising a plurality of microphones.” ...... 183
`Dependent Claim 39 ...................................................................... 184
`1.
`Claim 39: “The speech recognition system of claim 38,
`further comprising the buffer memory.” ............................... 184
`OPINIONS REGARDING GROUND 9: CLAIMS 42-44 ARE
`XVII.
`OBVIOUS OVER SMYTH, MOZER, NGUYEN, VENSKO, COMEAU,
`ICHIKAWA, AND SESHADRI .......................................................................... 184
`A.
`Dependent Claim 42 ...................................................................... 184
`1.
`Claim 42: “A laptop or tablet personal computer
`comprising the speech recognition system of claim 39.” ..... 184
`Dependent Claim 43 ...................................................................... 185
`1.
`Claim 43: “A mobile phone comprising the speech
`recognition system of claim 39.” .......................................... 185
`
`B.
`
`B.
`
`Amazon / Zentian Limited
`Exhibit 1003
`Page 8
`
`

`

`C.
`
`Declaration of Chris Schmandt
`Patent No. 10,839,789
`Dependent Claim 44 ...................................................................... 186
`1.
`Claim 44: “An electronic entertainment product
`comprising the speech recognition system of claim 39.” ..... 186
`OPINIONS REGARDING GROUND 10: CLAIM 45 IS OBVIOUS
`XVIII.
`OVER SMYTH, MOZER, NGUYEN, VENSKO, COMEAU, ICHIKAWA, AND
`STEINBISS 188
`A.
`Dependent Claim 45 ...................................................................... 188
`1.
`Claim 45: “An appliance comprising the speech
`recognition system of claim 39.” .......................................... 188
`XIX. CONCLUSION ............................................................................................ 189
`
`Amazon / Zentian Limited
`Exhibit 1003
`Page 9
`
`

`

`Declaration of Chris Schmandt
`Patent No. 10,839,789
`
`CLAIM LISTING
`
`Claim 1:
`Claim 1[Pre] An acoustic coprocessor for processing data associated with an
`audio signal, comprising:
`1(a) first interface for receiving at least one feature vector, wherein the feature
`vector is determined from the audio signal;
`1(b) an acoustic model memory for storing an acoustic model defining a
`plurality of acoustic states;
`1(c) a calculating apparatus for calculating distances indicating a similarity
`between the at least one feature vector and respective acoustic states of the acoustic
`model read from the acoustic model memory; and
`1(d) a second interface for sending at least one distance calculated by the
`calculating apparatus;
`1(e) wherein the calculating apparatus and the acoustic model memory are
`fabricated on a single integrated circuit.
`Claim 2:
`The acoustic coprocessor of claim 1, wherein the acoustic model contains
`Gaussian parameters and the distances are probabilities.
`Claim 4:
`The acoustic coprocessor of claim 2, wherein the second interface for sending
`distances is a serial bus.
`Claim 6:
`The acoustic coprocessor of claim 4, wherein the acoustic coprocessor, in
`response to receiving a feature vector, autonomously calculates the distance for an
`associated acoustic state of the acoustic model.
`Claim 7:
`The acoustic coprocessor of claim 6, further comprising a result memory,
`wherein the distance for an associated acoustic state of the acoustic model is stored
`in the result memory.
`Claim 8:
`The acoustic coprocessor of claim 7, wherein the result memory is configured
`so that distances calculated from a first feature vector are sent over the second
`interface for sending while at least one distance calculated from a second feature
`vector is being stored in the one or more result memories.
`Claim 9:
`Claim 9[Pre] A speech recognition system comprising the acoustic
`coprocessor of claim 7, the speech recognition system further comprising at least
`one CPU, wherein:
`
`Amazon / Zentian Limited
`Exhibit 1003
`Page 10
`
`

`

`Declaration of Chris Schmandt
`Patent No. 10,839,789
`9(a) the CPU executes software to perform or initiate a speech recognition
`word search using the distances calculated by the acoustic coprocessor in response
`to receiving the first feature vector, and
`9(b) a time period for the speech recognition word search overlaps a time
`period for the acoustic coprocessor to calculate the distance in response to receiving
`the second feature vector.
`Claim 10:
`Claim 10[Pre] An acoustic coprocessor, comprising:
`10(a) a first interface for receiving at least one feature vector;
`10(b) a calculating apparatus for calculating distances indicating a similarity
`between the at least one feature vector and at least one acoustic state of an acoustic
`model read from an acoustic model memory; and
`10(c) a second interface for sending at least one distance calculated by the
`calculating apparatus.
`Claim 11:
`The acoustic coprocessor of claim 10, wherein the calculating apparatus and
`the acoustic model memory are fabricated on a single integrated circuit.
`Claim 12:
`The acoustic coprocessor of claim 10, wherein the acoustic model contains
`Gaussian parameters and the distances are probabilities.
`Claim 13:
`The acoustic coprocessor of claim 10, wherein the acoustic model memory is
`a FLASH memory.
`Claim 14:
`The acoustic coprocessor of claim 10, wherein the second interface for
`sending distances is a serial bus.
`Claim 16:
`Claim 16 [Pre] A speech recognition system comprising the acoustic
`coprocessor of claim 10, the speech recognition system further comprising at least
`one CPU, wherein:
`16(a) the CPU executes software to perform or initiate a speech recognition
`word search using one or more distances calculated by the acoustic coprocessor in
`response to receiving a first feature vector, and
`16(b) a time period for the speech recognition word search overlaps with a
`time period for the acoustic coprocessor to calculate one or more distances in
`response to receiving a second feature vector.
`Claim 17:
`The acoustic coprocessor of claim 10, wherein the acoustic coprocessor is
`capable of, in response to receiving a feature vector, autonomously calculating the
`distances for each acoustic state of the acoustic model.
`
`Amazon / Zentian Limited
`Exhibit 1003
`Page 11
`
`

`

`Declaration of Chris Schmandt
`Patent No. 10,839,789
`
`Claim 18:
`The acoustic coprocessor of claim 17, further comprising one or more result
`memories configured to store the distances for each acoustic state of the acoustic
`model.
`Claim 20:
`The acoustic coprocessor of claim 18, wherein the calculating apparatus is
`capable of computing distances using acoustic models representing Gaussian
`probability distributions.
`Claim 21:
`The acoustic coprocessor of claim 20, wherein the acoustic model uses a
`neural network to calculate at least one of the distances.
`Claim 22:
`The acoustic coprocessor of claim 21, wherein the acoustic coprocessor is
`capable of generating an interrupt in response to a completion of calculation of a set
`of distances.
`Claim 23:
`The acoustic coprocessor of claim 22, wherein the acoustic model memory
`includes a plurality of storage locations within a static random access memory
`(SRAM).
`Claim 24:
`The acoustic coprocessor of claim 23, wherein the SRAM and the calculating
`apparatus are fabricated on a single integrated circuit.
`Claim 25:
`The acoustic coprocessor of claim 22, further comprising a status register,
`wherein the acoustic coprocessor is capable of setting one or more bits in the status
`register to values that indicate the completion of calculation of the set of distances.
`Claim 26:
`The acoustic coprocessor of claim 25, wherein the acoustic coprocessor
`further comprises a direct memory access (DMA) controller capable of copying
`distances from the one or more result memories to a buffer memory.
`Claim 27:
`The acoustic coprocessor of claim 26, wherein the DMA controller is capable
`of copying distances in a plurality of sequential bursts.
`Claim 28:
`Claim 28[Pre] A speech recognition system comprising the acoustic
`coprocessor of claim 26, wherein:
`28(a) the speech recognition system further comprises at least one processor
`configured to execute software to perform or initiate a speech recognition word
`search using one or more distances calculated by the acoustic coprocessor in
`response to receiving a first feature vector, and
`
`Amazon / Zentian Limited
`Exhibit 1003
`Page 12
`
`

`

`Declaration of Chris Schmandt
`Patent No. 10,839,789
`28(b) a time period for the speech recognition word search overlaps with a
`time period for the acoustic coprocessor to calculate one or more distances in
`response to receiving a second feature vector.
`Claim 29:
`The speech recognition system of claim 28, wherein the acoustic coprocessor
`and the at least one processor are fabricated on a single integrated circuit.
`Claim 35:
`The acoustic coprocessor of claim 26, wherein the acoustic model includes
`parameters represented with integer data.
`Claim 37:
`A speech recognition system comprising the acoustic coprocessor of claim 27,
`the speech recognition system further comprising at least one microphone.
`Claim 38:
`A speech recognition system comprising the acoustic coprocessor of claim 27,
`the speech recognition system further comprising a plurality of microphones.
`Claim 39:
`The speech recognition system of claim 38, further comprising the buffer
`memory.
`Claim 42:
`A laptop or tablet personal computer comprising the speech recognition
`system of claim 39.
`Claim 43:
`A mobile phone comprising the speech recognition system of claim 39.
`Claim 44:
`An electronic entertainment product comprising the speech recognition
`system of claim 39.
`Claim 45:
`An appliance comprising the speech recognition system of claim 39.
`
`Amazon / Zentian Limited
`Exhibit 1003
`Page 13
`
`

`

`Declaration of Chris Schmandt
`Patent No. 10,839,789
`
`I, Christopher Schmandt, declare as follows:
`
`I.
`
`INTRODUCTION AND QUALIFICATIONS
`1.
`I am over the age of 21 and am competent to make this declaration.
`
`A. Educational Background and Professional Experience
`2.
`I retired several years ago after a 40-year career at the Massachusetts
`
`Institute of Technology (“MIT”); for most of that time I was employed as a Principal
`
`Research Scientist at the Media Laboratory. In that role I also served as faculty for
`
`the MIT Media Arts and Sciences academic program. I was a founder of the Media
`
`Laboratory, a research lab which now spans two buildings.
`
`3.
`
`I received my B.S. degree in Electrical Engineering and Computer
`
`Science from MIT in 1978, and my M.S. in Visual Studies (Computer Graphics) also
`
`from MIT. I was employed at MIT since 1980, initially at the Architecture Machine
`
`Group which was an early computer graphics and interactive systems research lab.
`
`In 1985, I helped found the Media Laboratory and continued to work there until
`
`retirement. I was director of a research group titled “Living Mobile.” My research
`
`spanned distributed communication and collaborative systems, with an emphasis on
`
`multi-media and user interfaces, with a strong focus on speech-based systems. I have
`
`over 70 published conference and journal papers and one book in the field of speech
`
`technology and user interaction.
`
`Amazon / Zentian Limited
`Exhibit 1003
`Page 14
`
`

`

`Declaration of Chris Schmandt
`Patent No. 10,839,789
`For the first fifteen years of my career, my research emphasized speech
`
`4.
`
`recognition and speech user interfaces. I built the first conversational computer
`
`system utilizing speech recognition and synthesis (“Put That There”) starting in
`
`1980. I continued to innovate speech user interfaces using recognition, text-to-
`
`speech synthesis, and recorded audio in a wide variety of projects. I built one of the
`
`first graphical user interfaces for audio editing, employing keyword recognition on
`
`voice memos in 1982 (Intelligent Ear). I built the first research-grade unified
`
`messaging system, which combined text and voice messages into a single inbox,
`
`with speech recognition over the phone for remote access, and a graphical user
`
`interface for desktop access in 1983 (Phone Slave). Along with my students we built
`
`the first system for real time spoken driving directions, including speech-accessible
`
`maps of Cambridge, Massachusetts in 1987 (Back Seat Driver). We built some of
`
`the earliest speech-based personal assistants for managing messages, calendar,
`
`contacts, etc. (Conversational Desktop 1985, Chatter 1993, MailCall 1996). We built
`
`quite a few systems employing speech recognition in handheld mobile devices
`
`(ComMotion 1999, Nomadic Radio 2000, Impromptu 2001, and Symphony 2004,
`
`for example). We applied speech recognition to large bodies of everyday
`
`conversations captured with a wearable device and utilized as a memory aid
`
`(Memory Prosthesis 2004). We used speech recognition on radio newscasts to build
`
`a personalized version of audio newscasts (Synthetic News Radio, 1999) and also
`
`Amazon / Zentian Limited
`Exhibit 1003
`Page 15
`
`

`

`Declaration of Chris Schmandt
`Patent No. 10,839,789
`investigated adding speech recognition to a mouse-based window system a few years
`
`earlier.
`
`5.
`
`I was later awarded the prestigious Association for Computing
`
`Machinery (ACM) Computer Human Interface (CHI) Academy membership
`
`specifically for those years of work pioneering speech user interfaces.
`
`6.
`
`In the course of my research, I built a number of speech recognition
`
`client/server distributed systems, with the first being in 1985. Much of the initial
`
`motivation for a server architecture was that speech recognition required expensive
`
`digital signal processing hardware that we could not afford to put on each computer,
`
`so a central server with the required hardware was used. Later versions of the speech
`
`recognition server architecture allowed certain computers to perform specialized
`
`tasks serving a number of client computers providing voice user interfaces, either on
`
`screens or over telephone connections.
`
`7.
`
`Because of my early work with distributed speech systems, I served for
`
`several years in the mid-1990s with a working group on the impact of multimedia
`
`systems on the Internet reporting to the Internet Engineering Task Force (IETF) and
`
`later the Internet Activities Board (IAB). This work impacted emerging standards
`
`such as Session Initiation Protocol (SIP).
`
`8.
`
`In my faculty position I taught graduate level courses in speech
`
`technology and user interaction design, and directly supervised student research and
`
`Amazon / Zentian Limited
`Exhibit 1003
`Page 16
`
`

`

`Declaration of Chris Schmandt
`Patent No. 10,839,789
`theses at the Bachelors, Masters, and PhD level. I oversaw the Masters and PhD
`
`thesis programs for the entire Media Arts and Sciences academic program during
`
`my more senior years. I also served on the Media Laboratory intellectual property
`
`committee for many years.
`
`II. METHODOLOGY; MATERIALS CONSIDERED
`9.
`I have relied upon my education, knowledge and experience with
`
`speech technology and speech recognition systems, as well as the other materials as
`
`discussed in this declaration in forming my opinions.
`
`10.
`
`For this work, I have been asked to review U.S. Patent No. 10,839,789
`
`(“the ’789 Patent”) (Ex. 1001) including the specification and claims, and the ’789
`
`Patent’s prosecution history (“’789 File History”) (Ex. 1002). In developing my
`
`opinions relating to the ’789 Patent, I have considered the materials cited or
`
`discussed herein, including those itemized in the Exhibit Table below.
`
`Description
`Exhibit
`Exhibit 1001 U.S