Case 1:19-cv-01410-MN Document 22 Filed 10/09/19 Page 1 of 71 PageID #: 349
`
`IN THE UNITED STATES DISTRICT COURT
`FOR THE DISTRICT OF DELAWARE
`
`C.A. No. 1:19-cv-01410-MN
`
`JURY TRIAL DEMANDED
`
`)
`)
`)
`)
`)
`)
`)
`)
`
`))
`
`
`)
`)
`)
`)
`
`VB Assets, LLC,
`
`Plaintiff,
`
`v.
`
`Amazon.com, Inc.; Amazon.com LLC; Amazon
`Web Services, Inc.; A2Z Development Center,
`Inc. d/b/a Lab126; Rawles LLC; AMZN Mobile
`LLC; AMZN Mobile 2 LLC; Amazon.com
`Services, Inc. f/k/a Amazon Fulfillment Services,
`Inc.; and Amazon Digital Services LLC,
`
`Defendants.
`
`FIRST AMENDED COMPLAINT FOR PATENT INFRINGEMENT
`
`VB Assets, LLC (“Plaintiff” or “VoiceBox”) hereby alleges as follows for its First
`
`Amended Complaint against Defendants Amazon.com, Inc.; Amazon.com LLC; Amazon Web
`
`Services, Inc.; A2Z Development Center, Inc. d/b/a Lab126; Rawles LLC; AMZN Mobile LLC;
`
`AMZN Mobile 2 LLC; Amazon.com Services, Inc. f/k/a Amazon Fulfillment Services, Inc.; and
`
`Amazon Digital Services LLC (collectively, “Defendants” or “Amazon”):
`
`

`

`Case 1:19-cv-01410-MN Document 22 Filed 10/09/19 Page 2 of 71 PageID #: 350
`
`NATURE OF THE ACTION
`
`1.
`
`VoiceBox, through its predecessor companies VoiceBox Technologies
`
`Corporation and VoiceBox Technologies, Inc. (collectively “VoiceBox Technologies”),
`
`pioneered voice-based search and commerce technology. It invented what Amazon itself has
`
`described as “Echo-like” products long before Amazon.
`
`2.
`
`In recognition of its many innovations, the U.S. Patent & Trademark Office
`
`awarded and issued the VoiceBox Patents.1 The innovations in these patents were fundamental to
`
`the development of voice commerce technology.
`
`3.
`
`VoiceBox Technologies’ opportunities to promote and build a business based on
`
`these patents were crushed when Amazon introduced the infringing Echo and Alexa Products2
`
`and used its enormous size and clout to poach dozens of VoiceBox Technologies’ engineers and
`
`scientists.
`
`4.
`
`VoiceBox has brought this case to hold Amazon accountable for its infringement
`
`of VoiceBox’s patent rights.
`
`1 “VoiceBox Patents” collectively refers to U.S. Patent Nos. 8,073,681 (“the ’681 patent”);
`9,015,049 (“the ’049 patent”); 9,626,703 (“the ’703 patent”); 7,818,176 (“the ’176 patent”);
`8,886,536 (“the ’536 patent”); and 9,269,097 (“the ’097 patent”).
`2 “Alexa Products” collectively refers to Amazon’s Alexa virtual assistant and offerings that
`include Alexa, including the Echo product line (such as Echo 1st Gen., Echo 2nd Gen., Echo Dot
`1st Gen., Echo Dot 2nd Gen., Echo Dot 3rd Gen., Echo Dot Kids Edition, Echo Show 1st Gen.,
`Echo Show 2nd Gen., Echo Show 5, Echo Spot, Echo Plus 1st Gen., Echo Plus 2nd Gen., Echo
`Auto, and Echo Look), Amazon’s Alexa apps, Music apps, and Shopping apps on a smartphone
`or other mobile device, Amazon’s Alexa cloud, Alexa Voice Services, and Amazon.com
`website, and any other device, app, or instrumentality that includes, provides access to, or works
`with Alexa (such as Amazon Tap, Amazon Dash Wand, Echo Wall Clock, AmazonBasics
`Microwave, Amazon SmartPlug, Amazon Fire TV Sticks, Amazon Fire TVs, Amazon Fire TV
`Cubes, and Amazon Fire and Fire HD tablets) as well as software, hardware, and cloud
`infrastructure associated with any of the foregoing. Plaintiff reserves the right to expand upon or
`otherwise modify the above list during the discovery process in this case and is in no way
`limiting the scope of the accused products to what is currently listed.
`
`2
`
`

`

`Case 1:19-cv-01410-MN Document 22 Filed 10/09/19 Page 3 of 71 PageID #: 351
`
`THE PARTIES
`
`5.
`
`VB Assets, LLC, VoiceBox, is a limited liability company organized under the
`
`laws of Delaware and has its principal place of business at 1229A 120th Ave. NE, Bellevue, WA
`
`98005.
`
`6.
`
`On information and belief, Amazon.com, Inc. is a corporation incorporated in
`
`Delaware and has a principal place of business at 410 Terry Avenue North, Seattle, WA, 98109.
`
`On information and belief, Amazon.com, Inc. is the ultimate parent company of the other
`
`companies that make up Amazon, and is responsible for making, using, selling, offering for sale
`
`and/or importing Alexa Products.
`
`7.
`
`On information and belief, Amazon.com LLC is a limited liability company
`
`organized under the laws of Delaware and has a principal place of business at 410 Terry Avenue
`
`North, Seattle, WA, 98109. On information and belief, Amazon.com LLC includes as members
`
`various operating companies, which make, use, sell, offer for sale, and/or import Alexa Products.
`
`8.
`
`On information and belief, Amazon Web Services, Inc. is a corporation
`
`incorporated in Delaware and has a principal place of business at 410 Terry Avenue North,
`
`Seattle, WA, 98109. On information and belief, Amazon Web Services, Inc. provides Alexa cloud
`
`computing platforms, that are Alexa Products or are for use with one or more Alexa Products.
`
`9.
`
`On information and belief, A2Z Development Center, Inc. d/b/a Lab126 is a
`
`corporation incorporated in Delaware and has its principal place of business at 1120 Enterprise
`
`Way Sunnyvale, CA 94089. On information and belief, A2Z Development Center, Inc. d/b/a
`
`Lab126 performed research and development, including for one or more Alexa Products.
`
`10.
`
`On information and belief, Rawles LLC is a limited liability company formed
`
`under the laws of Delaware and has its principal place of business at 103 Foulk Road, Suite 100,
`
`3
`
`

`

`Case 1:19-cv-01410-MN Document 22 Filed 10/09/19 Page 4 of 71 PageID #: 352
`
`Wilmington, DE 19803. On information and belief, Rawles LLC performed research and
`
`development, including for one or more Alexa Products.
`
`11.
`
`On information and belief, AMZN Mobile LLC is a limited liability company
`
`organized under the laws of Delaware and has a principal place of business near Seattle, WA. On
`
`information and belief, AMZN Mobile LLC provides mobile apps that are Alexa Products or are
`
`for use with Alexa Products.
`
`12.
`
`On information and belief, AMZN Mobile 2 LLC is a limited liability company
`
`organized under the laws of Delaware and has a principal place of business near Seattle, WA. On
`
`information and belief, AMZN Mobile 2 LLC provides mobile apps that are Alexa Products or
`
`are for use with Alexa Products.
`
`13.
`
`On information and belief, Amazon.com Services, Inc. f/k/a Amazon Fulfillment
`
`Services, Inc. is a corporation incorporated in Delaware and has a principal place of business at
`
`410 Terry Ave. N. Seattle, WA 98109. On information and belief, Amazon.com Services, Inc.
`
`f/k/a Amazon Fulfillment Services, Inc. participated in the sale or offer for sale of one or more
`
`Alexa Products.
`
`14.
`
`On information and belief, Amazon Digital Services LLC is a limited liability
`
`company organized under the laws of Delaware and has a principal place of business at 410
`
`Terry Ave. N, Seattle, WA 98109. On information and belief, Amazon Digital Services LLC has
`
`participated in the sale or offer for sale of one or more Alexa Products and/or has provided digital
`
`services and content for use by Alexa Products.
`
`JURISDICTION AND VENUE
`
`15.
`
`This Court has original jurisdiction over the subject matter of this action under 28
`
`U.S.C. §§ 1331 and 1338(a) because the action arises under the patent laws of the United States.
`
`4
`
`

`

`Case 1:19-cv-01410-MN Document 22 Filed 10/09/19 Page 5 of 71 PageID #: 353
`
`16.
`
`Personal jurisdiction over each Defendant is proper in this District based on one
`
`or more of the following: its presence in this judicial district; it has availed itself of the rights and
`
`benefits of the laws of Delaware; or it has derived substantial revenue from sales of Alexa
`
`Products in Delaware and it has systematic and continuous business contacts with Delaware.
`
`Each Defendant was incorporated in Delaware and/or formed under the laws of Delaware and
`
`Amazon designs Alexa Products, which are advertised, offered for sale, sold, and used in
`
`Delaware.
`
`17.
`
`Venue is proper in this district under 28 U.S.C. § 1400(b) and 28 U.S.C. §§
`
`1391(b)(1), (b)(2). For purposes of § 1400(b), each Defendant was incorporated in Delaware
`
`and/or formed under the laws of Delaware and therefore resides within this District. For purposes
`
`of § 1391(b)(1), (b)(2), Amazon resides in the District of Delaware by virtue of being
`
`incorporated in Delaware and/or formed under the laws of Delaware.
`
`FACTUAL BACKGROUND
`
`A.
`
`18.
`
`VoiceBox Technologies Invents Groundbreaking Voice Technology
`
`In 2001, three brothers, Mike, Rich, and Bob Kennewick founded VoiceBox
`
`Technologies to bring natural language understanding (“NLU”) to a wide array of computer
`
`applications. They recognized that the typical computer speech-recognition systems forced
`
`human operators to adhere to a limited number of rigid speech prompts, typically through verbal
`
`menus of a so-called “Command and Control” system. These rigid prompts limited how systems
`
`were used and inhibited the widespread adoption of speech-recognition systems. The brothers
`
`believed that VoiceBox Technologies could become the first company to improve voice
`
`recognition systems to enable people to naturally and effectively interact with computer speech
`
`systems.
`
`5
`
`

`

`Case 1:19-cv-01410-MN Document 22 Filed 10/09/19 Page 6 of 71 PageID #: 354
`
`19.
`
`From its inception, VoiceBox Technologies engaged in intense research efforts to
`
`develop its NLU technology. As part of these efforts, VoiceBox Technologies achieved a
`
`significant milestone when it developed an early prototype called “Cybermind.” As demonstrated
`
`on the local Seattle-area television news (called King5 news), Cybermind was a voice-controlled
`
`speaker that could provide weather, recipes, sports scores, calendar updates, or play a song.3
`
`Figure 1: Cybermind Prototype
`
`20.
`
`VoiceBox Technologies’ groundbreaking work did not go unrecognized. After
`
`learning about VoiceBox Technologies’ technology, Toyota hired it to build a sophisticated NLU
`
`speech interface for its Lexus automobiles. VoiceBox Technologies built the voice and NLU
`
`capability for Toyota’s award-winning Entune multimedia system.
`
`21.
`
`As part of the development effort of an NLU interface for Lexus, VoiceBox
`
`Technologies demonstrated a personal assistant called “Alexus” that showcased the power of its
`
`Conversational Voice technology. On information and belief, the VoiceBox Technologies
`
`3https://www.youtube.com/watch?v=DDcRyPnvWhw
`
`6
`
`

`

`Case 1:19-cv-01410-MN Document 22 Filed 10/09/19 Page 7 of 71 PageID #: 355
`
`“Alexus” concept was introduced to the public more than six months before Amazon announced
`
`“Alexa.”
`
`Figure 2: “Alexus” Demonstration
`
`22.
`
`Throughout its research and development efforts, VoiceBox Technologies realized
`
`that its technology could be deployed in a wide range of applications from connected home to
`
`mobile personal assistants.
`
`Figure 3: Connected Home
`
`Figure 4: Mobile
`
`23.
`
`By January 2012, VoiceBox Technologies was a leader in NLU and
`
`conversational voice technology. Leading companies throughout the world, including Toyota,
`
`Lexus, TomTom, Pioneer, Chrysler, Dodge, and Magellan used VoiceBox Technologies’ award-
`
`7
`
`

`

`Case 1:19-cv-01410-MN Document 22 Filed 10/09/19 Page 8 of 71 PageID #: 356
`
`winning and patented contextual speech technology. VoiceBox Technologies had software
`
`applications that ran on smart speakers, in-car systems, smartphones, smart TVs, computers,
`
`tablets, e-readers, and personal navigation devices.
`
`24.
`
`In 2013, the Institute of Electrical and Electronics Engineers (“IEEE”) ranked
`
`VoiceBox Technologies number 13 in patent power for the computer software industry.
`
`VoiceBox Technologies had become the leader in conversational Artificial Intelligence (“AI”),
`
`including Voice Recognition (VR), NLU, and AI services.
`
`B.
`
`25.
`
`Amazon Takes VoiceBox’s Technology
`
`In 2011, VoiceBox Technologies contacted Amazon to explore a potential
`
`business relationship where VoiceBox Technologies would provide core NLU services to
`
`Amazon. Amazon’s corporate development department expressed interest and asked for
`
`“company and/or product overview slides” to facilitate an October 7, 2011 teleconference. In
`
`response, VoiceBox Technologies provided Amazon with a presentation that described its award-
`
`winning patented technology and explicitly referred to VoiceBox Technologies’ “patented
`
`Contextual Speech Technology.” Slides from that presentation are reproduced below:
`
`Figure 5: 2011 Slides
`
`8
`
`

`

`Case 1:19-cv-01410-MN Document 22 Filed 10/09/19 Page 9 of 71 PageID #: 357
`
`26.
`
`Amazon was so impressed by the technology VoiceBox Technologies presented
`
`on October 7, 2011 that its representative emailed VoiceBox Technologies the next business day
`
`to invite VoiceBox to visit Amazon’s offices on October 19, 2011. That meeting was with
`
`Douglas Booms, Amazon’s Vice President of Worldwide Corporate Development, as well as
`
`engineers and product/business development members of Amazon’s devices and digital teams.
`
`The email from Amazon stated that this was “the right audience to discuss [VoiceBox
`
`Technologies’] personal digital assistant and underlying conversational voice technology.” On
`
`information and belief, in addition to Mr. Booms, the following Amazon personnel attended the
`
`meeting: Nick Komorous (Director, Corporate Development), Ian Freed (VP, Amazon Devices),
`
`Greg Hart (VP, Digital), Al Lindsay (Director, Software Engineer), Frederic Deramat (Senior
`
`Principal Engineer), and John Thimsen (Software Developer). On information and belief, another
`
`Amazon engineer from Cupertino attended by teleconference.
`
`27.
`
`Two days after the meeting at Amazon, on Friday October 21, 2011, Amazon’s
`
`Mr. Komorous emailed VoiceBox Technologies and asked to visit the company’s office for a
`
`“deeper dive.” Mr. Komorous requested that this meeting occur the very next week. VoiceBox
`
`Technologies agreed to host Amazon’s personnel for a meeting at VoiceBox Technologies’ office
`
`on October 26, 2011. On information and belief, from Amazon, Marcello Typrin (Product
`
`Management / Business Development), Frederic Deramat (Senior Principal Engineer), John
`
`Thimsen (Software Developer), and Sean Fitz (Software Developer) joined Mr. Komorous at the
`
`meeting.
`
`28.
`
`In advance of the meeting, Mr. Komorous sent a detailed set of technical
`
`questions that would help Amazon’s “tech team understand the scope and [r]ange of things
`
`[Amazon] can try to tackle with VoiceBox[] as a partner.” Mr. Komorous also indicated that
`
`9
`
`

`

`Case 1:19-cv-01410-MN Document 22 Filed 10/09/19 Page 10 of 71 PageID #: 358
`
`Amazon’s culture was “engineering heavy” and asked that VoiceBox Technologies “have
`
`engineering and speech representation at the meeting.”
`
`29.
`
`The October 26, 2011 meeting at VoiceBox Technologies’ office ran from 10 am
`
`until 12 noon with some Amazon engineers staying even later until around 2:30 pm. The meeting
`
`included a tour with additional demos, a review of the technical architecture for VoiceBox
`
`Technologies’ server software, and plans for next generation products.
`
`30.
`
`During the meeting, VoiceBox Technologies presented a deck of 42 slides to the
`
`visiting Amazon personnel. The slides provided even more detail about VoiceBox Technologies’
`
`patented technology and informed Amazon that VoiceBox Technologies had 12 patents at the
`
`time with an additional 14 pending applications. The slide deck included the following slide
`
`regarding the ’176 patent, which is asserted in this lawsuit.
`
`10
`
`

`

`Case 1:19-cv-01410-MN Document 22 Filed 10/09/19 Page 11 of 71 PageID #: 359
`
`Figure 6: Slide from 2011 Presentation
`
`31.
`
`The slide deck VoiceBox Technologies presented to Amazon at the October 26,
`
`2011 meeting also proposed a business arrangement where VoiceBox Technologies would
`
`provide “Voice Services” to Amazon. The VoiceBox Technologies Voice Services from the
`
`presentation are shown below:
`
`11
`
`

`

`Case 1:19-cv-01410-MN Document 22 Filed 10/09/19 Page 12 of 71 PageID #: 360
`
`Figure 7: Slide from 2011 Presentation
`
`32.
`
`On information and belief, some of the Amazon personnel involved in the 2011
`
`meetings became technical leaders for Amazon’s Alexa Products while others became high-level
`
`executives with close working relationships with the senior leadership of Amazon. On
`
`information and belief, one of Mr. Freed’s past positions was Technical Advisor to CEO, a
`
`position that has been described as CEO Jeff Bezos’s “shadow” advisor. On information and
`
`belief, Mr. Freed then went on to become Vice President, Amazon Devices where he led a team
`
`of engineers working on Amazon’s Alexa Products. On information and belief, Mr. Hart has also
`
`been Mr. Bezos’s “shadow” advisor—he held the Technical Advisor to CEO position around
`
`2011. On information and belief, around 2011, Al Lindsey was promoted to Vice President
`
`12
`
`

`

`Case 1:19-cv-01410-MN Document 22 Filed 10/09/19 Page 13 of 71 PageID #: 361
`
`managing the Alexa Engine Software team. On information and belief, around 2011, Mr.
`
`Deramat was promoted to the position Vice President & Distinguished Engineer for Amazon
`
`Alexa. On information and belief, around 2011, Mr. Thimsen was promoted to the position
`
`Director of Engineering for Amazon Echo. As for Mr. Typrin, he states on his LinkedIn page that
`
`he is “[o]ne of the founding members of the team that shaped the vision and direction for
`
`Amazon's Echo and Alexa Voice Services.”
`
`33.
`
`A couple days after the last meeting in 2011, VoiceBox Technologies sent an
`
`email to Amazon asking for Amazon’s feedback. Mr. Komorous from Amazon replied that
`
`Amazon was “still discussing internally how contextual speech / cybermind 2012 could play a
`
`part in [Amazon’s] future.” Amazon did not provide the results of these discussions and did not
`
`pursue a business relationship with VoiceBox Technologies. Instead, on information and belief,
`
`Amazon decided to build its Alexa Products—without telling VoiceBox Technologies or asking
`
`permission to use VoiceBox Technologies’ patented technology.
`
`34.
`
`In 2014, Amazon announced the launch of Alexa, a virtual assistant, along with
`
`the first-generation Echo product, a smart speaker. Amazon’s Alexa and first-generation Echo
`
`product were strikingly similar to the patented technology that VoiceBox Technologies showed
`
`Amazon in 2011.
`
`35.
`
`In 2016, Amazon abruptly hired Philippe Di Cristo, who was VoiceBox
`
`Technologies’ Chief Scientist. While at VoiceBox Technologies, Mr. Di Cristo gained knowledge
`
`of the company’s voice technology and had full access to VoiceBox Technologies’ intellectual
`
`property. As Mr. Di Cristo explains on his LinkedIn Page, he had worked on an “Amazon Echo-
`
`like system” while at VoiceBox Technologies. Exhibit A at 3. On information and belief, Mr. Di
`
`13
`
`

`

`Case 1:19-cv-01410-MN Document 22 Filed 10/09/19 Page 14 of 71 PageID #: 362
`
`Cristo has helped design and implement VoiceBox Technologies’ patented technology into
`
`Amazon’s Echo and other Alexa Products.
`
`36.
`
`On information and belief, Mr. Di Cristo played an active role in soliciting
`
`additional VoiceBox employees to join Amazon. For example, shortly after Mr. Di Cristo joined
`
`Amazon from VoiceBox Technologies, Amazon ramped-up its efforts to recruit VoiceBox
`
`Technologies employees.
`
`37.
`
`On January 10, 2017, Amazon hosted an “Evening with the Leadership of
`
`Amazon Voice & Advanced Shopping,” which Amazon expressly described as an “invite only
`
`networking event for Voice Box employees . . . to talk . . . about opportunities at Amazon.” The
`
`director of Amazon’s Voice & Advanced NUI Shopping group— on information and belief, the
`
`group Mr. Di Cristo joined—sent an invite for the event to a large number of VoiceBox
`
`Technologies employees and indicated that Amazon was “building a world-class speech & NLU
`
`engineering team” and “[y]our profile looks quite relevant and we’d love to talk to you and see if
`
`there’s a fit.” For this event, Amazon rented out Seastar, the premier seafood restaurant near
`
`VoiceBox Technologies’ office.
`
`38.
`
`On information and belief, Mr. Di Cristo was originally scheduled to be a speaker
`
`at the event targeting VoiceBox Technologies employees—but abruptly withdrew. The event
`
`emphasized Amazon’s need for VoiceBox Technologies’ NLU technology. On information and
`
`belief, around this time, Amazon faced numerous NLU challenges and had only completed a
`
`small portion of work toward Amazon’s goal to build conversational dialog interaction into
`
`Alexa Products.
`
`39.
`
`On January 17, 2017, after Amazon’s rampant poaching of VoiceBox
`
`Technologies employees came to light, Mike Kennewick, CEO of VoiceBox Technologies, sent a
`
`14
`
`

`

`Case 1:19-cv-01410-MN Document 22 Filed 10/09/19 Page 15 of 71 PageID #: 363
`
`letter to Jeff Bezos, CEO of Amazon, to propose a business solution. A true and correct copy of
`
`that letter is attached as Exhibit B. The letter explains that, at the time, VoiceBox Technologies
`
`had “a deep portfolio of technology and IP, including a large number of significant patents not
`
`only in NLU but also in Voice[]Commerce, running on over 200 million devices.”
`
`40.
`
`On information and belief, Mr. Booms, Amazon’s Vice President of Worldwide
`
`Corporate Development, responded to VoiceBox Technologies’ January 17, 2017 letter on behalf
`
`of Mr. Bezos. In Amazon’s response, Mr. Booms requested a meeting at VoiceBox Technologies’
`
`offices for purposes of “go[ing] fairly deep on the technology, data, customer relationships.”
`
`41.
`
`On February 2, 2017, the parties met at VoiceBox Technologies’ office. Amazon
`
`came with a team of technologists from its Alexa program, including on information and belief,
`
`Manoj Sindhwani (Director Alexa), Karthik Ramakrishnan (Senior Manager, Alexa software),
`
`Nikko Strom (Scientist, Alexa), and Deepesh Mohnani (Alexa Voice Services Product
`
`Management) joined by Mr. Booms. VoiceBox Technologies provided a detailed technical
`
`presentation, which included information about patents and pending applications then owned by
`
`VoiceBox Technologies. Slides in the presentation listed all VoiceBox Technologies patents and
`
`published applications at the time, including the ’681 patent, the ’049 patent, the ’176 patent, the
`
`’536 patent, and the ’097 patent. The February 2, 2017 slides also reproduced a highlighted claim
`
`from the ’681 patent and a highlighted claim from the ’176 patent.
`
`42.
`
`Following the meeting, Mr. Booms emailed VoiceBox Technologies to request
`
`even more technical details. Then, on February 20, 2017, Amazon specifically asked VoiceBox
`
`Technologies for a list of all patents owned by the company.
`
`43.
`
`In March of 2017, VoiceBox Technologies hosted another meeting with Amazon
`
`personnel at VoiceBox Technologies’ office. The Amazon attendees included senior executives
`
`15
`
`

`

`Case 1:19-cv-01410-MN Document 22 Filed 10/09/19 Page 16 of 71 PageID #: 364
`
`and managers responsible for Alexa. VoiceBox Technologies informed Amazon by email after
`
`the meeting that VoiceBox Technologies had “[p]atents that could be useful as the market goes
`
`mainstream.” Shortly, thereafter, Mr. Komorous from Amazon emailed VoiceBox Technologies,
`
`writing that Amazon had been “[poring] through the material” VoiceBox Technologies provided
`
`and that Amazon had created yet another, even more detailed, list of requests.
`
`44.
`
`In April of 2017, VoiceBox Technologies shared a written summary of VoiceBox
`
`Technologies’ patent portfolio with Amazon. The written summary explained that VoiceBox
`
`Technologies’ patents covered “core & fundamental areas.” The summary showcased the ’176
`
`patent and the ’703 patent, and identified the ’536 patent and ’097 patent by number. The
`
`summary further explained that “[v]oice advertising & conversational e-commerce are essential
`
`to the core strategies and future revenue streams of many of the world’s leading technology
`
`companies, including: . . . Amazon . . . .”
`
`OVERVIEW OF VOICEBOX’S PATENTS-IN-SUIT
`
`The ’681 and ’049 Patents
`United States Patent Number 8,073,681 (“the ’681 patent”), entitled “System and
`
`45.
`
`Method for a Cooperative Conversational Voice User Interface,” was duly and legally issued on
`
`December 6, 2011, and names Larry Baldwin, Tom Freeman, Michael Tjalve, Blane Ebersold,
`
`and Chris Weider as the inventors. Attached as Exhibit C is a true and correct copy of the ’681
`
`patent.
`
`46.
`
`The ’681 patent claims, among other things, a system for providing a cooperative
`
`conversational voice user interface, comprising: a voice input device configured to receive an
`
`utterance during a current conversation with a user; and a conversational speech engine, wherein
`
`the conversational speech engine includes one or more processors configured to: accumulate
`
`16
`
`

`

`Case 1:19-cv-01410-MN Document 22 Filed 10/09/19 Page 17 of 71 PageID #: 365
`
`short-term shared knowledge about the current conversation, wherein the short-term shared
`
`knowledge includes knowledge about the utterance received during the current conversation;
`
`accumulate long-term shared knowledge about the user, wherein the long-term shared knowledge
`
`includes knowledge about one or more past conversations with the user; identify a context
`
`associated with the utterance from the short-term shared knowledge and the long-term shared
`
`knowledge; infer additional information about the utterance from the short-term shared
`
`knowledge and the long-term shared knowledge in response to determining that the utterance
`
`contains insufficient information to complete a request in the identified context; establish an
`
`intended meaning for the utterance within the identified identify a context based on the
`
`additional information inferred about the utterance; and generate a response to the utterance
`
`based on the intended meaning established within the identified context.
`
`47.
`
`48.
`
`VoiceBox is the assignee of the entire right, title, and interest in the ’681 patent.
`
`United States Patent Number 9,015,049 (“the ’049 patent”), entitled “System and
`
`Method for a Cooperative Conversational Voice User Interface,” was duly and legally issued on
`
`April 21, 2015, and names Larry Baldwin, Tom Freeman, Michael Tjalve, Blane Ebersold, and
`
`Chris Weider as the inventors. Attached as Exhibit E is a true and correct copy of the ’049 patent.
`
`49.
`
`The ’049 patent claims, among other things, a system for facilitating
`
`conversation-based responses, the system comprising: one or more physical processors
`
`programmed with one or more computer program instructions such that, when executed, the one
`
`or more computer program instructions cause the one or more physical processors to: receive a
`
`natural language utterance during a conversation between a user and the system; identify a first
`
`model that includes short-term knowledge about the conversation, wherein the short-term
`
`knowledge is based on one or more prior natural language utterances received during the
`
`17
`
`

`

`Case 1:19-cv-01410-MN Document 22 Filed 10/09/19 Page 18 of 71 PageID #: 366
`
`conversation; identify, based on the short-term knowledge, context information for the natural
`
`language utterance; determine, based on the context information, an interpretation of the natural
`
`language utterance; and generate, based on the interpretation of the natural language utterance, a
`
`response to the natural language utterance.
`
`50.
`
`51.
`
`VoiceBox is the assignee of the entire right, title, and interest in the ’049 patent.
`
`Voice user interface systems in existence before the inventions of the ’681 and
`
`’049 patents were typically of the “Command and Control” type. Such systems used verbal
`
`menus to restrict information that a person can provide at a given point. For example, the voice
`
`system would present the list of available options either verbally and/or on a screen. The user
`
`could then respond by speaking the menu item. Such a system could include numerous menus
`
`that the user would have to get through in order to convey the desired information to the system
`
`and/or to cause the system to take the desired action.
`
`52.
`
`The inventors recognized a significant problem with the Command and Control
`
`systems in that users would have to memorize exact words and phrases in order to interact with
`
`the system. This required significant learning because the user had to know which words and
`
`phrases to use in order to make a request of a Command and Control voice user interface system.
`
`Additionally, the process of stepping through menus could be time-consuming and, in some
`
`cases might dissuade a user from utilizing the voice-based system.
`
`53.
`
`To overcome the shortcoming of prior art systems, the inventors provided a
`
`solution that used an “Automatic Speech Recognizer” (or ASR) to generate a preliminary
`
`interpretation and provide that preliminary interpretation to a “conversational speech engine” for
`
`further processing. The conversational speech engine—which was not well-understood, routine,
`
`or conventional—could be implemented locally on a user device or remotely on a server. In
`
`18
`
`

`

`Case 1:19-cv-01410-MN Document 22 Filed 10/09/19 Page 19 of 71 PageID #: 367
`
`certain embodiments, the conversational speech engine included a conversational language
`
`processor, voice search engine / free form voice search module with context domain agents, and
`
`a context determination process. The conversational speech engine communicates with databases
`
`to generate an adaptive conversational response.
`
`54.
`
`Through the use of a conversational speech engine, the ’681 and ’049 patents
`
`advantageously rely on conversational responses which, in some embodiments, use short-term
`
`and long-term shared knowledge about user utterances to determine a context for the request,
`
`infer additional information about a request, and provide an adaptive conversational response.
`
`55.
`
`The inventors were thereby able to improve the functioning of voice user interface
`
`systems which improved the operation of those systems in an unconventional manner. For
`
`example, the innovations in the ’681 and ’049 patents allowed a user to converse naturally with a
`
`voice user interface system instead of “dumbing down” their requests to match the simple sets of
`
`instructions that existing Command and Control systems required. In this regard, one of the
`
`problems faced by the inventors was necessarily rooted in voice user interface technology
`
`specifically arising in the realm of voice user interface systems. Indeed, the commercial success
`
`and industry accolades provide objective evidence as to Voicebox’s innovative approach through
`
`the use unconventional technology.
`
`56.
`
`The ’681 patent describes and claims a system for providing a cooperative
`
`conversational voice user interface with the above-described conversational speech engine. The
`
`conversational speech engine has a processor that accumulates both short-term and long-term
`
`shared knowledge. The short-term shared knowledge includes knowledge about an utterance
`
`received during a current conversation while the long-term shared knowledge includes
`
`knowledge about one or more past conversations with the user.
`
`19
`
`

`

`Case 1:19-cv-01410-MN Document 22 Filed 10/09/19 Page 20 of 71 PageID #: 368
`
`57.
`
`The known prior art in the field of voice user interfaces neither taught
`
`accumulating short-term and long-term shared knowledge nor expressed any appreciation for the
`
`substantial advantages associated with utilizing this shared knowledge for various purposes in a
`
`conversational speech engine. Such uses include to identify context, infer additional information
`
`about an utterance that contains insufficient information to complete a request, establish an
`
`intended meaning for an utterance within the context based on the additional information
`
`inferred about the utterance, and generate a response based on the intended meaning established
`
`within the identified context. In this regard, accumulating and using both short-term and long-
`
`term shared knowledge was not well-understood, routine, or conventional and stands in sharp
`
`contrast to the conventional and routine approach of command and control systems that require a
`
`user to use rigid menus to establish context before making a request.
`
`58.
`
`During prosecution of the ’681 patent, the examiner rejected numerous
`
`application claims as being unpatentable over Kargman (US 2005/0015