Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`
`
`
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`
`
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`
`
`
`NIANTIC, INC.,
`Petitioner
`
`v.
`
`NANT HOLDINGS IP, LLC,
`Patent Owner
`
`
`
`U.S. Patent No. 10,403,051 B2
`Issue Date: September 3, 2019
`
`
`
`Title: Interference Based Augmented Reality Hosting Platforms
`
`PETITION FOR INTER PARTES REVIEW
`OF U.S. PATENT NO. 10,403,051 B2
`
`
`
`
`

`

`TABLE OF CONTENTS
`
`
`Page
`
`
`I. MANDATORY NOTICES UNDER §42.8(A)(1) ......................................... 1
`A.
`Real Party-In-Interest under §42.8(b)(1) .............................................. 1
`B.
`Related Matters under §42.8(b)(2) ....................................................... 1
`C.
`Lead and Back-Up Counsel under §42.8(b)(3) .................................... 1
`D.
`Service Information .............................................................................. 3
`E.
`Power of Attorney ................................................................................ 3
`FEE PAYMENT ............................................................................................. 4
`II.
`III. REQUIREMENTS UNDER §§ 42.104 AND 42.108 AND
`CONSIDERATIONS UNDER §§ 314(A) AND 325(D) ............................... 4
`A. Grounds for Standing under 37 C.F.R. § 42.104(a) ............................. 4
`B.
`Identification of Challenge Under 37 C.F.R. § 42.104(b) and
`Statement of Precise Relief Requested ................................................ 4
`Considerations under §§ 314 and 325(d) ............................................. 5
`C.
`37 C.F.R. § 42.108(c) ........................................................................... 5
`D.
`IV. OVERVIEW OF THE ’051 PATENT ........................................................... 6
`A.
`Level of Ordinary Skill in the Art ........................................................ 6
`B.
`Specification Overview ........................................................................ 6
`CLAIM CONSTRUCTION ......................................................................... 10
`V.
`VI. THE CHALLENGED CLAIMS ARE UNPATENTABLE ......................... 11
`A.
`Summary and Date Qualification of the Prior Art ............................. 12
`1.
`Yu [Ex. 1003] ........................................................................... 12
`2.
`Sanz-Pastor [Ex. 1004] ............................................................ 14
`3. Mullen [Ex. 1005] .................................................................... 16
`B. Ground 1: Single-Reference Obviousness of Claims 1, 5-9, 18,
`22-23, 27-29, 34, 36, 38, 43 Over Yu ................................................ 18
`1.
`Claim 1 ..................................................................................... 18
`
`
`
`
`
`-i-
`
`
`
`

`

`TABLE OF CONTENTS
`(continued)
`
`Page
`
`2.
`
`3.
`
`4.
`
`5.
`
`6.
`
`Claim 5: “The system of claim 1, wherein the AR server
`is a remote server coupled with the AR capable device
`via a wireless network.” ........................................................... 43
`Claim 6: “The system of claim 1, wherein the relevant
`AR object is caused to be rendered based on a position of
`the AR capable device relative to the environment.” .............. 45
`Claim 7: “The system of claim 1, wherein the relevant
`AR object is caused to be rendered based on an
`orientation of the AR capable device relative to the
`environment.” ........................................................................... 45
`Claim 8: “The system of claim 1, wherein the relevant
`AR object is caused to be rendered within an AR game.”....... 46
`Claim 9: “The system of claim 1, wherein the relevant
`AR object is caused to be rendered by superimposing a
`visual image of at least one relevant AR object over an
`image of the environment.” ...................................................... 47
`Claim 18: “The system of claim 1, wherein the presence
`of the relevant AR object is altered to include a non-
`visible presence.” ..................................................................... 48
`Claim 22: “The system of claim 1, wherein the
`determination of whether to alter presence of the relevant
`AR object depends on a time.” ................................................ 49
`Claim 23: “The system of claim 22, wherein the presence
`alteration of the relevant AR object changes with the
`time.” ........................................................................................ 50
`10. Claim 27: “The system of claim 1, wherein the AR server
`is further configured to enable the AR capable device to
`populate the environment with at least one of the relevant
`AR objects.” ............................................................................. 51
`11. Claim 28: “The system of claim 1, wherein the
`environment comprises an AR environment.” ......................... 52
`
`7.
`
`8.
`
`9.
`
`-ii-
`
`
`
`
`
`
`
`
`
`

`

`TABLE OF CONTENTS
`(continued)
`
`Page
`
`12. Claim 29: “The system of claim 28, wherein the AR
`environment includes real-world elements and virtual
`elements.” ................................................................................. 54
`13. Claim 34: “The system of claim 1, wherein the AR server
`is further configured to enable the AR capable device to
`have an interaction with the rendered relevant AR
`object.” ..................................................................................... 54
`14. Claim 36: “The system of claim 1, wherein the mobile
`device is a cell phone.” ............................................................ 54
`15. Claim 38: “The system of claim 1, wherein the mobile
`device is a tablet computer.” .................................................... 55
`Independent claim 43 ............................................................... 55
`16.
`C. Ground 2: Obviousness of Claims 1, 5-11, 15, 18, 22-23, 27-
`29, 34, 36, 38, and 43 Over Yu in View of Sanz-Pastor .................... 57
`1.
`Independent Claim 1 ................................................................ 58
`2.
`Claim 10: “The system of claim 1, wherein the relevant
`AR object is caused to be rendered within an overlapping
`augmented reality among multiple AR capable devices.” ....... 64
`Claim 11: “The system of claim 10, wherein the
`overlapping augmented reality comprises a team-based
`augmented reality.” .................................................................. 65
`Claim 15: “The system of claim 1, wherein the relevant
`AR object is caused to be rendered according to a haptic
`format.” .................................................................................... 65
`Dependent Claims 5-9, 18, 22-23, 27-29, 34, 36, and 38,
`and Independent Claim 43 ....................................................... 66
`D. Ground 3: Obviousness of Claims 1, 5-11, 15, 18, 22-29, 34-
`36, 38, 43 Over Yu in View of Sanz-Pastor, in Further View of
`Mullen................................................................................................. 67
`1.
`Claim 8: “The system of claim 1, wherein the relevant
`AR object is caused to be rendered within an AR game.”....... 67
`
`3.
`
`4.
`
`5.
`
`-iii-
`
`
`
`
`
`
`
`
`
`

`

`TABLE OF CONTENTS
`(continued)
`
`Page
`
`
`
`2.
`
`Claim 24: “The system of claim 1, wherein the AR server
`is further configured to enable the AR capable device to
`conduct a commercial transaction with a commerce
`engine.” .................................................................................... 69
`Claim 25: “The system of claim 24, wherein the
`commercial transaction includes an exchange of virtual
`currency.” ................................................................................. 70
`Claim 26: “The system of claim 24, wherein the
`commercial transaction includes a transfer of real-world
`funds.” ...................................................................................... 71
`Dependent claim 35 ................................................................. 71
`5.
`VII. CONCLUSION ............................................................................................. 76
`
`3.
`
`4.
`
`
`
`
`
`-iv-
`
`
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`List of Exhibits
`
`
`
`
`Description of Document
`Ex. No.
`1001 U.S. Patent No. 10,403,051 (“’051 patent” or “’051”)
`1002 Declaration of Dr. Michael Zyda (“Zyda”)
`1003 U.S. Patent Application Publication No. 2010/0066750 A1 to Yu et
`al. (“Yu”)
`1004 U.S. Patent Application Publication No. 2007/0242131 A1 to Sanz-
`Pastor et al. (“Sanz-Pastor”)
`1005 U.S. Patent Application Publication No. 2006/0105838 A1 to Mullen
`(“Mullen”)
`
`1006
`1007
`1008
`
`1009
`
`Reserved
`
`Reserved
`
`Excerpts from Oliver Bimber and Ramesh Raskar, Chapter 1: A Brief
`Introduction to Augmented Reality, in Spatial Augmented Reality:
`Merging Real and Virtual World (2005)
`
`Paul Milgram and Fumio Kishino, A Taxonomy of Mixed Reality
`Visual Displays, IEICE Transactions on Information Systems, E77-D
`(12):1321-29 (1994)
`
`1010
`
`Jean-Marie Normand et al., A new typology of augmented reality
`applications, AH ’12 Proceedings of the 3rd Augmented Human Int’l
`Conference (Mar. 2012)
`1011 U.S. Patent No. 3,050,870 (“Heilig”)
`1012
`
`Ivan E. Sutherland, A head-mounted three dimensional display,
`AFIPS ’68 (Fall, Part I): Proceedings of the December 9-11, 1968
`Fall Joint Comput. Conf., Part I, pp. 757-64 (1968)
`
` 
`
`
`
`‐i‐ 
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`List of Exhibits
`
`
`
`Description of Document
`Ex. No.
`1013 Grégory Maubon, A little bit of history from 2006: Nokia MARA
`project, https://www.augmented-reality.fr/2009/03/un-petit-peu-
`dhistoire-de-2006-projet-mara-de-nokia/
`1014 U.S. Patent No. 6,553,310 (“Lopke”)
`1015 U.S. Patent Application Publication No. 2002/0044152 (“Abbott”)
`1016
`
`Excerpts from Lester Madden, Professional Augmented Reality
`Browsers for Smartphones, Programming for junaio, Layar, and
`Wikitude (2011)
`
`1017
`
`1018
`
`1019
`
`Jonathan Raper et al., Applications of location-based services: a
`selected review, J. Location Based Servs., 1(2):89-111 (2007)
`
`Paul G. Savage, Blazing Gyros: The Evolution of Strapdown Inertial
`Navigation Technology for Aircraft, J. of Guidance, Control, and
`Dynamics, 36(3):637-55 (2013)
`
`Jeong Won Kim, A Step, Stride and Heading Determination for the
`Pedestrian Navigation System, J. Global Positioning Sys., 3(1-2):273-
`79 (2004)
`
`1020
`
`Press Release, Apple Reinvents the Phone with iPhone (Jan. 9, 2007),
`https://www.apple.com/newsroom/2007/01/09Apple-Reinvents-the-
`Phone-with-iPhone/
`1021 U.S. Patent No. 8,745,494 (“Spivak”)
`1022 U.S. Patent No. 5,848,373 (“DeLorme”)
`1023 Michael R. Macedonia et al., Exploiting Reality with Multicast
`Groups, Virtual Reality (1995)
`1024 U.S. Patent No. 6,882,933 (“Kondou”)
`1025
`
`Reserved
`
`
` 
`
`
`
`‐ii‐ 
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`List of Exhibits
`
`
`
`Description of Document
`Ex. No.
`1026 U.S. Patent No. 9,317,133 (“Korah”)
`1027 Duy-Nguyen Ta et al., SURFTrac: Efficient Tracking and Continuous
`Object Recognition using Local Feature Descriptors, 2009 IEEE
`Conference on Computer Vision and Pattern Recognition, pp. 2937-
`44 (2009)
`
`1028
`
`1029
`
`Carsten Magerkurth et al., Pervasive Games: Bringing Computer
`Entertainment Back to the Real World, ACM Comput. In Entm’t,
`3(3):1-19 (July 2005)
`
`Bruce Thomas et al., ARQuake: An Outdoor/Indoor Augmented
`Reality First Person Application, IEEE Int’l Symposium on Wearable
`Comput., pp. 139-46 (2000)
`
`1030
`
`Bruce Thomas et al., First Person Indoor/Outdoor Augmented Reality
`Application: ARQuake, Personal and Ubiquitous Computing, 6:75-86
`(2002)
`1031 U.S. Patent Application Publication No. 2004/0164897 (“Treadwell”)
`1032 Michael Zyda, From Visual Simulation to Virtual Reality to Games,
`IEEE, 38(9):25-32 (Sept. 2005)
`1033 Michael Zyda, Creating a Science of Games, Commc’ns of the ACM,
`50(7):27-29 (July 2007)
`
`1034
`1035
`1036
`1037
`1038
`
` 
`
`
`
`Reserved
`
`Reserved
`
`Reserved
`
`Reserved
`
`Excerpts from NantWorks’ Disclosure of Asserted Claims and
`Infringement Contentions and Ex. 1, dated 1/28/2021, served in
`
`‐iii‐ 
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`List of Exhibits
`
`
`
`Ex. No.
`
`1039
`
`1040
`1041
`1042
`
`Description of Document
`Nantworks, LLC v. Niantic, Inc., Case No. 3:20-cv-06262-LB (N.D.
`Cal.)
`
`Excerpts from Microsoft Corporation, Microsoft Computer
`Dictionary (2002)
`
`Reserved
`
`Reserved
`
`Patent L.R. 4-3 Joint Claim Construction and Prehearing Statement,
`ECF No. 61, dated 5/18/2021, filed in Nantworks, LLC v. Niantic,
`Inc., Case No. 3:20-cv-06262-LB (N.D. Cal.)
`
`1043
`
`Brett Zarda, EA Games to Incorporate Real-Time Weather, WIRED
`(Aug. 17, 2007), https://www.wired.com/2007/08/madden/
`1044 Michael Zyda et al., Designing a Massively Multiplayer Online
`Game/Research Testbed Featuring AI-Driven NPC Communities,
`Proceedings of the Sixth AAAI Conf. on Artificial Intel. & Digit.
`Ent., pp. 108-11 (2010)
`1045 U.S. Patent Application Publication No. 2008/0262910 (“Altberg”)
`1046
`
`Proof of Service of Summons and Complaint, ECF No. 12, dated
`Sept. 8, 2020, filed in Nantworks, LLC v. Niantic, Inc., Case No.
`3:20-cv-06262-LB (N.D. Cal.)
`
`1047
`
`Third Amended Complaint for Patent Infringement, ECF No. 50,
`dated Feb. 23, 2021, filed in Nantworks, LLC v. Niantic, Inc., Case
`No. 3:20-cv-06262-LB (N.D. Cal.)
`
` 
`
`
`
`‐iv‐ 
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`This is a petition for inter partes review of claims 1, 5-11, 15, 18, 22-29, 34-
`
`36, 38, and 43 of U.S. Patent No. 10,403,051 (Ex. 1001) (“’051 Patent” or “’051”)).
`
`I. MANDATORY NOTICES UNDER §42.8(A)(1)
`A. Real Party-In-Interest under §42.8(b)(1)
`Niantic, Inc. (“Niantic” or “Petitioner”) is the real party-in-interest to this IPR
`
`Petition.
`
`B. Related Matters under §42.8(b)(2)
`The ’051 Patent is the subject of pending litigation involving Petitioner:
`
`NantWorks, LLC, and Nant Holdings IP, LLC v. Niantic, Inc., Case No. 3:20-cv-
`
`06262-LB (N.D. Cal.). No trial date has been set.
`
`Petitioner was first served on September 8, 2020. (Ex. 1046.) The Third
`
`Amended Complaint in that action alleges that Niantic infringes the ’051 Patent.
`
`(Ex. 1047, ¶¶1, 20-34, 62-86.)
`
`C. Lead and Back-Up Counsel under §42.8(b)(3)
`Petitioner provides the following designation of counsel.
`
`LEAD COUNSEL
`
`BACK-UP COUNSEL
`
`Heidi L. Keefe (Reg. No. 40,673)
`hkeefe@cooley.com
`
`COOLEY LLP
`ATTN: Patent Group
`1299 Pennsylvania Ave. NW, Suite 700
`Washington, DC 20004
`Tel: (650) 843-5001
` 
`
`
`
`Lauren J. Krickl (Reg. No. 70,261)
`lkrickl@cooley.com
`
`COOLEY LLP
`ATTN: Patent Group
`1299 Pennsylvania Avenue NW
`Suite 700
`Washington D.C. 20004
`
`-1-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`
`LEAD COUNSEL
`
`BACK-UP COUNSEL
`
`Fax: (650) 849-7400
`
`
`
`
`
`
`
`
`
`
`
` 
`
`Tel: (650) 843-5065
`Fax: (650) 849-7400
`Dena Chen (Admission pro hac vice to be
`requested)
`dchen@cooley.com
`
`COOLEY LLP
`ATTN: Patent Group
`1299 Pennsylvania Ave. NW, Suite 700
`Washington D.C. 20004
`Tel: (650) 843-5135
`Fax: (650) 849-7400
`Alexandra D. Leeper (Admission pro hac
`vice to be requested)
`aleeper@cooley.com
`
`COOLEY LLP
`ATTN: Patent Group
`1299 Pennsylvania Avenue NW
`Suite 700
`Washington D.C. 20004
`Tel: (650) 843-5376
`Fax: (650) 849-7400
`Patrick W. Lauppe (Admission pro hac
`vice to be requested)
`plauppe@cooley.com
`
`COOLEY LLP
`ATTN: Patent Group
`1299 Pennsylvania Avenue NW
`Suite 700
`Washington D.C. 20004
`Tel: (650) 843-5833
`Fax: (650) 849-7400
`
`-2-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`D.
`Service Information
`This Petition is being served by Federal Express to the attorneys of record for
`
`the ’051 Patent, Mauriel Kapouytian Woods LLP, 15 W. 26th Street, Floor 7, New
`
`York, NY 10010. This Petition is also being served on litigation counsel for Patent
`
`Owner1 identified in the Certificate of Service. Petitioner consents to electronic
`
`service at the addresses provided above for lead and back-up counsel.
`
`E.
`Power of Attorney
`Filed concurrently per 37 C.F.R. § 42.10(b).
`
`
`1 The assignment records for the ’051 Patent show Nant Holdings IP, LLC as the
`
`sole Patent Owner. This is consistent with NantWorks’ assertion in the underlying
`
`litigation that Nant Holdings IP “solely owns the Asserted Patents,” with
`
`NantWorks, LLC as “the exclusive licensee of the Asserted Patents.” (Ex. 1047,
`
`¶19.) To the extent any ambiguity exists regarding whether NantWorks, LLC is an
`
`owner of the ’051 Patent, Petitioner is serving this Petition on counsel of record for
`
`Nant Holdings IP, LLC and NantWorks, LLC (collectively, “NantWorks”).
`
`Petitioner reserves the right to correct any clerical error.
`
` 
`
`
`
`-3-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`II.
`FEE PAYMENT
`This Petition requests review of claims 1, 5-11, 15, 18, 22-29, 34-36, 38, and
`
`43 of the ’051 patent. A payment of $44,875 is submitted herewith. See 37 C.F.R.
`
`§ 42.15(a). This Petition meets the fee requirements of 35 U.S.C. §312(a)(1).
`
`III. REQUIREMENTS UNDER §§ 42.104 AND 42.108 AND CONSIDERATIONS UNDER
`§§ 314(A) AND 325(D)
`A. Grounds for Standing under 37 C.F.R. § 42.104(a)
`Petitioner certifies that the ’051 Patent is available for IPR and that Petitioner
`
`is not barred or otherwise estopped.
`
`B.
`
`Identification of Challenge Under 37 C.F.R. § 42.104(b) and
`Statement of Precise Relief Requested
`Petitioner requests institution of IPR based on the following grounds:
`
`Ground
`
`Claims
`
`Basis for Challenge under §103(a)
`
`1
`
`2
`
`3
`
`1, 5-9, 18, 22-
`23, 27-29, 34,
`36, 38, 43
`
`1, 5-11, 15, 18,
`22-23, 27-29,
`34, 36, 38, 43
`
`1, 5-11, 15, 18,
`22-29, 34-36,
`38, 43
`
`Yu (Ex. 1003)
`
`Yu (Ex. 1003) and Sanz-Pastor (Ex. 1004)
`
`Yu (Ex. 1003), Sanz-Pastor (Ex. 1004), and Mullen
`(Ex. 1005)
`
`Submitted with this Petition is the Declaration of Dr. Michael Zyda (Ex. 1002)
`
`(“Zyda”), a qualified technical expert. (Zyda, ¶¶1-9, Ex. A.)
`
` 
`
`
`
`-4-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`C. Considerations under §§ 314 and 325(d)
`§314(a): Petitioner is aware of no prior petitions challenging the ’051
`
`Patent. Petitioner filed this IPR petition shortly after the parties’ joint claim
`
`construction brief was filed on May 18, 2021, and before NantWorks’ opening claim
`
`construction brief is scheduled to be filed (July 2, 2021). The Board should exercise
`
`its discretion to institute this inter partes review under §314(a) because the litigation
`
`is in its relatively early stages, no trial date has been set, and there is no substantial
`
`overlap of the issues and arguments presented in this Petition with those presented
`
`at the district court. See Apple v. Fintiv, IPR2020-00019, Paper 15 (P.T.A.B.
`
`May 13, 2020). For example, this Petition challenges six claims not asserted in
`
`district court, and presents grounds for obviousness that rely on primary prior art
`
`references not asserted in district court.
`
`§325(d): None of the prior art references relied upon in Petitioner’s Grounds
`
`were cited during prosecution of the ’051 Patent. Accordingly, the challenges
`
`presented in this Petition rely on new art and grounds of unpatentability different
`
`from any presented during prosecution.
`
`D.
`37 C.F.R. § 42.108(c)
`The Board should institute inter partes review of claims 1, 5-11, 15, 18, 22-
`
`29, 34-36, 38, and 43 because this Petition establishes a reasonable likelihood of
`
`prevailing with respect to each challenged claim. See 35 U.S.C. § 314(a).
`
` 
`
`
`
`-5-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`IV. OVERVIEW OF THE ’051 PATENT
`A. Level of Ordinary Skill in the Art
`A person of ordinary skill (“POSA”) as of April 20112 would have possessed
`
`at least a Master of Science in the areas of electrical engineering or computer science
`
`(or equivalent degree), with some working knowledge of augmented reality, mobile
`
`gaming, and the associated technologies; or, alternatively, a Bachelor of Science in
`
`computer science (or equivalent degree) with at least two years of experience in the
`
`aforementioned areas. (Zyda, ¶¶12-16.)
`
`B.
`Specification Overview
`The ’051 Patent is directed to augmented reality (“AR”) systems that
`
`“present[] [] virtual objects along side [sic] real-world elements.” (’051, 1:32-33.)
`
`The claimed AR systems “adjust the presence of the [AR] objects within the [AR]
`
`experience” based on a user’s physical location. (Id., Abstract, 4:10-12, 16:9-16.)
`
`The ’051 Patent acknowledges that systems and methods for modifying the
`
`appearance of virtual objects within augmented reality scenes were known. (Id.,
`
`
`2 The ’051 Patent was filed on November 9, 2018, claiming priority to a succession
`
`of continuation application and one divisional application dating back to June 30,
`
`2011. The ’051 Patent also claims priority to U.S. Provisional Application No.
`
`61/473,324 (Ex. 1001), filed on April 8, 2011.
`
` 
`
`
`
`-6-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`1:37-41, 2:3-27.)
`
`Figure 1 is a “schematic of an augmented reality ecosystem” according to the
`
`’051 Patent (’051, 4:35):
`
`(Id., Fig. 1.) As shown, the AR ecosystem “include[s] one or more databases
`
`including AR object repositories 140 storing AR objects 142[.]” (Id., 5:63-66.) The
`
`system “obtain[s] a digital representation of a scene” local to an “AR-capable
`
`
`
`device” such as a mobile device. (Id., 3:63-4:16.) The digital representation can
` 
`
`
`
`-7-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`include “[e]lement attributes,” which include “myriad types of attributes” such as
`
`“location information (e.g., relative location to elements 390 of a scene,
`
`triangulation, GPS coordinates, etc.)[.]” (Id., 16:4-19.)
`
`The system can also “determine a context related to the scene based on the
`
`digital representation and pertaining to the target object.” (’051, 4:10-12.) The ’051
`
`Patent uses the term “context” broadly, stating that “[c]ontexts can take on many
`
`different forms and can be defined as desired.” (Id., 8:23-24; see also id., 18:66-
`
`19:1 (referring to “other contexts pertaining to circumstances of scene 595”).)
`
`Determining a “context” in the ’051 Patent refers to using information such as the
`
`device’s location, the user’s identity, and the time of day to inform the system’s
`
`determination of what AR objects to present to the user. (See, e.g., id., 5:10-13
`
`(“[C]ontemplated infrastructures determine a relevant augmented reality context
`
`from environment data representing a real-world environment local to an AR-
`
`capable devices.”), 9:21-26 (“[O]ne could still have a distinct context from other
`
`local devices, possibly based on device use, user identity or preferences,
`
`authorization, authentication, interference among other elements in AR ecosystem
`
`100, or other attributes.”), 17:27-30 (“As scene changes with time, context 332 can
`
`ebb or flow, or even shift focus…from a first context to a second context.”).)
`
`Once the system has identified a “context,” the system “identif[ies] a set of
`
`relevant AR objects from available AR objects with respect to the context based on
`
` 
`
`
`
`-8-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`a derived interference among elements (e.g., real-world elements, virtual elements,
`
`etc.)[,]” and presents those relevant AR objects to the AR-capable device. (’051,
`
`4:13-19.)
`
`The presence of the relevant AR objects can be “enhanced” or “suppressed”
`
`depending on “the nature of the relevant AR objects[], the context, or other factors
`
`relating to the scene.” (’051, 18:18-21.) “At a most basic level, presence could
`
`simply mean relevant AR objects are present (enhanced) or not present
`
`(suppressed).” (Id., 18:21-23.) Figure 4 provides another example where AR object
`
`446A with “enhanced presence” appears larger, while AR object 446B with
`
`“suppressed presence” appears smaller:
`
` 
`
`
`
`-9-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`
`
`
`(Id., Fig. 4.)
`
`V. CLAIM CONSTRUCTION
`As of the Petition’s filing, the claim construction process in the underlying
`
`litigation is ongoing, and the district court has made no claim construction ruling.
`
`The parties dispute the construction of several claim terms. (See Ex. 1042.) For
`
` 
`
`
`
`-10-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`purposes of the present Petition, Petitioner has identified these potential disputes in
`
`the following analysis so that the Board can be aware of them, but respectfully
`
`submits that they need not be expressly resolved by the Board for purposes of
`
`evaluating Petitioner’s proposed Grounds. As the detailed analysis below will
`
`demonstrate, the prior art would satisfy the constructions proposed by either party.
`
`VI. THE CHALLENGED CLAIMS ARE UNPATENTABLE
`This Petition challenges claims 1, 5-11, 15, 18, 22-29, 34-36, 38, and 43
`
`(collectively, the “Challenged Claims”), based on three Grounds of obviousness.3
`
`Ground 1 is a single-reference obviousness ground based on Yu alone, and addresses
`
`claims 1, 5-9, 18, 22-23, 27-29, 34, 36, 38, and 43. Ground 2 builds upon Ground 1
`
`by adding Sanz-Pastor, and further addresses claims 10, 11, and 15. Ground 3 builds
`
`upon Ground 2 by adding Mullen, and further addresses claims 24-26 and 35.
`
`
`3 Claims 10, 11, 15, 18, 34, and 35 are not asserted in the underlying district court
`
`litigation. (See Ex. 1038, NantWorks’ Disclosure of Asserted Claims and
`
`Infringement Contentions, at 1.)
`
` 
`
`
`
`-11-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`A.
`Summary and Date Qualification of the Prior Art4
`
`Yu [Ex. 1003]
`Yu, titled “Mobile Virtual and Augmented Reality System,” describes an AR
`
`system where “[a] user can create ‘virtual graffiti’ [] that will be left for a particular
`
`device to view as part of an augmented-reality scene.” (Yu, Abstract.) The virtual
`
`graffiti is an object that overlays the real scene, thus augmenting reality. Yu qualifies
`
`as prior art under § 102(b) because it was published March 18, 2010, more than one
`
`year before the earliest date to which the ’051 Patent claims priority.
`
`The virtual graffiti in Yu can be “assigned to a particular physical location”
`
`and “uploaded to a network server[.]” (Yu, Abstract.) “When a device that is
`
`allowed to view the graffiti is near the location [to which the virtual graffiti is
`
`assigned], the graffiti will be downloaded to the device and displayed as part of an
`
`augmented-reality scene.” (Id.) Figure 2 illustrates three scenes where the virtual
`
`graffiti is either not present (left scene) or present (middle and right scenes) atop the
`
`“real-world door”:
`
`
`4 As explained by Dr. Zyda, each of the prior art references described herein are
`
`enabling. (Zyda, ¶¶511-513.)
`
` 
`
`
`
`-12-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`
`
`
`(Id., Fig. 2, ¶0024.)
`
`Yu further discloses that the presence of the virtual graffiti, again overlaid
`
`atop the real-world door, can be modified based on ambient-light conditions. (Yu,
`
`Abstract, ¶¶0004, 0015, 0028, 0063-0068.) Figure 3 provides an example where
`
`“the shadowing of a virtual object may be allowed to change based on, for example,
`
`the position of the sun” (id., ¶0026):
`
` 
`
`
`
`-13-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`
`
`
`(Id., Fig. 3, ¶0027.) The presence of the virtual graffiti also changes based on “the
`
`physical position and orientation (i.e., viewing direction) of the device.” (Yu,
`
`¶0032.)
`
`
`Sanz-Pastor [Ex. 1004]
`Sanz-Pastor describes an AR system where users can add “virtual sticky
`
`notes” (objects) to a particular physical location within an AR scene. (Sanz-Pastor,
`
`¶0070, Abstract.) The point is for people to be able to communicate about, and on
`
`top of, real-life scenes using those virtual messages. Sanz-Pastor qualifies as prior
`
`art under § 102(b) because it was published October 18, 2007, more than one year
`
`before the earliest date to which the ’051 Patent claims priority.
` 
`-14-
` 
`
`
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`Sanz-Pastor explains that users view the virtual sticky note messages “by
`
`panning the [user] device, revealing the message’s real world location as icons and
`
`text overlaid on top of the camera input on the display.” (Sanz-Pastor, Abstract.)
`
`Additionally, Sanz-Pastor provides implementation details for “simulat[ing]
`
`occlusion effects.” (Id.; see also, e.g., id., ¶0051 (“Message management module
`
`202 determines a geometric intersection from a device’s location to the coordinate
`
`of each message, and by comparing the resulting range with the actual distance
`
`between the two points, determines whether the message is visible from the device’s
`
`position.”).) Figure 4 depicts an example where the virtual messages are overlaid
`
`on the real-world image to first responders in emergency situations:
`
` 
`
`
`
`-15-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`
`
`
`(Id., Fig. 4, ¶¶0044-0045.)
`
`Sanz-Pastor further discloses that the user device can present virtual messages
`
`“in an attenuated fashion by making them transparent when drawn using a different
`
`color coding.” (Sanz-Pastor, ¶0029.)
`
` Mullen [Ex. 1005]
`Mullen discloses AR systems providing “location-based games” where “a
`
`user’s physical location correlates to the virtual location of a virtual character on a
`
`virtual playfield.” (Mullen, Abstract, ¶0006.) Mullen qualifies as prior art under §
`
`102(b) because it was published May 18, 2006, more than one year before the earliest
`
` 
`
`
`
`-16-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`date to which the ’051 Patent claims priority.
`
`Figure 1 of Mullen illustrates a handheld location-based game system and
`
`accompanying playmat:
`
`
`(Mullen, Fig. 1, ¶¶0010, 0025, 0045-0052.) Mullen renders “[v]irtual objects”
`
`within the “AR game system.” (Id., ¶¶0022, 0072, 0075-0076.) Mullen also
`
`discloses “computer-controlled character positioning and movement based on
`
`detected landscape objects[.]” (Id., ¶0030, Fig. 6.)
`
`Mullen further discloses a “virtual currency system such that any user can
`
`exchange real money for the virtual currency system, with the game manufacturer
`
`taking a percentage cut, and the virtual currency can be used to buy/sell items.”
`
` 
`
`
`
`-17-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`(Mullen, ¶0056.) A user can then be “provided with the option to then convert the
`
`virtual currency back to real currency (e.g., the U.S. dollar) with the game
`
`manufacturer again taking a percentage of the conversion.” (Id.)
`
`B. Ground 1: Single-Reference Obviousness of Claims 1, 5-9, 18, 22-
`23, 27-29, 34, 36, 38, and 43 Over Yu
`1.
`Claim 1
`(a)
`“An augmented reality
`comprising:” (Preamble)
`To the extent this preamble is limiting, it is satisfied by Yu. Yu discloses an
`
`(AR) platform system
`
`AR platform system where “[a] user can create ‘virtual graffiti’ that will be left for
`
`a particular device to view as part of an augmented-reality scene.” (Yu, ¶0014; see
`
`also id., Title (“Mobile Virtual and Augmented Reality System”).)5 Yu’s “virtual
`
`graffiti” is an “AR object” because it is a virtual object that is “present[ed] . . .
`
`along[]side real-world elements” (’051, 1:32-33), such as a “real-world door” to a
`
`restaurant. (Yu, ¶0024; Zyda, ¶100.) Figure 2 depicts an AR scene where the AR
`
`object (virtual graffiti) is superimposed on a real-world door:
`
`
`5 Unless noted otherwise, all emphasis and highlighting have been added by
`
`Petitioner.
`
` 
`
`
`
`-18-
`
` 
`
`

`

`Petition for Inter Partes Review of
`U.S. Patent No. 10,403,051 B2
`
`
`
`
`(Yu, Fig. 2; see also id., Fig. 3, ¶0027.) Yu further discloses that the virtual graffiti
`
`can comprise “images, audio and video clips, etc.” (Yu, ¶0025; cf. ’051, 19:66-
`
`20:21.) Like the ’051 Patent (’051, 3:58-4:1), Yu describes “an apparatus for
`
`receiving and displaying virtual graffiti [AR objects] as part of an augmented-reality
`
`scene.” (Yu, ¶0020; Zyda, ¶106.)
`
`(b