`
`Michael Zachary
`Direct 650.384.4683
`mzachary@kenyon.com
`
`Andrews Kurth Kenyon LLP
`1801 Page Mill Road
`Suite 210
`Palo Alto, CA 94304-1216
`650.384.4700
`Fax 650.384.4701
`
`May 10, 2017
`
`Matthew Bernstein, Esq.
`Perkins Coie, LLP
`11988 El Camino Real, Suite 350
`San Diego, CA 92130-2594
`
`
`Dear Matt:
`
`Further to my letters dated March 23, 2017, in which I informed Microsoft that three new
`Bradium patents would soon be issued by the U.S. Patent Office, I write to inform Microsoft that
`the patents have now been issued. In particular, U.S. Patent 9,635,136 issued on April 25, 2017,
`U.S. Patent 9,641,644 issued on May 2, 2017, and U.S. Patent 9,641,645 issued on May 2, 2017.
`Copies of all three patents are attached. As further stated below, Bradium believes that
`Microsoft is infringing all three patents. I also write to address some of the comments in your
`letter dated April 20, 2017, which you wrote in response to my March 23, 2017, letters.
`
`Notice of Infringement of U.S. Patent Nos. 9,635,136, 9,641,644, and 9,641,645
`
`Bradium believes that Microsoft has been infringing, and continues to infringe, one or
`more claims of U.S. Patent Nos. 9,635,136 (the “’136 patent”), 9,641,644 (the “’644 patent”),
`and 9,641,645 (the “’645 patent”). Microsoft has been on notice regarding these patents at least
`since the date of my March 23, 2017, letters.
`
`Based on its current investigation, Bradium identifies claim 10 of the ’136 patent, claim
`23 of the ’644 patent, and claim 1 of the ’645 patent as examples of infringed claims. Bradium
`further identifies as infringing products at least those products noted in my March 23, 2017,
`letters.
`
`Bradium intends to seek the Court’s authorization to add these newly-issued patents to
`the litigation entitled Bradium Technologies LLC v. Microsoft Corporation (D. Del. 15-0031-
`RGA), as soon as the Court permits.
`
`Contentions in Your Letter Dated April 20, 2017
`
`
`
`1.
`
`Microsoft’s Invalidity Contentions
`
`Your letter attacks the US Patent Office for allowing Application No. 14/970,526, now
`issued as the ’644 patent, which is surprising given that the Examiner was provided with and
`
`
`
`ANDREWS KURTH KENYON LLP
`Austin Beijing Dallas Dubai Houston London New York Research Triangle Park Silicon Valley The Woodlands Washington, DC
`
`Microsoft Corp. Exhibit 1026
`
`

`

`Matthew Bernstein, Esq.
`May 10, 2017
`Page 2
`
`
`carefully considered Microsoft’s arguments and asserted prior art. In addition, your letter
`mischaracterizes the Examiner’s statement of the basis for allowing the patent. The Examiner
`specifically stated in allowing Claims 1-65 that “[w]hen considering the claims as a whole,
`particularly how these elements interact with the claimed update parcels, the claims are
`found to be novel and non-obvious over the prior art.” Notice at Page 2, Paragraph 4 (emphasis
`added). Microsoft’s arguments regarding the alleged errors of the Patent Office are unsupported
`by the evidence and therefore frivolous.
`
`Your letter also persists in attacking the validity of the ’794 patent, which is one the
`patents already in suit, not one of the new patents. Microsoft’s continued attack on the patent is
`also surprising, given that Microsoft challenged the patent in the Patent Office, received a fair
`hearing, and lost. Further, we note that Microsoft abandoned its appeal to the Federal Circuit on
`this patent, an admission of defeat on the merits.
`
`
`
`2.
`
`Microsoft’s Contentions Regarding Its Infringement
`
`Your letter does not deny that Microsoft infringes the new patents, and instead attacks
`Bradium for not providing detailed infringement contentions. Your letter misconstrues the
`purpose of Image Processing’s March 23, 2017, letters, which was to provide Microsoft with
`notice of the expected issuance of the patents and of Bradium’s allegations of infringement. As
`your letter implicitly acknowledges, Microsoft is well aware that it infringes the new patents as
`well as the previously-issued patents that are asserted in the lawsuit. As you also well know,
`Bradium has already detailed the evidence of such infringement in its 101 pages of infringement
`contentions served on Microsoft in the pending lawsuit. Those contentions were provided after
`Microsoft was required under Court order to make the code for its software available to Bradium
`for review. Again, Microsoft’s position is frivolous.
`
`Microsoft’s further contention that it does not infringe the ’794 patent, which is not one
`of the new patents, is also without basis. Microsoft’s infringement is clearly demonstrated in the
`detailed infringement contentions provided in the litigation.
`
`Microsoft’s Contentions Regarding Unenforceability
`
`Your letter continues in the same vein to make baseless and scurrilous accusations that
`the patents are “unenforceable.” As you know, the individual who made these accusations
`refused to show up for his deposition where his statements could be tested under oath.
`Microsoft’s continued reliance on these baseless accusations is shameful and indicative of its
`weak position.
`
`***
`
`As requested in the March 23, 2017, letters, Bradium demands that Microsoft cease and
`desist from its infringement of the newly issued patents, as well as the earlier patents that are
`already the subject of the lawsuit.
`
`Microsoft Corp. Exhibit 1026
`
`

`

`Matthew Bernstein, Esq.
`May 10, 2017
`Page 3
`
`
`
`
`Enclosures
`
`Very truly yours,
`
`Michael Zachary
`
`Microsoft Corp. Exhibit 1026
`
`

`

`a2) United States Patent
`US 9,635,136 B2
`(0) Patent No.:
`*Apr. 25, 2017
`(45) Date of Patent:
`Levanon et al.
`
`US009635136B2
`
`(54)
`
`OPTIMIZED IMAGE DELIVERY OVER
`LIMITED BANDWIDTH COMMUNICATION
`CHANNELS
`
`(71)
`
`Applicant: Bradium Technologies LLC, Suffern,
`NY (US)
`
`(72)
`
`Inventors:
`
`Isaac Levanon, Raanana (IL); Yonatan
`Lavi, Raanana (IL)
`
`(73)
`
`Assignee: BRADIUM TECHNOLOGIES LLC,
`Suffern, NY (US)
`
`(*)
`
`Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`This patent is subject to a terminal dis-
`claimer.
`
`(21)
`
`Appl. No.: 15/343,052
`
`(22)
`
`Filed:
`
`Nov. 3, 2016
`
`Prior Publication Data
`
`(58) Field of Classification Search
`CPC .... HO4N 1/40068; HO4N 1/4172; HO4N 1/64;
`HO4N 21/234345; HO4N 21/234363;
`(Continued)
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`4,682,869 A *
`
`4,972,319 A *
`
`7/1987 Itoh wee GO6T 9/004
`358/426.12
`11/1990 Delorme ............ GO9B 29/007
`340/990
`
`(Continued)
`
`OTHER PUBLICATIONS
`
`Declaration of Yonatan Lavi, Exhibit 1017 with exhibits A-E filed
`in PTAB Case No. IPR2016-00448, all pages.
`
`Primary Examiner — David Lazaro
`(74) Attorney, Agent, or Firm — Anatoly S. Weiser, Esq.;
`Techlaw LLP.
`
`(65)
`
`(63)
`
`(51)
`
`(52)
`
`US 2017/0054830 Al
`
`Feb. 23, 2017
`
`(57)
`
`ABSTRACT
`
`Related U.S. Application Data
`
`Continuation of application No. 15/281,037, filed on
`Sep. 29, 2016, which is a continuation of application
`(Continued)
`
`Int. Cl
`GO6F 15/16
`FOAL 29/08
`
`(2006.01)
`(2006.01)
`.
`(Continued)
`
`US. Cl.
`CPC vresecessees HO4L 67/327 (2013.01); GO6F 3/14
`(2013.01), GO6T 3/4092 [7003201301y
`
`Large-scale images are retrieved over network communica-
`tions channels for display on a client device by selecting an
`update image parcel relative to an operator controlled image
`viewpoint to display via the client device. A request
`is
`prepared for the update image parcel and associated with a
`request queue for subsequent issuance over a communica-
`tions channel. The update image parcel is received from the
`communications channel and displayed as a discrete portion
`.
`:
`:
`:
`of the predetermined image. The update image parcel opti-
`mally hasa fixed pixel array size, is received in a single and
`or plurality ofnetwork data packets, and werethe fixed pixel
`array may be constrained to a resolution less than or equal
`to the resolution of the client device display.
`
`(Continued)
`
`27 Claims, 5 Drawing Sheets
`
`
`LOCAL PARCEL
`|
`
`Bara Store
`
`
`
`
`‘ARCEL DATA
`a
`i
`TA
`Par
`METWORK orem P
`U
`eecccemeh
`PARCEL Dat
`commend
`cet Da
`S DISPLAY
` | RECEIVE/REQUEST |
` |
`PROCESSING
`Renpermc
`|”
`
`
`
`
`
`[Paoey |
`
`
`| Veewne Frusaa
`1] Reouesy
`fet
`
`|
`CONTROL
`
`_ Queer |
`
`NAVIGATION COMMANDS
`
`48
`
`Microsoft Corp. Exhibit 1026
`
`Microsoft Corp. Exhibit 1026
`
`

`

`US 9,635,136 B2
`
`Page 2
`
`Related U.S. Application Data
`
`No. 14/970,526, filed on Dec. 15, 2015, which is a
`continuation of application No. 14/547,148, filed on
`Nov. 19, 2014, now Pat. No. 9,253,239, which is a
`continuation of application No. 13/027,929, filed on
`Feb. 15, 2011, now Pat. No. 8,924,506, which is a
`continuation-in-part of application No. 12/619,643,
`filed on Nov. 16, 2009, now Pat. No. 7,908,343,
`which is a continuation of application No. 10/035,
`987, filed on Dec. 24, 2001, now Pat. No. 7,644,131.
`Provisional application No. 60/258,465, filed on Dec.
`27, 2000, provisional application No. 60/258,466,
`filed on Dec. 27, 2000, provisional application No.
`60/258,467,
`filed on Dec. 27, 2000, provisional
`application No. 60/258,468, filed on Dec. 27, 2000,
`provisional application No. 60/258,488, filed on Dec.
`27, 2000, provisional application No. 60/258,489,
`filed on Dec. 27, 2000.
`
`(60)
`
`(51)
`
`Int. Cl.
`
`G06T 3/40
`G06F 3/14
`G09G 5/00
`GO6T 11/60
`GO6T 15/04
`GO6F 17/24
`GO6T 19/00
`US. Cl.
`
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2011.01)
`(2006.01)
`(2011.01)
`
`CPC wee GO6F 17/241 (2013.01); GO6T 11/60
`(2013.01); GO6T 15/04 (2013.01); GO6T
`19/003 (2013.01); GO6T 2207/10032
`(2013.01); GO9G 2350/00 (2013.01); GO9G
`2370/02 (2013.01); GO9G 2370/16 (2013.01)
`Field of Classification Search
`CPC ves. HO4N 21/25825; GO6F 17/30241; GO6F
`3/04815; GO6T 3/4092; GO6T 19/003
`USPC veces 709/202, 203, 217, 218, 230, 231;
`382/305, 232; 345/428, 581, 625
`See application file for complete search history.
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,559,936 A *
`
`5,613,051 A *
`
`9/1996 Poulter
`
`........... GO6F 17/30017
`345/428
`3/1997 Todice wee GO6F 3/14
`345/428
`
`(52)
`
`(58)
`
`(56)
`
`5,929,860
`
`A *
`
`7/1999
`
`5,995,903
`
`A *
`
`11/1999
`
`6,167,442
`
`A *
`
`12/2000
`
`6,212,301
`
`BI*
`
`4/2001
`
`6,246,797
`
`BI*
`
`6/2001
`
`6,285,317
`
`BI*
`
`9/2001
`
`6,314,452
`
`BI*
`
`11/2001
`
`6,326,965
`
`BI*
`
`12/2001
`
`6,345,279
`6,346,938
`
`BI*
`BI*
`
`2/2002
`2/2002
`
`6,397,259
`
`BI*
`
`5/2002
`
`6,449,639
`
`BI*
`
`9/2002
`
`6,496,189
`
`BI*
`
`12/2002
`
`6,525,732
`
`BI*
`
`2/2003
`
`6,608,628
`
`BI*
`
`8/2003
`
`6,608,933
`
`BI*
`
`8/2003
`
`6,625,309
`
`BI*
`
`9/2003
`
`6,704,024
`
`B2*
`
`3/2004
`
`6,704,791
`
`BI*
`
`3/2004
`
`6,711,297
`
`BI*
`
`3/2004
`
`6,754,365
`
`BI*
`
`6/2004
`
`6,801,665
`
`BI*
`
`10/2004
`
`6,882,755
`
`B2*
`
`4/2005
`
`6,898,311
`
`B2*
`
`5/2005
`
`6,970,604
`
`BI*
`
`11/2005
`
`7,644,131
`
`B2*
`
`1/2010
`
`8,924,506
`
`B2*
`
`12/2014
`
`Hoppe ose GO6T 9/001
`345/419
`Smith wo. GOI1C 21/00
`340/995.26
`Sutherland .......... GO6F 17/3028
`709/217
`Warner oo. GO06T 9/00
`382/232
`Castor vce HO4AN19/63
`375/E7.035
`ONg wees GOIC 21/3647
`340/995.2
`Dekel oc HO4N 19/647
`375/E7.045
`Castelli... GO6F 17/30241
`345/420
`eeccceeeereees GO6F 17/30905
`Li
`Chan wo. GO6F 3/04815
`345/419
`Lincke wc GO6F 17/3089
`TOT/E17.116
`Blumberg ......... GO6F 17/30905
`TO7/E17.118
`YaTON eeccceereeees GO6T 15/40
`345/419
`Gadh ween GO6T 15/20
`345/428
`ROSS veecccereseteteeees GO6T 17/20
`345/619
`Dowell 0... cece GO6T 9/007
`382/232
`ececcceeseteeteteenees GO6T 15/40
`345/418
`Robotham ou... GO6F 3/14
`345/581
`Hatvis oo. GO6T 15/20
`709/231
`Chang we. GO6T 1/00
`375/E7.065
`WEN ccc GO6T 1/0078
`382/100
`Atsumi oe HO4N 19/70
`375/E7.056
`Silverstein ......... HO4N 21/2662
`375/E7.011
`Whitehead 0... HO4N 1/64
`375/E7.184
`Chai wee HO4N 19/176
`375/E7.056
`Levanon wo. GO6F 3/14
`345/625
`Levanon oo... GO6F 3/1454
`345/625
`
`Li
`
`* cited by examiner
`
`Microsoft Corp. Exhibit 1026
`
`Microsoft Corp. Exhibit 1026
`
`

`

`U.S. Patent
`
`Apr.25, 2017
`
`Sheet 1 of 5
`
`US 9,635,136 B2
`
`
` | Bixary/XML 7
`IMAGE DATA
`
`
`
`
`
`SOURCE
`
`4g
`
`PRE-PROCESSED
`PARCELIMAGE DATA
`
`FIG. 2
`
`Microsoft Corp. Exhibit 1026
`
`
`
`Source
`Overlay Data
`
`34
`
`
`
`—p DAA
`CONVERSION |
`
`
`
`| NenworK
`|
`SERVER
`|
`8
`| Dara Store
`
`Microsoft Corp. Exhibit 1026
`
`

`

`U.S. Patent
`
`Apr. 25, 2017
`
`Sheet 2 of 5
`
`US 9,635,136 B2
`
`|
`Dara Store|P.
`
` A
`
`
`| Views Feustum [-
`CONTROL
`
`42a
`
`|
`parce Data
`|
`iP
`Nerwory Bence. Spe
`| RECEIVE/REQUEST |
`
`
`|) Pesoniy |
`1) Request i
`
`
`| Quiue |
`
`527
`
` | Loca Parcer
`ee ; AS
`
`
`—_Bd
`
`
`PARCEL Dat ond
`|
`
`PROCESSING
`
`48
`
`Parcer
`
`CEL DA po DISPLAY
`Dara
`|
`RENDERING
`
`NAVIGATION COMMANDS
`
`FIG. 3
`
`
`
`| Grapnics
`MICRO
`
`
`| Menore
`PROCESSOR
`
`
`
`|
`|
`
`
`
`60
`
`Microsoft Corp. Exhibit 1026
`
`Microsoft Corp. Exhibit 1026
`
`

`

`U.S. Patent
`
`Apr. 25, 2017
`
`Sheet 3 of 5
`
`US 9,635,136 B2
`
`82, |
`
`
`
`DETERMINE
`EXAMINE
` NAVIGATION
`
`
`VIEWABLE PARCELS
`EVENTS
`|
`For Upoare
`
`
`
`|
` “| Reouest Paacet
`
`
`(Issue Progmy
`|
`DOWNLOADS
`“\ Regus
`Of
`
`
`
`
`
` Snore PARCEL TO Jeo
`
`
`
`
`“(Receive Parc)
`
`
`| QuaoTree Nope |
`
`
`92
`
`94
`
`
`| SECT HiGHEST | a —
`|
`| PRIORITY REQUEST |
`ne
`| Warec
`|
`
`
`Request,
`
`Microsoft Corp. Exhibit 1026
`
`Microsoft Corp. Exhibit 1026
`
`

`

`U.S. Patent
`
`Apr. 25, 2017
`
`Sheet 4 of 5
`
`US 9,635,136 B2
`
`| Aniust Viewne |
`NAVIGATION__pf
` FRUSTUM
`EVENTS
`;
`LOCATION AND
`| ~ 429
`ORIENTATION
`
`
`
`
`
`
`PRIORITY
`SELECTION OF
`Nev? Papen
`
`RENDER IMAGE
`Parte. Td
`DispaY MEMORY
`
`
`
`
`
`
`__ ff Gvertay DATA
`Cooens Marcu P
`
`™ 404
`
`“426
`
`428
`
`| DispLay MEMORY |
`
`| Renper Overy |
`| Dato
`|
`
`
`
`
`Spuy Map into | 46
`
`: ASSIGH EXISTING |
`
`
`
`
`| PARCELS OF MAX |
`
`;
`tvadb
`|
`Trace THROUGH on "Yececosasmsssoamscossamssseca
`
`Quaoiree
`|
`PARCELS
`
` |
`
`FIG. 7
`
`mo 120
`
`START
`
`| Coear Prony |
`|
`Queue
`=F
`
`142
`
`
`|
`DEvERMINE
`|
`—| OPTIRAL DETAR : ps
`|
`teval
`|
`
`“144
`
`Povysons P
`
`|
`
`FIG. 8
`
`140
`
`148
`
`|
`
`PARCELS OF
`
`twat
`
`
`
`150
`
`Microsoft Corp. Exhibit 1026
`
`Microsoft Corp. Exhibit 1026
`
`

`

`U.S. Patent
`
`Apr. 25, 2017
`
`Sheet 5 of 5
`
`US 9,635,136 B2
`
`For EACH
`Pouycon P
`
`[Sup P OUTSIDE OF |
`
`| Views Frustun |
`
`DEFERMINE
`Gprimat Devan Lever L
`
`| DETERMINE CAMERA
`|
`Posmon
`
`|
`
`
` CupPTo
`|
`| VIEWING Rustam Pe.
`DETERMINE
`764
`Bounns
`|
`|
`NEAREST=BN
`
`PowconP of 184
`
`
`
`
`~~ 182
`
`
`
`
`
`
`
`
`
`
`
`
`
`
` SPLIT
`POLYGON P —_
`|
`
`Lp192
`\ 180
`
`|
`DETERMINE
`|
`| NEAREST POINT A J
`| On PotvconP P->
`a43508505000050050053
`
`"486
`
`oF
`(ompume
`|
`| Oprmat Dera §
`| Leb ForA [429
`
`Textune Mar7186
`
`Parcet To P
`Leven DB <b
`
` ir PARCEL
`
`
`{ 172
`: enceern'
` AbD PARCEL
` |
`Downioap
`|
`|
` Requesy
`|
`
`|
`oUF Parc
`;
`pla D>= bf
`
`/S
`
`460
`
`Creare CHILD
`Nove €
`
`176 ~
`
`
`DETERMINE
`
`
`
`
`| Request Priority |
`
`
`
`
` 78 — J ASSIGH REQUEST
`
`
`PRIORY
`
`FIG. 10
`
`Microsoft Corp. Exhibit 1026
`
`Microsoft Corp. Exhibit 1026
`
`

`

`US 9,635,136 B2
`
`1
`OPTIMIZED IMAGE DELIVERY OVER
`LIMITED BANDWIDTH COMMUNICATION
`CHANNELS
`
`PRIORITY CLAIMS/RELATED APPLICATIONS
`
`This application is a continuation of and claimspriority to
`USS. patent application Ser. No. 14/970,526, filed Dec. 15,
`2015, entitled OPTIMIZED IMAGE DELIVERY OVER
`LIMITED BANDWIDTH COMMUNICATION CHAN-
`
`NELS; this application is also a continuation of and claims
`priority to U.S. patent application Ser. No. 15/281,037, filed
`Sep. 29, 2016, entitled OPTIMIZED IMAGE DELIVERY
`OVER LIMITED BANDWIDTH COMMUNICATION
`
`CHANNELS; each of the U.S. patent application Ser. No.
`14/970,526 and Ser. No. 15/281,037 is a continuation of and
`claimspriority to U.S. patent application Ser. No. 14/547,
`148, filed Nov. 19, 2014, entitled OPTIMIZED IMAGE
`DELIVERY OVER LIMITED BANDWIDTH COMMUNI-
`CATION CHANNELS, now U.S. Pat. No. 9,253,239; which
`is a continuation of and claims priority to U.S. patent
`application Ser. No. 13/027,929, filed Feb. 15, 2011, entitled
`OPTIMIZED IMAGE DELIVERY OVER LIMITED
`BANDWIDTH COMMUNICATION CHANNELS, now
`USS. Pat. No. 8,924,506; which is a continuation-in-part of
`and claims priority to U.S. patent application Ser. No.
`12/619,643, filed on Nov. 16, 2009, entitled OPTIMIZED
`IMAGE DELIVERY OVER LIMITED BANDWIDTH
`COMMUNICATION CHANNELS, now U.S. Pat. No.
`7,908,343; which is a continuation of and claimspriority to
`USS. patent application Ser. No. 10/035,987, filed on Dec.
`24, 2001, entitled OPTIMIZED IMAGE DELIVERY OVER
`LIMITED BANDWIDTH COMMUNICATION CHAN-
`NELS, now U.S. Pat. No. 7,644,131; which claims the
`benefit under 35 U.S.C. §119(e) of U.S. Provisional Appli-
`cation Nos. 60/258,488, 60/258,489, 60/258,465, 60/258,
`468, 60/258,466, and 60/258,467, all filed Dec. 27, 2000.
`The disclosures of all the foregoing patent documents are
`incorporated herein by reference as if fully set forth herein,
`including Figures, Claims, and Tables. The present applica-
`tion is also related to application Ser. No. 10/035,981,
`entitled SYSTEM AND METHODS FOR NETWORK
`IMAGE DELIVERY WITH DYNAMIC VIEWINGFRUS-
`TUM OPTIMIZED FOR LIMITED BANDWIDTH COM-
`MUNICATION CHANNELS, Levanonetal., filed on Dec.
`24, 2001, now U.S. Pat. No. 7,139,794, issued on Nov. 21,
`2006, which is assigned to the Assignee of the present
`application.
`
`FIELD
`
`The disclosure is related to network based, image distri-
`bution systems and, in particular, to a system and methods
`for efficiently selecting and distributing image parcels
`through a narrowbandor otherwise limited bandwidth com-
`munications channel to support presentation of high-reso-
`lution images subject to dynamic viewing frustums.
`
`BACKGROUND
`
`The Internet and or other network systems may provide a
`unique opportunity to transmit for example complex images,
`typically large scale bit-maps, particularly those approach-
`ing photo-realistic levels, over large area and or distances. In
`common application, the images may be geographic, topo-
`graphic, and or other highly detailed maps. The data storage
`requirements and often proprietary nature of such images
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`could be such that conventional interests may be to transfer
`the images on an as-needed basis.
`In conventional fixed-site applications, the image data
`maybetransferred overa relatively high-bandwidth network
`to client computer systems that in turn, may render the
`image. Client systems may typically implement a local
`image navigation system to provide zoom and or pan
`functions based on user interaction. As well recognized
`problem with such conventional systems could be that full
`resolution image presentation may be subject to the inherent
`transfer latency of the network. Different conventional sys-
`tems have been proposed to reduce the latency affect by
`transmitting the image in highly compressed formats that
`support progressive resolution build-up of the image within
`the current client field of view. Using a transform com-
`pressed image transfer function increases the field of the
`image that can be transferred over a fixed bandwidth net-
`work in unit time. Progressive image resolution transmis-
`sion, typically using a differential resolution method, per-
`mits an approximate image to be quickly presented with
`image details being continuously added overtime.
`Tzou, in U.S. Pat. No. 4,698,689, describes a two-dimen-
`sional data transform system that supports transmission of
`differential coefficients to represent an image. Subsequent
`transmitted coefficient sets are progressively accumulated
`with prior transmitted sets to provide a succeedingly refined
`image. The inverse-transform function performed by the
`client computer is, however, highly compute intensive. In
`order to simplify the transform implementation and further
`reduce the latency of presenting any portion of an approxi-
`mate image, images are subdividedinto a regular array. This
`enables the inverse-transform function on the client, which
`is time-critical, to deal with substantially smaller coefficient
`data sets. The array size in Tzou is fixed, which leads to
`progressively larger coefficient data sets as the detail level of
`the image increases. Consequently, there is an inherently
`increasing latency in resolving finer levels of detail.
`An image visualization system proposed by Yapetal.,
`USS. Pat. No. 6,182,114, overcomes some of the foregoing
`problems. The Yap et al. system also employs a progressive
`encoding transform to compress the image transfer stream.
`The transform also operates on a subdivided image, but the
`division is indexed to the encoding level of the transform.
`The encoded transform coefficient data sets are, therefore, of
`constant size, which supports a modest improvementin the
`algorithmic performance of the inverse transform operation
`required on the client.
`Yap et al. adds utilization of client image panningor other
`image pointing input information to support a foveation-
`based operator to influence the retrieval order of the subdi-
`vided image blocks. This two-dimensional navigation infor-
`mation is used to identify a foveal region that is presumed
`to be the gaze point of a client system user. The foveation
`operator defines the corresponding image block as the center
`point of an ordered retrieval of coefficient sets representing
`a variable resolution image. The gaze point image block
`represents the area of highest image resolution, with reso-
`lution reduction as a function of distance from the gaze point
`determined by the foveation operator. This technique thus
`progressively builds image resolution at the gaze point and
`succeedingly outward based on a relatively compute inten-
`sive function. Shifts in the gaze point can be respondedto
`with relative speed by preferentially retrieving coefficient
`sets at and near the new foveal region.
`Significant problems remain in permitting the convenient
`and effective use of complex images by manydifferent types
`of client systems, even with the improvements provided by
`
`Microsoft Corp. Exhibit 1026
`
`Microsoft Corp. Exhibit 1026
`
`

`

`US 9,635,136 B2
`
`3
`the various conventional systems. In particular, the imple-
`mentation of conventional image visualization systems is
`generally unworkable for smaller, often dedicated or embed-
`ded, clients where use of image visualization would clearly
`be beneficial. Conventional approacheseffectively presume
`that client systems have an excess of computing perfor-
`mance, memory and storage. Small clients, however, typi-
`cally have restricted performance processors with possibly
`no dedicated floating-point support, little general purpose
`memory, and extremely limited persistent storage capabili-
`ties, particularly relative to common image sizes. A mobile
`computing device such as mobile phone, smart phone, tablet
`and or personal digital assistant (PDA) is a characteristic
`small client. Embedded, low-cost kiosk, automobile navi-
`gation systems andor Internet enabled I connected TV are
`other typical examples. Such systems are not
`readily
`capable, if at all, of performing complex, compute-intensive
`Fourier or wavelet transforms, particularly within a highly
`restricted memory address space.
`As a consequence of the presumption that the client is a
`substantial computing system, conventional image visual-
`ization systems also presumethat the client is supported by
`a complete operating system.
`Indeed, many expect and
`require an extensive set of graphics abstraction layers to be
`provided by the client system to support the presentation of
`the delivered image data. In general, these abstraction layers
`are conventionally considered required to handle the map-
`ping of the image data resolution to the display resolution
`capabilities of the client system. Thatis, resolution resolved
`image data provided to the client is unconstrained by any
`limitation in the client system to actually display the corre-
`sponding image. Consequently, substantial processor per-
`formance and memory can be conventionally devoted to
`handling image data that is not or cannot be displayed.
`Another problem is that small clients are generally con-
`strained to generally to very limited network bandwidths,
`particularly when operating under wireless conditions. Such
`limited bandwidth conditions may exist due to either the
`direct technological constraints dictated by the use of a low
`bandwidth data channel or indirect constraints imposed on
`relatively high-bandwidth channels by high concurrent user
`loads. Cellular connected PDAs and webphones
`are
`examples of small clients that are frequently constrained by
`limited bandwidth conditions. The conventionally realizable
`maximum network transmission bandwidth for such small
`devices may range from below one kilobit per second to
`several tens of kilobits per second. While Yap etal. states
`that the described system can work over low bandwidth
`lines, little more than utilizing wavelet-based data compres-
`sion is advanced as permitting effective operation at low
`communications bandwidths. While reducing the amountof
`data that must be carried from the server to the client is
`significant, Yap et al. simply relies on the data packet
`transfer protocols to provide for an efficient transfer of the
`compressed image data. Reliable transport protocols, how-
`ever, merely mask packet losses and the resultant, some-
`times extended recovery latencies. When such covered
`errors occur, however, the aggregate bandwidth of the con-
`nection is reduced andthe client system canstall waiting for
`further image data to process.
`Consequently, there remains a need for an image visual-
`ization system that can support small client systems, place
`few requirements on the supporting client hardware and
`software resources, and efficiently utilize low to very low
`bandwidth network connections.
`
`SUMMARY
`
`Thus, a general purpose of the present invention is to
`provide an efficient system and methods of optimally pre-
`
`4
`senting image data on client systems with potentially limited
`processing performance, resources, and communications
`bandwidth.
`
`This is achieved in the present invention by providing for
`the retrieval of large-scale images over network communi-
`cations channels for display on a client device by selecting
`an update image parcel relative to an operator controlled
`image viewpoint to display via the client device. A request
`is prepared for the update imageparcel and associated with
`a request queue for subsequent issuance over a communi-
`cations channel. The update image parcel is received from
`the communications channel and displayed as a discrete
`portion of the predetermined image. The update image
`parcel optimally has a fixed pixel array size, is received in
`a single and or plurality of network data packets, and were
`the fixed pixel array may be constrainedto a resolution less
`than or equal to the resolution of the client device display.
`An advantage of the present invention is that both image
`parcel data requests and the rendering of image data are
`optimized to address the display based on the display
`resolution of the client system.
`Another advantage of the present invention is that the
`prioritization of image parcel requests is based on an adapt-
`able parameter that minimizes the computational complexity
`of determining request prioritization and, in turn, the pro-
`gressive improvementin display resolution within the field
`of view presented on a client display.
`A further advantage of the present invention is that the
`client software system requires relatively minimal client
`processing power and storage capacity. Compute intensive
`numerical calculations are minimally required and image
`parcel data is compactly stored in efficient data structures.
`The client software system is very small and easily down-
`loaded to conventional computer systems or embedded in
`conventional dedicated function devices, including portable
`devices, such as PDAs, tablets and webphones.
`Still another advantage of the present invention is that
`image parcel data requests and presentation can be readily
`optimized to use low to very low bandwidth network con-
`nections. The software system of the present
`invention
`provides for re-prioritization of image parcel data requests
`and presentation in circumstances where the rate of point-
`of-view navigation exceeds the data requestrate.
`Yet another advantage of the present invention is that
`image parcel data rendering is performed without requiring
`any complex underlying hardware or software display sub-
`system. The client software system of the present invention
`includes a bit-map rendering engine that draws directly to
`the video memory of the display,
`thus placing minimal
`requirements on any underlying embeddedordisk operating
`system and display drivers. Complex graphics and anima-
`tion abstraction layers are not required.
`Still another advantage of the present invention is that
`image parcel block compression is used to obtain fixed size
`transmission data blocks. Image parcel data is recoverable
`from transmission data using a relatively simple client
`decompression algorithm. Using fixed size transmission data
`blocks enables image data parcels to be delivered to the
`client in bounded time frames.
`
`40
`
`65
`
`A yet further advantage of the present invention is that
`multiple data forms can be transferred to the client software
`system for concurrent display. Array overlay data, correlated
`positionally to the image parcel data and generally insensi-
`tive to image parcel resolution, can be initially or progres-
`
`Microsoft Corp. Exhibit 1026
`
`Microsoft Corp. Exhibit 1026
`
`

`

`US 9,635,136 B2
`
`5
`sively provided to the client for parsing and parallel pre-
`sentation on a client display image view.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`These and other advantages and features of the present
`invention will becomebetter understood upon consideration
`of the following detailed description of the invention when
`considered in connection with the accompanying drawings,
`in which like reference numerals designate like parts
`throughout the figures thereof, and wherein:
`FIG. 1 depicts a preferred system environment within
`which various embodiments of the present invention can be
`utilized;
`FIG. 2 is a block diagram illustrating the preparation of
`imageparcel and overlay data set that are to be stored by and
`served from a network server system in accordance with a
`preferred embodiment of the present invention;
`FIG. 3 is a block diagram of a client system image
`presentation system constructed in accordance with a pre-
`ferred embodiment of the present invention;
`FIG. 4 provides a data block diagram illustrating an
`optimized client image block processing path constructed in
`accordance with a preferred embodiment of the present
`invention;
`FIG. 5 is a process flow diagram showing a main pro-
`cessing thread implemented in a preferred embodiment of
`the present invention;
`FIG. 6 provides a process flow diagram showing a net-
`work request thread implementedin a preferred embodiment
`of the present invention;
`FIG.7 provides a process flow diagram showinga display
`image rendering thread implementedin a preferred embodi-
`ment of the present invention;
`FIG. 8 provides a process flow diagram showing the
`parcel map processing performedpreliminary to the render-
`ing of image data parcels in accordance with a preferred
`embodimentof the present invention;
`FIG. 9 provides a process flow diagram detailing the
`rendering and progressive prioritization of image parcel data
`download requests in accordance with a preferred embodi-
`ment of the present invention; and
`FIG. 10 provides a process flow diagram detailing the
`determination of an optimal detail level for image parcel
`presentation for a current viewing frustum in accordance
`with a preferred embodiment of the present invention.
`
`DETAILED DESCRIPTION OF ONE OR MORE
`EMBODIMENTS
`
`The preferred operational environment 10 of the present
`invention is generally shown in FIG. 1. A network server
`system 12, operating as a data store and server of imagedata,
`is responsive to requests received through a communications
`network, such as the Internet 14 generally and varioustiers
`of internet service providers (ISPs) including a wireless
`connectivity provider 16. Client systems, including conven-
`tional workstations and personal computers 18 and smaller,
`typically dedicated function devices often linked through
`wireless network connections, such as PDAs, webphones
`20, and automobile navigation systems,
`source image
`requests to the network server 12, provide a client display
`and enable image navigational input by a user of the client
`system. Alternately, a dedicated function client system 20
`may be connected through a separate or plug-in local
`networkserver 22, preferably implementing a small, embed-
`ded Web server, to a fixed or removable storage local image
`
`20
`
`25
`
`40
`
`45
`
`50
`
`6
`repository 24. Characteristically, the client system 18, 20
`displays are operated at some fixed resolution generally
`dependent on the underlying display hardware ofthe client
`systems 18, 20.
`The image navigation capability supported by the present
`invention encompasses a viewing frustum placed within a
`three-dimensional space over the