`
`Michael Zachary
`Direct 650.384.4683
`mzachary@kenyon.com
`
`Andrews Kurth Kenyon LLP
`1801 Page Mill Road
`Suite 210
`Palo Alto, CA 94304-1216
`650.384.4700
`Fax 650.384.4701
`
`May 10, 2017
`
`Matthew Bernstein, Esq.
`Perkins Coie, LLP
`11988 El Camino Real, Suite 350
`San Diego, CA 92130-2594
`
`
`Dear Matt:
`
`Further to my letters dated March 23, 2017, in which I informed Microsoft that three new
`Bradium patents would soon be issued by the U.S. Patent Office, I write to inform Microsoft that
`the patents have now been issued. In particular, U.S. Patent 9,635,136 issued on April 25, 2017,
`U.S. Patent 9,641,644 issued on May 2, 2017, and U.S. Patent 9,641,645 issued on May 2, 2017.
`Copies of all three patents are attached. As further stated below, Bradium believes that
`Microsoft is infringing all three patents. I also write to address some of the comments in your
`letter dated April 20, 2017, which you wrote in response to my March 23, 2017, letters.
`
`Notice of Infringement of U.S. Patent Nos. 9,635,136, 9,641,644, and 9,641,645
`
`Bradium believes that Microsoft has been infringing, and continues to infringe, one or
`more claims of U.S. Patent Nos. 9,635,136 (the “’136 patent”), 9,641,644 (the “’644 patent”),
`and 9,641,645 (the “’645 patent”). Microsoft has been on notice regarding these patents at least
`since the date of my March 23, 2017, letters.
`
`Based on its current investigation, Bradium identifies claim 10 of the ’136 patent, claim
`23 of the ’644 patent, and claim 1 of the ’645 patent as examples of infringed claims. Bradium
`further identifies as infringing products at least those products noted in my March 23, 2017,
`letters.
`
`Bradium intends to seek the Court’s authorization to add these newly-issued patents to
`the litigation entitled Bradium Technologies LLC v. Microsoft Corporation (D. Del. 15-0031-
`RGA), as soon as the Court permits.
`
`Contentions in Your Letter Dated April 20, 2017
`
`
`
`1.
`
`Microsoft’s Invalidity Contentions
`
`Your letter attacks the US Patent Office for allowing Application No. 14/970,526, now
`issued as the ’644 patent, which is surprising given that the Examiner was provided with and
`
`
`
`ANDREWS KURTH KENYON LLP
`Austin Beijing Dallas Dubai Houston London New York Research Triangle Park Silicon Valley The Woodlands Washington, DC
`
`Microsoft Corp. Exhibit 1026
`
`
`
`Matthew Bernstein, Esq.
`May 10, 2017
`Page 2
`
`
`carefully considered Microsoft’s arguments and asserted prior art. In addition, your letter
`mischaracterizes the Examiner’s statement of the basis for allowing the patent. The Examiner
`specifically stated in allowing Claims 1-65 that “[w]hen considering the claims as a whole,
`particularly how these elements interact with the claimed update parcels, the claims are
`found to be novel and non-obvious over the prior art.” Notice at Page 2, Paragraph 4 (emphasis
`added). Microsoft’s arguments regarding the alleged errors of the Patent Office are unsupported
`by the evidence and therefore frivolous.
`
`Your letter also persists in attacking the validity of the ’794 patent, which is one the
`patents already in suit, not one of the new patents. Microsoft’s continued attack on the patent is
`also surprising, given that Microsoft challenged the patent in the Patent Office, received a fair
`hearing, and lost. Further, we note that Microsoft abandoned its appeal to the Federal Circuit on
`this patent, an admission of defeat on the merits.
`
`
`
`2.
`
`Microsoft’s Contentions Regarding Its Infringement
`
`Your letter does not deny that Microsoft infringes the new patents, and instead attacks
`Bradium for not providing detailed infringement contentions. Your letter misconstrues the
`purpose of Image Processing’s March 23, 2017, letters, which was to provide Microsoft with
`notice of the expected issuance of the patents and of Bradium’s allegations of infringement. As
`your letter implicitly acknowledges, Microsoft is well aware that it infringes the new patents as
`well as the previously-issued patents that are asserted in the lawsuit. As you also well know,
`Bradium has already detailed the evidence of such infringement in its 101 pages of infringement
`contentions served on Microsoft in the pending lawsuit. Those contentions were provided after
`Microsoft was required under Court order to make the code for its software available to Bradium
`for review. Again, Microsoft’s position is frivolous.
`
`Microsoft’s further contention that it does not infringe the ’794 patent, which is not one
`of the new patents, is also without basis. Microsoft’s infringement is clearly demonstrated in the
`detailed infringement contentions provided in the litigation.
`
`Microsoft’s Contentions Regarding Unenforceability
`
`Your letter continues in the same vein to make baseless and scurrilous accusations that
`the patents are “unenforceable.” As you know, the individual who made these accusations
`refused to show up for his deposition where his statements could be tested under oath.
`Microsoft’s continued reliance on these baseless accusations is shameful and indicative of its
`weak position.
`
`***
`
`As requested in the March 23, 2017, letters, Bradium demands that Microsoft cease and
`desist from its infringement of the newly issued patents, as well as the earlier patents that are
`already the subject of the lawsuit.
`
`Microsoft Corp. Exhibit 1026
`
`
`
`Matthew Bernstein, Esq.
`May 10, 2017
`Page 3
`
`
`
`
`Enclosures
`
`Very truly yours,
`
`Michael Zachary
`
`Microsoft Corp. Exhibit 1026
`
`
`
`(12) United States Patent
`US 9,635,136 B2
`(10) Patent No.:
`Levanon et al.
`(45) Date of Patent:
`*Apr. 25, 2017
`
`US009635136B2
`
`(54)
`
`OPTIMIZED IMAGE DELIVERY OVER
`LIMITED BANDWIDTH COMMUNICATION
`CHANNELS
`
`(71)
`
`Applicant: Bradium Technologies LLC, Suffem,
`NY (US)
`
`(72)
`
`Inventors:
`
`Isaac Levanon, Raanana (IL); Yonatan
`Lavi, Raanana (IL)
`
`(73)
`
`Assignee:
`
`BRADIUM TECHNOLOGIES LLC,
`Suffern, NY (US)
`
`(*)
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`This patent is subject to a terminal dis-
`claimer.
`
`(21)
`
`Appl. No.: 15/343,052
`
`(22)
`
`Filed:
`
`Nov. 3, 2016
`
`Prior Publication Data
`
`(58) Field of Classification Search
`CPC
`H04N 1/40068; H04N 1/4172; H04N 1/64;
`H04N 21/234345; H04N 21/234363;
`(Continued)
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`4,682,869 A *
`
`4,972,319 A *
`
`7/1987 Itoh ....................... .. G06T 9/004
`358/426.12
`11/1990 Delorme ............ .. G09B 29/007
`340/990
`
`(Continued)
`
`OTHER PUBLICATIONS
`
`Declaration of Yonatan Lavi, Exhibit 1017 with exhibits A-E filed
`in PTAB Case No. IPR2016-00448, all pages.
`
`Primary Examiner 7 David Lazaro
`(74) Attorney, Agent, or Firm iAnatoly S. Weiser, Esq.;
`Techlaw LLP.
`
`(65)
`
`(63)
`
`(51)
`
`(52)
`
`US 2017/0054830 A1
`
`Feb. 23, 2017
`
`(57)
`
`ABSTRACT
`
`Related US. Application Data
`
`Continuation of application No. 15/281,037, filed on
`Sep. 29, 2016, which is a continuation of application
`
`Int Cl
`G08F 5/16
`H04L 29/08
`
`(continued)
`
`(2006.01)
`(2006.01)
`
`.
`(Commued)
`
`Large-scale images are retrieved over network communica-
`tions channels for display on a client device by selecting an
`update image parcel relative to an operator controlled image
`viewpoint to display via the client device. A request
`is
`prepared for the update image parcel and associated with a
`request queue for subsequent issuance over a communica-
`tions channel. The update image parcel is received from the
`communications channel and displayed as a discrete portion
`.
`.
`.
`.
`of the predetermined image. The update image parcel opti-
`mally has a fixed pixel array size, is received in a single and
`us C1'
`or plurality of network data packets, and were the fixed pixel
`CPC """""" " H04L 67/327 (201301); G06F 3/14
`array may be constrained to a resolution less than or equal
`(201301); G06T 3/4092 )G to the resolution of the client device display.
`(Continued)
`27 Claims, 5 Drawing Sheets
`
`
`
`’
`
`’A
`
` 1 3mm
`
`HAVEGAEIGH {:{IWN DS
`
`K40
`
`Microsoft Corp. Exhibit 1026
`
`
`
`1.6m PARCEi
`5
`r
`mm SHIRE
`
`
`
`PAR
`1A
`.
`ERCEL EM #3.
`
`?
`Y
`......
`
`Rams/Rama:
`REHEERBKG
`PQQGSSiHG
`
`44
`
`
`. 50
`
`‘JEWHG Home
`”
`
`.
`:
`{mm};
`
`
`mm:
`‘
`
`52 *
`
`"
`
`Microsoft Corp. Exhibit 1026
`
`
`
`US 9,635,136 B2
`
`Page 2
`
`Related U.S. Application Data
`
`No. 14/970,526, filed on Dec. 15, 2015, which is a
`continuation of application No. 14/547,148, filed on
`Nov. 19, 2014, now Pat. No. 9,253,239, which is a
`continuation of application No. 13/027,929, filed on
`Feb. 15, 2011, now Pat. No. 8,924,506, which is a
`continuation-in-part of application No. 12/619,643,
`filed on Nov. 16, 2009, now Pat. No. 7,908,343,
`which is a continuation of application No. 10/035,
`987, filed on Dec. 24, 2001, now Pat. No. 7,644,131.
`Provisional application No. 60/258,465, filed on Dec.
`27, 2000, provisional application No. 60/258,466,
`filed on Dec. 27, 2000, provisional application No.
`60/258,467,
`filed on Dec. 27, 2000, provisional
`application No. 60/258,468, filed on Dec. 27, 2000,
`provisional application No. 60/258,488, filed on Dec.
`27, 2000, provisional application No. 60/258,489,
`filed on Dec. 27, 2000.
`
`(60)
`
`Int. Cl.
`
`(51)
`
`5,929,860
`
`A*
`
`7/1999
`
`5,995,903
`
`A*
`
`11/1999
`
`6,167,442
`
`A*
`
`12/2000
`
`6,212,301
`
`B1*
`
`4/2001
`
`Warner
`
`6,246,797
`
`B1*
`
`6/2001
`
`6,285,317
`
`B1*
`
`9/2001
`
`6,314,452
`
`6,326,965
`
`6,345,279
`6,346,938
`
`B1*
`
`11/2001
`
`Dekel
`
`B1*
`
`12/2001
`
`Castelli
`
`B1*
`B1*
`
`2/2002
`2/2002
`
`6,397,259
`
`B1*
`
`5/2002
`
`6,449,639
`
`B1*
`
`9/2002
`
`6,496,189
`
`B1*
`
`12/2002
`
`6,525,732
`
`B1*
`
`2/2003
`
`6,608,628
`
`B1*
`
`8/2003
`
`8/2003
`
`Dowell
`
`Hoppe .................. .. G06T 9/001
`345/419
`Smith .................. .. G01C21/00
`340/995.26
`Sutherland ........ .. G06F 17/3028
`709/217
`................... .. G06T 9/00
`382/232
`Castor .................. .. H04N 19/63
`375/E7.035
`Ong ................. .. G01C 21/3647
`340/9952
`................ .. H04N 19/647
`375/E7.045
`........... .. G06F 17/30241
`345/420
`................... .. G06F 17/30905
`Li
`Chan ................. .. G06F 3/04815
`345/419
`Lincke .............. .. G06F 17/3089
`707/E17.116
`Blumberg ....... .. G06F 17/30905
`707/E17.118
`Yaron ................... .. G06T 15/40
`345/419
`Gadh .................... .. G06T 15/20
`345/428
`Ross ..................... .. G06T 17/20
`345/619
`................. .. G06T 9/007
`382/232
`......................... .. G06T 15/40
`345/418
`Robotham .............. .. G06F 3/ 14
`345/581
`Harris ................... .. G06T 15/20
`709/231
`Chang ..................... .. G06T 1/00
`375/E7.065
`Wen .................... .. G06T 1/0078
`382/100
`................ .. H04N19/70
`375/E7.056
`Silverstein ....... .. H04N 21/2662
`375/E7.011
`Whitehead ............. .. H04N1/64
`375/E7.184
`.................. .. H04N 19/ 176
`375/E7.056
`Levanon ................. .. G06F 3/ 14
`345/625
`Levanon ............. .. G06F 3/1454
`345/625
`
`Li
`
`Atsumi
`
`Chai
`
`G06T 3/40
`G06F 3/14
`G09G 5/00
`G06T11/60
`G06T15/04
`G06F 17/24
`G06T19/00
`U.S. Cl.
`
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2011.01)
`(2006.01)
`(2011.01)
`
`CPC .......... .. G06F 17/241 (2013.01); G06T11/60
`(2013.01), G06T15/04 (2013.01), G06T
`19/003 (2013.01), G06T 2207/10032
`(2013.01), G09G 2350/00 (2013.01), G09G
`2370/02 (2013.01), G09G 2370/16 (2013.01)
`Field of Classification Search
`CPC ...... .. H04N 21/25825; G06F 17/30241, G06F
`3/04815; G06T 3/4092, G06T 19/003
`USPC ............. .. 709/202, 203, 217, 218, 230, 231,
`382/305, 232; 345/428, 581, 625
`See application file for complete search history.
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,559,936 A *
`
`5,613,051 A *
`
`9/1996 Poulter
`
`........... .. G06F17/30017
`345/428
`3/1997 Iodice ..................... .. G06F 3/14
`345/428
`
`(52)
`
`(58)
`
`(56)
`
`6,608,933
`
`B1*
`
`6,625,309
`
`B1*
`
`9/2003
`
`6,704,024
`
`B2*
`
`3/2004
`
`6,704,791
`
`B1*
`
`3/2004
`
`6,711,297
`
`B1*
`
`3/2004
`
`6,754,365
`
`B1*
`
`6/2004
`
`6,801,665
`
`B1*
`
`10/2004
`
`6,882,755
`
`B2*
`
`4/2005
`
`6,898,311
`
`B2*
`
`5/2005
`
`6,970,604
`
`B1*
`
`11/2005
`
`7,644,131
`
`B2*
`
`1/2010
`
`8,924,506
`
`B2*
`
`12/2014
`
`* cited by examiner
`
`Microsoft Corp. Exhibit 1026
`
`Microsoft Corp. Exhibit 1026
`
`
`
`U.S. Patent
`
`Apr. 25, 2017
`
`Sheet 1 of 5
`
`US 9,635,136 B2
`
`
`
`311m um
`
`12
`
`:
`
`
`. Bzwasflxm
`EMA
`{WVERSEW ,
`
`” 3‘3
`
`
`
`50312211
`13mm Ema
`
`34
`
`
`
`
`
`5mm
`
`32
`
`18mm
`SEWER
`
`
`
`{9111;131:115
`_
`
`
`mman
`Fmfifimi {3m
`
`Fifi a 2
`
`Microsoft Corp. Exhibit 1026
`
`Microsoft Corp. Exhibit 1026
`
`
`
`U.S. Patent
`
`Apr. 25, 2017
`
`Sheet 2 of 5
`
`US 9,635,136 B2
`
` EeaiPéam
`:
`
`mam SHIRE
`
`‘
`
`42
`
`4a
`
`“:13 ...........
`
`Faxqémm
`
`RECEWEIREQEEES‘E‘
`
`E. ,,,,,,,,,
`
`‘ Wigwam
`.
`
`
`
`Paazaswafi
`
`52 ~‘
`
`
`
`fliogm
`
`
`
`Vsmxa favmsé
`52mm
`
`i mum.
`
`
`{lama
`
`
`
`. . . . . , , , , , .43.?
`
`Mfilfim
`
`REWWG
`
`E.......... “33..
`
`flavssmmé {ama‘am
`
`\ 49
`
`Fig, 3
`
`
`
`
`
`fimveax {Em Mm
`...............................
`,
`6t;
`
`I
`
`mfissgg
`
` 3§§§CRG
`
`mm mm 5,: am (am: 66
`
`53
`
`fig, fig
`
`Microsoft Corp. Exhibit 1026
`
`Microsoft Corp. Exhibit 1026
`
`
`
`U.S. Patent
`
`Apr. 25, 2017
`
`Sheet 3 of 5
`
`US 9,635,136 B2
`
`,,,,,,,,,,,,,,,,
`
`fimmmi
`
`,
`' Emmsisg
`
`
`
`32
`
`EVE“?
`3
`RR 1:99:23?
`
`8% m ..................................
`,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,5;
`'"""""
`
`
`52mm: mam
`
`
`“ism mam
`I
`fiawmms
`--
`REQQES‘KS
`
`
`
`mm
`Emma
`58
`{Emma
`
`
`
`
`Smaflmm‘m
`
`
`Réiisvzmm
`QMME Eéeai
`
`
`
`
`g9
`92
`
`Fig, g
`
`’
`
`86
`
`94
`
`
`
`
`
`Samfiieam
`
`
`Pfiiflfim’ Rm???
`QSSEEREWES?
`Wmm
`
`
`°
`"""""""""
`~~~~~~~~~~
`
`1% «j
`
`9 Wm
`
`
`F’fi
`
`66
`
`
`Microsoft Corp. Exhibit 1026
`
`Microsoft Corp. Exhibit 1026
`
`
`
`U.S. Patent
`
`Apr. 25, 2017
`
`Sheet 4 of 5
`
`US 9,635,136 B2
`
`fiavigm .........._
`gm“
`
`
`Amusmmma
`mm”
`5 women m
`QRSEMWW
`
`
`
`' :22
`
`-----------------
`
`,
`
`
`Pmszm
`fiiimm :3;
`i652? Pam.
`
`
`
`
`
`Rmmmei
`Mimi:
`
`fissm‘! 9mm ‘
`
`
`..............
`
`E amsfiima?
`a
`_
`
`HQ, “f
`
`f 1219
`
`' 124
`
`SM?
`
`1
`
`128
`
`126
`{am mam?
`
`633%
`................
`....................... W
`
`
`
`
`
`
`$39
`
`Rama: QVERW
`53mm
`{Bismmfiémv
`
`Bamms
`
`,,
`
`= 144
`
`.A v
`
`:;~,
`
`.
`
`mg. E.
`
`.....
`
`3
`
`Srmmma
`Pmmas?
`
`45
`
`
`
`Assam Emma
`
`Mam {ism
`
`
`
`a................“I ..............
`i
`E;
`..
`- >‘
`:........................................................................23
`
`{Emma
`
`
`
`
`Mam
`3;
`I ..................................
`,,,,,,,,,,,,,,,,l
`
`
`
`“348
`1 W:- gfi
`
`---------
`
`~
`
`53
`
`a g
`
`MG
`
`Microsoft Corp. Exhibit 1026
`
`Microsoft Corp. Exhibit 1026
`
`
`
`U.S. Patent
`
`Apr. 25, 2017
`
`Sheet 5 of 5
`
`US 9,635,136 B2
`
`£32 £53:
`mm P
`
`333? ? fimsms a?
`
`I Vzwm‘; mamas
`
`
`{mam
`’ Vtiwms mem ’
`Baum
`
`354
`
`
`
`EWREW?
`Ffifiifl m P
`
`
`
`5
`
`156
`
`g
`
`“PARCEL
`;
`.LEVEH} >22:
`
`
`
`
` 3F PAM:
`LEVEL 2) < i.
`
`
`{am {Haw
`
`WEE E
`i
`‘32
`
`2
`
`/
`
`16%}
`
`
`
`Am: mam.
`fiawwwm
`REQWSE‘
`
`
`
`
`
`
`1m .
`
`immast
`
`mam Pamm *
`
`
` 1",?8
`
`r
`éSSSSfi Emma:
`35
`3 Mam
`
`
`ammmze
`
`{Emma [Ema {Em i
`
`
`
`gamma (mm
`Emmi;
`
`?
`
`?
`
`'
`
`“£82
`
`
`
`fimmag
`MAKES?
`.,
`
`Rum P
`’ 384
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`g
`
`fimmm
`Nam? mm A
`as; warm?
`t ...............
`................
`
`;
`
`’
`
`;
`
`Cfifii’ufi
`Wigwam:
`maume
`
`"183
`
`
`
`
` SM"?
`Psmma?
`i..................................
`
`“332
`
`K “$88
`
`HG! m
`
`Microsoft Corp. Exhibit 1026
`
`Microsoft Corp. Exhibit 1026
`
`
`
`US 9,635,136 B2
`
`1
`OPTIMIZED IMAGE DELIVERY OVER
`LIMITED BANDWIDTH COMMUNICATION
`CHANNELS
`
`PRIORITY CLAIMS/RELATED APPLICATIONS
`
`This application is a continuation of and claims priority to
`US. patent application Ser. No. 14/970,526, filed Dec. 15,
`2015, entitled OPTIMIZED IMAGE DELIVERY OVER
`LIMITED BANDWIDTH COMMUNICATION CHAN-
`
`NELS; this application is also a continuation of and claims
`priority to US. patent application Ser. No. 15/281,037, filed
`Sep. 29, 2016, entitled OPTIMIZED IMAGE DELIVERY
`OVER LIMITED BANDWIDTH COMMUNICATION
`
`CHANNELS; each of the US. patent application Ser. No.
`14/970,526 and Ser. No. 15/281,037 is a continuation of and
`claims priority to US. patent application Ser. No. 14/547,
`148, filed Nov. 19, 2014, entitled OPTIMIZED IMAGE
`DELIVERY OVER LIMITED BANDWIDTH COMMUNI-
`CATION CHANNELS, now US. Pat. No. 9,253,239; which
`is a continuation of and claims priority to US. patent
`application Ser. No. 13/027,929, filed Feb. 15, 2011, entitled
`OPTIMIZED IMAGE DELIVERY OVER LIMITED
`BANDWIDTH COMMUNICATION CHANNELS, now
`US. Pat. No. 8,924,506; which is a continuation-in-part of
`and claims priority to US. patent application Ser. No.
`12/619,643, filed on Nov. 16, 2009, entitled OPTIMIZED
`IMAGE DELIVERY OVER LIMITED BANDWIDTH
`COMMUNICATION CHANNELS, now US. Pat. No.
`7,908,343; which is a continuation of and claims priority to
`US. patent application Ser. No. 10/035,987, filed on Dec.
`24, 2001, entitled OPTIMIZED IMAGE DELIVERY OVER
`LIMITED BANDWIDTH COMMUNICATION CHAN-
`NELS, now US. Pat. No. 7,644,131; which claims the
`benefit under 35 U.S.C. §119(e) of US. Provisional Appli-
`cation Nos. 60/258,488, 60/258,489, 60/258,465, 60/258,
`468, 60/258,466, and 60/258,467, all filed Dec. 27, 2000.
`The disclosures of all the foregoing patent documents are
`incorporated herein by reference as if fully set forth herein,
`including Figures, Claims, and Tables. The present applica-
`tion is also related to application Ser. No. 10/035,981,
`entitled SYSTEM AND METHODS FOR NETWORK
`IMAGE DELIVERY WITH DYNAMIC VIEWING FRUS-
`TUM OPTIMIZED FOR LIMITED BANDWIDTH COM-
`MUNICATION CHANNELS, Levanon et al., filed on Dec.
`24, 2001, now US. Pat. No. 7,139,794, issued on Nov. 21,
`2006, which is assigned to the Assignee of the present
`application.
`
`FIELD
`
`The disclosure is related to network based, image distri-
`bution systems and, in particular, to a system and methods
`for efficiently selecting and distributing image parcels
`through a narrowband or otherwise limited bandwidth com-
`munications channel to support presentation of high-reso-
`lution images subject to dynamic viewing frustums.
`
`BACKGROUND
`
`The Internet and or other network systems may provide a
`unique opportunity to transmit for example complex images,
`typically large scale bit-maps, particularly those approach-
`ing photo-realistic levels, over large area and or distances. In
`common application, the images may be geographic, topo-
`graphic, and or other highly detailed maps. The data storage
`requirements and often proprietary nature of such images
`
`2
`
`could be such that conventional interests may be to transfer
`the images on an as-needed basis.
`In conventional fixed-site applications, the image data
`may be transferred over a relatively high-bandwidth network
`to client computer systems that in turn, may render the
`image. Client systems may typically implement a local
`image navigation system to provide zoom and or pan
`functions based on user interaction. As well recognized
`problem with such conventional systems could be that full
`resolution image presentation may be subject to the inherent
`transfer latency of the network. Different conventional sys-
`tems have been proposed to reduce the latency affect by
`transmitting the image in highly compressed formats that
`support progressive resolution build-up of the image within
`the current client field of view. Using a transform com-
`pressed image transfer function increases the field of the
`image that can be transferred over a fixed bandwidth net-
`work in unit time. Progressive image resolution transmis-
`sion, typically using a differential resolution method, per-
`mits an approximate image to be quickly presented with
`image details being continuously added over time.
`Tzou, in US. Pat. No. 4,698,689, describes a two-dimen-
`sional data transform system that supports transmission of
`differential coefficients to represent an image. Subsequent
`transmitted coefficient sets are progressively accumulated
`with prior transmitted sets to provide a succeedingly refined
`image. The inverse-transform function performed by the
`client computer is, however, highly compute intensive. In
`order to simplify the transform implementation and further
`reduce the latency of presenting any portion of an approxi-
`mate image, images are subdivided into a regular array. This
`enables the inverse-transform function on the client, which
`is time-critical, to deal with substantially smaller coefficient
`data sets. The array size in Tzou is fixed, which leads to
`progressively larger coefficient data sets as the detail level of
`the image increases. Consequently, there is an inherently
`increasing latency in resolving finer levels of detail.
`An image visualization system proposed by Yap et al.,
`US. Pat. No. 6,182,114, overcomes some of the foregoing
`problems. The Yap et al. system also employs a progressive
`encoding transform to compress the image transfer stream.
`The transform also operates on a subdivided image, but the
`division is indexed to the encoding level of the transform.
`The encoded transform coefficient data sets are, therefore, of
`constant size, which supports a modest improvement in the
`algorithmic performance of the inverse transform operation
`required on the client.
`Yap et al. adds utilization of client image panning or other
`image pointing input information to support a foveation-
`based operator to influence the retrieval order of the subdi-
`vided image blocks. This two-dimensional navigation infor-
`mation is used to identify a foveal region that is presumed
`to be the gaze point of a client system user. The foveation
`operator defines the corresponding image block as the center
`point of an ordered retrieval of coefficient sets representing
`a variable resolution image. The gaze point image block
`represents the area of highest image resolution, with reso-
`lution reduction as a function of distance from the gaze point
`determined by the foveation operator. This technique thus
`progressively builds image resolution at the gaze point and
`succeedingly outward based on a relatively compute inten-
`sive function. Shifts in the gaze point can be responded to
`with relative speed by preferentially retrieving coefficient
`sets at and near the new foveal region.
`Significant problems remain in permitting the convenient
`and effective use of complex images by many different types
`of client systems, even with the improvements provided by
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Microsoft Corp. Exhibit 1026
`
`Microsoft Corp. Exhibit 1026
`
`
`
`US 9,635,136 B2
`
`3
`the various conventional systems. In particular, the imple-
`mentation of conventional image visualization systems is
`generally unworkable for smaller, often dedicated or embed-
`ded, clients where use of image visualization would clearly
`be beneficial. Conventional approaches effectively presume
`that client systems have an excess of computing perfor-
`mance, memory and storage. Small clients, however, typi-
`cally have restricted performance processors with possibly
`no dedicated floating-point support, little general purpose
`memory, and extremely limited persistent storage capabili-
`ties, particularly relative to common image sizes. A mobile
`computing device such as mobile phone, smart phone, tablet
`and or personal digital assistant (PDA) is a characteristic
`small client. Embedded, low-cost kiosk, automobile navi-
`gation systems and or Internet enabled I connected TV are
`other typical examples. Such systems are not
`readily
`capable, if at all, of performing complex, compute-intensive
`Fourier or wavelet transforms, particularly within a highly
`restricted memory address space.
`As a consequence of the presumption that the client is a
`substantial computing system, conventional image visual-
`ization systems also presume that the client is supported by
`a complete operating system.
`Indeed, many expect and
`require an extensive set of graphics abstraction layers to be
`provided by the client system to support the presentation of
`the delivered image data. In general, these abstraction layers
`are conventionally considered required to handle the map-
`ping of the image data resolution to the display resolution
`capabilities of the client system. That is, resolution resolved
`image data provided to the client is unconstrained by any
`limitation in the client system to actually display the corre-
`sponding image. Consequently, substantial processor per-
`formance and memory can be conventionally devoted to
`handling image data that is not or cannot be displayed.
`Another problem is that small clients are generally con-
`strained to generally to very limited network bandwidths,
`particularly when operating under wireless conditions. Such
`limited bandwidth conditions may exist due to either the
`direct technological constraints dictated by the use of a low
`bandwidth data channel or indirect constraints imposed on
`relatively high-bandwidth channels by high concurrent user
`loads. Cellular connected PDAs and webphones
`are
`examples of small clients that are frequently constrained by
`limited bandwidth conditions. The conventionally realizable
`maximum network transmission bandwidth for such small
`devices may range from below one kilobit per second to
`several tens of kilobits per second. While Yap et al. states
`that the described system can work over low bandwidth
`lines, little more than utilizing wavelet-based data compres-
`sion is advanced as permitting effective operation at low
`communications bandwidths. While reducing the amount of
`data that must be carried from the server to the client is
`significant, Yap et al. simply relies on the data packet
`transfer protocols to provide for an efficient transfer of the
`compressed image data. Reliable transport protocols, how-
`ever, merely mask packet losses and the resultant, some-
`times extended recovery latencies. When such covered
`errors occur, however, the aggregate bandwidth of the con-
`nection is reduced and the client system can stall waiting for
`further image data to process.
`Consequently, there remains a need for an image visual-
`ization system that can support small client systems, place
`few requirements on the supporting client hardware and
`software resources, and efficiently utilize low to very low
`bandwidth network connections.
`
`SUMMARY
`
`Thus, a general purpose of the present invention is to
`provide an efficient system and methods of optimally pre-
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`
`senting image data on client systems with potentially limited
`processing performance, resources, and communications
`bandwidth.
`
`This is achieved in the present invention by providing for
`the retrieval of large-scale images over network communi-
`cations channels for display on a client device by selecting
`an update image parcel relative to an operator controlled
`image viewpoint to display via the client device. A request
`is prepared for the update image parcel and associated with
`a request queue for subsequent issuance over a communi-
`cations channel. The update image parcel is received from
`the communications channel and displayed as a discrete
`portion of the predetermined image. The update image
`parcel optimally has a fixed pixel array size, is received in
`a single and or plurality of network data packets, and were
`the fixed pixel array may be constrained to a resolution less
`than or equal to the resolution of the client device display.
`An advantage of the present invention is that both image
`parcel data requests and the rendering of image data are
`optimized to address the display based on the display
`resolution of the client system.
`Another advantage of the present invention is that the
`prioritization of image parcel requests is based on an adapt-
`able parameter that minimizes the computational complexity
`of determining request prioritization and, in turn, the pro-
`gressive improvement in display resolution within the field
`of view presented on a client display.
`A further advantage of the present invention is that the
`client software system requires relatively minimal client
`processing power and storage capacity. Compute intensive
`numerical calculations are minimally required and image
`parcel data is compactly stored in efficient data structures.
`The client software system is very small and easily down-
`loaded to conventional computer systems or embedded in
`conventional dedicated function devices, including portable
`devices, such as PDAs, tablets and webphones.
`Still another advantage of the present invention is that
`image parcel data requests and presentation can be readily
`optimized to use low to very low bandwidth network con-
`nections. The software system of the present
`invention
`provides for re-prioritization of image parcel data requests
`and presentation in circumstances where the rate of point-
`of-view navigation exceeds the data request rate.
`Yet another advantage of the present invention is that
`image parcel data rendering is performed without requiring
`any complex underlying hardware or software display sub-
`system. The client software system of the present invention
`includes a bit-map rendering engine that draws directly to
`the video memory of the display,
`thus placing minimal
`requirements on any underlying embedded or disk operating
`system and display drivers. Complex graphics and anima-
`tion abstraction layers are not required.
`Still another advantage of the present invention is that
`image parcel block compression is used to obtain fixed size
`transmission data blocks. Image parcel data is recoverable
`from transmission data using a relatively simple client
`decompression algorithm. Using fixed size transmission data
`blocks enables image data parcels to be delivered to the
`client in bounded time frames.
`
`A yet further advantage of the present invention is that
`multiple data forms can be transferred to the client software
`system for concurrent display. Array overlay data, correlated
`positionally to the image parcel data and generally insensi-
`tive to image parcel resolution, can be initially or progres-
`
`Microsoft Corp. Exhibit 1026
`
`Microsoft Corp. Exhibit 1026
`
`
`
`US 9,635,136 B2
`
`5
`sively provided to the client for parsing and parallel pre-
`sentation on a client display image view.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`These and other advantages and features of the present
`invention will become better understood upon consideration
`of the following detailed description of the invention when
`considered in connection with the accompanying drawings,
`in which like reference numerals designate like parts
`throughout the figures thereof, and wherein:
`FIG. 1 depicts a preferred system environment within
`which various embodiments of the present invention can be
`utilized;
`FIG. 2 is a block diagram illustrating the preparation of
`image parcel and overlay data set that are to be stored by and
`served from a network server system in accordance with a
`preferred embodiment of the present invention;
`FIG. 3 is a block diagram of a client system image
`presentation system constructed in accordance with a pre-
`ferred embodiment of the present invention;
`FIG. 4 provides a data block diagram illustrating an
`optimized client image block processing path constructed in
`accordance with a preferred embodiment of the present
`invention;
`FIG. 5 is a process flow diagram showing a main pro-
`cessing thread implemented in a preferred embodiment of
`the present invention;
`FIG. 6 provides a process flow diagram showing a net-
`work request thread implemented in a preferred embodiment
`of the present invention;
`FIG. 7 provides a process flow diagram showing a display
`image rendering thread implemented in a preferred embodi-
`ment of the present invention;
`FIG. 8 provides a process flow diagram showing the
`parcel map processing performed preliminary to the render-
`ing of image data parcels in accordance with a preferred
`embodiment of the present invention;
`FIG. 9 provides a process flow diagram detailing the
`rendering and progressive prioritization of image parcel data
`download requests in accordance with a preferred embodi-
`ment of the present invention; and
`FIG. 10 provides a process flow diagram detailing the
`determination of an optimal detail level for image parcel
`presentation for a current viewing frustum in accordance
`with a preferred embodiment of the present invention.
`
`DETAILED DESCRIPTION OF ONE OR MORE
`EMBODIMENTS
`
`The preferred operational environment 10 of the present
`invention is generally shown in FIG. 1. A network server
`system 12, operating as a data store and server of image data,
`is responsive to requests received through a communications
`network, such as the Internet 14 generally and various tiers
`of internet service providers (ISPs) including a wireless
`connectivity provider 16. Client systems, including conven-
`tional workstations and personal computers 18 and smaller,
`typically dedicated function devices often linked through
`wireless network connections, such as PDAs, webphones
`20, and automobile navigation systems,
`source image
`requests to the network server 12, provide a client display
`and enable image navigational input by a user of the client
`system. Alternately, a