Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 1 of 28 PageID 99
`
`
`
`Exhibit 2
`U.S. Patent No. 9,349,183
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 2 of 28 PageID 100
`
`US009349183B1
`
`(12) United States Patent
`Douglas et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 9,349,183 B1
`May 24, 2016
`
`(54) METHOD AND APPARATUS FOR THREE
`DIMIENSIONAL VIEWING OF IMAGES
`
`(*) Notice:
`
`(76) Inventors: David Byron Douglas, Winter Park, FL
`(US); Robert E. Douglas, Winter Park,
`FL (US)
`Subject to any disclaimer, the term of this
`atent is extended or adiusted under 35
`p
`U.S.C. 154(b) by 1335 days.
`(21) Appl. No.: 12/176.569
`
`6,108,005. A * 8/2000 Starks et al. .................. 345/419
`6,115.449 A * 9/2000 Jang et al. ....................... 378/41
`6,124,977 A * 9/2000 Takahashi ......
`... 359,636
`6,476,607 B1 * 1 1/2002 Dannels et al.
`... 324,309
`6,532,008 B1* 3/2003 Guralnick ......
`... 345,419
`6,862,364 B1* 3/2005 Berestov .....
`... 382,132
`7,298,372 B2 * 1 1/2007 Pfister et al. ...
`... 345,424
`7,647,593 B2 *
`1/2010 Matsumoto ...
`... 718, 106
`7,822,265 B2. 10/2010 Berretty .......
`... 382,154
`7,840,047 B2 11/2010 Boing et al. .........
`... 38.2/128
`8,228,327 B2 * 7/2012 Hendrickson et al. ........ 345,420
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`(22) Filed:
`
`Jul. 21, 2008
`
`DE
`
`EP 1843296 A1 * 5, 2006 .............. GO6T 11.00
`
`Related U.S. Application Data
`(63) Continuation-in-part of application No. 1 1/941,578,
`filed on Nov. 16, 2007.
`(60) Provisional application No. 60/877,931, filed on Dec.
`28, 2006.
`(51) Int. Cl.
`G09G 5/14
`G09G 5/16
`G06K 9/78
`G06K 9/80
`G06T 700
`52) U.S. C.
`(52)
`CPC .................................... G06T 770075 (2013.01)
`(58) Field of Classification Search
`None
`See application file for complete search history.
`
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4472,737 A * 9/1984 Iwasaki ........................... 34.8/51
`5,049.987 A * 9/1991 Hoppenstein ...
`... 348/48
`5,233,458 A * 8/1993 Moffitt et al. ...
`359,227
`5,682,437 A * 10/1997 Okino et al. .................. 382/100
`6,034,716 A * 3/2000 Whiting et al. ................. 348/36
`
`
`
`OTHER PUBLICATIONS
`Martin et al. “Stereographic Viewing of 3D Ultrasound Images: A
`Novelty or a Tool?” IEEE Ultrasonics Symposium (1995) pp. 1-4.*
`(Continued)
`
`Primary Examiner — Chan Park
`Assistant Examiner — Mia M Thomas
`(74) Attorney, Agent, or Firm — Anderson Gorecki &
`Rouille LLP
`
`ABSTRACT
`(57)
`A method, apparatus and computer program product for
`three-dimensional viewing of images is presented. Embodi
`9.
`ges 1Sp
`ments of the invention provide a process for combining slices
`generated by medical imaging devices to create a Volume of
`interest and then present this Volume in a three-dimensional
`representation to a head display unit so that the user can
`obtain a holistic view of the patient. Key image processing
`techniques are applied which enable the user to rotate and
`view the volume of interest from alternative viewpoints; to
`enable tissue subtraction to facilitate unobstructed viewing of
`a region of interest; to identify differing tissues with color
`schematics; to Zoom in for optimal viewing; and to view a
`moving image of a Volume of interest.
`
`18 Claims, 14 Drawing Sheets
`
`3:TS Wes:::::RS:Roys &xis & 8A
`Sces
`
`ArrassississCRRssssss Wor
`SRs:
`
`s::sec:8aa88:a: 888 &SLex is sis
`
`displays, Naisa assailay
`it is a sass rer sistery
`sys ease of Tsiriaiwississists, review resist for
`88: Fayetty-F8:RS:
`
`-
`
`a:18
`
`m is
`
`X:SPAYMits, 8:33, Aki Aseca: Testate'ssasses
`He 8XTs: Wisy:ENSAEt. EE 8, PNSFRERGE EYs
`&:W8;&R38TRs &:38kti Biafr His
`E:ASERSA::f3 Risse: A RES
`8:sissal:Assist Sar
`
`rm as
`
`ser sectiisasts for serAasaitersssss retress
`skis RER:
`
`As
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 3 of 28 PageID 101
`
`US 9,349,183 B1
`Page 2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`2011/0107270 A1* 5/2011 Wang et al. ................... 715,850
`2011/0109620 A1* 5/2011 Hong et al. ...
`... 345,419
`2011/O194728 A1* 8, 2011 Kutcka et al.
`382/100
`2011/0273543 A1* 11/2011 Ushio et al. ..................... 348.54
`2012,0008734 A1
`1/2012 Thomson et al. ................. 378/4
`2012,0008735 A1
`1/2012 Maurer et al. ..
`... 378.5
`2012/0038631 A1* 2/2012 Mayhew et al.
`345/419
`2012/0056998 A1* 3/2012 Kang et al.......
`... 348/47
`2012/0120207 A1* 5, 2012 Shimazaki et al. .
`... 34.8/51
`2012/0162219 A1 ck
`6, 2012 Kobayashi et al
`345/419
`2299926. A
`.292 Nahm, ..........
`600,411
`29292,528 A
`.292 pani
`... 606, 41
`292922. A
`2292 Wir. 382,131
`22.22395 A
`22 Miller- 348/36
`2012/0242569 A1* 9/2012 Hamagishi.
`345,156
`2012fO269424 A1* 10, 2012 Ebata et al. ..
`382,154
`2013,0003020 A1
`1/2013 Koehler et al. .
`... 353.7
`2,392 A. 629 Swon.
`... 348.54
`2013,0182085 A1* 7, 2013 Ziarati .........
`... 34.8/51
`2014f0065663 A1* 3, 2014 Vasquez et al. .
`435/29
`2014/0176685 A1
`6/2014 Oikawa et al. ..
`... 34.8/51
`2014/0253698 A1* 9, 2014 Evans et al. ..
`... 348.54
`
`
`
`8,567,954 B2 * 10/2013 Koehler et al. ................... 353.7
`2002/0101658 A1* 8/2002 Hoppenstein
`359,466
`2003/0026474 A1
`2/2003 Yano ............................. 382,154
`2003/0107644 A1* 6/2003 Choi ............................... 348/49
`2003/01941 19 A1 * 10/2003 Manjeshwar et al.
`382,131
`2003/021872O A1 ck
`11, 2003 Morita et al.
`351/222
`2004/022363.6 A1* 11, 2004 Edic et al. ...
`382,131
`2004/0254454 A1* 12, 2004 Kockro ....
`600,424
`2005, 0096530 A1* 5, 2005 Daw et al. ...
`600,408
`2005/0148848 A1* 7/2005 Guang et al.
`600/407
`2005/0244050 A1* 11, 2005 Nomura et al.
`382,154
`2005/0278408 A1* 12/2005 Matsumoto ................... TO9/200
`2006/0013472 A1* 1/2006 Kagitani ....................... 382,154
`2006,0077204 A1* 4, 2006 Pfister et al.
`345,424
`2006/0171028 A1* 8, 2006 Okawa et al.
`359,463
`2006/017.7133 A1* 8, 2006 Kee ................
`382,173
`2006/0210111 A1* 9, 2006 Cleveland et al. .
`382,103
`2006/0268104 A1* 11/2006 Cowan et al. ..
`... 348/42
`2007/0058249 A1* 3, 2007 Hirose et al. .......
`359,464
`2007, 0147671 A1* 6, 2007 DiVincenzo et al. ........ 38.2/128
`2008.0025584 A1
`1/2008 Kunz et al. .........
`382/128
`Hakeretal. “Nondistorting Falttening Maps and the 3D Visualization
`2008/0055.305 A1* 3, 2008 Blank et al. .
`345,419
`of Colon CT Images' IEEE Transactions on Medical Imaging vol. 19.
`2008.0117233 A1* 5, 2008 Mather et al.
`345,690
`No. 7, Jul. 2000 pp. 1-6.*
`29929.3468. A
`22999 Benadet al.
`... 378.98
`.
`.
`.
`38885 3.
`A. ck 658 E. al. stal .
`. 3. Interrante et al. “Strategies for Effectively Visualizing 3D Flow with
`.
`.
`.
`.
`.
`
`2009, O232275 A1* 9, 2009 SE . .
`.
`378/40
`Volume LIC” IEEE Visualization Conference (1997) pp. 1-4.*
`.
`.
`.
`.
`.
`2010/01948.61 A1* 8/2010 Hoppenstein . ... 348/48
`Peterson et al. “Volvulus of Gastrointestinal Tract” Radiographics
`2010/02O1785 A1* 8, 2010 Lantin .........
`348/46,
`209; 29 pp. 1281-1293.*
`2010, O246911 A1* 9, 2010 Rabben et al.
`382,131
`2011/0026808 A1
`2/2011 Kim et al. ..................... 382,154
`
`OTHER PUBLICATIONS
`
`* cited by examiner
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 4 of 28 PageID 102
`
`U.S. Patent
`
`May 24, 2016
`
`Sheet 1 of 14
`
`US 9,349,183 B1
`
`24
`
`22
`
`
`
`
`
`Digital
`Recording
`Device
`
`
`
`General
`Purpose
`Processor
`
`image to eye
`
`s
`
`20
`
`AGILE 1.
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 5 of 28 PageID 103
`
`U.S. Patent
`
`May 24, 2016
`
`Sheet 2 of 14
`
`US 9,349,183 B1
`
`3 will
`
`
`
`Šess3
`issists
`:
`sy is
`
`so-it
`axis&
`essee
`gigs:f
`R.E.
`resis
`swice
`
`
`
`ses
`
`axiar
`Sty
`
`3&tefa
`is &se
`sciesses
`
`FIGURE 2
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 6 of 28 PageID 104
`Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 6 of 28 PageID 104
`
`U.S. Patent
`
`May 24, 2016
`
`Sheet 3 of 14
`
`US 9,349,183 B1
`
`PEER-«‘9‘
`
`‘1 {3‘3 LEVP
`
`‘
`
`'
`
`,
`
`'
`
`~
`
`_
`
`V
`
`.
`
`e
`
`112'
`
`1M
`
`x
`;
`
`.
`
`-
`
`-
`
`3f
`
`2‘" \\"\\.‘.\\__\
`
`"3”»._..‘.-¥.,...\x”;"yfl
`
`New: Diagram notw
`
`seaie
`
`FiGURE SEQ
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 7 of 28 PageID 105
`
`U.S. Patent
`
`May 24, 2016
`
`Sheet 4 of 14
`
`US 9,349,183 B1
`
`-
`
`SS.
`
`S S.
`
`.
`
`
`
`
`
`'.
`s
`s
`- w Ns.S.
`\s SSS s^
`S
`Y.
`SY
`Ns |
`saw N
`AN sW
`s
`
`
`
`
`
`i. s f
`
`See
`ReVR
`
`y
`
`Siste, isgrams sistia
`S&e
`
`FIGURE 3B
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 8 of 28 PageID 106
`
`U.S. Patent
`
`May 24, 2016
`
`Sheet 5 of 14
`
`US 9,349,183 B1
`
`
`
`Six
`
`Ostiff tisstä
`% - 30 isg
`
`(city
`theta
`Court
`theta 8 (a sign deg
`
`S$8
`
`Right
`Stief
`t
`Crisian || tieta
`theta 8 || s -at des
`
`city:
`thtas
`S.SE ties
`
`Raw atta - rax
`*ferirsty & Stieg
`Ray aipts st; REy.
`a sick 888 &t's
`x 338
`R}s >& E:
`*Reirassy - 5 deg
`
`
`
`Rei alpis a tax
`royaay 8 is des
`key apias, Evr
`8's sits 89888 &
`S waiti:
`
`
`
`S.
`
`R8 at a it
`so: itally at 5 deg
`
`S.S.
`
`FIGURE
`
`Note: )i&sf8 it not to
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 9 of 28 PageID 107
`
`U.S. Patent
`
`May 24, 2016
`
`Sheet 6 of 14
`
`US 9,349,183 B1
`
`
`
`is
`
`FIGURE 5
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 10 of 28 PageID 108
`
`U.S. Patent
`
`May 24, 2016
`
`Sheet 7 of 14
`
`US 9,349,183 B1
`
`
`
`
`
`
`
`S •rs al
`SediaS
`
`
`
`
`
`
`
`N
`
`igure is
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 11 of 28 PageID 109
`
`U.S. Patent
`
`May 24, 2016
`
`Sheet 8 of 14
`
`US 9,349,183 B1
`
`:S is 8 disaics is tie
`-isits: if its
`to Sey-C83 citate fire
`woxe: af intergst Y
`
`
`
`sis
`XX 3
`
`is is tie distaire it; i.e. x
`direction for EVF's point going
`straight forwardig the s-coordinate
`i e Foxe & Sest
`
`witH is is is sistance is tie
`3-8 scies: $88 the Sw's $33 to the
`& RS SERS: Es Sixes & Reesi
`
`Wixe: a siegest
`
`FGLIRE 7
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 12 of 28 PageID 110
`
`U.S. Patent
`
`May 24, 2016
`
`Sheet 9 of 14
`
`US 9,349,183 B1
`
`s: T
`
`
`
`SESSRCS SS EF
`
`ERES FRS SEC F SEE
`SRES
`
`RSS SE SES RRSSSSSSS
`SSRES
`
`E::SE F
`
`SEER SE8 GS FREE SCES
`
`SSES:
`
`E PRS FSR A. E. E.E.
`
`SSESS88,
`
`38.3 FSS: Ex RS SE
`
`SPAY38s, 38 A SEA SF&Y 38 it is, AS ERASE FC: R SELEF's
`EYE ASE &
`E & S.A. W.E.8s A&E, SE WE POS FOR
`SS
`EF EYE SS
`E
`SEE SESRES
`
`—
`
`iii, Aki SASE &R E risis' ASE &
`s:Sir AYERS, 8 ft.
`S$ 3.38 SSSRS & Ess, E SS PBS FRSE RSS EYE.
`83 ESSE 88.S. s. SSRES: E SSE F.
`S.E
`SF SES: E
`E
`R S
`R SERE SEES
`$SESSESSAGE SS SER
`
`s:
`
`s:
`
`a:S
`
`s
`
`4.
`
`a /"
`
`SER SEE: sisges F&R SSC-RASE A&E POSSESS RES REAL -
`F: FRE FEREE
`
`FGURE 8
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 13 of 28 PageID 111
`
`U.S. Patent
`
`May 24, 2016
`
`Sheet 10 of 14
`
`US 9,349,183 B1
`
`as sis Y
`
`SER SEESS & ERA. We has asse
`
`SS
`
`ass
`
`SSXSSSE33&
`
`8 SERESS & Si3&REA888.
`SSSE :
`ERSE WEFF8& &e:S
`
`E.
`
`'
`
`CE is SS SYS 8&SE : :
`RSSSAYS38, it is $38, &h: $88.88
`& SER&AEE 8:338g &888, he w88 OSN
`F 8: Et EYE &SR
`88S &
`ESSES
`
`Y
`
`i.S.S
`
`RiSF&Yixias, 8 is hits. Ah Sia&E FSR SER3 EYE BASEE 8
`HEATERNATE WEWSNG AR&E, SHE wisw Point For HS Right /
`EYE, &
`its Woits F EN SEREST ARE: 888& S.E is as For
`SSE SEES YES
`Six
`F&R
`R3
`EYE PR838 &
`A SERS & Fiese-Eise&SCRia: SA&E to Se SSEF:
`
`SSS
`
`
`
`FGRE
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 14 of 28 PageID 112
`
`U.S. Patent
`
`May 24, 2016
`
`Sheet 11 of 14
`
`US 9,349,183 B1
`
`SS Y
`
`is 38&GE : B & BiRs & EES FRCS Se Y
`SEE&SS $88S is
`AE SEE S F3
`r $8&E
`
`s
`
`
`
`Y.
`E.
`, AS Sir Sage Fir is
`,
`SSFA's $83, S:
`BASE: $;
`it is a WE88& Assi, Sie's Sifroit F&R fi
`EEE
`SE
`3: SES RERS
`
`Bisr AY.8s, is tie 3. A sit SREB sage F&R HE Right EYE
`SASEE 88 SS 8: A YEYi*is Assis, Sie Wii Fif F&R
`R&G
`EYE & S.E W8388 firS & SEREERS -
`FERES S&S FREEF EE 8:
`EFEEE 8: FO
`SE ris SS RE F$8 SSSE3 SEE-3SS$83 St
`E SER
`
`FURE
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 15 of 28 PageID 113
`
`U.S. Patent
`
`May 24, 2016
`
`Sheet 12 of 14
`
`US 9,349,183 B1
`
`SS was v
`
`sorting woxes of this is ess syA prers R's Y of this woxE. Y
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`SS-S8. CRS C
`
`S S SSESS SS
`RE:
`E
`
`Siâ& Sr R ESSF, SY
`R
`3
`ii., &
`EiSP&Y 88s, SR St.
`tiss Axists, is gist R&S FRHS
`BASE &
`Sis
`is
`ESE, SE 38 SSS
`SES
`
`SF&Y's, $88: 83, a C&CRE 8&E F&R SSE Rigi YE
`SASEE 8: Eta SRS &K&E, SS-S WE P8 FOR SHS
`883: SY, & 8:
`EFORS: Or 88 SSSESS &
`F.
`R$8; Sie
`it
`is ris S.
`Ler
`fa
`& S&S
`R RAR's fir
`S. R.
`E. S.
`&
`E feli&ESS; S&S
`SR
`
`Fe RE fif
`
`SSS
`
`SS3
`
`SS
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 16 of 28 PageID 114
`
`U.S. Patent
`
`May 24, 2016
`
`Sheet 13 of 14
`
`US 9,349,183 B1
`
`S. \
`
`SEESS & E S FSS FR & EF EYE
`
`y
`
`m m m m m m m m
`
`SEES: S, NE
`
`SE, SS FRS R3
`
`EYE
`
`-- S.
`
`RSPAYixes, iN T8E situ, AN Rags FQR THE LEFT EYE BASEE. QN HS- SS
`SSSAWSE: $88 & Rese is he $8 OSS FORES Fee is:
`SESS BS
`ESSES
`
`E:Sr. &Yihies, $8 E-3. & R8A&E F&R THE RSS EYE BASE Q8
`The 38's WESS;g A&E E. 's
`E:: geSF 8S 3R Se Rigi
`EYE. &S is fi:S$ 3 SSSS & Sier8 Shie Biase for
`SEESF EYSS
`E85: SE RSS SSS
`RE-S$88.8
`E SER
`
`888
`
`FRE
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 17 of 28 PageID 115
`
`U.S. Patent
`
`May 24, 2016
`
`Sheet 14 of 14
`
`US 9,349,183 B1
`
`SS TY
`
`SAS$8 & S. SSAGES. SASS8 SF-88 & Gifs: SSS $88. w
`F888 & '8& S$8 SF SEESS
`
`SS: AY::G. & a Shree 8RSS:CSA - SSEE-SO3& A8&SR, SE y - SS
`SES FSSES OF E is 8:SES & EiSEE
`SSR. E. RSFRSSESSESSESSESS
`f St
`y:E SESSSS C&SS CES: SE
`
`N SSS
`is Si SSS S&S & FSR. & R&S R&es A
`SSSSG A 3
`Sef of $88 &&ES SiS is 'g's, The Se's CF is geS &
`BSRSS
`SE ESSSSSSSSS
`E S $SE
`
`FGLIRE 3
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 18 of 28 PageID 116
`
`US 9,349,183 B1
`
`1.
`METHOD AND APPARATUS FOR THREE
`DIMENSIONAL VIEWING OF IMAGES
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`The present application is a continuation-in-part of U.S.
`patent application Ser. No. 1 1/941,578, filed Nov. 16, 2007
`which claims the benefit of U.S. Provisional Patent Applica
`tion No. 60/877,931, filed on Dec.28, 2006, the disclosures of
`which are incorporated herein by reference in their entirety
`
`10
`
`BACKGROUND
`
`2
`several CT slices present on the same or adjacent monitors).
`An example slice thickness of 1.25 mm would require
`approximately 500 slices in the viewing of the chest and
`abdomen. This process is inherently slow. Furthermore, in the
`example of a small pulmonary nodule which can be 2-3 mm,
`each slice must be carefully scrutinized. This is a very time
`consuming and labor intensive searching process.
`Systems designed to produce stereoscopic imagery have
`depended on the use of true Stereo pairs of images created by
`complex and costly optical systems. Attempts have been
`made to convert two-dimensional images to three-dimen
`sional images using shuttering with image shifting (e.g., U.S.
`Pat. No. 5,510,832). Such techniques have not produced
`three-dimensional imagery having sufficient quality for
`detailed medical examination purposes. Other transforma
`tions of Such pairs of images from one encoding method to
`another has been also been difficult and costly because they
`generally require depth information and computation. It has
`been necessary to generate Stereo pairs of images using two
`separate cameras or a single camera with special lenses. Such
`arrangements are costly and difficult to use.
`The current process of viewing cross-sections relies on the
`radiologist being able to mentally construct a holistic view.
`The radiologist must be able to piece together multiple slices,
`and rotate them in order to gain a representation of a portion
`of the image. Although several programs have been able to
`construct a three-dimensional representation within the data
`base, the user cannot see this representation in three-dimen
`sional. Several current programs can layer images from dif
`ferent slices onto the same 2D screen. Thus there is a
`component of an x, y, and Z axis in the viewingfield. However,
`the user cannot distinguish the distance in the y-axis (in the
`dimension projecting into and out of the image). Thus, the
`user does not have depth perception when viewing an image.
`In some cases using current technology, faint tissue anoma
`lies can be missed when looking at a whole series of multiple
`gray scale shaded images. The deficiency includes that such a
`faint anomaly would tend to blend in with other tissues of
`approximately equal grayscale.
`Embodiments of the invention significantly overcome Such
`deficiencies and provide mechanisms and techniques that
`provide a process for combining slices generated by medical
`imaging devices to create a Volume of interest and then pre
`senting this Volume in a three-dimensional representation to a
`Head Display Unit (HDU) so that the Radiologist/Medical
`Professional (R/MP, also referred to herein as a user) can
`obtain a holistic view of the patient. Key image processing
`techniques are applied which enable the user: to rotate and
`view the volume of interest from alternative viewpoints; to
`enable tissue subtraction to facilitate unobstructed viewing of
`a region of interest; to identify differing tissues with color
`schematics; and to Zoom in for optimal viewing.
`In a particular embodiment of a method for providing
`three-dimensional viewing of images by a user, the method
`includes selecting a Volume of interest from a collection of
`image slices and arranging the slices corresponding to the
`Volume of interest. The method also includes selecting an
`initial viewing angle of the slices, selecting a viewpoint for a
`left eye and selecting a viewpoint for a right eye. Additionally,
`the method includes displaying, in ahead display unit (HDU),
`animage for the left eye based on the initial viewing angle, the
`view point for the left eye and the volume of interest; and
`displaying, in the HDU, an image for the right eye based on
`the initial viewing angle, the view point for the right eye, and
`the Volume of interest and wherein the image for the left eye
`and the image for the right eye produce a three-dimensional
`image to the user.
`
`15
`
`30
`
`40
`
`Over the past several decades, the field of medical imaging
`has made many advances. In the 1950s, the principals of
`Magnetic Resonance (MR) were initially investigated. The
`fundamental premise of MR is that different materials reso
`nate at different magnetic field strengths. Magnetic Reso
`nance Imaging (MRI) was researched in the 1970s and tested
`clinically on patients in 1980. In 1984, MRI was approved by
`the Food and Drug Administration (FDA) for clinical use.
`Since then, this imaging modality has grown rapidly in popu
`larity.
`Computed Tomography (CT) imaging (also called CAT
`25
`scanning for Computed Axial Tomography) was invented in
`1972. Both gamma rays and X-rays were used in conjunction
`with a detector mounted on a special rotating frame to gen
`erate the image slices. Then a digital computer generates
`detailed cross sectional images. The original CT scan took
`hours to acquire a single slice of image data and more than 24
`hours to reconstruct this data into a single image. Today’s
`state-of-the-art CT systems can acquire a single image in less
`than a second and reconstruct the image instantly.
`In the 1970s, digital imaging techniques were imple
`35
`mented with the first clinical use and acceptance of the CT
`scanner. Analog to digital converters and computers were also
`adapted to conventional fluoroscopic image intensifier/TV
`systems in the 1970s as well. The key benefits of the digital
`technology include the fact that digital X-ray images can be
`enhanced and manipulated with computers, and the fact that
`digital images can be sent via a network to other workstations
`and computer monitors so that many people can share the
`information and assist in the diagnosis.
`Other recent developments include Positron Emission
`Tomography (PET), Single Photon Emission Computed
`Tomography (SPECT), and functional MRI (fMRI). PET is
`a nuclear medicine medical imaging technique which pro
`duces a three-dimensional image or map of functional pro
`cesses in the body. SPECT is a nuclear medicine tomographic
`imaging technique using gamma rays. It is very similar to
`conventional nuclear medicine planar imaging using a
`gamma camera. However, it is notable to provide true three
`dimensional information. This information is typically pre
`sented as cross-sectional slices through the patient, but can be
`freely reformatted or manipulated as required. Functional
`magnetic resonance imaging (fMRI) is the use of MRI to
`measure the hemodynamic response related to neural activity
`in the brain or spinal cord of humans or other animals. It is one
`of the most recently developed forms of neuroimaging.
`
`45
`
`50
`
`55
`
`60
`
`SUMMARY
`
`Conventional mechanisms such as those explained above
`suffer from a variety of deficiencies. One such deficiency is
`that when a radiologist views a CT scan, the limitations
`include viewing a single slice at a time (though there may be
`
`65
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 19 of 28 PageID 117
`
`US 9,349,183 B1
`
`4
`It is to be understood that the embodiments of the invention
`can be embodied strictly as a Software program, as Software
`and hardware, or as hardware and/or circuitry alone. Such as
`within a data communications device. The features of the
`invention, as explained herein, may be employed in data
`processing devices and/or Software systems for Such devices.
`Note that each of the different features, techniques, con
`figurations, etc. discussed in this disclosure can be executed
`independently or in combination. Accordingly, the present
`invention can be embodied and viewed in many different
`ways. Also, note that this Summary section herein does not
`specify every embodiment and/or incrementally novel aspect
`of the present disclosure or claimed invention. Instead, this
`Summary only provides a preliminary discussion of different
`embodiments and corresponding points of novelty over con
`ventional techniques. For additional details, elements, and/or
`possible perspectives (permutations) of the invention, the
`reader is directed to the Detailed Description section and
`corresponding figures of the present disclosure as further
`discussed below.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The foregoing will be apparent from the following more
`particular description of preferred embodiments of the inven
`tion, as illustrated in the accompanying drawings in which
`like reference characters refer to the same parts throughout
`the different views. The drawings are not necessarily to scale,
`emphasis instead being placed upon illustrating the principles
`of the invention.
`FIG. 1 is a block diagram a computer system that performs
`three-dimensional viewing of images in accordance with
`embodiments of the invention;
`FIG. 2 is a block diagram of the system indicating flow of
`data and the like;
`FIG. 3A is a diagram showing left and right eye viewing
`angles:
`FIG. 3B is a diagram showing a user-selectable conver
`gence point,
`FIG. 4 shows a volume of interest comprised of a series of
`slices selected by a user,
`FIG. 5 is a diagram showing a bird’s eye view demonstrat
`ing angle theta;
`FIG. 6 is a diagram demonstrating the Volumetric data in
`the grey cylinder,
`FIG. 7 is a diagram demonstrates the hypotenuses hyp1 and
`hyp2;
`FIG. 8: depicts a flow diagram of a particular embodiment
`of a method of providing three-dimensional viewing of
`images in accordance with embodiments of the invention;
`FIG.9 depicts a flow diagram of a particular embodiment
`of a method of viewing an alternative viewing angle;
`FIG. 10 depicts a flow diagram of a particular embodiment
`of a method of filtering an image:
`FIG. 11 depicts a flow diagram of a particular embodiment
`of a method of applying colors to an image;
`FIG. 12 depicts a flow diagram of a particular embodiment
`of a method of Zooming in on an image; and
`FIG. 13 depicts a flow diagram of a particular embodiment
`of a method of providing a moving image of a Volume of
`interest.
`
`DETAILED DESCRIPTION
`
`65
`
`Mechanisms and techniques that provide a process for
`combining slices generated by medical imaging devices to
`create a Volume of interest and then presenting this Volume in
`
`3
`Other embodiments include a computer readable medium
`having computer readable code thereon for providing three
`dimensional viewing of images by a user. The computer read
`able medium includes instructions for selecting a Volume of
`interest from a collection of image slices and instructions for
`arranging the slices corresponding to the Volume of interest.
`The computer readable medium also includes instructions for
`selecting an initial viewing angle of the slices, instructions for
`selecting a viewpoint for a left eye and instructions for select
`ing a viewpoint for a right eye. Additionally, the computer
`readable medium includes instructions for displaying, in a
`head display unit (HDU), an image for the left eye based on
`the initial viewing angle, the view point for the left eye and the
`Volume of interest; and instructions for displaying, in the
`HDU, an image for the right eye based on the initial viewing
`angle, the view point for the right eye, and the Volume of
`interest and wherein the image for the left eye and the image
`for the right eye produce a three-dimensional image to the
`USC
`Still other embodiments include a computerized device,
`configured to process all the method operations disclosed
`herein as embodiments of the invention. In such embodi
`ments, the computerized device includes a memory system, a
`processor, communications interface in an interconnection
`mechanism connecting these components. The memory sys
`tem is encoded with a process that provides three-dimen
`sional viewing of images by a user as explained herein that
`when performed (e.g. when executing) on the processor,
`operates as explained herein within the computerized device
`to perform all of the method embodiments and operations
`explained herein as embodiments of the invention. Thus any
`computerized device that performs or is programmed to per
`form processing explained herein is an embodiment of the
`invention.
`Other arrangements of embodiments of the invention that
`are disclosed herein include Software programs to perform
`the method embodiment steps and operations Summarized
`above and disclosed in detail below. More particularly, a
`computer program product is one embodiment that has a
`computer-readable medium including computer program
`logic encoded thereon that when performed in a computer
`ized device provides associated operations providing three
`dimensional viewing of images by a user as explained herein.
`The computer program logic, when executed on at least one
`processor with a computing system, causes the processor to
`perform the operations (e.g., the methods) indicated hereinas
`embodiments of the invention. Such arrangements of the
`invention are typically provided as Software, code and/or
`other data structures arranged or encoded on a computer
`readable medium such as an optical medium (e.g.,
`CD-ROM), floppy or hard disk or other a medium such as
`firmware or microcode in one or more ROM or RAM or
`PROM chips or as an Application Specific Integrated Circuit
`(ASIC) or as downloadable software images in one or more
`modules, shared libraries, etc. The software or firmware or
`other Such configurations can be installed onto a computer
`ized device to cause one or more processors in the comput
`erized device to perform the techniques explained herein as
`embodiments of the invention. Software processes that oper
`ate in a collection of computerized devices, such as in a group
`of data communications devices or other entities can also
`provide the system of the invention. The system of the inven
`tion can be distributed between many software processes on
`several data communications devices, or all processes could
`run on a small set of dedicated computers, or on one computer
`alone.
`
`5
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-2 Filed 09/16/20 Page 20 of 28 PageID 118
`
`US 9,349,183 B1
`
`5
`a three-dimensional representation to a Head Display Unit
`(HDU) so that the Radiologist/Medical Professional (R/MP.
`also referred to herein as a user) can obtain a holistic view is
`described. Key image processing techniques are applied
`which enable the user to rotate and view the volume of interest
`from alternative viewpoints; to enable tissue subtraction to
`facilitate unobstructed viewing of a region of interest, to
`identify differing tissues with color schematics; to Zoom in
`for optimal viewing; and to provide a moving image of a
`Volume of interest.
`Referring to FIG. 1, a top-level diagram of a system 10 is
`shown. System 10 includes an imaging device 12 Such as a
`Computed Tomography scan. Alternatively, the equipment
`could be an MRI, fMRI, or PET, etc. The imaging device 12
`is in communication with a digital recording device 14. The
`digital recording device records each slice of imagery
`together with the metadata Such as Subject, time, position of
`the gurney and position of the Electromagnetic (EM) trans
`mitter and receiver arrays.
`A general purpose processor 16 interacts with the digital
`recording device 14 based on inputs it receives from the user.
`Also in communication with general purpose processor 16 is
`printer/film developer 26. Printer/film developer 26 enables a
`hard copy of the viewed image to be captured and printed.
`As shown, the general purpose processor 16 generates an
`image for the right eye 18 and an image for the left eye 20. The
`image to right eye 18 is based on the parameters set by the
`user, and the computed image is sent to the right eye. The
`image to left eye 20 is based on the parameters set by the user,
`the computed image 20 is sent to the left eye. It should be
`noted that this viewpoint position is offset from the position
`used for the right eye image.
`The images are sent to a head display unit (HDU) 22 worn
`by the user 24. While a head display unit is shown and
`described, it should be appreciated that the present invention
`could also be utilized with other display units, including but
`not limited to, a display unit incorporating polarized lenses, a
`display unit wherein multiplexed images are viewed via shut
`tered lenses, virtual reality displays having a display unit with
`unique left and right eye pixel displays, and other types of
`40
`three-dimensional (3D) displays as would be known to one of
`reasonable skill in the art.
`The head display unit 22 displays image 18 to the right eye.
`Note that multiple slices have been Stacked, creating a Volu
`metric mass so that the image seen by the right eye is the
`volume of interest selected by the user. The head display unit
`22 also displays image 20 to the left eye similar to the manner
`of which the right image was displayed. This produces a
`three-dimensional image to the user 24.
`Referring now to FIG. 2, a block diagram of the system 50
`comprising one particular embodiment of the invention is
`shown. The diagram indicates flow of data, imagery, and the
`like according to the legend. Medical imaging device 12
`utilizes EM energy that emanates from the device 12 and is
`also received by the device 12. The medical imaging device
`12 receives imaging commands from the controller 52 and
`provides imagery data to the data recording. The Medical
`device controller 52 provides control commands to the medi
`cal imaging device 12. A power Supply 54 provides power to
`the medical imaging device 12.
`Digital recording device 14 records each of the slices of
`data collected by the medical imaging device 12, and may
`also be used to record medical records meta data. Digital
`recording device 14 interacts/exchanges data with the Gen
`eral purpose processor 16.
`65
`General purpose processor 16 interacts with the digital
`recording device 14 based on inputs it receives from the user
`
`50
`
`6
`through the Graphical User Interface (GUI) 58. The GUI 58
`allows interaction between the user with the general purpose
`processor 16 to initiate functions such as Zoom, rotate, filter
`tissue, and apply color schematics.
`The head display unit 22 includes a left head display unit
`