Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 1 of 32 PageID 159
`
`
`
`Exhibit 4
`U.S. Patent No. 9,980,691
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 2 of 32 PageID 160
`
`THIRUTTUULIHUTU
`
`US009980691B2
`
`( 12 ) United States Patent
`Douglas et al .
`
`( 10 ) Patent No . :
`( 45 ) Date of Patent :
`
`US 9 , 980 , 691 B2
`May 29 , 2018
`
`( 71 )
`
`( * ) Notice :
`
`( 54 ) METHOD AND APPARATUS FOR THREE
`DIMENSIONAL VIEWING OF IMAGES
`Applicants : David Byron Douglas , Winter Park , FL
`( US ) ; Robert E . Douglas , Winter Park ,
`FL ( US )
`( 72 ) Inventors : David Byron Douglas , Winter Park , FL
`( US ) ; Robert E . Douglas , Winter Park ,
`FL ( US )
`Subject to any disclaimer , the term of this
`patent is extended or adjusted under 35
`U . S . C . 154 ( b ) by 0 days . days .
`( 21 ) Appl . No . : 14 / 877 , 442
`( 22 ) Filed :
`Oct . 7 , 2015
`Prior Publication Data
`( 65 )
`US 2016 / 0026266 A1
`Jan . 28 , 2016
`Related U . S . Application Data
`Continuation - in - part of application No . 12 / 176 , 569 ,
`filed on Jul . 21 , 2008 , now Pat . No . 9 , 349 , 183 , which
`( Continued )
`
`( 63 )
`
`( 51 ) Int . Cl .
`G06K 9 / 00
`A61B 6 / 00
`
`( 52 )
`
`( 2006 . 01 )
`( 2006 . 01 )
`( Continued )
`U . S . CI .
`CPC . . . . . . . . . . . . . . A61B 6 / 466 ( 2013 . 01 ) ; A61B 5 / 055
`( 2013 . 01 ) ; A61B 6 / 022 ( 2013 . 01 ) ; A61B 6 / 462
`( 2013 . 01 ) ; GO2B 27 / 0172 ( 2013 . 01 ) ; G06T
`19 / 00 ( 2013 . 01 ) ; A61B 6 / 467 ( 2013 . 01 ) ; A61B
`6 / 501 ( 2013 . 01 ) ; A61B 6 / 503 ( 2013 . 01 ) ;
`( Continued )
`( 58 ) Field of Classification Search
`CPC . . . . . . . . A61B 5 / 055 ; A61B 6 / 022 ; A61B 6 / 462 ;
`
`( 56 )
`
`A61B 6 / 466 ; A61B 6 / 467 ; A61B 6 / 501 ;
`A61B 6 / 503 ; A61B 6 / 504 ; G02B
`27 / 0172 ; GO2B 2027 / 0134 ; GO2B
`2027 / 0187 ; GO2B 27 / 0093 ; G02B
`27 / 2207 ; GO6T 19 / 00
`See application file for complete search history .
`References Cited
`U . S . PATENT DOCUMENTS
`6 , 066 , 095 A *
`5 / 2000 Morsy . . . . . . . . . . . . . . . . . GO6T 7 / 0016
`600 / 438
`6 , 100 , 862 A
`*
`8 / 2000 Sullivan . . . . . . . . . . . . . . . G06T 15 / 00
`345 / 424
`( Continued )
`OTHER PUBLICATIONS
`Jason et al “ Pictorial Depth cues for outdoor Augmented Reality ” ,
`IEEE 2005 . *
`Primary Examiner — Amara Abdi
`( 74 ) Attorney , Agent , or Firm — Anderson Gorecki LLP
`( 57 )
`ABSTRACT
`A method , apparatus and computer program product for
`three - dimensional viewing of images is presented . Embodi
`ments of the invention provide a process for combining
`slices generated by medical imaging devices to create a
`volume of interest and then present this volume in a three
`dimensional representation to a head display unit so that the
`user can obtain a holistic view of the patient . Key image
`processing techniques are applied which enable the user to
`rotate and view the volume of interest from alternative
`viewpoints ; to enable tissue subtraction to facilitate unob
`structed viewing of a region of interest ; to identify differing
`tissues with color schematics ; to zoom in for optimal view
`ing ; and to view a moving image of a volume of interest .
`21 Claims , 17 Drawing Sheets
`
`12
`
`. . .
`
`TWITTER
`
`Digital
`Recording
`Bes
`
`General
`Purpose
`Processor
`L - - - - - - - -
`- - - - -
`
`24
`
`???? ta kgyz
`
`Image to . Leye
`
`.
`
`.
`
`16
`
`s 26
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 3 of 32 PageID 161
`
`US 9 , 980 , 691 B2
`Page 2
`
`( 51 )
`
`Related U . S . Application Data
`is a continuation - in - part of application No . 11 / 941 ,
`578 , filed on Nov . 16 , 2007 , now Pat . No . 8 , 384 , 771 .
`( 60 ) Provisional application No . 60 / 877 , 931 , filed on Dec .
`28 , 2006 .
`Int . Cl .
`G06T 19 / 00
`A61B 6 / 02
`A61B 5 / 055
`GO2B 27 / 00
`GO2B 27 / 22
`GO2B 27 / 01
`( 52 ) U . S . CI .
`CPC . . . . . . . . . . A61B 6 / 504 ( 2013 . 01 ) ; GO2B 27 / 0093
`( 2013 . 01 ) ; GO2B 27 / 2207 ( 2013 . 01 ) ; G02B
`2027 / 0134 ( 2013 . 01 ) ; G02B 2027 / 0187
`( 2013 . 01 )
`
`( 2011 . 01 )
`( 2006 . 01 )
`( 2006 . 01 )
`( 2006 . 01 )
`( 2018 . 01 )
`( 2006 . 01 )
`
`( 56 )
`
`References Cited
`U . S . PATENT DOCUMENTS
`6 , 692 , 441 B1 *
`2 / 2004 Poland . . . . . . . . . . . . . . . . . . . . .
`7 , 193 , 626 B2 *
`3 / 2007 Otani . . . . . . . . . . . .
`
`GO6T 15 / 08
`128 / 916
`GO1C 11 / 06
`345 / 418
`
`8 / 2002 Kelts . . . . . . . . . . . . . . . . . . . . G06F 3 / 0481
`2002 / 0112237 A1 *
`725 / 39
`2004 / 0204644 Al * 10 / 2004 Tsougarakis . . . . . . . . . . . GO1R 33 / 56
`600 / 410
`2005 / 0062684 Al *
`3 / 2005 Geng . . . . . . . . . . . . . . . . . . . . G06F 3 / 0346
`345 / 32
`2005 / 0065423 Al *
`3 / 2005 Owen . . . . . . . . . . . . . . . . . . . . G16H 40 / 63
`600 / 407
`2006 / 0033992 A1 *
`2 / 2006 Solomon . . . . . . . . . . . . . . GO2B 27 / 017
`359 / 462
`2007 / 0115204 A1 *
`5 / 2007 Budz . . . . . . . . . . . . . . . . G06T 15 / 08
`345 / 6
`2007 / 0116357 A1 *
`5 / 2007 Dewaele . . . . . . . . . . . . GO6K 9 / 3233
`382 / 173
`2007 / 0165927 A1 *
`7 / 2007 Muradyan . . . . . . . . . . . . . . . GO6K 9 / 40
`382 / 128
`2007 / 0279435 A1 * 12 / 2007 Ng . . . . . . . . . . . . . . . . . . . . G06F 3 / 011
`345 / 624
`2008 / 0033240 A1 *
`2 / 2008 Hoffman . . . . . . . . . . . . . A61B 90 / 36
`600 / 109
`2008 / 0037843 A1 *
`2 / 2008 Fu . . . . . . . . . . . . . . . . . . . . . GO6T 15 / 08
`382 / 128
`2008 / 0100612 A1 *
`5 / 2008 Dastmalchi . . . . . . . . . . . . A61B 3 / 102
`345 / 418
`2009 / 0324052 A1 * 12 / 2009 Nowinski . . . . . . . . . . . . . . A61B 6 / 504
`382 / 134
`
`* cited by examiner
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 4 of 32 PageID 162
`
`U . S . Patent
`
`May 29 , 2018
`
`Sheet 1 of 17
`
`US 9 , 980 , 691 B2
`
`?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
`
`.
`
`. .
`
`.
`
`. . ' '
`
`- . . . ' . ' . ' .
`
`min
`
`YYYYYYYYYYYYYYYYYY
`
`wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww
`
`* * * * *
`
`* 8
`
`??s to Keys
`
`image to Leye
`
`Digital
`Recording
`Bevice
`
`General
`Purpose
`Processor
`
`FIGURE 1
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 5 of 32 PageID 163
`
`U . S . Patent
`
`U . S . Patent
`
`May 29 , 2018
`
`Sheet 2 of 17
`
`US 9 , 980 , 691 B2
`
`133
`
`Medical
`Imaging
`Device
`
`54
`
`Power
`supply
`
`Medical
`Device
`Controller
`
`Digital
`Recording
`Device
`
`General
`Purpose
`Processor
`26
`
`56
`
`Power
`Supply
`
`GUI
`
`18 , 20
`KDUL
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`NY
`
`Power Supply
`3D image
`Printer /
`Film
`Data Flow
`Developer
`EM energy
`
`FIGURE 2
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 6 of 32 PageID 164
`
`tent
`
`May 29 , 2018
`
`Sheet 3 of 17
`
`US 9 , 980 , 691 B2
`
`ben 102
`
`106
`
`ting
`
`mm 108
`
`112
`
`1044
`
`104
`
`1101
`om
`
`bom
`
`LEVP
`
`REVO
`
`Note : Diagram not to
`Scale
`
`FIGURE 3A
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 7 of 32 PageID 165
`
`U . S . Patent
`
`May 29 , 2018
`
`Sheet 4 of 17
`
`US 9 , 980 , 691 B2
`
`LEVER
`
`REVD
`
`+ * + - + * + - + * * * *
`
`* *
`
`mwanamannannig
`
`Note : Diagram not to
`scale
`
`FIGURE 3B
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 8 of 32 PageID 166
`
`U . S . Patent
`
`May 29 , 2018
`
`Sheet 5 of 17
`
`US 9 , 980 , 691 B2
`
`mwen 26a
`
`Column theta = + 20 deg
`Right HDU
`Colunm theta =
`Column theta = - 20 deg
`
`- 26b
`
`Column theta = + 20 deg
`Left HDU
`
`
`
`Column theta ! Column - - 20 deg
`
`theta = 0
`
`
`
`not to scale Note : Diagram
`
`FIGURE 4
`
`A
`
`1 .
`
`154
`
`
`
`Row alpha - max * nominally + 15 deg
`
`
`
`
`
`alpha = 0 ; LEVP and Row slice have same
`Z value
`
`
`
`Row alpha - min * nominally - 15 deg
`
`- 152
`
`1L
`
`1992 - - - -
`
`
`
`
`
`alpha = 0 ; REVP and Row slice have same
`
`Row alpha - min * nominally - 15 deg
`Row alpha - max * nominally + 15 deg
`
`Z value
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 9 of 32 PageID 167
`
`U . S . Patent
`
`May 29 , 2018
`
`Sheet 6 of 17
`
`US 9 , 980 , 691 B2
`
`5 cm
`
`102 102
`
`35 cm
`
`Posterior
`
`Laera
`
`Media
`
`Amenor
`
`LEVP
`
`104
`
`FIGURE 5
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 10 of 32 PageID 168
`
`U . S . Patent
`
`May 29 , 2018
`
`Sheet 7 of 17
`
`US 9 , 980 , 691 B2
`
`theta
`
`- 104
`ARTIDO
`
`LEVP de
`mart ap
`
`W
`
`* * *
`
`Anterior
`
`Lateral
`
`Medial
`
`102
`Posterior
`
`Distally
`
`.
`
`Proximally
`
`Art
`
`* *
`
`Figure 6
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 11 of 32 PageID 169
`
`U . S . Patent
`
`May 29 , 2018
`
`Sheet 8 of 17
`
`US 9 , 980 , 691 B2
`
`This is the distance in the
`y - direction from the LEVP
`to the y - coordinate of the
`voxel of interest
`LEVP
`
`104 annoncen
`
`theta
`
`waarnemendur
`alpha Hyp1
`
`This is the distance in the x
`direction from LEVP ' s point looking
`straight forward to the x - coordinate
`of the voxel on interest
`
`This is the distance in the
`Z - direction from the LEVP ' s point to the
`z corrdinate of the voxel of interest
`
`hyp2
`
`" Voxel of interest
`
`FIGURE 7
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 12 of 32 PageID 170
`
`U . S . Patent
`
`May 29 , 2018
`
`Sheet 9 of 17
`
`US 9 , 980 , 691 B2
`
`400
`
`SELECTING A VOLUME OF INTEREST FROM A COLLECTION OF IMAGE
`JUM
`SLICES
`
`ARRANGING THE SLICES CORRESPONDING TO THE VOLUME OF
`INTEREST
`
`SELECTING AN INITIAL VIEWING ANGLE OF THE SLICES
`
`???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
`
`SELECTING A VIEW POINT FOR A LEFT EYE
`
`SELECTING A VIEW POINT FOR A RIGHT EYE
`VE
`
`DISPLAYING , IN A HEAD DISPLAY UNIT ( HDU ) , AN IMAGE FOR THE LEFT
`EYE BASED ON THE INITIAL VIEWING ANGLE , THE VIEW POINT FOR
`SAID LEFT EYE AND THE VOLUME OF INTEREST
`
`402
`
`404
`
`406
`
`m . 408
`
`mm 410
`
`412
`
`-
`
`414
`
`w 416
`
`DISPLAYING , IN THE HDU , AN IMAGE FOR THE RIGHT EYE BASED ON
`THE INITIAL VIEWING ANGLE , THE VIEW POINT FOR THE RIGHT EYE ,
`AND THE VOLUME OF INTEREST AND WHEREIN THE IMAGE FOR THE
`LEFT EYE AND THE IMAGE FOR SAID RIGHT EYE PRODUCE A THREE
`DIMENSIONAL IMAGE TO THE USER
`?????
`?????
`
`
` ????? ????? ????
`?????
`?????
`??????
`
`??????
`
`
`
`????? ???????
`
`??
`
`USER SEELCT IMAGE ( S ) FOR STORAGE AND POSSIBLE RETREIVAL
`FOR FUTURE REFERENCE
`
`FIGURE 8
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 13 of 32 PageID 171
`
`U . S . Patent
`
`May 29 , 2018
`
`Sheet 10 of 17
`
`US 9 , 980 , 691 B2
`
`450 -
`
`USER SELECTS ALTERNATIVE VIEWING ANGLE
`
`REORIENTING A VOLUME OF INTEREST IN ACCORDANCE WITH THE
`ALTERNATE VIEWING ANGLE
`
`DISPLAYING , IN THE HOU , AN IMAGE FOR THE LEFT EYE BASED ON THE
`ALTERNATE VIEWING ANGLE , THE VIEW POINT FOR THE LEFT EYE AND
`THE VOLUME OF INTEREST
`
`DISPLAYING , IN THE HDU , AN IMAGE FOR THE RIGHT EYE BASED ON
`THE ALTERNATE VIEWING ANGLE , THE VIEW POINT FOR THE RIGHT
`EYE , AND THE VOLUME OF INTEREST AND WHEREIN THE IMAGE FOR
`THE LEFT EYE AND THE IMAGE FOR THE RIGHT EYE PRODUCE AN
`ALTERNATE THREE - DIMENSIONAL IMAGE TO THE USER
`
`FIGURE 9
`
`mw 452
`
`www 456
`
`amanna 458
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 14 of 32 PageID 172
`
`U . S . Patent
`
`May 29 , 2018
`
`Sheet 11 of 17
`
`US 9 , 980 , 691 B2
`
`500
`
`g
`
`SELECTING ITEMS OF THE IMAGE TO BE SUBTRACTED FROM THE
`IMAGE TO PRODUCE A FILTERED IMAGE
`
`DISPLAYING , IN THE HDU , A FILTERED IMAGE FOR THE LEFT EYE
`BASED ON THE INITIAL VIEWING ANGLE , THE VIEW POINT FOR THE
`EYE AND THE VOLUME OF INTEREST
`
`DISPLAYING , IN THE HDU , A FILTERED IMAGE FOR THE RIGHT EYE
`BASED ON THE INITIAL VIEWING ANGLE , THE VIEW POINT FOR THE
`RIGHT EYE , AND THE VOLUME OF INTEREST AND WHEREIN THE
`FILTERED IMAGE FOR THE LEFT EYE AND THE FILTERED IMAGE FOR
`THE RIGHT EYE PRODUCE A FILTERED THREE - DIMENSIONAL IMAGE TO
`THE USER
`
`FIGURE 10
`
`van 502
`
`Koinonimninonio 504
`
`mawan 506
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 15 of 32 PageID 173
`
`U . S . Patent
`
`May 29 , 2018
`
`Sheet 12 of 17
`
`US 9 , 980 , 691 B2
`
`550 man
`
`SORTING VOXELS OF THE ITEMS BY A PROPERTY OF THE VOXEL
`
`APPLYING COLORS TO GROUPS OF SORTED VOXELS TO OBTAINA
`COLORED IMAGE
`
`DISPLAYING , IN THE HDU , A COLORED IMAGE FOR THE LEFT EYE
`BASED ON THE INITIAL VIEWING ANGLE , THE VIEW POINT FOR THE
`LEFT EYE AND THE VOLUME OF INTEREST
`
`DISPLAYING , IN THE HDU , A COLORED IMAGE FOR THE RIGHT EYE
`BASED ON THE INITIAL VIEWING ANGLE , THE VIEW POINT FOR THE
`RIGHT EYE , AND THE VOLUME OF INTEREST AND WHEREIN THE
`COLORED IMAGE FOR SAID LEFT EYE AND SAID COLORED IMAGE FOR
`THE RIGHT EYE PRODUCE A COLORED THREE - DIMENSIONAL IMAGE TO
`THE USER
`
`FIGURE 11
`
`552
`
`po www 554 554
`
`- 556
`556
`
`mm 558
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 16 of 32 PageID 174
`
`U . S . Patent
`
`May 29 , 2018
`
`Sheet 13 of 17
`
`US 9 , 980 , 691 B2
`
`600 -
`
`SELECTING A NEW VIEW POINT FOR A LEFT EYE
`
`SELECTING A NEW VIEW POINT FOR A RIGHT EYE
`
`DISPLAYING , IN THE HOU , AN IMAGE FOR THE LEFT EYE BASED ON THE
`INITIAL VIEWING ANGLE , THE NEW VIEW POINT FOR THE LEFT EYE AND
`THE VOLUME OF INTEREST
`
`
`
` ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?????
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
` ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ? ?
`
`
`
`
`
`
`
`
`
`- 604
`
`en 606
`
`munun 608
`
`??
`
`. ??
`
`.
`
`: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : .
`
`:
`
`: :
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`DISPLAYING , IN THE HDU , AN IMAGE FOR THE RIGHT EYE BASED ON
`THE INITIAL VIEWING ANGLE , THE NEW VIEW POINT FOR THE RIGHT
`EYE , AND THE VOLUME OF INTEREST AND WHEREIN THE IMAGE FOR
`THE LEFT EYE AND THE IMAGE FOR THE RIGHT EYE PRODUCE A
`THREE - DIMENSIONAL IMAGE TO THE USER
`FIGURE 12
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 17 of 32 PageID 175
`
`U . S . Patent
`
`May 29 , 2018
`
`Sheet 14 of 17
`
`US 9 , 980 , 691 B2
`
`650 -
`
`
`
`- ???????
`
`
`
`
`
` ????? - ???? - ????? - ??????
`
`
`
`
`
`
`
`
`
`
`
` ????? - ?? -
`
`
`
`
`
` ??? ????? - ???? - - ??? -
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`????? -
`
`? -
`
`?
`
`
`
`????? -
`
`
`
` ????? - ?? -
`
`
`
`
`
` ????? - ??? -
`
`
`
`
`
`
`
`????? -
`
`? -
`
`?
`
`
`
`????? -
`
`
`
` ????? - ?? -
`
`
`
`
`
` ? ?? -
`
`
`
`????? -
`
`
`
` ???? ????? - - ???? - ????? - ??? - ??? -
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
` ????? - ?? -
`
`
`
`
`
` ????? - ????? - ??? -
`
`
`
`
`
`
`
`
`
`
`
` ????? - ??? -
`
`
`
`
`
`
`
` ????? - ?? -
`
`
`
`
`
` ????? - ?? -
`
`
`
`
`
`???? -
`
`652
`
`STACKING A SET OF IMAGES TAKEN WITHIN A GIVEN TIME INTERVAL
`TO FORM A VOLUME OF INTEREST
`
`DISPLAYING , IN A THREE DIMENSIONAL STEREOSCOPIC MANNER , THE
`SET OF IMAGES OF THE VOLUME OF INTEREST IN A CONSECUTIVE
`MANNER TO REPRESENT THE VOLUME OF INTEREST AND HOW THE
`VOLUME OF INTEREST CHANGES OVER TIME
`
`pe
`
`654
`
`FORMING ALOOP OF THE IMAGES SUCH THAT AFTER A LAST IMAGE
`OF SET OF IMAGES IS DISPLAYED , THE SET OF IMAGES ARE
`DISPLAYED AGAIN FROM THE BEGINNING IMAGE TO THE LAST IMAGE
`
`656
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 18 of 32 PageID 176
`
`U . S . Patent
`
`May 29 , 2018
`
`Sheet 15 of 17
`
`US 9 , 980 , 691 B2
`
`700
`
`192222222222222222222222222
`
`S EDERT
`
`7777
`
`1 . : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : .
`
`: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : .
`
`: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
`
`:
`
`: : : : : : : : : : : : : : : :
`
`: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :
`
`: . : . : . : . : . . . . : .
`
`: : : : : : :
`
`. .
`
`: : : : : : :
`. . .
`
`: : : : : :
`
`.
`
`.
`
`.
`
`V
`
`Figure 14
`
`

`

`now . - - - - - - -
`
`atent
`
`US 9 , 980 , 691 B2
`
`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 19 of 32 PageID 177
`
`May 29 , 2018
`
`Sheet 16 of 17
`
`710 mm
`
`Figure 15
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 20 of 32 PageID 178
`
`U . S . Patent
`
`May 29 , 2018
`
`Sheet 17 of 17
`
`US 9 , 980 , 691 B2
`
`120
`
`722
`
`Figure 16
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 21 of 32 PageID 179
`
`US 9 , 980 , 691 B2
`
`that when a radiologist views a CT scan , the limitations
`METHOD AND APPARATUS FOR THREE
`DIMENSIONAL VIEWING OF IMAGES
`include viewing a single slice at a time ( though there may be
`several CT slices present on the same or adjacent monitors ) .
`CROSS REFERENCE TO RELATED
`An example slice thickness of 1 . 25 mm would require
`APPLICATIONS
`5 approximately 500 slices in the viewing of the chest and
`abdomen . This process is inherently slow . Furthermore , in
`The present application is a continuation - in - part of U . S .
`the example of a small pulmonary nodule which can be 2 - 3
`patent application Ser . No . 12 / 176 , 569 , filed on Jul . 21 , 2008 mm , each slice must be carefully scrutinized . This is a very
`which is a continuation - in - part of U . S . patent application
`time - consuming and labor intensive searching process .
`Ser . No . 11 / 941 , 578 , filed Nov . 16 , 2007 which claims the
`Systems designed to produce stereoscopic imagery have
`benefit of U . S . Provisional Patent Application No . 60 / 877 , 10
`depended on the use of true stereo pairs of images created by
`931 , filed on Dec . 28 , 2006 , the disclosures of which are
`complex and costly optical systems . Attempts have been
`incorporated herein by reference in their entirety .
`made to convert two - dimensional images to three - dimen
`sional images using shuttering with image shifting ( e . g . ,
`BACKGROUND
`15 U . S . Pat . No . 5 , 510 , 832 ) . Such techniques have not pro
`Over the past several decades , the field of medical imag
`duced three - dimensional imagery having sufficient quality
`ing has made many advances . In the 1950s , the principals of
`for detailed medical examination purposes . Other transfor
`Magnetic Resonance ( MR ) were initially investigated . The
`mations of such pairs of images from one encoding method
`fundamental premise of MR is that different materials reso -
`to another has been also been difficult and costly because
`nate at different magnetic field strengths . Magnetic Reso - 20 they generally require depth information and computation . It
`nance Imaging ( MRI ) was researched in the 1970s and
`has been necessary to generate stereo pairs of images using
`tested clinically on patients in 1980 . In 1984 , MRI was
`two separate cameras or a single camera with special lenses .
`approved by the Food and Drug Administration ( FDA ) for
`Such arrangements are costly and difficult to use .
`clinical use . Since then , this imaging modality has grown
`The current process of viewing cross - sections relies on
`rapidly in popularity .
`25 the radiologist being able to mentally construct a holistic
`Computed Tomography ( CT ) imaging ( also called CAT
`view . The radiologist must be able to piece together multiple
`scanning for Computed Axial Tomography ) was invented in
`slices , and rotate them in order to gain a representation of a
`1972 . Both gamma rays and x - rays were used in conjunction
`portion of the image . Although several programs have been
`with a detector mounted on a special rotating frame to
`able to construct a three - dimensional representation within
`generate the image slices . Then a digital computer generates 30 the database , the user cannot see this representation in
`detailed cross sectional images . The original CT scan took
`three - dimensional . Several current programs can layer
`hours to acquire a single slice of image data and more than
`images from different slices onto the same 2D screen . Thus
`24 hours to reconstruct this data into a single image . Today ' s
`there is a component of an x , y , and z axis in the viewing
`state - of - the - art CT systems can acquire a single image in
`field . However , the user cannot distinguish the distance in
`less than a second and reconstruct the image instantly .
`35 the y - axis ( in the dimension projecting into and out of the
`In the 1970s , digital imaging techniques were imple -
`image ) . Thus , the user does not have depth perception when
`mented with the first clinical use and acceptance of the CT
`viewing an image .
`scanner . Analog to digital converters and computers were
`In some cases using current technology , faint tissue
`also adapted to conventional fluoroscopic image intensifier /
`anomalies can be missed when looking at a whole series of
`TV systems in the 1970s as well . The key benefits of the 40 multiple gray scale shaded images . The deficiency includes
`digital technology include the fact that digital X - ray images
`that such a faint anomaly would tend to blend in with other
`can be enhanced and manipulated with computers , and the
`tissues of approximately equal grayscale .
`fact that digital images can be sent via a network to other
`Embodiments of the invention significantly overcome
`workstations and computer monitors so that many people
`such deficiencies and provide mechanisms and techniques
`can share the information and assist in the diagnosis .
`45 that provide a process for combining slices generated by
`Other recent developments include Positron Emission
`medical imaging devices to create a volume of interest and
`Tomography ( PET ) , Single Photon Emission Computed
`then presenting this volume in a three - dimensional repre
`Tomography ( SPECT ) , and functional MRI ( f - MRI ) . PET is
`sentation to a Head Display Unit ( HDU ) so that the Radi
`a nuclear medicine medical imaging technique which pro -
`ologist / Medical Professional ( R / MP , also referred to herein
`duces a three - dimensional image or map of functional 50 as a user ) can obtain a holistic view of the patient . Key
`processes in the body . SPECT is a nuclear medicine tomo
`image processing techniques are applied which enable the
`graphic imaging technique using gamma rays . It is very
`user : to rotate and view the volume of interest from alter
`similar to conventional nuclear medicine planar imaging
`native viewpoints ; to enable tissue subtraction to facilitate
`using a gamma camera . However , it is not able to provide
`unobstructed viewing of a region of interest ; to identify
`true three - dimensional information . This information is typi - 55 differing tissues with color schematics ; and to zoom in for
`cally presented as cross - sectional slices through the patient ,
`optimal viewing .
`but can be freely reformatted or manipulated as required .
`In a particular embodiment of a method for providing
`Functional magnetic resonance imaging ( f - MRI ) is the use
`three - dimensional viewing of images by a user , the method
`of MRI to measure the hemodynamic response related to
`includes selecting a volume of interest from a collection of
`neural activity in the brain or spinal cord of humans or other 60 image slices and arranging the slices corresponding to the
`animals . It is one of the most recently developed forms of
`volume of interest . The method also includes selecting an
`initial viewing angle of the slices , selecting a viewpoint for
`neuroimaging .
`a left eye and selecting a viewpoint for a right eye . Addi
`tionally , the method includes displaying , in a head display
`65 unit ( HDU ) , an image for the left eye based on the initial
`viewing angle , the view point for the left eye and the volume
`of interest ; and displaying , in the HDU , an image for the
`
`SUMMARY
`Conventional mechanisms such as those explained above
`suffer from a variety of deficiencies . One such deficiency is
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 22 of 32 PageID 180
`
`US 9 , 980 , 691 B2
`
`between many software processes on several data commu
`right eye based on the initial viewing angle , the view point
`nications devices , or all processes could run on a small set
`for the right eye , and the volume of interest and wherein the
`of dedicated computers , or on one computer alone .
`image for the left eye and the image for the right eye produce
`a three - dimensional image to the user .
`It is to be understood that the embodiments of the inven
`Other embodiments include a computer readable medium 5 tion can be embodied strictly as a software program , as
`having computer readable code thereon for providing three -
`software and hardware , or as hardware and / or circuitry
`dimensional viewing of images by a user . The computer
`alone , such as within a data communications device . The
`readable medium includes instructions for selecting a vol
`features of the invention , as explained herein , may be
`ume of interest from a collection of image slices and
`employed in data processing devices and / or software sys
`instructions for arranging the slices corresponding to the 10 tems for such devices .
`volume of interest . The computer readable medium also
`Note that each of the different features , techniques , con
`includes instructions for selecting an initial viewing angle of
`figurations , etc . discussed in this disclosure can be executed
`the slices , instructions for selecting a viewpoint for a left eye
`independently or in combination . Accordingly , the present
`and instructions for selecting a viewpoint for a right eye .
`invention can be embodied and viewed in many different
`Additionally , the computer readable medium includes 15 ways . Also , note that this summary section herein does not
`instructions for displaying , in a head display unit ( HDU ) , an
`specify every embodiment and / or incrementally novel
`image for the left eye based on the initial viewing angle , the
`aspect of the present disclosure or claimed invention .
`view point for the left eye and the volume of interest ; and
`Instead , this summary only provides a preliminary discus
`instructions for displaying , in the HDU , an image for the
`sion of different embodiments and corresponding points of
`right eye based on the initial viewing angle , the view point 20 novelty over conventional techniques . For additional details ,
`for the right eye , and the volume of interest and wherein the
`elements , and / or possible perspectives ( permutations ) of the
`image for the left eye and the image for the right eye produce
`invention , the reader is directed to the Detailed Description
`a three - dimensional image to the user .
`section and corresponding figures of the present disclosure
`Still other embodiments include a computerized device ,
`as further discussed below .
`configured to process all the method operations disclosed 25
`BRIEF DESCRIPTION OF THE DRAWINGS
`herein as embodiments of the invention . In such embodi
`ments , the computerized device includes a memory system ,
`a processor , communications interface in an interconnection
`The foregoing will be apparent from the following more
`mechanism connecting these components . The memory sys -
`particular description of preferred embodiments of the
`tem is encoded with a process that provides three - dimen -
`30 invention , as illustrated in the accompanying drawings in
`sional viewing of images by a user as explained herein that
`which like reference characters refer to the same parts
`when performed ( e . g . when executing ) on the processor ,
`throughout the different views . The drawings are not nec
`operates as explained herein within the computerized device
`essarily to scale , emphasis instead being placed upon illus
`to perform all of the method embodiments and operations
`trating the principles of the invention .
`explained herein as embodiments of the invention . Thus any 35
`FIG . 1 is a block diagram a computer system that per
`computerized device that performs or is programmed to
`forms three - dimensional viewing of images in accordance
`perform processing explained herein is an embodiment of
`with embodiments of the invention ;
`FIG . 2 is a block diagram of the system indicating flow of
`the invention .
`Other arrangements of embodiments of the invention that
`data and the like ;
`are disclosed herein include software programs to perform 40
`FIG . 3A is a diagram showing left and right eye viewing
`the method embodiment steps and operations summarized
`angles ;
`above and disclosed in detail below . More particularly , a
`FIG . 3B is a diagram showing a user - selectable conver
`computer program product is one embodiment that has a
`gence point ;
`computer - readable medium including computer program
`FIG . 4 shows a volume of interest comprised of a series
`logic encoded thereon that when performed in a computer - 45 of slices selected by a user ;
`ized device provides associated operations providing three -
`FIG . 5 is a diagram showing a bird ' s eye view demon
`dimensional viewing of images by a user as explained
`strating angle theta ;
`herein . The computer program logic , when executed on at
`FIG . 6 is a diagram demonstrating the volumetric data in
`least one processor with a computing system , causes the
`the grey cylinder ;
`processor to perform the operations ( e . g . , the methods ) 50
`FIG . 7 is a diagram demonstrates the hypotenuses hyp1
`indicated herein as embodiments of the invention . Such
`arrangements of the invention are typically provided as
`FIG . 8 : depicts a flow diagram of a particular embodiment
`software , code and / or other data structures arranged or
`of a method of providing three - dimensional viewing of
`encoded on a computer readable medium such as an optical
`images in accordance with embodiments of the invention ;
`medium ( e . g . , CD - ROM ) , floppy or hard disk or other a
`FIG . 9 depicts a flow diagram of a particular embodiment
`medium such as firmware or microcode in one or more ROM
`of a method of viewing an alternative viewing angle ;
`or RAM or PROM chips or as an Application Specific
`FIG . 10 depicts a flow diagram of a particular embodi
`Integrated Circuit ( ASIC ) or as downloadable software
`ment of a method of filtering an image ;
`images in one or more modules , shared libraries , etc . The
`FIG . 11 depicts a flow diagram of a particular embodi
`software or firmware or other such configurations can be 60 ment of a method of applying colors to an image ;
`installed onto a computerized device to cause one or more
`FIG . 12 depicts a flow diagram of a particular embodi
`processors in the computerized device to perform the tech -
`ment of a method of zooming in on an image ;
`niques explained herein as embodiments of the invention .
`FIG . 13 depicts a flow diagram of a particular embodi
`Software processes that operate in a collection of comput -
`ment of a method of providing a moving image of a volume
`erized devices , such as in a group of data communications 65 of interest ;
`devices or other entities can also provide the system of the
`FIG . 14 shows a screen shot of a three dimensional ( 3D )
`invention . The system of the invention can be distributed
`cursor in a first position ;
`
`55
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-4 Filed 09/16/20 Page 23 of 32 PageID 181
`
`US 9 , 980 , 691 B2
`FIG . 15 shows a screen shot of a 3D cursor in a second
`provides imagery data to the data recording . The Medical
`position ; and
`device controller 52 provides control commands to the
`FIG . 16 shows a screen shot of a 3D cursor in a third
`medical imaging device 12 . A power supply 54 provides
`power to the medical imaging device 12 .
`position .
`Digital recording device 14 records each of the slices of
`DETAILED DESCRIPTION
`data collected by the medical imaging device 12 , and may
`also be used to record medical records meta data . Digital
`recording device 14 interacts / exchanges data with the Gen
`Mechanisms and techniques that provide a process for
`eral purpose processor 16 .
`combining slices generated by medical imaging devices to
`General purpose processor 16 interacts with the digital
`create a volume of interest and then presenting this volume 10
`recording device 14 based on inputs it receives from the user
`in a three - dimensional representation to a Head Display Unit
`through the Graphic