Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 1 of 32 PageID 127
`
`
`
`Exhibit 3
`U.S. Patent No. 9,473,766
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 2 of 32 PageID 128
`
`US0094.73766B2
`
`(12) United States Patent
`Douglas et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 9.473,766 B2
`Oct. 18, 2016
`
`(54) METHOD AND APPARATUS FOR THREE
`DIMIENSIONAL VIEWING OF IMAGES
`
`(71) Applicants: David Byron Douglas, Winter Park, FL
`(US); Robert E. Douglas, Winter Park,
`FL (US)
`(72) Inventors: David Byron Douglas, Winter Park, FL
`(US); Robert E. Douglas, Winter Park,
`FL (US)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 17 days.
`(21) Appl. No.: 14/313,398
`
`(*) Notice:
`
`(22) Filed:
`
`Jun. 24, 2014
`
`(65)
`
`Prior Publication Data
`US 2014/0307O67 A1
`Oct. 16, 2014
`
`Related U.S. Application Data
`(63) Continuation-in-part of application No. 12/176,569,
`filed on Jul. 21, 2008, now abandoned, which is a
`continuation-in-part of application No. 11/941,578,
`filed on Nov. 16, 2007, now Pat. No. 8,384,771.
`(60) Provisional application No. 60/877,931, filed on Dec.
`28, 2006.
`
`(2006.01)
`(2006.01)
`(2006.01)
`
`(51) Int. Cl.
`G06K 9/00
`G06T 700
`H04N I3/04
`(52) U.S. Cl.
`CPC ........ H04N 13/044 (2013.01); H04N 13/0434
`(2013.01); H04N 13/0459 (2013.01)
`(58) Field of Classification Search
`None
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,708.359 A *
`
`5,200,819 A * 4/1993 Nudelman ........... A61B 1/O008
`257/E31115
`5,402,191 A * 3/1995 Dean ...................... GO3B 35/16
`348/E13,022
`1/1998 Gregory ................. GOR 33.54
`324,307
`6,002,518 A * 12/1999 Faris .................. GO2B 27/0093
`345,32
`6,108,005. A * 8/2000 Starks ................ GO2B 27/22O7
`345/419
`6,449,309 B1* 9/2002 Tabata ............... GO2B 27,2264
`348.53
`6,545,650 B1 * 4/2003 Yamada ............. GO2B 27/0093
`345/7
`6,580,448 B1* 6/2003 Stuttler .................. A61B 3,113
`348/46
`6,676,259 B1* 1/2004 Trifilo .................. GO2B 27 O17
`348/E13.038
`7,524,053 B2 * 4/2009 Lipton ..................... GO2C 7/12
`351/159.56
`7,808.449 B2 * 10/2010 Neidrich ................. A63F 13/02
`345/6
`8,233,103 B2 * 7/2012 MacNaughton ... H04N 13/0438
`345/419
`
`(Continued)
`Primary Examiner — Stephen R Koziol
`Assistant Examiner — Mia M Thomas
`(74) Attorney, Agent, or Firm — Anderson Gorecki LLP
`(57)
`ABSTRACT
`A method and apparatus for three dimensional viewing of
`images is presented. Left eye viewing point (LEVP) imagery
`is passed through a left eye filter to obtain a filtered LEVP
`imagery. Right eye viewing point (REVP) imagery is passed
`through a right eye filter to obtain a filtered REVP imagery.
`The filtered LEVP imagery is projected on a display and the
`filtered REVP imagery is projected on the display. A user
`wearing polarized glasses is able to view the filtered LEVP
`imagery and the filtered REVP imagery as three-dimen
`sional imagery on the display. The three dimensional imag
`ery can be augmented.
`18 Claims, 17 Drawing Sheets
`
`480 y
`
`: selecting Avolume of iNTEREst From a collection of AGE-T
`
`SCS
`- - - - - - - - - - - - - - - - - - - - - - - - - -
`
`ARRANGING THE SLICES coRRESPONDING TO THEvoluntE of a
`NEREST
`
`a04
`
`--------------------------- -
`SEECSI&AN INIA WEWIWGANGE). The SLCES
`:
`
`t
`SELETEN3A wisw FollTRA LETEE
`
`498
`
`Y
`
`SELECTING AWENPI&T FRARETEYE
`
`y
`
`-------------------------
`
`4.
`
`:
`
`:
`
`:
`
`: DISPLAYING, IN A HEADDISPLAY UNIT (HDU), ANIMAGE FOR THE LEF:/ '
`EYE BASED ON THE NIIAL VEwins ANGLE. THE VIEW Point FoR
`SAFTYEAH WE PIN:RES
`
`DSPAYing, HE HUANIASE FORTHERIHEE BASED ON
`THE initial viewing ANGLE, THE viewpoist For The right EYE, 31
`AND THE volume of iNTEREST AND WHEREIN THE IMAGE FOR THE
`Las YEAY THE A3 FORSAs
`Yearc A. Hr.
`RES3&BAE SciE USER
`:
`
`:
`
`: USER SEELCT IMAGES FOR STORAGE AND possiblERETREIVAL /
`
`FRLstre referenice
`
`4-6
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 3 of 32 PageID 129
`
`US 9,473,766 B2
`Page 2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`
`
`2005/0030621 A1
`
`8,363,096 B1* 1/2013 Aguirre .............. HO4N 13,0434
`348/42
`8,542.326 B2 * 9/2013 MacNaughton ...... GO2F 1/1335
`349,15
`D692,941 S * 1 1/2013 Klinar .......................... D16/309
`9,077.982 B2* 7/2015 Rha .........
`HO4N 13,0434
`9,094,676 B1* 7/2015 Schutten
`HO4N 13.0429
`9,338.445 B2* 5/2016 Atkins ...
`... GO9G 3/003
`2005.0017938 A1* 1/2005 O'Donnell ......... HO4N 13,0422
`345,88
`2/2005 Takahashi .......... GO2B 27,2214
`359,464
`2005/O151152 A1* 7, 2005 Miller .................... GO2B, 27.26
`257/103
`2006/0210147 A1* 9/2006 Sakaguchi ............. GO1C 11,06
`382,154
`2006/0227103 A1* 10, 2006 Koo ................... HO4N 13,0468
`345,156
`2006/0238441 A1* 10/2006 Benjamin .......... HO4N 13.0429
`345.8
`2007.003583.0 A1* 2, 2007 Matveev ............ GO2B 27,2228
`359,464
`2007/0085902 A1* 4, 2007 Walker ............... HO4N 13,0497
`34.8/51
`
`2012/0013711 A1
`
`2012/0019636 A1
`
`2009, 0231697 A1* 9, 2009 Marcus .................... HO4N 3/10
`359,465
`2009/0304232 A1* 12/2009 Tsukizawa ............. A61B 3,113
`382,103
`1/2012 Tamir ................... HO4N 13,026
`348/46
`1/2012 Gefen ................ HO4N 13/0497
`34.8/55
`2012/0113235 A1* 5, 2012 Shintani ............. HO4N 13/0022
`34.8/51
`2012/O127284 A1* 5, 2012 Bar-Zeev ............. GO2B 27 O17
`348.53
`2012fO190439 A1* 7, 2012 Nourbakhsh ...... GO2B 27,2264
`463,31
`8, 2012 Sakai ................. GO2B 27,2264
`349.13
`2012/0287361 A1* 11/2012 Sugihara ................ GO2B, 27.26
`349.15
`2013,005783.0 A1* 3, 2013 Tsai ................... GO2B 27,2214
`353/7
`2013/0242063 A1* 9, 2013 Matsumoto ............ HO4N 13/04
`34.8/51
`2013/0278727 A1* 10, 2013 Tamir ................. HO4N 13,0048
`348/47
`2014/0347726 A1* 11/2014 Yang ...................... GO2B, 27.26
`359,465
`
`2012/0206665 A1
`
`* cited by examiner
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 4 of 32 PageID 130
`
`U.S. Patent
`
`US 9,473,766 B2
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 5 of 32 PageID 131
`
`U.S. Patent
`
`Oct. 18, 2016
`
`Sheet 2 of 17
`
`US 9,473,766 B2
`
`
`
`
`
`Medical
`Imaging
`Device
`
`14
`
`Digital
`Recording —
`Device
`
`/
`
`56
`
`
`
`
`
`General
`Purpose
`Processor - GUI
`
`— Power Supply
`<-- 3D image
`Printer |
`8 Data Flow
`film
`Developer
`EM energy ir
`
`--~i-
`
`s
`
`--------------------------
`
`
`
`FIGURE 2
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 6 of 32 PageID 132
`
`Sheet 3 of 17
`
`US 9,473,766 B2
`
`
`
`f3
`
`Scae
`
`Floure 34
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 7 of 32 PageID 133
`
`U.S. Patent
`
`Oct. 18, 2016
`
`Sheet 4 of 17
`
`US 9,473,766 B2
`
`
`
`y
`
`Note: Ciagram not to
`scale
`
`FIGURE 3B
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 8 of 32 PageID 134
`Case 6:20-cv-01699-GAP-DCI
`Document 1-3 Filed 09/16/20 Page 8 of 32 PageID 134
`
`U.S. Patent
`
`Oct. 18, 2016
`
`Sheet 5 of 17
`
`US 9,473,766 B2
`
`2:200“
`
`
`
`
`
`
`
` 28m9Ho:ESmmE”202
`
`VWEDGE
`
`1EE-mcgm>>o
`
`newwe-3&5an
`
`
`
`
`.119:8m>mc8%vcm
`_3m;
`
`n_>m:”on2%.26m
`w1111111111111111111111111111“
`
`
`
`
`
`ou39:
`55:00
`
`:39:cEgoo
`m68ow,n;
`mEmmm>mfimoxmvcm
`1SE-mgm301W
`
`68me,EEEEOcL
`
`
`55:00£5.60m
`
`
`
`
` Mmonom-"mos8+”W99:99:
`
`mmvme+>=m£EoF
`
`me-mcgm>>om
`
`
`
`a>mm”on2%26m_
`
`m:_m>N
`
`SQ:m
`
`Ec:_oo
`
` uSQ:29m
`no:
`
`
`
`
`
`
`
`mwN
`
`an
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 9 of 32 PageID 135
`
`U.S. Patent
`
`Oct. 18, 2016
`
`Sheet 6 of 17
`
`US 9,473,766 B2
`
`3 :
`
`
`
`102
`
`A
`
`3 is
`
`FIGURE 5
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 10 of 32 PageID 136
`
`U.S. Patent
`
`Oct. 18, 2016
`
`Sheet 7 Of 17
`
`US 9,473,766 B2
`
`
`
`Antefior
`
`Posterior
`
`
`
`Praxially
`
`Figure 6
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 11 of 32 PageID 137
`
`U.S. Patent
`
`Oct. 18, 2016
`
`Sheet 8 of 17
`
`US 9,473,766 B2
`
`his is the distace if the
`y-direction from the EVP
`to they-Coordinate of the theta
`w8Xe of Fitefest
`
`This is the dista FCe if the X
`direction fro:
`EVP's positiooking
`straight forward to the X-Cogdinate
`of the WCXe of itefest
`
`
`
`Ef
`
`104
`
`-- This is the distance in the
`z-direction from the LEVP's point to the
`Z Corrdinate of the VOxel of interest
`
`WXe of interest
`
`FIGURE 7
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 12 of 32 PageID 138
`
`U.S. Patent
`
`Oct. 18, 2016
`
`Sheet 9 Of 17
`
`US 9,473,766 B2
`
`SELECTING AWOLUME OF INTEREST FROM A COLLECTION OF IMAGE /
`SCES
`:
`
`402
`
`ARRANGING THE SLECES CORRESPONDING TO THE WOLUME OF /
`INTEREST
`
`SELECTING AN ENTIAL VIEWING ANGE OF THE SCES
`
`t
`
`SELECTING AWEWPONT FOR A LEFT EYE
`
`------------------------------------------------------------------------------------a------------------------------------------------------------------------------------
`
`
`
`- 46
`
`/ 48
`
`DISPLAYING, INA HEAD DISPLAY UNIT (HDU), ANIMAGE FOR THE LEFT M
`EYE BASED ON THE ENTIAL VIEWING ANGLE, THE VIEW POINT FOR
`SAD LEFT EYE AND THE WOLUME OF INTEREST
`
`412
`
`DISPLAYING, IN THE HDU, ANIMAGE FOR THE RIGHT EYE BASED ON
`THE INITIAL VIEWING ANGLE, THE VIEW POINT FOR THE RIGHT EYE,
`AND THE VOLUME OF INTEREST AND WHEREIN THE IMAGE FOR THE
`LEFT EYE AND THE IMAGE FOR SAID RIGHT EYE PRODUCEA THREE
`DMENSONAMAGE TO THE USER
`
`:
`
`414
`
`/
`
`USER SEELCT IMAGE(S) FOR STORAGE AND POSSIBLE RETREVAL /
`FOR FUTURE REFERENCE
`:
`
`416
`
`FIGURE 8
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 13 of 32 PageID 139
`
`U.S. Patent
`
`Oct. 18, 2016
`
`Sheet 10 of 17
`
`US 9,473,766 B2
`
`c 452
`USER SELECTS AERNATIVE VIEWING ANGLE
`Y
`
`r! 454
`
`REORIENTING AWOLUME OF INTEREST INACCORDANCE WITH THE /
`ALTERNATE WEWENG ANGE
`:
`
`DISPLAYING, IN THE HDU, AN IMAGE FOR THE LEFT EYE BASED ON THE /
`ALTERNATE VIEWING ANGLE, THE VIEW POINT FOR THE LEFT EYE AND
`THE VOLUME OF INTEREST
`
`:
`
`456
`
`DISPLAYING, IN THE HDU, AN IMAGE FOR THE RIGHT EYE BASED ON
`: THE ALTERNATE VIEWING ANGLE, THE VIEW POINT FOR THE RIGHT : /
`EYE, AND THE VOLUME OF INTEREST AND WHEREIN THE IMAGE FOR
`THE LEFT EYE AND THE IMAGE FOR THE RIGHT EYE PRODUCE AN :
`ALTERNATE THREE-DIN ENSIONAL MAGE O HE USER
`
`:
`
`458
`
`FIGURE 9
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 14 of 32 PageID 140
`
`U.S. Patent
`
`Oct. 18, 2016
`
`Sheet 11 of 17
`
`US 9,473,766 B2
`
`SELECTING TEMS OF THE IMAGE TO BE SUBTRACTED FROM THE /
`MAGE O PRODUCE A FLEREO MAGE
`:
`
`8
`
`502
`
`DISPLAYING, IN THE HOU, A FILTERED IMAGE FOR THE LEFT EYE
`BASED ON THE INITIAL VIEWING ANGLE, THE VIEW POINT FOR THE
`EYE AND HE WOLUE OF INTEREST
`
`M
`
`504
`
`DISPLAYING, IN THE HDU, A FILTERED IMAGE FOR THE RIGHT EYE
`BASED ON THE INITIAL VIEWING ANGLE, THE VIEW POINT FOR THE
`RIGHT EYE, AND THE VOLUME OF INTEREST AND WHEREIN THE
`FILTERED IMAGE FOR THE LEFT EYE AND THE FILTERED IMAGE FOR
`THE RIGHT EYE PRODUCE AFTERED THREE-DEMENSIONAL IMAGE TO
`THE USER
`
`
`
`506
`
`FIGURE 10
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 15 of 32 PageID 141
`
`U.S. Patent
`
`Oct. 18, 2016
`
`Sheet 12 of 17
`
`US 9,473,766 B2
`
`550 y
`
`SORTING voxELS OF THE TEMS BY A PROPERTY OF THE voxEL Y
`:-----------------------------------------------------!-----------------------------------------------------------------------------
`APPLYING COLORS TO GROUPS OF SORTED voxELS TO OBTAIN A /
`COLORED IMAGE
`
`554.
`
`i
`
`DISPLAYING, IN THE HDU, A colorED IMAGE FOR THE LEFT EYE /
`BASED ON THE INITIAL VIEWING ANGLE, THE VIEW POINT FOR THE :
`LEFT EYE AND THE WOUVE OF ENTEREST
`
`DISPLAYING, IN THE HDU, A coLoRED IMAGE FOR THE RIGHT EYE
`BASED ON THE INTIAL VIEWING ANGLE, THE VIEW POINT FOR THE
`RIGHT EYE, AND THE VOLUME OF INTEREST AND WHEREIN THE /
`COLORED IMAGE FOR SAID LEFT EYE AND SAID COLORED IMAGE FOR:
`THE RIGHT EYE PRODUCE A colorED THREE-DIMENSIONAL IMAGE To:
`:
`THE USER
`:
`
`558
`
`FIGURE 11
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 16 of 32 PageID 142
`
`U.S. Patent
`
`Oct. 18, 2016
`
`Sheet 13 of 17
`
`US 9,473,766 B2
`
`SELECNG A NEW WIEW POINT FOR A LEFT EYE
`
`t
`
`SELECTING A NEW WIEW PONT FOR A RIGHT EYE
`
`-- 602
`
`- 604
`
`DISPLAYING, IN THE HDU, ANIMAGE FOR THE LEFT EYE BASED ON THE /
`INTIAL VIEWING ANGLE, THE NEW VIEW POINT FOR THE LEFT EYE AND
`THE VOLUNE OF ENERES
`
`:
`
`606
`
`DISPLAYING, IN THE HDU, ANIMAGE FOR THE RIGHT EYE BASED ON .
`THE INITIAL VIEWING ANGLE, THE NEW VIEW POINT FOR THE RIGHT
`EYE, AND THE WOLUME OF INTEREST AND WHEREN THE IMAGE FOR
`THE LEFT EYE AND THE IMAGE FOR THE RIGHT EYE PRODUCE A :
`THREE-DEMENSONAMAGE TO THE USER
`
`:
`
`608
`
`/
`
`FIGURE 12
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 17 of 32 PageID 143
`
`U.S. Patent
`
`Oct. 18, 2016
`
`Sheet 14 of 17
`
`US 9,473,766 B2
`
`650
`
`\
`
`STACKING ASET OF IMAGES TAKEN WITHIN A GIVEN TIME INTERVAL /
`TO FORMA VOLUME OF ENTERES
`
`:
`
`DISPLAYING, IN A THREE DEMENSIONAL STEREOSCOPIC MANNER, THE
`: SET OF IMAGES OF THE VOLUME OF INTEREST IN A CONSECUTIVE
`MANNER TO REPRESENT THE VOLUME OF INTEREST AND HOW THE
`WOLUVE OF ENTEREST CHANGES OVER TIME
`
`FORMING A LOOP OF THE IMAGES SUCH THAT AFTER A LAST IMAGE /
`OF SET OF IMAGES IS DISPLAYED, THE SET OF IMAGES ARE
`:
`DISPLAYED AGAIN FROM THE BEGINNING IMAGE TO THE LAST IMAGE .
`
`FIGURE 13
`
`652
`
`654
`
`656
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 18 of 32 PageID 144
`
`U.S. Patent
`
`Oct. 18, 2016
`
`Sheet 15 Of 17
`
`US 9,473,766 B2
`
`72
`
`722
`
`
`
`78
`
`FIGURE 14
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 19 of 32 PageID 145
`
`U.S. Patent
`
`Oct. 18, 2016
`
`Sheet 16 of 17
`
`US 9,473,766 B2
`
`800 N.
`
`PASSING LEFT EYE VIEWING POINT (LEVP) IMAGERY
`THROUGH A LEFT EYE FILTER TO OBTAN AFTERED
`LEVP MAGERY
`
`PASSING RIGHT EYE VIEWING POINT (REVP) IMAGERY
`THROUGH A RIGHT EYE FILTER TO OBTAN A
`FTERED REVP MAGERY
`
`THE LEFT EYE FLTER AND THE RIGHT EYE FILTER
`COMPRESE ORTHOGONAL LINEAR POLARZATION
`FHTERS, AND WHEREIN THE POLARIZED GLASSES
`COMPRISE LINEAR POLARIZED GLASSES
`
`WHEREN THE LEFT EYE FLTER AND THE RIGHT
`EYE FILTER COMPRISE CIRCULAR POLARIZATION
`FHLTERS, AND WHEREIN THE POLARIZED GLASSES
`COMPRESE CIRCULAR POLARIZED GASSES
`
`WHEREN THE CIRCULAR POLARIZATION
`FTERSEACH FURTHER COMPRESESA
`OUARTER WAVE PLATE
`
`THE METHOD OF CAM WHEREN THE LEFT EYE
`FTER ANO THE RIGHT EYE FILTER COMPRISE
`ANAGLYPH COLOR FILTERS, AND WHEREEN THE
`POLAREZEO GE ASSES COMPRISE ANAGLYPH COLOR
`FTER GASSES
`
`PROJECTING THE FILTERED LEVP MAGERY ON A
`DISPLAY
`
`
`
`
`
`PROJECTENG THE FLTERED REVP MAGERY ON THE
`DISPLAY
`
`WHEREEN AUSER WEARNG POLARIZED GASSESS
`ABE OVEW THE FILTERED EVP MAGERY AND
`THE FILTERED REVP MAGERY AS THREE
`DIMENSIONAL MAGERY ON THE DISPLAY
`
`FIGURE 15
`
`8.
`
`82
`
`84
`
`85
`
`88
`
`82O
`
`8.
`
`824
`
`826
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 20 of 32 PageID 146
`
`U.S. Patent
`
`Oct. 18, 2016
`
`Sheet 17 Of 17
`
`US 9,473,766 B2
`
`900 Na
`
`
`
`
`
`
`
`
`
`
`
`
`
`INTERLEAVING LEFT EYE VIEWING POINT (LEVP)
`IMAGERY AND RIGHT EYE VIEWING POINT (REVP)
`MAGERY TO PRODUCE INTERLEAVED IMAGERY
`
`PROFECTING THE INTEREAVED MAGERY ON A
`DISPLAY
`
`WHEREN AUSER WEARNG ACTIVE SHUTTER
`GLASSES ISABLE TO WIEW THE INTERLEAVED
`IMAGERY AS THREE DIMENSIONAL MAGERY ON THE
`DISPLAY
`
`FIGURE 16
`
`9.
`
`12
`
`914.
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 21 of 32 PageID 147
`
`US 9,473,766 B2
`
`1.
`METHOD AND APPARATUS FOR THREE
`DIMENSIONAL VIEWING OF IMAGES
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`The present application is a continuation-in-part of U.S.
`patent application Ser. No. 12/176,569, Filed on Jul. 21,
`2008 which is a continuation-in-part of U.S. patent appli
`cation Ser. No. 11/941,578, filed Nov. 16, 2007 which claims
`the benefit of U.S. Provisional Patent Application No.
`60/877,931, filed on Dec. 29, 2006, the disclosures of which
`are incorporated herein by reference in their entirety.
`
`BACKGROUND
`
`10
`
`15
`
`2
`that when a radiologist views a CT scan, the limitations
`include viewing a single slice at a time (though there may be
`several CT slices present on the same or adjacent monitors).
`An example slice thickness of 1.25 mm would require
`approximately 500 slices in the viewing of the chest and
`abdomen. This process is inherently slow. Furthermore, in
`the example of a small pulmonary nodule which can be 2-3
`mm, each slice must be carefully scrutinized. This is a very
`time-consuming and labor intensive searching process.
`Systems designed to produce stereoscopic imagery have
`depended on the use of true Stereo pairs of images created by
`complex and costly optical systems. Attempts have been
`made to convert two-dimensional images to three-dimen
`sional images using shuttering with image shifting (e.g.,
`U.S. Pat. No. 5,510,832). Such techniques have not pro
`duced three-dimensional imagery having Sufficient quality
`for detailed medical examination purposes. Other transfor
`mations of Such pairs of images from one encoding method
`to another have been also been difficult and costly because
`they generally require depth information and computation. It
`has been necessary to generate stereo pairs of images using
`two separate cameras or a single camera with special lenses.
`Such arrangements are costly and difficult to use.
`The current process of viewing cross-sections relies on
`the radiologist being able to mentally construct a holistic
`view. The radiologist must be able to piece together multiple
`slices, and rotate them in order to gain a representation of a
`portion of the image. Although several programs have been
`able to construct a three-dimensional representation within
`the database, the user cannot see this representation in
`three-dimensional. Several current programs can layer
`images from different slices onto the same 2D screen. Thus
`there is a component of an x, y, and Z axis in the viewing
`field. However, the user cannot distinguish the distance in
`the y-axis (in the dimension projecting into and out of the
`image). Thus, the user does not have depth perception when
`Viewing an image.
`In some cases using current technology, faint tissue
`anomalies can be missed when looking at a whole series of
`multiple gray scale shaded images. The deficiency includes
`that such a faint anomaly would tend to blend in with other
`tissues of approximately equal grayscale.
`Embodiments of the invention significantly overcome
`Such deficiencies and provide mechanisms and techniques
`that provide a process for combining slices generated by
`medical imaging devices to create a volume of interest and
`then presenting this volume in a three-dimensional repre
`sentation to a Head Display Unit (HDU) so that the Radi
`ologist/Medical Professional (R/MP, also referred to herein
`as a user) can obtain a holistic view of the patient. Key
`image processing techniques are applied which enable the
`user: to rotate and view the volume of interest from alter
`native viewpoints; to enable tissue subtraction to facilitate
`unobstructed viewing of a region of interest, to identify
`differing tissues with color schematics; and to Zoom in for
`optimal viewing.
`In a particular embodiment of a method for providing
`three-dimensional viewing of images by a user, the method
`includes selecting a volume of interest from a collection of
`image slices and arranging the slices corresponding to the
`Volume of interest. The method also includes selecting an
`initial viewing angle of the slices, selecting a viewpoint for
`a left eye and selecting a viewpoint for a right eye. Addi
`tionally, the method includes displaying, in a head display
`unit (HDU), an image for the left eye based on the initial
`viewing angle, the view point for the left eye and the volume
`of interest; and displaying, in the HDU, an image for the
`
`Over the past several decades, the field of medical imag
`ing has made many advances. In the 1950s, the principals of
`Magnetic Resonance (MR) were initially investigated. The
`fundamental premise of MR is that different materials reso
`nate at different magnetic field strengths. Magnetic Reso
`nance Imaging (MRI) was researched in the 1970s and
`tested clinically on patients in 1980. In 1984, MRI was
`approved by the Food and Drug Administration (FDA) for
`clinical use. Since then, this imaging modality has grown
`rapidly in popularity.
`25
`Computed Tomography (CT) imaging (also called CAT
`scanning for Computed Axial Tomography) was invented in
`1972. Both gamma rays and X-rays were used in conjunction
`with a detector mounted on a special rotating frame to
`generate the image slices. Then a digital computer generates
`detailed cross sectional images. The original CT scan took
`hours to acquire a single slice of image data and more than
`24 hours to reconstruct this data into a single image. Today's
`state-of-the-art CT systems can acquire a single image in
`less than a second and reconstruct the image instantly.
`In the 1970s, digital imaging techniques were imple
`mented with the first clinical use and acceptance of the CT
`scanner. Analog to digital converters and computers were
`also adapted to conventional fluoroscopic image intensifier/
`TV systems in the 1970s as well. The key benefits of the
`digital technology include the fact that digital X-ray images
`can be enhanced and manipulated with computers, and the
`fact that digital images can be sent via a network to other
`workstations and computer monitors so that many people
`can share the information and assist in the diagnosis.
`Other recent developments include Positron Emission
`Tomography (PET), Single Photon Emission Computed
`Tomography (SPECT), and functional MRI (fMRI). PET is
`a nuclear medicine medical imaging technique which pro
`duces a three-dimensional image or map of functional
`processes in the body. SPECT is a nuclear medicine tomo
`graphic imaging technique using gamma rays. It is very
`similar to conventional nuclear medicine planar imaging
`using a gamma camera. However, it is not able to provide
`true three-dimensional information. This information is typi
`cally presented as cross-sectional slices through the patient,
`but can be freely reformatted or manipulated as required.
`Functional magnetic resonance imaging (fMRI) is the use
`of MRI to measure the hemodynamic response related to
`neural activity in the brain or spinal cord of humans or other
`animals. It is one of the most recently developed forms of
`neuroimaging.
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`SUMMARY
`
`Conventional mechanisms such as those explained above
`suffer from a variety of deficiencies. One such deficiency is
`
`65
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 22 of 32 PageID 148
`
`US 9,473,766 B2
`
`4
`between many Software processes on several data commu
`nications devices, or all processes could run on a small set
`of dedicated computers, or on one computer alone.
`It is to be understood that the embodiments of the inven
`tion can be embodied strictly as a software program, as
`Software and hardware, or as hardware and/or circuitry
`alone. Such as within a data communications device. The
`features of the invention, as explained herein, may be
`employed in data processing devices and/or software sys
`tems for Such devices.
`Note that each of the different features, techniques, con
`figurations, etc. discussed in this disclosure can be executed
`independently or in combination. Accordingly, the present
`invention can be embodied and viewed in many different
`ways. Also, note that this Summary section herein does not
`specify every embodiment and/or incrementally novel
`aspect of the present disclosure or claimed invention.
`Instead, this Summary only provides a preliminary discus
`sion of different embodiments and corresponding points of
`novelty over conventional techniques. For additional details,
`elements, and/or possible perspectives (permutations) of the
`invention, the reader is directed to the Detailed Description
`section and corresponding figures of the present disclosure
`as further discussed below.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The foregoing will be apparent from the following more
`particular description of preferred embodiments of the
`invention, as illustrated in the accompanying drawings in
`which like reference characters refer to the same parts
`throughout the different views. The drawings are not nec
`essarily to Scale, emphasis instead being placed upon illus
`trating the principles of the invention. The drawings are not
`necessarily to scale, emphasis instead being placed upon
`illustrating the principles of the invention.
`FIG. 1 is a block diagram a computer system that per
`forms three-dimensional viewing of images in accordance
`with embodiments of the invention.
`FIG. 2 is a block diagram of the system indicating flow of
`data and the like.
`FIG. 3A is a diagram showing left and right eye viewing
`angles.
`FIG. 3B is a diagram showing a user-selectable conver
`gence point.
`FIG. 4 shows a volume of interest comprised of a series
`of slices selected by a user.
`FIG. 5 is a diagram showing a bird's eye view demon
`strating angle theta.
`FIG. 6 is a diagram demonstrating the Volumetric data in
`the grey cylinder.
`FIG. 7 is a diagram demonstrates the hypotenuses hyp1
`and hyp2.
`FIG. 8: depicts a flow diagram of a particular embodiment
`of a method of providing three-dimensional viewing of
`images in accordance with embodiments of the invention.
`FIG.9 depicts a flow diagram of a particular embodiment
`of a method of viewing an alternative viewing angle.
`FIG. 10 depicts a flow diagram of a particular embodi
`ment of a method of filtering an image.
`FIG. 11 depicts a flow diagram of a particular embodi
`ment of a method of applying colors to an image.
`FIG. 12 depicts a flow diagram of a particular embodi
`ment of a method of Zooming in on an image.
`FIG. 13 depicts a flow diagram of a particular embodi
`ment of a method of providing a moving image of a Volume
`of interest.
`
`3
`right eye based on the initial viewing angle, the view point
`for the right eye, and the volume of interest and wherein the
`image for the left eye and the image for the right eye produce
`a three-dimensional image to the user.
`Other embodiments include a computer readable medium
`having computer readable code thereon for providing three
`dimensional viewing of images by a user. The computer
`readable medium includes instructions for selecting a Vol
`ume of interest from a collection of image slices and
`instructions for arranging the slices corresponding to the
`volume of interest. The computer readable medium also
`includes instructions for selecting an initial viewing angle of
`the slices, instructions for selecting a viewpoint for a left eye
`and instructions for selecting a viewpoint for a right eye.
`Additionally, the computer readable medium includes
`instructions for displaying, in a head display unit (HDU), an
`image for the left eye based on the initial viewing angle, the
`view point for the left eye and the volume of interest; and
`instructions for displaying, in the HDU, an image for the
`right eye based on the initial viewing angle, the view point
`for the right eye, and the volume of interest and wherein the
`image for the left eye and the image for the right eye produce
`a three-dimensional image to the user
`Still other embodiments include a computerized device,
`configured to process all the method operations disclosed
`herein as embodiments of the invention. In such embodi
`ments, the computerized device includes a memory system,
`a processor, communications interface in an interconnection
`mechanism connecting these components. The memory sys
`tem is encoded with a process that provides three-dimen
`sional viewing of images by a user as explained herein that
`when performed (e.g. when executing) on the processor,
`operates as explained herein within the computerized device
`to perform all of the method embodiments and operations
`explained herein as embodiments of the invention. Thus any
`computerized device that performs or is programmed to
`perform processing explained herein is an embodiment of
`the invention.
`Other arrangements of embodiments of the invention that
`are disclosed herein include Software programs to perform
`the method embodiment steps and operations Summarized
`above and disclosed in detail below. More particularly, a
`computer program product is one embodiment that has a
`computer-readable medium including computer program
`logic encoded thereon that when performed in a computer
`ized device provides associated operations providing three
`dimensional viewing of images by a user as explained
`herein. The computer program logic, when executed on at
`least one processor with a computing system, causes the
`processor to perform the operations (e.g., the methods)
`indicated herein as embodiments of the invention. Such
`arrangements of the invention are typically provided as
`Software, code and/or other data structures arranged or
`encoded on a computer readable medium Such as an optical
`medium (e.g., CD-ROM), floppy or hard disk or other a
`55
`medium such as firmware or microcode in one or more ROM
`or RAM or PROM chips or as an Application Specific
`Integrated Circuit (ASIC) or as downloadable software
`images in one or more modules, shared libraries, etc. The
`Software or firmware or other Such configurations can be
`installed onto a computerized device to cause one or more
`processors in the computerized device to perform the tech
`niques explained herein as embodiments of the invention.
`Software processes that operate in a collection of comput
`erized devices, such as in a group of data communications
`devices or other entities can also provide the system of the
`invention. The system of the invention can be distributed
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`60
`
`65
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-3 Filed 09/16/20 Page 23 of 32 PageID 149
`
`US 9,473,766 B2
`
`5
`FIG. 14 is a block diagram of an environment for per
`forming three-dimensional viewing of images in accordance
`with embodiments of the invention.
`FIG. 15 is a flow diagram of a particular embodiment of
`a method of viewing three dimensional images.
`FIG. 16 is a flow diagram of another particular embodi
`ment of a method of viewing three dimensional images.
`
`DETAILED DESCRIPTION
`
`5
`
`10
`
`15
`
`6
`The images are sent to a head display unit (HDU) 22 worn
`by the user 24. While a head display unit is shown and
`described, it should be appreciated that the present invention
`could also be utilized with other display units, including but
`not limited to, a display unit incorporating polarized lenses,
`a display unit wherein multiplexed images are viewed via
`shuttered lenses, virtual reality displays having a display
`unit with unique left and right eye pixel displays, and other
`types of three-dimensional (3D) displays as would be known
`to one of reasonable skill in the art.
`The head display unit 22 displays image 18 to the right
`eye. Note that multiple slices have been stacked, creating a
`Volumetric mass so that the image seen by the right eye is the
`volume of interest selected by the user. The head display unit
`22 also displays image 20 to the left eye similar to the
`manner of which the right image was displayed. This
`produces a three-dimensional image to the user 24.
`Referring now to FIG. 2, a block diagram of the system
`50 comprising one particular embodiment of the invention is
`shown. The diagram indicates flow of data, imagery, and the
`like according to the legend. Medical imaging device 12
`utilizes EM energy that emanates from the device 12 and is
`also received by the device 12. The medical imaging device
`12 receives imaging commands from the controller 52 and
`provides imagery data to the data recording. The Medical
`device controller 52 provides control commands to the
`medical imaging device 12. A power Supply 54 provides
`power to the medical imaging device 12.
`Digital recording device 14 records each of the slices of
`data collected by the medical imaging device 12, and may
`also be used to record medical records meta data. Digital
`recording device 14 interacts/exchanges data with the Gen
`eral purpose processor 16.
`General purpose processor 16 interacts with the digital
`recording device 14 based on inputs it receives from the user
`through the Graphical User Interface (GUI) 58. The GUI 58
`allows interaction between the