Case 6:20-cv-01699-GAP-DCI Document 1-1 Filed 09/16/20 Page 1 of 24 PageID 75
`
`
`
`Exhibit 1
`U.S. Patent No. 8,384,771
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-1 Filed 09/16/20 Page 2 of 24 PageID 76
`
`US008384771B1
`
`(12) United States Patent
`Douglas
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,384,771 B1
`Feb. 26, 2013
`
`(54) METHOD AND APPARATUS FOR THREE
`DIMIENSIONAL VIEWING OF IMAGES
`
`(*) Notice:
`
`(76) Inventor: David Byron Douglas, Winter Park, FL
`(US)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 1312 days.
`(21) Appl. No.: 11/941,578
`
`(22) Filed:
`
`Nov. 16, 2007
`Related U.S. Application Data
`(60) Provisional application No. 60/877,931, filed on Dec.
`28, 2006.
`
`(51) Int. Cl.
`(2006.01)
`H04N I3/04
`(52) U.S. Cl. ............................... 348/53: 348/51: 348/42
`(58) Field of Classification Search ........................ None
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`4,472,737 A * 9, 1984 Iwasaki ........................... 348,51
`4,952,024. A * 8/1990 Gale ................
`... 34.8/53
`5,200,819 A * 4, 1993 Nudelman et al.
`... 348/46
`5,682,172 A * 10/1997 Travers et al. .......
`... 34.8/53
`5,696,521 A * 12/1997 Robinson et al.
`... 34.8/53
`6,108,005. A * 8/2000 Starks et al. .....
`... 348/42
`6,115.449 A * 9/2000 Jang et al. ...
`... 34.8/51
`6,580,448 B1* 6/2003 Stuttler ...
`... 348/46
`2006/0210147 A1* 9/2006 Sakaguchi .......
`... 348/42
`2006/0238441 A1* 10/2006 Benjamin et al. ................. 345.8
`
`
`
`JP
`JP
`JP
`
`FOREIGN PATENT DOCUMENTS
`2008220406 A * 9, 2008
`2009000167 A * 1, 2009
`2009018048 A * 1, 2009
`
`OTHER PUBLICATIONS
`Martin, RW et al. “Stereographic Viewing of 3DUltrasound Images:
`A Novelty or a Tool?” 1995 IEEE Ultrasonics Symposium. IEEE
`Press. 1431-1434.
`Haker, Steven et al. “Nondistorting Flattening Maps and the 3-D
`Visualization of Colon CT Images.” IEEE Transactions on Medical
`Imaging. vol. 19, No. 7. Jul. 2000. 665-670.*
`“Fast Inspection of Contents of a Volume of 3D Data.” IBMTechnical
`Disclosure Bulletin. Feb. 1, 1994. vol. 37, Issue 2A. 4 Pages.*
`* cited by examiner
`Primary Examiner — Jeffrey R Swearingen
`(74) Attorney, Agent, or Firm — Anderson Gorecki &
`Rouille LLP
`
`ABSTRACT
`(57)
`A method, apparatus and computer program product for three
`dimensional viewing of images is presented. Embodiments of
`the invention provide a process for combining slices gener
`ated by medical imaging devices to create a Volume of interest
`and then present this Volume in a three-dimensional represen
`tation to a head display unit so that the user can obtain a
`holistic view of the patient. Key image processing techniques
`are applied which enable the user to rotate and view the
`volume of interest from alternative viewpoints; to enable
`tissue subtraction to facilitate unobstructed viewing of a
`region of interest; to identify differing tissues with color
`schematics; and to Zoom in for optimal viewing.
`21 Claims, 12 Drawing Sheets
`
`4-y
`
`sELECTING AWOLUMEoFNTEREST FROMA collecTION OF IMAGE /
`SCES
`
`402
`
`ArraNeigh scs correspong The Woute of
`Interest
`
`-
`
`SLECTING ANNAL WEWING ANSE OF THESEs -o
`
`SELECTING AWPINFRA LEFT
`
`SELECTING AWEWPOINT FORARGHTEE
`
`-
`
`/ 41
`
`DisPLAYING, INAHEADDISPLAY UNIT (Hou, ANIMAGE FOR THE LEFT/
`YEBASED ON THE INITIALWIEWING ANGLE, THEWIEW Point For
`SADLEYEANTHE WOLUE OF INTEREST
`
`12
`
`DisPAYING INTHEHDUANIMAGE FORTHERIGHTYEBASED ON
`the NITIAL WEWING ANGLE, THEWIEWPOINTForthrighter,
`ANT woula of NTrst And HRTHAG Fort
`EFTEYE AND THE MASE FORSADRGHTEEPRODUCEATHREE
`DIENSIONAIAE THESER
`
`/ 41
`
`users ELCTIMAGE(s)FoRsToRAGE ANDPossIBLE RETREIWAL
`FORFUTUREREFERENCE
`
`/ a16
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-1 Filed 09/16/20 Page 3 of 24 PageID 77
`
`U.S. Patent
`
`US 8,384,771 B1
`
`
`
`
`
`
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-1 Filed 09/16/20 Page 4 of 24 PageID 78
`
`U.S. Patent
`
`Feb. 26, 2013
`
`Sheet 2 of 12
`
`US 8,384,771 B1
`
`- -?
`
`12
`
`14
`
`Medical
`Device
`Controller
`/
`R Digital
`ecording — H 56
`Device
`
`N
`(as Medical
`madin
`ea) E.
`/ 54
`POWer
`Supply
`
`16
`
`POWer
`Supply
`
`/
`General
`Purpose
`Processor R- GUI
`
`/
`
`58
`
`Power Supply
`-26
`s—D-3D image
`Printer?
`-> Data Flow
`Film
`Developer
`see
`EM energy
`t
`
`Y
`
`1829,
`(L)
`
`-n
`HDU
`(R) —
`
`FIGURE 2
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-1 Filed 09/16/20 Page 5 of 24 PageID 79
`
`U.S. Patent
`
`Feb. 26, 2013
`
`sheet 3 of 12
`
`
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-1 Filed 09/16/20 Page 6 of 24 PageID 80
`Case 6:20-cv-01699-GAP-DCI
`Document 1-1 Filed 09/16/20 Page 6 of 24 PageID 80
`
`U.S. Patent
`
`Feb. 26, 2013
`
`Sheet4of12
`
`US 8,384,771 B1
`
`
`
`
`
`
`«8|\
`
`
`
`
`
`
`
`29m
`
`CEDOCEDO_ono:_0
`
`99:5:2899:
`
`
`
`
`
` omow8+nu39:mosom-“
`
`
`
`>
`
`
`
`
`
`
`
`
`
`
` 088B“0:E3920”202
`
`VHEDGE
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`«2I/
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
` o:_m>NAIoEmwm>mcmo__wvcm
`
`
`n_>m_._”on2%26m
`
`own9->=mEEoE
`
`EE-22m26m
`N2.|/
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`foEmwm>mcoo__mvcm
`
`n_>m_m”on2%26m
`
`
`
`own3-2:82:05
`EE-22m26m
`
`_
`
`me-22m26m
`
`
`
`mowm:+>__mc_Eoc.,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`cE:_oO
`sou|\
`
`
`
`so:
`
`:3
`
`39:
`
`mos8+n
`
`
`
`
`
`
`ou99:9%om-u
`:E:_oo905cE:_oO
`
`
`
`
`
`
`own9+265E090
`
`me-22m26m
`
`m:_m>N
`
`
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-1 Filed 09/16/20 Page 7 of 24 PageID 81
`
`U.S. Patent
`
`Feb. 26, 2013
`
`Sheet 5 of 12
`
`US 8,384,771 B1
`
`C.
`
`102
`
`
`
`104
`
`FIGURE 5
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-1 Filed 09/16/20 Page 8 of 24 PageID 82
`
`U.S. Patent
`
`Feb. 26, 2013
`
`Sheet 6 of 12
`
`US 8,384,771 B1
`
`Latefa
`
`102
`
`a Posterior
`
`S
`
`S
`
`S
`
`
`
`
`
`ea a
`
`
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-1 Filed 09/16/20 Page 9 of 24 PageID 83
`
`U.S. Patent
`
`Feb. 26, 2013
`
`Sheet 7 of 12
`
`US 8,384,771 B1
`
`r.
`
`SS Es S3 a tise
`y-direction for the E.
`
`-
`
`
`
`W{}xe fifts:ses
`
`E.
`
`104
`
`, fist:
`
`This is the d sing ...
`direction from EVE S 3: . looking
`&f Es Sisse: ; sees
`
`-H This is the distance in the
`z-direction from the LEVP's point to the
`ZCOrrdinate of the VOXel of interest
`
`f8 festest
`
`FIGURE 7
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-1 Filed 09/16/20 Page 10 of 24 PageID 84
`
`U.S. Patent
`
`Feb. 26, 2013
`
`Sheet 8 of 12
`
`US 8,384,771 B1
`
`400
`
`SELECTING AWOLUME OF INTEREST FROMA COLLECTION OF IMAGE /
`SLICES
`
`ARRANGING THE SLICES CORRESPONDING TO THE VOLUME OF
`INTEREST
`
`/
`
`402
`
`404
`
`SELECTING AN INITIAL VIEWING ANGLE OF THE SLICES
`
`SELECTING AVIEW POINT FOR A LEFT EYE
`
`SELECTING AVIEW POINT FOR ARIGHT EYE
`
`/ 4O6
`
`/ 408
`
`- 410
`
`DISPLAYING, INAHEADDISPLAY UNIT (HDU), ANIMAGE FOR THE LEFT /
`EYE BASED ON THE INITIAL VIEWING ANGLE, THE VIEW POINT FOR
`SAID LEFT EYE AND THE VOLUME OF INTEREST
`
`412
`
`DISPLAYING, IN THE HDU, ANIMAGE FOR THE RIGHT EYE BASED ON
`THE INITIAL VIEWING ANGLE, THE VIEWPOINT FOR THE RIGHTEYE,
`AND THE VOLUME OF INTEREST AND WHEREIN THE IMAGE FOR THE
`LEFT EYE AND THE IMAGE FOR SAID RIGHT EYE PRODUCE ATHREE
`DIMENSIONAL IMAGE TO THE USER
`
`414
`
`/
`
`USER SEELCT IMAGE(S) FOR STORAGE AND POSSIBLE RETREIVAL /
`FORFUTURE REFERENCE
`
`416
`
`FIGURE 8
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-1 Filed 09/16/20 Page 11 of 24 PageID 85
`
`U.S. Patent
`
`Feb. 26, 2013
`
`Sheet 9 of 12
`
`US 8,384,771 B1
`
`450 \
`
`USER SELECTS ALTERNATIVE VIEWING ANGLE
`
`y
`
`REORIENTING AWOLUME OF INTEREST INACCORDANCE WITH THE /
`ALTERNATE VIEWING ANGLE
`
`454
`
`- 452
`
`DISPLAYING, IN THE HDU, ANIMAGE FOR THE LEFT EYE BASED ON THE /
`ALTERNATE VIEWING ANGLE, THE VIEWPOINT FOR THE LEFT EYE AND
`THE VOLUME OF INTEREST
`
`456
`
`DISPLAYING, IN THE HDU, ANIMAGE FOR THE RIGHT EYE BASED ON
`THE ALTERNATE VIEWING ANGLE, THE VIEWPOINT FOR THE RIGHT
`EYE, AND THE VOLUME OF INTEREST AND WHEREIN THE IMAGE FOR
`THE LEFT EYE AND THE IMAGE FOR THE RIGHT EYE PRODUCE AN
`ALTERNATE THREE-DIMENSIONAL IMAGE TO THE USER
`
`458
`
`/
`
`FIGURE 9
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-1 Filed 09/16/20 Page 12 of 24 PageID 86
`
`U.S. Patent
`
`Feb. 26, 2013
`
`Sheet 10 of 12
`
`US 8,384,771 B1
`
`500
`
`\
`
`SELECTING TEMS OF THE IMAGE TO BE SUBTRACTED FROM THE /
`IMAGE TO PRODUCE AFILTERED IMAGE
`
`y
`
`502
`
`DISPLAYING, IN THE HDU, AFILTERED IMAGE FOR THE LEFT EYE
`BASED ON THE INITIAL VIEWING ANGLE, THE VIEWPOINT FOR THE
`EYE AND THE VOLUME OF INTEREST
`
`/
`
`504
`
`DISPLAYING, IN THE HDU, AFILTERED IMAGE FOR THE RIGHTEYE
`BASED ON THE INITIAL VIEWING ANGLE, THE VIEW POINT FOR THE
`RIGHTEYE, AND THE VOLUME OF INTEREST AND WHEREIN THE
`FILTERED IMAGE FOR THE LEFT EYE AND THE FILTERED IMAGE FOR
`THE RIGHT EYE PRODUCEAFILTERED THREE-DIMENSIONAL IMAGETO
`THE USER
`
`/
`
`506
`
`FIGURE 10
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-1 Filed 09/16/20 Page 13 of 24 PageID 87
`
`U.S. Patent
`
`Feb. 26, 2013
`
`Sheet 11 of 12
`
`US 8,384,771 B1
`
`550
`
`\
`
`SORTINGWOXELS OF THE ITEMS BY A PROPERTY OF THE WOXEL
`
`/ 552
`
`APPLYING COLORS TO GROUPS OF SORTED WOXELS TO OBTAIN A /
`COLORED IMAGE
`
`554
`
`DISPLAYING, IN THE HDU, A COLORED IMAGE FOR THE LEFT EYE
`BASED ON THE INITIAL VIEWING ANGLE, THE VIEW POINT FOR THE
`LEFT EYE AND THE VOLUME OF INTEREST
`
`/
`
`556
`
`y
`y
`
`DISPLAYING, IN THE HDU, A COLORED IMAGE FOR THE RIGHT EYE
`BASED ON THE INITIAL VIEWING ANGLE, THE VIEW POINT FOR THE / 558
`RIGHTEYE, AND THE VOLUME OF INTEREST AND WHEREIN THE
`COLORED IMAGE FOR SAID LEFT EYE AND SAID COLORED IMAGE FOR
`THE RIGHT EYE PRODUCE A COLORED THREE-DIMENSIONAL IMAGE TO
`THE USER
`
`FIGURE 11
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-1 Filed 09/16/20 Page 14 of 24 PageID 88
`
`U.S. Patent
`
`Feb. 26, 2013
`
`Sheet 12 of 12
`
`US 8,384,771 B1
`
`600
`
`\
`
`SELECTING A NEW VIEWPOINT FORA LEFT EYE
`
`SELECTING A NEW VIEW POINT FOR ARIGHT EYE
`
`/ 602
`
`604
`
`/
`
`DISPLAYING, IN THE HDU, ANIMAGE FOR THE LEFT EYE BASED ON THE /
`INITIAL VIEWING ANGLE, THE NEWVIEWPOINT FOR THE LEFT EYE AND
`THE VOLUME OF INTEREST
`
`606
`
`DISPLAYING, IN THE HDU, ANIMAGE FOR THE RIGHT EYE BASED ON
`THE INITIAL VIEWING ANGLE, THE NEW VIEWPOINT FOR THE RIGHT
`EYE, AND THE VOLUME OF INTEREST AND WHEREIN THE IMAGE FOR
`THE LEFT EYE AND THE IMAGE FOR THE RIGHT EYE PRODUCE A
`THREE-DIMENSIONAL IMAGE TO THE USER
`
`608
`
`/
`
`FIGURE 12
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-1 Filed 09/16/20 Page 15 of 24 PageID 89
`
`US 8,384,771 B1
`
`1.
`METHOD AND APPARATUS FOR THREE
`DIMENSIONAL VIEWING OF IMAGES
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`The present application claims the benefit of U.S. Provi
`sional Patent Application No. 60/877.931, filed on Dec. 28,
`2006, which is incorporated herein by reference in its entirety.
`
`10
`
`BACKGROUND
`
`2
`approximately 500 slices in the viewing of the chest and
`abdomen. This process is inherently slow. Furthermore, in the
`example of a small pulmonary nodule which can be 2-3 mm,
`each slice must be carefully scrutinized. This is a very time
`consuming and labor intensive searching process.
`Systems designed to produce stereoscopic imagery have
`depended on the use of true Stereo pairs of images created by
`complex and costly optical systems. Attempts have been
`made to convert two-dimensional images to three-dimen
`sional images using shuttering with image shifting (e.g., U.S.
`Pat. No. 5,510,832). Such techniques have not produced
`three-dimensional imagery having sufficient quality for
`detailed medical examination purposes. Other transforma
`tions of Such pairs of images from one encoding method to
`another has been also been difficult and costly because they
`generally require depth information and computation. It has
`been necessary to generate Stereo pairs of images using two
`separate cameras or a single camera with special lenses. Such
`arrangements are costly and difficult to use.
`The current process of viewing cross-sections relies on the
`radiologist being able to mentally construct a holistic view.
`The radiologist must be able to piece together multiple slices,
`and rotate them in order to gain a representation of a portion
`of the image. Although several programs have been able to
`construct a three-dimensional representation within the data
`base, the user cannot see this representation in three-dimen
`sional. Several current programs can layer images from dif
`ferent slices onto the same 2D screen. Thus there is a
`component of an x, y, and Z axis in the viewingfield. However,
`the user cannot distinguish the distance in the y-axis (in the
`dimension projecting into and out of the image). Thus, the
`user does not have depth perception when viewing an image.
`In some cases using current technology, faint tissue anoma
`lies can be missed when looking at a whole series of multiple
`gray scale shaded images. The deficiency includes that such a
`faint anomaly would tend to blend in with other tissues of
`approximately equal grayscale.
`Embodiments of the invention significantly overcome Such
`deficiencies and provide mechanisms and techniques that
`provide a process for combining slices generated by medical
`imaging devices to create a Volume of interest and then pre
`senting this Volume in a three-dimensional representation to a
`Head Display Unit (HDU) so that the Radiologist/Medical
`Professional (R/MP, also referred to herein as a user) can
`obtain a holistic view of the patient. Key image processing
`techniques are applied which enable the user: to rotate and
`view the volume of interest from alternative viewpoints; to
`enable tissue subtraction to facilitate unobstructed viewing of
`a region of interest; to identify differing tissues with color
`schematics; and to Zoom in for optimal viewing.
`In a particular embodiment of a method for providing
`three-dimensional viewing of images by a user, the method
`includes selecting a Volume of interest from a collection of
`image slices and arranging the slices corresponding to the
`Volume of interest. The method also includes selecting an
`initial viewing angle of the slices, selecting a viewpoint for a
`left eye and selecting a viewpoint for a right eye. Additionally,
`the method includes displaying, in ahead display unit (HDU),
`animage for the left eye based on the initial viewing angle, the
`view point for the left eye and the volume of interest; and
`displaying, in the HDU, an image for the right eye based on
`the initial viewing angle, the view point for the right eye, and
`the Volume of interest and wherein the image for the left eye
`and the image for the right eye produce a three-dimensional
`image to the user.
`Other embodiments include a computer readable medium
`having computer readable code thereon for providing three
`
`15
`
`25
`
`30
`
`Over the past several decades, the field of medical imaging
`has made many advances. In the 1950s, the principals of
`Magnetic Resonance (MR) were initially investigated. The
`fundamental premise of MR is that different materials reso
`nate at different magnetic field strengths. Magnetic Reso
`nance Imaging (MRI) was researched in the 1970s and tested
`clinically on patients in 1980. In 1984, MRI was approved by
`the Food and Drug Administration (FDA) for clinical use.
`Since then, this imaging modality has grown rapidly in popu
`larity.
`Computed Tomography (CT) imaging (also called CAT
`scanning for Computed Axial Tomography) was invented in
`1972. Both gamma rays and X-rays were used in conjunction
`with a detector mounted on a special rotating frame to gen
`erate the image slices. Then a digital computer generates
`detailed cross sectional images. The original CT scan took
`hours to acquire a single slice of image data and more than 24
`hours to reconstruct this data into a single image. Today's
`state-of-the-art CT systems can acquire a single image in less
`than a second and reconstruct the image instantly.
`In the 1970s, digital imaging techniques were imple
`mented with the first clinical use and acceptance of the CT
`scanner. Analog to digital converters and computers were also
`35
`adapted to conventional fluoroscopic image intensifier/TV
`systems in the 1970s as well. The key benefits of the digital
`technology include the fact that digital X-ray images can be
`enhanced and manipulated with computers, and the fact that
`digital images can be sent via a network to other workstations
`and computer monitors so that many people can share the
`information and assist in the diagnosis.
`Other recent developments include Positron Emission
`Tomography (PET), Single Photon Emission Computed
`Tomography (SPECT), and functional MRI (fMRI). PET is
`a nuclear medicine medical imaging technique which pro
`duces a three-dimensional image or map of functional pro
`cesses in the body. SPECT is a nuclear medicine tomographic
`imaging technique using gamma rays. It is very similar to
`conventional nuclear medicine planar imaging using a
`gamma camera. However, it is notable to provide true three
`dimensional information. This information is typically pre
`sented as cross-sectional slices through the patient, but can be
`freely reformatted or manipulated as required. Functional
`magnetic resonance imaging (fMRI) is the use of MRI to
`measure the hemodynamic response related to neural activity
`in the brain or spinal cord of humans or other animals. It is one
`of the most recently developed forms of neuroimaging.
`
`40
`
`45
`
`50
`
`55
`
`SUMMARY
`
`Conventional mechanisms such as those explained above
`suffer from a variety of deficiencies. One such deficiency is
`that when a radiologist views a CT scan, the limitations
`include viewing a single slice at a time (though there may be
`several CT slices present on the same or adjacent monitors).
`An example slice thickness of 1.25 mm would require
`
`60
`
`65
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-1 Filed 09/16/20 Page 16 of 24 PageID 90
`
`US 8,384,771 B1
`
`4
`within a data communications device. The features of the
`invention, as explained herein, may be employed in data
`processing devices and/or Software systems for Such devices.
`Note that each of the different features, techniques, con
`figurations, etc. discussed in this disclosure can be executed
`independently or in combination. Accordingly, the present
`invention can be embodied and viewed in many different
`ways. Also, note that this Summary section herein does not
`specify every embodiment and/or incrementally novel aspect
`of the present disclosure or claimed invention. Instead, this
`Summary only provides a preliminary discussion of different
`embodiments and corresponding points of novelty over con
`ventional techniques. For additional details, elements, and/or
`possible perspectives (permutations) of the invention, the
`reader is directed to the Detailed Description section and
`corresponding figures of the present disclosure as further
`discussed below.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The foregoing will be apparent from the following more
`particular description of preferred embodiments of the inven
`tion, as illustrated in the accompanying drawings in which
`like reference characters refer to the same parts throughout
`the different views. The drawings are not necessarily to scale,
`emphasis instead being placed upon illustrating the principles
`of the invention.
`FIG. 1 is a block diagram a computer system that performs
`three-dimensional viewing of images in accordance with
`embodiments of the invention;
`FIG. 2 is a block diagram of the system indicating flow of
`data and the like;
`FIG. 3 is a diagram showing left and right eye viewing
`angles:
`FIG. 4 shows a volume of interest comprised of a series of
`slices selected by a user,
`FIG. 5 is a diagram showing a bird’s eye view demonstrat
`ing angle theta;
`FIG. 6 is a diagram demonstrating the Volumetric data in
`the grey cylinder,
`FIG. 7 is a diagram demonstrates the hypotenuses hyp1 and
`hyp2;
`FIG. 8: depicts a flow diagram of a particular embodiment
`of a method of providing three-dimensional viewing of
`images in accordance with embodiments of the invention;
`FIG.9 depicts flow diagram of a particular embodiment of
`a method of viewing an alternative viewing angle;
`FIG. 10 depicts flow diagram of a particular embodiment
`of a method of filtering an image:
`FIG. 11 depicts flow diagram of a particular embodiment
`of a method of applying colors to an image; and
`FIG. 12 depicts flow diagram of a particular embodiment
`of a method of Zooming in on an image.
`
`DETAILED DESCRIPTION
`
`Mechanisms and techniques that provide a process for
`combining slices generated by medical imaging devices to
`create a Volume of interest and then presenting this Volume in
`a three-dimensional representation to a Head Display Unit
`(HDU) so that the Radiologist/Medical Professional (R/MP.
`also referred to herein as a user) can obtain a holistic view is
`described. Key image processing techniques are applied
`which enable the user to rotate and view the volume of interest
`from alternative viewpoints; to enable tissue subtraction to
`
`3
`dimensional viewing of images by a user. The computer read
`able medium includes instructions for selecting a Volume of
`interest from a collection of image slices and instructions for
`arranging the slices corresponding to the Volume of interest.
`The computer readable medium also includes instructions for
`selecting an initial viewing angle of the slices, instructions for
`selecting a viewpoint for a left eye and instructions for select
`ing a viewpoint for a right eye. Additionally, the computer
`readable medium includes instructions for displaying, in a
`head display unit (HDU), an image for the left eye based on
`the initial viewing angle, the view point for the left eye and the
`Volume of interest; and instructions for displaying, in the
`HDU, an image for the right eye based on the initial viewing
`angle, the view point for the right eye, and the Volume of
`interest and wherein the image for the left eye and the image
`for the right eye produce a three-dimensional image to the
`USC
`Still other embodiments include a computerized device,
`configured to process all the method operations disclosed
`herein as embodiments of the invention. In such embodi
`ments, the computerized device includes a memory system, a
`processor, communications interface in an interconnection
`mechanism connecting these components. The memory sys
`tem is encoded with a process that provides three-dimen
`sional viewing of images by a user as explained herein that
`when performed (e.g. when executing) on the processor,
`operates as explained herein within the computerized device
`to perform all of the method embodiments and operations
`explained herein as embodiments of the invention. Thus any
`computerized device that performs or is programmed to per
`form processing explained herein is an embodiment of the
`invention.
`Other arrangements of embodiments of the invention that
`are disclosed herein include Software programs to perform
`the method embodiment steps and operations Summarized
`above and disclosed in detail below. More particularly, a
`computer program product is one embodiment that has a
`computer-readable medium including computer program
`logic encoded thereon that when performed in a computer
`ized device provides associated operations providing three
`dimensional viewing of images by a user as explained herein.
`The computer program logic, when executed on at least one
`processor with a computing system, causes the processor to
`perform the operations (e.g., the methods) indicated hereinas
`embodiments of the invention. Such arrangements of the
`invention are typically provided as Software, code and/or
`other data structures arranged or encoded on a computer
`readable medium such as an optical medium (e.g.,
`CD-ROM), floppy or hard disk or other a medium such as
`firmware or microcode in one or more ROM or RAM or
`PROM chips or as an Application Specific Integrated Circuit
`(ASIC) or as downloadable software images in one or more
`modules, shared libraries, etc. The software or firmware or
`other Such configurations can be installed onto a computer
`ized device to cause one or more processors in the comput
`erized device to perform the techniques explained herein as
`embodiments of the invention. Software processes that oper
`ate in a collection of computerized devices, such as in a group
`of data communications devices or other entities can also
`provide the system of the invention. The system of the inven
`tion can be distributed between many software processes on
`several data communications devices, or all processes could
`run on a small set of dedicated computers, or on one computer
`alone.
`It is to be understood that the embodiments of the invention
`can be embodied strictly as a software program, as Software
`and hardware, or as hardware and/or circuitry alone. Such as
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`

`

`Case 6:20-cv-01699-GAP-DCI Document 1-1 Filed 09/16/20 Page 17 of 24 PageID 91
`
`US 8,384,771 B1
`
`5
`facilitate unobstructed viewing of a region of interest, to
`identify differing tissues with color schematics; and to Zoom
`in for optimal viewing.
`Referring to FIG. 1, a top-level diagram of a system 10 is
`shown. System 10 includes an imaging device 12 Such as a
`Computed Tomography scan. Alternatively, the equipment
`could be an MRI, fMRI, or PET, etc. The imaging device 12
`is in communication with a digital recording device 14. The
`digital recording device records each slice of imagery
`together with the metadata Such as Subject, time, position of
`the gurney and position of the Electromagnetic (EM) trans
`mitter and receiver arrays.
`A general purpose processor 16 interacts with the digital
`recording device 14 based on inputs it receives from the user.
`Also in communication with general purpose processor 16 is
`printer/film developer 26. Printer/film developer 26 enables a
`hard copy of the viewed image to be captured and printed.
`As shown, the general purpose processor 16 generates an
`image for the right eye 18 and an image for the left eye 20. The
`image to right eye 18 is based on the parameters set by the
`user, and the computed image is sent to the right eye. The
`image to left eye 20 is based on the parameters set by the user,
`the computed image 20 is sent to the left eye. It should be
`noted that this viewpoint position is offset from the position
`used for the right eye image.
`The images are sent to a head display unit 22 worn by the
`user 24. The head display unit 22 displays image 18 to the
`right eye. Note that multiple slices have been stacked, creat
`ing a Volumetric mass so that the image seen by the right eye
`is the volume of interest selected by the user. The head display
`unit 22 also displays image 20 to the left eye similar to the
`manner of which the right image was displayed. This pro
`duces a three-dimensional image to the user 24.
`Referring now to FIG. 2, a block diagram of the system 50
`comprising one particular embodiment of the invention is
`shown. The diagram indicates flow of data, imagery, and the
`like according to the legend. Medical imaging device 12
`utilizes EM energy that emanates from the device 12 and is
`also received by the device 12. The medical imaging device
`12 receives imaging commands from the controller 52 and
`provides imagery data to the data recording. The Medical
`device controller 52 provides control commands to the medi
`cal imaging device 12. A power Supply 54 provides power to
`the medical imaging device 12.
`45
`Digital recording device 14 records each of the slices of
`data collected by the medical imaging device 12, and may
`also be used to record medical records meta data. Digital
`recording device 14 interacts/exchanges data with the Gen
`eral purpose processor 16.
`50
`General purpose processor 16 interacts with the digital
`recording device 14 based on inputs it receives from the user
`through the Graphical User Interface (GUI) 58. The GUI 58
`allows interaction between the user with the general purpose
`processor 16 to initiate functions such as Zoom, rotate, filter
`tissue, and apply color schematics.
`The head display unit 22 includes a left head display unit
`which receives an image from general purpose processor 16
`and displays an image of the Volume of interest as it would be
`seen from a left eye perspective of the user. Similarly, the head
`display unit includes a right head display unit which receives
`an image from general purpose processor 16 and displays
`image of the volume of interest as it would be seen from a
`right eye perspective of the user. This produces a three-di
`mensional image to the user 24.
`Also shown is power supply 56. Power supply 56 supplies
`power to the various electronic equipment Such as digital
`
`30
`
`6
`recording device 14, general purpose processor 16, head dis
`play unit 22 and printer/film developer 26.
`Referring now to FIG. 3, left and right eye viewing angles
`are shown. Each eye will see the image from a different angle.
`The brain will interpret the left eye viewing angle's image and
`the right eye's viewing angle image together to give depth
`perception. Thus, a three-dimensional image will be seen. In
`this example, circle 102 represents a slice of image data.
`Triangles 106 and 108 extending from Left Eye Viewing
`Perspective (LEVP) 104 represent the angles (in the X-Y
`plane; theta and theta) from the viewpoint to the Voxels
`within the field of view. Triangles 112 and 114 extending
`from Right Eye Viewing Perspective (REVP) 110 represents
`the angles (in the X-Y plane; theta and theta) from the
`viewpoint to the voxels within the field of view (FOV).
`The viewpoints receive light from theta (horizontal) and
`alpha (vertical) angles within the field of view. In this diagram
`the field of view has a limit of 40 degrees spanning the
`horizontal and 30 degrees spanning the vertical. This birds
`eye view of the diagram shows the cones as triangles. These
`triangles represent geometric shapes from which to gather the
`Volumetric data and present the data instereoscopic fashion to
`the head display unit (HDU). The circle demonstrates an axial
`slice through the data. Note that standard reformatted slices
`(i.e., coronal vs. saggital vs. oblique) may be used as the slices
`for the process described above.
`FIG. 3 demonstrates the cross-section in the x-y plane at
`the slice at the same Z-value for the LEVP and REVP. It
`should be noted that from the viewpoints, other viewing
`angles will pass through multiple slices. It should also be
`noted that the viewing points may be altered so that three
`dimensional stereoscopic presentation may be obtained from
`any position and at any angle at the Volumetric data.
`Referring now to FIG. 4, the user selects a volume of
`interest, which is comprised of a series of slices. These slices
`may be obtained from any medical imaging device view of the
`patient. For example, they may be obtained from an axial,
`saggital, coronal or any oblique view. These slices are then
`stacked one upon another in the sequence in which they were
`generated. The pixels in the L HDU 26b and R HDU 26a are
`shown as 152 and 154 respectively. Each pixel in the display
`comes form a particular theta-alpha ray (shown in FIG. 3).
`Since theta in this example is the horizontal dimension and a
`FOV of 40 degrees is used in this example, the span of pixels
`in the horizontal direction represents pixels from -20 degrees
`(left limit in FOV) to +20 degrees (right limit in FOV). Since
`alpha in this example is the vertical dimension and a FOV of
`30 degrees is used in this example, the span of pixels in the
`vertical direction represents pixels from -15 degrees (down
`ward limit in FOV) to +15 degrees (upper limit FOV).
`Referring now to FIG. 5, a bird's eye view demonstrating
`angle theta is shown. Theta is the angle from the Left Eye
`Viewing Perspective (LEVP) 104 to the various voxels 102 in
`the Volumetric data set. In this example, the angle theta is
`angle in the x-y plane from the line extending exactly along
`the anterior-posterior direction to the LEVP to the voxels of
`interest in this slice. This will be the theta component for
`plotting a pixel based on the alpha-theta combination. Theta
`to the LEVP-Left-Most-Visible-Point is 0 degrees. Theta to
`the LEVP-Right-Most-Visible-Point is 18.4 degrees; thus,
`assuming the halfwidth display of 20 deg and 612 pixels, the
`LEVP-Right-Most-Visible-Point will be plotted at 559 (or
`53 pixels from the right most edge of the L HDU).
`FIG. 6 shows the volumetric data in the grey cylinder. The
`angles theta and alpha are shown extending from the LEVP
`104 to the voxels of interest. 102 Theta and alpha are angles
`from