United States Patent [19]
`Kalend et al.
`
`[54] APPARATUS RESPONSIVE TO MOVEMENT
`OF A PATIENT DURING TREATMENT/
`DIAGNOSIS
`
`[75] Inventors: Andre M. Kalend, Monroeville; Joel
`Greenberger, Sewickley; Karun B.
`Shimoga, Pittsburgh; Charalambos N.
`Athanassiou, Pittsburgh; Takeo
`Kanade, Pittsburgh, all of Pa.
`[73] Assignee: University of Pittsburgh of the
`Commonwealth System of Higher
`Education, Pittsburgh, Pa.
`
`[21] Appl. No.:715,834
`[22] Filed:
`Sep. 19, 1996
`[51] Int. Cl* ~~~~~~~~~~~ A61B 6/00
`[52] U.S. Cl. ….. 128/653.1
`[58] Field of Search ................................. 128/630, 653.1,
`128/660.03; 364/413.02, 413.13, 413.25,
`413.26; 356/375; 378/69, 205
`
`-
`
`[56]
`
`References Cited
`U.S. PATENT DOCUMENTS
`4,466,075 8/1984 Groch et al. ................... 364/413.26 X
`5,080,100 1/1992 Trotel ................................... 128/653.1
`5,103,823 4/1992 Acharya et al. ..................... 128/653.1
`5,214,711
`5/1993 Neely et al. ...........
`364/413.27 X
`5,295,483 3/1994 Nowacki et al. ......
`... 128/660.03
`5,389,101
`2/1995 Heilbrun et al. ...
`128/653.1 X
`5,398,684 3/1995 Hardy ................................... 128/653.1
`5,446,548 8/1995 Gerig et al.
`128/653.1 X
`5,482,042
`1/1996 Fujita ................................... 128/653.1
`5,558,430 9/1996 Bova et al. .......................... 128/653.1
`OTHER PUBLICATIONS
`Active Shape Models—“Smart Snakes’, T.F. Cootes and C.J.
`Taylor, pp. 267–275, Proceedings of European Conference
`on Computer Vision, Genoa, Italy, 1992.
`
`
`
`US005.727554A
`[11] Patent Number:
`[45] Date of Patent:
`
`5,727,554
`Mar. 17, 1998
`
`Training Models of Shape from Sets of Examples, T.F.
`Cootes, C.J. Taylor, D.H. Cooper, and J. Graham, pp. 8–18,
`Proceedings of European Conference on Computer Vision,
`Genoa, Italy, 1992.
`A Computational Framework and an Algorithm for the
`Measurement of Visual Motion, P. Anandan, pp. 283—310,
`International Journal of Computer Vision, 2, 1989.
`Feature Extraction from Faces Using Deformable Tem
`plates, A.L. Yuille, P.W. Hallinan, and D.S. Cohen, pp.
`99–111, International Journal of Computer Vision, 8:2,
`1992.
`Computer and Robot Vision, vol. I, R. M. Haralick and L. G.
`Shapiro, pp. 328–353, Library of Congress Cataloging-in
`—Publication Data, 1992.
`Motion Tracking with an Active Camera, D. Murray and A.
`Basu, pp. 449–459, IEEE Transactions on Pattern Analysis
`and Machine Intelligence, vol. 16, No. 5, May 1994.
`
`Primary Examiner—Francis Jaworski
`Attorney, Agent, or Firm—Richard V. Westerhoff, Eckert
`Seamans Cherin & Mellott, LLC
`[57]
`ABSTRACT
`A camera generates digital image signals representing an
`image of one or more natural or artificial fiducials on a
`patient positioned on treatment or diagnosis equipment. A
`processor applies multiple levels of filtering at multiple
`levels of resolution to repetitively determine successive
`fiducial positions. A warning signal is generated if move
`ment exceeds certain limits but is still acceptable for treat
`ment. Unacceptable displacement results in termination of
`the treatment beam. Tracking templates can be generated
`interactively from a display of the digital image signals or
`through automatic selection of an image having the median
`correlation to an initial template. A gating signal synchro
`nized to patient breathing can be extracted from the digital
`image signals for controlling the radiation beam generator.
`
`22 Claims, 12 Drawing Sheets
`
`Varian Exhibit 2004, Page 001
`
`

`
`U.S. Patent
`
`Mar 17, 1998
`
`Sheet 1 of 12
`
`5,727,554
`
`
`
`FIG. 1
`
`Varian Exhibit 2004, Page 002
`
`

`
`U.S. Patent
`
`Mar 17, 1998
`
`Sheet 2 of 12
`
`5,727,554
`
`25
`
`39
`
`FIG. 2
`
`
`
`
`
`
`
`
`
`
`
`
`
`PERFORM BRACKETING
`ON SELECTED POINT/MATCHES
`|N H|-RES IMAGE
`
`FILIER SECONDARY
`MATCHES/PONIS WITHIN THE
`SAME IMAGE NEIGHBORHOOD
`USING MIN–SUPPRESSION
`
`GO TO BLOCK 116
`
`FIG. 9
`
`Varian Exhibit 2004, Page 003
`
`

`
`U.S. Patent
`
`Mar 17, 1998
`
`Sheet 3 of 12
`
`5,727,554
`
`41 2.2%
`
`43
`
`FIG. 3
`
`13-A /
`
`5
`
`37
`
`==,
`
`NZJ-35
`T
`
`BEAM
`GENERATOR
`63
`
`DIGITIZER
`
`PATIENT MOTION
`DETECTOR
`
`GATING SIGNAL
`GENERATOR
`
`47
`
`|
`|
`|
`
`|
`|
`|
`|
`|
`|
`
`
`
`Varian Exhibit 2004, Page 004
`
`

`
`U.S. Patent
`
`Mar 17, 1998
`
`Sheet 4 of 12
`
`5,727,554
`
`
`
`Varian Exhibit 2004, Page 005
`
`

`
`U.S. Patent
`
`Mar 17, 1998
`
`Sheet 5 of 12
`
`5,727,554
`
`START
`
`DETECT FIDUCALS ON PATIENT 800X
`USING CURRENT CAMERA |MAGE
`110
`FINE-TUNE TRACKING TEMPLATES USING CURRENT
`CAMERA (MAGE FOR SPECIFIC PATIENT AND
`ENVIRONMNETAL CONDITIONS
`120
`Y/HAS USER TERMINATEDY-130
`MONITORING2
`N
`
`|
`
`0
`
`TRACK FIDUCIAL IN
`NEW IMAGE
`
`140
`
`|S
`FIDUCIAL LOST?
`
`ATTEMPT TO RE-ACQUIRE
`FIDUCIAL
`
`IS FIDUCIAL
`RE-ACQUIRED7
`
`PRESENT ALARM AND
`FEEDBACK SIGNALS
`
`REMOVE THAT FIDUCIAL
`FROM TRACKING
`
`
`
`-
`
`170
`
`
`
`
`
`
`
`
`
`
`
`
`
`Y
`
`( No D
`
`ARE THERE ANY
`FIDUCIALS LEFTP
`N
`ABORT
`
`210
`
`FIG. 6
`
`Varian Exhibit 2004, Page 006
`
`

`
`U.S. Patent
`
`Mar 17, 1998
`
`Sheet 6 of 12
`
`5,727,554
`
`|NTERACTIVELY
`SELECT
`FIDUCALS
`S 113
`
`|
`
`0
`
`USE
`"IDEALIZED"
`|MAGE TEMPLATES
`
`USE PRE-STORED
`|MAGE TEMPLATES
`FOR PATIENT
`
`111
`
`112
`
`
`
`SEARCH THE CURRENT |MAGE
`IN LOW-RES (1/3)
`FOR CAND|DATE MATCHES
`OF ALL TEMPLATES
`114
`
`L00ALIZE CANDIDATE MATCHES
`|N HIGH-RES IMAGE
`115
`
`SELECT THE K–BEST MATCHES
`AS THE MOST RELIABLE
`FIDUCIALS
`
`116
`
`USER EDIT POINTS
`
`117
`
`GO TO BLOCK 120
`
`FIG. 7
`
`Varian Exhibit 2004, Page 007
`
`

`
`U.S. Patent
`
`Mar 17, 1998
`
`Sheet 7 of 12
`
`5,727,554
`
`RASTER-SCAN THE IMAGE
`SELECTING POINTS USING
`py
`3)
`SPARSE-SAMPLING
`
`114.1
`
`
`
`RASTER-SCAN THE IMAGE
`SELECTING POINTS USING
`INTEREST OPERATORS FOLLOWED
`BY THRESHOLDING ALBEDO
`114.2
`
`14
`
`PERFORM TEMPLATE MATCHING
`FOR ALL TEMPLATES
`SELECTED IN 111–113 AT
`SELECTED IMAGE POINTS
`USING NORMALIZED CORRELATION
`114.3
`FILTER UN-WANTED
`PONT/MATCHES USING
`THRESHOLDING ON THE
`NORM-CORR VALUE [0,---,1]
`114.4
`
`
`
`
`
`PERFORM LOCALIZATION USING
`BRACKETING & INTERPOLATION
`ON REMAINING POINT/MATCHES
`114.5
`
`GO TO BLOCK 115
`
`F/G. 8
`
`Varian Exhibit 2004, Page 008
`
`

`
`U.S. Patent
`
`Mar 17, 1998
`
`Sheet 8 of 12
`
`5,727,554
`
`FOR
`EACH
`TEMPLATE
`FAMILY
`
`120
`
`SELECT "MEDIAN" POINT/MATCH
`FROM FIDUCIALS DETECTED
`USING THE SAME INITIAL
`TEMPLATE
`
`
`
`121
`ACQUIRE RELEWANT IMAGE
`PORTION AS THE
`NEW TEMPLATE
`
`
`
`
`
`
`
`
`
`
`
`
`
`122
`RECORD POSITION/INT-OPER
`ALBEDO/NORM-CORR FOR ALL
`RELEVANT POINT/MATCHES
`USING THE NEWLY ACQUIRED
`TEMPLATE
`
`
`
`123
`RECORD THE CURRENT SPATIAL
`PATTERN OF THE FIDUCIALS
`(POINT/MATCHES)
`
`124
`
`GO TO BLOCK 130
`
`FIG 10
`
`Varian Exhibit 2004, Page 009
`
`

`
`U.S. Patent
`
`Mar 17, 1998
`
`Sheet 9 of 12
`
`5,727,554
`
`ESTIMATE NEW POSITION
`OF FIDUCIAL
`
`
`
`14|
`
`LOW-RES IMAGE LOCALIZATION
`OF FIDUCIAL POSITON
`
`140
`-
`
`142
`H|-RES IMAGE LOCALIZATION
`OF FIDUCIAL POSITON
`
`143
`
`
`
`
`
`
`
`GO TO BLOCK 140
`
`FIG 11
`
`
`
`
`
`
`
`
`
`
`
`
`
`HAS FIDUCIAL BEEN
`LOCATED WITHIN SPECIFIED
`|MAGE WINDOW2
`
`
`
`CONSTANCY FILTERS
`1. NORM-CORR WALUE INTERWAL
`2. INT-OPER VALUE INTERVAL (ALBEDO)
`3. IMAGE-L|M||
`F|DUCIAL PASSED ALL OF THESE7
`Y
`
`GO TO BLOCK 200
`
`
`
`GO TO BLOCK 160
`
`FIG. 14
`
`Varian Exhibit 2004, Page 010
`
`

`
`U.S. Patent
`
`Mar. 17, 1998
`
`Sheet 10 of 12
`
`5,727,554
`
`RASTER-SCAN THE IMAGE
`WINDOW SELECTING POINTS
`USING "SPARSE-SAMPLING"
`
`
`
`
`
`
`
`RASTER-SCAN THE º E
`|NTEREST OPERATORS WITH WALU
`CLOSE TO THAT OF FIDU}|AL N
`PREVIOUS TRACKING STEPS
`1322
`
`PERFORM NORM-CORR TEMPLATE
`MATCHING AND SELECT THE
`BEST MATCH
`
`.
`132
`
`PERFORM BRACKETING ON THE
`POSITION OF THE BEST MATCH
`
`
`
`
`
`
`
`GO TO BLOCK 1.33
`
`FIG. 72
`
`PERFORM BRACKETING, IN THE HI-RES
`IMAGE ON THE POSITION OF THE
`CAND|DATE BEST MATCH
`
`
`
`
`
`
`
`
`
`CALCULATE NORM-CORR/INT-OPER/POS, OF
`THE BEST MATCH; IF MATCH FOUND
`153.2
`
`133
`
`
`
`CALCULATE SUB—PIXEL ACCURACY OF
`POS. IF NEEDED/SELECTED; IF MATCH FOUND
`133.3
`FILTER CCD-JITTER OUT OF POS.;
`|F NEEDED
`
`133.4
`
`
`
`GO TO BLOCK 134
`
`F/G, 73
`
`Varian Exhibit 2004, Page 011
`
`

`
`U.S. Patent
`
`Mar 17, 1998
`
`Sheet 11 of 12
`
`5,727,554
`
`ESTIMATE NEW POSITION
`OF FIDUCIAL
`
`181
`
`
`
`RASTER-SCAN THE IMAGE IN
`HI-RES, USING SPARSE
`SAMPLING, SELECT BEST
`MATCH; IF ANY FOUND
`
`180
`-
`
`PERFORM BRACKETING AROUND
`THE POSITION OF THE
`BEST MATCH; IF ANY FOUND
`183
`
`CALCULATE
`NORM-CORR/NI-OPER/POS
`OF FIDUCAL; IF ANY FOUND
`184
`CALCULATE SUB-PIXEL
`ACCURACY OF FIDUCIAL
`POSITION; IF NEEDED
`
`185
`
`UPDATE THE NUMBER OF
`SUCCESSIVE RE-ACQ. ATTEMPTS
`FOR THAT FIDUCIAL
`
`.
`
`186
`
`GO TO BLOCK 190
`
`FIG. 15
`
`Varian Exhibit 2004, Page 012
`
`

`
`U.S. Patent
`
`Mar 17, 1998
`
`Sheet 12 of 12
`
`5,727,554
`
`FROM BLOCK 150
`
`FROM BLOCK 190
`
`FROM BLOCK 170
`
`ESTIMATE DIRECTION AND DISTANCE
`TRAVELLED BY EACH CURRENTLY
`ACTIVELY TRACKED FIDUCIAL
`SINCE THE DETECTION STEP
`201
`COMPARE THE SPATIAL PATTERN
`OF THE ACTIVELY TRACKED
`FIDUCIALS WITH THE INITIAL AND
`PREVIOUS PATTERNS
`
`202
`ESTIMATE ANY "QUAS-PERIODIC"
`MOTION ASSOCIATED WITH THE
`INDIVIDUAL FIDUCIALS AND/OR
`THE SPATIAL PATTERN
`203
`COMPUTE AND DISPLAY ALARM
`WARNINGS, ALARM STATES, AND
`CONTROLLING FEEDBACK SIGNALS
`TO THE USER OR TO CONNECTED
`EQUIPMENT
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`204
`
`GO TO BLOCK 210
`
`FIG. 16
`
`Varian Exhibit 2004, Page 013
`
`

`
`5,727,554
`
`1
`APPARATUS RESPONSIVE TO MOVEMENT
`OF A PATIENT DURING TREATMENT/
`DLAGNOSIS
`
`10
`
`15
`
`20
`
`25
`
`BACKGROUND OF THE INVENTION
`1. Field of the Invention
`This invention relates to medical use of radiation for
`treatment and diagnosis, and more particularly to detection
`and response to patient movement during radiological treat
`ment and diagnosis.
`2. Background Information
`Conventional radiotherapy treatment relies on simple
`patient setup techniques. These techniques use stationary
`and a limited number of radiation fields, which are often
`much wider than the tumor or volume, thus effectively
`compensating for the possibility of a tumor geometric miss.
`Consequently, a substantial amount of healthy tissue is
`irradiated and becomes a radio-biological dose limiting
`factor in tumor control.
`Modern conformal dynamic radiotherapy attempts to
`overcome the above radio-biological limitation by tight
`margin conformation of radiation dose distribution tailored
`to the three-dimensional tumor volume by the use of
`computer-control multibeam conformal dynamic radio
`therapy (CCRT). Consequently, the accuracy in patient
`position, knowledge of the movement of a patient including
`substantial motion of internal organs such as with breathing
`is of primary importance. In addition to patient movement
`which would cause the tight beam to miss the tumor, it is
`important to be able to detect patient movement which could
`cause a collision between the patient and the linear
`accelerator, which is repeatedly repositioned to establish the
`multiple treatment beams.
`There is a need therefore for apparatus for detecting
`patient movement on radiological treatments/diagnostic
`equipment.
`There is a particular need for such apparatus which can
`detect submillimeter patient movement in real time.
`There is also a need for such apparatus which can detect
`patient movement initiated from various treatment positions.
`There is also a need for such apparatus which can detect
`patient movement under varying lighting conditions.
`There is a further need for such apparatus which can
`discriminate movement associated with patient breathing
`from other movement and accommodate therefor.
`
`30
`
`35
`
`45
`
`2
`As another aspect of the invention, the means generating
`an output includes means indicating movement of the at
`least one passive fiducial relative to at least one selected
`level of displacement. Preferably, the output means gener
`ates a warning that movement exceeds a first displacement
`and includes means providing a signal for terminating
`radiation treatment when the movement exceeds a second
`greater displacement. Preferably, the means providing an
`indication of movement includes a display generating an
`image of the patient and the fiducials, together with an
`indication of movement relative to the first and second
`displacements.
`As yet another aspect of the invention, the means deter
`mining movement of the passive fiducials includes means
`detecting movement associated with patient breathing and
`random movement. The movement associated with patient
`breathing can be used to generate a gating signal synchro
`nized to patient breathing. This gating signal can then be
`used to actuate the beam generator only during selected parts
`of the breathing cycle.
`BRIEF DESCRIPTION OF THE DRAWINGS
`A full understanding of the invention can be gained from
`the following description of the preferred embodiments
`when read in conjunction with the accompanying drawings
`in which:
`FIG. 1 is an isometric view of apparatus in accordance
`with the invention for implementing conformal dynamic
`radiotherapy.
`FIG. 2 is a plan view of a patient reclining on a couch
`which forms part of the apparatus of FIG. 1 and illustrating
`the placement of fiducials in accordance with the invention.
`FIG. 3 is a perspective view of a preferred fiducial used
`in implementation of the invention.
`FIG. 4 is a functional diagram illustrating implementation
`of the invention.
`FIG. 5 is an illustration of a display which is generated by
`the apparatus of FIG. 1 in implementation of the invention.
`FIGS. 6–16 are flow charts of software used in imple
`mentation of the invention.
`FIG. 17 is an illustration of an interest operator which can
`be used in implementation of the invention.
`DESCRIPTION OF THE PREFERRED
`EMBODIMENT
`FIG. 1 illustrates a radiotherapy treatment system 1 in
`which the invention is implemented. This system 1 includes
`a machine 3 having a gantry 5 pivotally mounted on a
`machine base 7 for rotation about a horizontal axis 9. The
`gantry 5 has a first arm 11 carrying a collimator 13 which
`directs a beam of high energy radiation 15, such as a beam
`of high energy photons, along a path which is perpendicular
`to and passes through an extension of the axis of rotation 9.
`This intersection is referred to as the isocenter 17. In some
`machines, a portal imager 19 is mounted on a second arm 21
`on the opposite end of the gantry in alignment with the
`radiation beam 15. The portal imager 19 records radiation
`which is not absorbed by the patient.
`The isocenter 17 serves as the origin of a coordinate
`system for room space. As can be seen, the X axis coincides
`with the axis of rotation 9 of the gantry. Thus, as the gantry
`5 rotates it defines a plane of treatment containing the Y and
`Z axes.
`The machine 3 further includes a patient positioning
`assembly 23, which includes a couch 25 mounted on a
`
`SUMMARY OF THE INVENTION
`These needs and others are satisfied by the invention
`which is directed to apparatus responsive to movement of a
`patient which identifies and tracks movement of at least one
`passive fiducial on the patient. The apparatus applies mul
`tiple levels of filtering which can include: correlation, pref
`erably normalized correlation, sparse sampling, bracketing
`and interpolation, and minima suppression to rapidly iden
`tify the location of the at least one fiducial. The multiple
`levels of filtering are applied at multiple levels of resolution
`of the digital image signals.
`Interest operators can be used in combination with tem
`plates to locate the positions of the passive fiducials. The
`templates can be selected interactively by a user from a
`display generated by the digital image signals. Alternatively,
`the template used for tracking is selected from images
`generated using an initial template. Rather than using the
`image which best matches the initial template, the template
`with a median match is selected.
`
`50
`
`55
`
`65
`
`Varian Exhibit 2004, Page 014
`
`

`
`5
`
`10
`
`15
`
`20
`
`25
`
`35
`
`45
`
`3
`support 27 for vertical, lateral and longitudinal movement
`relative to the support. The support 27 is mounted on a
`turntable 29, which has its axis 31 vertically aligned under
`the isocenter 17 and concentric with the Z axis. With this
`arrangement, the patient positioning assembly 23 has four
`degrees of freedom: translation in the X, Y and Z axes of
`room space and rotation about the Z axis. Thus, the patient
`is not rotated about the longitudinal axis of the couch or
`tilted about a horizontal axis extending transversely through
`the couch. However, with the addition of rotation of the
`gantry in the Y-Z treatment plane, the radiation beam 15 can
`be directed through a patient reclining on the couch 25 in
`any desired direction. A computer 33 controls movement of
`the patient positioning assembly 23 and the gantry 5 for
`establishing the progression of high energy treatment beams
`used in practicing conformal radiation therapy.
`As previously discussed, in conformal radiation therapy
`the beam 15 is tightly conformed by the collimator 13 to the
`specific tumor to be treated. Thus, movement of the patient
`on the couch 25 of the patient position assembly 23 can
`cause misalignment of the radiation beam 15 with the tumor.
`This not only degrades treatment of the tumor but also
`exposes surrounding healthy tissue to unwanted levels of
`radiation. In addition, normal breathing by the patient can
`cause movement of internal organs by an amount which
`would result in misalignment of the beam. For instance, a
`tumor on the lower portion of the lung can move several
`centimeters during normal breathing. Slight movement of
`the patient can be tolerated; however, treatment should be
`terminated if acceptable tolerances of movement are
`exceeded. Furthermore, excessive movement by the patient
`can also cause a collision between the patient and the gantry
`as the patient positioning assembly 23 and gantry are
`positioned for successive treatment beams.
`The invention employs a vision system 34 to measure and
`respond to patient movement. The vision system 34 includes
`at least one video camera 35. Preferably, multiple cameras
`are used. In the exemplary embodiment of the invention a
`first camera 35, is mounted on the first arm 11 of the gantry
`5 adjacent the collimator 13 and is aimed to capture an
`image of a patient 37 positioned on the couch 25, as shown
`in FIG. 2. As the camera 351 will be below the couch 25 for
`some positions of the gantry 5, a second camera 352 is fixed
`to the ceiling over the patient positioning assembly 23. The
`field of view of this camera 352 will be blocked when the
`gantry 5 is at the top of its arc. Thus, the patient is visible
`to at least one camera 35 at all times. Additional cameras 35
`could be provided, such as cameras laterally displaced from
`the patient positioning assembly 23 to provide more sensi
`tivity to movement along the axis of, for instance, the
`camera 352. However, as will be discussed below, a single
`camera can detect three-dimensional movement, including
`movement toward and away from the camera which is
`detected as a change in the size of the image.
`In the exemplary embodiment of the invention, natural or
`artificial fiducials are used to detect patient movement.
`Natural fiducials could be scars or other prominent features
`of the patient. The preferred fiducial 39 shown in FIG. 3 is
`a sphere 41 covered with a material having a lambertian
`surface. Such a surface is highly reflective under low light
`conditions, yet provides a uniform scattered reflection with
`no highlights. The sphere 41 is attached to the center of a
`non-reflective base 43 which is secured to the patient's skin,
`such as by an adhesive.
`In principle. only one fiducial 39 is required. As a prac
`tical matter, it is advantageous to provide multiple fiducials
`placed on the patient so as to detect any movement of the
`
`5,727,554
`
`4
`critical locations. Thus, as shown in FIG. 2, by way of
`example, four fiducials 39 are placed on the patient’s chest.
`Natural skin markings could be used in addition to the
`artificial fiducials shown in FIG, 3. If more than one camera
`35 is used, each tracks as many of the fiducials 39 as it can
`See.
`FIG. 4 is a functional diagram of the invention. The
`camera(s) 35 capture an image of the fiducials 39 on the
`patient 37 reclining on the patient positioning assembly 23.
`The image captured by the camera 35 is digitized by
`digitizer 45 to generate digital image signals. These digital
`image signals are 0 to 255 gray scale signals for each camera
`pixel. The digital image signals are processed by a processor
`which includes a patient motion detector 47. Patient motion
`detector 47 is implemented in the computer 49 shown in
`FIG. 1. The computer 49 includes a monitor 51 which
`generates a display 53, an example of which is shown in
`FIG. 5. The man machine interface 55 for the computer 49
`includes a keyboard 57 and a pointing device 59, such as a
`mouse or trackball.
`As will be discussed fully, the patient motion detector 47
`detects and identifies the fiducials 39 and then tracks their
`movement. Movement within a certain narrow tolerance is
`acceptable, while larger movements are unacceptable. Vis
`ible and/or audio warnings of these two classifications of
`movement can be generated. A gating signal generator 61
`responds to unacceptable movement to disable the beam
`generator 63. This unacceptable movement which would
`terminate the radiation beam can be movement which dis
`places the target tumor so that it is missed by the radiation
`beam, or could be movement which would cause a collision
`between the patient and the gantry 5 during movement of the
`machine from one treatment beam to the next. In the former
`case, the gating signal generator 61 could re-enable the beam
`generator, if the patient returns to the proper position. For
`instance, a large sigh could temporarily displace the target
`area by an unacceptable amount. In accordance with another
`aspect of the invention, the patient motion detector 47 can
`track patient breathing and extract such quasi-periodic
`motion from random patient motion. Gating of the beam
`generator can then be synchronized with patient breathing.
`For instance, a tumor on the lung could move up to 4 to 5
`centimeters during patient breathing. This is an unacceptable
`amount of movement. However, by synchronizing genera
`tion of the radiation beam with breathing, the tumor can be
`repetitively irradiated at a fixed position during the breathing
`cycle.
`As shown in FIG. 5, the display 53 presents an image of
`the patient 37 with the fiducials 39 appearing prominently.
`An indicator 65, such as the square shown, surrounds each
`fiducial and is color coded to indicate the state of motion of
`the fiducial. The fiducial with the largest displacement such
`as 39a is singled out by a distinctive marker, such as a red
`square 65a, while the remaining markers are green squares
`in the exemplary system. The display also includes a traffic
`light 67 having a green section 67g, a yellow section 67y and
`a red section 67r. When motion of the fiducials is within
`preferred tolerances, the green section 67g of the traffic light
`is on. For motion which is outside the normal range, but
`which is still acceptable, the yellow section 67y is on. The
`traffic light turns red when the motion of any of the fiducials
`is approaching the unacceptable. A scale 69 along the side of
`the display 53 indicates in bar graph form the percentage of
`maximum allowable displacement of the fiducial of maxi
`mum displacement. Thus, for instance, if the red light 67r is
`illuminated and the bar graph 71 indicates 80%, the fiducial
`with maximum displacement has moved by a distance which
`
`50
`
`55
`
`65
`
`Varian Exhibit 2004, Page 015
`
`

`
`5,727,554
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`5
`is four fifths of the way through the acceptable displacement.
`The green, yellow and red regions need not be equal as
`shown in the example.
`Detection of motion of a patient using passive fiducials
`requires an implementation which is robust enough to
`accommodate for the variations in the shapes, appearance
`and lighting conditions to which the fiducials are subjected
`and, at the same time, is fast enough to provide real time
`tracking of patient movement. The invention satisfies these
`requirements by utilization of successive levels of filtering
`and templates which are modified to accommodate for actual
`conditions. The result is a system which can track patient
`movement at 20 Hz or better.
`Flow charts of suitable software 100 for implementing the
`invention are illustrated in FIGS. 6–16. FIG. 6 illustrates the
`main routine of the software 100 and includes detecting
`fiducials on the patient’s body is in the current camera image
`at 110. As will be described, this is accomplished utilizing
`templates. The templates are then fine tuned at 120 for the
`specific patient and environmental conditions. As long as the
`user desires monitoring as determined at 130, a loop is
`entered in which each individual fiducial is tracked as
`indicated at 140. It is possible that a fiducial can be lost by
`the tracking system. This could occur, for instance, if the
`patient moves so that a fiducial is blocked from the camera’s
`view, or the patient moves a hand through the line of sight
`of the camera. Also, a fiducial may be temporarily lost by
`rapid movement or adverse lighting conditions. If a fiducial
`is lost, as determined at 150, a number of attempts can be
`made to reacquire it. If the fiducial is not reacquired within
`a reasonable time, however, it is removed from tracking as
`indicated by 160 and 170. If the selected number of attempts
`to reacquire, such as for example, five, have not been
`reached, an attempt is made to reacquire the fiducial at 180.
`If the fiducial is reacquired at 190, then a routine is run at
`200 to generate any alarm if needed, and gating signals for
`the accelerator or beam generator 63 as indicated at 200. As
`long as any fiducials remain to be tracked as indicated at
`210, the tracking loop is repetitively run.
`FIG. 7 illustrates the general routine 110 for detecting the
`fiducials 39 in the image represented by the digital image
`signals. As mentioned, templates are used to identify the
`locations of the fiducials. The templates indicate what the
`pattern of digital signals representing the fiducial should
`look like. The size of the templates used must be considered.
`Larger templates improve the accuracy but take longer to
`process. In the exemplary system, templates 40 pixels square
`have been utilized. There are several ways in which the
`templates can be generated. As indicated at 111 in FIG. 7,
`idealized image templates can be utilized. In addition to such
`idealized templates or in place thereof, pre-stored image
`templates for the patient can be used as indicated at 112.
`Such pre-stored templates are used, for instance, for natural
`fiducials such as scars. One template is used for each family
`of fiducials. For instance, if all of the fiducials are the
`preferred fiducials such as shown in FIG. 3, only one
`template is required because all of the fiducials in the family
`will generate a similar image.
`In addition, templates can be selected interactively by the
`user at 113. This is accomplished by using the mouse or
`trackball 59 to click on the center of a representation of the
`fiducial on the display 53.
`Where the idealized or pre-stored templates are utilized,
`a multiresolution pyramid is used to locate the fiducials in
`the image using the templates. Thus, as indicated at 114, a
`search is made of the current image in low resolution for
`
`6
`candidate matches of all template families. In the exemplary
`embodiment of the invention, one-third resolution is used at
`this point. Matches are made using a normalized correlation
`between the template and the image. The matches found in
`low resolution are then verified and localized in high reso
`lution at 115. The K best matches are then selected as the
`most reliable fiducials at 116 where K equals the number of
`fiducials to be tracked. The user is then given the opportunity
`at 117 to edit the detected location of fiducials found either
`through use of the idealized or pre-stored templates or
`templates generated interactively.
`The details of the low resolution detection routine per
`formed in block 114 of FIG. 7 is shown in FIG. 8. As shown
`at 114.1, the image can be raster scanned selecting points
`using sparse sampling. In raster scanning pixels are consid
`ered successively along each line, line-by-line in increments
`of one, while in sparse sampling the increment is greater
`than one. Alternatively, the image can be raster scanned as
`indicated at 114.2, selecting candidate points using interest
`operators followed by thresholding. Interest operators are
`simple patterns which emphasize gray scale characteristics
`of a particular fiducial. An example is shown in FIG. 17,
`where the fiducial is a light circle 73 on a dark background
`75. The interest operator 77 could be, for instance, the one
`pixel value 79 in the center having a gray scale value
`matching that of the light circle 73, and the four pixels 81 at
`the cardinal points having gray scale values similar to that of
`the background 75. Such interest operators permit rapid
`searching of the image and should be selected as to assure
`identifying all of the fiducials in the family. They will most
`likely also generate additional candidate points. Returning to
`FIG. 8, the interest operator generated value in the exem
`plary system is the relative albedo. The relative albedo of
`each point in the low resolution scan is compared to a
`threshold value to select candidate points.
`For each candidate point, a template matching is per
`formed at 1143, using a normalized correlation. Unwanted
`point matches are then filtered out at 114.4 using threshold
`ing on the normalized correlation value. In the exemplary
`embodiment, a normalized correlation of 0.75 was used as
`the threshold. Bracketing and interpolation are then used at
`114.5 to localize the remaining point/matches. In imple
`menting bracketing, a rectangular image window is selected
`within which the desired point match will definitely lie.
`Then by interpolating between the correlation values of
`points on the border of the selected window along with its
`center, a new estimate of the location of the point match is
`calculated. This process is repeated with successively
`smaller windows centered on the new estimate of the
`location of the point match until a singular point is reached.
`In the exemplary system, the interpolation is performed
`using a two-dimensional Gaussian distribution.
`FIG. 9 illustrates the techniques for verifying the candi
`date matches in high-resolution indicated at 115 in FIG. 7.
`Bracketing is performed on the selected matches in high
`resolution as indicated at 115.1. These points are then
`filtered at 115.2 within the same image neighborhood using
`minima suppression. In implementing minima suppression,
`for each point which has been a match, an area the size of
`the template is centered on the point. A point is selected as
`a further candidate match only if it is the best correlation
`with the template within the template window.
`An important aspect of the invention is the fine tuning of
`the tracking templates called for at 120 in FIG. 6. FIG. 10
`illustrates the details of fine tuning the templates. As indi
`cated at 121, the median point/match from fiducials detected
`using the same initial template is selected. For example, if
`
`45
`
`50
`
`55
`
`65
`
`Varian Exhibit 2004, Page 016
`
`

`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`7
`there are three point matches for a fiducial family, the match
`having the middle value of correlation is selected. Notice
`that the match with the best correlation is not selected as it
`is likely to eliminate some valid matches. This technique
`adapts the selection of the template to be used for tracking
`to the actual conditions existing at the time of the selection.
`The relevant image portion is then acquired as the new
`template at 122, and the position, the interest operator value
`and the normalized correlation for all relevant point/matches
`using this newly acquired template is then recorded at 123.
`The steps 121–123 are accomplished for each template
`family. Then, the current special pattern of all the fiducials
`determined by the point/matches, is recorded at 124.
`The program then enters the tracking loop at block 130 in
`FIG. 6. The routine for continuous tracking, which is called
`at 140 in FIG. 6 is illustrated in FIG. 11. The new position
`of the fiducial is estimated at 131 by projecting a ve