(12) United States Patent
`(12) United States Patent
`Gutkowicz-Krusin et al.
`Gutkowicz-Krusin et al.
`
`111111111111111111111111111110I110111111191fili11111111111111111111111111
`
`
`USOO6208749B1
`(10) Patent No.:
`US 6,208,749 B1
`US 6,208,749 B1
`(to) Patent No.:
`(45) Date of Patent:
`(45) Date of Patent:
`*Mar. 27, 2001
`*Mar. 27, 2001
`
`(54) SYSTEMS AND METHODS FOR THE
`(54) SYSTEMS AND METHODS FOR THE
`MULTISPECTRAL IMAGING AND
`MULTISPECTRAL IMAGING AND
`CHARACTERIZATION OF SKIN TISSUE
`CHARACTERIZATION OF SKIN TISSUE
`Y
`(75) Inventors: Dina Gutkowicz-Krusin, Princeton, NJ
`Inventors: Dina Gutkowicz-Krusin, Princeton, NJ
`(75)
`(US); Marek Elbaum, Dobbs Ferry;
`(US); Marek Elbaum, Dobbs Ferry;
`Michael Greenebaum, Brooklyn, both
`Michael Greenebaum, Brooklyn, both
`s
`s
`of NY (US); Adam Jacobs, Glen
`R. (Sam Jacobs, Glen
`Ridge, NJ (US)
`1Clge,
`
`(*) Notice:
`( * ) Notice:
`
`(73) Assignee: Electro-Optical Sciences, Inc.,
`(73) ASSignee: Esters, sinces Inc.,
`Irvington, NY (US)
`glon,
`This patent issued on a continued pros-
`This patent issued on a continued pros-
`ecution application filed under 37 CFR
`ecution application filed under 37 CFR
`1.53(d), and is subject to the twenty year
`1.53(d), and is subject to the twenty year
`patent term provisions of 35 U.S.C.
`patent term provisions of 35 U.S.C.
`154(a)(2)
`154(a)(2).
`
`Subject to any disclaimer, the term of this
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`U.S.C. 154(b) by 0 days.
`(21) Appl. No.: 09/032,450
`(21) Appl. No.: 09/032,450
`(22) Filed:
`Feb. 27, 1998
`(22) Filed:
`Feb. 27, 1998
`O
`O
`Related U.S. Application Data
`Related U.S. Application pal d
`(60)
`b
`(60) Provisional application No. 60/039,218, filed on Feb. 28,
`Provisiona app ication No. 60/039,218, filed on Feb. 28,
`1997, and provisional application No. 60/039,407, filed on
`17 gyisional application No. 60/039,407, filed on
`Feb. 28, 1997.
`• 4Y-2 7
`(51) Int. Cl. .............................. G06K 9/00; GO6K 9/34;
` GO6K 9/00; GO6K 9/34;
`(51) Int. C1.7
`GO1J 3/40
`G01J 3/40
`(52) U.S. Cl. .......................... 382/128; 382/165; 382/173;
`(52) U.S. Cl.
` 382/128; 382/165; 382/173;
`356/303
`356/303
`(58) Field of Search ..................................... 382/128, 162,
`(58) Field of Search
` 382/128, 162,
`382/173, 274, 165; 358/504; 356/303,346
`382/173, 274, 165; 358/504; 356/303, 346
`
`(56)
`(56)
`
`References Cited
`References Cited
`U.S. PATENT DOCUMENTS
`U.S. PATENT DOCUMENTS
`
`3,335,716
`3,335,716
`4,170,987
`4,170,987
`
`8/1967 Alt et al. .
`8/1967 Alt et al..
`10/1979 Anselmo et al. .................... 600/475
` 600/475
`10/1979 Anselmo et al.
`
`
`
`4,236,082
`4,236,082
`4,505,583
`4,505,583
`4,556,057
`4,556,057
`
`11/1980 Butler ................................ 250/461.1
`11/1980 Butler
`250/461.1
`3/1985 Konomi
`356/73
`3/1985 Konomi ......
`... 356/73
`12/1985 Hiruma et al.
`600/476
`12/1985 Hiruma et al. ....................... 600/476
`(List continued on next page.)
`(List continued on next page.)
`FOREIGN PATENT DOCUMENTS
`FOREIGN PATENT DOCUMENTS
`0359433
`3/1990 (EP)
` A61B/5/05
`0359433
`3/1990 (EP) ................................ A61B/5/05
`0650694
`5/1995 (EP) ................................ A61B/S/00
`0650694
`5/1995 (EP)
` A61B/5/00
`OTHER PUBLICATIONS
`OTHER PUBLICATIONS
`S.X. Zhao and T. Lu, “The Classification of the Depth of
`S.X. Zhao and T. Lu, "The Classification of the Depth of
`Burn Injury Using Hybrid Neural Network'; IEEE Confer
`Burn Injury Using Hybrid Neural Network"; IEEE Confer-
`ence on Engineering in Medicine & Biology Society, ISBN
`ence on Engineering in Medicine & Biology Society, ISBN
`7803–2475–7, vol. 1, pp. 815–816, Jul., 1997.
`7803-2475-7, vol. 1, pp. 815-816, Jul., 1997.
`I. Koren, A. Laine, F. Taylor, M. Lewis, “Interactive Wavelet
`I. Koren, A. Laine, F. Taylor, M. Lewis, "Interactive Wavelet
`Processing and Techniques Applied to Digital Mammogra
`Processing and Techniques Applied to Digital Mammogra-
`phy”; IEEE Conference Proceedings, ISBN 07–7803–3
`phy"; IEEE Conference Proceedings, ISBN 07-7803-3
`192-3, vol. 3, pp. 1415-1418, Mar. 1996.
`192-3, vol. 3, pp. 1415-1418, Mar. 1996.
`(List continued on next page.)
`(List continued on next page.)
`Primary Examiner Amelia Au
`Primary Examiner Amelia Au
`ASSistant Examiner Mehrdad Dastouri
`Assistant Examiner—Mehrdad Dastouri
`(74) Attorney, Agent, or Firm Morgan & Finnegan, L.L.P.
`(74) Attorney, Agent, or Firm—Morgan & Finnegan, L.L.P.
`(57)
`ABSTRACT
`(57)
`ABSTRACT
`Systems and methods for the multispectral imaging of skin
`Systems and methods for the multispectral imaging of skin
`tissue enables automatic characterization of the condition of
`tissue enables automatic characterization of the condition of
`a region of interest of the skin, based on direct digital
`a region of interest of the skin, based on direct digital
`imaging of the region of interest or the digitization of color
`imaging of the region of interest or the digitization of color
`photographic Slides of the region of interest, illuminated by
`photographic slides of the region of interest, illuminated by
`appropriately filtered light. Preferably, digital image at a
`appropriately filtered light. Preferably, a digital image at a
`low spectral band is automatically Segmented and that
`low spectral band is automatically segmented and that
`Segmented mask is used to Segment the other images by a
`segmented mask is used to segment the other images by a
`digital processor. Parameters related to the texture,
`digital processor. Parameters related to the texture,
`9.
`p
`asymmetry, blotchineSS and border irregularities are also
`asymmetry, blotchiness and border irregularities are also
`automatically estimated. The region of interest is automati
`automatically estimated. The region of interest is automati-
`cally characterized by the digital processor, based on those
`cally characterized by the digital processor, based on those
`parameters. The region of interest may include a skin lesion,
`parameters. The region of interest may include a skin lesion,
`in which case the present invention enables the character-
`In which CSC the present invention enables the character
`ization of the lesion as malignant or benign.
`ization of the lesion as malignant or benign.
`
`73 Claims, 16 Drawing Sheets
`73 Claims, 16 Drawing Sheets
`
`5
`50
`
`Skin
`Skin
`lesion
`lesion
`
`54
`
`54
`
`y
`A2
`
`X.1
`1
`
`56
`
`56
`
`y
`A3
`
`57
`
`57 W. viv
`4
`
`•
`
`•
`
`•
`
`Digital
`Digital
`Images
`Images
`
`52
`
`52
`
`58
`58
`
`60
`60
`
`segmenta
`Segmenta-
`don Mask
`tion Mask
`W
`Segrnteci
`
`62
`62 y
`Segmented
`
`84
`64
`
`65
`65
`Segmented
`
`•
`
`•
`
`•
`
`seated seated Seged series
`66 Compute Lesion ---
`66 Compute Lesion
`Parameters
`Parameters
`
`68
`68
`
`CASSFER
`
`70
`
`Yes
`Yes
`
`No
`
`Petitioner's Exhibit 1013
`Page 1 of 37
`
`

`

`US 6,208,749 B1
`US 6,208,749 B1
`Page 2
`Page 2
`
`U.S. PATENT DOCUMENTS
`U.S. PATENT DOCUMENTS
`
` 600/476
`9/1988 Suzuki
`4,768,513
`9/1988 Suzuki ................................. 600/476
`4,768,513
`9/1988 Suzaki et al.
` 382/128
`4,773,097
`9/1988 Suzaki et al. ..
`... 382/128
`4,773,097
`4/1989 Sekiguchi ............................... 348/68
`4,821,117
`4/1989 Sekiguchi
` 348/68
`4,821,117
`1/1990 Leffell et al.
` 250/461.2
`4,894,547
`1/1990 Leffel et al. ...
`250/461.2
`4,894,547
`6/1990 Alfano et al.
` 600/477
`4,930,516
`6/1990 Alfano et al. ..
`... 600/477
`4,930,516
`9/1990 Zeng et al. ........................... 600/476
`4,957,114
`9/1990 Zeng et al.
` 600/476
`4,957,114
`4/1991 Suzuki et al.
` 600/317
`5,003,977
`4/1991 Suzuki et al. ........................ 600/317
`5,003,977
`5/1991 Kenet et al.
` 382/128
`5,016,173
`5/1991 Kenet et al. ...
`... 382/128
`5,016,173
`8/1991 Jeffcoat et al.
` 600/342
`5,036,853
`8/1991 Jeffcoat et al.
`... 600/342
`5,036,853
`10/1992 Page ...............
`... 356/350
`5,157,461
`10/1992 Page
` 356/350
`5,157,461
`12/1992 Daikuzono
` 600/476
`5,174,297
`12/1992 Daikuzono ........................... 600/476
`5,174,297
`8/1993 Kenet
` 600/300
`5,241,468
`8/1993 Kenet ................................... 600/300
`5,241,468
`11/1994 Martens et al.
` 600/477
`5,363,854
`11/1994 Martens et al.
`... 600/477
`5,363,854
`11/1994 Alfano et al.
` 600/477
`5,369,496
`11/1994 Alfano et al. ..
`... 600/477
`5,369,496
`4/1995 Salb
` 600/317
`5,408,996
`4/1995 Salb ..................................... 600/317
`5,408,996
`6/1995 Richards-Kortum et al.
` 600/477
`5,421,337
`6/1995 Richards-Kortum et al. ....... 600/477
`5,421,337
`5/1996 Tsuruoka et al.
` 382/128
`5,515,449
`5/1996 Tsuruoka et al. .............
`... 382/128
`5,515,449
`6/1996 Lee
` 382/257
`5,528,703
`6/1996 Lee ....................
`... 382/257
`5,528,703
`1/1997 MacAulay et al. ...
`... 600/478
`5,590,660
`1/1997 MacAulay et al.
` 600/478
`5,590,660
`5,660,982 * 8/1997 Tryggvason et al. .................... 435/6
`5,660,982 * 8/1997 Tryggvason et al.
` 435/6
`5,699,798
`12/1997 Hochman et al.
` 600/420
`5,699,798
`12/1997 Hochman et al. ...
`... 600/420
`12/1997 Vari et al.
` 600/473
`5,701,902
`5,701,902
`12/1997 Variet al. ..........
`... 600/473
`5,717,791
`2/1998 Labaere et al.
` 382/274
`5,717,791
`2/1998 Labaere et al. ....
`... 382/274
`5,740,268
`4/1998 Nishikawa et al.
` 382/132
`5,740,268
`4/1998 Nishikawa et al.
`... 382/132
`5,749,830
`5/1998 Kaneko et al.
` 600/160
`5,749,830
`5/1998 Kaneko et al. ....
`... 600/160
`5,784,162
`7/1998 Cabib et al.
` 356/346
`5,784,162
`7/1998 Cabib et al. ...
`... 356/346
`5,799,100
`8/1998 Clarke et al.
` 382/132
`5,799,100
`8/1998 Clarke et al. ........................ 382/132
`
`
`
`OTHER PUBLICATIONS
`OTHER PUBLICATIONS
`T. Lee, V. Ng, D. McLean, A. Coldman, R. Gallagher and J.
`T. Lee, V. Ng, D. McLean, A. Coldman, R. Gallagher and J.
`Sale, “A Multi-Stage Segmentation Method for Images of
`Sale, "A Multi-Stage Segmentation Method for Images of
`Skin Lesions'; IEEE Conference Proceedings on Commu
`Skin Lesions"; IEEE Conference Proceedings on Commu-
`nication, Computers and Signal Processing, pp. 602-605,
`nication, Computers and Signal Processing, pp. 602-605,
`Feb. 1995.
`Feb. 1995.
`B.F. Jones and P. Plassman, “An Instrument to Measure the
`B.F. Jones and P. Plassman, "An Instrument to Measure the
`Dimensions of Skin Wounds'; IEEE Transactions on Bio
`Dimensions of Skin Wounds"; IEEE Transactions on Bio-
`medical Engineering, vol. 42, No. 5, pp. 464–470, May
`medical Engineering, vol. 42, No. 5, pp. 464-470, May
`1995.
`1995.
`R.T.J. Bostock, E. Claridge, A.J. Harget, and P.N. Hall,
`R.T.J. Bostock, E. Claridge, A.J. Harget, and P.N. Hall,
`“Towards A Neural Network Based System For Skin Cancer
`"Towards A Neural Network Based System For Skin Cancer
`Diagnosis”; IEEE International Conference on Artificial
`Diagnosis"; IEEE International Conference on Artificial
`Neural Networks, pp. 215–219, ISBN 0-85296–573–7,
`Neural Networks, pp. 215-219, ISBN 0-85296-573-7,
`1993.
`1993.
`M. Herbin, A. Venot, J.Y. Devaux and C. Piette, “Color
`M. Herbin, A. Venot, J.Y. Devaux and C. Piette, "Color
`Quantitation Through Image Processing in Dermatology”;
`Quantitation Through Image Processing in Dermatology";
`IEEE Transactions on Medical Imaging, vol. 9, No. 1, pp.
`IEEE Transactions on Medical Imaging, vol. 9, No. 1, pp.
`262-269, Sep. 1990.
`262-269, Sep. 1990.
`"Border irregularity: atypical moles Versus melanoma', C.L.
`"Border irregularity: atypical moles versus melanoma", C.L.
`Huang et al., EurJ Dermatol, vol. 6, pp. 270-273, Jun. 1996.
`Huang et al., Eur J Dermatol, vol. 6, pp. 270-273, Jun. 1996.
`“In vivo Spectrophotometric Evaluation of Neoplastic and
`"In vivo Spectrophotometric Evaluation of Neoplastic and
`Non-Neoplastic Skin Pigmented Lesions. III. CCD Cam
`Non-Neoplastic Skin Pigmented Lesions. III. CCD Cam-
`era-Based Reflectance Imaging”, R. Marchesini et al., Pho
`era-Based Reflectance Imaging", R. Marchesini et al., Pho-
`tochemistry and Photobiology, vol. 62, No. 1, pp. 151-154;
`tochemistry and Photobiology, vol. 62, No. 1, pp. 151-154;
`1995.
`1995.
`“The Morphologic Criteria of the Pseudopod in Surface
`"The Morphologic Criteria of the Pseudopod in Surface
`Microscopy', S.W. Menzies, et al., Arch Dermatol, vol. 131,
`Microscopy", S.W. Menzies, et al., Arch Dermatol, vol. 131,
`pp. 436–440, Apr. 1995.
`pp. 436-440, Apr. 1995.
`“A rudimentary System for automatic discrimination among
`"A rudimentary system for automatic discrimination among
`basic skin lesions on the basis of color analysis of Video
`basic skin lesions on the basis of color analysis of video
`images, H. Takiwaki et al., Journal of the American Acad
`images", H. Takiwaki et al., Journal of the American Acad-
`emy of Dermatology, vol. 32, No. 4, pp. 600-604, Apr. 1995.
`emy of Dermatology, vol. 32, No. 4, pp. 600-604, Apr. 1995.
`“Topodermatographic Image Analysis for Melanoma
`"Topodermatographic
`Image Analysis
`for Melanoma
`Screening and the Quantitative ASSessment of Tumor
`Screening and the Quantitative Assessment of Tumor
`Dimension Parameters of the Skin', H. Voigt et al., Cancer,
`Dimension Parameters of the Skin", H. Voigt et al., Cancer,
`vol. 75, No. 4, Feb. 15, 1995.
`vol. 75, No. 4, Feb. 15,1995.
`
`"Application of an artificial neural network in epilumines
`"Application of an artificial neural network in epilumines-
`cence microScopy pattern analysis of pigmented Skin
`cence microscopy pattern analysis of pigmented skin
`lesions: a pilot study', M. Binder et al., British Journal of
`lesions: a pilot study", M. Binder et al., British Journal of
`Dermatology 130; pp. 460–465; 1994.
`Dermatology 130; pp. 460-465; 1994.
`“Computer image analysis in the diagnosis of melanoma',
`"Computer image analysis in the diagnosis of melanoma",
`A. Greene et al., Journal of the American Academy of
`A. Greene et al., Journal of the American Academy of
`Dermatology; vol. 31, No. 6, pp. 958-964, 1994.
`Dermatology; vol. 31, No. 6, pp. 958-964,1994.
`“Computerized Digital Image Analysis: An Aid for Mela
`"Computerized Digital Image Analysis: An Aid for Mela-
`noma Diagnosis”, A.J. Sober et al., The Journal of Derma
`noma Diagnosis", A.J. Sober et al., The Journal of Derma-
`tology, vol. 21, pp. 885-890, 1994.
`tology, vol. 21, pp. 885-890,1994.
`“Neural Network Diagnosis of Malignant Melanoma From
`"Neural Network Diagnosis of Malignant Melanoma From
`Color Images”, F. Ercal et al., IEEE Transactions of Bio
`Color Images", F. Ercal et al., IEEE Transactions of Bio-
`medical Engineering, vol. 41, No. 9, pp. 837-845, Sep.
`medical Engineering, vol. 41, No. 9, pp. 837-845, Sep.
`1994.
`1994.
`“The ABCD rule of dermatology”, F. Nachbar et al., Journal
`"The ABCD rule of dermatology", F. Nachbar et al., Journal
`of the American Academy of Dermatology, Vol. 3, No. 4, pp.
`of the American Academy of Dermatology, vol. 3, No. 4, pp.
`551–559, Apr. 1994.
`551-559, Apr. 1994.
`"Evaluation of different image acquisition techniques for a
`"Evaluation of different image acquisition techniques for a
`computer vision System in the diagnosis of malignant mela
`computer vision system in the diagnosis of malignant mela-
`noma, T. Schindewolf et al., Journal of the American
`noma", T. Schindewolf et al., Journal of the American
`Academy of Dermatology, vol. 31, No. 1, pp. 33–41, Jul.
`Academy of Dermatology, vol. 31, No. 1, pp. 33-41, Jul.
`1994.
`1994.
`“Detection of Skin Tumor Boundaries in Color Images”, F.
`"Detection of Skin Tumor Boundaries in Color Images", F.
`Ercal et al., IEEE Transactions of Medical Imaging, vol. 12,
`Ercal et al., IEEE Transactions of Medical Imaging, vol. 12,
`No. 3, pp. 624–627, Sep. 1993.
`No. 3, pp. 624-627, Sep. 1993.
`"Automatic Color Segmentation Algorithms with Applica
`"Automatic Color Segmentation Algorithms with Applica-
`tion to Skin Tumor Feature Identification”, S.E. Umbagh et
`tion to Skin Tumor Feature Identification", S.E. Umbagh et
`al., IEEE Engineering in Medicine and Biology, pp. 75-82,
`al., IEEE Engineering in Medicine and Biology, pp. 75-82,
`Sep., 1993.
`Sep., 1993.
`“Comparison of classification rates for conventional and
`"Comparison of classification rates for conventional and
`dermatoscopic images of malignant and benign melanocytic
`dermatoscopic images of malignant and benign melanocytic
`lesions using computerized colour image analysis”, T.
`lesions using computerized colour image analysis", T.
`Schindewolf et al., Eur J Dermatol, vol. 3, No. 4, pp.
`Schindewolf et al., Eur J Dermatol, vol. 3, No. 4, pp.
`299-303, May 1993.
`299-303, May 1993.
`“Classification of Melanocytic Lesions with Color and Tex
`"Classification of Melanocytic Lesions with Color and Tex-
`ture Analysis Using Digital Image Processing, T. Schinde
`ture Analysis Using Digital Image Processing", T. Schinde-
`Wolf et al., The International Academy of Cytology, Ana
`wolf et al., The International Academy of Cytology, Ana-
`lytical and Quantitative Cytology and Histology, Vol. 15,
`lytical and Quantitative Cytology and Histology, vol. 15,
`No. 1, pp. 1-11, Feb. 1993.
`No. 1, pp. 1-11, Feb. 1993.
`“Clinical Diagnosis of Pigmented Lesions. Using Digital
`"Clinical Diagnosis of Pigmented Lesions Using Digital
`Epiluminescence Microscopy, R.O. Kenet et al., Arch
`Epiluminescence Microscopy", R.O. Kenet et al., Arch
`Dermatol, vol. 129, pp. 157–174; Feb. 1993.
`Dermatol, vol. 129, pp. 157-174; Feb. 1993.
`“Optical properties of human dermis in vitro and in Vivo”,
`"Optical properties of human dermis in vitro and in vivo",
`R. Graaffet al., Applied Optics, vol. 32, No. 4, pp. 435-447,
`R. Graaff et al., Applied Optics, vol. 32, No. 4, pp. 435-447,
`Feb. 1, 1993.
`Feb. 1,1993.
`“Automatic Detection of Irregular Borders in Melanoma and
`"Automatic Detection of Irregular Borders in Melanoma and
`Other Skin Tumors”, J.E. Golston et al., Computerized
`Other Skin Tumors", J.E. Golston et al., Computerized
`Medical Imaging and Graphics, vol. 16, No. 3, pp. 199-203,
`Medical Imaging and Graphics, vol. 16, No. 3, pp. 199-203,
`1992.
`1992.
`“Automatic Detection of Asymmetry in Skin Tumors”, W.V.
`"Automatic Detection of Asymmetry in Skin Tumors", W.V.
`Stoecker et al., Computerized Medical Imaging and Graph
`Stoecker et al., Computerized Medical Imaging and Graph-
`ics, vol. 16, No. 3, pp. 191-197, 1992.
`ics, vol. 16, No. 3, pp. 191-197,1992.
`“Results obtained by using a computerized image analysis
`"Results obtained by using a computerized image analysis
`System designed as an aid to diagnosis of cutaneous mela
`system designed as an aid to diagnosis of cutaneous mela-
`noma, N. Cascinelli et al., Melanoma Research, Vol. 2, pp.
`noma", N. Cascinelli et al., Melanoma Research, vol. 2, pp.
`163-170, 1992.
`163-170,1992.
`“An Automatic Color Segmentation Algorithm with Appli
`"An Automatic Color Segmentation Algorithm with Appli-
`cation to Identification of Skin Tumor Borders', S.E.
`cation to Identification of Skin Tumor Borders", S.E.
`Umbaugh et al., Computerized Medical Imaging and Graph
`Umbaugh et al., Computerized Medical Imaging and Graph-
`ics, vol. 16, No. 3, pp. 227-235, May-Jun. 1992.
`ics, vol. 16, No. 3, pp. 227-235, May-Jun. 1992.
`"Automatic Color Segmentation of Images with Application
`"Automatic Color Segmentation of Images with Application
`to Detection of Variegated Coloring in Skin Tumors”, S.W.
`to Detection of Variegated Coloring in Skin Tumors", S.W.
`Umbaugh et al., IEEE Engineering in Medicine and Biology
`Umbaugh et al., IEEE Engineering in Medicine and Biology
`Magazine, Dec. 1989, pp. 43–52.
`Magazine, Dec. 1989, pp. 43-52.
`
`Petitioner's Exhibit 1013
`Page 2 of 37
`
`

`

`US 6,208,749 B1
`US 6,208,749 B1
`Page 3
`Page 3
`
`“Multispectral Imaging of Burn Wounds: A New Clinical
`"Multispectral Imaging of Burn Wounds: A New Clinical
`Instrument for Evaluating Burn Depth”, M.A. Afromowitz
`Instrument for Evaluating Burn Depth", M.A. Afromowitz
`et al., IEEE Transactions on Biomedical Engineering, Vol.
`et al., IEEE Transactions on Biomedical Engineering, vol.
`35, No. 10, pp. 842-850; Oct. 1988.
`35, No. 10, pp. 842-850; Oct. 1988.
`“In Vivo epiluminescence microScopy of pigmented Skin
`"In vivo epiluminescence microscopy of pigmented skin
`lesions. I. Pattern analysis of pigmented Skin lesions”, H.
`lesions. I. Pattern analysis of pigmented skin lesions", H.
`Pelhamberger et al., Journal of American Academy of Der
`Pehamberger et al., Journal of American Academy of Der-
`matology, vol. 17, No. 4, pp. 571-583, Oct. 1987.
`matology, vol. 17, No. 4, pp. 571-583, Oct. 1987.
`“In Vivo epiluminescence microScopy of pigmented Skin
`"In vivo epiluminescence microscopy of pigmented skin
`lesions. II. Diagnosis of Small pigmented Skin lesions and
`lesions. II. Diagnosis of small pigmented skin lesions and
`early detection of malignant melanoma’, A. Steiner et al.,
`early detection of malignant melanoma", A. Steiner et al.,
`Journal of the American Academy of Dermatology, Vol. 17,
`Journal of the American Academy of Dermatology, vol. 17,
`No. 4, pp. 584-591; Oct. 1987.
`No. 4, pp. 584-591; Oct. 1987.
`“The Optics of Human Skin", R.R. Anderson et al., The
`"The Optics of Human Skin", R.R. Anderson et al., The
`Journal of Investigative Dermatology, vol. 77, No. 1, pp.
`Journal of Investigative Dermatology, vol. 77, No. 1, pp.
`13-19; Jul 1981.
`13-19; Jul. 1981.
`“Melanin, a unique biological absorber, M.L. Wolbarsht,
`"Melanin, a unique biological absorber", M.L. Wolbarsht,
`Applied Optics, vol. 20, No. 13, pp. 2184-2186; Jul. 1,
`Applied Optics, vol. 20, No. 13, pp. 2184-2186; Jul. 1,
`1981.
`1981.
`“The Wavelet Transform, Time-Frequency Localization and
`"The Wavelet Transform, Time—Frequency Localization and
`Signal Analysis”, I. Daubechies, IEEE Trans Inform Theory,
`Signal Analysis", I. Daubechies, IEEE Trans Inform Theory,
`vol. 36, No. 5, pp. 961-1005; Sep. 1990.
`vol. 36, No. 5, pp. 961-1005; Sep. 1990.
`
`“Wavelet in Medicine and Biology”, Aldroubi et al., C&C
`"Wavelet in Medicine and Biology", Aldroubi et al., C&C
`Press, NY, pp. 11-15, 1996.
`Press, NY, pp. 11-15, 1996.
`"Singularity detection and processing with wavelets, S.
`"Singularity detection and processing with wavelets", S.
`Mallat et al., IEEE Trans Inform Theory 38:617-643; 1992.
`Mallat et al., IEEE Trans Inform Theory 38:617-643; 1992.
`“Wavelets and Applications”, S. Mallat et al., S. Verlag, Y.
`"Wavelets and Applications", S. Mallat et al., S. Verlag; Y
`Meyer (ed.) NY pp. 207–284; 1992.
`Meyer (ed.) NY pp. 207-284; 1992.
`“Characterization of Signals from multiScale edges', S.
`"Characterization of signals from multiscale edges", S.
`Mallat et al., IEEE Trans Patt and Mech Int’l; 14:710–732;
`Mallat et al., IEEE Trans Patt and Mech Intl; 14:710-732;
`1992.
`1992.
`“Introduction to Statistical Pattern Recognition”, K. Fuku
`"Introduction to Statistical Pattern Recognition", K. Fuku-
`maga, Academic Press, Boston, pp. 90-96, 125, 219–221;
`maga, Academic Press, Boston, pp. 90-96, 125, 219-221;
`1990.
`1990.
`“Image Features From Phase Congruency”, P. Kovesi, Uni
`"Image Features From Phase Congruency", P. Kovesi, Uni-
`versity of Western Australia, pp. 1-30; Technical Report 9/4,
`versity of Western Australia, pp. 1-30; Technical Report 9/4,
`Revised Jun. 1995.
`Revised Jun. 1995.
`
`* cited by examiner
`* cited by examiner
`
`Petitioner's Exhibit 1013
`Page 3 of 37
`
`

`

`U.S. Patent
`U.S. Patent
`
`Mar. 27, 2001
`Mar. 27, 2001
`
`Sheet 1 of 16
`Sheet 1 of 16
`
`US 6,208,749 B1
`US 6,208,749 B1
`
`3
`3
`
`4
`4
`
`WHITE LIGHT
`WHITE LIGHT
`SOURCE
`SOURCE
`
`NARROW
`NARROW
`FILTERS
`FILTERS
`
`5
`5
`-------
`
`FIBEROPTIC
`FBEROPTIC
`ILLUMNATOR
`ILLUMINATOR
`
`12
`12
`
`
`
`COMPUTER
`DIGITAL
`/
`DIGITAL
`.1-
`INTERFACE
`INTERFACE
`
`A.
`
`v
`MEMORY —
`
`12a
`
`12c
`
`_ DIGITAL
`PROCESSOR
`
`12C this DISPLAY
`
`19
`
`6
`6
`MONOCHROMATIC
`MONOCHROMATIC
`CAMERA
`CAMERA
`
`47 s
`
` D
`D
`
`.
`•
`•
`
`> CONTROL SIGNALS
`KEY: -
`KEY: >
`CONTROL SIGNALS
`LIGHT SIGNALS
`--0.
`-> LGHT SIGNALS
`—
`r> DIGITAL DATA
`-
`D DIGITAL DATA
`
`HIGH-RESOLUTION
`HGH-RESOLUTION
`MULTI-SPECTRAL
`MULTI-SPECTRAL
`DIGITAL IMAGES
`DIGITAL IMAGES
`FIG. la
`FIG. 1 a
`
`2 \
`2
`
`
`
`FIG. lb
`
`UV
`
`Y\
`.Z\_
` Y\
`./\.
`.....,/"N_
`
`..,\__
`IR
`
`VISIBLE
`VISIBLE
`
`Petitioner's Exhibit 1013
`Page 4 of 37
`
`

`

`Waled *S11
`U.S. Patent
`
`Toot 'a ..0 ,\
`
`91 Jo Z lamIS
`
`HI 617LNOV9 Sa US 6,208,749 B1
`
`12a
`
`12b
`
`12c
`
`
`
`
`
`
`
`
`
`13 13
`\
`PHOTO
`CAMERA
`
`
`
`15
`\
`35-MM
`ARCHIVE
`
`17
`
`16
`
`BAN DPASS
`FILTERS
`
`MONO-
`CHROME
`CAMERA
`
`I
`
`1
`
`r -
`
`_ _ I
`
`I
`
`I ____ 1 3-CHIP 1
`I
`CCD
`I
`— —7 — — j
`20
`
`18
`
`DIGITAL
`IMAGES
`
`12
`
`COMPUTER
`
`DIGITAL
`INTERFACE
`
`.
`MEMORY
`
`:
`
`DIGITAL
`PROCESSOR
`
`FIG. 1c
`
`,
`19 ----d DISPLAY
`
`Petitioner's Exhibit 1013
`Page 5 of 37
`
`

`

`Waled *S11.
`U.S. Patent
`
`Toot 'a .Jutv
`
`91 Jo £ 13311S
`
`US 6,208,749 B1
`HI 617LNOV9 Sfl
`
`
`
`
`TO
` ► COMPUTER 12
`
`23
`
`FROM
` < COMPUTER 12
`
`30a
`
`F
`24
`
`(%-,•.\\\.s>
`SKIN LESION
`
`
`
`
`
`_.........„...----.°
`
`'"'"".
`
`2
`
`FIG. 2
`
`GN
`9
`
`22
`
`NOISET
`NIXIS - — — — — — — — — — — — — — — — — —|-
`
`r
`
`25
`
`FROM
`COMPUTER 12
`
`FROM
`COMPUTER 12
`
`ZL HELTldWOO
`
`Petitioner's Exhibit 1013
`Page 6 of 37
`
`

`

`Waled *S11
`
`Toot 'a ..0 ,\
`
`91 Jo 17 loolIS
`
`HI 617LNOV9 Sa
`
`Step 2
`
`Steps
`3 & 4
`
`Step 5
`USER CHOICE OF
`BANDS FOR COLOR
`VISUALIZATION
`
`Step 6
`WEIGHT R,G,B CHANNELS
`-c>
`IN INVERSE PROPORTION
`
`- -
`
`TO <1 gray strip'
`
`DISPLAY
`IMAGES
`
`FIG. 3a
`
`DARK SCENE
`
`MOTOR
`
`t
`11111
`DIFFUSE WHITE
`REFERENCE TARGET
`LESION
`
`SKIN
`GRAY STRIP
`
`CAPTURE N
`DARK-FIELD
`IMAGES
`
`AVERAGE
`AND STORE
`RESULT, I D
`
`- - - - Step 1
`
`CAPTURE N'
`WHITE FLAT-FIELD
`IMAGES IN EACH
`SPECTRAL BAND
`
`—f Cl>
`
`STORE AVERAGE
`RESULT FOR EACH
`SPECTRAL BAND, Iwi
`
`Step 3
`CAPTURE IMAGES OF
`SKIN AND GRAY STRIP
`IN EACH SPECTRAL
`BAND, Is, , i-1,2,...M
`
`Step 4 NORMALIZE:
`A
`
`S —ID
`1
`--> IS -->(
`ID
`I
`CALCULATE < !gray strip>i
`
`*‘2° -1),
`
`A
`
`Petitioner's Exhibit 1013
`Page 7 of 37
`
`

`

`Waled *S11
`U.S. Patent
`
`Toot `LZ *JRAI
`
`91 JO S laNS
`
`Tif 6171NOV9 Sa US 6,208,749 B1
`
`
`
`
`,
`v
`
`0
`
`-,
`i
`
`•
`
`•
`
`Skin
`lesion
`
`50
`
`09
`
`54
`
`y
`
`A.2
`
`Xi
`
`
`
`56
`
`57
`
`X3
`
`A.4
`
`I
`Segmenta-
`tion Mask
`i
`Segmented
`24.,1
`
`v
`62
`Segmented
`2,2
`
`64
`Segmented
`
`sw
`
`65
`Segmented
`X4
`
`•
`
`•
`
`•
`
`Digital
`Images
`
`52
`
`58
`
`60
`
`Compute Lesion
`Parameters
`
`68
`
`CLASSFIER
`
`70
`
`Yes
`
`v No
`Fig. 3b
`
`Petitioner's Exhibit 1013
`Page 8 of 37
`
`

`

`U.S. Patent
`U.S. Patent
`
`Mar. 27, 2001
`Mar. 27, 2001
`
`Sheet 6 of 16
`Sheet 6 of 16
`
`US 6,208,749 B1
`US 6,208,749 B1
`
`YAGNANT MANOVA
`MALIGNANT MELANOMA
`
`14
`
`1.1.
`
`1. O
`
`0
`
`O
`0
`
`.
`
`.
`
`100
`100
`50
`50
`NTENSTTY LEVE,
`INTENSITY LEVEL
`Fig. 4(a)
`Fig. 4(a)
`
`AYPCAL VANOCYC NWUS
`ATYPICAL MELANOCYTIC NEVUS
`
`14
`
`1. 24
`
`12
`
`io
`1. O
`
`8
`
`6
`
`4
`
`2
`
`O
`0
`
`0
`O
`
`150
`150
`
`mop
`
`•MM
`
`se
`
`150
`150
`
`100
`50
`OO
`50
`NTNSY EVEL
`INTENSITY LEVEL
`Fig. 4(b)
`Fig. 4(b)
`
`Petitioner's Exhibit 1013
`Page 9 of 37
`
`

`

`U.S. Patent
`U.S. Patent
`
`Mar. 27, 2001
`Mar. 27, 2001
`
`Sheet 7 of 16
`Sheet 7 of 16
`
`US 6,208,749 B1
`US 6,208,749 B1
`
`YAGNANT YEANOVA
`MALIGNANT MELANOMA
`
`I.
`
`.
`
`OEM
`
`.
`
`,I
`
`l
`
`I
`
`I
`
`I
`
`1
`
`1
`
`O
`
`50
`50
`
`100
`100
`NTENSTTY LEVEL
`INTENSITY LEVEL
`Fig. 5(a)
`Fig. 5(a)
`
`150
`150
`
`200
`200
`
`ATYPICAL MELANOCYTIC NEVUS
`ATYPCA, MEANOCYC NEWUS
`
`O
`0
`
`50
`50
`
`100
`100
`INTENSITY LEVEL
`NTNSTY LOVE
`Fig. 5(b)
`Fig. 5(b)
`
`150
`150
`
`200
`200
`
`Petitioner's Exhibit 1013
`Page 10 of 37
`
`

`

`U.S. Patent
`U.S. Patent
`
`Mar. 27, 2001
`Mar. 27, 2001
`
`Sheet 8 of 16
`Sheet 8 of 16
`
`US 6,208,749 B1
`US 6,208,749 B1
`
`ANAN EANO.
`MALIGNANT MELANOMA
`
`ATYPICAL MELANOCYTIC NEVUS
`AYPCA, EAECYC NEW US
`
`
`
`Fig. 6(a)
`
`Fig. 6(d)
`
`Fig. 6(b)
`
`Fig. 6(e)
`
`Fig. 6(c)
`
`Fig. 8(f)
`
`Petitioner's Exhibit 1013
`Page 11 of 37
`
`

`

`U.S. Patent
`U.S. Patent
`
`Mar. 27, 2001
`Mar. 27, 2001
`
`Sheet 9 of 16
`Sheet 9 of 16
`
`US 6,208,749 B1
`US 6,208,749 B1
`
`AGNix EN:
`MALIGNANT MELANOMA
`
`ATYPICAL MELANOCYTIC NEVUS
`YC, ENCYC NES
`
`Fig. 7(a)
`
`Fig. 7(d)
`
`xt
`
`t
`
`t l
`
`o
`
`t s a
`
`i
`
`a
`
`t 4
`
`Fig. (b)
`Fig. 7(1))
`
`
`
` AMOVINNWOMARIM,
`
`Fig. (e)
`Fig. 7(e)
`
`Fig. 7(e)
`
`Fig. 7(f)
`
`Petitioner's Exhibit 1013
`Page 12 of 37
`
`

`

`U.S. Patent
`U.S. Patent
`
`Mar. 27, 2001
`Mar. 27,2001
`
`Sheet 10 of 16
`Sheet 10 of 16
`
`US 6,208,749 B1
`US 6,208,749 B1
`
`MALIGNANT MEANOMA
`MALIGNANT MELANOMA
`
`ATPCAL MEANCYTIC NEWS
`ATYPICAL MELANOCYTIC NEVUS
`
`
`
`Pig. 8(a)
`
`Fig. 8(e)
`
`Fig. 8(b)
`
`Fig. 8(0
`
`fg. 8(c)
`
`Fig. 8(g)
`
`Fig. 8(d)
`
`rig. 8(h)
`
`Petitioner's Exhibit 1013
`Page 13 of 37
`
`

`

`U.S. Patent
`U.S. Patent
`
`Mar. 27, 2001
`Mar. 27, 2001
`
`Sheet 11 of 16
`Sheet 11 of 16
`
`US 6,208,749 B1
`US 6,208,749 B1
`
`J-5
`
`J-4
`J-4
`
`J-3
`J-3
`
`J-2
`J-2
`
`J-1
`J-1
`
`J
`
`J-1
`J+1
`
`J+2
`J+2
`
`J-3
`J+3
`
`J+4
`J+4
`
`J-5
`J+5
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`0
`
`0
`
`0
`
`1
`
`1
`
`1
`
`1
`
`1
`
`0
`
`0
`
`0
`
`1
`
`1
`
`0
`
`0
`
`0
`
`1
`
`2
`
`0
`
`0
`
`0
`
`3
`
`1
`
`0
`
`1
`
`2
`
`0
`
`0 —3 —5 —3
`
`1 —5 —8 —5
`
`0 —3 —5 —3
`
`0
`
`0
`
`1
`
`1
`
`0
`
`2
`
`1
`
`0
`
`1
`
`3
`
`0
`
`0
`
`0
`
`2
`
`1
`
`0
`
`1
`
`1
`
`0
`
`0
`
`0
`
`1
`
`0
`
`0
`
`0
`
`1
`
`1
`
`0
`
`0
`
`0
`
`1
`
`1
`
`1
`
`1
`
`1
`
`0
`
`0
`
`0
`
`-3 -2 - 1
`1-3 1-2 1-1
`
`I
`
`+ 1 + 2 +3
`1+1 1+2 1+3
`
`Fig. 9
`Fig. 9
`
`Petitioner's Exhibit 1013
`Page 14 of 37
`
`

`

`U.S. Patent
`U.S. Patent
`
`Mar. 27, 2001
`Mar. 27, 2001
`
`Sheet 12 of 16
`Sheet 12 of 16
`
`US 6,208,749 B1
`US 6,208,749 B1
`
`MAGNANT MELANOMA
`MALIGNANT MELANOMA
`
`ATYPICAL MELANOCYTIC NEVUS
`ATYPICAL MEANOCYTEC NEW US
`
`
`
`Fig. 10(a)
`
`Fig. 10(d)
`
`S.
`
`SX
`
`s&
`
`& 3. :
`
`Fig. 10(b)
`
`Fig. 10(e)
`
`Fig. 10(c)
`
`Fig. 10(f)
`
`Petitioner's Exhibit 1013
`Page 15 of 37
`
`

`

`U.S. Patent
`U.S. Patent
`
`Mar. 27, 2001
`Mar. 27, 2001
`
`Sheet 13 of 16
`Sheet 13 of 16
`
`US 6,208,749 B1
`US 6,208,749 B1
`
`
`
`
`
`
`
`
`
`
`
`
`
`PARAMETER.
`PARAMETER
`
`Asymmetry:
`Asymmetry:
`
`Blotchiness:
`
`Border:
`
`Texture:
`
`Abin
`Abin
`Ab
`Ab
`As
`Ag
`Ar
`
`Bib
`
`Big
`
`Blr
`Cb
`Cb
`C
`Cr
`Cl
`Cl
`B
`B
`Gb
`Gb
`T1
`Tlb
`Tlg
`T1g
`T2g
`
`T2r
`T3b
`b
`T3g
`
`T4b
`
`T4g
`
`T4. 4.
`
`T5b
`
`T6b
`
`
`
`Diagnostic
`Diagnostic
`accuracy (%)
`accuracy (%)
`52
`
`Sensitivity Specificity
`Specificity
`Sensitivity
`(%)
`(%)
`86
`
`71
`
`68
`
`73
`
`78
`
`56
`
`68
`
`68
`
`80
`
`80
`
`78
`
`66
`
`76
`
`88
`
`61
`
`61 1.
`
`66
`
`73
`
`80
`
`63
`
`68
`
`56
`
`76
`
`90
`
`84
`
`82
`
`89
`
`90
`
`72
`
`75
`
`55
`
`57
`
`70
`
`81
`
`56
`
`49
`
`90
`
`76
`
`72
`
`59
`
`74
`
`69
`
`83
`
`58
`
`46
`
`48
`
`50
`
`62
`
`45
`45
`
`40
`40
`
`42
`42
`
`37
`37
`
`38
`
`44
`44
`
`44
`44
`
`36
`
`38
`
`31.9
`
`49
`
`41
`
`43
`
`39
`
`
`
`38
`
`38
`
`39
`
`36
`
`38
`
`Fig. 11
`Fig. 11
`
`Petitioner's Exhibit 1013
`Page 16 of 37
`
`

`

`U.S. Patent
`U.S. Patent
`
`Mar. 27, 2001
`Mar. 27, 2001
`
`Sheet 14 of 16
`Sheet 14 of 16
`
`US 6,208,749 B1
`US 6,208,749 B1
`
`41 malignant melanomas and 104 atypical melanocytic nevi
`41 malignant melanomas and 104 atypical melanocytic nevi
`i
`1
`i
`O malignant melanoma
`• malignant melanoma
`o atypical melanocytic nevus
`o atypical melanocytic nevus
`
`
`
`1.4
`
`I
`
`'
`
`I
`
`I
`
`'
`
`I
`
`I
`
`'
`
`4•1
`
`•
`
`•
`
`I
`
`••
`
`Al
`
`_
`
`_
`
`1.3
`
`Sensitivity = 100%
`— Sensitivity = 100%
`Specificity = 84.6%
`Specificity = 84.6%
`
`1. 2
`
`0
`o
`
`o
`
`•
`
`•
`
`•
`
`•
`
`•
`
`•
`
`•
`•
`
`0
`
`o
`
`o
`
`o
`
`o
`
`•
`
`•
`
`••
`•
`
`•
`
`•
`
`•
`
`•
`
`•
`
`et
`
`• _
`•
`
`•
`
`•
`
`O 0
`
`" 0 0 "3 0
`
`°O
`0
`0
`
`so
`
`0
`
`0
`
`°
`
`0
`
`1)
`
`U 0
`
`0
`
`CID
`D i p
`
`o
`6)
`
`0
`0
`0
`
`0
`0 00
`0
`
`0.9
`
`—
`
`0.8
`
`—
`
`0
`o °0
`o
`
`0 0
`i
`20
`2O
`
`t
`
`ab
`O 0
`O
`
`O
`
`0
`
`°
`
`0 00 0
`0 0 0
`
`0 0 0
`1" 00 U4)
`0
`0 0
`
`•
`•
`
`•
`•
`
`•
`
`•
`a
`
`00
`
`•
`• as a
`
`0
`0 00
`0 o
`
`0
`0
`
`0
`
`o
`
`0
`
`o
`
`000
`0
`0
`
`0
`
`0
`
`o
`
`0
`
`i °
`40
`40
`
`.
`
`i
`1
`80
`80
`8O
`60
`LESION NUMBER
`SION NUMBER
`
`1
`100
`100
`
`1
`120
`120
`
`1
`140
`140
`
`Fig. 12
`Fig. 12
`
`Petitioner's Exhibit 1013
`Page 17 of 37
`
`

`

`U.S. Patent
`U.S. Patent
`
`Mar. 27, 2001
`Mar. 27, 2001
`
`Sheet 15 of 16
`Sheet 15 of 16
`
`US 6,208,749 B1
`US 6,208,749 B1
`
`24 superficial spreading melanomas (SSM)
`24 superficial spreading melanomas (SSM)
`16 melanomas in-situ (MIS)
`18 melanomas in —situ (MIS)
`i . . I ,
`1 . I
`1 1 1 1 1 1 1 1 1
`' 1 1 1
`1 '
`
`1 1 1 1 1 1 1 1
`
`. 1
`
`1
`
`I
`
`I
`
`i
`
`I
`
`• SSM
`• o MIS
`
`0
`
`- o
`
`•
`
`•
`
`•
`
`•
`
`•
`
`I
`
`I
`
`•
`
`••
`
`•
`
`•
`
`•
`
`•
`
`•
`
`•
`
`•
`
`•
`
`• I.
`
`•
`
`•
`
`•
`
`o 0
`
`o
`
`0
`
`0
`
`00
`
`0
`
`0
`
`0
`
`0
`0
`
`1.2
`1.2
`
`
`
`O.8
`0.8
`
`O. O4
`
`> O. 4
`
`-0
`
`— 0.8
`- 0.8
`
`- 1.2
`— 1.2
`
`1
`
`1
`
`1 1 1 1
`
`.
`5
`
`•
`
`—
`
`Classification Accuracy
`Classification Accuracy
`95.8% for SSM
`95.8% for SSM
`O
`87.5% for MIS
`87.5% for MIS
`. 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
`i
`10
`15
`20
`25
`30
`35
`40
`SON NUMBER
`LESION NUMBER
`
`l
`
`Fig. 13
`Fig. 13
`
`Petitioner's Exhibit 1013
`Page 18 of 37
`
`

`

`U.S. Patent
`U.S. Patent
`
`Mar. 27, 2001
`Mar. 27, 2001
`
`S