1111111111111111 IIIIII IIIII 1111111111 11111 1111111111 111111111111111 111111111111111 11111111
`US 20180188108Al
`
`c19) United States
`c12) Patent Application Publication
`FAWZY
`
`c10) Pub. No.: US 2018/0188108 Al
`Jul. 5, 2018
`(43) Pub. Date:
`
`(54) METHOD AND APPARATUS FOR SPECTRAL
`REFLECTANCE IMAGING USING DIGITAL
`CAMERAS
`
`(71) Applicant: Zycor Labs Inc., Richmond (CA)
`
`(72)
`
`Inventor: Yasser FAWZY, Vancouver (CA)
`
`(21) Appl. No.: 15/703,981
`
`(22) Filed:
`
`Sep. 13, 2017
`
`Related U.S. Application Data
`
`(60) Provisional application No. 62/396,730, filed on Sep.
`19, 2016.
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`G0JJ 3102
`G0JJ 3/28
`G0JJ 3/10
`
`(2006.01)
`(2006.01)
`(2006.01)
`
`(52) U.S. Cl.
`CPC ............. G0JJ 31027 (2013.01); G0JJ 3/2823
`(2013.01); G0JJ 2003/102 (2013.01); G0JJ
`3/10 (2013.01); G0JJ 310232 (2013.01)
`ABSTRACT
`(57)
`A method and spectral light-based apparatus with an embed(cid:173)
`ded (built-in) spectral calibration module for acquiring
`multi-spectral reflectance images from a digital camera are
`disclosed. The apparatus may be an attachment device,
`which may be integrated with a consumer digital camera
`(such as smartphone camera), and may measure and/or
`estimate spectral reflectance and true color values for an
`object recorded by the camera. An example apparatus com(cid:173)
`prises an array of monochromatic light sources, preferably
`pulsed LEDs, irradiating in a time-multiplexed manner to
`generate light spectra in the range of 400 nm-1000 nm, an
`optical lens to limit the field of view of the attached camera,
`an electro-mechanical shutter or plate with its inner (reflec(cid:173)
`tion) surface coated with a diffuse reflectance standard to
`ensure flat spectral response, and an interface module for
`synchronizing the time-multiplexed light spectra with the
`coated shutter opening and closing and with the digital
`frames acquired by the camera, such that the true spectral
`reflectance and true color value of an object can be mea(cid:173)
`sured.
`
`Petitioner's Exhibit 1008
`Page 1 of 19
`
`

`

`Patent Application Publication
`
`Jul. 5, 2018 Sheet 1 of 10
`
`US 2018/0188108 Al
`
`Electrical
`Synchronization/
`Triggering
`
`Spectral Sampling
`Module
`
`Digital Camera
`
`104
`
`White
`Reflectance(cid:173)
`Coated Electro(cid:173)
`mechanical
`shutter
`
`FIG. 1A
`
`FIG. 1B
`
`Petitioner's Exhibit 1008
`Page 2 of 19
`
`

`

`Patent Application Publication
`
`Jul. 5, 2018 Sheet 2 of 10
`
`US 2018/0188108 Al
`
`102
`
`DIGITAL CAMERA
`
`s---___. \
`
`jf
`
`206
`
`21
`
`252
`
`J202
`
`204
`
`FIG. 2A
`
`!250
`
`260
`
`FIG. 28
`
`Petitioner's Exhibit 1008
`Page 3 of 19
`
`

`

`Patent Application Publication
`
`Jul. 5, 2018 Sheet 3 of 10
`
`US 2018/0188108 Al
`
`302
`
`FIG. 3A
`
`!350
`
`FIG. 3B
`
`Petitioner's Exhibit 1008
`Page 4 of 19
`
`

`

`Patent Application Publication
`
`Jul. 5, 2018 Sheet 4 of 10
`
`US 2018/0188108 Al
`
`FIG. 4A
`
`---
`
`FIG. 4B
`
`Petitioner's Exhibit 1008
`Page 5 of 19
`
`

`

`Patent Application Publication
`
`Jul. 5, 2018 Sheet 5 of 10
`
`US 2018/0188108 Al
`
`08
`
`502
`
`roo
`
`INTERFACE
`
`LED DRIVING
`CIRCUIT
`CONTROLLER
`
`DIGITAL CAMERA
`CONTROLLER
`
`FIG. 5
`
`Petitioner's Exhibit 1008
`Page 6 of 19
`
`

`

`Patent Application Publication
`
`Jul. 5, 2018 Sheet 6 of 10
`
`US 2018/0188108 Al
`
`FIG. 6A
`
`FIG. 68
`
`FIG. 6C
`
`Petitioner's Exhibit 1008
`Page 7 of 19
`
`

`

`Patent Application Publication
`
`Jul. 5, 2018 Sheet 7 of 10
`
`US 2018/0188108 Al
`
`roo
`
`SEQUENTIAL
`ILLUMINATION OF
`TISSUE AT DIFFERENT
`MONOCHROMATIC
`WAVELENGTHS
`
`SHUTTER SPECTRAL
`MEASUREMENTS
`
`TISSUE
`SPECTRAL
`IMAGES
`
`REFLECTANCE
`- - ESTIMATION 1 - - - - 1 1
`
`TISSUE
`REFLECTANCE
`IMAGES (VS.
`WAVELENGTH}
`
`INVERSE
`MODEL
`ALGORITHM
`
`LESION
`CONTRAST
`ESTIMATOR
`
`TISSUE
`REFLECTANCE
`IMAGES(VS.
`DEPTH)
`
`718
`
`FIG. 7
`
`Petitioner's Exhibit 1008
`Page 8 of 19
`
`

`

`Patent Application Publication
`
`Jul. 5, 2018 Sheet 8 of 10
`
`US 2018/0188108 Al
`
`Electrical
`Synchronization/
`Triggering
`
`Standard
`Reflectance(cid:173)
`Coated Electro(cid:173)
`mechanical
`Shutter
`
`RGB Digital
`Camera
`
`02
`
`FIG. SA
`
`85°\.
`
`A
`
`A
`
`FIG. 8B
`
`Petitioner's Exhibit 1008
`Page 9 of 19
`
`

`

`Patent Application Publication
`
`Jul. 5, 2018 Sheet 9 of 10
`
`US 2018/0188108 Al
`
`908
`
`UPDATED
`REFLECTANCE
`ESTIMATOR
`
`COATED
`SHUTTER
`REFLECTANCE
`ESTIMATOR
`
`06
`
`YES
`
`Hschanged?
`
`NO
`
`INITIAL
`REFLECTANCE
`CALIBRATION
`ESTIMATOR
`
`FIG. 9
`
`Petitioner's Exhibit 1008
`Page 10 of 19
`
`

`

`Patent Application Publication
`
`Jul. 5, 2018 Sheet 10 of 10 US 2018/0188108 Al
`
`FIG. 10A
`
`1004
`
`LIGHT FROM
`SURROUNDINGS
`
`FIG. 10B
`
`Petitioner's Exhibit 1008
`Page 11 of 19
`
`

`

`US 2018/0188108 Al
`
`Jul. 5, 2018
`
`1
`
`METHOD AND APPARATUS FOR SPECTRAL
`REFLECTANCE IMAGING USING DIGITAL
`CAMERAS
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`[0001] This application claims the benefit under 35 U.S.C.
`§ 119 of U.S. Application No. 62/396,730 filed 19 Sep. 2017
`and entitled METHOD AND APPARATUS FOR SPEC(cid:173)
`TRAL REFLECTANCE IMAGING USING DIGITAL
`CAMERAS which is hereby incorporated herein by refer(cid:173)
`ence for all purposes.
`
`FIELD
`
`[0002] This invention relates to imaging. Some embodi(cid:173)
`ments provide spectral intensity calibration devices for
`cameras. Such devices may be provided as camera acces(cid:173)
`sories or may be built into cameras. applications of invention
`include multi-spectral imaging. Some embodiments provide
`camera-based devices configured for multi-spectral imaging
`and color shade measurements.
`
`BACKGROUND
`
`[0003] Spectral reflectivity is the fraction oflight reflected
`by an object. This fraction typically varies with the wave(cid:173)
`length of light incident on the object. Spectral reflectivity
`can be used to characterize the surface of an object. Digital
`cameras can be used to estimate the spectral reflectance of
`a surface. However, digital cameras are not optimized for
`measuring spectral reflectance or for quantitative color mea(cid:173)
`surements.
`[0004] Most digital cameras capture digital color images
`in a trichromatic fashion using three distinct sets of detec(cid:173)
`tors. The detectors may, for example, be complementary
`metal-oxide semiconductor ("CMOS") or charge-coupled
`device ("CCD") sensors. Each set of detectors is sensitive to
`different wavelengths (colors) of light. Typical digital cam(cid:173)
`eras have detectors sensitive to each of: red (R), green (G),
`and blue (B) light. Such cameras ("RGB digital cameras")
`are incapable of reproducing hi-fidelity color images which
`are required in fields such as tele-dermatology. In addition,
`RGB digital cameras have a limited color gamut and cannot
`reproduce a full range of colors and shades. These inherent
`limitations are more pronounced when the surface of an
`object contains reddish colors.
`[0005] Because of the limitations ofRGB digital cameras,
`RGB digital cameras cannot be used where accurate mea(cid:173)
`surement and/or reproduction of colors is required. For
`example, RGB digital cameras have limited application in
`color reproduction for tele-medicine, image archiving and
`follow-up, color matching and measurements, and tissue
`characterization and quantification in cancer characteriza(cid:173)
`tion.
`[0006] One method for obtaining accurate information
`about the color of a surface using a RGB digital camera is
`to use the camera to obtain images while illuminating the
`surface with spectral light at different wavelengths. The
`spectral light comprises a set of spectral bands, preferably
`narrow bands, centered on each of the different wavelengths.
`The spectral reflectance images are then acquired using an
`RGB camera. If the sensitivity of the sensors of the RGB
`camera at the wavelengths of the spectral light are known
`and the intensity of illumination in the different bands of the
`
`spectral light are known then the resulting set of images can
`be processed to determine the spectral reflectance of the
`surface.
`[0007] This method is cost effective but requires a spec(cid:173)
`trometer and a dedicated lab facility to characterize the
`spectral sensitivity of digital camera sensors. It further
`requires pre-measurement calibration using a calibration
`target with a known reflectance at each wavelength, e.g. a
`white 99% diffuse reflectance disk. The pre-measurement
`calibration using a white diffuse reflectance disk ensures the
`effective spectral calibration of a camera sensor, especially
`when used to acquire spectral images in a non-lab-controlled
`environment.
`[0008] There remains a need for digital cameras which can
`provide accurate spectral images of a target, even in dynami(cid:173)
`cally changing viewing or illumination conditions. There is
`a particular need for such cameras that are cost effective
`enough for widespread application in fields such as tele(cid:173)
`dermatology, color matching and the like.
`
`SUMMARY
`
`[0009] This invention has a number of aspects. These
`aspects may be combined but may also be applied individu(cid:173)
`ally or in sub-combinations. These aspects include, without
`limitation:
`[0010] multi-spectral imaging attachments for use with
`digital cameras. Such attachments may include light
`sources and calibration devices;
`[0011] digital cameras with built-in multi-spectral
`imaging capabilities;
`[0012]
`calibration devices for use with digital cameras,
`such devices may include reflective surfaces that can be
`selectively placed in the field of view of the camera.
`Such calibration devices may be applied to determine
`intensities of different bands of spectral light for multi(cid:173)
`spectral imaging or to characterize ambient light;
`[0013]
`software (which may be in the form of firmware
`or and app, for example) useful for performing multi(cid:173)
`spectral imaging using a digital camera;
`[0014] methods for calibrating digital cameras; and
`[0015] methods for acquiring multi-spectral images
`using digital cameras.
`[0016] One example aspect provides a fully portable appa(cid:173)
`ratus that can be integrated with or attached to a consumer/
`commercial digital camera to provide a spectral imaging and
`color value measurement device.
`[0017] One aspect of the invention provides a spectral
`light source having an embedded (built-in) spectral intensity
`calibration module combined with a digital camera-based
`device, for use in multi-spectral imaging and/or true color
`shade measurements. The spectral light source and calibra(cid:173)
`tion module may be provided as an accessory or add-on to
`the digital camera or may be built in to the digital camera.
`The digital camera may be a purpose-built camera or a
`camera that forms part of a computing device such as a
`camera of a smartphone or tablet or an accessory camera
`attached to a personal computer, for example.
`[0018] Another aspect of the invention provides cameras
`equipped with shutters having surfaces that have a known,
`preferably flat spectral response. The shutters, may, for
`example, be coated with a white or grey surface coating.
`Such shutters may be used to calibrate the cameras. Such
`shutters may be built into a camera or added as an accessory.
`Some embodiments include a control unit that processes
`
`Petitioner's Exhibit 1008
`Page 12 of 19
`
`

`

`US 2018/0188108 Al
`
`Jul. 5, 2018
`
`2
`
`images of the shutter illuminated by ambient light to char(cid:173)
`acterize the ambient light or images of the shutter illumi(cid:173)
`nated by spectral light to characterize the spectral light.
`[0019] Another aspect of the invention provides a multi(cid:173)
`spectral imaging system. The multi-spectral imaging system
`comprises a digital camera having an imaging lens. A
`spectral intensity calibration module includes an electro(cid:173)
`mechanical shutter coated with a material having a known
`reflectance (e.g. a white or grey coating) and a control
`module. A spectral light source is arranged to illuminate an
`object in a field of view of the lens or the surface of the
`electromechanical shutter depending on whether the shutter
`is open or closed. The control module synchronizes electri(cid:173)
`cal triggering of the light source with opening or closing of
`the shutter.
`[0020] Another aspect of the invention provides a multi(cid:173)
`spectral imaging system. The multi-spectral imaging system
`comprises a digital camera having an imaging lens; a
`spectral light source; and a control module. The control
`module synchronizes electrical triggering of the light source
`to emit bands of one or more different wavelengths with
`acquisition of an image by the camera.
`[0021] An example spectral light apparatus accessory
`comprises a housing which encloses a spectral light source
`operative to produce different monochromatic wavelengths;
`an optical lens system aligned with an optical axis of a
`camera such as a smartphone camera; and an electro(cid:173)
`mechanical shutter (such as a leaf camera shutter) covering
`a distal opening in the housing. The spectral light source
`may be controlled to emit different wavelengths in a time(cid:173)
`multiplexed manner. The housing limits the field of view of
`the camera to the area defined by the distal opening in the
`housing. The shutter can selectively cover or leave open the
`distal opening of the housing. A surface of the shutter facing
`the camera is coated with a standard diffuse reflectance
`coating to produce a known, preferably flat, spectral
`response suitable for spectral calibration. The optical lens
`system controls the optical coupling of a lens of the smart(cid:173)
`phone camera to ensure that it captures only the reflected
`light originating from the interaction of the spectral light
`with the object or surface being imaged. The housing
`comprises a built-in and time-synchronized spectral calibra(cid:173)
`tion module, which performs spectral calibration under the
`same viewing conditions as the acquired spectral images of
`the object or surface being imaged.
`[0022] The spectral light source may comprise an array of
`light-emitting diodes ("LEDs"). Control circuits may elec(cid:173)
`trically synchronize operation of the LEDs with the smart(cid:173)
`phone or digital camera trigger control, and the opening or
`closing of the white reflectance-coated mechanical shutter.
`Analysis software such as a function built into firmware of
`the camera or a mobile app uses the acquired synchronized
`frames to calculate the spectral reflectance of the object or
`surface that is illuminated by the light source. The analysis
`software uses the acquired frames obtained from the embed(cid:173)
`ded coated shutter when it is closed, and uses the frames
`obtained from the object or surface when the shutter is
`opened, in order to estimate spectral reflectance and/or color
`values of the object or surface being imaged.
`[0023] Another aspect of the invention provides a spectral
`light apparatus that can be used with a digital camera, such
`as a smartphone camera. This apparatus allows a digital
`camera to acquire spectral reflectance images and color
`values from an object or surface.
`
`[0024] Another aspect of the invention provides methods
`for multi-spectral imaging.
`[0025] A further aspect of the present invention provides
`a digital camera suitable for skin imaging, and in particular
`for generating depth-resolved skin images from acquired
`spectral images.
`[0026] Various embodiments of the present invention have
`medical and/or non-medical applications, including per(cid:173)
`forming tele-medicine procedures (such as tele-dermatol(cid:173)
`ogy), performing color matching, improving color image
`synthesis and realism, performing computer-aided diagnosis
`("CAD"), and quantifying and characterizing tissue/material
`properties and changes.
`[0027] Further aspects and example embodiments are
`illustrated in the accompanying drawings and/or described
`in the following description.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0028] The accompanying drawings illustrate non-limit(cid:173)
`ing example embodiments of the invention.
`[0029] FIG. lA is a schematic block diagram of an
`example apparatus for estimating reflectance image spectra
`using a multi-spectral imaging system with an embedded
`(built-in) white reflectance-coated shutter calibration mod(cid:173)
`ule.
`[0030] FIG. 1B shows an example reflectance spectrum
`for the FIG. lA white reflectance-coated shutter calibration
`module.
`[0031] FIGS. 2Aand 2B are schematic cross section views
`of example spectral imaging attachment devices. The device
`of FIG. 2A is adapted for macro-imaging. The device of FIG.
`2B is adapted for distance imaging.
`[0032] FIGS. 3A and 3B are schematic views of a multi(cid:173)
`spectral imaging camera with a built-in white reflectance(cid:173)
`coated shutter calibration module.
`[0033] FIGS. 4A and 4B respectively illustrate example
`spectral imaging attachments for a smartphone and a web(cid:173)
`cam or digital camera.
`[0034] FIG. 5 is a block diagram illustrating a system for
`electronic synchronization between an electro-optical
`attachment and a smartphone/digital camera processor.
`[0035] FIGS. 6A, 6B, and 6C illustrate example shutter
`types suitable for white reflectance-coating.
`[0036] FIG. 7 is a process flow diagram illustrating the
`operation of a mobile app or algorithm for quantifying the
`depth of skin lesions.
`[0037] FIG. SA is a schematic block diagram of an
`example RGB digital camera having an embedded standard
`reflectance-coated shutter calibration module.
`[0038] FIG. 8B is an example of the FIG. SA standard
`reflectance-coated shutter calibration module, with some
`example possible reflectance spectra.
`[0039] FIG. 9 is a process flow diagram illustrating a
`dynamic spectral calibration algorithm.
`[0040] FIGS. l0A and 10B are schematic illustrations of
`an example digital camera having an embedded standard
`reflectance-coated shutter calibration module.
`
`DETAILED DESCRIPTION
`
`[0041] Throughout the following description, specific
`details are set forth in order to provide a more thorough
`understanding of the invention. However, the invention may
`be practiced without these particulars. In other instances,
`
`Petitioner's Exhibit 1008
`Page 13 of 19
`
`

`

`US 2018/0188108 Al
`
`Jul. 5, 2018
`
`3
`
`well known elements have not been shown or described in
`detail to avoid unnecessarily obscuring the invention.
`Accordingly, the specification and drawings are to be
`regarded in an illustrative, rather than a restrictive sense.
`[0042] One embodiment of the present invention relates to
`a digital camera-based device having a built-in spectral
`intensity calibration module. This device may be attached to
`any digital camera including, for example, a smartphone
`camera. The device allows the digital camera to perform
`multi-spectral imaging and/or color shade measurement.
`The device may dynamically re-calibrate the digital camera
`when viewing or illuminations conditions change.
`[0043] The general principles of spectral imaging using a
`calibration surface when producing images will now be
`described, followed by a description of some specific
`example embodiments of the invention.
`[0044] A digital camera system equipped to perform
`multi-spectral imaging may apply the following principle.
`The measured signal reflected from some target (TARGT)
`and recorded by the digital camera system can be described
`by the following model:
`
`l TARG/J,x,y)~fF(/c)E(/c)S(/c)R(/c)aA,
`
`where A is the wavelength of illumination, IrARGr is the
`measured spectral reflectance of the target at a certain
`location and point in time, F(A) is the response of the optical
`system (including the response of the optical lens and/or
`filters), E(A) is the spectrum of the light source, S(A) is the
`camera sensitivity, and R(A) is the target's real reflectance
`spectrum.
`[0045] A sequentially flashed light (such as the light
`emitted by a pulsed LED ring), illuminating both a target and
`a calibration surface at the same distance as the target, may
`be described by the following model:
`
`:E,J TARGrtic,t 1,x,y)~LF1 (ic,t ),x,y)E 1 (lc,t ),x,y)S 1 (/c,t ),x,y)
`R(ic,t1,x,y);
`
`:E.lcal(ic,t2,X,y)~IT2(ic,t2,X,y)E2(ic,t2,x,y)S2(ich,x,y)R(/c,
`12,x,y);
`[0046] where Ical is the measured reflectance spectrum of
`the calibration surface (which may comprise a white-coated
`electro-mechanical shutter for example), Rand Lare respec(cid:173)
`tively the real reflectance spectra of the target and calibration
`surface, the subscript 1 refers to properties of the target and
`the calibration surface at a first time, and the subscript 2
`refers to properties of the target and the calibration surface
`at a second, later time. The difference between t 1 and t2 (i.e.
`the length of time between sequential flashes of illuminating
`light) may be one the order of seconds or milliseconds, for
`example.
`[0047] For simultaneous (or sequential) imaging of the
`target surface and the calibration surface at the same location
`as the target, the model becomes:
`
`F 1(ic,t1,x,y)~F2(ich,x,y)~F1(/c);
`
`R(/c)~[:E.J TARGp@caz]L(/c).
`
`[0048] To measure the true target reflectance ( or true color
`image of the target), the reflectance L(A) of the calibration
`surface should have a known spectral response within the
`measured wavelength range. Additionally, measurement of
`
`the target and the calibration surface should be done under
`similar illumination and viewing conditions. It is convenient
`but not mandatory for the known spectral response of the
`calibration surface to be flat over the measured wavelength
`range.
`[0049] Some embodiments of the invention integrate a
`calibration module within a digital camera system. Such a
`built-in calibration module may estimate the effective spec(cid:173)
`tral sensitivities of the detectors in the camera at the place
`and time that the camera is used to image a target.
`In some embodiments, a calibration module uses
`[0050]
`images of a calibration surface coated with a standard flat
`response diffuse reflectance coating such as SpectraR(cid:173)
`eflect™ diffuse reflectance coating (which is a water-based
`barium sulfate coating).
`In some embodiments a calibration surface having
`[0051]
`a known reflectance at different wavelengths is provided on
`an inside face of a shutter covering the lens of a digital
`camera. The spectral response of the shutter may be estab(cid:173)
`lished by coating the inner surface of the shutter with a
`standard reflectance coating so that the real spectral response
`L(A) is a known, preferably flat, spectral response (that is,
`L(A) is preferably relatively constant over the desired wave(cid:173)
`length range).
`[0052] To correct for variations with wavelength in a
`spectral response of a calibration surface ( e.g. a coated
`shutter) the calibration module can use a mapping table
`which indicates a difference between L(A) and the reflec(cid:173)
`tance of a diffuse reflectance standard at different wave(cid:173)
`lengths. The difference corresponds to an input value in a
`looknp table ("LUT"). The LUT based algorithm then
`corrects the measured spectral response of the calibration
`surface, by transforming the input value into an output
`value.
`[0053] Some embodiments automatically set a camera to
`acquire images of an object illuminated with light of one or
`more wavelengths and images of a calibration surface illu(cid:173)
`minated in the same manner. For example, the camera may
`automatically acquire images of an object and images of a
`calibration surface ( e.g. the inside of a shutter) in alternation.
`The shutter may be automatically opened and closed to
`facilitate such a sequence of images. Other possibilities
`include:
`acquiring images of an object between two
`[0054]
`images of a calibration surface ( each of these images
`may be acquired under the same illumination condi(cid:173)
`tions); or
`images of an calibration surface
`acquiring
`[0055]
`between two images of an object (each of these images
`may be acquired under the same illumination condi(cid:173)
`tions); or
`acquiring a sequence of images of a calibration
`[0056]
`surface and an object with the images of the calibration
`surface acquired within a short time interval of the
`images of the object (collectively the sequence may
`include one or more images of the object acquired
`under each of a plurality of illumination conditions and
`one or more images of the calibration surface acquired
`under each of the same plurality of illumination con(cid:173)
`ditions).
`[0057] Synchronizing shutter reflectance images with the
`acquired spectral images of the target may provide a feed(cid:173)
`back control to ensure that the digital camera system is
`spectrally calibrated during measurement.
`
`Petitioner's Exhibit 1008
`Page 14 of 19
`
`

`

`US 2018/0188108 Al
`
`Jul. 5, 2018
`
`4
`
`[0058] For an RGB color digital camera, an illuminator
`may be controlled to perform simultaneous double or triple
`band illumination to speed up spectral image acquisition and
`measurements. Each triple band illumination may include
`light of three wavelength bands with each of the wavelength
`bands detectable by one of the RGB sensors of the digital
`camera.
`[0059] With these principles in mind, some non-limiting
`example embodiments of the invention will now be
`described.
`[0060] FIG. lA illustrates a multi-spectral imaging system
`100 with an embedded (built-in) reflectance-coated shutter
`calibration module. System 100 includes an electro-me(cid:173)
`chanical shutter 104 coated with a material that provides a
`known, preferably reasonably high reflectivity. For example,
`the inner surface of the shutter may be white or gray. The
`shutter is embedded (built-in) within the housing of multi(cid:173)
`spectral imaging system 100 together with a spectral sam(cid:173)
`pling module comprising imaging lens 106; and a control
`module 108 to synchronize electrical triggering of the light
`source with opening or closing of shutter 104. Electro(cid:173)
`mechanical shutter 104 has a known spectral response, such
`as the spectrum 110 shown in FIG. 1B.
`In operation, system 100 sends a trigger signal
`[0061]
`from the control module 108 to close electro-mechanical
`shutter 104. System 100 sends another trigger signal from
`the control module 108 to spectral sampling module 106 to
`illuminate the closed shutter 104 with spectral light. System
`100 then produces a series of multi-spectral images Is(A,x,
`y,t) of the closed shutter 104. System 100 then sends a
`trigger signal from the control module 108 to open the white
`or grey coated electro-mechanical shutter 104. When the
`white or grey coated electro-mechanical shutter 104 is open,
`system 100 produces multi-spectral images It(A,x,y,t) of a
`target object. System 100 may then process the acquired
`images to estimate the spectral reflectance of the surface of
`the target object using the following equation: Rt(A,x,y,t)=It
`(A,x,y,t)/Is(A,x,y,t). This estimation may be performed on a
`pixel-by-pixel basis.
`[0062] System 100 may be configured to acquire a series
`of multi-spectral images It(A,x,y,t) of the surface of a target
`object and a series of multi-spectral images Is(A,x,y,t) of the
`inner surface of electro-mechanical shutter 104 within a
`short time frame. In each image, the surface of a target object
`and the shutter 104 may be illuminated by light of a different
`wavelength ( or different combination of discrete wave(cid:173)
`lengths). These images may then be processed on a per-pixel
`basis, to determine a spectral calibration which relates a
`detected light intensity to a reflectance of a target object at
`a given wavelength, using the following equation: Rt(A,x,
`y, t )= It(A,x,y, t )/Is(A,x,y, t).
`[0063] System 100 may be provided as an attachment to
`any standard digital camera 102, such as a smartphone
`camera.
`[0064] Synchronization between operation of the camera
`to acquire images, operation of the light source of spectral
`sampling module 106 and operation of shutter 104 may be
`facilitated by software running on the camera or on a
`computer that controls the camera. For example, where the
`camera is the camera of a smartphone the software may
`comprise an app running on the smartphone or may be built
`into an operating system of the smart phone. where the
`camera is a stand-alone digital camera the software may be
`included in firmware of the digital camera, where the camera
`
`is an accessory connected to a computer the software may
`comprise application software executing on the computer.
`The software may communicate to control module 108 by a
`wired interface ( e.g. by sending signals to control module
`108 by way of a USB interface or other digital data interface
`provided as part of the camera or a computer controlling the
`camera) or by way of a wireless interface ( e,g, by way of a
`Bluetooth™ or WiFi or other wireless signal transmitted
`from a wireless interface of the camera or a computer
`controlling the camera and received by a wireless interface
`connected to control module 108.
`[0065]
`In a non-limiting example embodiment, the soft(cid:173)
`ware executing on the camera or associated computer con(cid:173)
`trols the camera to take a series of images ( either as still
`images or a series of frames of a video image). For each
`image the software sends a synchronization signal to control
`module 108. Control module 108 controls light source of
`spectral sampling module 106 to emit light of one or more
`colors desired for the current image and controls shutter 104
`to be open or closed depending on whether the current image
`is intended to be a calibration image or an image of the
`object.
`[0066] Control of which images are to be calibration
`images and what colors of light illuminate each image may
`be in control module 108 or in the software that controls the
`camera. Where control over these parameters are in the
`software that controls the camera then the software may
`include information specifying whether shutter 104 should
`be open or closed and what spectral bands should be enabled
`for illumination either as part of the synchronization signals
`or in a separate communication.
`[0067] The software may automatically associate together
`the acquired images. The software may optionally perform
`analysis of the acquired images.
`[0068] FIGS. 2A and 2B illustrate multi-spectral imaging
`attachments for a smartphone camera. FIG. 2A shows an
`attachment useful for macro imaging, such as dermoscopy
`imaging in the field of dermatology. FIG. 2B shows an
`attachment useful for distance imaging, such as clinical
`imaging in the field of dermatology.
`[0069] As shown in FIG. 2A, apparatus 200 includes a
`device housing 202 that contains: a light source 204 com(cid:173)
`prising one or more monochromatic light sources for sup(cid:173)
`plying spectral light illumination; an optical lens system 208
`for controlling the field of view of digital camera 102 and for
`capturing reflected spectral light; and an electrically-con(cid:173)
`trolled mechanical shutter or plate 212 coated with standard
`white reflectance for calibrating the spectral response of
`digital camera 102. Apparatus 200 further includes an elec(cid:173)
`tronic interface module 206 that selects the monochromatic
`light source(s) of light source 204 to be triggered and
`synchronizes the triggering sequence with the frames
`acquired by the attached digital camera 102. The synchro(cid:173)
`nization may be achieved through an electrical communi(cid:173)
`cation protocol. For example, the acquisition time of image
`frames by the camera may be synchronized with the times
`that spectral light source 204 is on.
`[0070] A communication protocol may initiate the on
`signal for spectral light source 204 when the attached digital
`camera 102 receives a trigger signal to acquire images.
`Another communication protocol may synchronize the fre(cid:173)
`quency of the on time for spectral light with the frame rate
`of the attached digital camera 102. Apparatus 200 further
`includes a hood 210 to collect and diffuse light. In addition,
`
`Petitioner's Exhibit 1008
`Page 15 of 19
`
`

`

`
`
`US 2018/0188108 Al
`
`Jul. 5, 2018
`
`5
`
`
`hood 210 is useful as a fixed 216
`
`spacer between a target
`
`ments, processing is performed in all or in part by a
`lens system 208 to facilitate focusing
`
`(e.g. skin) and optical
`
`
`smartphone app ( or a web app) that runs on the device
`216 for macro-imaging.
`camera 102 instead ofby a separate processing
`light to the target
`hosting
`unit.
`
`[0071] Light source 204 provides
`
`light in the visible or in
`
`[0077] Optical lens system 208 may comprise
`macro
`
`
`
`
`
`the visible and near-infrared region, and may be a single unit
`
`
`
`
`lenses with variable optical fields of view. The macro lenses