USOO7349574B1
`
`(12) United States Patent
`Sodini et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,349,574 B1
`Mar. 25, 2008
`
`(54) SYSTEM AND METHOD FOR PROCESSING
`NON-LINEAR IMAGE DATA FROMA
`DIGITAL MAGER
`
`(*) Notice:
`
`(75) Inventors: Charles G. Sodini, Belmont, MA (US);
`Jason Y. Sproul, Watertown, MA (US);
`Edward T. Chang, Cambridge, MA
`(US)
`(73) Assignee: Sensata Technologies, Inc., Attleboro,
`MA (US)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 931 days.
`(21) Appl. No.: 10/685,126
`(22) Filed:
`Oct. 14, 2003
`Related U.S. Application Data
`(60) Provisional application No. 60/417,862, filed on Oct.
`11, 2002.
`
`(51) Int. Cl.
`(2006.01)
`G06K 9/00
`(52) U.S. Cl. ....................................... 382/168; 34.5/589
`(58) Field of Classification Search ........ 382/168-172,
`382/312,321; 34.5/589-605
`See application file for complete search history.
`References Cited
`
`(56)
`
`U.S. PATENT DOCUMENTS
`6.421,084 B1
`7/2002 Chang et al.
`9, 2003 Minemier
`6,614,562 B1
`7,184,066 B2*
`2/2007 Elliot et al. ................. 345,694
`OTHER PUBLICATIONS
`“A 256x256 CMOS Imaging Array with Side Dynamic Range
`Pixels and Column-Parallel Digital Output.” Decker et al. IEEE
`Journal of Solid-State Circuits. Dec. 1998, vol. 33, No. 12.
`
`“Autobrite: Method for Controlling the Dynamic Range of an Image
`Sensor.” SMAL Camera Technologies. Dec. 7, 2001.
`“Color Filter Array Recovery Using a Threshold-based Variable
`Number of Gradients.” Chang et al. Compaq Computer Corppora
`tion. Cambridge Research Lab, Cambridge, MA., 1999.
`
`* cited by examiner
`Primary Examiner Jose L. Couso
`(74) Attorney, Agent, or Firm—Russell E. Baumann
`
`(57)
`
`ABSTRACT
`
`A system and method process non-linear image data, still or
`Video, from a digital imager. Noise generated by analog-to
`digital converters is filtered from a pixel of digital image
`data. Moreover, the effects of single pixel defects in the
`imager are eliminated by clamping a predetermined pixel of
`image data within the window when the value of the
`predetermined pixel is greater than a maximum value of the
`image data of neighboring pixels or less than a minimum
`value of the image data of neighboring pixels. Ripples in
`image data are reduced by eliminating the effects of single
`pixel defects before filtering for crosstalk caused by elec
`trical crosstalk between sensor elements in an imager. Dark
`current is removed from image data generated by an imager
`by Subtracting a fraction of a determined dark current value
`from all image data generated by the imager to compensate
`for nonlinearities in dark current across the imager. The
`image data is white balanced by creating a set of scalar color
`adjustments from determined average color values and con
`straining the set of Scalar adjustments to plausible lighting
`conditions to prevent overcompensation on images having
`large regions of similar hue. Lastly, utilization of a fixed set
`of intensity levels is optimized by remapping and restrech
`ing the image data to create new luma values for each pixel.
`
`13 Claims, 10 Drawing Sheets
`
`
`
`MAGER
`
`ANALOG
`TO
`DGITAL
`CONVERTER
`
`COLOR
`MAGE
`PROCESSING
`
`IMAGE DATA
`REMAPPNG
`
`NOSE
`FILTERING 1. '
`
`APPLE v. RED.COM
`
`Page 1 of 27
`
`Apple Ex. 1013
`
`

`

`U.S. Patent
`
`Mar. 25, 2008
`
`Sheet 1 of 10
`
`US 7,349,574 B1
`
`
`
`5) NISSCHOONHOEI
`
`GIOVJANI
`
`GH5OVINI
`
`NOIJLOTHRIRHOO
`
`TV LIÐICI
`
`RIGHLAIGHANOO
`
`
`
`(LIV JOJ?) I TRIQ OIH
`
`
`
`?OTVNV ?NTRIGH LTI HYHQIÐVINI
`
`OJL
`
`
`
`
`
`
`
`
`
`
`
`
`
`APPLE v. RED.COM
`
`Page 2 of 27
`
`Apple Ex. 1013
`
`

`

`U.S. Patent
`
`Mar. 25, 2008
`
`Sheet 2 of 10
`
`US 7,349,574 B1
`
`
`
`
`
`RIGH LÈRIGHANOO
`
`TV LIÐICI
`
`OJL
`
`?OTVNV
`
`YI?IÐVWI
`
`(HSION
`
`
`
`?NTRIGH LTIJI
`
`?ÐNI, I, IVWORTH V LVCI GIÐ
`VINI
`
`YHOTOO
`
`?IÐVINI
`
`?NISSCHOORIAI
`
`
`
`Z ETRIQ OIH
`
`APPLE v. RED.COM
`
`Page 3 of 27
`
`Apple Ex. 1013
`
`

`

`U.S. Patent
`
`Mar. 25, 2008
`
`Sheet 3 of 10
`
`US 7,349,574 B1
`
`?7 (ETRIQ OIH
`
`
`
`£ €TRIQ OIH
`
`00 I
`
`APPLE v. RED.COM
`
`Page 4 of 27
`
`Apple Ex. 1013
`
`

`

`s”U
`
`aM
`
`5
`
`ehSm
`
`01f04
`
`m9,437,m
`
`1B
`
`4mEDGE
`
`t852829:39:48mmmmemxm823%M5:;P8
`
`
`2,oémgmm
`
`
`
`m.~32:cm
`
`a29349257:5
`
`mmHOZ
`
`Uémhdm
`
`ED
`
`HZmMMDO
`
`A<>Ozma
`
`MQ<HmmOMU
`
`ZOHHODQmE
`
`5
`
`mm
`
`mm
`
`moqoo
`
`
`
`zofiommmoumm
`
`wmmMEEOOmQ
`
`$32):
`
`<H<Q
`
`vm
`
`APPLE V. REDCOM
`
`Page 5 of 27
`
`Apple EX. 1013
`
`APPLE v. RED.COM
`
`Page 5 of 27
`
`Apple Ex. 1013
`
`
`
`

`

`Ps”U
`
`t
`
`r._aM
`
`00m
`
`hS
`
`01f05
`
`5,943,7SU
`
`1B
`
`M0EDGE
`
`n5586mmmoz<q<mmHEB3mm
`
`40528onH<mmezoo
`
`
`
`2,wémmgmm%~2szom
`
`wZOFEEMEZHa
`
`mmHOZ
`
`SEW—DE
`
`mmmyEEOOmQ
`
`m0<~>=
`
`<H<Q
`
`Em
`
`HZEDU
`
`i>02mm
`
`5
`
`vm
`
`mm
`
`5qu
`
`
`
`295528mm
`
`MéHmmOMU
`
`ZOHHUDQmE
`
`mm
`
`APPLE V. REDCOM
`
`Page 6 of 27
`
`Apple EX. 1013
`
`APPLE v. RED.COM
`
`Page 6 of 27
`
`Apple Ex. 1013
`
`
`
`
`

`

`U.S. Patent
`
`Mar. 25, 2008
`
`Sheet 6 of 10
`
`US 7,349,574 B1
`
`
`
`to
`
`bo
`
`1 animefiel
`
`Fq ft 7 C Converthwark Art)
`
`APPLE v. RED.COM
`
`Page 7 of 27
`
`Apple Ex. 1013
`
`

`

`U.S. Patent
`
`Mar. 25, 2008
`
`Sheet 7 of 10
`
`US 7,349,574 B1
`
`Loll wave-rou
`mo Te Pust ty
`M7A af2 FMJC
`Coast radlera
`
`
`
`
`
`
`
`
`
`
`
`7764 ws ae?a
`Cowraee
`Azewc7 roa
`65ue CATO-2
`Cl2C 7
`
`APPLE v. RED.COM
`
`Page 8 of 27
`
`Apple Ex. 1013
`
`

`

`U.S. Patent
`
`Mar. 25, 2008
`
`Sheet 8 of 10
`
`US 7,349,574 B1
`
`
`
`to
`
`f
`
`t
`
`t
`
`is to
`
`finae
`
`Tise
`
`F-1 Gu (26 7
`
`APPLE v. RED.COM
`
`Page 9 of 27
`
`Apple Ex. 1013
`
`

`

`U.S. Patent
`
`Mar. 25, 2008
`
`Sheet 9 of 10
`
`US 7,349,574 B1
`
`0.8
`
`0.4
`
`0.2
`
`O
`
`1
`
`2
`
`3
`
`4.
`
`5
`
`Cullu in A A lord
`rap T J sity leuel
`
`16-wee !C)
`
`
`
`a?
`wA
`U et F(Avia
`F2em r146-era
`
`Ccm fee SS
`fiss A Lurie V.
`wd (22
`
`APPLE v. RED.COM
`
`Page 10 of 27
`
`Apple Ex. 1013
`
`

`

`U.S. Patent
`
`Mar. 25, 2008
`
`Sheet 10 of 10
`
`US 7,349,574 B1
`
`
`
`FRAME Stre
`Wait was few sets
`
`get level to be
`
`
`
`APPLE v. RED.COM
`
`Page 11 of 27
`
`Apple Ex. 1013
`
`

`

`US 7,349,574 B1
`
`1.
`SYSTEMAND METHOD FOR PROCESSING
`NON-LINEAR IMAGE DATA FROMA
`DIGITAL IMAGER
`
`PRIORITY INFORMATION
`
`This application claims priority from U.S. Provisional
`Patent Application Ser. No. 60/417,862, filed on Oct. 11,
`2002. The entire contents of U.S. Provisional Patent Appli
`cation Ser. No. 60/417,862, are hereby incorporated by
`reference.
`
`FIELD OF THE PRESENT INVENTION
`
`The present invention is directed to processing non-linear
`image data, still or video, from a digital imager. More
`specifically, the present invention is directed to providing
`noise removal processes, linear mapping processes, and
`white balance processes for preparing non-linear image data
`from a digital imager for display or further analysis.
`
`10
`
`15
`
`BACKGROUND OF THE PRESENT
`INVENTION
`
`25
`
`30
`
`35
`
`45
`
`Conventionally, image data from digital imagers are pro
`cessed in Some fashion to provide a quality digital repre
`sentation of the scene being imaged. FIG. 1 illustrates a
`conventional image process pipeline from the imager to a
`display system or other system that requires a quality digital
`representation of the imaged scene.
`As illustrated in FIG. 1, an imager 1, comprising a
`plurality of photosensitive elements commonly referred to as
`pixels, the physical realization of the pixels being either a
`plurality of phototransistors or a plurality of photodiodes
`functioning as light-detecting elements; images a scene and
`produces a plurality of Voltages. In operation a conventional
`pixel is first reset with a reset Voltage that places an
`electronic charge across the capacitance associated with:
`e.g., a photodiode. Electronic charge produced by the pho
`todiode when exposed to illumination, causes charge of the
`40
`diode capacitance to dissipate in proportion to the incident
`illumination intensity of the scene. At the end of an exposure
`period, the change in diode capacitance charge is detected
`and the photodiode is reset. The amount of light detected by
`the photodiode is computed as the difference between the
`reset Voltage and the Voltage corresponding to the final diode
`capacitance charge.
`The Voltages generated by the imager 1 are fed to an
`analog to digital converter 3. The analog to digital converter
`3 converts each analog Voltage to a digital image data word
`having a digital value in a range from 0, which convention
`ally represents no illumination, to Some maximum value that
`represents illumination saturation. If the width of the digital
`image data word were set to 8-bits, the range of digital
`values that a digital image data word could realize would be
`in the range 0 to 255, Obeing no illumination and 255 being
`saturation.
`The digital image data word is then filtered by a filtering
`subsystem 5. The filtering subsystem 5, in most conven
`tional systems, modifies the digital image data to compen
`sate for any imager characteristic artifacts that were pro
`duced by the imager 1 or to compensate the digital image
`data for defective pixels in the imager 1.
`After filtering, the filtered digital image data is processed
`in an image processing Subsystem 7 wherein the digital
`image data is modified to compensate for dark current,
`crosstalk, and/or white balance. The filtered digital image
`
`50
`
`55
`
`60
`
`65
`
`2
`data may also be decompressed in the image processing
`subsystem 7 if the sensor data received from the imager has
`been compressed. The processed digital image data is further
`corrected in an image correction Subsystem 9. The image
`correction Subsystem 9 may provide gamma correction,
`color correction, and/or sharpening. The corrected digital
`image data is then fed to a number of possible Subsystems
`that can further prepare the image data for display, printing,
`or analysis.
`Notwithstanding the various stages of filtering, process
`ing, and correcting in a conventional image data pipeline,
`the image data may still have various quality issues with
`respect to noise introduced by the imager or other compo
`nents along the pipeline. Moreover, utilizing conventional
`image processing systems, like the one illustrated in FIG. 1,
`the image data may not be properly compensated for with
`respect to the lighting conditions used to image the scene or
`the sensor color responsivity.
`It is, therefore, desirable to provide an image-processing
`pipeline that produces image data of a high quality and that
`Substantially removes any noise or artifacts introduced by
`the imager or other components along the pipeline. It is also
`desirable to provide an image processing pipeline that
`produces image data of a high quality and that Substantially
`compensates the image data with respect to the lighting
`conditions used to image the scene or the sensor color
`responsivity.
`
`SUMMARY OF THE PRESENT INVENTION
`
`A first aspect of the present invention is a method for
`removing undesirable signal characteristics from a pixel of
`still digital image data generated by analog-to-digital con
`verters. The method applies a predetermined Voltage to each
`analog-to-digital converter to generate test outputs; calcu
`lates, from the test outputs, a relative offset value for each
`analog-to-digital converter, determines a maximum relative
`offset value from the calculated relative offset values; stores
`the maximum relative offset value; stores the calculated
`relative offset values such that each analog-to-digital con
`verter has an associated stored calculated relative offset
`value; adds the stored maximum relative offset value to a
`pixel of still digital image data generated by an analog-to
`digital converter; and subtracts the stored calculated relative
`offset value associated with the analog-to-digital converter
`from the sum of the stored maximum relative offset value
`and the pixel of still digital image data generated by the
`associated analog-to-digital converter.
`A second aspect of the present invention is a method for
`removing undesirable signal characteristics from a pixel of
`Video digital image data generated by analog-to-digital
`converters. The method applies a predetermined Voltage to
`each analog-to-digital converter to generate test outputs;
`calculates, from the test outputs, a relative offset value for
`each analog-to-digital converter, compares the calculated
`relative offset value for an associated analog-to-digital con
`verter with a previously stored calculated relative offset
`value for the associated analog-to-digital converter, incre
`ments the previously stored calculated relative offset value
`by a predetermined amount if it is determined that the
`calculated relative offset value for the associated analog-to
`digital converter is greater than the previously stored cal
`culated relative offset value for an associated analog-to
`digital converter and storing a new relative offset value for
`an associated analog-to-digital converter therefrom; decre
`ments the previously stored calculated relative offset value
`for the associated analog-to-digital converter by the prede
`
`APPLE v. RED.COM
`
`Page 12 of 27
`
`Apple Ex. 1013
`
`

`

`US 7,349,574 B1
`
`10
`
`3
`termined amount if it is determined that the calculated
`relative offset value is less than the previously stored cal
`culated relative offset value for the associated analog-to
`digital converter and storing a new relative offset value for
`an associated analog-to-digital converter therefrom; deter- 5
`mines a maximum relative offset value from the stored
`relative offset values and storing the determined maximum
`relative offset value; adds the stored maximum relative
`offset value to a pixel of video digital image data generated
`by an analog-to-digital converter, and Subtracts the stored
`relative offset value associated with the analog-to-digital
`converter from the sum of the stored maximum relative
`offset value and the pixel of video digital image data
`generated by the associated analog-to-digital converter.
`A third aspect of the present invention is a method for 15
`eliminating effects of single pixel defects in the imager. The
`method provides a predetermined size window of neighbor
`ing pixels and clamps a predetermined pixel of image data
`within the window when the value of the predetermined
`pixel is greater than a maximum value of the image data of
`neighboring pixels or less than a minimum value of the
`image data of neighboring pixels.
`A fourth aspect of the present invention is a method for
`removing dark current from image data generated by an
`imager. The method shields a pair of columns of pixels at an
`edge of an image array; reads image data from the shielded
`pair of columns; averages the read image data to generate a
`dark current value; and subtracts a fraction of the dark
`current value from all image data generated by the imager to
`compensate for nonlinearities in dark current across the
`imager.
`A fifth aspect of the present invention is a method for
`white balancing image data from an imager. The method
`samples a predetermined pattern of predetermined sized
`pixel blocks; chooses a pixel for each color from each
`sampled pixel block; averages the chosen pixels of a par
`ticular to determine an average color value for the color;
`creates a set of Scalar color adjustments from the average
`color values, a scalar color adjustment for each color,
`constrains the set of scalar adjustments to plausible lighting
`conditions to prevent overcompensation on images having
`large regions of similar hue; normalizes the constrained set
`of Scalar adjustments; and applies the normalized set of
`Scalar adjustments to the image data.
`Another aspect of the present invention is a method for
`optimally utilizing a fixed set of intensity levels of a video
`display to represent a wide dynamic range data captured by
`an imager. The method determines a target grey value;
`determines a power curve from the determined target grey
`value; limits the determined power curve to a range of 50
`minimum and maximum power curves, the range of mini
`mum and maximum power curves corresponding to an
`original compression curve utilized by the imager, remaps a
`luma value of each pixel using the limited power curve;
`determines a minimum luma value BP, for a frame of 55
`image data; creates a histogram from the remapped luma
`values; determines an ideal black point value BP and an
`ideal white point value WP from the created histogram and
`determined minimum luma value; and restretches the image
`data, using the determined ideal black point value and an 60
`ideal white point value, to create new luma values for each
`pixel.
`A further aspect of the present invention is a method of
`optimally utilizing a fixed set of intensity levels of a video
`display to represent a wide dynamic range data captured by
`an imager. The method determines a target grey value;
`determines a power curve from the determined target grey
`
`4
`value; converts luma values into logarithmic luma values,
`the logarithmic luma values being independent of an input
`data range; determines a power function based upon a ratio
`between a desired logarithmic luma average and an actual
`logarithmic luma average; determines a maximum gain
`limit; determines a correct luma value; determines a luma
`correction value from the correct luma value and the actual
`luma value; and applies the luma correction value to each
`pixel input luma value to generate a corrected pixel input
`luma value.
`Another aspect of the present invention is a method for
`determining a maximum input value for a captured scene.
`The method determines a range value corresponding to a
`determined maximum input value from a previous frame and
`a white point value; determines a scale value based upon a
`ratio of the range value and a predetermined portion of a
`number of bins B in a histogram; determines a shift value
`based on the determined scale value; determines a center
`point C for the histogram to be the white point when the
`determined maximum input value from a previous frame is
`greater than or equal to the white point; allows a centerpoint
`for the histogram to move between the white point and a
`point equal to half the white point when the determined
`maximum input value from a previous frame is less than the
`white point; generates a histogram for incoming frame data;
`finds a white point bini', such that no more than 1% of the
`pixels are in bins above the white point bini': determines
`the desired white point W to be equal to i'--C-(B/2)+1; and
`adjusts a gain factor for the next frame based upon the
`determined desired white point W.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The present invention may take form in various compo
`nents and arrangements of components, and in various steps
`and arrangements of steps. The drawings are only for
`purposes of illustrating a preferred embodiment and are not
`to be construed as limiting the present invention, wherein:
`FIG. 1 is a block diagram showing a conventional image
`processing pipeline for processing digital image data from
`an imager;
`FIG. 2 is a block diagram showing an image processing
`pipeline for processing digital image data from an imager
`according to the concepts of the present invention;
`FIG. 3 illustrates a 3x5 smoothing filter window for
`substantially eliminating the effects of defective pixels
`within the imager, according to the concepts of the present
`invention;
`FIG. 4 graphically illustrates constraints for providing
`white balance compensation according to the concepts of the
`present invention;
`FIG. 5 is a flowchart showing one embodiment of the
`image processing pipeline for processing digital image data
`from an imager according to the concepts of the present
`invention;
`FIG. 6 is a flowchart showing another embodiment of the
`image processing pipeline for processing digital image data
`from an imager according to the concepts of the present
`invention;
`FIG. 7 is a histogram of illumination intensity levels of an
`imaged scene;
`FIG. 8 is a block diagram of the adaptive sensitivity, on
`a pixel-by-pixel basis, control system;
`FIG. 9 shows calculating the timing of the transition
`points for a transfer control function;
`
`25
`
`30
`
`35
`
`40
`
`45
`
`65
`
`APPLE v. RED.COM
`
`Page 13 of 27
`
`Apple Ex. 1013
`
`

`

`US 7,349,574 B1
`
`5
`FIG. 10 shows a desired map of output voltage versus
`illumination intensity level for a scene imaged to produce
`the transfer control function of FIG. 9;
`FIG. 11 illustrates a flowchart showing the operations of
`an illumination intensity mapping controller; and
`FIG. 12 illustrates a flowchart showing the operations of
`a transfer control function generator circuit.
`
`DETAILED DESCRIPTION OF THE PRESENT
`INVENTION
`
`6
`were produced by the imager 1, for defective pixels in the
`imager 1, and for directional noise.
`In a preferred embodiment of the present invention, the
`digital image data is initially filtered in the noise filtering
`subsystem 50 to remove column noise associated with the
`imager. The column noise filtering process of the present
`invention uses image data output by the imager during an
`analog to digital conversion test mode to remove imager
`specific noise from the raw sensor data.
`At the beginning of each imager idle cycle, an analog to
`digital conversion test is run. The imager idle cycle is
`initiated in response to certain events. Examples of Such
`events in a still imager are powering up the imager, taking
`a picture, inserting/removing a card, and/or selecting options
`on the imager. Moreover, examples of Such events in a video
`imager are powering up the imager and/or during blanking
`row periods (in between every frame of video data).
`With respect to the utilization of the concepts of the
`present invention in conjunction with a still imager, for each
`column, all of the pixels in that column are assigned to an
`analog-to-digital converter associated with that column.
`During normal operations, each of the pixels of a particular
`column is switched “ON” in sequence so as to connect to the
`analog-to-digital converter associated with that column.
`For example, during a 0.05 millisecond period of an
`image capture routine, a pixel of the column will be con
`nected to the analog-to-digital converter associated with that
`column. During the next 0.05 millisecond period of an
`image capture routine, the next pixel in that column will be
`connected to the analog-to-digital converter associated with
`that column, and so on.
`During the analog-to-digital converter test mode, all of
`the pixels are disconnected from the analog-to-digital con
`verter. After disconnection, a constant Voltage is applied to
`all of the analog-to-digital converters, and the outputs are
`used to calculate the relative analog-to-digital converter
`offsets. It is assumed that the analog-to-digital converters all
`have the same gain factor, but different offsets. The captured
`offsets define the column fixed pattern noise e.
`To avoid clipping, the present invention, according to one
`embodiment thereof, initially adds the maximum column
`fixed pattern noise max(e?) to each pixel of digital image
`data and then subtracts the column fixed pattern noise et,
`associated with the column of the pixel wherein i is the
`column number of the pixel. This column noise filtering
`process produces digital image data wherein imager-specific
`noise, the relative analog-to-digital converter offsets, has
`substantially been removed.
`In another embodiment of the present invention, to avoid
`clipping, the present invention initially adds the average
`column fixed pattern noise ave(e) to each pixel of digital
`image data and then Subtracts the column fixed pattern noise
`et associated with the column of the pixel whereini is the
`column number of the pixel. This column noise filtering
`process produces digital image data wherein imager-specific
`noise, the relative analog-to-digital converter offsets, has
`substantially been removed.
`On the other hand, with respect to the utilization of the
`concepts of the present invention in conjunction with a video
`imager, for each column, all of the pixels in that column are
`assigned to an analog-to-digital converter associated with
`that column. During normal operations, each of the pixels of
`a particular column is switched “ON” in sequence so as to
`connect to the analog-to-digital converter associated with
`that column.
`In the video embodiment, the present invention, according
`to one embodiment thereof, disconnects all the pixels from
`
`5
`
`10
`
`15
`
`25
`
`30
`
`35
`
`The present invention will be described in connection
`with preferred embodiments; however, it will be understood
`that there is no intent to limit the present invention to the
`embodiments described herein. On the contrary, the intent is
`to cover all alternatives, modifications, and equivalents as
`may be included within the spirit and scope of the present
`invention as defined by the appended claims.
`For a general understanding of the present invention,
`reference is made to the drawings. In the drawings, like
`reference have been used throughout to designate identical
`or equivalent elements. It is also noted that the various
`drawings illustrating the present invention are not drawn to
`scale and that certain regions have been purposely drawn
`disproportionately so that the features and concepts of the
`present invention could be properly illustrated.
`As noted above, it is desirable to provide an image
`processing pipeline that produces image data of a high
`quality and that Substantially removes any noise or artifacts
`introduced by the imager or other components along the
`pipeline. It is also desirable to provide an image-processing
`pipeline that produces image data of a high quality and has
`Substantially compensated the image data with respect to the
`lighting conditions used to image the scene or the sensor
`color responsivity. Moreover, it is desirable to provide an
`image-processing pipeline that produces image data of a
`high quality and has substantially limited the amount of
`stretching done to flat images.
`FIG. 2 illustrates a block diagram showing an image
`processing pipeline according to the concepts of the present
`invention. As illustrated in FIG. 2, an imager 1, comprising
`a plurality of photosensitive elements commonly referred to
`as pixels, images a scene and produces a plurality of
`Voltages. In operation, the pixel is first reset with a reset
`Voltage that places an electronic charge across the capaci
`tance associated with; e.g., a photodiode. Electronic charge
`produced by the photodiode, when exposed to illumination,
`causes charge of the diode capacitance to dissipate in
`proportion to the incident illumination intensity of the scene.
`At the end of an exposure period, the change in diode
`capacitance charge is detected and the photodiode is reset.
`The amount of light detected by the photodiode is computed
`as the difference between the reset voltage and the voltage
`corresponding to the final diode capacitance charge.
`The Voltages generated by the imager 1 are fed to an
`analog to digital converter 3. The analog to digital converter
`3 converts each analog Voltage to a digital image data word
`having digital value in a range from 0, which represents no
`illumination, to some maximum value that represents illu
`mination saturation. If the width of the digital image data
`word were set to 8-bits, the range of digital values that a
`digital image data word could realize would be in the range
`0 to 255, O being no illumination and 255 being saturation.
`The digital image data word is then filtered by a noise
`filtering subsystem 50. The noise filtering subsystem 50
`modifies the digital image data to compensate for any
`imager characteristic artifacts, such as column noise, that
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`APPLE v. RED.COM
`
`Page 14 of 27
`
`Apple Ex. 1013
`
`

`

`US 7,349,574 B1
`
`10
`
`15
`
`25
`
`30
`
`35
`
`7
`the analog-to-digital converter during the analog-to-digital
`converter test mode, the analog-to-digital converter test
`mode typically occurring during the blanking row periods
`between the every capture of a frame of image data. After
`disconnection, a constant Voltage is applied to all of the
`analog-to-digital converters, and the outputs are used to
`calculate the relative analog-to-digital converter offsets. The
`present invention uses the outputs by the imager in the
`analog-to-digital converter test mode to remove imager
`specific noise from the raw sensor data.
`In another embodiment, the present invention disconnects
`all the pixels from the analog-to-digital converter during the
`analog-to-digital converter test mode, the analog-to-digital
`converter test mode occurring during the blanking row
`periods between the every capture of a frame of image data.
`After disconnection, two fixed Voltages (a low Voltage and
`a high Voltage) are separately applied to all of the analog
`to-digital converters, and the outputs are used to calculate
`the analog-to-digital converters relative gains and relative
`offsets.
`It is noted that column noise, in the video embodiment,
`should be very constant for a particular image, so the present
`invention imposes a smoothing filter on the relative analog
`to-digital converter offsets to remove random noise over
`time. As the relative analog-to-digital converter offset data
`for each frame is captured, the relative analog-to-digital
`converter offset data is compared with the stored relative
`analog-to-digital converter offset data (initially Zero). If the
`new relative analog-to-digital converter offset data is larger
`than the stored value for a given column, the stored value is
`incremented by 0.5 prior to the start of the next frame.
`Conversely, if the new value is less than the stored value, the
`stored value is decremented by 0.5 before the next frame.
`The stored offset value for a particular column defines the
`column fixed pattern noise et for that column.
`As in the still embodiment, to avoid clipping in the video
`embodiment, the present invention, according to one
`embodiment thereof, initially adds the maximum column
`fixed pattern noise max(e?) to each pixel of digital image
`data and then subtracts the column fixed pattern noise et,
`40
`associated with the column of the pixel wherein i is the
`column number of the pixel. This column noise filtering
`process produces digital image data wherein imager-specific
`noise, the relative analog-to-digital converter offsets and/or
`gains, has substantially been removed.
`In another video embodiment of the present invention, to
`avoid clipping, the present invention initially adds the aver
`age column fixed pattern noise ave(e) to each pixel of
`digital image data and then Subtracts the column fixed
`pattern noise et associated with the column of the pixel
`wherein i is the column number of the pixel. This column
`noise filtering process produces digital image data wherein
`imager-specific noise, the relative analog-to-digital con
`verter offsets and/or gains, has substantially been removed.
`In a further embodiment, the present invention determines
`if the imager is in a still image mode or a video mode. If the
`imager is in a still image mode, the present invention adds
`the maximum column fixed pattern noise max(e?) to each
`pixel of digital image data and then Subtracts the column
`fixed pattern noise et associated with the column of the
`pixel wherein i is the column number of the pixel to avoid
`clipping. On the other hand, if the image is in a video mode,
`the present invention adds the average column fixed pattern
`noise ave(e) to each pixel of digital image data and then
`Subtracts the column fixed pattern noisee,
`associated with
`the column of the pixel wherein i is the column number of
`the pixel to avoid clipping.
`
`50
`
`45
`
`55
`
`60
`
`65
`
`8
`It is further noted that for each embodiment described
`above, dark rows of pixels or shielded fixed voltage rows
`can also be utilized as the source of column noise data to be
`used to calculate the relative analog-to-digital converter
`offsets and/or gains. More specifically, column noise data
`from dark rows are preferable for removing artifacts known
`as the “virtual row band,” “Tint band' or “ADC test mode
`band.
`After the column noise filtering process, the filtered
`digital image data may be processed by the noise filtering
`subsystem 50 so as to eliminate the effect of single pixel
`defects in the imager. Due to manufacture defects and
`eventual pixel failure, an imager may have numerous defec
`tive pixels that may be stuck at arbitrary Voltages, may float,
`and may be either linear or logarithmic in response depend
`ing on the location of the flaw or defect. These defective
`pixels can cause the linear Scaling process to waste valuable
`bits just to represent a few excessively bright pixels th