`
`Exhibit 6
`
`
`
`Case 2:20-cv-00234-JRG Document 1-6 Filed 07/13/20 Page 2 of 10 PageID #: 124
`
`(12) United States Patent
`Whiteside
`
`USOO6516147B2
`(10) Patent No.:
`US 6,516,147 B2
`(45) Date of Patent:
`Feb. 4, 2003
`
`(54) SCENE RECOGNITION METHOD AND
`SYSTEM USING BRIGHTNESS AND
`RANGING MAPPING
`(75) Inventor: George D. Whiteside, Lexington, MA
`(US)
`
`(73) Assignee: Polaroid Corporation, Waltham, MA
`(US)
`
`(*) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days
`a --
`(21) Appl. No.: 09/740,416
`(22) Filed:
`Dec. 19, 2000
`(65)
`Prior Publication Data
`US 2001/0031142 A1 Oct. 18, 2001
`• us/2
`Related U.S. Application Data
`(60) Provisional application No. 60/172,883, filed on Dec. 20,
`1999.
`(51) Int. Cl. ................................................ G03B 15/02
`(52) U.S. Cl. ......................... 396/61; 396/100; 396/106;
`396/159; 396/165; 396/187; 396/234
`
`
`
`(58) Field of Search .......................... 396/61, 165, 155,
`396/157, 159, 195, 233, 234, 65, 100, 106,
`121, 122, 123
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`... . . . . . . . . . . . . . 396/155
`4,304,476 A * 12/1981 Bloom
`6,167.200 A * 12/2000 Yamaguchi et al. .......... 396/65
`* cited by examiner
`Primary Examiner David M. Gray
`ASSistant Examiner Rochelle Blackman
`(57)
`ABSTRACT
`A method of controlling exposure of a Scene image com
`prising the following Steps: sensing a scene for image data;
`d
`tat
`f a bright
`f th
`erIVing values representauve OI a brignuneSS map oI ine
`Scene, Sensing the Scene for image data, deriving values
`representative of a range map; comparing the range map
`with the Scene brightness map; and controlling the exposure
`by controlling artificial illumination upon the Scene. In an
`alternate aspect, a System is provided for controlling expo
`Sure of a Scene image comprising the following elements: at
`least one Sensor assembly; a Source of artificial illumination;
`and a processing means
`p
`9.
`10 Claims, 4 Drawing Sheets
`
`
`
`Case 2:20-cv-00234-JRG Document 1-6 Filed 07/13/20 Page 3 of 10 PageID #: 125
`
`U.S. Patent
`
`Feb. 4, 2003
`
`Sheet 1 of 4
`
`US 6,516,147 B2
`
`
`
`2011 ( START
`203
`
`IRSTROBE
`
`205
`
`2O7
`
`209
`
`211
`
`213
`
`215
`
`AUTO FOCUS
`
`SET CCD TO
`HIGHRES.
`
`BRIGHTNESS MAP
`
`SET CCD TO
`LOWRES.
`
`R STROBE
`
`RANGING MAP
`
`217
`COMPARE BRIGHTNESS
`RANGING MAPS
`SET FLASH
`
`219
`
`221
`
`TAKE PICTURE
`
`223
`
`END
`
`FIG. 2
`
`
`
`Case 2:20-cv-00234-JRG Document 1-6 Filed 07/13/20 Page 4 of 10 PageID #: 126
`
`U.S. Patent
`
`Feb. 4, 2003
`
`Sheet 2 of 4
`
`US 6,516,147 B2
`
`301
`
`303
`
`
`
`)
`39.0
`
`R(639,479)
`
`R (639,211)
`
`FIG. 7B
`
`
`
`Case 2:20-cv-00234-JRG Document 1-6 Filed 07/13/20 Page 5 of 10 PageID #: 127
`
`U.S. Patent
`
`Feb. 4, 2003
`
`Sheet 3 of 4
`
`US 6,516,147 B2
`
`
`
`
`
`470-S
`
`420
`
`4301
`
`
`
`
`
`
`
`
`
`
`
`25
`A-f
`s 3.
`y S
`N
`W %
`2 | EN/St
`2 TR
`
`
`
`f3-A-
`
`
`
`2S4
`
`/
`
`
`
`Case 2:20-cv-00234-JRG Document 1-6 Filed 07/13/20 Page 6 of 10 PageID #: 128
`
`U.S. Patent
`
`Feb. 4, 2003
`
`Sheet 4 of 4
`
`US 6,516,147 B2
`
`
`
`
`
`Case 2:20-cv-00234-JRG Document 1-6 Filed 07/13/20 Page 7 of 10 PageID #: 129
`
`US 6,516,147 B2
`
`1
`SCENE RECOGNITION METHOD AND
`SYSTEM USING BRIGHTNESS AND
`RANGING MAPPING
`
`CROSS-REFERENCE TO RELATED
`APPLICATION
`The present application claims the benefit of copending
`U.S. provisional patent application, Ser. No. 60/172,883
`filed in the U.S. Patent and Trademark Office on Dec. 20,
`1999.
`
`BACKGROUND OF THE INVENTION
`The present invention pertains to an automated exposure
`control unit and method for image recording devices and is,
`particularly, adapted for use in conjunction with cameras
`employing Strobes for exposing photographic film of the
`Self-developing type.
`Capturing a properly exposed Scene on film is often
`difficult and a wide variety of techniques have been put forth
`in the prior art to achieve that end in an economical manner.
`Proper exposure is especially difficult when a photographer
`is confronted with a wide variety of Scene lighting and
`Subject conditions. Correct exposure is even more difficult
`when the exposure employs an electronic Strobe for illumi
`nating the Subject and Scene. The foregoing difficulties are
`compounded when exposing Self-developing film in low
`cost mass-market cameras, wherein conventional techniques
`of improving film picture quality done during film proceSS
`ing cannot be utilized and Sophisticated electronic Systems
`do not make economic Sense.
`Despite the problems associated with obtaining correct
`exposures, there are several known Successful techniques for
`dealing with the above issues by providing enhanced expo
`Sure for a variety of Scene lighting and Subject conditions.
`For instance, commonly-assigned, U.S. Pat. No. 4,192.587
`describes a proportional fill flash System for varying the time
`at which a Source of artificial illumination is energized
`relative to an exposure interval by determining the range of
`a Subject and to thereafter utilize the ranging information to
`vary the time in which the flash is fired so that it fires at an
`aperture corresponding to Subject range. Commonly
`assigned U.S. Pat. No. 4,255,030 describes a proportional
`fill flash System utilizing quenching of a quench Strobe at an
`appropriate time without determining Subject range based on
`the integration of both ambient and flash. Commonly
`assigned U.S. Pat. No. 4,285,584 describes a proportional
`fill flash System utilizing a photometer having three Zones in
`its overall field of view for distinguishing between
`background, central, and foreground portions of the Scene.
`The output of the different Segments of the Scene are
`compared to increase or decrease the exposure depending on
`whether the scene is more heavily illuminated in the back
`ground or foreground. Commonly assigned U.S. Pat. No.
`4,423,936 describes an exposure control unit utilizing a
`photometer having a multi-sensor array that detects both
`Subject range and ambient light intensity. A comparison of
`range measurements identifies that Scene area having the
`nearest object to the ambient light intensity measurements of
`Subject and non-Subject areas in order to classify Scene
`lighting conditions and then Select a corresponding program
`which controls the operation of the System to vary the ratio
`of ambient to artificial light contributions to exposure.
`Despite the Success of the above approaches in addressing
`the foregoing concerns particularly in regard to Self
`developing instant cameras, there is nevertheless a continu
`ing desire to improve upon the efficiency and costs of
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`obtaining proper exposure not only with cameras of the
`foregoing kind but also with digital cameras as well. In
`regard to digital cameras, it is highly desirable to effect
`highly quality printing utilizing low cost methodologies. For
`instance, in a camera with a digital capture mode the use of
`a multi-sensor array to minimize the problem adds to the
`overall cost to a System. Moreover, it is highly desirable to
`achieve the foregoing in a variety of Scene lighting condi
`tions wherein artificial illumination makes a contribution.
`
`SUMMARY OF THE INVENTION
`An object of the invention is to control the photographic
`exposure automatically Such that the Subject and non-Subject
`areas of a Scene are correctly exposed in an economical and
`efficient manner.
`It is another object of the invention to provide a digital
`capture System which can utilize the material already in the
`System.
`To achieve this and other objects, the invention is directed
`to a method of controlling exposure to a photosensitive
`element by controlling illumination of the Scene through a
`comparison of brightness and range mapping. The method
`comprises: (a) sensing image data from the Scene including
`Scene brightness from a first Set of a plurality of regions in
`the Scene; (b) forming a brightness map of the Scene in
`accordance with the brightneSS data corresponding to the
`first set of regions; (c) sensing range data from a second set
`of regions in the Scene; (d) forming a range map to deter
`mine a Subject in the Scene; and, (e) comparing the range
`map with the Scene brightness map for determining a
`relationship between Scene brightness and the Subject bright
`ness; and, (f) controlling the exposure by controlling artifi
`cial illumination upon the Scene, whereby a relationship of
`ambient and artificial illumination is generally maintained.
`The invention is further directed to a system for control
`ling exposure by controlling the relationship between ambi
`ent and artificial illumination during exposure. The System
`comprises a Sensor for Sensing image data including Scene
`brightness from a first Set of a plurality of regions in a Scene;
`a Sensor for Sensing range data from a Second Set of regions
`in the Scene; and, processing means for (i) defining a
`brightness map of the Scene in accordance with the bright
`ness data corresponding to the first set of regions, (ii)
`defining a range map of the Scene in accordance to the
`Second set of regions to determine a Subject in the Scene, (iii)
`comparing the range map with the Scene brightness map for
`determining a relationship between Scene brightness and the
`Subject range; and, (iv) controlling the exposure during an
`exposure interval by controlling artificial illumination upon
`the Scene, whereby a relationship of ambient and artificial
`illumination is generally maintained.
`AS a result of the foregoing System and method, when the
`ambient brightness readings of both the Scene background
`and Scene Subject are measured, control of the ambient
`exposure of the background is controlled by the ambient
`background readings and the exposure of the Subject is
`controlled by a comparing the Subject brightness to the Scene
`brightness and by controlling the amount of artificial illu
`mination directed at the Subject by the Source of artificial
`illumination to make up the difference in brightness values
`in order to give correct exposure for both the Scene back
`ground and the Subject. Accordingly, both the Scene Subject
`and Scene background are welleXposed.
`BRIEF DESCRIPTION OF THE DRAWINGS
`The invention will now be set forth in detail with refer
`ence to the drawings, in which:
`
`
`
`Case 2:20-cv-00234-JRG Document 1-6 Filed 07/13/20 Page 8 of 10 PageID #: 130
`
`US 6,516,147 B2
`
`3
`FIG. 1 shows a block diagram of a camera according to
`the present invention;
`FIG. 2 shows a block diagram of operation of the camera
`of FIG. 1;
`FIG. 3 shows a scene of which a photograph is to be
`taken;
`FIG. 4 shows a plurality of regions used in range map
`ping,
`FIG. 5 shows the scene of FIG. 3 Superimposed on the
`plurality of regions of FIG. 4;
`FIG. 6 shows a ranging map;
`FIG. 7A shows a macroscopic view of a plurality of
`regions used in brightness mapping,
`FIG. 7B shows an enlarged portion of the macroscopic
`view of FIG. 7A and shows discrete regions;
`FIG. 8 shows the scene in which the subject is illuminated
`more dimly than the background;
`FIG. 9 shows a scene in which the subject is illuminated
`more brightly than the background;
`FIG. 10 is an elevation view of another embodiment of an
`exposure control device with, however, portions removed
`for clarity in illustrating its construction and one mode of
`operation;
`FIG. 11 is a view similar to FIG. 10, but illustrating the
`device in yet another mode of operation; and,
`FIG. 12 is a cross-sectional view illustrating the system of
`FIG 10.
`
`15
`
`25
`
`4
`subject and the background. With this flash setting, the
`picture is taken in Step 221, and the operation ends in Step
`223.
`The operation of forming the ranging map in Step 215 will
`be explained with reference to FIGS. 3-6. FIG. 3 shows a
`view of CCD 105 in low resolution divided into sixteen
`exemplary regions R1-R16, although, as noted above, the
`number of regions capable of being formed by the CCD
`typically r ranges from ten to 500 in low resolution. FIG. 5
`shows the same scene as in FIG. 3 except that it is divided
`into the Sixteen regions corresponding to regions R1-R16 of
`FIG. 4 and thus shows how CCD 105 in low resolution
`divides scene 301 into the regions. In each of the regions
`R1-R16 of FIG. 5, the range is determined, and a near or far
`determination is made for each of the regions. As a result,
`the ranging map 601 of FIG. 6 is obtained. In this ranging
`map 601, regions R1-R5, R8, R9, R12, R13, and R16
`corresponding mostly to the background 305 of the scene
`301, are determined to be far regions, while regions R6, R7,
`F10, R11, R14 and R15, corresponding mostly to the subject
`303, are determined to be near regions. Of course, it is not
`necessary to use a binary near/far distinction; instead, vary
`ing distance ranges could be used.
`While the exemplary embodiment forms the ranging map
`with the CCD in low resolution, it forms the brightness map
`with the CCD in high resolution, which will be explained
`with reference to FIGS. 7A and 7B. In high resolution, each
`pixel discernible by the CCD can be a separate region;
`alternatively, a resolution intermediate between the low
`resolution and the maximum resolution of the CCD can be
`used. If the CCD is capable of VGA resolution (640 pixels
`across by 480 pixels down), the regions range from R(0,0)
`in the upper left corner to R(639.0) in the upper right corner
`to R(0.479) in the lower left corner to R(639,479) in the
`lower right corner, as indicated in FIG. 7A. The portion of
`the CCD framed in dotted lines in FIG. 7B is shown enlarged
`in FIG. 7B, which shows discrete regions ranging from
`R(631.200) to R(631.211) to R(639,200) to R(639.211).
`The operation of comparison in step 217 will now be
`explained with reference to FIGS. 6, 8 and 9. As explained
`above, FIG. 8 shows a situation in which the subject is
`illuminated more dimly than the background, while FIG. 9
`shows the opposite situation. The brightness levels in the
`brightness map obtained in step 209 are compared with
`ranging map 601. If the near regions are darker than the far
`regions, the situation in FIG. 8 is recognized, while if the
`opposite is true, the situation in FIG. 9 is recognized. The
`flash is Set accordingly to contribute to the exposure. For
`instance, in Some circumstances if the Subject is significantly
`frontlit, the flash need not be fired.
`FIGS. 10-12 illustrate portions of an exposure control
`unit 400. The exposure control unit 400 is similar that
`described in commonly assigned U.S. patent application,
`Ser. No. 09/133,661 filed Aug. 2, 1998. Hence, only those
`portions of the exposure control unit 400 which are neces
`sary to understand the present invention will be described
`Since a detailed description thereof is incorporated herein
`and made a part hereof. The unit 400 includes an aperture/
`lens disc 420 that is rotatably mounted in the housing
`assembly 402. The aperture/lens disc 420 is indexed under
`the control of a stepper motor 466 by means of a spur gear
`430 that meshes with a gear 468. A set of different sized
`apertures 422, 426 are selectively rotated into and out of
`coincidence with the CCD image sensor 442. Although not
`shown, the apertures 422, 426 can be provided with lenses
`of a Suitable variety Such as close-up, wide angle, or
`telephoto.
`
`40
`
`45
`
`50
`
`35
`
`DETAILED DESCRIPTION
`FIG. 1 shows a camera 101 according to the present
`invention. The camera 101 includes a quenchable electronic
`strobe unit or other suitable strobe unit 103, and a photo
`responsive area sensor 105, such as the CCD. A filter
`element 107, Such as a filter Sold under the model number
`CM 500, is disposed in front of the CCD 105 and is moved
`into one of two positions by motor 111 Such that light
`directed toward the CCD 105 is intercepted by either a
`visible-passing filter 109 V or an infrared-passing filter
`109IR. Alternatively, filter element 107 can be disposed in
`front of the strobe 103. An auto-focus device 113, Such as
`any auto-focus device known in the art moves a lens 115 into
`the appropriate focus position. The components are con
`nected to a microprocessor 117 or other suitable control
`device to control the operations of the camera.
`The camera of FIG. 1 operates as shown in the flow chart
`of FIG. 2. The operation starts in step 201. An IR wink or
`strobe pulse is emitted in step 203 by the strobe 103 (or
`another Suitable IR Source which may be separately
`provided) to allow an auto-focus operation in step 205 by the
`autofocus device 113. It will be appreciated that an auto
`focus operation need not be performed, but rather a set lens
`can be used that if desired could be manually set. The CCD
`55
`is Set to high resolution in Step 207 So that a brightness map
`of the Scene (Subject and background) to be photographed
`can be formed in step 209. In the exemplary embodiment,
`the visible-passing filter 109 V is located in front of the CCD
`105 so that the brightness map is formed with visible light.
`In step 211, the CCD is set to low resolution, and another IR
`Wink or Strobe pulse is emitted in Step 213 So that a ranging
`map of the scene can be formed in step 215. In the exem
`plary embodiment, the infrared-passing filter 1091R is
`located in front of the CCD 105. The brightness and ranging
`maps are compared in Step 217, and the flash is Set in Step
`219 to provide the appropriate level of backfill for the
`
`60
`
`65
`
`
`
`Case 2:20-cv-00234-JRG Document 1-6 Filed 07/13/20 Page 9 of 10 PageID #: 131
`
`US 6,516,147 B2
`
`S
`A filter assembly 460 is rotationally mounted on a shaft
`461 within the housing 402 before the lens assembly 410 and
`is biased by a torsion Spring 470 to rest against a Segment
`412 of the housing assembly 402, whereby a visible light
`pass filter element 462 is positioned to be in overlying
`relationship to an aperture 426. During pre-exposure to
`obtain the Scene brightness mapping, the image Sensor 442
`is enabled and the visible pass filter element 462 allows
`Visible light from the Scene to be passed to the Sensor,
`whereby Scene brightness measurements for each image
`Sensing region of the Sensor can be achieved. These image
`Sensing regions of the Sensor, of course, correspond to Scene
`portions that are to be Sensed for establishing the Scene
`brightness map. The Signals from the Sensor are forwarded
`to a System controller (not shown).
`To effect a range determining function while Still in the
`pre-exposure mode, the aperture/lens disc 420 is rotated in
`a counterclockwise direction, whereby a tab 423 on the disc
`drives the filter assembly 460 against the bias of the spring
`So that an infrared pass filter element 464 is placed in
`overlying relationship to the CCD image sensor 442, while
`the aperture 426 is now in overlying relationship to the
`image Sensor. In this step, the image Sensor 442 can be
`operated in a low resolution mode for determining a range
`map distinguishing Subject areas relative to the nonSubject
`areas The flash is operated to illuminate the Scene and the
`resulting reflected Scene brightness will pass through the IR
`pass filter 464 to the sensor 442; whereby range information
`for each Sensor region can be determined consistant with the
`Wink IR ranging technique described in commonly-assigned
`U.S. Pat. No. 4,785,322. Also, the present invention envi
`Sions the use of differently configured image acquisition
`modules with a variety of techniques for presenting an IR
`filter over Such a module, Such as by moving an IR filter in
`front of the acquisition module by means of a stepper motor
`or Solenoid.
`A System controller (not shown) includes a logic circuit
`board includes a micro-controller circuit that which receives
`electrical signals from the various camera elements and, in
`turn, controls operation of the Stepper motor and the CCD as
`well as Strobe and camera shutter mechanism logic circuit
`board. The logic circuit board includes a microprocessor that
`is operable for decoding Signals from, for instance, the
`Sensor for the Scene brightness and range determining Steps
`during different modes of operation. The logic circuit
`includes a conventional, electronically erasable memory
`Section which includes appropriate numbers of look-up
`tables, each of which employ combinations of the exposure
`parameters of Subject range, flash mode Selection, pre
`exposure Scene brightness information to define the desired
`memory cell address. The present invention contemplates
`establishing the Scene brightness and ranging maps, as well
`as comparing the maps to provide a relationship between the
`two that controls the strobe firing intervals for each and
`every combination of the Scene brightness and ranging
`maps. The logic circuit will control when the strobe will be
`fired and quenched during an exposure cycle So that a
`desired fill flash ratio between ambient and flash is main
`tained despite wide variances in Scene lighting and Subject
`ranges that exist.
`Although illustrative embodiments of the invention have
`been set forth, those skilled in the art will recognize that
`other embodiments can be realized within the scope of the
`invention. For example, the image Sensing CCD can be
`operated in only one resolution, So that the ranging and
`brightness information can be taken Simultaneously. In other
`words, a Single Set of data can be used to determine
`
`5
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`brightness and ranging. Also, the System and method
`described above do not have to be implemented in a camera,
`but find utility in any art in which correct illumination of
`Subjects under a variety of lighting conditions are an issue.
`What is claimed is:
`1. A method of controlling exposure of a Scene image
`comprising the Steps of;
`(a) sensing a Scene for image data including Scene bright
`neSS data from at least a first Set of a plurality of regions
`of the Scene including a Subject region;
`(b) deriving values representative of a brightness map of
`the Scene in accordance with Scene brightness data
`values corresponding to each of a first Set of regions,
`(c) sensing the Scene for image data including range data
`from at least a Second Set of regions in the Scene;
`(d) deriving values representative of a range map in
`accordance with range data values corresponding to
`each of the Second Set of regions and utilizing the
`values representative of a range map to determine a
`Subject in the Scene; and,
`(e) comparing the range map with the Scene brightness
`map for determining a relationship between Scene
`brightness and the Subject brightness, and,
`(f) controlling the exposure by controlling artificial illu
`mination upon the Scene, whereby a relationship of
`ambient and artificial illumination is generally obtained
`based on the relationship between Scene brightness and
`the Subject brightness.
`2. The method of claim 1 wherein said step of controlling
`the artificial illumination controls firing intervals of a Strobe
`for each and every combination of the Scene brightness and
`ranging maps.
`3. The method of claim 1 wherein one of the brightness
`Sensing or range Sensing is performed in a first Sensor
`resolution mode and the other of the range Sensing or
`brightness Sensing is performed in a Second Sensor resolu
`tion mode.
`4. The method of claim 1 wherein the brightness and
`range Sensing are performed in one resolution mode.
`5. The method of claim 1 wherein said range and bright
`neSS Sensing Steps are performed generally simultaneously.
`6. A System of controlling exposure of a Scene image, the
`System comprises:
`at least one Sensor assembly for Sensing image data
`including Scene brightness from a first Set of a plurality
`of regions in a Scene, Said Sensor assembly is operable
`for Sensing range data from a Second Set of generally
`independent regions in the Scene; a Source of artificial
`illumination; and,
`processing means for:
`(i) defining a brightness map of the Scene in accordance
`with the brightness data corresponding to each of the
`regions in the first Set of regions,
`(ii) defining a range map of the Scene in accordance to
`the Second set of regions to determine a Subject in the
`Scene,
`(iii) comparing the range map with the Scene brightness
`map for determining a relationship between Scene
`brightness and the Subject range; and,
`(iv) controlling the exposure by controlling a strobe
`artificial illumination upon the Scene, whereby a
`relationship of ambient and artificial illumination is
`generally obtained based on the relationship between
`Scene brightness and the Subject brightness.
`7. The system of claim 6 wherein said sensor senses one
`of the Scene brightness and the other of the Scene ranging in
`
`
`
`Case 2:20-cv-00234-JRG Document 1-6 Filed 07/13/20 Page 10 of 10 PageID #: 132
`
`US 6,516,147 B2
`
`7
`a first resolution mode and Said Sensor Senses the other of the
`Subject range and Scene brightness in a Second resolution
`mode.
`8. The system of claim 6 wherein said sensor is operable
`for Sensing the Scene brightness and ranging in a Single
`resolution mode.
`9. The system of claim 6 wherein said sensor assembly
`includes an infrared pass filter assembly that is operable in
`one condition to allow ambient and artificial illumination
`
`8
`from a Scene to impinge on Said Sensor, and in another
`condition allows infrared from the Scene to impinge on Said
`SCSO.
`10. The System of claim 6 wherein Said processing means
`5 is operable for
`controlling the artificial illumination by a controlling a
`Strobe for each and every combination of the Scene
`brightneSS and ranging maps.
`
`k
`
`k
`
`k
`
`k
`
`k
`
`