I IIIII IIIIIIII Ill lllll lllll lllll lllll lllll lllll lllll lllll 111111111111111111
`US009100604 B2
`
`c12) United States Patent
`Nakano et al.
`
`(IO) Patent No.:
`(45) Date of Patent:
`
`US 9,100,604 B2
`* Aug. 4, 2015
`
`(54) ELECTRIC CAMERA
`
`(71) Applicant: HITACHI MAXELL, LTD., Osaka (JP)
`
`(72)
`
`Inventors: Takahiro Nakano, Tokyo (JP); Ryuji
`Nishimura, Tokyo (JP); Toshiro
`Kinugasa, Tokyo (JP)
`
`(2013.01); H04N 5/23258 (2013.01); H04N
`5/23274 (2013.01); H04N9/045 (2013.01)
`(58) Field of Classification Search
`CPC . H04N 5/335; H04N 5/3741; H04N 5/23248;
`H04N 5/23258; H04N 5/23274; H04N 5/3454;
`H04N 9/045; H04N 3/155
`See application file for complete search history.
`
`(73) Assignee: HITACHI MAXELL, LTD., Osaka (JP)
`
`(56)
`
`References Cited
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by O days.
`
`This patent is subject to a terminal dis(cid:173)
`claimer.
`
`(21) Appl. No.: 14/264,243
`
`(22) Filed:
`
`Apr. 29, 2014
`
`(65)
`
`Prior Publication Data
`
`US 2014/0232915 Al
`
`Aug. 21, 2014
`
`Related U.S. Application Data
`
`(60) Continuation of application No. 13/681,495, filed on
`Nov. 20, 2012, now Pat. No. 8,736,729, which is a
`continuation of application No. 12/845,266, filed on
`Jul. 28, 2010, now Pat. No. 8,339,493, which is a
`
`(Continued)
`
`(30)
`
`Foreign Application Priority Data
`
`Jan. 11, 2000
`
`(JP) ................................. 2000-006064
`
`(51)
`
`Int. Cl.
`H04N5/374
`H04N5/232
`H04N9/04
`(52) U.S. Cl.
`CPC ......... H04N 513741 (2013.01); H04N 5/23248
`
`(2011.01)
`(2006.01)
`(2006.01)
`
`U.S. PATENT DOCUMENTS
`
`4,054,915 A
`5,170,249 A
`
`10/ 1977 Sugihara
`12/ 1992 Ohtsubo et al.
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`JP
`JP
`
`11/1992
`04-323973
`10/1997
`9-270959
`(Continued)
`
`Primary Examiner - Luong T Nguyen
`(74) Attorney, Agent, or Firm - Mattingly & Malur, PC
`
`ABSTRACT
`(57)
`An electric camera includes an image sensing device with a
`light receiving surface having N vertically arranged pixels
`and an arbitrary number of pixels arranged horizontally, N
`being equal to or more than three times the number of effec(cid:173)
`tive scanning lines M of a display screen of a television
`system, a driver to drive the image sensing device to vertically
`mix or cull signal charges accumulated in individual pixels of
`K pixels to produce, during a vertical effective scanning
`period of the television system, a number of lines of output
`signals which corresponds to 1/K the number of vertically
`arranged pixels N of the image sensing device, K being an
`integer equal to or less than an integral part of a quotient ofN
`divided by M, and a signal processing unit having a function
`of generating image signals by using the output signals of the
`image sensing device.
`
`5 Claims, 8 Drawing Sheets
`
`17
`
`IMAGE
`UNSTABILITY
`DECISION
`CIRCUIT
`
`16a
`
`GYRO
`SENSOR
`
`GYRO
`SENSOR
`
`16b
`
`4
`
`14
`
`15
`
`MODE
`SELECTOR
`SWITCH
`
`RECORD
`BUTTON
`
`11
`
`RECORDING
`UNIT
`
`10
`
`~I ,_,
`I DEVICE
`1\ 3
`
`LENS
`
`2 APERTURE
`
`SENSING
`
`GAIN
`ADJUST
`CIRCUIT
`
`SIGNAL
`PROCESSING
`CIRCUIT
`
`VERTICAL
`INTER·
`POLATION
`CIRCUIT
`
`HORIZONTAL
`INTER·
`POLATION
`CIRCUIT
`
`ENCODER
`CIRCUIT
`
`OUTPUT
`
`5
`
`6
`
`7
`
`B
`
`9
`
`12
`
`13
`
`BlackBerry Corporation Exhibit 1003, pg. 1
`
`

`

`US 9,100,604 B2
`Page 2
`
`Related U.S. Application Data
`
`continuation of application No. 10/660,710, filed on
`Sep. 12, 2003, now Pat. No. 8,059,177, which is a
`divisionofapplicationNo. 09/520,836, filed on Mar. 8,
`2000, now Pat. No. 6,765,616.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,187,569 A
`5,287,192 A
`5,402,173 A *
`5,438,365 A *
`5,638,132 A *
`5,734,424 A
`5,828,406 A
`5,847,758 A
`
`2/1993 Tani
`2/1994 Iizuka
`.............. 348/322
`3/1995 Noguchi et al.
`8/1995 Yamashita et al ............. 348/297
`6/1997 Hokari et al. ................. 348/556
`3/ 1998 Sasaki
`10/1998 Parulski et al.
`12/1998 Iizuka
`
`6,195,125 Bl
`6,519,000 Bl
`6,661,451 Bl
`6,765,616 Bl*
`6,781,634 Bl*
`6,798,448 Bl
`6,906,746 B2
`6,970,191 Bl
`7,154,539 Bl
`7,352,391 Bl*
`7,403,226 B2 *
`8,059,177 B2
`
`2/2001 Udagawa et al.
`2/2003 Udagawa
`12/2003 Kijima et al.
`7/2004 Nakano et al. ................ 348/322
`8/2004 Shiozaki et al.
`.............. 348/374
`9/2004 Motono et al.
`6/2005 Hijishiri et al.
`11/2005 Sase et al.
`12/2006 Nishimura et al.
`4/2008 Hatakeyama .............. 348/223.1
`7/2008 Nakano et al. ................ 348/294
`11/2011 Nakano et al.
`
`FOREIGN PATENT DOCUMENTS
`
`JP
`JP
`JP
`
`11-004456
`11-187306
`11-355665
`
`1/1999
`7 /1999
`12/1999
`
`* cited by examiner
`
`BlackBerry Corporation Exhibit 1003, pg. 2
`
`

`

`"""' = = O'I = ~ = N
`
`'"'..c
`
`d r.,;_
`
`13
`
`... ?
`f-.-1 D/A I----OUTPUT
`
`0 ....
`....
`.....
`rJJ =(cid:173)
`
`('D
`('D
`
`QO
`
`9
`. r·~
`CIRCUIT ?
`POLATION II CIRCUIT
`I LI ENCODER
`INTER-
`HORIZONTAL
`
`12
`
`'
`
`CIRCUIT
`POLATION ~
`INTER(cid:173)
`
`8
`?
`
`VERTICAL
`
`0 ....
`'" ...
`~
`~
`
`N
`
`Ul
`
`~ = ~
`
`~
`~
`~
`•
`00
`~
`
`FIG.1
`
`CONTROL H RECORDING 1.-. 1 O
`
`r -
`
`UNIT
`
`CIRCUIT
`
`t
`
`t
`
`11~t
`
`I
`
`BUTTON
`RECORD
`?
`15
`
`SELECTOR
`
`SWITCH
`
`MODE
`?
`14
`
`UNSTABILITY
`
`CIRCUIT
`DECISION
`
`~
`
`i-
`
`IMAGE
`?
`17
`
`7
`
`6
`
`5
`
`LENS 2 APERTURE
`
`3
`
`\
`
`1
`
`CIR;UIT
`
`PROCESSING i-...
`
`?
`
`AID
`
`CIRCUIT
`ADJUST
`
`GAIN H H SIGNAL
`'
`
`*
`
`SENSING
`
`I DEVICE
`(\ I IMAGE
`
`CIRCUIT
`DRIVE
`
`t +
`4
`
`'
`
`16b
`(
`
`SENSOR
`GYRO
`
`SENSOR
`GYRO
`2_
`16a
`
`BlackBerry Corporation Exhibit 1003, pg. 3
`
`

`

`U.S. Patent
`
`Aug. 4, 2015
`
`Sheet 2 of 8
`
`US 9,100,604 B2
`
`OUTPUT
`
`34
`
`V2 V3
`
`FIG.2
`
`33
`
`FIG.4
`
`~1 A FIELD (n+1 )TH LINE
`
`I
`I
`
`;~ B FIELD nTH LINE
`
`I
`I
`
`~' A FIELD nTH LINE
`
`BlackBerry Corporation Exhibit 1003, pg. 4
`
`

`

`"""' = = O'I = ~ = N
`
`... -.c
`d r.,;_
`
`~
`
`('D
`('D
`
`0 ....
`.....
`rJJ =(cid:173)
`
`QO
`
`0 ....
`N
`......
`~
`~
`
`Ul
`
`~ = ~
`
`~
`~
`~
`•
`00
`~
`
`VERTICAL BLANKING PERIOD
`
`VERTICAL EFFECTIVE SCANNING PERIOD
`
`VERTICAL BLANKING PERIOD
`
`VERTICAL SCANNING PERIOD
`
`__ jl ________________ _fl_f1JlJL ____________________ fL_Jl~--f1_
`
`--lf----------------~--------------------1r-~--~
`
`__ ll ________________ _fl_f1JlJL ____________________ Il_Jl~--f1_
`
`FIG.3
`
`---~
`
`LOW LEVEL
`
`V3 ( MIDDLE LEVEL fl_ Jl
`lf-lf
`
`V2 MIDDLE LEVEL
`
`[
`
`LOW LEVEL
`
`,--1-4--1-1
`
`-
`
`HIGH LEVEL
`
`LOW LEVEL
`
`T1
`
`T2
`
`V1 ( MIDDLE LEVEL IL Jl
`
`.J. ·1
`
`T6
`
`T4
`
`I~
`
`SCANNING PERIOD
`
`HORIZONTAL
`BLANKING PERIOD
`HORIZONTAL'.
`
`T5
`
`T3
`
`BlackBerry Corporation Exhibit 1003, pg. 5
`
`

`

`U.S. Patent
`
`Aug. 4, 2015
`
`Sheet 4 of 8
`
`US 9,100,604 B2
`
`FIG.5
`
`AREAB
`
`AREA A
`
`1 960 J
`
`I ~12ao~ I
`
`~1600~
`
`FIG.6
`
`-AREAA
`
`-AREAS
`
`- - A FIELD GRAVITY CENTER OF (n+1)TH LINE
`
`1
`I
`I
`I
`;..
`
`-------· B FIELD GRAVITY CENTER OF nTH LINE
`
`I
`I
`
`I ~' - - A FIELD GRAVITY CENTER OF nTH LINE
`
`BlackBerry Corporation Exhibit 1003, pg. 6
`
`

`

`13
`
`12
`)
`
`9
`)
`
`8
`)
`
`7
`)
`
`6
`)
`
`OUTPUT
`
`~ D/A
`
`CIRCUIT
`ENCODER
`
`-•
`
`HORIZONTAL
`
`CIRCUIT
`POLATION
`
`INTER-
`
`....
`
`CIRCUIT
`POLATION
`
`VERTICAL
`
`INTER-
`
`CIRCUIT
`
`SIGNAL
`,
`
`PROCESSING ~
`
`~ = ~
`
`~
`~
`~
`•
`00
`~
`
`FIG.7
`
`---10
`
`RECORDING
`
`CONTROL
`
`t,
`
`11 """'\. '
`
`CIRCUIT -UNIT
`
`I
`
`I
`
`BUTTON
`RECORD
`
`SWITCH
`
`SELECTOR
`
`MODE
`
`SWITCH
`CHANGE
`
`I
`
`VIEW ANGLE
`
`15
`
`14
`
`18
`
`"""' = = O'I = ~ = N
`
`'"'..c
`
`d r.,;_
`
`0 ....
`Ul
`.....
`rJJ =(cid:173)
`
`('D
`('D
`
`QO
`
`0 ....
`'" ...
`~
`~
`
`N
`
`Ul
`
`SENSING --. ADJUST -AID -
`
`CIRCUIT
`DRIVE
`
`) +
`4
`
`.
`
`UNSTABILITY
`
`CIRCUIT
`DECISION
`
`IMAGE
`
`I/
`
`i
`
`1 6b
`(
`
`....
`
`....
`
`SENSOR
`GYRO
`
`SENSOR
`GYRO
`I
`
`10a
`
`5
`)
`
`CIRCUIT
`
`GAIN
`
`LENS 2 APERTURE
`
`3
`
`\
`
`1
`
`)
`
`IMAGE
`
`JI DEVICE
`n I +
`
`BlackBerry Corporation Exhibit 1003, pg. 7
`
`

`

`U.S. Patent
`
`Aug. 4, 2015
`
`Sheet 6 of 8
`
`US 9,100,604 B2
`
`FIG.8
`
`- - - A FIELD GRAVITY CENTER OF (n+ 1 )TH LINE
`
`-,
`l
`.. --------- B FIELD GRAVITY CENTER OF nTH LINE
`r
`.
`_)
`A FIELD GRAVITY CENTER OF nTH LINE
`
`FIG.9
`
`AREA A
`
`AREAB
`
`AREA A : 960HIGH x 1280WIDE
`
`r 1200 D
`l~
`
`AREA 8 : 720HIGH x 960WIDE
`
`AREA C : 480HIGH x 640WIDE
`
`-1-- -1600 - -.. 1
`
`BlackBerry Corporation Exhibit 1003, pg. 8
`
`

`

`U.S. Patent
`
`Aug. 4, 2015
`
`Sheet 7 of 8
`
`US 9,100,604 B2
`
`FIG.1 O
`
`-
`
`!Yr
`
`IY[
`
`IYC
`-
`
`. . . . . .
`
`IY[
`
`IYr
`
`IYI
`
`-
`IYC
`
`IY[
`
`IY[
`. -
`.
`.
`. .........
`.
`.
`.
`IY[
`
`LYJ 1 - -
`IYC
`
`)
`33
`
`.--
`iGc
`
`IGC
`
`IGC
`. 1 - -
`.
`.
`.
`.
`.
`
`IGC
`
`IG[
`
`~ [
`
`FIG.12
`
`OUT(cid:173)
`PUT
`
`34
`
`32
`V1 V2 V3 V4 VS V
`6
`
`,__
`
`30
`
`\J1
`
`IC[
`
`IC-[
`
`[g ,__
`
`. -. . . . .
`
`ICC
`
`IC-[
`fee
`
`,___
`
`I
`
`- - - - nTH LINE
`
`- - - - (n+ 1 )TH LINE
`
`- - - - (n+2)TH LINE
`
`- - - - n'TH LINE
`- - - - (n'+ 1 )TH LINE
`- - - - (n'+2)TH LINE
`- - - - (n'+3)TH LINE
`- - - - (n'+4)TH LINE
`
`(BEFORE INTERPOLATION)
`
`(AFTER INTERPOLATION)
`
`FIG.13A
`
`FIG.138
`
`Ye w Cy
`Ye w Cy
`Ye w Cy
`Ye w Cy
`Ye w Cy
`
`R G B
`
`R G B
`R G 8
`R G B
`R G B
`
`BlackBerry Corporation Exhibit 1003, pg. 9
`
`

`

`"""' = = O'I = ~ = N
`
`'"'..c
`
`d r.,;_
`
`0 ....
`QO
`.....
`rJJ =(cid:173)
`
`('D
`('D
`
`QO
`
`0 ....
`'" ...
`~
`~
`
`N
`
`Ul
`
`~ = ~
`
`~
`~
`~
`•
`00
`~
`
`~
`
`VERTICAL BLANKING PERIOD
`
`VERTICAL SCANNING PERIOD
`
`VERTICAL EFFECTIVE SCANNING PERIOD
`
`VERTICAL BLANKING PERIOD
`
`~
`
`---ir-----------------~----------------------ir---1.f-~----ir
`---ir-----------------~---------------------~----ir--~----ir
`
`---u-------------------~----------------------ir----ir--~---~
`-u-----------------~--------------------~-~-~-~
`
`FIG.11
`
`T1
`
`T2
`
`n ___ n __
`n __ _r1_
`ir----r
`J--u--(cid:173)
`ir--u--
`ir--u--(cid:173)
`
`LOWLEVEL
`
`( MIDDLE LEVEL
`
`V6
`
`LOW LEVEL
`
`V5 MIDDLE LEVEL
`
`[
`
`HIGHLEVEL
`LOW LEVEL
`
`MIDDLE LEVEL
`LOW LEVEL
`
`V4
`(
`
`V3 MIDDLE LEVEL
`
`[
`
`HIGHLEVEL
`LOW LEVEL
`
`MIDDLE LEVEL
`LOW LEVEL
`
`MIDDLE LEVEL
`
`HIGHLEVEL
`
`V2
`(
`
`[
`
`V1
`
`___ rl_ ________________ ____I1____ _____________________ }-i ___ rl_ _ _____Il_fl. ___ f1
`___ f1 ________________ ____1l____ _____________________ s-l ___ f1_~ ___ f1
`
`-1 .. 1
`
`T6
`
`T4
`
`I~
`
`SCANNING PERIOD
`
`HORIZONTAL
`BLANKING PERIOD
`HORIZONTAL
`
`T3
`
`BlackBerry Corporation Exhibit 1003, pg. 10
`
`

`

`US 9,100,604 B2
`
`1
`ELECTRIC CAMERA
`
`CROSS REFERENCE TO RELATED
`APPLICATION
`
`This application is a continuation of U.S. application Ser.
`No. 13/681,495, filed Nov. 20, 2012, which is a continuation
`of U.S. application Ser. No. 12/845,266, filed Jul. 28, 2010,
`now U.S. Pat. No. 8,339,493, issued Dec. 25, 2012, which is
`a continuation of U.S. application Ser. No. 10/660,710, filed
`Sep. 12, 2003, now U.S. Pat. No. 8,059,177, issued Nov. 15,
`2011, and is related to U.S. application Ser. No. 10/660, 711,
`filed Sep. 12, 2003, now U.S. Pat. No. 7,403,226, issued Jul.
`22, 2008, both of which are divisional applications of U.S.
`application Ser. No. 09/520,836, filed Mar. 8, 2000, now U.S.
`Pat. No. 6,765,616, issued Jul. 20, 2004, the subject matter of
`all the above is incorporated by reference herein.
`
`BACKGROUND OF THE INVENTION
`
`The present invention relates to a photography related to
`video cameras, camcorders, digital still cameras and others
`using a solid-state image sensing device, and more particu(cid:173)
`larly to an electric camera using a solid-state image sensing
`device with a large number of pixels.
`Electric cameras using solid-state image sensors such as
`CCDs (charge-coupled devices) include a so-called video
`camera or camcorder for taking moving images and a so(cid:173)
`called digital still camera for taking still images. In recent
`years, video cameras with a still image taking function and
`digital still cameras with a moving image taking function
`have become available.
`In a video camera to photograph moving images, it is
`generally assumed that the video is viewed on a display such
`as television monitor and thus the camera is designed to
`produce output signals conforming to a television system
`such as NTSC and PAL. Therefore, the effective number of
`vertically arranged pixels or picture elements on the image
`sensing device used in such a camera is determined to enable
`television signals to be generated. The NTSC system, for
`example, performs interlaced scanning on two fields, each of
`which has an effective scanning line number of about 240
`lines (the number of scanning lines actually displayed on the
`monitor which is equal to the number of scanning lines in the
`vertical blanking period subtracted from the total number of
`scanning lines in each field). To realize this, the image sensing
`device has about 480 pixel rows as the standard effective
`number of vertically arranged pixels. That is, the signals of
`two vertically adjoining pixels in each field are mixed
`together inside or outside the image sensing device to gener(cid:173)
`ate about 240 scanning lines, and the combinations of pixels
`to be cyclically mixed together are changed from one field to
`another to achieve the interlaced scanning.
`Some image sensing devices to take moving images
`according to the NTSC system have an area of pixels for
`image stabilization added to the area of effective pixel area,
`thus bringing the effective number of vertically arranged
`pixels to about 480 or more. In this case, an area beyond 480th
`pixels is read out at high speed during the vertical blanking
`period and therefore the signals thus read out are not used as 60
`effective signals. Therefore, the video signals can only be
`generated from those signals coming from the area of about
`480 vertically arranged pixels. When such a camera is used to
`photograph a still image, it is relatively easy to generate a
`static image signal conforming to, for example, JPEG (Joint 65
`Photographic Expert Group) from the signals coming from
`the same pixel area that is used to take a moving image. A
`
`2
`problem remains, however, that the number of vertically
`arranged pixels obtained is limited to around 480, making it
`impossible to produce more detailed static image signals.
`In a camera having an image sensing device with the area of
`5 pixels for image stabilization mentioned above, a method of
`alleviating this problem may involve using the entire area of
`effective pixels including the area of image stabilization pix(cid:173)
`els in photographing a still image. Even when photographing
`a still image, however, the photographed image needs to be
`10 monitored for check and, for that purpose, it is necessary to
`generate signals conforming to the television system from
`signals read out from all effective pixels.
`An example of such a conventional camera has been pro(cid:173)
`posed in JP-A-11-187306. In the camera disclosed in this
`15 publication, signals from all the effective pixels are read out
`taking two or more times the field period of the television
`system, stored in a memory means such as a field memory,
`and then subjected to interpolation processing for transfor(cid:173)
`mation into signals conforming to the field cycle and hori-
`20 zontal scan cycle of television.
`This conventional camera, however, requires a large pro(cid:173)
`cessing circuit, such as field memory, for signal conversion.
`Another drawback is that the image sensing device readout
`cycle is a plurality of times the field cycle, degrading the
`25 dynamic resolution. Even with the use of this circuit, the
`number of pixels obtained as the static image signals is lim(cid:173)
`ited to the number of effective pixels used for moving videos
`plus the area of image stabilization pixels.
`In a digital still camera designed for taking still images,
`30 there has been a trend in recent years toward an increasing
`number of pixels used on the moving video image sensing
`device in order to obtain higher resolution static image sig(cid:173)
`nals. When taking a moving image or monitoring the video, it
`is necessary to generate signals that conform to the television
`35 system. The number of pixels on such an image sensing
`device, however, does not necessarily match the number of
`scanning lines of the television system and therefore some
`form of conversion means is required.
`The conversion means may involve, as in the video camera
`40 with the area of image stabilization pixels, reading out signals
`from the image sensing device taking a longer time than the
`field cycle and interpolating them to generate television sig(cid:173)
`nals. This method has, in addition to the problem described
`above, a drawback that the readout cycle increases as the
`45 number of pixels increases, further degrading the dynamic
`resolution.
`To mitigate this problem, JP-A-9-270959 discloses an
`apparatus which mixes together or culls the pixel signals
`inside the image sensing device to reduce the number of
`50 signals to be read and therefore the read cycle. Although this
`apparatus alleviates the problem of the degraded dynamic
`resolution, it requires a large processing circuit such as field
`memory to perform time-axis transformation to generate sig(cid:173)
`nals conforming to the television system and the image sens-
`55 ing device itself needs to have a special structure for perform(cid:173)
`ing desired mixing and culling.
`
`SUMMARY OF THE INVENTION
`
`The present invention relates to a photography of video
`cameras, camcorders, digital still cameras and others using a
`solid-state image sensing device, and more particularly to an
`electric camera using a solid-state image sensing device with
`a large number of pixels.
`The conventional electric cameras, as described above,
`have drawbacks that when taking a still picture by using a
`video camera, the number of pixels is not sufficient and that
`
`BlackBerry Corporation Exhibit 1003, pg. 11
`
`

`

`US 9,100,604 B2
`
`4
`FIG.10 is a schematic diagram showing the structure of an
`image sensing device in a third embodiment of the electric
`camera according to the present invention.
`FIG. 11 is a drive pulse timing diagram in the third embodi-
`5 ment of the electric camera of the invention.
`FIG. 12 is a schematic diagram showing an interpolation
`operation in the third embodiment of the electric camera of
`the invention.
`FIGS. 13A and 13B are schematic diagrams showing the
`10 arrangement of color filters in the image sensing device in a
`fourth embodiment of the electric camera according to the
`present invention.
`
`DESCRIPTION OF THE EMBODIMENTS
`
`3
`when taking a moving image with a still camera, the associ(cid:173)
`ated circuit inevitably increases and the dynamic image qual-
`ity deteriorates. Taking both moving and static images of
`satisfactory quality with a single camera is difficult to
`achieve. In addition to solving the above problems, to obtain
`good dynamic picture quality by using an image sensing
`device having a large number of pixels intended for still
`images requires extracting a pixel area that is used to realize
`an image stabilizing function. The conventional art and cam(cid:173)
`eras do not offer a means to accomplish this function.
`An object of the present invention is to provide an electric
`camera that solves these problems and which uses an image
`sensing device with a sufficient number of pixels for still
`images and enables taking of highly detailed still images and 15
`a moving video taking with reduced image quality degrada(cid:173)
`tion without increasing circuitry such as field memory. It is
`also an object of the present invention to provide an electric
`camera that can also realize the image stabilizing function.
`According to one aspect of this invention, the electric cam- 20
`era to realize the above objectives has: an image sensing
`device with a light receiving surface having N vertically
`arranged pixels and an arbitrary number of pixels arranged
`horizontally, N being equal to or more than three times the
`number of effective scanning lines M of a display screen of a 25
`television system; a driver to drive the image sensing device
`to vertically mix or cull signal charges accumulated in indi(cid:173)
`vidual pixels of every K pixels to produce a number oflines of
`output signals which corresponds to the number of effective
`scanning lines M, K being at least one of integers equal to or 30
`less than an integral part of a quotient of N divided by M ( a
`number of lines of output signals corresponds to 1/K the
`number of vertically arranged pixels N of the image sensing
`device); and a signal processing unit to generate image sig- 35
`nals by using the output signals of the image sensing device.
`As explained above, since this invention eliminates the
`limit on the number of vertically arranged pixels, an electric
`camera can be provided which enables taking of highly
`detailed still images and a satisfactory moving video taking 40
`by using an image sensing device with a large enough pixel
`number even for still images.
`
`Now embodiments of the present invention will be
`described by referring to the accompanying drawings. FIG. 1
`is a block diagram showing the configuration of one embodi(cid:173)
`ment of an electric camera according to the invention.
`In FIG. 1, reference number 1 represents a lens, 2 an
`aperture, 3 an image sensing device, 4 a drive circuit, 5 a gain
`adjust circuit, 6 an analog-digital (AID) conversion circuit, 7
`a signal processing circuit, 8 a vertical interpolation circuit to
`perform interpolation in a vertical direction, 9 a horizontal
`interpolation circuit to perform interpolation in a horizontal
`direction, 10 a recording unit including recording media such
`as magnetic tape, semiconductor memory and optical disk to
`record a video signal, 11 a control circuit to control these
`constitutional elements according to the operating state, 12 an
`encoder circuit to modulate the video signal into a standard
`television signal, 13 a digital-analog (D/A) conversion cir(cid:173)
`cuit, 14 a mode selector switch to change over the operation
`mode between the moving video taking and the still image
`taking, 15 a record button to start or stop the recording, 16a
`and 16b gyro sensors to detect vertical image-unstability and
`lateral image-unstability, respectively, and 17 an image-un(cid:173)
`stability decision circuit to determine the image-instability
`from signals output from the gyro sensors.
`In the above configuration, light coming from the lens 1
`through the aperture 2 is focused on a light receiving surface
`of the image sensing device 3 where it is converted into an
`electric signal. In this embodiment the image sensing device
`45 3 is of a CCD type. FIG. 2 shows the structure of this image
`sensing device 3. In FIG. 2, denoted 30 are pixels each formed
`of a photodiode, which are arranged horizontally and verti(cid:173)
`cally in a grid pattern. On these grid-arrayed pixels three types
`of color filters that pass yellow (Ye), green (G) and cyan (Cy),
`50 respectively, are arranged in such a way that the combination
`of these three colors is repeated horizontally every three pix(cid:173)
`els and that the filters of the same colors are lined vertically in
`so-called vertical stripes. Although an arbitrary number of
`pixels may be used, this embodiment has an array of 1200
`55 pixels vertically and 1600 pixels horizontally. A vertical
`transferunit 32 is a CCD which is driven by three phase pulses
`Vl, V2, V3. This CCD has a three-gate structure in which
`each pixel corresponds to three phase pulses and thus can
`vertically transfer a signal charge of each pixel independently.
`60 Transfer gates 31 for transferring the charge of each pixel to
`the vertical transfer unit 3 2 are commonly connected to a gate
`of the vertical transfer unit 32 that corresponds to the V2
`pulse. An operation to transfer the charge from each pixel to
`the vertical transfer unit 32 in response to a peak value of the
`65 pulse applied to the commonly connected gate and an opera(cid:173)
`tion to transfer the charge vertically are performed separately.
`A horizontal transfer unit 33 horizontally transfers the
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a block diagram showing the configuration of a
`first embodiment of an electric camera according to the
`present invention.
`FIG. 2 is a schematic diagram showing the structure of an
`image sensing device in the first embodiment of the electric
`camera of the invention.
`FIG. 3 is a drive pulse timing diagram in the first embodi(cid:173)
`ment of the electric camera of the invention.
`FIG. 4 is a schematic diagram showing a mixing operation
`in the first embodiment of the electric camera of the invention.
`FIG. 5 is a schematic diagram showing a readout area in the
`first embodiment of the electric camera of the invention.
`FIG. 6 is a schematic diagram showing a mixing operation
`in the first embodiment of the electric camera of the invention.
`FIG. 7 is a block diagram showing the configuration of a
`second embodiment of an electric camera according to the
`present invention.
`FIG. 8 is a schematic diagram showing a mixing operation
`in the second embodiment of the electric camera of the inven(cid:173)
`tion.
`FIG. 9 is a schematic diagram showing a readout area in the
`second embodiment of the electric camera of the invention.
`
`BlackBerry Corporation Exhibit 1003, pg. 12
`
`

`

`US 9,100,604 B2
`
`5
`
`5
`charges supplied from the vertical transfer units 32 and out(cid:173)
`puts them successively through an output amplifier 34 from
`the output terminal.
`Referring back to FIG. 1, the operation performed when
`the moving video mode is selected by the mode selector
`switch 14 will be explained. The number of vertically
`arranged pixels on the image sensing device in this embodi(cid:173)
`ment is 1200, so if the number of effective scanning lines in
`the field of the NTSC system is assumed to be 240 lines, then
`vertically mixing five pixels ( = 1200 pixel rows/240 scanning
`lines) can match the number of lines of output signals from
`the image sensing device to the number of effective scanning
`lines.
`However, in this embodiment, to realize the image stabi(cid:173)
`lizing function described later, four vertically arranged pixels 15
`are mixed together during motion image taking mode. When
`four vertically arranged pixels are to be cyclically mixed
`together, the signals from the area of 960 pixels (=240 scan(cid:173)
`ning linesx4 pixels) out of the 1200 vertically arranged pixels
`are used as effective signals and the remaining 240 pixels
`(=1200 (all pixels)-960 (effective pixels)) are not used for
`image forming. FIG. 3 shows the timing of a vertical drive
`pulse for the image sensing device in this operation mode,
`with Vl, V2 and V3 representing three phase drive pulses
`applied to each gate of the CCD or vertical transfer unit 32.
`In FIG. 3, in a period Tl included in the vertical blanking
`period, the drive pulse V2 is held high to transfer the signal
`charge accumulated in each pixel to under the V2 gate of the
`vertical CCD. Next, in a period T2, while the V2 pulse is still
`at middle level, the V3 pulse is raised from low level to middle
`level; next, while the V3 pulse is at middle level, the V2 pulse
`is changed from middle level to low level, after which the Vl
`pulse is changed from low level to middle level; next, while
`the Vl pulse is at middle level, the V3 pulse is changed from
`middle level to low level, after which the V2 pulse is changed 35
`to middle level and finally the Vl pulse is changed from
`middle level to low level. With this sequence of pulse opera(cid:173)
`tions, the signal charges under the V2 gate for one pixel row
`are transferred and held again under the V2 gate.
`By repeating this series of operations, the signal charges 40
`for a desired number of pixel rows can be transferred. In FIG.
`3, during a period T3 included in the vertical blanking period
`before the vertical effective scanning period (the vertical
`scanning period minus the vertical blanking period which
`corresponds to the actually displayed image) and during a 45
`period T4 included in the vertical blanking period after the
`vertical effective scanning period, the above transfer opera(cid:173)
`tion for one pixel row is repeated a total of 240 times to
`transfer the signal charges of the 240 pixel rows not used for
`image generation to the horizontal transfer unit 33 during the 50
`vertical blanking period. For example, if this transfer opera(cid:173)
`tion is performed 120 times during the period T3 and 120
`times during the period T 4, the signal charges from upper 120
`pixel rows and lower 120 pixel rows on the light receiving
`surface are transferred to the horizontal transfer unit 33 dur- 55
`ing the period T3 and period T4 within the vertical blanking
`period. During each of the subsequent periods TS and T6 in
`the vertical blanking period, the horizontal transfer unit 33 is
`driven for a predetermined period to output the charges trans(cid:173)
`ferred to the horizontal transfer unit 33 from the output ter- 60
`minal. These charges are not used as valid signals as they are
`output during the vertical blanking period.
`Next, in the vertical effective scanning period of FIG. 3, the
`above one-pixel-row transfer operation is performed four
`times during each horizontal blanking period to transfer the 65
`signal charges of four pixel rows to the horizontal transfer unit
`33 where they are mixed together. Then, during a horizontal
`
`6
`effective scanning period (the horizontal scanning period
`minus the horizontal blanking period which corresponds to
`the actually displayed image), the horizontal transfer unit 33
`is driven to read out the signal charges from the horizontal
`transfer unit to produce an output signal conforming to the
`television system. If the above operation is performed on the
`A field and if, on the B field, the number of pixel rows
`transferred during the period T3 is set to 122 rows and that
`during the period T 4 is set to 118 rows, then the combination
`10 of four pixel rows to be cyclically mixed together shifts by
`two rows between the two fields, thus allowing the interlaced
`scanning to be performed as shown in FIG. 4. (FIG. 4 shows
`the light receiving surface of the image sensing device and its
`relation to the displayed screen is vertically inverted.)
`Let us return to FIG. 1. The output signal from the image
`sensing device 3 is adjusted in gain by the gain adjust circuit
`5 and then converted by the AID conversion circuit 6 into a
`digital signal. The digital signal is then processed by the
`signal processing circuit 7 that performs color signal process-
`20 ing and luminance signal processing, such as generation of
`color signals, gamma correction, white balance processing
`and outline enhancement. The image sensing device in this
`embodiment has an array of vertical stripes of yellow (Ye),
`green (G) and cyan (Cy) color filters, so the color signals for
`25 Ye, Gand
`Cy are obtained as a series of color points from one line of
`output signals at all times no matter how many pixels are
`vertically combined. From these color signals three primary
`color signals R, G, B can be obtained from the following
`30 calculations.
`
`G~G
`
`The R, G and B signals undergoes the white balance pro(cid:173)
`cessing and gamma correction processing in the signal pro(cid:173)
`cessing circuit 7 and are then converted into color difference
`signals such as R-Y, B-Y or U and V. The luminance signals
`and the color difference signals are then entered through the
`vertical interpolation circuit 8 into the horizontal interpola(cid:173)
`tion circuit 9. In this operation state the signals just pass
`through the vertical interpolation circuit 8 without being pro(cid:173)
`cessed. The horizontal interpolation circuit 9 performs inter(cid:173)
`polation on the signals in the horizontal direction.
`FIG. 5 shows the light receiving surface of the image
`sensing device. As described above, in the operating state of
`this embodiment, the signals read out during the vertical
`effective scanning period correspond to an area having 960 of
`the 1200 vertically arranged pixels and a horizontal width of
`1600 pixels, as shown shaded at A in FIG. 5. If the entire light
`receiving surface of the image sensing device has a 4-to-3
`(width to height) aspect ratio, the shaded area A is more
`laterally elongate than this aspect ratio. Hence, if the signals
`of all horizontal pixels of the light receiving surface are dis(cid:173)
`played, for example, on an NTSC standard television monitor
`with the 4-to-3 aspect ratio, the image displayed is com(cid:173)
`pressed horizontally and looks vertically elongate, compared
`with the original image. It is therefore necessary to output
`during the horizontal effective scanning period only those
`signals coming from a pixel area with the horizontal width
`conforming to the aspect ratio of the television system, as
`shown by a shaded area B. When the television system has an
`4-to-3 as