I lllll llllllll II llllll lllll lllll lllll lllll lllll lllll lllll 111111111111111111111111111111111
`US 20040126019Al
`
`(19) United States
`(12) Patent Application Publication
`Ikebe et al.
`
`(10) Pub. No.: US 2004/0126019 Al
`Jul. 1, 2004
`( 43) Pub. Date:
`
`(54)
`
`IMAGING DEVICE, IMAGE PROCESSING
`METHOD, AND RECORDING MEDIUM
`
`(22) Filed:
`
`Sep.30,2003
`
`(76)
`
`Inventors: Keiichi lkebe, Kanagawa (JP); Takao
`Inoue, Kanagawa (JP); Akira
`Takahashi, Kanagawa (JP); Takashi
`Maki, Kanagawa (JP); Taku Kodama,
`Kanagawa (JP); Ikuko Yamashiro,
`Kanagawa (JP); Takanori Yano,
`Kanagawa (JP); Takeshi Koyama,
`Tokyo (JP); Shin Aoki, Kanagawa (JP);
`Hiroyuki Sakuyama, Tokyo (JP)
`
`Correspondence Address:
`BLAKELY SOKOLOFF TAYLOR & ZAFMAN
`12400 WILSHIRE BOULEVARD, SEVENTH
`FLOOR
`LOS ANGELES, CA 90025 (US)
`
`(21)
`
`Appl. No.:
`
`10/676,535
`
`(30)
`
`Foreign Application Priority Data
`
`Sep. 30, 2002
`
`(JP) ...................................... 2002-286044
`
`Publication Classification
`
`(51)
`Int. CI.7 ............................... G06K 9/36; G06K 9/46
`(52) U.S. Cl. ......................................... 382/232; 348/231.2
`
`(57)
`
`ABSTRACT
`
`An image processing method includes obtaining raw image
`data of an object by generating the object by an imager that
`generates an image of the object with color decomposition,
`decomposing the raw image data into a plurality of color
`planes such that each of the color planes comprises pixel
`information of a color pertinent to the color plane, and
`compressing data of each color plane.
`
`START
`
`SET COMPRESSION RATIO 1&2
`
`S400
`
`GET ONE PLANE
`
`IRREVERSIBLE COMPRESSION
`WITH RATIO 1 OR 2
`
`NO
`
`S401
`
`S402
`
`S403
`
`RECORD ON A MEDIUM
`
`S404
`
`END
`
`C&A Marketing, Inc. Exhibit 1004 Page 1
`
`

`

`Patent Application Publication
`
`Jul. 1, 2004 Sheet 1 of 13
`
`US 2004/0126019 Al
`
`.._,..
`0
`.,...
`(
`>-
`~
`a.
`0
`
`en -
`
`J ..
`
`LC')
`.,...
`0
`(
`>-
`a:
`0
`~
`w
`~
`
`c::
`0
`UJ (./)
`'-' en
`<(UJ
`::;EO -o
`c::
`a.
`
`:E a:::
`0
`•
`enOLL
`o'cn
`o<(z
`<(
`c::
`I-
`
`..
`
`r
`
`..
`
`r
`
`....
`-
`
`....
`
`-
`-
`
`r--
`
`M
`0 .,...
`
`"'--
`
`N
`0 .,...
`
`"'--
`
`.,...
`0 .,...
`
`"'--
`
`.,...
`.,...
`?
`.......... -
`LL
`
`...
`, .....
`
`~
`
`...
`
`en
`0
`.,...
`\,.,
`
`....
`....
`
`-
`0
`. a:
`UJ
`
`...
`....
`
`0 .,...
`.,...
`?
`:::> -
`
`~
`
`.
`~ 0
`UJ
`~
`
`...
`,-..
`
`'w WO
`co
`00
`~o .,.....
`~ 00
`Ow
`~o
`
`r
`
`cc
`0 .,...
`(
`a::
`UJ
`:E ::J
`WO
`.
`~ (./)a::
`>-1-
`enz
`0
`0
`
`...
`
`~
`
`~
`
`0 .,...
`(
`
`...J
`0 ...J
`C::: LU
`1-Z
`Z<(
`oa.
`0
`
`tf3DVl/'JI
`
`~
`
`...
`-
`
`,....._,
`
`0
`0 .,...
`
`S~Il.dO 3DVl/'JI
`
`r--
`
`C&A Marketing, Inc. Exhibit 1004 Page 2
`
`

`

`jBAYER ARRAY I
`
`LOW CORRELATION
`
`LOW CORRELATION
`
`LOW CORRELATION
`
`LOW CORRELATION
`
`. ·~~ DIVIDE TO PLANES
`
`I R PLANE I
`
`1-G-~ PLANE I
`
`FIG.2A
`
`FIG.28
`
`FIG.2C
`
`O
`
`I
`
`I
`
`f
`
`I
`
`.... i
`
`z
`················~·····
`0 . . . . .
`a :5
`~ c--~--i
`..... IBl .... 1
`. . . '
`:
`:
`:
`:
`-w :·
`....,..
`...&... a::
`:_. ___ :
`:
`'
`R i
`! ..... i
`a:
`.....
`0
`0
`HIGH
`CORRELATION
`~
`z
`················~·····
`'
`'
`.
`'
`'
`0
`i= : ..... f ...... ~-----:-----i
`.
`.
`~m .:R:R:····t····j
`•
`0:::
`c:: R
`0
`0
`
`..,...
`..L.<(
`
`I
`I
`
`I
`I
`
`'
`'
`
`f
`I
`
`'
`'
`
`I
`O
`
`'
`'
`
`I
`I
`
`'
`
`····~·····
`:
`:
`..... : ..... ;
`HIGH
`CORRELATION
`
`•
`I Gb PLANE I
`HIGH
`CORRELATION
`
`18 PLANE I
`HIGH
`CORRELATION z
`····-~··· 0
`l B
`i=
`:·····
`.... '
`J: <(
`B
`: CJ _J
`:
`:
`:
`:
`r····i81···· B
`'
`'
`'
`'
`'-w
`:i:: a:::
`~ .... a....
`a:::
`:
`:
`:
`:
`:
`0
`0
`: ..... i .....•..... : ..... i
`
`z
`0
`i=
`J::5 Qw
`
`I
`
`0:::
`0:::
`0
`0
`
`z
`0
`i=
`I:5 Qw
`J: 0:::
`0:::
`0
`0
`
`HIGH
`CORRELATION
`CLOSE GAP
`~
`z
`.......... , ...... ,. ...... .,.. .......
`z
`················~·····
`'
`.
`'
`.
`'
`'
`'
`'
`'
`'
`0
`0
`' ' ' ' '
`i= 1-····i·····f·····i·-···i
`i= 1-····i·····f·····l·····i
`' . ' . '
`....,.. :5
`~w ·--+·--1
`I :5
`: .... + .... i
`c5 ~i· i
`S: w Gr.Gr
`i
`:
`····-~----~
`0:::
`'
`0:::
`a::: G G
`a:::
`i
`i
`0
`' ...... : .....•
`0
`0
`0
`HIGH
`CORRELATION
`
`I
`I
`
`I
`I
`
`I
`I
`
`I
`I
`
`I
`I
`
`I
`I
`
`I
`I
`
`I
`I
`
`I
`f
`
`I
`I
`
`~
`
`·B·B
`.
`
`····~·····
`1 1
`. ... .:.. ...•
`HIGH
`CORRELATION
`
`~
`z
`················~·····
`'
`.
`'
`'
`'
`0
`....,.. :5 . ' . ' .
`i= 1-····i·····f·····l·····i
`
`I
`I
`
`I
`I
`
`I
`I
`
`I
`I
`
`I
`I
`
`~1 a-::r.:i
`
`HIGH
`CORRELATION
`
`(')
`
`~ .....
`
`(')
`
`~ .....
`
`""C
`~ .....
`~ = .....
`~ "Cl -....
`.... 0 =
`~
`O' -....
`.... 0 =
`~ = :-
`"'"" ~
`N
`8 .i;;..
`'Jl =(cid:173)~
`~ .....
`N
`0 .....,
`"'"" ~
`
`d
`'Jl
`N c c
`~ c
`"'"" N
`O'I c
`"'""
`\C
`>
`"'""
`
`C&A Marketing, Inc. Exhibit 1004 Page 3
`
`

`

`I OBLIQUE ARRAY I
`LOW CORRELATION ~ ~ ~ ~
`
`FIG.3A
`
`LOW CORRELATION
`
`:"'·- ....
`
`•
`[ G PLANE I
`.. ---...... ---....
`... ---....
`:. ··@:~---~G: : __ _)
`z @:G:· ··@:G:
`. . . . --~·· ..
`0
`i=~: ··@:.: ··@:_: ··@:.: .)
`:5
`·G:· .. ·~> .. ··~: .. "~· ···.
`I
`CJ
`.
`. .
`. .
`• .
`•
`-w . . . '•
`..
`:
`a::: ·G·--~G
`.. "G ··@G ... •
`a:::
`. . . . . . .
`I
`0 o
`
`HIGH
`CORRELATION
`~
`. .
`
`FIG.38
`
`z
`Q
`
`FIG.3C
`
`~
`.
`
`-----
`........
`........
`.........
`
`'
`
`z! '
`-I-
`0
`I:S
`Qw
`I ct:
`a:::
`0
`0
`
`HIGH
`CORRELATION
`
`HIGH
`CORRELATION
`
`LOW CORRELATION LOW CORRELATION
`t
`I R PLANE I .
`~ (> ~-("';
`("·~··\·(··~ "\···
`. ; .
`. ,.., @
`····: ··\-~··: [: :::R:·
`: .... (-J(. i.\ ~<·) ">
`,.. .. --, __ ,,·--~--\-.. ,'
`?.f6i/){~
`
`'
`
`HIGH
`CORRELATION
`
`DIVIDE INTO PLANES
`
`~·~~~
`I B PLANE I
`... (}.@.() .. @
`..... :
`: : :.:
`:.
`) ....
`....
`..,-' '··~B -·
`"
`.·· B ·\.)-~i>·\
`.......... , ... B .. , .. ··'.··@
`:
`::
`::
`\ .. :\/::; ' . .><:.'
`
`: :
`
`::
`
`:
`
`CLOSE GAP
`
`HIGH
`HIGH
`CORRELATION CORRELATION
`~
`. . .
`. '
`.
`. . .
`. . .
`,~,..-J·····t····-f·····
`
`z
`
`0 -I-
`
`I:S
`Qw
`In::
`a:::
`0
`0
`
`HIGH
`CORRELATION
`
`(')
`
`~ .....
`
`""C
`~ .....
`~ = .....
`~ "Cl -....
`.... 0 =
`~
`O' -....
`.... 0 =
`~ = :-
`'"""' ~
`N
`8 .i;;..
`'Jl =(cid:173)~
`~ .....
`
`(')
`
`~ .....
`
`~
`0 .....,
`'"""' ~
`
`d
`'Jl
`N c c
`~ c
`'"""' N
`O'I c
`'"""'
`\C
`>
`'"""'
`
`C&A Marketing, Inc. Exhibit 1004 Page 4
`
`

`

`, -
`
`LINE ARRAY
`
`HIGH CORRELATION
`
`HIGH CORRELATION
`
`\__} ~
`
`I
`·~G)lR?·G· B >G:<;R~: :·G:· B
`,::o:::ff~!'. :9: s ::G:: :1t.:: ::9:: s
`:·G:·n~~j:i :G: B ::G-::~~R:Fd:: B
`::q:. :~~R~:r:::: s ::G:: )3} ::o:: s
`LOW CORRELATION
`LOW CORRELATION
`t
`[R-PLANE I
`
`FIG.4A
`
`FIG.48
`
`FIG.4C
`
`z
`0
`i==
`
`:c a::
`-UJ
`a::
`0
`O
`
`..
`
`•••••
`
`[GPLANE]
`.'-G>
`:·G:·I
`l:G:~
`....
`. ..
`....
`a~ .. -'G-:1
`·(l:·
`....
`..
`1:d:
`-:a-:1
`1-q:
`. . . .
`..
`<cf
`:G:
`HIGH
`CORRELATION
`~
`5 ::~:: ::9:: ::~{ :S\
`i==
`a~~~-+-····-
`-UJ
`.....
`:c 0::
`0:::
`0 o HIGH
`CORRELATION
`
`(')
`
`~ .....
`
`""C
`~ .....
`~ = .....
`~ "Cl -....
`.... 0 =
`~
`O' -....
`.... 0 =
`~ = :-
`"'"" ~
`N
`8 .i;;..
`rFJ. =(cid:173)~
`~ .....
`
`(')
`
`~ .....
`
`.i;;..
`0
`
`"""
`"'"" ~
`
`d
`rFJ.
`N c c
`~ c
`"'"" N
`O'I c
`"'""
`\C
`>
`"'""
`
`DIVIDE INTO PLANES
`
`• [8 Pl.A:NE-]
`
`z
`0
`I
`:c i== CJ5
`
`-UJ
`:Co::
`0:::
`0
`0
`
`z
`0
`i==
`:c~
`QUJ
`:c 0::: a::
`
`0
`0
`
`B
`B
`B
`B
`
`B
`B
`B
`B
`
`RELATIVELY HIGH
`CORRELATION
`
`_____ .. ____ ...._ ____ _
`
`B B
`B B
`B B
`B B
`
`~
`-----.. ---- ...... -----
`. ---·r. ----~-. -. --
`. .
`'
`'
`-----.. ---- ...... -----
`'
`'
`-----L-----·------
`RELATIVELY HIGH
`CORRELATION
`
`l:G::
`..
`:(}::
`·G:·
`. .. . .
`:G~:
`
`.
`
`g~ i~
`
`j't:
`
`:c ~ [
`
`0
`0
`
`II
`
`·H=tJ
`
`RELATIVELY HIGH
`CORRELATION
`~
`CLOSE GAP
`
`~ ~[ !ii1:::::1::•:1:::::
`
`0
`0
`
`RELATIVELY HIGH
`CORRELATION
`
`C&A Marketing, Inc. Exhibit 1004 Page 5
`
`

`

`Patent Application Publication
`
`Jul. 1, 2004 Sheet 5 of 13
`
`US 2004/0126019 Al
`
`FIG.5
`
`G M G M
`y c y c
`G M G M
`y c y c
`
`FIG.6
`
`y M c G y M c G
`y M c G y M c G
`y M c G y M c G
`y M c G y M c G
`
`C&A Marketing, Inc. Exhibit 1004 Page 6
`
`

`

`Patent Application Publication
`
`Jul. 1, 2004 Sheet 6 of 13
`
`US 2004/0126019 Al
`
`FIG.7
`
`START
`
`GET ONE PLANE
`
`REVERSIBLE COMPRESSION
`
`NO
`
`YES
`
`RECORD TO MEDIUM
`
`END
`
`S200
`
`S201
`
`S202
`
`S203
`
`C&A Marketing, Inc. Exhibit 1004 Page 7
`
`

`

`Patent Application Publication
`
`Jul. 1, 2004 Sheet 7 of 13
`
`US 2004/0126019 Al
`
`FIG.8
`
`START
`
`GET ONE PLANE
`
`REVERSIBLE OR IRREVERSIBLE
`COMPRESSION
`
`NO
`
`----
`
`FINISHED?
`
`YES
`
`S300
`
`S301
`
`S302
`
`RECORD ON A MEDIUM
`
`S303
`
`END
`
`C&A Marketing, Inc. Exhibit 1004 Page 8
`
`

`

`Patent Application Publication
`
`Jul. 1, 2004 Sheet 8 of 13
`
`US 2004/0126019 Al
`
`FIG.9
`
`START
`
`SET COMPRESSION RATIO 1&2
`
`S400
`
`GET ONE PLANE
`
`IRREVERSIBLE COMPRESSION
`WITH RATIO 1 OR 2
`
`NO
`
`S401
`
`S402
`
`S403
`
`YES
`
`RECORD ON A MEDIUM
`
`S404
`
`END
`
`C&A Marketing, Inc. Exhibit 1004 Page 9
`
`

`

`Patent Application Publication
`
`Jul. 1, 2004 Sheet 9 of 13
`
`US 2004/0126019 Al
`
`FIG.10
`
`START
`
`SET COLOR COMPRESSION RATIO
`
`S500
`
`GET ONE COLOR PLANE
`
`S501
`
`IRREVERSIBLE COMPRESSION
`WITH CORRESPONDING
`COMPRESSION RATIO
`
`NO
`
`S502
`
`S503
`
`YES
`
`RECORD ON A MEDIUM
`
`S504
`
`END
`
`C&A Marketing, Inc. Exhibit 1004 Page 10
`
`

`

`Patent Application Publication
`
`Jul. 1, 2004 Sheet 10 of 13
`
`US 2004/0126019 Al
`
`T""""
`
`T"""" .
`
`(!) -LL
`
`LO en
`
`M en
`
`ww:c~
`z
`o_J
`:so
`co
`o<
`a.oww
`..,_zO::§
`-w0 0
`co
`u.
`
`,....
`en
`
`C&A Marketing, Inc. Exhibit 1004 Page 11
`
`

`

`Patent Application Publication
`
`Jul. 1, 2004 Sheet 11 of 13
`
`US 2004/0126019 Al
`
`FIG.12A
`
`FIG.128
`
`OLL
`
`(ORIGINAL IMAGE TILE)
`
`1 LL
`
`1 HL
`
`1LH
`
`1HH
`
`DECOMPOSITION LEVEL=O
`
`DECOMPOSITION LEVEL=1
`
`FIG.12C
`
`FIG.120
`
`2LL 2HL
`
`2LH 2HH
`
`1HL
`
`1LH
`
`1HH
`
`2HL
`(2)
`
`3LL 3HL
`(0)
`(1)
`3LH 3HH
`(1)
`(1)
`2LH 2HH
`(2)
`(2)
`
`1LH
`(3)
`
`1HL
`(3)
`
`1HH
`(3)
`
`DECOMPOSITION LEVEL=2
`
`DECOMPOSITION LEVEL=3
`
`C&A Marketing, Inc. Exhibit 1004 Page 12
`
`

`

`Patent Application Publication Jul. 1, 2004 Sheet 12 of 13
`
`US 2004/0126019 Al
`
`FIG.13
`
`CODE FORMAT GENETRAL SCHEMA
`I soc I MAIN HEADER I
`
`I SOT I TILE HEADER I SOD I · 1 TITLE DATA
`
`I SOT I TILE HEADER I SOD I
`
`I SOT I Tl LE HEADER I SOD I
`
`EOC I
`
`C&A Marketing, Inc. Exhibit 1004 Page 13
`
`

`

`Patent Application Publication
`
`Jul. 1, 2004 Sheet 13 of 13
`
`US 2004/0126019 Al
`
`FIG.14
`
`102
`
`101
`
`106
`
`105
`
`104
`
`103
`
`C&A Marketing, Inc. Exhibit 1004 Page 14
`
`

`

`US 2004/0126019 Al
`
`Jul. 1, 2004
`
`1
`
`IMAGING DEVICE, IMAGE PROCESSING
`METHOD, AND RECORDING MEDIUM
`
`[0001] The present application claims priority to the cor(cid:173)
`responding Japanese Application No. 2002-286044, filed on
`Sep. 30, 2002, the entire contents of which are hereby
`incorporated by reference.
`
`BACKGROUND OF THE INVENTION
`
`[0002] The present invention relates to imaging devices
`for generating color images such as digital still cameras,
`digital video cameras, stand-alone type image scanners,
`image scanners integrated to copying machines, and the like.
`More particularly, the present invention relates to an imag(cid:173)
`ing device that uses an imager of the type of that generates
`an image with color decomposition.
`
`[0003] A CCD-type or MOS-type imager (imaging ele(cid:173)
`ment) for use in digital still cameras or digital video cameras
`generally has color filters for color decomposition on an
`imaging surface thereof and captures an image with color
`decomposition. Thus, each pixel therein provides informa(cid:173)
`tion about only one color component. Thus, generally con(cid:173)
`duct color interpolating processing is performed for each of
`the pixels in the imager for interpolating image information
`of two other color components, by using the image infor(cid:173)
`mation of the nearby pixels, such that the image information
`for all the color components is acquired for each of the
`pixels in the imager. With this, color pixel data is obtained
`for all the pixels on the imager.
`
`[0004] During the foregoing interpolation processing,
`other signal processing such as white-balance processing,
`gamma correction processing, edge enhancement process(cid:173)
`ing, and the like, are generally conducted simultaneously.
`
`[0005] The color image data thus processed is generally
`recorded on a recording medium after compression accord(cid:173)
`ing to a compression encoding algorithm such as JPEG.
`
`[0006] On the other hand, there are cameras that record
`image date without compression.
`
`[0007] Further, there are some high-grade cameras such as
`single-lens reflex type digital still cameras that have the
`function or mode of raw-data recording for image data
`recording. In this raw-data recording mode, the output signal
`of the imager is recorded on the recording medium after
`conversion to digital signals (raw image data) without
`applying data compression.
`
`[0008] With regard to the technology of compression of
`image information obtained by an imager, there is proposed
`a method and apparatus that uses primary color filters of a
`Bayer array (see FIG. 2A) in Japanese Laid-Open Patent
`Application 2002-516540. According to this technology,
`data compression is applied independently to four signals,
`i.e., a differential signal of adjacent R and G pixels, a G
`signal of the G pixel, a differential signal of adjacent B and
`G pixels, and a G signal of a G pixel.
`
`[0010] Meanwhile, it should be noted that the content of
`the signal processing mentioned above depends on the
`manufacturer of the imaging device and the details of the
`processing are usually not disclosed. Thus, in the case a user
`wishes to conduct a retouch process to the image data
`according to the object or purpose of generating pictures, or
`according to the preference of the user, by using a retouch
`software, it is preferable that the user can use the row image
`data, which is free from the effect of signal processing that
`is conducted inside the camera. Further, it is preferable that
`the effect of the signal processing in the camera does not
`appear in such a retouch process even in the case that
`non-reversible compression of image data has been con(cid:173)
`ducted in the camera.
`
`[0011] A raw image data recording mode is the mode for
`meeting the demands of the user. On the other hand, such a
`row image data recording mode has a drawback in that the
`amount of the image data to be stored in the recording
`medium becomes enormous. In case the imager has three
`million pixels, for example, the image data for a single
`picture may be several megabytes or more, and the effi(cid:173)
`ciency of using the recording medium is degraded seriously.
`
`In the case of the technology described in the
`[0012]
`foregoing Japanese Laid-Open Patent Application 2002-
`516540, op. cit., too, there is conducted a processing for
`calculating the R-G and B-G difference signals, and thus,
`problems arise that are similar to the case of signal process(cid:173)
`ing conducted inside the camera as noted above.
`
`SUMMARY OF THE INVENTION
`
`[0013] A method and apparatus for image processing is
`disclosed. In one embodiment, the color imaging device
`comprises an imager that generates an image with color
`decomposition. The imager produces raw image data as a
`result of generating the image. The color imaging device
`also comprises a raw image data decomposing unit to
`decompose the raw image data into multiple color planes
`such that each color plane includes data of pixels of the same
`color in the form of sub-color image data. The data com(cid:173)
`pressing unit compresses the sub-color image data for each
`of the multiple color planes.
`
`BRIEF DESCRIPTION OF THF DRAWINGS
`[0014] FIG.1 is a block diagram of one embodiment of an
`image processing system;
`
`[0015] FIGS. 2A-2C are diagrams illustrating a Bayer
`array of primary color filters and decomposition of raw
`image data into color planes;
`
`[0016] FIGS. 3A-3C are diagrams illustrating an oblique
`array of primary color filters and decomposition of raw
`image data into color planes;
`
`[0017] FIGS. 4A-4C are diagrams illustrating a line array
`of primary color filters and decomposition of raw image data
`into color planes;
`
`In the art of digital still cameras and digital video
`[0009]
`cameras, JPEG or MPEG technology have been used com(cid:173)
`monly for compression of image data, while JPEG 2000
`(ISO/IEC FCD 15444-1) or Motion-JPEG 2000 (ISO/IEC
`FCD 1544-3) is drawing attention as a new compressive
`encoding algorithm that can replace the foregoing conven(cid:173)
`tional algorithms.
`
`[0018] FIG. 5 is a diagram showing a Bayer array of
`complementary color filters;
`
`[0019] FIG. 6 is a diagram showing a line array of
`complementary filters;
`
`[0020] FIG. 7 is a flow diagram illustrating Mode A
`operation of the imaging device of FIG. 1;
`
`C&A Marketing, Inc. Exhibit 1004 Page 15
`
`

`

`US 2004/0126019 Al
`
`Jul. 1, 2004
`
`2
`
`[0021] FIG. 8 is a flow diagram illustrating Mode B
`operation of the imaging device of FIG. 1;
`
`[0022] FIG. 9 is a flow diagram illustrating Mode C
`operation of the imaging device of FIG. 1;
`
`[0023] FIG. 10 is a flow diagram illustrating Mode D
`operation of the imaging device of FIG. 1;
`
`[0024] FIG. 11 is a block diagram illustrating a compres(cid:173)
`sion encoding algorithm according to JPEG 2000;
`
`[0025] FIGS. 12A-12D are diagrams showing a two(cid:173)
`dimensional wavelet conversion;
`
`[0026] FIG. 13 is a diagram showing format of encoded
`data of JPEG 2000; and
`
`[0027] FIG. 14 is a diagram showing a computer that can
`conduct the image processing of one embodiment of the
`present invention.
`
`DETAILED DESCRIPTION OF IBE
`INVENTION
`
`[0028] Accordingly, one embodiment of the present inven(cid:173)
`tion provides an imaging device wherein the foregoing
`problems are eliminated.
`
`In the case of the technology described in the
`[0029]
`foregoing Japanese Laid-Open Patent Application 2002-
`516540, op. cit., too, there is conducted a process for
`calculating the R-G and B-G difference signals, and thus,
`there arises problems similar to the case of signal processing
`conducted inside the camera as noted above.
`
`[0030] Another and more specific embodiment of the
`present invention provides an imaging device of the type
`that generates an image with color decomposition, where the
`image data is compressed efficiently and simultaneously
`without undergoing the influence of the signal processing
`conducted inside the imaging device.
`
`[0031] Another embodiment of the present invention pro(cid:173)
`vides a color imaging device that includes an imager that
`generates an image with color decomposition, where the
`imager produces raw image data as a result of generating the
`image; a data decomposing unit to decompose the raw image
`data into a plurality of color planes such that each color
`plane includes data of pixels of the same color in the form
`of sub-color image data; and a data compressing unit to
`compress the sub-color image data for each of the plurality
`of color planes.
`
`In one embodiment of the color imaging device
`[0032]
`noted above, the data compressing unit conducts a reversible
`data compression for each of the plurality of color planes.
`
`In another embodiment of the color imaging device
`[0033]
`noted above, the data compression unit applies a reversible
`data compression to a specific color plane that provides a
`relatively large influence on the resolution of a reproduced
`image and further a non-reversible data compression to the
`other color planes that provide less influence on the resolu(cid:173)
`tion of the reproduced image.
`
`In another embodiment of the color imaging device
`[0034]
`of the present invention noted above, the data compression
`unit applies a non-reversible compression to all of the
`plurality of color planes.
`
`In another embodiment of the color imaging device
`[0035]
`of the present invention noted above, a controller is provided
`for controlling the compression ratio for a specific color
`plane that provides a larger effect on the resolution of the
`reproduced image, independently to the compression ratio
`for other color planes in which the effect on the resolution
`of the reproduced image is smaller.
`
`In another embodiment of the present invention,
`[0036]
`the controller sets, in the case a user of the color imaging
`device attaches importance to resolution of reproduced
`images, the compression ratio of the specific color plane to
`be smaller than a standard compression ratio.
`
`In another embodiment of the present invention,
`[0037]
`the controller sets, in the case a user of the color imaging
`device attaches importance to resolution of reproduced
`images, the compression ratio of the specific color plane to
`be smaller than a standard compression ratio and the com(cid:173)
`pression ratio of the other color planes to be larger than the
`standard compression ratio.
`
`In another embodiment of the present invention,
`[0038]
`the controller sets, in the case a user of the color imaging
`device attaches importance to color reproducibility of repro(cid:173)
`duced images, the compression ratio of the specific color
`plane to be smaller than a standard compression ratio.
`
`In another embodiment of the present invention,
`[0039]
`the controller sets, in the case a user of the color imaging
`device attaches importance to color reproducibility of repro(cid:173)
`duced images, the compression ratio of the specific color
`plane to be smaller than a standard compression ratio and the
`compression ratio of other color planes to be larger than the
`standard compression ratio.
`
`In a further embodiment of the present invention,
`[0040]
`the data compression unit applies a non-reversible compres(cid:173)
`sion to all of the plurality of color planes, and a controller
`is provided that controls a compression ratio of each of the
`color planes independently.
`
`In a further embodiment of the present invention,
`[0041]
`there is further provided an information acquiring unit that
`acquires information for determining the compression ratio
`for each of the color planes, and wherein the controller
`determines the compression ratio of each of the color planes
`based on the information acquired by the information
`acquiring unit.
`
`In a further embodiment of the present invention,
`[0042]
`the information acquiring unit acquires information about
`the proportion of high frequency components for each color,
`and wherein the controller sets the compression ratio of the
`color plane in which the proportion of the high-frequency
`component is smallest to be higher than a standard com(cid:173)
`pression ratio.
`
`In a further embodiment of the present invention,
`[0043]
`the information acquiring unit acquires evaluation of white(cid:173)
`balance, and wherein the controller determines whether the
`proportion of the color component is large or small based on
`the evaluation, the controller further setting the compression
`ratio of the color plane of which white-balance is determined
`as being small to be larger than a standard compression ratio.
`
`In a further embodiment of the present invention,
`[0044]
`the controller sets the compression ratio for each color plane
`based on an instruction of a user of the color imaging device.
`
`C&A Marketing, Inc. Exhibit 1004 Page 16
`
`

`

`US 2004/0126019 Al
`
`Jul. 1, 2004
`
`3
`
`[0045]
`In a further embodiment of the present invention,
`the compression unit compresses data of each color plane
`according to a compression encoding algorithm in compli(cid:173)
`ance with JPEG 2000.
`
`[0046] Another embodiment of the present invention pro(cid:173)
`vides an image processing process, comprising: obtaining
`raw image data by an imager that generates an image of an
`object with color decomposition; decomposing the raw
`image data into a plurality of color planes, where each of the
`color planes comprising pixel information of a color perti(cid:173)
`nent to the color plane; and compressing data of each color
`plane.
`
`[0047]
`In one embodiment of the present invention, the
`operation of compressing the color plane is conducted by a
`reversible compression process for all of the color planes.
`
`[0048]
`In a further embodiment of the present invention,
`the compression operation is conducted such that only a
`specific color plane which provides a large influence on a
`reproduced image is subjected to a reversible compression
`process while the remaining color planes are subjected to an
`irreversible compression process.
`
`[0049]
`In a further embodiment of the present invention,
`the compression operation is conducted with an irreversible
`process for all of the color planes.
`
`[0050]
`In a further embodiment of the present invention, a
`compression ratio for a specific color plane that provides a
`large influence on the resolution of a reproduced image is set
`independently with respect to a compression ratio of other
`color planes in which the influence on the resolution of the
`reproduced image is smaller.
`
`[0051]
`In a further embodiment of the present invention,
`the process further includes setting a compression ratio for
`each of the color planes independently.
`
`[0052]
`In a still further embodiment of the present inven(cid:173)
`tion, the process further includes acquiring information for
`determining the compression ratio for each of the color
`planes from the raw image data, where the compression ratio
`is determined by the acquired information.
`
`[0053]
`In a yet another embodiment of the present inven(cid:173)
`tion, the compression is conducted according to a compres(cid:173)
`sion encoding algorithm in compliance with JPEG 2000.
`
`[0054] According to the present invention, high compres(cid:173)
`sion efficiency is achieved as compared with the case of
`compressing the raw image data itself because of the com(cid:173)
`pression conducted for each of the color planes divided out
`or decomposed from the raw image data. Thereby, it should
`be noted that strong correlation existing between adjacent
`pixels in such color planes is utilized. As the encoded data
`thus obtained is derived directly from the raw image data,
`which is free from signal processing such as color interpo(cid:173)
`lation processing, white balance processing, gamma correc(cid:173)
`tion processing, edge enhancement processing, and the like,
`conducted in the camera, the encoded data thus obtained is
`also free from the effect of such signal processing. Because
`of the high compression ratio, the image data obtained by
`one embodiment of the present invention has a compact size
`and the efficiency of utilization of recording medium is
`improved substantially when the encoded image data of the
`present invention is recorded on a recording medium as
`compared with the case of recording the raw image data
`
`itself or as compared with the case of recording the raw
`image data with straightforward compression.
`[0055] By conducting a reversible compression for all of
`the color planes, it becomes possible to reproduce the image
`data exactly identical with the raw image data, which is free
`from the effect of signal processing noted above. Because of
`the strong correlation existing between adjacent pixels,
`embodiments of the present invention can achieve high
`compression efficiency even when such a reversible com(cid:173)
`pression process is employed.
`[0056] By employing reversible compression to a specific
`color plane, which provides a profound effect on the repro(cid:173)
`duced image quality, and applying irreversible compression
`to the remaining color planes, it becomes possible to reduce
`the file size of the obtained encoded image data while
`reducing, and potentially minimizing, the deterioration in
`the reproduced image quality.
`[0057] Further, by employing irreversible compression for
`all of the color planes, it becomes possible to reduce the file
`size of the encoded image data further and the efficiency of
`utilization of the recording medium is improved further.
`[0058]
`In the case of employing irreversible compression
`for all of the color planes, it becomes possible to improve the
`efficiency of compression while reducing, and potentially
`minimizing, the degradation of resolution or color repro(cid:173)
`ducibility of reproduced image, by changing the compres(cid:173)
`sion ratio between the color planes independently such that
`a compression ratio smaller than a standard compression
`ratio is applied to a specific color plane that provides a
`profound effect on the resolution or color reproducibility of
`reproduced image.
`[0059] By constructing the imaging device such that the
`user can specify the compression ratio, it becomes possible
`to conduct image processing according to the object of the
`picturing. Alternatively, the compression ratio can be set
`automatically by extracting information about setting of the
`compression ratio from the raw image data.
`[0060] By using JPEG 2000 compression encoding algo(cid:173)
`rithm for the compression, the degradation of image quality
`is suppressed in an embodiment of the present invention and
`high efficiency of compression is achieved at the same time
`even in the case of conducting reversible compression.
`Further, by conducting post quantization process, it becomes
`possible to adjust the compression ration without conducting
`a recompression process. Thereby, the compression can be
`achieved more easily with a target compression ratio.
`[0061] Other objects and further features of the present
`invention will become apparent from the following detailed
`description of embodiments of the present invention when
`read in conjunction with the attached drawings.
`[0062]
`In the description below, JPEG 2000 (ISO/IEC
`FCD 15444-1) is used for the compression encoding algo(cid:173)
`rithm of still images. Further, Motion-JPEG 2000 (ISO/IEC
`FCD 1544-3) is used for the compression encoding algo(cid:173)
`rithm of motion pictures. In Motion-JPEG 2000, each of
`continuous still images is handled as a frame, and each
`frame (still picture) is subjected to compression encoding in
`compliance to JPEG 2000. With regard to JPEG 2000,
`reference should be made to Noguchi, Y., "Next Generation
`Image Encoding Method JPEG 2000", Triceps, Tokyo, Feb.
`13, 2001 (in Japanese).
`
`C&A Marketing, Inc. Exhibit 1004 Page 17
`
`

`

`US 2004/0126019 Al
`
`Jul. 1, 2004
`
`4
`
`[0063] Hereinafter, general summary of this technology
`will be provided.
`
`[0064] FIG. 11 shows a simplified block diagram for
`illustrating the algorithm of JPEG 2000.
`
`[006S]
`In JPEG 2000, the image data subjected to com(cid:173)
`pression encoding (image data of each frame in the case of
`handling a motion picture) is divided into rectangular non(cid:173)
`overlapping regions called tiles for each component, and the
`image processing is conducted on such tiles. Further, it is
`possible that the tile may have the size of a full image. In this
`case, the division of the image into tiles is not performed.
`
`[0066] Next, a two-dimensional wavelet conversion (dis(cid:173)
`crete wavelet conversion DW1) is applied to each tile in
`each component (step Sl).
`
`[0067] FIG. 12 illustrates the wavelet conversion for the
`case the decomposition level number is 3.
`
`[0068] As a result of the two-dimensional wavelet con(cid:173)
`version for the original image tile shown in FIG. 12A
`(decomposition level=O), the tile is divided into sub-bands
`lLL, lHL, lLH and lHH shown in FIG. 12B.
`
`[0069] Further, by applying a two-dimensional wavelet
`conversion to the coefficients of the sub-band lLL, the
`sub-band is divided into sub-bands 2LL, 2HL, 2LH and
`2HH as represented in FIG. 12C. Further, by applying a
`two-dimensional wavelet conversion to the coefficients of
`the sub-band 2LL, the sub-band 2LL is divided into sub(cid:173)
`bands 3LL, 3HL, 3LH and 3HH as represented in FIG. 12D.
`In FIG. 12D, the numeral in the bracket represents the
`resolution level.
`
`[0070] The wavelet coefficients obtained by such a recur(cid:173)
`sive divide (octave divide) of low-frequency components
`(LL sub-band coefficient) are then quantized for each sub(cid:173)
`band (step 2). In JPEG 2000, both of reversible (loss-free)
`compression and irreversible (lossy) compression are pos(cid:173)
`sible. In the case of using reversible compression, the step
`width of quantization is 1, and no quantization is made in
`this stage.
`
`[0071] After the quantization, each of the sub-band coef(cid:173)
`ficients is subjected to an entropy encoding process (step
`S3). In this entropy encoding process, an encoding technique
`called EB COT (embedded block coding with optimized
`truncation) is used, where the encoding process includes
`block dividing, coefficient modeling and binary arithmetic
`encoding. As a result of the entropy encoding, the bit plane
`of each sub-band coefficient is encoded from the upper plane
`to the lower plane for each block called code block.
`
`[0072] The last t