COPY
`
`REDCOM.007Xl
`
`PATENT
`
`IN THE UNITED STATES PATENT AND TRADEMARK OFFICE
`
`Inventor
`
`App. No.
`
`Filed
`
`For
`
`James Jannard, et al.
`
`90/012550
`
`September 13, 2012
`
`VIDEO CAMERA
`
`Examiner
`
`Tran, Henry N.
`
`Art Unit
`
`ConfNo.
`
`3992
`
`1159
`
`CERTIFICATE OF EFS WEB
`TRANSMISSION
`
`I hereby certify that this correspondence, and any
`other attachment noted on the automated
`Acknowledgement Receipt, is being transmitted
`from within the Pacific Time zone to the
`Commissioner for Patents via the EFS Web server
`on:
`
`January 31 2014
`(Date)
`
`/Michael Guiliana/
`Michael A. Guiliana, Reg. no. 42, 611
`
`DECLARATION OF GRAEME NATTRESS
`
`Commissioner for Patents
`P.O. Box 1450
`Alexandria, VA 22313-1450
`
`Dear Sir:
`
`I, Thomas Graeme Nattress, declare that:
`
`I. BACKGROUND
`
`1.
`
`I am a lead camera systems architect at Red.com, Inc. (dba Red Digital Camera) ("RED"),
`
`the assignee of U.S. Patent No. 8,174,560 ("the 560 patent"), which is a subject of the
`
`present reexamination proceeding.
`
`I am also a listed inventor on the '560 patent and an
`
`inventor on seven additional issued patents that belong to RED.
`
`2.
`
`In addition to my position at RED, I am founder and co-owner of Nattress Productions Inc.,
`
`where I have been developing and selling a range of custom image processing software
`
`solutions since 2004. Before joining RED I held positions from about 2001 to 2004 as Vice
`
`President of Research and Development at Noitaminanimation Inc. and Cinerio
`
`-1-
`
`RED.COM Ex. 2024, Page 1 of 34
`
`RED.COM Ex. 2024
`Apple v. RED.COM
`IPR2019-01065
`
`

`

`Application No.:
`Filing Date:
`
`90/012550
`September 13, 2012
`
`Entertainment, Inc., where I developed custom animation software for internal company
`
`use. At Noitaminanimation I worked with digital cinematography cameras to do matching
`
`of computer animation to real footage.
`
`3.
`
`I hold a Bachelor's of Science in Mathematics from the University of Newcastle upon
`
`Tyne, and a Postgraduate Certificate in Education from the University of Sunderland.
`
`During my university years, I was heavily involved with video production, and this is where
`
`my passion and interest in motion imagery really began to take form. During this time I
`
`was an early adopter of non-linear video editing and special effects software, and was a
`
`professional reviewer of such software for Content Creation Europe magazine.
`
`4.
`
`I have also edited a number of books in the digital video field. One of these books-The
`
`Fi/maker's Handbook: A Comprehensive Guide for the Digital Age, by Steven Ascher and
`
`Edward Pincus-is widely acknowledged as the "bible" of film and video production.
`
`5, At RED, my responsibilities include designing and implementing image processing
`
`algorithms, pipelines and workflow.
`
`I have been intimately involved in the process of
`
`designing and building all of RED's camera models from the ground up.
`
`6.
`
`I joined RED in 2005, and based upon the investment of an enormous amount of effort and
`
`resources, we created the first ever digital cinematography camera which was capable of
`
`capturing and recording compressed RAW image data, on board, at 2k and higher
`
`resolution, and at a frame rate of at least about 23 frames per second. Specifically, our
`
`cameras compress and record raw digital image data having a resolution of at least 2k
`
`(including 4k) into a storage device of the camera (e.g., carried on or within a portable
`
`RED.COM Ex. 2024, Page 2 of 34
`
`

`

`Application No.:
`Filing Date:
`
`90/012550
`September 13, 2012
`
`housing of the camera) at a frame rate of at least about twenty-three frames per second,
`
`where the data remains substantially visually lossless upon decompression.
`
`7.
`
`I have reviewed all of the outstanding rejections in the Office Actions dated October 31,
`
`2013 and March 29, 2013, as well as any references cited in the September 13, 2012
`
`Request for Reexamination. The cumulative list of references includes:
`
`• U.S. Patent Publication No. 2006/0061822 to Sung et al. ("Sung '822")
`
`• U.S. Patent Publication No. 2007/0092149 to Sung et al. ("Sung '149")
`
`• On the Dependency Between Compression and Demosaicing in Digital Cinema,
`by D. Menon, et al. ("Menon")
`
`• U.S. Patent Publication No. 2010/0014590 to Smith et al. ("Smith")
`
`• English Language Patent Abstract of Japanese Published Application No.
`JP-06054239 to Nishimura ("Nishimura")
`
`• U.S. Patent Publication No. 4,450,487 to Koide ("Koide")
`
`• The ISO Standard 15444 (Part I): Information Technology - JPEG 2000 Image
`Coding System: Core Coding, pages i-v, xiv, 1-11, and 120-122 (ISO/IEC
`15444-1) ("JPEG 2000")
`
`• Single-sensor Camera Image Compression by Lukac et al., IEEE Transactions on
`Consumer Electronics, Vol. 52, pages 299-307 ("Lukac")
`
`• U.S. Patent No. 6,825,876 to Easwar et al. ("Easwar")
`
`8.
`
`I have carefully reviewed the entire text of each of the above references, as well as any
`
`comments by the Examiner relating to the references, where applicable. The following is
`
`an explanation of certain deficiencies of each of the references as it relates to the claims of
`
`the '560 patent.
`
`9.
`
`Those of ordinary skill in the art of cinema-grade digital movie camera design and video
`
`image processing, at the time of filing of the '560 patent, typically had a 4-year university
`
`degree in a field such as Mathematics as well as at least two years of experience with
`
`high-end digital photography equipment and processing techniques used in broadcast, mass
`-3-
`
`RED.COM Ex. 2024, Page 3 of 34
`
`

`

`Application No.:
`FHing Date:
`
`90/012550
`September 13, 2012
`
`publication or cinema production. Additionally, one could acquire such a level of skill,
`
`without a university degree, through at least two years hands-on involvement with high-end
`
`digital photography equipment and processing techniques used during participation in
`
`broadcast, mass publication or cinema productions along with associated on-the-job
`
`training.
`
`II. DESCRIPTION OF TYPICAL PRIOR ART DATA FLOW FOR HIGH END
`
`PHOTOGRAPHY
`
`10. To understand the reasoning of those of ordinary skill in the art of cinema-grade movie
`
`cameras, it is important to consider the needs, requirements, and expectations of the
`
`customer base of those of skill in the art. For example, a camera user in the cinema
`
`production market must face the reality of the costs required to prepare a scene for
`
`"filming" a cinema production, as well as the unique requirements of cinema recording.
`
`The filming of some scenes in modem movies can generate $100,000 per hour of costs for
`
`preparation and support during actual filming. During filming, all of that cost is transferred
`
`into the media used for "filming". As such, the recording system associated with the
`
`camera must reliably capture the desired motion video at a level of quality that can
`
`withstand intense scrutiny.
`
`11. The final goal of a movie production effort is to provide a motion video that can be
`
`enlarged to larger-than-life scales, on screens that can be 50 feet wide or larger, without any
`
`audience member being distracted by errors or artifacts, throughout the length of the
`
`production which is often over 2 hours long. To achieve that level quality and consistency,
`
`RED.COM Ex. 2024, Page 4 of 34
`
`

`

`Application No.:
`Filing Date:
`
`90/012550
`September 13, 2012
`
`nearly every frame of a cinema production is studied by experts to ensure the absence or
`
`successful elimination of undesirable characteristics found in the recorded video.
`
`12. At the time we filed the '560 patent, the cinema industry did not employ any video
`
`compression techniques for processing digital video recordings, either in camera or
`
`downstream. Thus, when we released the first RED camera, cinema industry professionals
`
`were reluctant to even attempt to use a cinema-grade camera that employed any
`
`compression, let alone an unprecedented technique of compressing and storing raw
`
`mosaiced image data (e.g., Bayer pattern image data or other color filter array image data)
`
`on board the camera. This is because the best known compression techniques, which were
`
`designed for use with full color plane, demosaiced image data has already proven
`
`inadequate for cinema applications. Thus, the risks of wasted production costs were too
`
`great to attempt to use unproven techniques in actual cinema projects.
`
`III. DESCRIPTION OF TYPICAL PRIOR ART DATA FLOW FOR HIGH END
`
`PHOTOGRAPHY
`
`13. Set forth below is a basic explanation of video image processing in the context of the
`
`concerns of one of ordinary skill in the art of cinema quality video image data capture and
`
`processing.
`
`Image
`Sensor
`
`El~!El
`LJ LJ LJ
`
`Compr
`ession
`
`14.
`
`In the flowchart set forth above, an original image of light is captured by an image sensor.
`
`In this case, the image sensor includes a Bayer-pattern sensor in which each point of the
`
`-5-
`
`RED.COM Ex. 2024, Page 5 of 34
`
`

`

`Application No.:
`Filing Date:
`
`90/012550
`September 13, 2012
`
`sensor can only detect the luminosity of a single color, either red, blue, or green.
`
`Additionally, there are twice as many green locations as there are red or blue.
`
`15. The values for each of these colors can be separated and represented as three different
`
`planes, each having values in certain locations and blanks in others. Thus, each one of
`
`these planes is incomplete. In order to create a complete image, all the missing blue data
`
`must be determined, all of the missing red data must be determined, and all the missing
`
`green values must be determined. That process is usually referred to as "de-mosaicing" and
`
`in some texts as "interpolation".
`
`16. One aspect of these incomplete color planes noted above is that the original raw values of a
`
`single color from the image sensor correspond to points on an image that are not spatially
`
`continuous. Thus, for example, the values of the green data from an incomplete raw color
`
`plane of Bayer-pattern data, the values are spaced apart from one another and are staggered.
`
`Aggregating those values into a block of continuous values results in an effective shifting
`
`of some of the data, resulting in artificial zipper-like discontinuities at image edges.
`
`Compression algorithms are designed to include predictors. Such predictors are designed
`
`for spatially continuous image data. Thus, when compression algorithms are applied
`
`directly to Bayer-pattern data, the algorithms do not function as intended, there will be
`
`more entropy in the residual and such data is less compressible.
`
`17. Typical data workflow processes applied to non-cinema, lower-end video camera data
`
`included a de-mosaic process of the image data prior to compression. Such demosaicing
`
`techniques allow other entropy reducing techniques to be applied subsequently, and
`
`generally improve how certain steps of compression algorithms perform, including de-
`
`-6-
`
`RED.COM Ex. 2024, Page 6 of 34
`
`

`

`Application No.:
`Filing Date:
`
`90/012550
`September 13, 2012
`
`correlation transforms, spatial transforms and predictors. Such demosaicing techniques
`
`include algorithms of the following types, as just a few examples:
`
`• Bilinear interpolation-based
`
`• Edge-directed interpolation
`
`• Alternating projection-based interpolation
`
`• Homogeneity-directed demosaicing
`
`• Primary-consistent soft-decisioning demosaicing
`
`• Successive approximation with edge-weighted improvement
`
`• Demosaicing with directional filtering and a aposteriori decision
`
`• Variance of color difference
`
`•
`
`Interpolation using spatial/spectral correlation+ adaptive median filtering
`
`• Direction linear minimum mean square error estimation
`
`• Local polynomial approximation
`
`• Adaptive filtering-based demosaicing
`
`18. Additionally, other processes that are often used for further enhancing demosaiced video
`
`data also improve compressibility.
`
`For example, such other processes include the
`
`following:
`
`• Color correction (sometimes alternatively referred to as "color compensation")
`(e.g., application of color matrix)
`
`RED.COM Ex. 2024, Page 7 of 34
`
`

`

`Application No.:
`Filing Date:
`
`90/012550
`September 13, 2012
`
`• Tonal processing (e.g., application of tonal curves, contrast enhancement)
`
`• Gamma processing (sometimes alternatively referred to as "gamma correction")
`
`19. Those of ordinary skill in the art understood that these processes improved compressibility.
`
`For example, one article published in May 2006, around the time of filing of the '560
`
`patent, concluded that "both JPEG and JPEG 2000 produced the best results in
`
`conjunction with powerful ED and CCA (demosaicing) schemes." Single-Sensor Camera
`
`Image Compression, Lukac, et al., IEEE Transactions on Consumer Electronics, Vol. 52,
`
`No. 2, May 2006 ("Lukac", discussed below), page 302, right hand column, last full
`
`paragraph, lines 6-8 (emphasis and parenthetical added). Thus, Lukac states that more
`
`powerful demosaicing improves compression. Lukac additionally concluded that "at high
`
`compression ratios [sic] (straightforward [mosaic] image compression) produced the worst
`
`results." Lukac, page 303, left hand column, lines 15-16 (emphasis added).
`
`20. Those of ordinary skill in the art also understood that color correction (matrix-based and
`
`other color correction techniques) enhances compressibility because it makes the image a
`
`much closer approximation (or often identical) to the final rendering intent of the image.
`
`Tonal processing (curves, contrast enhancement) also enhances compressibility because it
`
`makes the image a much closer approximation (or often identical) to the final rendering
`
`intent of the image. Gamma processing places the image in a more perceptually linear
`
`domain and hence any lossy compression that relies on the human visual system to disguise
`
`artifacts will be more effective. All of the above noted processes, however, can only be
`
`performed on demosaiced data, not mosaiced data. There was no accepted use of the
`
`-8-
`
`RED.COM Ex. 2024, Page 8 of 34
`
`

`

`Application No.:
`Filing Date:
`
`90/012550
`September 13, 2012
`
`above-listed terms in the context of performing the corresponding operations on mosaiced
`
`image data.
`
`21. Although the application of demosaicing plus other processes, such as those noted above,
`
`were believed to provide the best results for compression, none resulted in a compression
`
`technique that could be used for cinema production. Those of ordinary skill in the art at the
`
`time of filing the '560 patent were well aware that the cinema industry considered the best
`
`compression techniques to be unusable for cinema production. Specifically, those of
`
`ordinary skill in the art were aware that the best designed compression techniques used in,
`
`for example, DVDs, web video and low-end digital cameras, resulted in "artifacts" so
`
`severe that no one in the cinema industry would even attempt to use any video image
`
`compression technique on a cinema project.
`
`IV. REJECTION OF CONVENTIONAL THINKING
`
`22. As Mr. J annard and I studied the perceived problems and other barriers to the use of
`
`compression in cinema movie cameras, we realized that the very techniques believed to
`
`provide the best compression results left lasting effects in the video image data that appear
`
`as artifacts and also unacceptably limited downstream editability. More specifically, with
`
`regard to compression techniques, we realized that in any of the known compression codecs
`
`used for video image data compression, the resulting decompressed value of one pixel is
`
`affected by the values of surrounding data points during the compression process. For
`
`example, when Bayer-sensor data is demosaiced prior to compression, the missing values
`
`for each color are filled-in with values output from the chosen demosaicing technique.
`
`Techniques such as color correction techniques, which are considered as compression-
`
`RED.COM Ex. 2024, Page 9 of 34
`
`

`

`Application No.:
`FHing Date:
`
`90/012550
`September 13, 2012
`
`enhancing, shift the values of each color of each pixel. As such, even before compression,
`
`the interpolated values affect the original raw mosaiced values through the color correction
`
`process. During compression, the compression codec considers all of the values to produce
`
`the compressed version of the data. The codec uses various techniques for characterizing
`
`the data in a way that best fits the data input into the codec, which includes both the
`
`original raw mosaiced data, shifted by way of the color correction technique noted above,
`
`as well as the interpolated data, which has also been shifted by the color correction
`
`technique. Thus, the interpolated data affects the processing and analysis of the original
`
`raw mosaiced data values. Upon decompression, the effects of the interpolated data values
`
`and the color correction technique on the original raw mosaiced data values cannot be
`
`removed; those effects have been "baked-in" and the raw image data is irreversibly
`
`compromised. We realized that it was those "baked-in" effects that prevented the best
`
`conventional compression techniques from being used in cinema production.
`
`23. Thus, at the time of filing the '560 patent, many cinema industry professionals including
`
`those of ordinary skill in the art of cinema-quality camera design, rejected the use of on(cid:173)
`
`camera compression as a useful data-rate control tool because the best conventional
`
`approaches to compression had not provided results that could be used for cinema
`
`production.
`
`24. To arrive at our invention of a cinema camera that compresses raw, mosaiced video image
`
`data on board the camera at cinema quality resolutions and frame rates, we moved in a
`
`direction opposite to the conventional thinking that demosaicing combined with other
`
`processes produce better compression results. This conventional view would lead one to
`
`-10-
`
`RED.COM Ex. 2024, Page 10 of 34
`
`

`

`Application No.:
`Filing Date:
`
`90/012550
`September 13, 2012
`
`believe that attempting to compress raw mosaiced video data, which is incomplete and
`
`spatially non-continuous, which could not be processed with the best compressibility(cid:173)
`
`improving techniques such as color compensation and other techniques, would result in
`
`worse results than those produced by compression of demosaiced and enhanced data. This
`
`view was confirmed in the Lukac article which states "at high compression ratios [sic]
`
`(straightforward [mosaic] image compression) produced the worst results." Lukac, page
`
`303, left hand column, lines 15-16 (emphasis added). Consistent with the conventional
`
`thinking, many were skeptical that our camera would perform sufficiently well to be
`
`useable in the cinema environment.
`
`25. At the time of filing the '560 patent, one of ordinary skill in the art would have also faced
`
`the reality that none of the implementation guides for the then-available compression
`
`codecs, such as JPEG 2000, provided guidance for compressing raw mosaiced data.
`
`Rather, those guides only included guidance for compressing demosaiced, color corrected,
`
`tonally processed, gamma corrected video image data.
`
`26. As of the filing date of the 560 patent, those of ordinary skill in the art of the cinema video
`
`camera market would not have found any credible reason, in light of all the prior art noted
`
`below, to attempt to create a camera as recited in the present claims.
`
`27. Thus, one of ordinary skill in the art in the environment of cinema quality video image
`
`processing would not have found it obvious to develop a technique for implementing
`
`known compression techniques, such as JPEG 2000, to operate on non-demosaiced data
`
`(for which there is no guidance in the codec guides) and to build a system that could
`
`perform such a technique on board a cinema-grade camera, i.e., a camera that has at least
`
`-11-
`
`RED.COM Ex. 2024, Page 11 of 34
`
`

`

`Application No.:
`Filing Date:
`
`90/012550
`September 13, 2012
`
`2K horizontal resolution at 23 frames per second to store that compressed raw mosaiced
`
`image data on board the camera. Even where some prior art references include references
`
`to the possibility of compressing mosaiced data in the context of low end imagery
`
`applications, such references do not provide a reason for one of ordinary skill in the art to
`
`develop a cinema-grade camera with on-board storage and compression of raw mosaiced
`
`image data.
`
`28. Additionally, those of ordinary skill in the art at the time of filing of the '560 patent would
`
`have concluded that (1) certain references described in greater detail below necessarily and
`
`inherently include a de-mosaicing process prior to video image data compression, (2)
`
`hypothetical proposals for compressing mosaiced image data in the context of low-end,
`
`previously demosaiced and enhanced imagery, were insufficient to make it obvious to
`
`develop an approach that could be used for compressing raw mosaiced image data at a
`
`resolution of at least 2K at a frame rate of at least 23 frames per second on board a camera,
`
`and
`
`(3)
`
`the accepted wisdom and some published research counseled against
`
`implementation of compression of raw mosaiced image data for high-end uses.
`
`V. U.S. PATENT APPLICATION PUBLICATION NO. 2006/0061822 TO SUNG ET AL.
`
`("SUNG '822")
`
`29. The Sung '822 reference does not disclose compression of raw, mosaiced video image data.
`
`Firstly, the use of the term "raw" in the Sung '822 reference the following quote from the
`
`text of the '822 reference makes it mathematically certain that Sung '822 refers to mosaiced
`
`data as "raw data", as follows:
`
`-12-
`
`RED.COM Ex. 2024, Page 12 of 34
`
`

`

`Application No.:
`Filing Date:
`
`90/012550
`September 13, 2012
`
`Taking a 3 minion pixels digital camera as an example, it requires a total
`of 9.0 minion bytes of density of memory to store the raw data before
`sending them into an image or video compression engine.
`
`Sung '822, 'lI [0007] (emphasis added).
`
`30. Because the above quote states that 9 million bytes of memory are required to store 3
`
`million pixels, that mathematically means that there are 3 bytes of information per pixel,
`
`i.e., three colors per pixel. However, that is three times the amount of information that
`
`comes out of a sensor chip in a camera.
`
`31. The pixel arrangement used in the sensor 50 of Sung '822 is illustrated in Figure 5 thereof
`
`(reproduced below). The sensor 50 illustrated in Figure 5 is a "Bayer pattern" image sensor
`
`which is configured to detect only one color of light for each pixel location. Additionally,
`
`as is typical with Bayer-pattern sensors, there are twice as many green pixels as there are
`
`red or blue.
`
`R
`
`Fig. 6A
`
`32. At paragraphs [0041] and [0042], Sung '822 explains that the data from the image sensor
`
`50 (Figure 5) is then processed for better compressibility using techniques illustrated in
`
`Figure 6A and 6B. With regard to Figure 6A, Sung '822 explains that the values of the "G
`
`plane 62" can be subtracted from the values of the red "plane 60". However, in order to a
`
`-13-
`
`RED.COM Ex. 2024, Page 13 of 34
`
`

`

`Application No.:
`Filing Date:
`
`90/012550
`September 13, 2012
`
`green "plane" from a red "plane", there would need to be some correlation between those
`
`values. But, as shown in Figure 5, there are twice as many green values as red or blue
`
`values. Thus, one of ordinary skill in the art would understand that the R plane 60 and G
`
`plane 62 of Figure 6A are actually full interpolated color "planes" of an image. In other
`
`words, the data from the pixels of Figure 50 has been fully interpolated so that for each
`
`individual pixel, whether it is associated with red, green or blue, the values of the other two
`
`colors have been calculated from other surrounding data,
`
`i.e., by "interpolation".
`
`Accordingly, where the image sensor 50 includes 3 million pixels, 9 million bytes of
`
`memory would be recorded; one byte of memory used for each color of each pixel.
`
`33. Thus, it is clear from the text of the Sung '822 that a full de-mosaic step must necessarily
`
`be performed between the data flow represented by Figures 5 and 6A of the Sung '822
`
`reference. Further, one of ordinary skill in the art would conclude that such a de-mosaicing
`
`step would be used in preparing the data from Figure 5 for the process illustrated in Figure
`
`6A.
`
`34. The term "de-mosaic" does not appear in the text of the Sung '822 reference. Thus, one of
`
`ordinary skill in the art, when reading the disclosure of the Sung '822 reference, would
`
`assume that the description of the de-mosaic process performed on the data from the sensor
`
`50 illustrated in Figure 5 before the operations illustrated in Figure 6A has been omitted.
`
`Otherwise, the text of the Sung '822 reference would not make sense. Further, the lack of
`
`any further discussion of de-mosaicing data in the text of the Sung '822 reference
`
`reinforces the conclusion that the de-mosaicing of the data from the image sensor 50 occurs
`
`prior to the procedure illustrated in Figure 6A.
`
`-14-
`
`RED.COM Ex. 2024, Page 14 of 34
`
`

`

`Application No.:
`Filing Date:
`
`90/012550
`September 13, 2012
`
`35. The conclusion that there is a de-mosaicing process performed on the data from the image
`
`sensor 50 prior to the operation illustrated in Figure 6A is further reinforced in the
`
`description of Figure lOB in paragraph [0060] of Sung '822. Specifically, paragraph
`
`[0060] of Sung '822 states the following (emphasis added):
`
`The captured raw image data is then delivered to an image processing unit
`color
`like gamma correction,
`functions
`includes
`116 which
`compensation and interpolation. The image pixels after the imaging
`processing unit 116 are processed by a compression engine 118 then
`stored temporarily before they are sent to another image compression
`engine 120 for data reduction so that smaller size files are available for
`storing in a flash memory 124 or other recording medium or being
`transmitted via a network.
`
`36.
`
`In this context, the word "interpolation" will be understood by one of ordinary skill in the
`
`art to correspond to a de-mosaicing of the data from the image sensor 50 to form the color
`
`planes illustrated in Figure 6A, specifically, the "R plane 60 and G plane 62". Sung '822,
`
`'JI [0042]. Thus, Sung '822 describes compressing demosaiced image data, not mosaiced
`
`image data.
`
`37. This understanding by those of ordinary skill that the word "interpolation" in this context
`
`would correspond to de-mosaicing is evidenced by the following excerpts from Circles of
`
`Confusion, a well-known industry technology handbook authored by Alan Roberts and
`
`published by the European Broadcasting Union:
`
`All these patterns (referring to Bayer and mosaic patterns in general)
`require interpolation to reconstruct RGB signals at full resolution.
`
`Circles of Confusion, page 127 (emphasis and parenthetical added), included as
`Exhibit A.
`
`In a single sensor camera there is only a single color value available at
`each cell position so, to create RGB signals, interpolation is needed.
`
`-15-
`
`RED.COM Ex. 2024, Page 15 of 34
`
`

`

`Application No.:
`Filing Date:
`
`90/012550
`September 13, 2012
`
`Circles of Confusion, page 128 (emphasis added), included as Exhibit A.
`
`38. Moreover, gamma correction and color compensation can only be performed on full color
`
`resolution, demosaiced image data, and cannot be performed on mosaiced data. Thus, the
`
`above reference to gamma correction and color compensation in Sung '822 prior to
`
`compression further supports the conclusion that the process described in Sung' 822 must
`
`include compressing demosaiced image data and does not include compressing raw
`
`mosaiced data.
`
`39. The fact that the above-listed operations can only be performed on fully processed RGB
`
`data is commonly understood in the art. For example, with respect to color correction/color
`
`compensation, although sensors perceive red, green and blue light via dyes in the color
`
`filter array, the dyes do not match the response of the human visual system. Color
`
`correction matrices are designed to correct for this by adjusting hues in the image so that
`
`the colors in the scene appear correct to the viewer. In order to adjust hues in the image,
`
`the color correction matrix must operate on full color resolution, demosaiced image data.
`
`For example, to add or subtract some percentage of green or blue from red for a pixel to
`
`alter the hue of that pixel from a slightly orange-ish red to a pure firetruck red, intensity
`
`values for all three colors--red, green, and blue--are needed for that pixel, and the image
`
`data must therefore be demosaiced. Moreover, color correction matrices are also designed
`
`to increase image saturation in order to counteract a corresponding reduction in image
`
`saturation that results from a degree of color cross-talk due to overlap in the color response
`
`of the dyes in the color filter array. In order to increase image saturation and counteract the
`
`color cross-talk, all of the color channels are needed for each pixel. For instance, to
`
`-16-
`
`RED.COM Ex. 2024, Page 16 of 34
`
`

`

`Application No.:
`Filing Date:
`
`90/012550
`September 13, 2012
`
`subtract a percentage of red and blue from green for a particular pixel, values for all three
`
`colors are needed for that pixel, and the image data must therefore be demosaiced.
`
`40.
`
`It is also well understood that gamma correction can only be applied to full color resolution,
`
`demosaiced image data. This follows from the way image data is processed and displayed.
`
`In particular, electronic displays are designed to receive and process a full color resolution
`
`(demosaiced), color corrected, gamma corrected image signal, such as an RGB or Y'CbCr
`
`image signal. In order to properly display the image signal, the display applies an inverse
`
`gamma correction curve. And because image processing operations are generally not
`
`commutative, the display must perform the inverse gamma curve on the image data as it
`
`existed immediately prior to the application of the original gamma correction operation.
`
`That is, the display can only properly process the image data if gamma correction was the
`
`last operation performed on the image data, after demosaicing, as well as after other
`
`operations such as color processing, tonal processing, etc.
`
`41.
`
`It is also well known that tonal processing can only be applied to full color resolution,
`
`demosaiced image data. This is because tonal processing will appear visually very different
`
`depending on the RGB color space that the tonal processing is done in. In particular, for
`
`the tonal processing to appear visually correct and as intended on a display, the tonal
`
`processing must be done in calibrated, full color resolution, demosaiced, RGB space.
`
`42. The fact that the above-listed operations can only be performed on full color resolution,
`
`demosaiced data is also well documented in industry publications. By way of example, the
`
`following excerpt from Introduction to Color Imaging Science shows that "colour
`
`correction" is applied after "CFA interpolation" (demosaicing):
`
`-17-
`
`RED.COM Ex. 2024, Page 17 of 34
`
`

`

`Application No.:
`Filing Date:
`
`90/012550
`September 13, 2012
`
`The many steps of signal processing typically include: (1) input signal
`shaping; (2) analog-to-digital conversion; (3) pixel-offset correction; (4)
`pixel gain correction; (5) pixel defect correction; (6) CFA interpolation;
`(7) exposure (density) balance; (8) colour (white) balance; (9) colour
`correction (3x3 matrix); (10) output signal processing.
`
`Hsien-Che Lee, Introduction to Color Imaging Science, pages 506-507, included
`as Exhibit B.
`
`43. The following graphic from another publication shows that "RGB blending", which would
`
`be understood to be equivalent to color correction, as well as gamma correction, are
`
`performed after demosaicing ("CFA interpolation"), i.e. on demosaiced data.
`
`Bl1'1Ckl11val -(cid:157)
`AdJ111.tman1
`
`Ga1mna
`Correctim1
`
`(=
`
`RBS
`Blending
`
`Nola
`~~--", White
`Retl11ttlo11 ~· Balance:
`D
`
`¢::::
`
`tiA
`lnttrpolation
`
`ROBtoYCC ¢::.:.
`CQl!vereion
`
`n
`
`Edgt
`Enltanee
`
`=:> Con1rnt =✓·
`E.1,1tanct1
`
`Faist Chro111~
`Suppreulo11
`
`Jianping Zhou, Getting the Most Out of Your Image-Processing Pipeline, page 3
`(http://www.ti.com/lit/wp/sprvlO~!wrvl05.pdf}, included as Exhibit C.
`
`44. Similarly,
`
`the Wikipedia
`
`page
`
`for
`
`"Color
`
`Image
`
`Pipeline",
`
`includes
`
`the following graphic depicting "3x3 matrix" (i.e. color correction) and "Gamma correct"
`
`operations performed after a "Debayer" (de