`
`
`
`
`
`
`
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`__________________________________
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`__________________________________
`
`APPLE INC.,
`Petitioner,
`
`v.
`
`RED.COM, LLC,
`Patent Owner.
`__________________________________
`
`Case No. IPR2019-01065
`Patent No. 9,245,314
`__________________________________
`
`DECLARATION OF THOMAS GRAEME NATTRESS IN SUPPORT OF
`PATENT OWNER RED.COM, LLC PRELIMINARY RESPONSE
`
`
`
`
`RED.COM Ex. 2001
`Apple v. RED.COM
`IPR2019-01065
`
`
`
`Apple v. RED.COM
`Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065
`
`
`I, Thomas Graeme Nattress, declare and state as follows:
`
`I. INTRODUCTION
`
`1.
`
`I am a lead camera systems architect at RED.COM (“RED”), the
`
`assignee of U.S. Patent Nos. 9,230,299 (“the ’299 patent”) and 9,245,314 (the ’314
`
`patent). I am also a listed inventor on the ’299 and ’314 patents, and an inventor
`
`on over twenty-five additional issued patents assigned to RED. I am submitting
`
`this declaration in connection with Patent Owner Preliminary Responses to
`
`IPR2019-010164 and IPR2019-01065, filed by Petitioner Apple Inc., relating to
`
`the ’299 and ’314 patents, respectively.
`
`2.
`
`I executed a previous declaration, which was submitted on January 31,
`
`2014, in connection with a reexamination proceeding on U.S. Patent No. 8,174,560
`
`(the “’560 patent”), assigned to RED and for which I am a named inventor (the
`
`“2014 declaration”). A copy of my 2014 declaration can be found at pages 312-
`
`345 in Petitioner’s Ex. 1002 in IPR2019-01064. I understand that Petitioner’s Ex.
`
`1002 in IPR2019-01064 is a copy of the prosecution history of the ’299 patent. A
`
`true and correct copy of my 2014 declaration is also attached hereto as Exhibit
`
`2024 for use and reference in IPR2019-01065, as my 2014 declaration does not
`
`appear in Ex. 1002 (’314 patent prosecution history) in IPR2019-01065.
`
`3.
`
`A detailed summary of my biography and credentials can be found in
`
`my 2014 declaration at paragraphs 1-6.
`
`-1-
`
`
`
`Apple v. RED.COM
`Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065
`
`
`4.
`
`In my 2014 declaration, I testified regarding the inventions claimed in
`
`the ’560 patent. In particular, paragraphs 10-28 in my 2014 declaration describe
`
`the state of the art at the time of the filing of the ’560 patent, and the deficiencies in
`
`technology and conventional thinking that were overcome by the inventions
`
`claimed in the ’560 patent. Those paragraphs relate to the April 11, 2007 time
`
`frame, which is the earliest filing date to which the ’560 patent claims priority.
`
`The ’299 and ’314 patents claim priority to the ’560 patent, and to the same April
`
`11, 2007 provisional application as the ’560 patent. Thus, paragraphs 10-28 in my
`
`2014 declaration refer to the deficiencies in technology and conventional thinking
`
`that were overcome by the inventions in claimed in the ’299 and ’314 patents.
`
`II. REDUCTION TO PRACTICE OF THE
`RED ONE MOTION PICTURE CAMERAS
`
`5.
`
`I first met Mr. Jim Jannard in December of 2005. At that meeting, we
`
`discussed his desire to create the first ever digital motion picture camera that could
`
`record compressed digital motion video at cinema quality levels, including 4K. I
`
`was intrigued by the possibilities, because combining the ease and flexibility post-
`
`production of digital video while maintaining cinema-quality frame rate and
`
`resolution would be a game-changer in the world of movie making. But I also
`
`knew that digital compression was highly disfavored in the movie camera industry
`
`-2-
`
`
`
`Apple v. RED.COM
`Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065
`
`due to its resulting artifacts and lower resolution, which were unacceptable for big-
`
`screen cinema viewing.
`
`6.
`
`To solve this problem, one key area I researched was the use of an
`
`image sensor with a Bayer-pattern filter. This was one of several unconventional
`
`avenues we explored at RED. At the time, such sensors were associated with
`
`lower-quality consumer-grade video cameras and derided as incapable of providing
`
`cinema-quality video due to artifact and resolution issues. In contrast, the industry
`
`consensus held that cinema quality cameras would need to utilize three sensors,
`
`with a prism to split red, green and blue light to each sensor. However, we
`
`believed that the benefits of a Bayer-pattern image sensor could be optimized if the
`
`image data remained in raw, mosaiced format for compression. We believed that
`
`such a data workflow could allow the raw digital files to operate as a digital
`
`negative, and would provide all of the post-production flexibility of being able to
`
`manipulate the original raw data.
`
`Developing REDCODE for the RED ONE Cameras
`
`7.
`
`Immediately following my December 2005 meeting with Mr. Jannard,
`
`I began working on the design of RED’s first commercial digital motion picture
`
`camera that would become known as the RED ONE. This work would last all
`
`throughout 2006 and into 2007 when we commercially launched the RED ONE
`
`-3-
`
`
`
`Apple v. RED.COM
`Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065
`
`video camera. My title on the RED ONE project was Problem Solver, and it
`
`remains my title to this day.
`
`8.
`
`As I explained in paragraphs 13-21 of my 2014 declaration, the
`
`typical flowchart for prior art video image data capture and processing involved
`
`demosaicing video data captured by a Bayer-pattern sensor prior to compressing
`
`that data for storage. Paragraphs 22-28 of my 2014 declaration explain how we
`
`rejected
`
`that conventional
`
`thinking by
`
`implementing known compression
`
`techniques, such as JPEG 2000, in such a way as to operate on non-demosaiced
`
`Bayer-pattern image data, a type of image data these techniques were not designed
`
`to work on.
`
`9.
`
`Accordingly, my work on the RED ONE camera was based on a video
`
`image processing pipeline that did not include a conventional demosaicing step
`
`prior to compression. Rather, the data flow upon which I based my research and
`
`design work for the RED ONE camera generally operated along the following
`
`four-step sequence for processing and compressing raw Bayer-pattern video image
`
`data: First, raw image data was collected by the image sensor, which was a
`
`Mysterium CMOS image sensor chip with a Bayer-pattern pixel filter, i.e., a sensor
`
`chip that detected only one data value for each of the green, red and blue pixel
`
`locations. The raw mosaiced data was outputted by the image sensor at a
`
`resolution of at least 2K and at least 23 frames per second. Second, the raw Bayer-
`
`-4-
`
`
`
`Apple v. RED.COM
`Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065
`
`pattern image data was then sent to a Xilinx processing FPGA chip for pixel
`
`correction and processing of the raw Bayer-pattern image data. Third, the pixel-
`
`corrected and processed raw Bayer-pattern video image data was then sent to
`
`Analog Devices compression chips, which utilized a mathematically lossy wavelet
`
`compression codec known as JPEG 2000, to compress the processed raw Bayer-
`
`pattern video image data. Fourth, the compressed raw Bayer-pattern image video
`
`data was sent to a memory device, by way of a SATA port, for storage as a raw
`
`data file that at one point in development was given a .JIM file extension in
`
`homage to Mr. Jannard and the new file type that RED had created. We referred to
`
`the above programing for the RED ONE cameras, and the resulting raw
`
`compressed data files that it generated, as REDCODE.
`
`10. Much of my research and design work that went into the RED ONE
`
`focused on the second step above, i.e., how to process the raw Bayer-pattern video
`
`image data before compression in way would optimize the decompressed and
`
`demosaiced output. This was a key step in REDCODE because, without pre-
`
`processing, JPEG 2000 compression was otherwise incapable of processing raw
`
`Bayer-pattern image data in a useful way. To solve this problem, I devised and
`
`tested a number of methods for processing the raw Bayer-pattern video image data
`
`prior to compression. I ultimately devised two methods in particular that were
`
`-5-
`
`
`
`Apple v. RED.COM
`Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065
`
`employed on the RED ONE cameras, referred to as GAS (green average
`
`subtraction) and pre-emphasis.
`
`11. The GAS processing technique uses an averaging of green pixel
`
`values to produce low noise, co-sited estimated green values. By way of
`
`background, the highest quality demosaic algorithms work on a large kernel to
`
`provide high quality estimates of edge direction. The edge direction is perturbable
`
`from changes to the data in the input image data to the demosaic algorithm to a
`
`much greater extent than the small local average used in the GAS method. In the
`
`GAS method, green pixels located adjacent or in close proximity to a target pixel
`
`are averaged to calculate an average green value, which is then subtracted from the
`
`value of the target pixel, whether the target pixel is red or blue. For example, a
`
`distinct average green value may be calculated for each target pixel location, then
`
`those distinct average green values are subtracted from the values of the
`
`corresponding distinct red and blue pixels. This subtraction can result in a large
`
`number of zeros, for example, in areas of an image that are black, white or shades
`
`of gray; where the values generated by the image sensor for all three colors are
`
`equal.
`
`12. The pre-emphasis processing technique can be used to help preserve
`
`image detail. This technique helps tune the compression of pixel values through a
`
`pre-emphasis curve that changes the gradient in different sensor code value regions
`
`-6-
`
`
`
`Apple v. RED.COM
`Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065
`
`to either emphasize or de-emphasize those pixel values relative to others in the
`
`image. For example, this technique can be used to spread-apart (increase gradient)
`
`pixel values in some ranges of values and squeeze-together (reduce gradient)
`
`values in one or more other ranges. The spreading-apart of values preferentially
`
`preserves the variations in values, and thus image details, in the ranges where the
`
`values are spread apart so that the corresponding details better survive the
`
`compression process. In one implementation, the pre-emphasis curve is set to
`
`spread apart pixel values corresponding to the darkest image regions, whereas the
`
`relatively brightest image regions were squeezed together. This allowed for
`
`compression that better preserved the details in the darkest areas of the image,
`
`which can otherwise often be washed out. After sufficient testing to find the most
`
`ideal curve, keeping in mind the unfixed rendering intent of the raw image data, I
`
`settled on a simple power law curve with adjustable black offset. This curve
`
`resulted in low complexity and a single tuning variable, among other benefits.
`
`13. The REDCODE programming described above formed the backbone
`
`of the RED ONE cameras, including the two RED ONE cameras known as Boris
`
`and Natasha.
`
`The Boris RED ONE Motion Picture Camera
`
`14. RED’s work refining and debugging the RED ONE continued
`
`throughout 2006 and into 2007. With the goal of debuting the RED ONE at the
`
`-7-
`
`
`
`Apple v. RED.COM
`Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065
`
`upcoming NAB 2007 show taking place in Las Vegas on April 14-19, 2007,
`
`several RED team members tested a RED ONE motion picture camera, nicknamed
`
`Boris, on March 8, 2007 at RED’s Lake Forest, California, headquarters. I was
`
`present for that testing, as was Jim Jannard, Jarred Land, Ted Schilowitz, David
`
`Macintosh and Stuart English, among others. I documented the event by taking
`
`photos with my personal camera. True and correct copies of seven of those photos
`
`are attached hereto as Exhibits 2002-2008.
`
`15. To confirm the dates of the photos that I took of the testing of Boris, I
`
`ran ExifTool on the photo files to collect the metadata for the photos. Attached
`
`hereto as Exhibit 2009 is a true and correct copy of the output of the metadata of
`
`files on my camera, including those relating to the photos those I took of the
`
`testing of Boris at RED headquarters. Exhibits 2002-2008, respectively,
`
`correspond to IMG_4291, IMG_4327, IMG_4317, IMG_4338, IMG_4354,
`
`IMG_4364 and IMG_4365, which all indicate a “Date/Time Original” of March 8,
`
`2007. As an aside, the timestamp associated with the “Date/Time Original”
`
`indication reports time for the Eastern time zone, which corresponds to my home
`
`time zone at the time in Ottawa, Canada.
`
`16. Exhibit 2002, reproduced below, shows Jim Jannard inspecting Boris.
`
`-8-
`
`
`
`Apple v. RED.COM
`Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065
`
`
`17. Exhibit 2003, reproduced below, shows, from left to right, David
`
`Macintosh, Jim Jannard, Ted Schilowitz, Jarred Land and Stuart English standing
`
`next to Boris after it had been set up on a tripod to conduct test footage.
`
`
`
`
`
`-9-
`
`
`
`Apple v. RED.COM
`Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065
`
`
`18. Exhibit 2004, reproduced below, is a close-up shot of Boris on the
`
`tripod. David Macintosh can be seen in the background. This photo also shows a
`
`wire running from Boris to a live monitor (out of frame) on which we viewed the
`
`live feed from Boris. Also visible in this photo is the internal framework for
`
`Boris’s hardware circuitry, as Boris’s cover is removed. This photo also shows
`
`that a section of Boris between the body and lens had a black powder coating,
`
`while the remainder of the exterior was milled aluminum.
`
`
`
`
`
`
`
`-10-
`
`
`
`Apple v. RED.COM
`Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065
`
`
`19. Exhibit 2005, reproduced below, is a photo looking head-on into the
`
`front of Boris on the tripod. In the background of the photo, from left to right, is
`
`David Macintosh, Jarred Land and Ted Schilowitz. The lens of my camera is
`
`visible in the reflection of the lens on Boris. The reflection appears to show me
`
`adjusting the focus of my camera as Jarred Land simultaneously adjusts the focus
`
`on Boris.
`
`
`
`
`
`
`
`
`
`-11-
`
`
`
`Apple v. RED.COM
`Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065
`
`
`20. Exhibit 2006, reproduced below, shows Ted Schilowitz posing next
`
`to a ColorChecker chart, also referred to as a Macbeth chart, while Boris is
`
`recording video of the shot. In the upper-right-hand foreground, an image of Ted
`
`Schilowitz can be seen in the monitor connected to Boris showing the live feed of
`
`the video.
`
`
`21. After recording test footage from Boris, we took the compressed raw
`
`Bayer-pattern video image data that Boris had saved to file, and played it back on a
`
`computer in decompressed and demosaiced form to examine the results. Exhibit
`
`
`
`-12-
`
`
`
`Apple v. RED.COM
`Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065
`
`2007, reproduced below, shows Ted Schilowitz reviewing the output of the Boris
`
`test footage taken of himself.
`
`
`
`22. The results of the test footage were a resounding success. Visual
`
`inspection of the playback of the video files recorded by Boris confirmed that it
`
`had recorded compressed raw mosaiced video image data that had been outputted
`
`in visually lossless form with no visible compression artifacts after decompression
`
`and demosaicing.
`
`23. While the visually lossless nature of the Boris test footage was evident
`
`upon visual inspection alone, confirmation of its high resolution and frame rate
`
`was also confirmed by the computer program used to display the Boris video
`
`-13-
`
`
`
`Apple v. RED.COM
`Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065
`
`footage. Although the output parameters of the test footage of Ted Schilowitz
`
`cannot be seen in the photo above, a photo of the output of test footage taken of
`
`David Macintosh provides a clearer image. That photo, Exhibit 2008, is
`
`reproduced below with a yellow circle annotation to indicate the output
`
`parameters.
`
`
`
`
`
`
`
`
`
`
`
`
`
`-14-
`
`
`
`Apple v. RED.COM
`Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065
`
`
`24. Below is an enlargement of the output parameters contained in the
`
`annotated circle in Exhibit 2008.
`
`
`
`
`
`The output parameters read as follows:
`
`Format Settings:
`
`Format Custom: W: 4096 H: 2048
`
`Aspect Custom: Scale: 1.0000
`
`Framerate: 24.000
`
`Show: Full res. High.
`
`-15-
`
`
`
`Apple v. RED.COM
`Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065
`
`At the top of the enlargement, the “.jim” file extension we used to denote
`
`compressed raw Bayer-pattern video image data created by REDCODE can be
`
`seen.
`
`25. These parameters report an output resolution of 4096 x 2048, which
`
`denotes 4K resolution. The framerate is reported as 24 frames per second. Based
`
`on these parameters, and my inspection of the visually lossless video shot by Boris,
`
`I knew by at least March 8, 2007 that our invention would work for its intended
`
`purpose. My understanding was then further confirmed later in March when Sir
`
`Peter Jackson successfully used Boris and its companion camera, Natasha, to shoot
`
`his “Crossing the Line” film.
`
`The April 2, 2007 Release Note for Boris and Natasha
`
`26. As part of my research and development of REDCODE, I instructed
`
`Wind River on a regular basis and conveyed how REDCODE needed to operate on
`
`the RED ONE cameras.
`
`27. Wind River would routinely provide documentation detailing the
`
`progress of the tasks we had assigned them. Attached as Exhibit 2010 is a true
`
`and correct copy of a Release Note, dated April 2, 2007, that Wind River provided
`
`to RED in connection with the Boris and Natasha cameras. This document was
`
`typical of the documentation that Wind River would generate on the RED ONE
`
`-16-
`
`
`
`Apple v. RED.COM
`Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065
`
`project, and provide to RED on or near the date indicated on the cover. RED has
`
`maintained a copy of Exhibit 2010 since the RED ONE project.
`
`28. The cover of the April 2, 2007 Release Note states that it was
`
`“Prepared for Diamond.” Diamond was Wind River’s code name for RED. The
`
`cover also indicates that the document is a “Release Note” for “Sundance.”
`
`Sundance was Wind River’s code name for the RED ONE. Page four of the April
`
`2, 2007 Release Note indicates that the “released cameras are designated as
`
`‘Natasha’ and ‘Boris.’”
`
`29. Page five of the April 2, 2007 Release Note states, “Captured .jim
`
`files were verified by Diamond’s Graeme and David Macintosh to produce
`
`expected quality and the expected results from the new RED/BLUE pre-emphasis
`
`feature.” I am the Graeme referred to in this sentence. The “new RED/BLUE pre-
`
`emphasis feature” refers to an adjustment to the pre-emphasis functionality already
`
`running in REDCODE on Boris and Natasha. The “.jim” files are the compressed
`
`raw Bayer-pattern video image data produced by REDCODE running on Boris and
`
`Natasha.
`
`Huffman Compression
`
`30. As mentioned previously, the RED ONE cameras, including Boris and
`
`Natasha, relied upon JPEG2000 compression, which is a mathematically lossy
`
`compression technique. Huffman compression, as recited in the claims of the ’299
`
`-17-
`
`
`
`Apple v. RED.COM
`Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065
`
`patent, refers to a lossless compression technique that has been well known for
`
`many decades.
`
`31. A lossless compression technique provides a lower compression ratio
`
`than lossy compression. While this difference impacts the write speed and
`
`memory on the hard drive, it does not lower the visual quality of the output upon
`
`decompression. Because the REDCODE on Boris and Natasha produced visually
`
`lossless output with a lossy compression technique, a person of ordinary skill in the
`
`art would know that RECODE would successfully produce visually lossless output
`
`using a lossless compression routine like Huffman compression. This result
`
`follows necessarily because a lossless compression technique removes any
`
`potentiality for compression artifacts caused by lossy compression.
`
`
`
`I declare that all statements made herein of my own knowledge are true and
`
`that all statements made on information and belief are believed to be true, and
`
`further that these statements were made with the knowledge that willful, false
`
`statements and the like so made are punishable by fine or imprisonment, or both,
`
`under Section 1001 of Title 18 of the United States Code.
`
`-18-
`
`
`
`Apple v. RED.COM
`Declaration of Graeme Nattress re POPR - IPR2019-010164 and IPR2019-01065
`
`Executed on August 15, 2019 at Acton, Ontario, Canada.
`
`30703838
`
`-19-
`
`
Accessing this document will incur an additional charge of $.
After purchase, you can access this document again without charge.
Accept $ Charge
Still Working On It
This document is taking longer than usual to download. This can happen if we need to contact the court directly to obtain the document and their servers are running slowly.
Give it another minute or two to complete, and then try the refresh button.
A few More Minutes ... Still Working
It can take up to 5 minutes for us to download a document if the court servers are running slowly.
Thank you for your continued patience.
This document could not be displayed.
We could not find this document within its docket. Please go back to the docket page and check the link. If that does not work, go back to the docket and refresh it to pull the newest information.
Your account does not support viewing this document.
You need a Paid Account to view this document. Click here to change your account type.
Your account does not support viewing this document.
Set your membership
status to view this document.
With a Docket Alarm membership, you'll
get a whole lot more, including:
- Up-to-date information for this case.
- Email alerts whenever there is an update.
- Full text search for other cases.
- Get email alerts whenever a new case matches your search.
One Moment Please
The filing “” is large (MB) and is being downloaded.
Please refresh this page in a few minutes to see if the filing has been downloaded. The filing will also be emailed to you when the download completes.
Your document is on its way!
If you do not receive the document in five minutes, contact support at support@docketalarm.com.
Sealed Document
We are unable to display this document, it may be under a court ordered seal.
If you have proper credentials to access the file, you may proceed directly to the court's system using your government issued username and password.
Access Government Site
