(12) United States Patent
`Presler
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 9,565.419 B2
`Feb. 7, 2017
`
`USO09565419B2
`
`(54) DIGITAL CAMERA SYSTEM FOR
`RECORDING, EDITING AND VISUALIZING
`IMAGES
`
`(76) Inventor: Ari M. Presler, Niskayuna, NY (US)
`(*) Notice:
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 2199 days.
`(21) Appl. No.:
`12/595,811
`(22) PCT Filed:
`Apr. 14, 2008
`(86). PCT No.:
`PCT/US2O08/060272
`S 371 (c)(1),
`(2), (4) Date: Oct. 13, 2009
`(87) PCT Pub. No.: WO2008/128205
`PCT Pub. Date: Oct. 23, 2008
`
`(65)
`
`Prior Publication Data
`US 2010/011 1489 A1
`May 6, 2010
`
`Related U.S. Application Data
`(60) Provisional application No. 60/923,339, filed on Apr.
`13, 2007.
`
`(51) Int. Cl.
`HO)4N 5/907
`HO)4N 9/79
`H04N 9/804
`GO3B 7/56
`H04N I3/02
`
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(Continued)
`
`(52) U.S. Cl.
`CPC .......... H04N 13/0203 (2013.01); G03B 17/56
`(2013.01); H04N 5/225 (2013.01); H04N
`5/232 (2013.01); H04N 5/772 (2013.01);
`H04N 5/765 (2013.01); H04N 5/781
`
`(2013.01); H04N 5/85 (2013.01); H04N 5/907
`(2013.01); H04N 9/7921 (2013.01); H04N
`9/804 (2013.01)
`
`(58) Field of Classification Search
`CPC ... H04N 13/0203; H04N 5/225; H04N 5/232;
`H04N 5/765; H04N 5/772; H04N 5/781;
`H04N 5/85, H04N 5/907; H04N 9/7921;
`H04N 9/804; G03B 17/56
`USPC ......................... 725/105; 348/340, 46, 211.2
`See application file for complete search history.
`References Cited
`
`(56)
`
`U.S. PATENT DOCUMENTS
`
`1/1967 Bing et al. .................... 396,541
`3,295.425 A *
`3,927,269 A * 12/1975 Yoshino et al. ................ 348.24
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`JP
`JP
`WO
`
`2002223385 A
`2007096412 A
`WO-95095.10 A1
`
`8, 2002
`4/2007
`4f1995
`
`OTHER PUBLICATIONS
`
`Silicon imaging, Silicon Imaging SI-1920HD MegaCamera 1080P/
`24/30/60
`12-bit
`High-Definition
`Digital Camera, 2004,
`XP002682243, whole document.
`(Continued)
`Primary Examiner — Twyler Haskins
`Assistant Examiner — Fayez. Bhuiyan
`(74) Attorney, Agent, or Firm — Matthew J. Spark; Zuber
`Lawler & Del Duca
`ABSTRACT
`(57)
`A digital camera system (20), as illustrated in FIG. 1,
`includes an optical assembly (22) to gather light (24) from
`a desired scene (26), a modular imaging Subsystem (28)
`aligned with the optical assembly (22), and an image pro
`cessing, recording and display Subsystem (34).
`44 Claims, 12 Drawing Sheets
`
`
`
`38
`
`SOFTWARE
`
`VAGE
`SENSOR
`
`FRAME
`GRABBER
`
`
`
`PROCESSOR
`
`DSPAY
`
`SCENE
`u1
`
`42
`
`USER
`
`STORAGE
`
`44
`
`APPLE v. RED.COM
`
`Page 1 of 29
`
`Apple Ex. 1005
`
`

`

`US 9,565.419 B2
`Page 2
`
`(51) Int. Cl.
`
`EN2.
`
`HO)4N 5/77
`HO)4N 5/765
`H04N 5/781
`HO4N 5/85
`
`3.08:
`
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`
`(56)
`
`References Cited
`
`2005, 0169532 A1* 8, 2005 Yamana et al. ............... 382,200
`ck
`
`388-365. A 338 XEA".O. S3
`
`7, 2006 Hsieh et al.
`2006, O164533 A1
`2006/0176951 A1* 8, 2006 Berman et al. .......... 375,240.01
`2006/0233236 A1* 10, 2006 Labrozzi et al. ........ 375,240.03
`SSA '838. SM. O'57,368
`ang et al. ..
`2008, 0231630 A1* 9, 2008 Shenkar et al. .
`... 345,419
`2009/O115901 A1* 5, 2009 Winter et al. ................. 348,565
`a
`
`U.S. PATENT DOCUMENTS
`
`OTHER PUBLICATIONS
`
`4,281,909 A *
`4,667,226 A
`5,221,964 A *
`6, 198,505 B1*
`6,788,338 B1
`6,829,391 B2*
`7,113,203 B1*
`7,738,008 B1*
`2003/O193571 A1*
`2004/O198101 A1*
`2004/0218099 A1*
`
`8, 1981
`5, 1987
`6, 1993
`3, 2001
`9, 2004
`12, 2004
`9, 2006
`6, 2010
`10, 2003
`10, 2004
`11, 2004
`
`Ishibashi et al. ............... 352/25
`Glenn
`Chamberlain et al. ....... 348/373
`Turner et al. .............. 348.222.1
`Dinev et al.
`Comaniciu et al. .......... 382.243
`Wu et al. .....
`348.20799
`Ball .............................. 348.159
`Schultz et al.
`348.20799
`Rapp ............
`... 439,701
`Washington .................. 348,571
`
`
`
`ARRI, ARRIFLEX D-20, 2006, D-20: p. 1-12.*
`Debra Kaufman, New Products & Services, Feb. 2004, Amerian
`Cinematographer, pp. 91-94.*
`ARRI, ARRIFLEX d-20, Sep. 1, 2004, ARRI, pp. 1-12.*
`EP08745798.2 Extended Search Report dated Sep. 14, 2012.
`Webpage: Silicon Imaging SI-1920HD MegaCamera, 2003.
`Webpage: Silicon Imaging and P+S Technik, 2012.
`Web document: Red One Camera, 2012.
`
`* cited by examiner
`
`APPLE v. RED.COM
`
`Page 2 of 29
`
`Apple Ex. 1005
`
`

`

`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 1 of 12
`
`US 9,565.419 B2
`
`82
`
`
`
`-}NHOS
`
`APPLE v. RED.COM
`
`Page 3 of 29
`
`Apple Ex. 1005
`
`

`

`U.S. Patent
`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 2 of 12
`
`US 9,565.419 B2
`US 9,565,419 B2
`
`70
`
`SOFTWARE
`
`72
`
`64
`
`In RECORD
`
`START/STOP
`
`SYNCWIRING
`
`66
`
`FRAME
`
`GRABBER &CONTROLLER
`SENSOR
`
`A
`
`UNIT
`
`
`
`
`
`
`
`28
`3052
`
` %
`
`8
`
`illlllllllllll
`OWN
`BOVflEiBiNi
`
`24
`
`54
`
`FIG.2
`
`89
`68
`
`SENSORS
`
`APPLE V. REDCOM
`
`Page 4 of 29
`
`Apple EX. 1005
`
`APPLE v. RED.COM
`
`Page 4 of 29
`
`Apple Ex. 1005
`
`

`

`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 3 of 12
`
`US 9,565.419 B2
`
`88
`
`89 I
`
`06
`
`Z8
`
`
`
`
`
`
`
`
`
`
`
`NOW SN
`
`IEEI
`
`89
`
`APPLE v. RED.COM
`
`Page 5 of 29
`
`Apple Ex. 1005
`
`

`

`U.S. Patent
`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 4 of 12
`
`US 9,565.419 B2
`US 9,565,419 B2
`
`.. >—
`:3 <1
`m .J
`1 Q~
`Q (I)
`I -
`C)
`
`*5
`g Q.
`4: i-
`Q: :3
`o
`
`00
`Q
`N
`
`
`
`= m
`12::
`<
`g
`P
`LL
`O
`‘0
`
`0
`N
`N
`
`N
`
`
`
`z
`9 o:
`P o
`a:
`< {/2
`Lu w
`9 03
`§ CO
`<{ CC 05 Z “J
`EE 38
`(3
`g E
`0
`0
`
`,4
`c6 0
`Q (I
`2 1*"
`>é
`w o
`
`\
`
`N
`co
`
`V‘
`
`L5
`LT:
`
`\o
`
`O N
`
`N
`m
`
`g
`N
`
`N
`
`NDH
`
`:93
`
`
`
`
`
`
`
`
`
`N
`
`.«J
`
`Q ”.1 5 go
`
`CE
`
`9, 1:.ZZ
`w :3
`00
`W
`
`I Egg“
`
`
`
`
`:3 3
`{IllIlllllllmllilllllllll
`05 O
`HIIHIIIHIIHIIIHIHIIHHI
`N :: 0 ‘0
`
`F" Z
`0 Lu
`HIIHIIlfllmlllllllillmll EU)
`
`C)
`Q
`
`N
`
`Eg
`
`APPLE V. REDCOM
`
`Page 6 of 29
`
`Apple EX. 1005
`
`APPLE v. RED.COM
`
`Page 6 of 29
`
`Apple Ex. 1005
`
`

`

`U.S. Patent
`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 5 of 12
`
`US 9,565.419 B2
`US 9,565,419 B2
`
`HD—SD?
`
`DESPLAY
`
`110
`
`SOFTWARE-m-
`
`106
`
`104
`
`
`
`REMOTJEZQSENSORANDFRAMEGRABBERCAMERAMODULE
`
`
`
`
`
`
`
`
`
`
`
`05
`(D
`QC
`
`01
`
`....
`
`0 §
`
`ill!Illilllllllllllllllfllll
`
`Wm
`
`mImmuunmmum
`
`«w»
`unulmmunnmmum
`
`NN
`
`32
`GRABBER &COMMUNICAWON
`PROCESSOR
`
`
`
`
`
`
`
`FRAME
`
`SYNC& CONTROL
`
`82
`
`FIG.5
`
`82
`
`837WOHLNOO
`
`808N318 8
`
`
`LINK,SYNC& CONTROL
`
`DATA
`
`
`
`
`(0
`CC
`
`
`
`00
`
`02L
`
`u
`(/3
`
`APPLE V. REDCOM
`
`Page 7 of 29
`
`Apple EX. 1005
`
`APPLE v. RED.COM
`
`Page 7 of 29
`
`Apple Ex. 1005
`
`

`

`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 6 of 12
`
`US 9,565.419 B2
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`0£ I
`
`APPLE v. RED.COM
`
`Page 8 of 29
`
`Apple Ex. 1005
`
`

`

`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 7 of 12
`
`US 9,565.419 B2
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`99,
`
`3OWN
`NON SN3
`a C2
`
`APPLE v. RED.COM
`
`Page 9 of 29
`
`Apple Ex. 1005
`
`

`

`OSU.
`
`m
`
`b.e
`
`7,
`
`2B
`
`
`
`
`
`
`
`MMzmwmowIODO...PSufi/”3wGzfiwmoomma<mm§<o$25500wdmOE
`
`e\QEQQQ
`
`FmmH
`
`
`0m.8vvmmNM.»zQEQzaESOomammmoomammafimI...I.n.”ewkw<oo<0mm
`
`
`7mmHEmaN:21‘$38JoEzouIL1I...05%.20.53mWm“2r~.I..l
`szwEm:mango:<mw2<oOzgooo_.fmo<m9w.555moSwa.waaz0w4m<>0§mm
`
`
`
`
`
`22mI"...11'...SmIa:...Mm
`Bmdffigg07:5Ozmmmwooma><.Em_omQOUmégmWm.h5255.532
`
`
`
`
`
`WMN.mm<efirn~0m423PnWNwI....“wzrmoimmkzémmn
`@232.memzmmzmwm3x$330:3m.3O..OWnO
`
`
`
`
`
`
`
`
`................................Nw.n_....xxxxxxxxxxxxx><I~Qm~0
`
`mo5%;
`
`
`
`emN.
`
`(29.52i\/5,mvN..5”,ozaém«.300a53:.Dwdx6504232
`
`
`
`
`zoEBZDmSa9szow.ozamOOmmxmozpmzMomcwwS0:ngU/l,
`.80;&
`
`
`
`.01;MEEUDEQEZ45395mm>63me
`
`$52.52
`
`mmmzmmzs’.IIIIIIIIIIIIfl
`
`APPLE V. REDCOM
`
`Page 10 of 29
`
`Apple EX. 1005
`
`APPLE v. RED.COM
`
`Page 10 of 29
`
`Apple Ex. 1005
`
`
`
`

`

`U.S. Patent
`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 9 of 12
`
`US 9,565.419 B2
`US 9,565,419 B2
`
`mm5:5;mm
`
`
`
`mbfiommbfiom
`
`mm
`
`mewhman
`
`ZOMh<N3<DmM>
`
`ozamoommwmm.
`
`vm\mmm
`
`mmvm
`
`vm
`
`Nam
`
`NN
`
`vmN
`
`
`
`
`
`ZOmk<N3<Dw~>.mOZEKOOMKxmoghmzOmOMKMRW
`
`
`
`
`
`
`
`
`
`
`
`
`
`ZOEwOQ<mm§<0Ommmhw
`
`manI
`
`
`
`JOthOUw0<hw
`
`wa
`
`OmmwwmOm.
`
`ZOK<N3<DQ>
`
`OZEmOmew
`
`Nmm
`
`0.DE
`
`APPLE V. REDCOM
`
`Page 11 of 29
`
`Apple EX. 1005
`
`APPLE v. RED.COM
`
`Page 11 of 29
`
`Apple Ex. 1005
`
`
`
`
`

`

`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 10 of 12
`
`US 9,565.419 B2
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`APPLE v. RED.COM
`
`Page 12 of 29
`
`Apple Ex. 1005
`
`

`

`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 11 of 12
`
`US 9,565.419 B2
`
`22
`
`
`
`
`
`y:-CAMERA & SEREO-3E SYSEM WITH NEWORKANB
`6O 3O BROACAS NFRASRCBRE
`2
`3.
`
`
`
`90
`
`&
`
`138
`
`88
`
`
`
`E.
`e
`O
`a sm
`o
`5 (2 :
`T
`g
`U
`2
`N.
`2
`s: E2 E
`
`E
`
`
`
`
`
`
`
`
`
`SENSOR
`N
`
`Es
`GRASSR
`&
`cCWMCA
`N
`RCCESSOR
`
`
`
`AA NK,
`SYNC 8.
`CORC.
`
`68
`
`82
`
`22
`
`Stereo Rig
`
`
`
`- BROADCAS
`FASR3CRE
`Y-------
`
`SENSOR
`
`
`
`CAT-5
`
`
`
`
`
`SENSOR
`
`68
`
`
`
`
`
`
`
`22
`
`
`
`
`
`RAE
`GRAB8Er
`&
`COMAN CON
`OCESSR
`
`X
`32
`
`1 Oa
`
`WOBE SEREO
`RECORDER
`
`
`
`
`
`
`
`
`
`Cof
`NEWORK
`
`
`
`EAAA ANASEr
`& COO GANG
`
`naging
`Mode
`
`
`
`NEWORKRCORENG, EO;NG
`AYSACK & WSAAN
`
`--S3
`SAY
`
`SEEO 3D
`OROA
`OSPAY
`
`APPLE v. RED.COM
`
`Page 13 of 29
`
`Apple Ex. 1005
`
`

`

`U.S. Patent
`
`Feb. 7, 2017
`
`Sheet 12 of 12
`
`US 9,565.419 B2
`
`SCONOVR SOF WARE DAGRAM
`
`FIG. 12
`
`4.08
`
`{NPU
`RECEVE
`
`444
`
`SORAGE
`
`READ FROM
`SORAGE
`
`442
`
`414
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`iDENTFY
`CAMERA (S)
`
`4O2
`
`/
`
`ADS
`SENGS &
`CA38 raCN
`
`OAD
`CABRAONS
`
`CARE
`MAGE &
`MEAAA
`
`MAGE
`CORRECTON &
`PACKNG
`
`RAM BUFFER
`
`426
`
`436
`
`434
`
`NEWORK
`
`RECEVE
`SREAM &
`CONTRO
`
`
`
`
`
`
`
`RECORD f
`SREAMNG
`
`ENCODNG, PACKING
`& BFERNG
`(eg, CNEFORM CODEC)
`43O
`
`
`
`
`
`
`
`
`
`428
`
`4.38
`
`WE
`REWEW
`
`PAYE3ACK
`DECODE
`
`
`
`APPLY
`WETADAA
`(eg. ROAS SPEEDGRADE)
`42O
`
`WRTE FLE &
`CONANER
`(AVI, MOV, SV)
`432
`
`SREAMNG
`
`44O
`
`DSPAY
`WSAZAON
`MAGE PROCESSENG
`422
`
`424
`
`SORAGE
`
`NETWORK
`
`O.JPU
`
`OUTPUT 2
`
`APPLE v. RED.COM
`
`Page 14 of 29
`
`Apple Ex. 1005
`
`

`

`US 9,565,419 B2
`
`1.
`DIGITAL CAMERA SYSTEM FOR
`RECORDING, EDITING AND VISUALIZING
`IMAGES
`
`CROSS-REFERENCE TO RELATED
`APPLICATION(S)
`The present application is a U.S. National Phase applica
`tion of and claims priority from PCT/US2008/060272, filed
`on Apr. 14, 2008 designating the United States, which in turn
`was based on and claimed priority from U.S. Provisional
`Patent Application No. 60/923,399 filed Apr. 13, 2007 both
`of which applications are incorporated herein by reference.
`
`FIELD OF THE INVENTION
`
`10
`
`15
`
`The present invention is related, in general, to a digital
`cinema camera system. More particularly, the present inven
`tion is related to a digital cinema camera system for record
`ing, editing and Visualizing images.
`
`BACKGROUND OF THE INVENTION
`
`2
`tools or metadata integration. These digital cinema cameras
`do not utilize existing broadcast infrastructure to transmit
`multi-resolution data and have complex workflows with
`respect to stereo 3D and multi-camera content acquisition,
`broadcast and network transmission either live or in a post
`production process. These digital cinema cameras are lim
`ited to 1920x1080 image sensor pixel arrays that require the
`use of a multiple sensor prism block which, in turn, requires
`use of complex and expensive optics. These digital cinema
`cameras utilize dedicated hardware functions with no or
`limited image processing flexibility or upgrade capability.
`Dedicated hardware functions utilized by these digital cin
`ema cameras include video processing to perform non
`reversible color space transformations or Sub-sampling to
`formats, such as YUV 4:2:2 and 4:4:4, as standard broadcast
`signals. These digital cinema cameras implement a variety
`of proprietary compression and coding schemes that intro
`duce visible image artifacts, especially when projected on
`large screens. While a number of these digital cinema
`cameras can generate preview imagery for display on an
`electronic viewfinder, these digital cinema cameras can only
`do so with limited resolution or visualization tools. High
`resolution outputs from these digital cinema cameras are
`restricted to transmission in SMPTE standard resolution and
`formats. These digital cinema cameras often output imagery
`to be record on restrictive, proprietary or large external
`storage devices. These storage devices include a tape storage
`system having only linear data access, Non-Volatile Flash or
`RAM drives with limited storage, and multiple hard disk
`drive RAID storage systems which are often non-portable
`and whose media cannot be easily removed or transported
`for direct use in other systems. Also, the files stored on these
`storage devices have limited color correction, image pro
`cessing or post production metadata integration.
`In recent years, many digital still cameras or dual-mode
`video and still camcorders have also been developed which
`use single image sensors with color filter arrays. These
`digital still cameras and camcorder devices do use higher
`resolution sensors (e.g., HD (1920x1080) camcorders, digi
`tal single-lens reflex camera (DSLR) are now 10 MP and
`higher). However, these digital still cameras and camcorders
`have slow readout architectures (e.g., a DSLR may only
`shoot four (4) frames per second) and can only achieve Video
`rate preview at low resolution (e.g., 640x480) or standard
`definition (e.g., VGA 640x480 at thirty (30) frames per
`second) using Sub-Sampling or windowing techniques.
`These digital still cameras and camcorders use dedicated
`hardware functions or targeted function digital signal pro
`cessors (DSP) to perform image processing to interpolate
`and colorize the raw image data from the image sensor.
`These digital still cameras and camcorders compress the
`colorized images for storage; but the compressing process
`performed by these devices prevents access to the original
`full raw image pixel data for later processing, analysis or
`colorization. In addition, the interpolation and color pro
`cessing applied to the source raw data in those devices
`initially generates data sets that are larger than the Source
`raw data which, in turn, requires the application of higher
`compression to fit the data sets into a target storage capacity.
`This typically results in a reduction in image quality com
`pared to the original image or a coded version of the raw
`data.
`A few single sensor cameras have been developed for use
`in 2K and 4K acquisitions in raw format. However, these
`cameras use dedicated hardware or targeted function DSPs
`to perform image processing to interpolate, colorize and
`display preview quality output and simultaneously compress
`
`25
`
`30
`
`35
`
`40
`
`For many years, film cameras were the only option for
`capturing cinema quality motion pictures. The time require
`ments and costs related to shooting and processing motion
`picture images on film stock and then transferring those
`images into a digital form have created a need for motion
`picture cameras that capture high definition or cinema
`resolution imagery directly in a digital form. The advent of
`Digital Cinemas, cost effective Stereo 3D Digital cinema
`projection systems and establishment of Digital Cinema
`Initiative SMPTE Standards has fueled the need for more
`content creation for delivery at 2K, 4K and Stereo formats.
`Accordingly, there is a need for a digital camera system
`that meets the needs described above that reduces costs.
`There is also a need for a digital camera system that
`leverages digital processing and visualization tools. There is
`a further need for a digital camera system that provides user
`feedback and metadata collection when shooting special
`effects, compositions and stereo or multi-camera content.
`There is an additional need for a digital camera system that
`improves flexibility in networked collaboration, enables
`separated imaging block and recording, has a simple work
`flow with metadata management and, importantly, maintains
`45
`film-like reproduction qualities and camera operation. There
`is also a need for a digital camera system that not only
`provides the capabilities described above but can also utilize
`existing cabling infrastructure, broadcast signal monitoring
`and transmission systems. There is a need for a digital
`camera system that mixes broadcast standard digital sources
`into a recording and visualization system, as well as generate
`broadcast standard and network Streaming outputs for 2D
`and 3D content.
`In the past few years, while several digital cinema cam
`55
`eras have emerged on the market these digital cinema
`cameras are complex designs with limited connectivity that
`are only able to address a limited set of the needs described
`above. For example, these digital cinema cameras are
`incompatible with existing cable infrastructure. Also, these
`60
`digital cinema cameras either completely lack network man
`agement or are capable of only minimal network manage
`ment (i.e., only simple controls that lack full image stream
`ing or metadata management). Further, these digital cinema
`cameras lack the ability to capture or record multi-sensor 2K
`or 4K image data using a single control application. Addi
`tionally, these digital cinema cameras lack visualization
`
`50
`
`65
`
`APPLE v. RED.COM
`
`Page 15 of 29
`
`Apple Ex. 1005
`
`

`

`US 9,565,419 B2
`
`10
`
`15
`
`25
`
`30
`
`35
`
`40
`
`3
`the raw sensor image data for later digital editing and
`grading. Also, the compression method and metadata
`employed by these cameras foreclose the dynamic retrieval
`of alternative resolution or representations at different bit
`rates during recording for network streaming, remote grad
`ing or adaptive editing. Due to their architectures, these
`single sensor cameras must apply high compression to fit
`data into target internal storage capacity devices. Also, due
`to their architectures, these single sensor cameras lack the
`ability to transmit the imager raw content over existing
`broadcast or network infrastructure cabling for remote moni
`toring, networking, recording or visualization. These single
`sensor cameras cannot process captured signals with prere
`corded content or broadcast format signals for live compo
`sition, Switching, grading, mixing into virtual sets or adding
`graphic overlays based on extracted metadata or analytics.
`These single sensor cameras also lack the ability to manage,
`control or record multi-sensor imagers, which may be
`remotely connected to a recording system.
`In recent years, there has been an interest in producing
`digital cinema quality 3D stereographic, wide-dynamic and
`immersive content using multiple imagers. This has created
`a need for more efficient modular and scalable cameras and
`workflow solutions. There is a further need for a digital
`camera system having a precise Synchronization mechanism
`to enable images to be mixed or stitched without motion
`artifacts. While digital camera systems have been used to
`produce this type of content, these camera systems suffer
`from the same system limitations as the cameras described
`above. These camera systems are mostly comprised of
`stand-alone cameras, each with individual controls, viewing
`and recording systems, with no integration mechanism other
`than a common sync signal (i.e., there is no communication
`between camera controls or viewing and recording settings).
`These camera systems are large and bulky Such that the
`camera systems cannot be placed very close together physi
`cally, as is required for short inter-ocular distances in 3D
`Stereographic or for creating hemispherical views where
`cameras need to be placed as close together as possible from
`a common center point. When shooting thru mirrors and
`beam splitters, rigs (i.e., a combination of digital cameras,
`optics and mounting platform) become more cumbersome
`and difficult to use in handheld shooting environments.
`Finally, these camera systems lack a comprehensive set of
`image processing, visualization, positioning control, record
`45
`ing, playback, communications and display tools for use in
`Such high-definition multi-camera systems.
`
`SUMMARY OF THE INVENTION
`
`The present invention as described herein discloses a
`digital camera system that captures scalable resolution,
`bit-depth and frame rate raw or color processed images from
`one or more modular imaging modules at precise film or
`Video rates, can utilize industry standard cabling infrastruc
`ture for transmitting either the raw sensor data or processed
`raw on the same or different links, provides a mechanism for
`timing synchronization of exposure and readout cycles from
`multiple imaging modules, uses a unified software or opera
`tor interface to control the capture, processing and non
`60
`destructive visualization from one or more imaging mod
`ules, can optionally combine the live imagery with
`previously stored imagery or computer generated virtual sets
`and simultaneously record the raw, broadcast format, or
`visualization processed imagery in its original or near origi
`nal representation. The processor can be used to compress
`the imagery with an encoder, which can generate multiple
`
`50
`
`55
`
`65
`
`4
`streams one for the purpose of recording at highest quality
`and optionally additional streams at lower data rates for
`remote transmission. It enables the recording of one or
`multiple image streams using a common removable storage
`device or across multiple devices for increased throughput.
`The recording can make use of a single file containing the
`streams from multiple imaging modules with metadata to
`enable the selective playback of one or more streams. The
`output processing can include mixing the imagery from the
`multiple streams for display on standard computer or broad
`cast monitoring devices or processed for output on special
`ized stereographic displays that require formatting and Syn
`chronization from dual image streams. Utilizing metadata
`encoded in the recorded stream or generated thru user input
`during playback the relative position, color transformation
`and format of the dual streams, representing the left and
`right eye content, can be adjusted to change the stereo
`graphic effect and depth perception on these displays.
`This invention enables reduced complexity for capturing
`imagery from one or more image modules, enables remote
`image sensing and frame grabbing with transmission using
`existing industry standard broadcast and networking infra
`structure, improves storage and processing efficiency, pro
`vides increased flexibility and tools visualization, network
`ing, analysis and mixing of prerecorded or computer
`generated data, and delivers unique display modes 2D and
`3D representation of the multiple streams during live,
`recording, playback or post processing. The disclosed digital
`camera system may include optics, one or more imaging
`modules, a frame grabber, a processor, software, user input
`mechanism, a display, synchronization mechanism, net
`working means and storage means. In addition, a configu
`ration is disclosed for a portable digital camera and record
`ing system capable of HD. 2K and 4K stereo-3D or wide
`dynamic multi-image acquisition using two image sensing
`modules and separated image processing, recording and
`display Subsystem.
`Other features and advantages of the present invention
`will become apparent from the following more detailed
`description, taken in conjunction with the accompanying
`drawings, which illustrate, by way of example, the prin
`ciples of the invention.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The accompanying drawings illustrate the invention.
`Throughout the drawings like reference numbers indicate
`like exemplary elements, components, or steps. In Such
`drawings:
`FIG. 1 is a block diagram of a digital camera system
`embodying the present invention;
`FIG. 2 is a diagram of an embodiment of the digital
`camera system;
`FIG. 3 is a diagram of an alternate embodiment of the
`digital camera system;
`FIG. 4 is a diagram of an additional embodiment of the
`digital camera system covering a remote sensor and frame
`grabber camera module;
`FIG. 5 is a diagram of another embodiment of the digital
`camera system covering a remote three-sensor and frame
`grabber camera module;
`FIG. 6 is a diagram of yet another embodiment of the
`digital camera system;
`FIG. 7 is a diagram of an alternate embodiment of the
`digital camera system;
`
`APPLE v. RED.COM
`
`Page 16 of 29
`
`Apple Ex. 1005
`
`

`

`US 9,565,419 B2
`
`6
`
`5
`
`optics, one or more imaging modules, a frame grabber, a
`FIG. 8 is a diagram of another embodiment of the digital
`processor, software, user input mechanism, a display, syn-
`camera system a mobile docking camera and processing
`chronization mechanism, networking means and storage
`system;
`FIG. 9 is a diagram of another embodiment of the digital
`means. In addition, a configuration is disclosed for a por-
`5 table digital camera and recording system capable of HD,
`camera system covering stereo 3D network recording and
`visualization;
`2K and 4K stereo-3D or wide-dynamic multi-image acqui-
`FIG. 10 is a diagram of another embodiment of the digital
`sition using two image sensing modules and separated image
`camera system covering a mobile stereo camera and record-
`processing, recording and display subsystem.
`ing system;
`An embodiment of the present invention in the form of a
`10 digital camera system 20
`FIG. 11 is a diagram of another embodiment of the digital
`is
`illustrated in FIG. 1 and
`camera system covering a multi-camera and stereo-3D sys-
`described below in order to provide an overview of the
`tem with network and broadcast infrastructure; and
`camera system 20 as well as various components of the
`FIG. 12 is a flow chart for SiliconDVR software associ-
`system 20 and their respective functions. The camera system
`ated with the present invention.
`15 20 includes an optical assembly 22 to gather light 24 from
`a desired scene 26. The system 20 also includes a modular
`DETAILED DESCRIPTION OF THE
`imaging subsystem 28 aligned with the optical assembly 22
`PREFERRED EMBODIMENTS
`to receive light 24 gathered and/or modified by the optical
`assembly 22. The modular imaging subsystem 28 comprises
`As shown in FIGS. 1-12 for purposes of illustration, the
`20 one or more imagers 30 and at least one frame grabber 32.
`present invention resides in a digital camera system that
`The imager 30 captures high definition raw images at film or
`captures scalable resolution, bit-depth and frame rate raw or
`video rates for HD, 2K and 4K, cinema quality production.
`color processed images from one or more modular imaging
`The imager 30 can come in various forms including, without
`modules at precise film or video rates. The present invention
`limitation, at least one pixilated image sensor unit having
`utilizes industry standard cabling infrastructure for trans-
`25 one or more arrays of pixels, a pickup tube, a semiconductor
`mitting either the raw sensor data or processed raw on the
`detector or the like. The pixilated image sensor unit can
`same or different links. The present invention provides a
`come in various forms including, without limitation, a
`mechanism for timing synchronization of exposure and
`complimentary metal-oxide semiconductor (CMOS) active-
`readout cycles from multiple imaging modules. The present
`pixel image sensor, a metal-oxide semiconductor (MOS)
`invention also provides a unified software or operator inter-
`30 active-pixel image sensor, a charge-coupled device (CCD),
`face to control the capture, processing and non-destructive
`a contact image sensor (CIS) or other pixilated detection
`visualization from one or more imaging modules. The
`devices. A single image sensor 30 may include color filters
`present invention can optionally combine live imagery with
`that are used to capture a representation of the full color
`previously stored imagery or computer generated virtual
`images.
`sets, while simultaneously recording the raw, broadcast
`The optical assembly 22 includes optics ( e.g., lenses). The
`format, or visualization processed imagery in its original or 35
`system 20 includes a lens mount (not shown) that intercon-
`near original representation. The present invention discloses
`nects the optical assembly 22 and the modular imaging
`a processor that can be used to compress the imagery with
`subsystem 28. The lens mount can come in various forms
`an encoder, which can generate multiple streams one for the
`including, without
`limitation, a fixed optical
`interface
`purpose ofrecording at highest quality and optionally addi-
`mount, an interchangeable lens optical interface mounting
`tional streams at lower data rates for remote transmission. 40
`system or the like. Thus, the lens mount can provide for film
`The present invention enables the recording of one or
`or video lenses to be removably connected to the modular
`multiple image streams using a common removable storage
`imaging subsystem 28. The interchangeable lens mount is a
`device or across multiple devices for increased throughput.
`precise mounting surface and locking mechanism, which
`The recording can make use of a single file containing the
`enables field exchange of the lens mount to support the use
`streams from multiple imaging modules with metadata to 45
`of a variety of industry standard lenses such as, PL, Nikon-F,
`enable the selective playback of one or more streams. The
`Panavision, Leica, C and Canon. An interchange lens mount
`output processing can include mixing the imagery from the
`with an integrated optic enables the use of B4 optics
`multiple streams for display on standard computer or broad-
`(originally designed for use with three-sensor prism cam-
`cast monitoring devices or processed for output on special-
`eras) on a single-sensor based camera unit. In the alternative,
`ized stereographic displays that require formatting and syn- 50
`the image sensor unit may have an integrated lens.
`chronization from dual image streams. Utilizing metadata
`The image sensor unit 30 includes a plurality of adjust-
`encoded in the recorded stream or generated thru user input
`ment mechanisms (not shown) to adjust the position of the
`during playback the relative position, color transformation
`image sensor unit relative to the optical center of lens
`and format of the dual streams, representing the left and
`projection and/or to adjust the co-planarity of a sensing plate
`right eye content, can be adjusted to change the stereo- 55
`(i.e., the surface which holds the sensor circuit board of the
`graphic effect and depth perception on these displays.
`image sensor) relative to the optical interface mount. The
`The present invention enables reduced complexity for
`image sensor unit 30 also includes a mechanism for back
`capturing imagery from one or more image modules,
`focus adjustment. Any of the adjustment mechanisms can
`enables remote image sensing and frame grabbing with
`include an electronic positioning device for remote opera-
`transmission using existing industry standard broadcast and 60
`tion for moving the sensor(s), optics, camera(s) or rig(s). In
`networking infrastructure, improves storage and processing
`the alternative, the image sensor unit 30 may be integrated
`efficiency, provides increased flexibility and tools visualiza-
`with an optical beam splitter or rotating shutter mechanism
`tion, networking, analysis and mixing of prerecorded or
`to enable the use of an optical viewfinder while continuing
`computer generated data, and delivers unique display modes
`to acquire imagery. In another alternative, an electronic
`2D and 3D representation of the multiple streams during 65
`display unit can be mounted into the optical beam splitter
`live, recording, playback or post processing. The present
`mechanism to enable selectable operation as an optical or
`invention discloses a digital camera system that includes
`
`APPLE v. RED.COM
`
`Page 17 of 29
`
`Apple Ex. 1005
`
`

`

`15
`
`7
`electronic viewfinder or as a combination optical viewfinder
`with virtual electronic image display.
`An optical device (not shown). Such as an RGB prism,
`may be positioned in front of the optical assembly 22 so that
`a plurality of pixilated sensor units 30 in the imaging
`Subsystem 28 capture color-separated channels. Alterna
`tively, a plurality of pixilated image sensor units 30 and
`beam splitting optics may also be used to capture a wide
`dynamic range representation of light 24 from the scene 26,
`where each pixilated image sensor unit 30 captures a range
`of Scene intensity (i.e., each sensor will have a bounded
`range of intensity that the sensor can accurately detect or
`measure based on the capacity of sensitivity and setting of
`the camera and its sensor (e.g., one cannot typically see
`details of both the Moon and the Sun in the same scene).
`Signals from each of the plurality of image sensor units 30
`can be processed and combined into a single image repre
`senting a wider range of scene intensity than can be accom
`plished with a single image sensor unit 30. Each image
`s