throbber
111111
`
`1111111111111111111111111111111111111111111111111111111111111111111111111111
`US 20110249073Al
`
`(19) United States
`c12) Patent Application Publication
`Cranfill et al.
`
`(10) Pub. No.: US 2011/0249073 A1
`Oct. 13, 2011
`(43) Pub. Date:
`
`(54) ESTABLISHING A VIDEO CONFERENCE
`DURING A PHONE CALL
`
`(76)
`
`Inventors:
`
`Elizabeth C. Cranfill, San
`Francisco, CA (US); Stephen 0.
`Lemay, San Francisco, CA (US);
`Joe S. Abuan, San Jose, CA (US);
`Hsi-Jung Wu, Sunnyvale, CA (US);
`Xiaosong Zhou, Campbell, CA
`(US); Roberto Garcia, JR.,
`Sunnyvale, CA (US)
`
`(21) Appl. No.:
`
`12/794,766
`
`(22) Filed:
`
`Jun.6,2010
`
`Related U.S. Application Data
`
`(60) Provisional application No. 61/321,871, filed on Apr.
`7, 2010.
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`H04N 7115
`
`(2006.01)
`
`(52) U.S. Cl. .............................. 348/14.02; 348/E07.078
`
`(57)
`
`ABSTRACT
`
`Some embodiments provide a method for initiating a video
`conference using a first mobile device. The method presents,
`during an audio call through a wireless communication net(cid:173)
`work with a second device, a selectable user-interface (UI)
`item on the first mobile device for switching from the audio
`call to the video conference. The method receives a selection
`of the selectable UI item. The method initiates the video
`conference without terminating the audio call. The method
`terminates the audio call before allowing the first and second
`devices to present audio and video data exchanged through
`the video conference.
`
`(1110)
`
`First Stage
`
`( 1115) Second Stage
`
`(1120)
`
`Third Stage
`
`1105
`
`1142
`1144 --~>+---+--
`
`1146
`
`1148
`
`1152
`
`1170
`
`1185
`
`1180
`
`1160
`
`GTL 1008
`PGR of U.S. Pat. No. 8,929,525
`
`0001
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 1 of 90
`
`US 2011/0249073 Al
`
`l()
`0
`N
`
`0
`........
`N
`
`I.()
`0
`"""'"
`
`0
`0
`N
`
`0
`0
`-r-
`
`0002
`
`

`

`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`Figure 3
`
`0
`\0
`0 .....
`N
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`1.,_---
`Encoder 1
`,
`"
`\ ___ 1
`
`....
`
`A
`
`Driver
`Encoder
`
`_j_j:J
`
`~ Video Processing Module
`
`325
`
`315
`
`\
`
`'4 lr
`
`j I'
`
`"'
`,
`
`-..
`.AI
`
`1~R-1
`(--.....
`
`Module
`Exchange
`
`Media
`
`,.
`..
`
`...
`...
`
`CIPU Driver
`
`"'
`~
`Images
`Camera
`
`\-:3
`1 CIPU
`..---,
`
`320
`
`310
`
`305
`
`300~
`
`0003
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`
`t Figure 4
`
`480
`....-
`
`CIPU Dtivcr
`
`405b
`
`405a
`
`Sensor
`Camera
`
`410b
`
`Array
`Pixel
`
`Array
`Pixel
`
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`(.H
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`475
`
`·-----
`
`Module
`Controller
`
`Auto Focus
`
`Auto Exposure
`
`465
`
`Sensor Module 1.-.
`
`445
`
`440
`
`Statistics Engine
`
`415
`
`<t
`~
`co
`
`~
`::.?
`ill
`
`417
`
`Buffer
`
`Line/Frame
`
`430
`
`\,. _________ _
`425
`
`420
`
`--.:-.:--
`
`----
`
`••• --
`
`'
`
`1
`
`485; :~{:,}:~~---
`
`470
`
`~
`
`400
`
`0004
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 4 of 90
`
`US 2011/0249073 Al
`
`.
`
`.
`
`..JII
`
`r
`
`fil""
`
`~
`
`"'
`c )-
`
`-t
`--
`fil"" :-
`
`~
`
`.
`c :::> !--"' iiJii: ::::
`
`..JII
`¥
`
`~ ..JII
`
`¥
`
`.
`~
`
`fil""
`
`..JII
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`:/
`
`0
`
`~
`I.()
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`y
`
`I.()
`.,...-
`I.()
`
`/
`
`I.()
`0
`I.()
`
`c :::> H iiJii: ::::
`
`~
`
`.
`c J ~ ::--
`
`..JII
`¥
`
`~
`
`..JII
`
`.
`.
`c J ~ --::--
`
`0
`N
`I.()
`
`I.()
`I.()
`I.()
`
`0
`I.()
`I.()
`
`0
`C'?
`I.()
`
`I.()
`N
`I.()
`
`0005
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 5 of 90
`
`US 2011/0249073 Al
`
`L{)
`co
`(.()
`
`0 co
`
`(.()
`
`L{)
`1'--
`(.()
`
`0
`1'--
`(.()
`
`L{)
`(.()
`(.()
`
`0
`(.()
`(.()
`
`L{)
`L{)
`(.()
`
`0
`L{)
`(.()
`
`L{)
`
`(.()
`
`"""
`
`0
`(.()
`
`"""
`
`L{)
`C")
`(.()
`
`0
`C")
`(.()
`
`L{)
`N
`(.()
`
`\
`\
`\
`\
`\
`\
`\
`\
`\
`\
`
`I
`I
`I
`I
`I
`
`/
`
`L{)
`0
`(.()
`
`/
`
`0 ...--
`
`(.()
`
`Y'
`
`L{)
`...-
`
`(.()
`
`0
`N
`(.()
`
`0006
`
`

`

`0
`\0
`0 .....
`0\
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Driver
`CIPU
`
`llo
`
`I
`I
`I
`I
`I
`
`Pipeline
`
`Backend Processing
`
`Storage
`
`I
`
`730 ---M Controller Module
`
`735
`
`725
`
`Frontend Processing~ 715
`
`~ Pipeline
`
`1
`
`.
`
`700
`
`/
`
`----------------------------------------------\
`
`Captured Image Processing Unit (CIPU)
`
`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`---
`
`--
`
`---
`
`----
`
`Figure 7
`---
`
`I '-----------------
`
`I
`I
`I
`1 ..
`
`720
`Pipeline ~
`
`Frontend Processing
`
`t
`
`-----
`
`) ... r
`
`t
`
`l
`
`I
`I
`I
`I
`
`405b
`
`Ca~~~o~~ I
`
`Array
`
`Sensor
`
`0007
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`-....l
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Figure 8
`
`820
`
`j
`Driver
`CIPU I
`
`480
`
`t I
`
`I
`I
`I
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`I
`I
`I
`
`800
`
`/
`
`Processing Pipeline
`
`Processing Pipeline
`
`_l
`
`I
`
`.....
`
`_...
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`I
`I
`I
`
`Array
`Pixel
`
`410b
`
`1
`
`Array
`Pixel
`
`j_
`
`Sensor
`Camera
`
`405b
`
`I
`
`-"'
`
`...
`
`Sensor
`Camera
`
`815
`
`~
`
`800
`
`0008
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`QO
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`950
`
`I
`
`• 1-
`
`960
`
`920
`
`Driver
`Encoder
`
`Interface
`Network
`
`• •I
`
`Video Conference Client
`\
`
`Figure 9
`
`945\
`
`+
`
`11111111
`
`Network Layer
`
`Managen1ent Layer
`
`Image Processing Layer
`
`910 •
`
`940
`
`935
`
`930
`
`~:E
`
`....... >
`0 u
`~ q
`
`-cl
`(j)
`0
`
`~
`(j)
`q
`u
`(j)
`
`.§
`........
`(j)
`
`925
`
`I 7
`
`905
`
`955
`
`--)
`ITNRI
`Module,._-
`Exchange
`
`Media
`
`...........
`
`CIPU Driver
`
`•
`Images
`Camera
`
`1 CIPU 1i~C
`r __ ,
`
`--\
`
`~
`
`900
`
`0009
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`\0
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Figure 10
`
`J
`
`I
`
`7. Video conference establishment
`
`I
`
`I
`
`5. Video Conference Acceptance
`
`6. Video1 Conference Acceptance
`
`conference
`accept video
`4. Request to
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`3. Video Conference R~quest
`
`2. Vid~o Conference Request
`
`1025
`
`Client -
`Conference
`
`Video
`
`Device 1020
`
`Server
`
`Conference
`[" Video
`
`1015
`
`video conference
`1. Request to start
`
`Client
`
`Conference
`
`Video
`
`Device 1005
`
`1010
`
`~
`
`1000
`
`0010
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`0
`....
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`l,~:r!i~j
`
`I~·. (?::--I
`'il
`\
`'
`:
`rr'
`\
`
`~._,.:'.
`
`' t<:
`
`1132
`
`1155
`
`Figure 11
`
`1180
`
`1185
`
`·.·.·.·.·.·.·.·.·.·.:.·.:.·.·.·.;.;.;.;.;.;.;.~
`
`Fifth Stage
`
`( 1130)
`
`Fourth Stage
`
`(1125}
`
`1146
`
`1144 _ ___tg_j-:.-j-l~ I
`1142---~
`
`I
`
`I
`
`Third Stage
`
`(1120)
`
`( 1115) Second Stage
`
`FirstStage
`
`(1110)
`
`1105
`
`0011
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`....
`....
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Figure 12
`
`End~
`
`(
`
`1280
`
`~
`
`1246
`
`Ill=~
`
`Video Conference
`
`Invitation
`
`Invitation
`
`Video Conference
`
`John Smith
`
`Third Stage
`
`(1220)
`
`Second Stage
`
`(1215)
`
`First Stage
`
`(121 0)
`
`1205
`
`1200
`
`0012
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`N
`....
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Figure 13
`
`1155
`
`1280
`
`~
`
`John Smith
`
`(1394) ThirdStage
`
`(1392) Second Stage
`
`First Stage
`
`(1390)
`
`1200
`
`0013
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`0
`\0
`0 .....
`....
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`(.H
`
`('D
`('D
`
`·!'~··, U
`
`' ·
`
`-
`
`'1:~ 1155 rll'
`
`n
`
`1480
`
`1447
`
`Jason
`
`Info
`
`Info
`
`Stage
`(1425) Fourth
`
`(1420) Stage
`Third
`
`1445
`
`1400
`
`0014
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 14 of 90
`
`US 2011/0249073 Al
`
`1500
`
`"\.
`
`Start
`
`1505
`
`Set bit rate at an initial bit rate
`
`Identify a set of network condition
`___
`1510
`--..... parameters received from the other
`device
`
`1535
`
`No
`
`Yes
`
`Defined
`amount of time
`elapsed?
`
`1530
`
`Increase bit rate by a defined
`amount
`
`1515
`
`1520
`
`Yes
`Has one-way
`atency deteriorated past>-----,
`a threshold
`amount?
`
`No
`
`Current bit
`rate has reached
`the bandwidth
`estimation
`bit rate?
`
`No
`
`Yes
`
`1525
`
`No
`
`Current
`bit rate has reached
`a defined maximum
`bit rate?
`
`Yes
`
`End
`
`Figure 15
`
`0015
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 15 of 90
`
`US 2011/0249073 Al
`
`1600
`
`'\.
`
`1665 ---
`
`Client Application
`
`16\
`
`Video Conference Module
`
`I 1604
`Video
`Conference
`Manager
`
`-
`
`r:::
`
`L 1606
`Image
`r:::
`~
`lBufferJ Processing
`Manager
`
`~
`Buffer j
`I 1608
`
`~ ..L1612
`
`I I 1614
`
`Networking
`Manager
`
`(
`
`LL1616
`
`(
`
`L 1618
`l Bufferj
`~
`l Buffer
`
`l r-- ~
`l BufferJ
`
`I
`I
`
`i-
`
`I
`
`1610
`
`~
`Bufferj
`--- -!------- ------
`1622
`Camera
`Video
`Source
`Compressor
`Module
`Module
`
`-/-1§?4__
`
`-;1~~6
`Decompressor r-:-
`
`-----
`
`Video
`
`I
`I
`
`Module
`
`I
`I
`"\ 1625:
`I
`Media Exchange Module
`I
`I
`------ -------------- -------------- -----~
`I
`1620
`
`CIPU Driver
`
`J
`"" 1630
`--- L-.---
`r
`I
`CIPU
`:
`1
`--~----)
`1650
`
`[ Encoder Driver
`
`Decoder Driver
`
`"" 1635
`--- ----
`Encoder
`
`r
`1
`I
`~~----
`1655
`
`I
`1
`J
`
`"" 1640
`--- ~---
`r
`1 Decoder
`I
`-~----
`1660
`
`I
`1
`J
`
`-- --
`(
`: Display
`
`'I
`:
`
`~~--~
`1645
`
`Figure 16
`
`0016
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 16 of 90
`
`US 2011/0249073 Al
`
`1600
`/
`
`Client Application
`
`/
`
`/ 1665
`
`1604
`
`1602---.._
`.... ,
`
`,.._,_\-:-!:--:----..Video Conference Module
`Video
`Conference
`Manager
`
`I I 1614
`
`Networking
`Manager
`
`..__ __,..
`
`; 1608
`
`Image
`Processing
`Manager
`
`I
`
`Media Exchange Module
`
`1620
`
`_f""igure 17
`
`l BufferJ
`
`I
`1705
`
`l BufferJ
`
`1710
`
`Audio Processing
`Manager
`
`-
`
`""' 1715
`
`[ Audio Driver
`
`]
`
`.J __ _ _ __ L _ 1720
`
`1725
`
`7
`
`I
`
`\ ~~\
`I ---
`<::;;::;' "
`1730
`
`0017
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 17 of 90
`
`US 2011/0249073 Al
`
`1800
`
`~
`
`Receive instructions to start a
`video conference
`
`1805
`
`Initialize a module that
`interacts with the video
`conference manager
`
`1810
`
`Yes
`
`1815
`
`Generate composite image for
`display
`
`1820
`
`I+ - - - - - - - ,
`
`1825
`
`No
`
`1840
`
`setting
`change?
`
`1835
`
`1845
`
`No
`
`Figure 18
`
`0018
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`QO
`....
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Figure 19
`
`1920 I
`
`1915 I
`
`Module
`History
`Motion
`
`Q) z
`~
`~ ~
`~
`0
`c
`I
`1li
`0
`C:>
`
`~
`I
`_......,
`
`Module
`
`Averaging
`
`Pixel
`
`0..
`....
`0
`(.)
`Q)
`(/)
`(/)
`Q)
`"'0
`E
`ro
`Ol
`Q)
`
`.__,
`ci
`~
`I
`_......,
`
`1910
`
`I
`
`Module
`
`Difference
`
`0
`&
`~ ~
`c
`(.)
`Q)
`E
`ro
`Ol
`Q)
`
`(2'
`
`Output Motion History
`
`TNR Manager
`
`Output Image
`
`1905
`
`Temporal Noise Reduction (TNR) Module
`
`R ~eference Image (R)
`
`nput Image (I)
`
`0 :s
`c
`I
`1li
`0
`....
`>.
`~
`I
`
`~
`
`\
`
`1900
`
`0019
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 19 of 90
`
`US 2011/0249073 Al
`
`2000
`
`\
`
`2015
`
`Receive an input image, reference image, and motion
`history
`
`2005
`
`Average pixels of
`input image with
`pixels of reference
`image
`
`2010
`
`2020
`
`Select a pixel in the input
`image
`
`No
`
`Is pixel's
`probability of motion
`below a particular
`threshold?
`
`2025
`
`2030
`
`Average pixel of input image
`with pixel of reference image
`
`2035
`
`Yes
`
`Any
`unselected
`pixels left in the
`input image?
`
`No
`
`2040
`
`Update motion history
`
`Figure 20
`
`End
`
`0020
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 20 of 90
`
`US 2011/0249073 Al
`
`2100
`
`~
`
`Start
`
`~
`
`Retrieve image from buffer
`
`v 2105
`
`~
`Perform exposure adjustment v 2110
`
`+
`
`Perform focus adjustment
`
`+
`
`Perform image resizing
`
`v 2115
`
`v 2120
`
`~
`Perform perspective correction v 2125
`+
`
`Adjust dynamic range
`
`v 2130
`
`+
`Determine rate controller parameters v 2135
`~
`
`to encode image
`
`End
`
`Figure 21
`
`0021
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 21 of 90
`
`US 2011/0249073 Al
`
`0
`-.;t
`C\1
`C\1
`
`(J)
`
`0) cu
`U5
`"0
`.....
`...c
`f-
`
`,..........
`0
`C\1
`C\1
`~
`
`(J)
`
`0) cu
`U5
`"0 c
`0
`()
`(J)
`(/)
`
`(J)
`
`0) cu
`U5
`......
`(f)
`.....
`LL
`
`-0 ...-
`
`C\1
`~
`
`0
`0
`C\1
`C\1
`
`I.{')
`0
`C\1
`C\1
`
`"' "' e ::s
`.tn
`u:
`
`0022
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`N
`N
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`2340
`
`,....
`....
`
`1155
`
`,....
`....
`2325
`
`Figure 23
`
`2330
`
`2305
`
`Third Stage
`
`(2320)
`
`(2315) Second Stage
`
`First Stage
`
`(231 0)
`
`2300
`
`0023
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 23 of 90
`
`US 2011/0249073 Al
`
`2400
`
`~
`
`Start
`
`~
`
`Receive data from accelerometer sensor
`
`~
`
`Receive data from gyroscope sensor
`
`v 2405
`
`v 2410
`
`~
`Determine an amount of perspective correction to perform v
`~
`
`based on data from accelerometer and gyroscope sensors
`
`Receive image captured by a camera
`
`2415
`
`v 2420
`
`~
`Modify image based on determined amount of perspective v
`~
`
`correction
`
`2425
`
`End
`
`Figure 24
`
`0024
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 24 of 90
`
`US 2011/0249073 Al
`
`0
`1'-
`L()
`C'\1
`
`Ol
`c
`
`Q) Q) --
`
`(f)
`(f)
`
`.~ 0
`( . ) -
`(.)
`(.)
`0
`:::l Q) Q)
`s..."O
`0.. 0 Q..s...
`(f)
`.._
`Q)~ .._ 0
`a>O
`Ol
`0..
`co
`E
`
`tl")
`~
`~
`;...
`::
`-~
`~
`
`.,....
`L()
`C'\1
`
`L()
`C'\1
`L()
`C'\1
`
`0
`C'\1
`L()
`C'\1
`
`0 .,....
`L()
`C'\1
`
`L()
`
`.....
`L()
`C'\1
`
`Ol c
`
`(f)
`(f)
`Q) Q)
`( . ) -
`0
`:::l
`s..."O
`0.. 0
`Q)~
`Ol co
`E
`
`L()
`<0
`L()
`C'\1
`
`L()
`1'-
`L()
`C'\1
`
`L()
`L()
`L()
`C'\1
`
`0
`c.o
`L()
`C'\1
`
`I
`
`L()
`0
`L()
`C'\1
`
`I
`
`0
`L()
`L()
`C'\1
`
`0025
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`Figure 26
`
`0
`\0
`0 .....
`Ul
`N
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Encoder
`
`2650
`
`Controller
`
`Rate
`
`2645
`
`2640
`
`2635
`
`2620
`
`I
`I
`
`Bitstream Manager
`
`Image Resizer
`
`--~------
`2610
`
`Processing Layer
`
`(
`
`2605
`
`Encoder Driver
`
`---------------------..
`
`2600 ~------------------------------------
`
`,.---
`
`0026
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 26 of 90
`
`US 2011/0249073 Al
`
`2700
`
`\
`
`Start
`
`+
`
`Receive image captured by a camera
`
`+
`
`Resize image
`
`+
`
`Send resized image to encoder
`
`v 2705
`
`v 2710
`
`v 2715
`
`manager
`
`+
`Send data indicating size of resized image to bitstream v 2720
`+
`Receive bitstream of encoded image from encoder v 2725
`+
`Insert data indicating size of resized image into bitstream v 2730
`+
`
`End
`
`Figure 27
`
`0027
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 27 of 90
`
`US 2011/0249073 Al
`
`,_
`a.>
`"0
`0 u
`a.>
`0
`
`'-a.>
`.!:::!
`(/)
`a.>
`0:::
`a.>
`0>
`Cll
`E
`
`.....
`a.>
`0>
`Cll
`c:
`Cll
`~
`E
`Cll
`a.> ,_
`......
`
`(/) -i:i5
`
`0
`C')
`<X)
`N
`
`lO
`......
`<X)
`N
`
`lO
`N
`<X)
`N
`
`0
`'<:!"
`<X)
`N
`
`lO
`'<:!"
`<X)
`N
`
`(/)
`
`lO
`0
`<X)
`
`I <D
`>.
`Cll
`N~~
`lii1 u 12
`10..
`I
`
`,_
`a.> >
`·;::
`0
`0>
`c:
`"0
`0 u
`a.>
`0
`
`I
`~----------------------------------
`
`0
`0
`<X)
`N
`
`0028
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 28 of 90
`
`US 2011/0249073 Al
`
`2900
`'\.
`
`Start
`
`~
`
`Receive bitstream of an encoded image
`
`~
`
`Extract data indicating size of image from bitstream
`
`+
`Send size data to image resizer
`
`~
`
`Send bitstream to decoder
`
`~
`
`Receive decoded image from decoder
`
`~
`
`Extract sub-image from decoded image based on size data
`
`~
`
`End
`
`2905
`
`2910
`
`2915
`
`2920
`
`2925
`
`2930
`
`Figure 29
`
`0029
`
`

`

`0
`\0
`0 .....
`\0
`N
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Rate Controller 2
`
`~ Buffer ~
`
`3015
`
`----
`-...._
`
`1'---
`,----
`
`Rate Controller 1
`
`3010
`
`Encoder Driver
`
`3070
`
`3060
`
`~
`
`3000
`
`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`Figure 30
`
`Video Encode,\__ 3032
`
`3030 ~
`
`Quantizer
`
`3040
`
`~304
`
`~3090
`
`022 ~ 3080~
`
`"'
`
`3020
`
`0030
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`Figure 31
`
`To Decoding Buffer
`
`To Video Conference
`
`Manager
`
`....
`(') ....
`0
`0 =
`....
`~ .....
`0" -....
`.... 0 = '"= =
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`(')
`
`~ .....
`
`(')
`
`0
`\0
`0 .....
`0
`(.H
`.....
`rFJ =- ('D
`....
`....
`
`0
`N
`
`('D
`
`~
`
`(.H
`
`Network
`
`Module
`Receiver
`
`~
`
`Manager
`
`Media Transport
`
`Manager
`Protocol
`Transport
`Virtual
`
`3125
`
`3120
`
`/
`
`Buffer
`
`Transmission
`
`Universal
`
`A
`
`Module
`
`Transmitter
`
`3115
`
`/
`
`r
`
`3122
`
`r
`
`Universal v
`
`Buffer Manager
`Transmission
`
`"-.,~
`
`Manager
`Protocol
`
`3105
`Manager
`
`Session Negotiating
`
`Networking Manager
`
`3130
`
`\
`
`\
`
`3135
`
`\
`
`\
`
`"'-3110
`
`From Encoded Video Buffer
`
`~
`
`3100
`
`0031
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`~
`
`Figure 32
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`0
`\0
`0 .....
`....
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`(.H
`
`('D
`('D
`
`3265
`
`Third Stage
`
`(3220)
`
`Second Stage
`
`(3215)
`
`First Stage
`
`(3210)
`
`3240
`
`3205
`
`3200
`
`0032
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`N
`(.H
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`~
`
`3335
`
`~
`
`3330
`
`Second Stage
`
`First Stage
`
`Figure 33
`
`~:~:~
`~
`
`:~
`
`'
`; .. \\..,....
`.\ .. ~)
`!:i./1
`
`\
`
`3250
`
`Second Stage
`
`First Stage
`
`3300
`
`0033
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`
`(.H
`(.H
`
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`3475
`
`+-~ /--~
`
`3465
`
`Figure 34
`
`-.
`
`Sixth Stage
`
`(3435)
`
`Fifth Stage
`
`(3430)
`
`Fourth Stage
`
`(3425)
`
`Third Stage
`
`(3420)
`
`Second Stage
`
`First Stage
`
`(3410)
`
`1105
`
`3400
`
`0034
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`
`.j;o.
`(.H
`
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`3567
`
`Figure 35
`
`......
`
`Sixth Stage
`
`(3535)
`
`Fifth Stage
`
`(3530)
`
`1155
`
`Third Stage
`
`(3420)
`
`Second Stage
`
`First Stage
`
`(3410)
`
`0035
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`Ul
`(.H
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Sixth Stage
`
`(3690)
`
`_..
`
`Fifth Stage
`
`(3685)
`
`.. !.!.!.!.!.~. -
`
`3567
`
`_..
`
`Fourih Stage
`
`3565
`
`Third Stage
`
`(3650)
`
`Second Stage
`
`First Stage
`
`(3640)
`
`1105
`
`3500
`
`0036
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 36 of 90
`
`US 2011/0249073 Al
`
`3500
`
`(3705) First Stage
`
`1105
`\
`
`(3710) Second Stage
`
`1170
`
`Local
`Rotate
`90'
`
`Remote t
`
`Rotate
`90'
`
`(3720) Fourth Stage
`
`l Remote
`
`Rotate
`90°
`
`(3715) Third Stage
`
`Local
`Rotate
`90'
`
`Figure 37
`
`0037
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`-....l
`(.H
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Figure 38
`
`3845
`
`3840 -- Jill""""
`
`3800
`
`(3820)
`
`Stage
`Fourth
`
`(3815)
`
`Stage
`Third
`
`(3810)
`
`Stage
`Second
`
`(3805)
`
`Stage
`First
`
`3825
`
`0038
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`Figure 39
`
`0
`\0
`0 .....
`QO
`(.H
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`(3920)
`
`Stage
`Fourth
`
`(3915)
`
`Stage
`Third
`
`(391 0)
`
`Stage
`Second
`
`(3805)
`
`Stage
`First
`
`3925
`
`0039
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`\0
`(.H
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Figure 40
`
`ll""'""
`
`-I
`
`;1
`
`,
`
`3865
`
`(4020)
`
`Stage
`Fourth
`
`(4015)
`
`Stage
`Third
`
`(4010)
`
`Stage
`Second
`
`(4005)
`
`4025
`
`Stage
`First
`
`3800
`
`0040
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`P,igure 41
`
`0
`\0
`0 .....
`0
`.j;o.
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Stage (4120)
`Fourth
`
`Stage (4115}
`Third
`
`Stage (411 0)
`Second
`
`Stage (3805)
`First
`
`41
`
`0041
`
`

`

`> ....
`
`(,H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`....
`.....
`rFJ =- ('D
`....
`0 ....
`
`.j;o.
`
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(,H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Figure42
`
`4260
`1155
`
`.. 1n~J ~I '
`
`i
`
`4275
`......
`
`4250
`
`4235 JP:;;D .·,.
`·Jr~
`
`II~P-D '"'ll 4230
`
`.
`
`'
`
`4270'
`
`·U
`i · · ·. · ~II
`c: • '
`
`·' .
`
`:
`
`(4220) St::~oe
`
`(4215)
`
`Second
`
`(421 0) Stage
`
`(4205)
`
`4225
`
`0042
`
`

`

`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`Figure 43
`
`0
`\0
`0 .....
`N
`.j;o.
`.....
`rFJ =- ('D
`....
`0 ....
`
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`(4325)
`
`Fifth Stage
`
`4365
`
`Fourth Stage
`
`(4205) =
`
`First Stage
`
`4200
`
`4225
`
`0043
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`
`(.H
`.j;o.
`
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Figure 44
`
`4446
`
`4450
`
`4260
`
`(4420)
`
`Stage
`Fourth
`
`(4415)
`
`Stage
`Third
`
`(4410)
`
`Stage
`Second
`
`(4405)
`
`4425
`
`Stage
`First
`
`0044
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 44 of 90
`
`US 2011/0249073 Al
`
`4500
`"'.
`
`4505
`
`4510
`
`Start
`
`Start video conference
`
`Send images from one
`camera to other device
`
`Yes
`
`End
`
`conference?
`
`No
`
`camera?
`
`4525
`
`Notify other device of switch
`camera
`
`4530
`
`4535
`
`Switch cameras
`
`Perform switch camera
`animation
`
`Figure 45
`
`0045
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`Ul
`.j;o.
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Figure 46
`
`..... ...
`
`4688
`
`,...
`.....
`
`~'-'-'-'-'-'-'-'-'-'-'-'-'-'-'-'-'-':"::>:::'":
`
`~;:,_,:,_,:,_,.:,:.~.~'~'-'-'-'-'-'-'-'-'-'-'-'-'-'-'-'~~~
`
`Stage
`Eighth
`
`(4645)
`
`Stage
`Seventh
`
`(4640)
`
`Stage
`Sixth
`
`(4635)
`
`Stage
`Fifth
`
`(4630)
`
`1132
`
`1160
`
`1170
`
`Fourth
`
`{4625)
`
`{4610)
`
`4600
`
`0046
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`0\
`.j;o.
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Figure 47
`
`4688
`
`4686
`
`..
`
`Stage
`Sixth
`
`(4645)
`
`Stage
`Fifth
`
`(4640)
`
`Stage
`Fourth
`
`(4635)
`
`1285
`
`1160
`
`1170
`
`(4630) Third Stage
`
`(4610)
`
`4600
`
`0047
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`-....l
`.j;o.
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`....
`....
`
`Figure 48
`
`4870
`
`4855~~
`
`Third Stage
`
`(4820)
`
`(4815) Second Stage
`
`First Stage
`
`\
`4805
`
`4800
`
`0048
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`QO
`.j;o.
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Figure 49
`
`4870
`
`....
`.....
`
`1285
`
`\
`4805
`
`4800
`
`Third Stage
`
`(4820)
`
`(4815) Second Stage
`
`First Stage
`
`0049
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 49 of 90
`
`US 2011/0249073 Al
`
`5000'\..
`
`5005
`
`Start video conference
`
`5010 '-----"~+=======-----.
`
`Receive images from one
`camera of the other device
`
`End
`
`Yes
`
`5020
`No
`
`5025
`
`Send switch camera request
`to other camera
`
`Receive images from one
`camera of the other device
`
`Figure 50
`
`Yes
`
`No
`
`Receive images from other
`camera of the other device
`
`5055
`
`Perform switch camera
`animation
`
`0050
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`0
`Ul
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Figure 51
`
`_.......
`
`~
`
`Stage
`Eighth
`
`(5145)
`
`Stage
`Seventh
`
`(5140)
`
`(5135) Sixth Stage
`
`Fifth
`
`(5130)
`
`5150
`
`Stage
`Fourth
`
`(5125)
`
`Stage
`Third
`
`(5120)
`
`Stage
`(5115) Second
`
`Stage
`First
`
`(511 0)
`
`5100
`
`0051
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`....
`Ul
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`Figure 52
`
`li
`
`l
`
`I
`
`-..H.,
`
`~
`Stage
`Sixth
`
`~
`
`"/;/'//////:-,
`
`~
`
`(5145)
`
`Stage
`Fifth
`
`:;-;-:-:•:•:•:•:•:•:•:•:•:•:•:•:•:•:•:•:•:•:•:•:•:•:•:•:•:•:•:•:•
`
`(5140)
`
`(5135) Fourth
`
`5282
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Third Stage
`
`(5130)
`
`Stage
`Second
`
`(5290)
`
`(511 0)
`
`5100
`
`0052
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`N
`Ul
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Figure 53
`
`II
`
`1
`
`.. ,.
`
`........~~h .·
`
`........
`
`........
`
`Stage
`Eighth
`
`(5145)
`
`Stage
`Seventh
`
`(5140)
`
`(5135) Sixth Stage
`
`Stage
`Fifth
`
`(5130)
`
`(5125)
`
`Third
`
`(5120)
`
`(511 0)
`
`5100
`
`0053
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`(.H
`Ul
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Figure 54
`
`II""
`...._
`
`II""
`...._
`
`{5435) Sixth Stage
`
`Fifth Stage
`
`(5430)
`
`Fourth Stage
`
`(5425)
`
`5440
`
`II""
`...._
`
`5470
`
`5445
`
`5465
`
`540
`
`Third Stage
`
`(5420)
`
`Second Stage
`
`(5415)
`
`First Stage
`
`(5410)
`
`5400
`
`0054
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 54 of 90
`
`US 2011/0249073 Al
`
`5500
`
`~
`
`Start
`
`5505
`
`Start video conference
`
`5510
`
`Receive video captured from
`one camera of the other device
`
`Yes
`
`End
`
`conference?
`
`5515
`
`5520
`
`No
`
`5525
`
`Send command to other
`device
`
`Figure 55
`
`0055
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 55 of 90
`
`US 2011/0249073 Al
`
`-0
`
`N
`<0
`L(')
`.._.-
`
`t
`
`L(')
`o;t
`CD
`l.()
`
`L(')
`N
`CD
`L(')
`
`L(')
`L(')
`.,.-
`..-
`
`Q.)
`0)
`
`(1j -(})
`
`-a
`c
`0
`t)
`Q.)
`(})
`
`-L(')
`
`.,.-
`CD
`...__,
`L(')
`
`Q.)
`
`0) co -(j) -(f)
`
`.....
`LL
`
`0
`0
`<0
`L(')
`
`L(')
`M
`<0
`L(')
`
`L(')
`0
`CD
`L(')
`
`<o
`
`l.t) e
`::1
`u:
`.t:D
`
`0056
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`0\
`Ul
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`....
`.....
`
`Fifth Stage
`
`(5730)
`
`Fourth Stage
`
`(5725)
`
`Figure 57
`
`5760
`
`5770
`
`......
`.....
`
`5750
`
`Third Stage
`
`(5720)
`
`(5715) Second Stage
`
`First Stage
`
`(5710)
`
`5700
`
`0057
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`0
`\0
`0 .....
`-....l
`Ul
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Fifth Stage
`
`(5830)
`
`Third Stage
`
`(5820)
`
`Figure 58
`
`5845
`
`5805
`
`0058
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 58 of 90
`
`US 2011/0249073 Al
`
`5900
`
`~
`
`Receive image captured by a camera
`
`v 5905
`
`Read pixel values in a defined region in the image v 5910
`
`Calculate average of pixel values
`
`v 5915
`
`Yes
`
`Average
`of pixel values
`equal to a particular
`defined value?
`
`5920
`
`No
`
`Adjust exposure value based on calculated average v 5925
`
`End
`
`Figure 59
`
`0059
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 59 of 90
`
`US 2011/0249073 Al
`
`6000 '\..
`
`6020
`
`6010 '\..
`
`6020
`
`6015 '\..
`
`6020
`
`6025
`
`6030
`
`6040
`
`6035
`.......
`
`6045
`
`Figure 60
`
`(2)x
`
`0060
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 60 of 90
`
`US 2011/0249073 Al
`
`6100
`
`~
`
`Start
`
`6105
`
`Start video conference
`
`6110
`
`Receive video captured from
`one camera of the other device
`
`End
`
`Yes
`
`End video
`conference?
`
`Auto Focus?
`
`6115
`
`6120
`
`No
`
`6140
`
`Send command to other
`device
`
`Figure 61
`
`0061
`
`

`

`Patent Application Publication
`
`Oct. 13, 2011 Sheet 61 of 90
`
`US 2011/0249073 Al
`
`t
`
`0
`-=:t
`N
`tO
`
`:
`-r-
`N:
`ta:
`......_.,:
`
`0
`0
`N
`tO
`
`1.(')
`0
`N
`tO
`
`lC')
`C')
`N
`tO
`
`0062
`
`

`

`> ....
`
`(.H
`-....l
`0
`\0
`.j;o.
`N
`0
`..._
`....
`0 ....
`N
`rFJ
`c
`
`US 2011/0249073 A1
`
`0
`\0
`0 .....
`N
`0\
`.....
`rFJ =(cid:173)
`....
`0 ....
`
`('D
`('D
`
`N
`
`Oct. 13, 2011 Sheet 62 0f 90
`
`~
`
`....
`(') ....
`0
`
`(.H
`
`.... 0 =
`.... 0 = '"= = 0" -....
`~ 'e -....
`('D = .....
`'"= ~ .....
`
`~ .....
`
`(')
`
`~ .....
`
`(')
`
`Patent Application Publication
`
`
`
`mocmLmEooucm
`
`wocmmeu—COOUCM
`
`Figure 63
`
`8£sz
`
`g.
`
`E?
`
`5835HEnmouxm
`
`$9

This document is available on Docket Alarm but you must sign up to view it.


Or .

Accessing this document will incur an additional charge of $.

After purchase, you can access this document again without charge.

Accept $ Charge
throbber

Still Working On It

This document is taking longer than usual to download. This can happen if we need to contact the court directly to obtain the document and their servers are running slowly.

Give it another minute or two to complete, and then try the refresh button.

throbber

A few More Minutes ... Still Working

It can take up to 5 minutes for us to download a document if the court servers are running slowly.

Thank you for your continued patience.

This document could not be displayed.

We could not find this document within its docket. Please go back to the docket page and check the link. If that does not work, go back to the docket and refresh it to pull the newest information.

Your account does not support viewing this document.

You need a Paid Account to view this document. Click here to change your account type.

Your account does not support viewing this document.

Set your membership status to view this document.

With a Docket Alarm membership, you'll get a whole lot more, including:

  • Up-to-date information for this case.
  • Email alerts whenever there is an update.
  • Full text search for other cases.
  • Get email alerts whenever a new case matches your search.

Become a Member

One Moment Please

The filing “” is large (MB) and is being downloaded.

Please refresh this page in a few minutes to see if the filing has been downloaded. The filing will also be emailed to you when the download completes.

Your document is on its way!

If you do not receive the document in five minutes, contact support at support@docketalarm.com.

Sealed Document

We are unable to display this document, it may be under a court ordered seal.

If you have proper credentials to access the file, you may proceed directly to the court's system using your government issued username and password.


Access Government Site

We are redirecting you
to a mobile optimized page.





Document Unreadable or Corrupt

Refresh this Document
Go to the Docket

We are unable to display this document.

Refresh this Document
Go to the Docket