Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 1 of 37 PageID #: 121
`Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 1 of 37 PageID #: 121
`
`EXHIBIT D
`
`EXHIBIT D
`
`

`

`Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 2 of 37 PageID #: 122
`
`111111111111111111111111111111111111111111111111111111111111111111111111111
`US00898211 OB2
`
`(12) United States Patent
`Saban et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,982,110 B2
`Mar.17,2015
`
`(54) METHOD FOR IMAGE TRANSFORMATION,
`AUGMENTED REALITY, AND
`TELEPERENCE
`
`(71) Applicant: EyesMatch Ltd., Road Town, Tortola
`(VG)
`
`(72)
`
`Inventors: Ofer Saban, Vienna, VA (US); Nissi
`Vilcovsky, Tokyo (JP)
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,230,039 A
`5,551,021 A
`
`7/1993 Grossman et al.
`8/1996 Harada et al.
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`(73)
`
`Assignee: Eyesmatch Ltd, Road Town, Tortola
`(VG)
`
`DE
`DE
`
`( *)
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`3/2001
`19943355 Al
`112002
`10031965 Al
`(Continued)
`
`OTHER PUBLICATIONS
`
`(21) Appl. No.: 14/253,831
`
`(22) Filed:
`
`Apr. 15, 2014
`
`(65)
`
`Prior Publication Data
`
`Aug. 14, 2014
`US 2014/0225978Al
`Related U.S. Application Data
`(63) Continuation-in-part of application No. 13/843,001,
`filed on Mar. 15, 2013, which is a continuation-in-part
`of application No. 13/088,369, filed on Apr. 17, 2011,
`now Pat. No. 8,624,883, which is a continuation of
`
`(51)
`
`Int. CI.
`G06F 31038
`G06T 19100
`
`(Continued)
`
`(2013.01)
`(2011.01)
`(Continued)
`
`(52) U.S. CI.
`CPC ................ G06T 191006 (2013.01); G02B 5108
`(2013.01); G06F 31011 (2013.01); G09F 19116
`(2013.01);
`
`(Continued)
`
`(58) Field of Classification Search
`CPC ............ G06F 3/01; G06F 3/011; G06F 3/005
`USPC ............... 345/204-215; 348/333.01; 434/395
`See application file for complete search history.
`
`Extended Search Report for European Patent Application No.
`06711263.1datedAug.18, 2011.
`(Continued)
`
`Primary Examiner - Vi jay Shankar
`(74) Attorney, Agent, or Firm -Nixon Peabody LLP;
`Joseph Bach, Esq.
`
`(57)
`
`ABSTRACT
`
`Computerized method for image transformation, augmented
`reality and telepresence. A camera generates a stream of
`images of the user; a processor detects the presence of the user
`in the stream of images and applies adaptive transformation
`mapping to the stream of images captured by the camera to
`generate modified images that appear to be captured from a
`different point of view of the camera's actual point of view.
`The modified images are displayed on a local and/or remote
`monitor for telepresence. The processor can also generate an
`avatar having body characteristics corresponding to body
`characteristics of a user appearing in the images. The proces(cid:173)
`sor may generate the avatar by analyzing registration pointers
`on the images of the camera. The processor may reside in the
`remote location, and the camera images transmitted over a
`network to the processor, and the modified images transmit(cid:173)
`ted back from the processor to the monitor.
`
`18Claims,16 Drawing Sheets
`
`102
`103
`~1!11-:
`
`.......... ""'_
`
`~~.........,mn;::,:.c:r:,~~:,•
`•
`.,..-)Oldll .....
`pbolsllhlrt .. llftlrlht-
`
`108
`
`104
`
`Scni8n1:m
`
`106
`
`105
`
`--~--
`
`Copy provided by USPTO from the PIRS Image Database on 06-17-2020
`
`

`

`Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 3 of 37 PageID #: 123
`
`US 8,982,110 B2
`Page2
`
`Related U.S. Application Data
`
`application No. 11/817,411, filed as application No.
`PCT/IL2006/000281 on Mar. 1, 2006, now Pat. No.
`7,948,481.
`
`(60) Provisional application No. 60/656,885, filed on Mar.
`1, 2005, provisional application No. 60/656,884, filed
`on Mar. 1, 2005, provisional application No. 61/738,
`957, filed on Dec. 18, 2012, provisional application
`No. 61/892,368, filed on Oct. 17, 2013, provisional
`application No. 61/862,025, filed on Aug. 4, 2013.
`
`(51)
`
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`(2011.01)
`(2006.01)
`(2006.01)
`(2011.01)
`(2006.01)
`(2006.01)
`
`Int. Cl.
`G02B 5/08
`G06F 3/01
`G09F 19/16
`G09F 27/00
`H04N 1/387
`H04N 1/62
`H04N 7/14
`G06T 11/00
`H04N 21/00
`G06T 3/00
`G06T 7/00
`G06T 15/20
`H04N 51262
`H04N 51272
`(52) U.S. Cl.
`CPC ................ G09F 27/00 (2013.01 ); H04N 1/387
`(2013.01); H04N 1/622 (2013.01); H04N 7/144
`(2013.01); G06T 11/00 (2013.01); H04N 21/00
`(2013.01); G06T 3/00 (2013.01); G06T 7/00
`(2013.01); G06T 15/205 (2013.01); H04N
`512624 (2013.01); H04N 512628 (2013.01);
`H04N 200512726 (2013.01)
`USPC ............................ 345/204; 345/212; 345/214
`
`( 56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,923,776 A
`5,937,081 A
`6,195,467 Bl
`6,366,694 Bl
`6,417,850 Bl
`7,500,755 B2 *
`7,874,481 B2 *
`7,874,681 B2
`7,948,481 B2
`8,000,727 Bl
`8,624,883 B2
`2002/0049546 Al
`2002/0196333 Al
`2003/0085866 Al
`2003/0110099 Al
`2003/0146901 Al
`2005/0018140 Al
`2005/0047629 Al
`2005/0259158 Al
`2006/0007303 Al
`2006/0158534 Al
`2006/0178902 Al
`
`7/1999 Kamgar-Parsi
`8/ 1999 O'Brill et al.
`21200 l Asimopoulos et al.
`4/2002 Acharya
`7/2002 Kang
`312009 Ishizaki et al. .................. 353/28
`1/2011 Silverbrook et al.
`. ........ 235/383
`1/2011 Huebner
`512011 Vilcovsky
`8/2011 Busmuan et al.
`112014 Vilcovsky
`4/2002 Shimomura
`1212002 Gorischek
`512003 Bimber et al.
`612003 Trajkovic et al.
`8/2003 Ryan
`1/2005 Ishizaki et al.
`312005 Farrell et al.
`1112005 Jacob et al.
`1/2006 Milton
`712006 Gotohda
`8/2006 Vicars et al.
`
`2007/0040033 Al
`2007/0120954 Al
`2007/0132863 Al
`2008/0056564 Al
`2008/0151092 Al
`2009/0051779 Al
`2009/0091710 Al
`2010/0097442 Al
`2010/0169411 Al
`2010/0190510 Al
`2010/0191578 Al
`2011/0199294 Al
`2011/0210970 Al
`201210120184 Al
`2012/0154872 Al
`2012/0169850 Al
`2012/0177284 Al
`2012/0229637 Al
`2012/0233089 Al
`2013/0083015 Al
`2013/0088562 Al
`2013/0229482 Al
`2014/0225977 Al
`2014/0226000 Al
`2014/0226900 Al
`
`212007 Rosenberg
`5/2007 Allen et al.
`6/2007 Deguchi
`3/2008 Lindbloom
`6/2008 Vilcovsky
`212009 Rolston
`4/2009 Huebner
`4/2010 Lablans
`7/2010 Colton et al.
`7/2010 Maranhas et al.
`7/2010 Tran et al.
`8/2011 Vilcovsky
`912011 Segawa
`512012 Fornell et al.
`612012 Reddy
`7/2012 Kimetal.
`7/2012 Wang
`9/2012 Mooradian et al.
`9/2012 Calman et al.
`412013 Hernandez
`412013 Hong et al.
`912013 Vilcovsky et al.
`8/2014 Vilcovsky et al.
`8/2014 Vilcovsky et al.
`8/2014 Saban et al.
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`EP
`EP
`WO
`WO
`WO
`
`1372092 Al
`1376207 Al
`1859432 A2
`00/22955 Al
`2006/092793 A2
`2014/100250 A2
`
`12/2003
`1/2004
`11/2007
`4/2000
`9/2006
`6/2014
`
`OTHER PUBLICATIONS
`
`Office Action for European PatentApplicationNo. 06711263.1 dated
`Jun. 21, 2012.
`Office Action for European PatentApplicationNo. 06711263.l dated
`Jun. 25, 2013.
`Decision to Refuse for European Patent Application No. 06711263.l
`dated Nov. 4, 2014.
`Office Action for U.S. Appl. No. 13/088,369 dated Nov. 2, 2012.
`Office Action for U.S. Appl. No. 13/088,369 dated Apr. 9, 2013.
`Notice of Allowance for U.S. Appl. No. 13/088,369 dated Sep. 3,
`2013.
`International Search Report and Written Opinion for PCT/IL2006/
`000281 dated Jun. 12, 2007.
`International Preliminary Report on Patentability for PCT/IL2006/
`000281 dated Sep. 20, 2007.
`Office Action for U.S. Appl. No. 11/817,411 mailed on Jul. 21, 2010.
`NoticeofAllowanceforU.S.Appl. No. 11/817,411 mailed on Feb. 3,
`2011.
`International Search Report and Written Opinion for PCT/US2013/
`076253 dated May 6, 2014.
`Notice of Allowance for U.S. Appl. No. 14/253,800 dated Oct. 24,
`2014.
`Invitation to Pay Additional Fees and, Where Applicable, Protest Fee
`for PCT/US2014/034333 dated Aug. 8, 2014.
`NoticeofAllowanceforU.S.Appl. No. 13/843,001 mailed on Dec. 5,
`2014.
`International Search Report and Written Opinion for PCT/US2014/
`034333 dated Dec. 17, 2014.
`Notice of Allowance for U.S. Appl. No. 14/253,827 dated Dec. 15,
`2014.
`* cited by examiner
`
`Copy provided by USPTO from the PIRS Image Database on 06-17-2020
`
`

`

`Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 4 of 37 PageID #: 124
`
`.......,
`('
`Video/still/IR
`Cam 20/30
`1 :n
`..I
`\
`'-101
`
`\..
`
`/
`
`/
`
`Screen 1:m
`
`l+t-t-
`
`_,,
`
`......-
`
`(")
`
`~
`1
`~ c:r
`'<
`2}i
`"'cl
`>-l
`0
`
`§'
`g-
`~
`s
`r§
`
`v I g
`
`0
`
`~ _,
`K>
`~
`0
`
`__
`
`_,,,
`
`103~
`
`Image grabbing module:
`Enhancement filters, format
`conversion, video frame separation,
`Image cropping or resize
`Image stitching if needed etc.
`-Cl 02
`Eyes-match transformation:
`Apply on the image the right mapping
`to match the camera point of view with i.--
`theoretical mirror point of view
`(user eyes reflection) and fill the blind
`pixels if there are after the mapping.
`104
`• \
`
`w_
`
`1
`+
`Augmented reality module
`e.g., virtual color and texture
`replacement virtual dressing,
`object insertion, etc.
`
`Trigger event module:
`User in front of the mirror
`Face recognition
`User gesture commands
`Item recognition
`Distance measurement
`User body measurements/
`assessment (height, age,
`weight, ethnic group, Sex etc.)
`Calculate User theoretical point of
`view in theoretical mirror.
`'-107
`
`108 \
`
`1 ~ rl" Cloud
`
`FIG 1
`
`•
`
`,-1 l O
`r
`Interface
`
`..___..
`
`_
`
`..._
`
`..... ~ Web/
`store
`_
`
`4
`
`-
`User
`s~art
`P one
`-
`
`~
`rJJ.
`•
`
`~ = ~ n> a
`
`~
`
`s::
`:"I .....
`;--l
`N = ......
`
`tll
`
`('!)
`('!)
`
`00 =(cid:173)
`.....
`.....
`0
`'""
`.....
`
`Q'I
`
`L1
`r:JJ.
`00
`~
`00
`N
`';....
`~
`Q
`Cd
`N
`
`Control element
`Control and management
`Set the camera for optimize quality
`~ Set other HW ~lements,
`ln~erface between. alg_onthm ~odules and
`higher code{apphcatJon/user interfaces.
`Push calibrated .data from factory
`into the algonthm elements.
`~
`Factory calibration
`Define the mapping transformation
`be~en .camera and user point of
`view m front of the screen.
`Calibration based distance, special location
`and user height or any combination.
`
`'---.----""I/
`\_ 106
`
`'-
`
`Video/still recording·
`Record single image or ~ort
`take based on SW control.
`-C 105
`Machine vision augmented reality modules
`
`I+-
`I+--
`109
`\-
`'-......
`
`

`

`Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 5 of 37 PageID #: 125
`
`n
`~
`>o
`~
`0.:
`(0
`i::>..
`0-
`'<
`c:::
`VJ
`"t:I
`>-l
`0
`i:t>
`0
`s
`st-(0
`~
`VJ a
`Jg
`u
`~
`~
`~
`
`(0
`
`0
`::i
`0
`'i"
`..,.
`
`N
`0
`N
`0
`
`Calibrated image/video after ~
`201
`EyesMatch: Distortion scale orientation
`and location correction
`
`202
`Element 1 :n election for segmentation and extraction
`(X, Y) in the pixel space.
`Can be multiple election points per object
`
`Election interface
`To choose object
`
`Model/Mask generation of the objects in the color
`space 20/grayscale or 30 in the RGB space
`Mask will be saved and associated to the original video
`
`203
`
`Rendering Module
`Get new color, texture per object and modify the mask
`
`Get the original video and render the modified object in
`
`New color texture election
`For object and background
`
`204
`
`205
`
`FIG. 2
`
`206~
`
`Interface
`
`I
`
`Cloud Database
`Video from the mirror
`
`Web/store
`User upload
`video
`
`User smart
`phone
`Election and video
`
`E-commerce
`Fabric, color,
`items etc.
`
`I
`
`I
`
`~
`rJl .
`
`'"'d
`~
`~
`~
`
`~
`~
`:"I
`
`;'I
`N
`~
`Ul
`
`00
`
`= ~
`-
`-
`-
`'""' -
`
`=-
`
`('D
`('D
`
`N
`Q
`
`c::l'I
`
`~
`rJ).
`QC
`\c
`QC
`N
`-..lo-'
`
`lo-' = Cd
`
`N
`
`

`

`Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 6 of 37 PageID #: 126
`
`n
`0
`~
`
`l
`~
`~
`c:;
`
`[/J
`
`~
`~
`~
`g-
`::s c;
`s
`~
`tJ
`~
`~
`§
`C>
`'?'
`:::;
`t0
`C>
`N
`C>
`
`(1)
`
`Calibrated image/video after
`EyesMatch: Distortion scale orientation
`and location correction
`
`302
`Element 1 :n election for segmentation and extraction
`(X, Y) in the pixel space.
`Can be multiple election points per object
`e.g., eyes, hair, skin, etc.
`
`306
`
`Election
`interface
`To choose
`body part for
`modification
`
`Model/Mask generation of the objects in the color
`space 2D/grayscale or 30 in the RGB space
`Mask will be saved and associated to the original video
`
`303
`
`307
`
`Original video
`after EyesMatch
`Body Distortion correction
`+
`Background after EyesMatch
`+
`e.g., head pose/body
`movement detector
`
`Rendering Module
`Get item 3D, current and required orientation info,
`(e.g., current eye pose and required correction)
`Get the video and render the modified object in
`304
`
`305
`
`FIG. 3
`
`Interface
`
`Cloud Database
`Video from
`the mirror
`
`......
`
`Web/store
`User upload
`video
`
`......
`User smart phone
`Election and video
`
`E-commerce
`Fabric, color,
`items, etc.
`
`~
`00
`•
`~
`~
`~
`
`~ = ~
`
`~
`~
`:'!
`"""'
`;.J
`N
`Q
`"""' Ul
`
`r:.ri =(cid:173)
`('!)
`~
`w
`<::>
`.....
`"""' 0\
`
`New color
`texture election
`For object and
`background
`
`d
`00.
`QO
`~
`QO
`N
`1-
`
`~ = Cd
`
`N
`
`

`

`Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 7 of 37 PageID #: 127
`
`U.S. Patent
`
`Mar.17,2015
`
`Sheet 4of16
`
`US 8,982,110 B2
`
`401
`
`Converting frame into a column
`vector !1xn] N=WldthxHeightx3
`
`Eliminating the brightness effect.
`As an example:
`1. Converting the RGB into XYZ
`2. Etch pixel, divide by the sum of the XYZ
`
`411
`Additional information other than
`color about the object that can help
`with the mask decision:
`1. Depth map e.g., IR, 30, RF camera
`or sensor (after some registration to
`actual image 20 or 30)
`2. Object shape boundaries texture
`and location characteristics
`3. Histogram stretching global or local
`4. Previous frames (likelihood pixel is
`part of the object)
`
`5.
`
`402
`
`403
`
`405
`
`Sample the color in the K(x,y), and
`perform the color transformation
`as in 403 for all K
`
`406
`
`Find the closest distance in the 30 color space of
`the first K(x,y) to any of the image pixels vector of
`403, app~ the decision with additional info from 411.
`Repeat for all K sample points. Save the minimum
`distance and remember the index K with the closest
`distance. Result will he labeled image
`
`Assign label = zero to all the distances > threshold
`Add 1 to the labeled image
`
`407
`
`408
`
`Zero all the nonrelevant labels and assign 1 to the relevant
`Optional: App~ morphological BW filter to clean noise and smooth the shape
`of the object or app~ any of the additional technique mentioned to improve
`the decision if a pixel is part of the object or not
`
`Apply the mask on the Color image to obtain the 2D/3D gray/color
`mask apply/cross the decision wtth addttional info from 411
`
`410
`
`409
`
`FIG. 4
`
`Copy provided by USPTO from the PIRS Image Database on 06-17-2020
`
`

`

`Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 8 of 37 PageID #: 128
`
`U.S. Patent
`
`Mar.17,2015
`
`Sheet 5 of16
`
`US 8,982,110 B2
`
`Video or image of body to analyze after the EyesMatch calibration and
`projection correction f (user distance, camera angle and he~ht, optics)
`501_)
`
`FIG. 5
`
`Additional information on the user. Gender ethnic group,
`type of object that he is wearing, previous info for comparison
`\._504
`
`511~ 1
`
`502-=:J.
`
`Optional for speed:
`Converting frame into a column
`vector [1xn] N=WidthxHeightx3
`
`503-=:J.
`~
`Eliminating the brightness effect.
`As an example:
`1. Converting the RGB into XYZ
`2. Etch pixel, divide by
`the sum of the XYZ
`
`1. Additional information on the user from
`other sensors or other ~atforms e.g., Depth
`Map, IR, 30, RF camera or sensor (after some
`regislration or cafibration to actual image 20 or 30)
`2. Items identification what the user is actual~ wearing
`3. lnfonnation from multiple or previous cuts or
`measurements or from other platforms or user itself
`4. Normal body proportion per sex/ethnic age etc. (will
`use the info for better searching of body part location)
`5. Calibration information of the image pixeUcm
`(after the image distortion and perspective correction)
`6. Other: Statistic medical information
`J
`I
`506""\_
`Apply multi angle/orientation edge detection methods to obtain rough body line
`j_
`Calculate the central mass and head location.
`Can be done in multiple techniques e.g., flipping left right and auto coloration.
`C507
`508~
`Segment the body according to body proportion statistic information. E.g., ratio between
`navel to height, bust to navel, etc. and searching for each cut the representative pixel
`I
`Translate the pixel cut into perimeter predication based on a single or multiple cuts }-
`1
`~509
`510~
`Analysis Interpretation: Obtain user body type, BMI, health risks, weight,
`user suggested outfit size, or outfit forms based on his body type estimation, etc.
`
`l
`
`t-E-
`
`Copy provided by USPTO from the PIRS Image Database on 06-17-2020
`
`

`

`Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 9 of 37 PageID #: 129
`
`U.S. Patent
`
`Mar.17,2015
`
`Sheet 6of16
`
`US 8,982,110 B2
`
`60~ J "--607
`.....-~Co-ntr-0-1 e-le-me-nt---.
`Control and management
`Set the camera for optimize
`quality. Set other HW
`elements, Interface between ~
`algortthm modules and
`higher code/application/
`user interfaces. Push
`calibrated data from factory
`into the algorithm elements.
`
`Trigger event module:
`User in front of the camera/mirror
`Face detection/recognition
`User gesture commands
`Item recognition
`Distance measurement
`User body measurements/
`assessment (heigh~ age,
`weigh~ ethnic group, Sex, etc.)
`
`I-+
`
`l
`
`Factory calibration
`Defme the mapping
`transformation between
`camera and user point of
`view in front of the screen.
`CaHbration based distance,
`special location and user
`height, etc.
`or any combination.
`\_609
`
`L610
`Interfaces
`
`~ Cloud
`
`I+
`
`I+
`
`Web/
`store
`
`User
`smart
`phone
`
`Security
`personal
`4-1 mobile or
`controlapp
`
`/'
`""\
`Video/still/IR
`Cam 20/30 l--
`1:n
`\_601
`
`\.
`
`\ £,
`
`l
`
`14--+
`
`I----
`
`Image grabbing module:
`Enhancement filters,
`fonnat conversion,
`video frame separation,
`Image cropping or resize
`Image stitching n needed, etc.
`603 ...,_
`-u02
`_l
`Eyes-match transformation:
`Apply on the image the right
`~---i mapping to match the camera
`point of view with user location/
`eyes location. (We obtained
`calibrated image or video
`L
`l
`604~
`Security scanning module
`Body line scanner (based edge
`detection and analysis)
`Cloth detection -
`correlation with Database
`Face recognHion
`Additional recognition of:
`item user carry, gender, color,
`origin impression, palm, eyes,
`fingerprinting, voice, hair, DNA etc.
`Additional sensors:
`microwave scanner, IR scanner,
`smell, wireless sniffer to trigger
`and detect user device ID
`
`605~ I
`
`Video/still recording:
`record single image or t+(cid:173)
`r - - short take based on I-(cid:173)
`SW control
`
`L606
`
`Screen 1:m
`In front of the user guiding him what to do or in a control room
`
`FIG. 6
`
`Copy provided by USPTO from the PIRS Image Database on 06-17-2020
`
`

`

`Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 10 of 37 PageID #: 130
`
`U.S. Patent
`
`Mar.17,2015
`
`Sheet 7of16
`
`US 8,982,110 B2
`
`Smart security device, or SW in the camera DSP
`
`"
`
`\.
`
`/
`Vldeo/stilUIR
`Cam 20/30 t---
`1 :n
`"-101
`
`1-E--+
`
`Image grabbing module:
`Enhancement filters,
`format conversion,
`video frame separation,
`Image cropping or resize
`Image stitching if needed, etc.
`\ .
`'-702
`703__:j_
`Eyes-match transformation:
`Apply on the image the right
`mapping to match the camera
`....- point of view with user location/ I-E(cid:173)
`eyes location. (We obtained
`calibrated image or video
`
`Trigger event module:
`User in front of the camera/mirror
`Face detection/recognition
`User gesture commands
`Item recognition
`Distance measurement
`User body measurements/
`assessment (height, age,
`weigh~ ethnic group, Sex etc.)
`
`708~ J \:707
`
`Interfaces
`wire or
`wireless access
`
`r-
`
`Cloud
`
`Web/
`store
`
`I+
`
`I+
`
`User
`smart
`phone
`
`Security
`personal
`~ mobileor
`control app
`
`FIG. 7
`
`Control element
`Control and management:
`Set the camera for optimize
`quality. Set other HW
`elements, Interface between ~
`algorithm modules and
`higher code/application/
`user interfaces. Push
`calibrated data from factory
`into the algorithm elements.
`
`l
`
`Factory calibration
`Define the mapping
`transfonnation between
`camera and user point of
`view in front of the screen.
`Calibration based distance,
`special location and user
`height, etc.
`or any combination.
`\_709
`
`t--
`
`Security scanning module
`Body line scanner (based edge
`detection and analysis) ~
`Cloth detection •
`correlation with Database
`Face recognition
`Additional recognition of:
`item user carry, gender, color,
`origin impression, palm, eyes,
`fingerprinting, voice, hair, DNA etc.
`Additional sensors:
`microwave scanner, IR scanner,
`smell, wireless sniffer to trigger
`and detect user device ID
`
`J
`70~""\
`1
`Lf-.
`Video/still recording:
`Optional augmented reality
`]
`module e.g. virtual color and
`record single image or
`texture replacement virtual ~
`short take based on
`dressing, object insertion, or
`SW control
`dilation or background
`\__705
`record and manipulate, etc.
`
`1+
`
`Copy provided by USPTO from the PIRS Image Database on 06-17-2020
`
`

`

`Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 11 of 37 PageID #: 131
`
`U.S. Patent
`
`Mar.17,2015
`
`Sheet 8of16
`
`US 8,982,110 B2
`
`• Q
`
`([J
`
`FIG. 8a
`
`FIG. 8b
`
`FIG. 8c
`
`Copy provided by USPTO from the PIRS Image Database on 06-17-2020
`
`

`

`Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 12 of 37 PageID #: 132
`
`U.S. Patent
`
`Mar.17,2015
`
`Sheet 9of16
`
`US 8,982,110 B2
`
`0
`
`FIG. 9
`
`Copy provided by USPTO from the PIRS Image Database on 06-17-2020
`
`

`

`Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 13 of 37 PageID #: 133
`
`U.S. Patent
`
`Mar.17,2015
`
`Sheet 10 of 16
`
`US 8,982,110 B2
`
`.
`.
`.o·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.
`
`. . ..
`.. . . . ..
`·············~
`..
`..
`..
`..
`...................
`. ..
`.. . . ..
`.. . . ..
`..
`..
`.. . . .. . , .........
`. ..
`. ..
`.. . ..
`..
`..
`..
`..
`..
`..
`..
`.. . ..
`.. . . .
`. .. . . ..
`
`.............
`.............
`
`\ :-:-:-:-:-:-:-:-:-:·:-:-:
`.::.:::
`\
`
`. ..
`
`. .................................................. .
`
`FIG. 10
`
`Copy provided by USPTO from the PIRS Image Database on 06-17-2020
`
`

`

`Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 14 of 37 PageID #: 134
`
`U.S. Patent
`
`Mar.17,2015
`
`Sheet 11 of 16
`
`US 8,982,110 B2
`
`Endomorph
`
`Ectomorph
`
`Mesomorph
`FIG. 11
`
`Banana
`
`Apple
`Pear
`FIG. 12
`
`Hourglass
`
`Copy provided by USPTO from the PIRS Image Database on 06-17-2020
`
`

`

`Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 15 of 37 PageID #: 135
`
`U.S. Patent
`
`Mar.17,2015
`
`Sheet 12of16
`
`US 8,982,110 B2
`
`0
`
`-----41--1301
`
`............
`.............
`............
`.............
`............
`.............
`............
`.............
`.............
`.............
`.............
`............
`............
`............
`............
`............
`.............
`.............
`:.·/::::///::
`
`::::::::::::::::::::::::
`. .......... .
`. .......... .
`············
`. .......... .
`. .......... .
`············
`. .......... .
`. .......... .
`············
`············
`. .......... .
`. ........... .
`. ........... .
`.............
`.............
`············
`............
`············
`
`1302
`
`1304
`
`1305
`
`1303
`
`FIG. 13
`
`Copy provided by USPTO from the PIRS Image Database on 06-17-2020
`
`

`

`Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 16 of 37 PageID #: 136
`
`U.S. Patent
`
`Mar.17,2015
`
`Sheet 13 of 16
`
`US 8,982,110 B2
`
`FIG. 14
`
`Copy provided by USPTO from the PIRS Image Database on 06-17-2020
`
`

`

`Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 17 of 37 PageID #: 137
`
`U.S. Patent
`
`Mar.17,2015
`
`Sheet 14 of 16
`
`US 8,982,110 B2
`
`1505
`
`REGISTl:R
`
`1515
`
`FIG. 15
`
`Copy provided by USPTO from the PIRS Image Database on 06-17-2020
`
`

`

`Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 18 of 37 PageID #: 138
`
`U.S. Patent
`
`Mar.17,2015
`
`Sheet 15 of 16
`
`US 8,982,110 B2
`
`FIG. 16
`
`Copy provided by USPTO from the PIRS Image Database on 06-17-2020
`
`

`

`Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 19 of 37 PageID #: 139
`
`U.S. Patent
`
`Mar.17,2015
`
`Sheet 16of16
`
`US 8,982,110 B2
`
`Mirror in idle mode.
`
`1
`
`User approaching to the mirror, presence sensor and
`tracking are locking and creating mirror experience.
`
`I
`
`Automatic authentication/or registration occurs, station will try to
`recognize the user and if it can't, a new account will be opened.
`Self authentication or combination of self and
`automatic/registration is also on embodiment.
`
`1
`
`Based on setting and configuration the mirror allows the user control
`and configuration of the local station in automatic, semi automatic,
`manual or remote control manner.
`User/remote assistant/local assistance can control the player by
`selecting thumbnails and e.g., start, stop, erase, record of the video.
`
`1701
`
`1702
`~
`
`1703
`~
`
`r-- 1704
`
`1
`
`User after installing application on his mobile device or from the web
`can share the videos with friends and advisers. In addition can get
`recommendation for other item sizes, colors, accessories. The ~ 1705
`recommendation can be supported with new image with the
`new item or different color and or some pricing incentive, etc.
`
`I
`
`User can get recommendation, coupons, access to
`E-commerce inventory, and other item and colors of items he liked.
`Can be linked to other relevant third party services and sites.
`Can get recommendation on sizes for E-commerce, see trends.
`User can see sharing from other customers.
`User can rate and observe trends of appearance.
`
`r--
`
`1706
`
`FIG. 17
`
`Copy provided by USPTO from the PIRS Image Database on 06-17-2020
`
`

`

`Case 1:21-cv-00111-UNA Document 1-4 Filed 01/28/21 Page 20 of 37 PageID #: 140
`
`US 8,982, 110 B2
`
`2
`video created by one or more cameras, with or without other
`sensors, into a mirror or video conference experience.
`
`SUMMARY
`
`The following summary of the disclosure is included in
`order to provide a basic understanding of some aspects and
`features of the invention. This summary is not an extensive
`overview of the invention and as such it is not intended to
`particularly identify key or critical elements of the invention
`or to delineate the scope of the invention. Its sole purpose is to
`present some concepts of the invention in a simplified form as
`a prelude to the more detailed description that is presented
`below.
`In this disclosure we describe a computerized technique
`that takes a video, a still, or a group of still images, before or
`after the transformation described in U.S. application Ser. No.
`13/843,001, and continue the computerized method to
`address additional functionalities, such as user interface, aug(cid:173)
`mented reality, color change, texture change, shape manipu(cid:173)
`lation of an object, body or background, and more. In addi-
`tion, the calibrated virtual camera approach allows for body
`line, body curve and body type measurements from a 2D or
`3D image or video.
`For the augmented reality capabilities, first a 2D and/or 3 D
`model or mask will be created for one or more items or
`elements (e.g., shirt, dress, pants, head, shoe, glass, entire
`body, and the like), then the model or mask manipulation can
`enable the augmented reality features like color, texture
`replacement and manipulation, or geometric measurement, or
`complete object replacement. The disclosed technique is dif-
`ferentiated from other techniques that are using only 3D
`camera (IR or dual camera). In the following disclosed tech(cid:173)
`niques one can build the model from a single regular camera
`and can improve the results with multiple cameras and addi(cid:173)
`tional information from IR, 3D camera or other sensors and
`information regarding the object sought to be manipulated.
`Some embodiments can include multi level user identifi(cid:173)
`cation. Specifically, the embodiments can include face rec-
`40 ognition improvement, user learning, and adaptation. Known
`methods of face recognition can utilize algorithms that can be
`very sensitive to face pose. In the present technique, a smooth
`experience of face pose can be created to accommodate dif-
`ferent camera locations and angles.
`Various embodiments can include code identification, e.g.,
`quick recognition (QR) code, lD Code, hidden code and the
`like. The embodiments can be adapted to discover codes from
`a relatively long distance with a relatively small image as
`compared to the image under projection or under other optic
`50 distortion. Also, it may include wireless identification, e.g.,
`NFC, WiFi wideband, microwave 3D, body access network
`(BAN) chip and the like. Wireless identification can be made
`from a mobile device, watch glass, microchip or any other
`carry on device or microchip. Other body measurement tech-
`55 niques may include fingerprinting, body identification, body
`type, eyes, palm recognition, X-ray correlation, body tem(cid:173)
`perature, body pulse, blood pressure and the like.
`In another aspect a user interface and methods to control
`and operate manually or automatically the virtual mirror
`60 capabilities are described.
`In additional embodiments the product mechanical design
`and appearance are disclosed, to enhance the usability and
`functionality and the user experience overall.
`In some embodiments, a non-transitory computer-readable
`65 storage medium for operating a monitor, a camera, and a
`processor, is provided and configured so as to display a mir(cid:173)
`ror-mimicking image on the monitor, and comprising: on a
`
`1
`METHOD FOR IMAGE TRANSFORMATION,
`AUGMENTED REALITY, AND
`TELEPERENCE
`
`CROSS-REFERENCE(S) TO RELATED
`APPLICATION(S)
`
`This application claims the benefit of, and priority to, U.S.
`Provisional Patent Application No. 61/862,025, filed on Aug.
`4, 2013, entitled "Provisional of Virtual calibrated camera 10
`capable to provide augmented reality features: e.g. Color,
`texture, shape manipulation of body items or background in
`virtual mirror or video conference. In addition capable of
`performing body measurements, body monitoring for com- 15
`mercial, security and healthcare applications," the entire dis(cid:173)
`closure of which is hereby incorporated herein by reference.
`This application claims the benefit of, and priority to, U.S.
`Provisional Patent Application No. 61/892,368, filed on Oct.
`17, 2013, entitled "Virtual Mirror Flow of Usage Control and 20
`User Interface" the entire disclosure of which is hereby incor(cid:173)
`porated herein by reference.
`This application is Continuation-in-Part of U.S. applica(cid:173)
`tion Ser. No. 13/843,001, filed Mar. 15, 2013, entitled
`"DEVICES, SYSTEMS AND METHODS OF CAPTUR- 25
`ING AND DISPLAYING APPEARANCES," which is Con(cid:173)
`tinuation-in-Part of U.S. application Ser. No. 13/088,369,
`filed Apr. 17, 2011, which is a continuation of U.S. applica(cid:173)
`tion Ser. No. 11/817,411, now U.S. Pat. No. 7,948,481, issued
`May 24, 2011, and which was a National Phase application of 30
`PCT International Application No. PCT/IL06/000281, Inter(cid:173)
`national Filing Date Mar. 1, 2006, which claims the benefit of
`U.S. Provisional Application No. 60/656,884, filed Mar. 1,
`2005, and U.S. Provisional Application No. 60/656,885, filed
`Mar. 1, 2005. U.S. application Ser. No. 13/843,001 further 35
`claims the priority benefit of U.S. Provisional Application
`No. 61/738,957, filed Dec. 18, 2012. The entire disclosures of
`all of the above listed applications are incorporated herein by
`reference.
`
`TECHNICAL FIELD
`
`The invention relates generally to imaging and display
`systems and, more particularly, to monitors, and interactive
`displays, e.g., in retail and/or service environments, medical 45
`or home situations, video conferencing, gaming, and the like.
`Specific implementations relate to virtualizing a mirror in
`situations where users expect to see a mirror, e.g., in trying on
`apparel. Another specific implementation relate to situations
`where a natural appearance is preferable over standard video
`image, such as in, e.g., video conferencing.
`
`BACKGROUND
`
`The conventional mirror (i.e., reflective surface) is the
`common and most reliable tool for an individual to explore
`actual self-appearance, in real time. A few alternatives have
`been proposed around the combination of a camera and a
`screell" to replace the conventional mirror. However, these
`techniques are not convincing and are not yet accepted as a
`reliable image of the individual as ifhe was looking at himself
`in a conventional mirror. This is mainly because the image
`generated by a camera is very different from an image gen(cid:173)
`erated by a mirror.
`In