Case 1:22-cv-01257-GBW Document 1-2 Filed 09/22/22 Page 1 of 29 PageID #: 80
`
`
`
`Exhibit 2
`
`
`
`
`
`
`

`

`Case 1:22-cv-01257-GBW Document 1-2 Filed 09/22/22 Page 2 of 29 PageID #: 81
`I 1111111111111111 11111 1111111111 111111111111111 IIIII IIIII IIIIII IIII IIII IIII
`US007336814B2
`
`c12) United States Patent
`Boca et al.
`
`(IO) Patent No.:
`(45) Date of Patent:
`
`US 7,336,814 B2
`Feb.26,2008
`
`(54) METHOD AND APPARATUS FOR
`MACHINE-VISION
`
`(75)
`
`Inventors: Remus F Boca, North Vancouver (CA);
`Babak Habibi, North Vancouver (CA);
`Mohammad Sameti, Coquitlam (CA);
`Simona Pescaru, Vancouver (CA)
`
`4,334,241 A
`4,437,114 A
`4,578,561 A
`4,613,942 A
`
`6/1982 Kashioka et al ............ 358/107
`3/1984 LaRussa ..................... 358/101
`3/1986 Corby, Jr. et al.
`...... 219/124.34
`9/1986 Chen .......................... 364/513
`
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`(73) Assignee: Braintech Canada, Inc., North
`Vancouver (CA)
`
`EP
`
`0 114 505
`
`8/1984
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by O days.
`
`(Continued)
`
`OTHER PUBLICATIONS
`
`(21) Appl. No.: 11/183,228
`
`(22) Filed:
`
`Jul. 14, 2005
`
`(65)
`
`Prior Publication Data
`
`US 2006/0088203 Al
`
`Apr. 27, 2006
`
`Related U.S. Application Data
`
`(60) Provisional application No. 60/587,488, filed on Jul.
`14, 2004.
`
`(51)
`
`Int. Cl.
`G06K 9/00
`(2006.01)
`(52) U.S. Cl. ...................... 382/141; 382/142; 382/143;
`382/144; 382/147; 382/152
`(58) Field of Classification Search ........ 382/141-153,
`382/155-160
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`3,986,007 A
`4,146,924 A
`4,219,847 A
`4,305,130 A
`
`10/1976
`3/1979
`8/1980
`12/1981
`
`Ruoff, Jr .................. 235/151.1
`Birk et al.
`.................. 364/513
`Pinkney et al.
`............. 358/126
`Kelley et al.
`............... 364/513
`
`"3D Robot Guidance with a Single Camera," ISRA Vision Systems
`AG, pp. 83-105, no date available.
`
`(Continued)
`
`Primary Examiner-Brian Le
`(74) Attorney, Agent, or Firm-Seed IP Law Group PLLC
`
`(57)
`
`ABSTRACT
`
`A system and method facilitate machine-vision, for example
`three-dimensional pose estimation for target objects, using
`one or more images sensors to acquire images of the target
`object at one or more positions, and to identify features of
`the target object in the resulting images. A set of equations
`is set up exploiting invariant physical relationships between
`features such as constancy of distances, angles, and areas or
`volumes enclosed by or between features. The set of equa(cid:173)
`tions may be solved to estimate a 3D pose. The number of
`positions may be determined based on the number of image
`sensors, number of features identified, and/or number of
`known physical relationships between less than all features.
`Knowledge of physical relationships between image sensors
`and/or between features and image sensors may be
`employed. A robot path may be transformed based on the
`pose, to align the path with the target object.
`
`39 Claims, 12 Drawing Sheets
`
`f/24
`
`140
`
`ALL IMAGES
`FROM ALL VIEWS
`
`TAKE NEXT VIEW (from I to m)
`
`TAKE NEXT IMAGE OF CURRENT VIEW
`(from I lo c)
`
`LOCATE FEATURES PREVIOUSLY TRAINED
`CORRESPONDING TO THE IMAGE SENSOR
`THAT ACQUIRED THE
`IMAGE
`
`EXTRACT FEATURE INFORMATION
`
`NO
`GO TO COMPUTE POSITION
`
`150
`
`

`

`Case 1:22-cv-01257-GBW Document 1-2 Filed 09/22/22 Page 3 of 29 PageID #: 82
`
`US 7,336,814 B2
`Page 2
`
`U.S. PATENT DOCUMENTS
`4/1987 Pryor .......................... 29/407
`4,654,949 A
`5/1989 Suzuki .................. 318/568.13
`4,835,450 A
`7/1990 McGee et al. .............. 364/513
`4,942,539 A
`5,083,073 A
`1/1992 Kato .......................... 318/577
`5,212,738 A *
`5/1993 Chande et al. .............. 382/152
`9/1994 Azuma et al.
`.............. 414/416
`5,350,269 A
`5,454,775 A
`10/1995 Cullen et al. ................. 483/16
`12/1997 Pryor .................... 364/167.01
`5,696,673 A
`9/1999 Pryor ......................... 382/154
`5,956,417 A
`9/1999 Bieman et al. ........ 318/568.15
`5,959,425 A
`5,978,521 A * 11/1999 Wallack et al. ............. 382/294
`6,044,183 A
`3/2000 Pryor ......................... 382/287
`6,115,480 A *
`9/2000 Washizawa ................. 382/103
`6,141,863 A
`11/2000 Hara et al. .................... 29/714
`1/2001 Pryor ...................... 29/407.04
`6,167,607 Bl
`6,211,506 Bl
`4/2001 Pryor et al. .............. 250/208.1
`6,301,763 Bl
`10/2001 Pryor ...................... 29/407.04
`6,341,246 Bl
`1/2002 Gerstenberger et al.
`.... 700/245
`6,466,843 Bl
`10/2002 Bonanni et al.
`............ 700/245
`6,594,600 Bl
`7/2003 Arnoul et al.
`................ 702/94
`9/2003 Huang et al. ............... 382/154
`6,628,819 Bl*
`6/2004 Fujita et al. ................ 700/245
`6,754,560 B2
`10/2004 Bachelder et al. .......... 382/294
`6,804,416 Bl
`11/2004 Habibi et al. ............... 700/259
`6,816,755 B2
`6/2006 Franke et al. ............... 356/604
`7,061,628 B2 *
`1/2003 Aliaga et al. .................. 703/2
`2003/0004694 Al
`2003/0007159 Al *
`1/2003 Franke et al. ............... 356/604
`2004/0114033 Al
`6/2004 Eian et al. .................... 348/42
`2004/0172164 Al
`9/2004 Habibi et al.
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`EP
`EP
`EP
`EP
`JP
`
`0151417
`0493612
`0763406 Bl
`0911603 Bl
`0951968
`1-124072
`
`8/1985
`7 /1992
`9/1997
`4/1999
`10/1999
`5/1989
`
`OTHER PUBLICATIONS
`
`3D Vision with One Camera, URL~http://neu.isravision.com/
`likecms/index.php?site~site.html&diFisra&nav~ 162,
`download
`date Apr. 12, 2005.
`Bejczy, A. K., "Challenges of Human-Robot Communication in
`Telerobotics," IEEE International Workshop on Robot and Human
`Communication, pp. 1-8, Nov. 11-14, 1996.
`Borotschnig, H., et al., "Appearance-Based Active Object Recog(cid:173)
`nition," Image and Vision Computing, 18:715-727, 2000.
`U.S. Appl. No. 60/413,180, filed Sep. 23, 2002, Eian et al.
`
`U.S. Appl. No. 60/587,488, filed Jul. 14, 2004, Boca et al.
`Denzler, J., et al., "Learning, Tracking and Recognition of 3D
`Objects," Proceedings of the International Conference on Intelli(cid:173)
`gent Robots and Systems (!ROS), XP000512662, 1:89-96, Munich,
`Sep. 12, 1994 .
`Huang, T., et al., "Uniqueness of3D Pose Under Weak Perspective:
`A Geometrical Proof," IEEE Transactions on Pattern Analysis and
`Machine Intelligence, 17(12):1220-1221, Dec. 1995.
`Ji, Q., et al., "An Integrated Linear Technique for Pose Estimation
`from Different Geometric Features," International Journal of Pattern
`Recognition and Artificial Intelligence, 13(5):705-733, Aug. 1999.
`Jia, Y-B., et al., "Sensing Polygon Poses by Inscription," in Pro(cid:173)
`ceedings of 1994 IEEE International Conference on Robotics and
`Automation, Los Alamitos, CA, May 8, 1994, pp. 1642-1649.
`Kim, W., "Computer Vision Assisted Virtual Reality Calibration,"
`URL~http://www-robotics.jpl.nasa.gov/publications/Won_Kirn/
`ra98_vrc.pdf.
`Kovacic, S., et al., "Planning Sequences of Views for 3-D Object
`Recognition and Pose Determination," Pattern Recognition,
`31(10):1407-1417, 1998.
`Liu, Y., et al., "Determination of Camera Location from 2D to 3D
`Line and Point Correspondences", IEEE Transaction on Pattern
`Analysis and Machine Intelligence, 12(1), Jan. 1990.
`Lu, C-P., et al., "Fast and Globally Convergent Pose Estimation
`from Video Images," Transactions on Pattern Analysis and
`Machine Intelligence, 22(6):610-622, Jun. 2000.
`Meyer, W.,"One-Eyed Robots With 3D Vision," ISRA Vision Sys(cid:173)
`tems AG, Press News, Release No. 16, Jun. 15, 2004, pp. 1-7 .
`Sanchez, A., et al., "Robot-Arm Pick and Place Behavior Program(cid:173)
`ming System Using Visual Perception," in Proceedings of the 15th
`International Conference on Pattern Recognition, Los Alamitos,
`CA, Sep. 3-7, 2000, 4:507-510.
`Sharma, R., "Visual Servoing with Independently Controlled Cam(cid:173)
`eras Using a Learned Invariant Representation," in Proceedings of
`the 37"' IEEE Conference on Decision & Control, Tampa, FL, Dec.
`1998, pp. 3263-3268.
`Tsai, R., et al., "A New Technique for Fully Autonomous and
`Efficient 3D Robotics Hand/Eye Calibration," in IEEE Transactions
`on Robotics and Automation, 5(3):345-358, Jun. 1989.
`Wei, G.-Q., et al., "Active Self-Calibration of Robotic Eyes and
`Hand-Eye Relationships With Model Identification," IEEE Trans(cid:173)
`actions on Robotics and Automation, 14(1):158-166, Feb. 1998.
`Wei, G-Q., et al., "Multisensory Visual Servoing By a Neural
`Network," IEEE Transactions on Systems, Man and Cybernetics,
`Part B: CYBERNETICS, 29(2):276-280, Apr. 1999.
`Zhang, Z., "A Flexible New Technique for Camera Calibration,"
`URL~http://research.microsoft.com/research/pubs/view.
`aspx?tr_id~212.
`
`* cited by examiner
`
`

`

`Case 1:22-cv-01257-GBW Document 1-2 Filed 09/22/22 Page 4 of 29 PageID #: 83
`
`U.S. Patent
`
`Feb.26,2008
`
`Sheet 1 of 12
`
`US 7,336,814 B2
`
`~
`
`~
`
`' ' '
`
`'
`
`~
`~
`I
`\, ,,,. .. /'
`,,"'
`
`/ /
`
`,,"
`
`,~
`
`~ ~
`
`~ co ;r
`
`~
`~
`
`~
`
`~
`"'or::t-
`
`C:) -)
`
`' ' ', _,._
`..-
`
`/
`I
`I
`~
`~
`
`'
`
`'
`
`' ,
`,
`
`

`

`Case 1:22-cv-01257-GBW Document 1-2 Filed 09/22/22 Page 5 of 29 PageID #: 84
`
`"'""' ~ = N
`00
`0--,
`-....l w w
`d r.,;_
`
`....
`0 ....
`N
`.....
`rJJ =- ('D
`
`('D
`
`N
`
`QO
`0
`0
`N
`~Cl's
`N
`?'
`('D
`"f'j
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`'--10
`
`I
`
`40a
`
`I
`
`I
`
`20
`
`18
`\
`
`li
`
`FIG. 2A
`
`I
`
`I
`
`I
`
`I
`
`14
`
`42b
`
`\__
`
`42c
`
`I
`
`\
`
`24a
`
`42g/9
`
`42i
`
`~42f
`
`42d
`
`/
`
`120---.....
`
`~~42e
`
`42h
`
`42c
`
`24c
`
`40b
`
`

`

`Case 1:22-cv-01257-GBW Document 1-2 Filed 09/22/22 Page 6 of 29 PageID #: 85
`
`"'""' ~ = N
`00
`0--,
`-....l w w
`d r.,;_
`
`N
`
`~
`
`('D
`('D
`
`....
`0 ....
`.....
`rJJ =(cid:173)
`
`QO
`0
`0
`N
`~Cl's
`N
`?'
`('D
`"f'j
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`'--10
`
`40a
`
`20
`
`FIG. 2B
`
`14
`
`24a
`
`44
`
`12a
`
`42h
`
`42f
`
`42d
`
`;r;::--42e
`
`24c
`
`-40b
`
`24b
`
`

`

`Case 1:22-cv-01257-GBW Document 1-2 Filed 09/22/22 Page 7 of 29 PageID #: 86
`
`U.S. Patent
`
`Feb.26,2008
`
`Sheet 4 of 12
`
`US 7,336,814 B2
`
`..Q
`~
`
`;§
`
`<..>
`
`~
`~ ~
`
`t:,
`~
`
`

`

`Case 1:22-cv-01257-GBW Document 1-2 Filed 09/22/22 Page 8 of 29 PageID #: 87
`
`U.S. Patent
`
`Feb.26,2008
`
`Sheet 5 of 12
`
`US 7,336,814 B2
`
`CALIBRATE IMAGE SENSOR(S)
`
`1100
`
`102
`
`----------
`
`TRAIN SYSTEM
`
`r---.- 104
`
`POSE ESTIMATION
`(DETERMINE TARGET OBJECT'S 3-D POSE) ~ 106
`
`FIG. 4
`
`1102
`
`INTRINSIC PARAMETER FOR
`DETERMINE
`EACH
`IMAGE SENSOR
`
`110
`
`----------
`
`DETERMINE POSE OF ONE OR MORE
`IMAGE SENSORS
`
`112
`
`----------
`
`FIG. 5
`
`

`

`Case 1:22-cv-01257-GBW Document 1-2 Filed 09/22/22 Page 9 of 29 PageID #: 88
`
`U.S. Patent
`
`Feb.26,2008
`
`Sheet 6 of 12
`
`US 7,336,814 B2
`
`1104
`
`ACQUIRE ONE VIEW AND TRAIN FEATURES ~120
`
`DETERMINE NUMBER OF ADDITIONAL VIEWS
`
`---..__,,121
`
`CHANGE RELATIVE OBJECT POSE TO
`IMAGE SENSOR(S) (RIGID) AND ACQUIRE ~122
`VIEW(S)
`
`EXTRACT FEATURES
`
`~124
`
`/128
`
`COMPUTE 3D FEATURE POSITIONS IN
`RESPECTIVE
`IMAGE SENSOR COORDINATE
`FRAMES (CREATE LOCAL MODELS)
`
`126
`
`/
`SPARSE MODEL {
`INFORMATION
`
`I
`I
`
`FIG. 6
`
`

`

`Case 1:22-cv-01257-GBW Document 1-2 Filed 09/22/22 Page 10 of 29 PageID #: 89
`
`"'""' ~ = N
`00
`0--,
`-....l w w
`d r.,;_
`
`....
`0 ....
`-....J
`.....
`rJJ =- ('D
`
`('D
`
`N
`
`QO
`0
`0
`N
`~Cl's
`N
`?'
`('D
`"f'j
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`FIG. 7
`
`I
`
`GO TO COMPUTE POSITION
`
`j= 1 .. c
`k=1 .. m
`i=1 .. nf
`Fi vk,cj (x,y)
`STORE
`
`150
`
`>
`IMAGES FOR CURRENT VIEW? >
`
`,,.-f 56
`
`cts4
`
`152
`
`l
`
`NO
`
`MORE VIEWS?
`
`t NO
`
`I
`
`YES (
`
`MORE
`
`YES (
`
`148
`
`146
`
`144
`
`INFORMATION
`
`EXTRACT FEATURE
`
`CORRESPONDING TO THE
`IMAGE SENSOR
`LOCATE FEATURES PREVIOUSLY TRAINED
`
`IMAGE
`
`THAT ACQUIRED THE
`
`(from 1 to c)
`IMAGE OF CURRENT VIEW
`
`TAKE NEXT
`
`l'---142
`
`TAKE NEXT VIEW (from 1 to m)
`
`FROM ALL VIEWS
`
`ALL IMAGES
`
`1124
`
`140
`
`

`

`Case 1:22-cv-01257-GBW Document 1-2 Filed 09/22/22 Page 11 of 29 PageID #: 90
`
`~
`a
`
`O
`QO
`
`g2
`
`QO
`0
`0
`N
`~a-.
`~
`O"
`('D
`"'f"l
`
`""""
`
`=
`;-
`~
`~
`e
`
`"'""' ~ = N
`00
`0--,
`-....l w w
`d r.,;_
`
`FIG. ;A ---J---------------------"
`
`-----------
`
`Fi vm,cc ( X, y) IMAGE COORDINATES .••
`
`LIST OF FEATURES INFORMATION FROM
`
`VIEW m, CAMERA c:
`
`•
`
`~ L
`,,-1
`I
`I
`I
`I
`I
`
`•
`
`•
`
`16~c
`/
`
`A
`
`i
`
`:
`______ _j
`
`____ J________________
`
`----
`
`L_ --------
`
`Fi vm,c1 ( X, y) IMAGE COORDINATES ...
`
`i
`
`I
`I
`
`i
`
`1
`
`FEATURES EXTRAcf rn FROM CAMERA c
`
`IMAGE COORDINATES...
`
`Fiv2,cc(x,y)
`
`LIST OF FEATURES INFORMATION FROM
`
`VIEW 2, CAMERA c:
`
`VIEW m
`
`i
`
`I
`I
`
`i
`• : • • • • •
`
`: VIEW 2
`
`I
`I
`
`LIST OF FEATURES INFORMATION FROM
`
`VIEW m, CAMERA 1:
`
`FEATURES EXTRAcfrn FROM CAMERA 1
`
`IMAGE COORDINATES...
`
`Fiv2,c1(x,y)
`
`/
`
`INFORMATION FROM
`
`LIST OF FEATURES
`
`VIEW 2, CAMERA 1:
`
`I
`'---------------' I
`1
`
`:
`:
`
`I
`I
`I
`I
`
`:
`:
`
`I
`I
`
`I
`I
`I
`I
`
`:
`:
`
`:
`I
`I
`I
`
`VIEW 1 CAMERA 1·
`
`'
`
`1
`
`FEATURES FROM
`
`f I v1 ,c1 (x y)
`
`•
`
`'
`
`I
`----------'--~ I
`
`IMAGE COORDINATES...
`
`fiv1,cc(x,y)
`
`:
`
`:
`
`IMAGE COORDINATES...
`
`Fiv1,c1(x,y)
`
`1
`I ~------------~ I
`
`LIST OF FEATURES INFORMATION FROM
`
`VIEW 1, CAMERA c:
`
`• : • • • • : •
`: VIEW 1 :
`
`~..,
`
`INFORMATION FROM
`
`LIST OF FEATURES
`
`VIEW 1, CAMERA 1:
`
`I
`I
`,--------------------,
`
`CAMERA c
`
`l /;
`1600
`
`CAMERA 1
`
`,,,--128
`
`

`

`Case 1:22-cv-01257-GBW Document 1-2 Filed 09/22/22 Page 12 of 29 PageID #: 91
`
`....
`0 ....
`.....
`rJJ =(cid:173)
`
`1,0
`
`('D
`('D
`
`QO
`0
`0
`N
`~Cl's
`N
`?'
`('D
`"f'j
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`"'""' ~ = N
`00
`0--,
`-....l w w
`d r.,;_
`
`FIG. BB
`
`N
`
`n=1 .. TOTAL NUMBER OF FEATURES FOR ALL IMAGE SENSOR(S)
`b=1 .. c
`a=1 .. m
`COORDINATE SYSTEMS
`RELATIVE TO THE RESPECTIVE
`IMAGE SENSOR
`P: 30 POSITION (x,y,z) OF A GIVEN FEATURE
`WHERE:
`pn va, cb
`
`170
`
`P(FEA TURE j, FEATURE i)
`
`SPARSE MODEL INFORMATION:
`
`126
`
`168
`
`FEATURES AND/OR IMAGE SENSOR(S)
`
`INFORMATION OF ALL THE
`
`THE 30 POSITION
`
`MARQUARDT) TO SOLVE THE ABOVE EQUATIONS FOR
`USE A MINIMIZATION ALGORITHM (LIKE LEVENBNERG
`
`164
`
`P(Fi vs,cp, ... Fi vs,cr) = P(Fi vt,cp, ... Fi vt,cr)
`
`IN ANOTHER VIEW
`
`PHYSICAL PARAMETERS
`
`ANO OTHER FEATURES
`IN ONE VIEW WITH THE SAME
`THE PHYSICAL PARAMETERS BETWEEN EACH FEATURE
`CONSTRUCT NONLINEAR EQUATIONS SET BY EQUATING
`
`

`

`Case 1:22-cv-01257-GBW Document 1-2 Filed 09/22/22 Page 13 of 29 PageID #: 92
`
`"'""' ~ = N
`00
`0--,
`-....l w w
`d r.,;_
`
`....
`0 ....
`0
`....
`.....
`rJJ =(cid:173)
`
`('D
`('D
`
`N
`
`QO
`0
`0
`N
`~Cl's
`N
`?'
`('D
`"f'j
`
`~ = ~
`
`~
`~
`~
`•
`00
`
`e •
`
`FIG. 9
`
`204
`
`202
`
`DETERMINE 3D OBJECT POSE
`
`USE TRANSFORMED FEATURES POSITIONS TO
`
`FRAME TO THE COMMON REFERENCE FRAME
`
`DESIRED VIEW ( p ... va, c ... ) FROM EACH CAMERA
`
`TRANSFORM FEATURES POSITIONS FOR THE
`
`n=1 .. TOTAL NUMBER OF FEATURES FOR ALL IMAGE SENSOR(S)
`b=1 .. c
`a=1 .. m
`COORDINATE SYSTEMS
`RELATIVE TO THE RESPECTIVE
`IMAGE SENSOR
`P: 3D POSITION (x,y,z) OF A GIVEN FEATURE
`WHERE:
`pn va, cb
`
`106a
`
`f
`
`200
`
`

`

`Case 1:22-cv-01257-GBW Document 1-2 Filed 09/22/22 Page 14 of 29 PageID #: 93
`
`U.S. Patent
`
`Feb.26,2008
`
`Sheet 11 of 12
`
`US 7,336,814 B2
`
`220
`
`1 106b
`
`pnva, cb
`WHERE:
`P: 3D POSITION (x,y,z) OF A GIVEN FEATURE
`RELATIVE TO THE RESPECTIVE
`IMAGE SENSOR
`COORDINATE SYSTEMS
`a=1 .. m
`b=1 .. c
`n=1 .. TOTAL NUMBER OF FEATURES FOR ALL IMAGE
`SENSOR(S)
`
`SELECT AT LEAST THREE NON COLINEAR
`FEATURES Fi 1 Fi 2 Fi3
`
`I
`
`I
`
`224a
`
`222
`
`224b
`
`TAKE SELECTED FEATURES
`Fi2
`Fi3
`( Fil
`va,c .. ,
`va,c .. ,
`va,c ..
`FROM THE VIEW THE OBJECT WAS
`IN THE OBJECT REFERENCE POSITION
`
`)
`
`TAKE SELECTED FEATURES
`Fi2
`Fi3
`( Fi1
`vb,c .. ,
`vb,c .. ,
`vb,c ..
`FROM THE VIEW THE OBJECT
`WAS
`IN THE CURRENT POSITION
`
`)
`
`226a
`
`226b
`
`TRANSFORM SELECTED FEATURES
`POSITIONS (P·" VO, c ... )
`FROM EACH CAMERA FRAME
`TO THE COMMON REFERENCE FRAME
`
`TRANSFORM SELECTED FEATURES
`POSITIONS ( p .. · vb, c ... )
`FROM EACH CAMERA FRAME
`TO THE COMMON REFERENCE FRAME
`
`228a
`
`228b
`
`IN THE SAME
`USE THESE POINTS
`COMMON REFERENCE FRAME TO
`BUILD OBJECT REFERENCE
`FRAME
`
`IN THE SAME
`USE THESE POINTS
`COMMON REFERENCE FRAME TO
`BUILD CURRENT OBJECT FRAME
`
`FIG. 10
`
`

`

`Case 1:22-cv-01257-GBW Document 1-2 Filed 09/22/22 Page 15 of 29 PageID #: 94
`
`U.S. Patent
`
`Feb.26,2008
`
`Sheet 12 of 12
`
`US 7,336,814 B2
`
`220
`
`f
`
`106c
`
`pn va, cb
`WHERE:
`P: 3D POSITION (x,y,z) OF A GIVEN FEATURE
`RELATIVE TO THE RESPECTIVE
`IMAGE SENSOR
`COORDINATE SYSTEMS
`a=1 .. m
`b=1 .. c
`n=1 .. TOTAL NUMBER OF FEATURES FOR ALL IMAGE
`SENSOR(S)
`
`SELECT AT LEAST THREE NON COUNEAR
`FEATURES Fi1 Fi2 Fi3
`'
`'
`
`224c
`
`222
`
`224b
`
`TAKE SELECTED FEATURES
`)
`Fi2
`Fi3
`(F i1
`va,c .. ,
`va,c .. ,
`va,c ..
`FROM THE VIEW THE OBJECT WAS
`IN THE PREVIOUS POSITION
`
`TAKE SELECTED FEATURES
`Fi2
`Fi3
`(Fi1
`vb,c .. ,
`vb,c .. ,
`vb,c ..
`FROM THE VIEW THE OBJECT
`WAS
`IN THE CURRENT POSITION
`
`)
`
`226a
`
`226b
`
`TRANSFORM SELECTED FEATURES
`POSITIONS (p ... va, c ... )
`FROM EACH CAMERA FRAME
`TO THE COMMON REFERENCE FRAME
`
`TRANSFORM SELECTED FEATURES
`POSITIONS (p ... vb, c ... )
`FROM EACH CAMERA FRAME
`TO THE COMMON REFERENCE FRAME
`
`228a
`
`228b
`
`IN THE SAME
`USE THESE POINTS
`COMMON REFERENCE FRAME TO
`BUILD OBJECT REFERENCE
`FRAME
`
`IN THE SAME
`USE THESE POINTS
`COMMON REFERENCE FRAME TO
`BUILD CURRENT OBJECT FRAME
`
`FIG. 11
`
`

`

`Case 1:22-cv-01257-GBW Document 1-2 Filed 09/22/22 Page 16 of 29 PageID #: 95
`
`US 7,336,814 B2
`
`1
`METHOD AND APPARATUS FOR
`MACHINE-VISION
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`This application claims benefit under 35 U.S.C. 119(e) to
`U.S. provisional application Ser. No. 60/587,488, filed Jul.
`14, 2004.
`
`BACKGROUND OF THE INVENTION
`
`5
`
`2
`including cameras and lasers, susceptibility to damage or
`misalignment when operating in industrial environments as
`well as posing of a potential safety hazard when laser light
`sources are deployed in proximity of human operators.
`Based on the above considerations it is desirable to devise
`a three-dimensional robot guidance system that eliminates
`the need for stereo camera pairs and the need for the use of
`structured light and specialized sensors. Such a system
`would increase accuracy, simplify setup and maintenance
`10 and reduce hardware costs.
`Prior methods have been developed that utilize a single
`camera to view each region/feature of the object in order to
`calculate the 3D pose of the object. For example see U.S.
`Pat. No. 4,942,539 McGee, and European Patent No.
`15 0911603Bl Ersu. However these and similar methods
`require the calibration of all cameras relative to a common
`coordinate frame such as a robot. In practice such a require(cid:173)
`ment is cumbersome and time-consuming to fulfill and
`difficult to automate. These methods also require a priori
`20 knowledge of the geometrical relationships between all
`object features used. One source for such data is object
`Computer Aided Design (CAD) models; however, such data
`files are often not readily available. In the absence of CAD
`data, past systems have relied on direct object measurement
`25 using a coordinate measurement machine or a robot
`equipped with a pointing device. This process is difficult and
`error prone especially in the case of large objects with
`features that are scattered in different regions.
`It is therefore highly desirable to develop a three-dimen-
`30 sional robot guidance system that in addition to eliminating
`the need for stereo cameras and lasers, also eliminates the
`need for inter-camera calibration and the need for a priori
`knowledge of geometrical relationships between all object
`features.
`
`1. Field of the Invention
`This disclosure relates to the field of machine vision,
`which may be useful in robotics, inspection or modeling.
`2. Description of the Related Art
`Increasingly more manufacturing operations are per(cid:173)
`formed with the aid of industrial robots. Robots that had
`traditionally been used as blind motion playback machines
`are now benefiting from intelligent sensor-based software to
`adapt to changes in their surroundings. In particular, the use
`of machine vision has been on the rise in industrial robotics.
`A typical vision guided robotic system analyzes image(s)
`from one or more cameras to arrive at such information as
`the position and orientation of a workpiece upon which the
`robotic tool is to operate.
`Early implementations of vision guided robots have pro(cid:173)
`vided only limited part pose information, primarily in the
`two-dimensional space whereby the movement of a given
`part is constrained to a planar surface. For example see U.S.
`Pat. No. 4,437,114 LaRussa. However, many robotic appli(cid:173)
`cations require the robot to locate and manipulate the target
`workpiece in three dimensions. This need has sparked many
`attempts at providing various three-dimensional guidance
`capabilities. In many past cases, this has involved using two 35
`or more cameras that view overlapping regions of the object
`of interest in what is known as a stereo configuration. The
`overlapping images or fields-of-view contain many of the
`same object features viewed from two or more vantage
`points. The difference amongst the apparent position of 40
`corresponding features in each of the images i.e., the par(cid:173)
`allax, is exploited by these methods to calculate the three
`dimensional coordinates of such features. For examples see
`U.S. Pat. No. 4,146,924 Birk et al., and U.S. Pat. No.
`5,959,425 Bieman et al.
`Many drawbacks exist that render stereo based systems
`impractical for industrial applications. The measurement
`error in such systems increases rapidly in response to image
`feature detection errors; these systems also require exactly
`known geometrical relationships between camera pairs. Fur- 50
`thermore stereo methods require the use of at least double
`the number of cameras which drives up the cost, complexity
`and the need for calibration.
`Other attempts at locating objects with multiple cameras
`in the past have taken advantage of video cameras in
`combination with laser light projectors that project various
`stationary or moving patterns such as stripes, cross-hairs and
`the like upon the object of interest. These systems typically
`involve a combination of lasers and cameras that must be
`calibrated relative to a common coordinate system and rely
`on specific assumptions about the geometry of the object of
`interest to work. For example see U.S. Pat. No. 5,160,977
`Utsumi.
`Drawbacks of such attempts include the need for expen(cid:173)
`sive specialized sensors as opposed to use of standard
`off-the-shelf hardware, the need for knowledge of exact
`geometric relationships between all elements of the system
`
`BRIEF SUMMARY OF INVENTION
`
`In one aspect, a method useful in machine-vision of
`objects comprises acquiring a number of images of a first
`view of a training object from a number of cameras; iden(cid:173)
`tifying a number of features of the training object in the
`acquired at least one image of the first view; employing at
`least one of a consistency of physical relationships between
`some of the identified features to set up a system of
`45 equations, where a number of unknowns is not greater than
`a number of equations in the system of equations; and
`automatically computationally solving the system of equa(cid:173)
`tions. The method may further determine a number of
`additional views to be obtained based at least in part on the
`number of image sensors, the number of features identified,
`the number of features having an invariant physical rela-
`tionship associated thereto, and a type of the invariant
`physical relationship associated with the features, sufficient
`to provide a system of equations and unknowns where the
`55 number of unknowns is not greater than the number of
`equations. Where the invariant physical relationships are
`distances, the number of views may, for example, be deter(cid:173)
`mined by computationally solving the equation m~(f2-f-
`2k-2r+6(c-ck))/(f2-3f)-1, where mis the number of views,
`60 f the number of features, k the number of known distances
`between pairs of the features, r is the number of rays with a
`known distance between a feature and an image sensor, c is
`the number of image sensors and ck is the number of known
`transformation between an imager sensor reference frame
`65 and a common reference frame.
`In another aspect, a machine-vision system comprises at
`least one image sensor operable to acquire images of a
`
`

`

`Case 1:22-cv-01257-GBW Document 1-2 Filed 09/22/22 Page 17 of 29 PageID #: 96
`
`US 7,336,814 B2
`
`3
`trammg object and of target objects; processor-readable
`medium storing instructions for facilitating machine-vision
`for objects having invariant physical relationships between
`a number of features on the objects, by: acquiring a number
`of images of a first view of a training object from a number
`of cameras; identifying a number of features of the training
`object in the acquired at least one image of the first view;
`employing at least one of a consistency of physical relation(cid:173)
`ships between some of the identified features to set up a
`system of equations, where a number of unknowns is not
`greater than a number of equations in the system of equa(cid:173)
`tions; and automatically computationally solving the system
`of equations; and a processor coupled to receive acquired
`images from the at least one image sensor and operable to
`execute the instructions stored in the processor-readable 15
`medium.
`In still another aspect, a processor readable medium stores
`instructions for causing a processor to facilitate machine(cid:173)
`vision for objects having invariant physical relationships
`between a number of features on the objects: by acquiring a 20
`number of images of a first view of a training object from a
`number of cameras; identifying a number of features of the
`training object in the acquired at least one image of the first
`view; employing at least one of a consistency of physical
`relationships between some of the identified features to set 25
`up a system of equations, where a number of unknowns is
`not greater than a number of equations in the system of
`equations; and automatically computationally solving the
`system of equations.
`In a yet another aspect, a method useful in machine-vision 30
`of objects comprises acquiring a number of images of a first
`view of a training object from a number of cameras; iden(cid:173)
`tifying a number of features of the training object in the
`acquired at least one image of the first view; associating
`parameters to less than all of the identified features which 35
`parameters define an invariant physical relationship between
`either the feature and at least one other feature, the feature
`and the at least one camera, or between the at least one
`camera and at least another camera where an invariant
`physical relationship between each one of the features and at 40
`least one other feature is not known when associating the
`parameters before a runtime; determining a number of
`additional views to be obtained based at least in part on the
`number of cameras, the number of features identified, and
`the number of features having parameters associated thereto, 45
`sufficient to provide a system of equations and unknowns
`where the number of unknowns is not greater than the
`number of equations; and acquiring at least one image of
`each of the number of additional views of the training object
`by the at least one camera; identifying at least some of the 50
`number of features of the training object in the acquired at
`least one image of the number of additional views of the
`training object.
`In even another aspect, a method useful in machine-vision
`for objects having invariant physical relationships between 55
`a number of features on the objects comprises in a pre(cid:173)
`runtime environment: acquiring at least one image of a first
`view of a training object by at least one image sensor;
`identifying a number of features of the training object in the
`acquired at least one image of the first view; and associating 60
`a number of parameters to less than all of the identified
`features which define an invariant physical relationship
`between the either the feature and at least one other feature
`or between the feature and the at least one image sensor;
`determining a number of additional views to be obtained
`based at least in part on the number of image sensors
`acquiring at least one image, the number of features of the
`
`4
`training object identified, the number of features having
`parameters associated therewith, and a type of invariant
`physical relationship associated with each of the parameter;
`acquiring at least one image of a second view of the training
`5 object by the at least one image sensor; and identifying at
`least some of the number of features of the training object in
`the acquired at least one image of the second view; and in
`at least one of a pre-run time environment or a runtime
`environment, computationally determining a local model
`10 using the identified features in each of a number of respec(cid:173)
`tive image sensor coordinate frames.
`In still another aspect, a machine-vision system comprises
`at least one image sensor operable to acquire images of a
`training object and of target objects; processor-readable
`medium storing instructions for facilitating pose estimation
`for objects having invariant physical relationships between
`a number of features on the objects, by: in a pre-runtime
`environment: acquiring at least one image of a first view of
`a training object by at least one image sensor; identifying a
`number of features of the training object in the acquired at
`least one image of the first view; and associating a number
`of parameters to less than all of the identified features which
`define an invariant physical relationship between the either
`the feature and at least one other feature or between the
`feature and the at least one image sensor; determining a
`number of additional views to be obtained based at least in
`part on the number of image sensors acquiring at least one
`image, the number of features of the training object identi(cid:173)
`fied, the number of features having parameters associated
`therewith, and a type of invariant physical relationship
`associated with each of the parameter; acquiring at least one
`image of a second view of the training object by the at least
`one image sensor; and identifying at least some of the
`number of features of the training object in the acquired at
`least one image of the second view; and in at least one of a
`pre-run time environment or a runtime environment, com(cid:173)
`putationally determining a local model using the identified
`features in each of a number of respective image sensor
`coordinate frames; and a processor coupled to receive
`acquired images from the at least one image sensor and
`operable to execute the instructions stored in the processor(cid:173)
`readable medium.
`In a further aspect, a processor readable medium stores
`instructions for causing a processor to facilitate machine(cid:173)
`vision for objects having invariant physical relationships
`between a number of features on the objects, by: in a
`pre-runtime environment: acquiring at least one image of a
`first view of a training object by at least one image sensor;
`identifying a number of features of the training object in the
`acquired at least one image of the first view; and associating
`parameters to less than all of the identified features which
`define a physical relationship between the either the feature
`and at least one other feature or between the feature and the
`at least one image sensor; and determining a number of
`additional views to be obtained based at least in part on the
`number of image sensors acquiring at least one image and
`the number of features of the training object identified;
`acquiring at least one image of a second view of the training
`object by the at least one image sensor; and identifying at
`least some of the number of features of the training object in
`the acquired at least one image of the second view; and in
`at least one of a pre-run time environment or a runtime
`65 environment, computationally determining a local model
`using the identified features in each of a number of respec(cid:173)
`tive image sensor coordinate frames.
`
`

`

`Case 1:22-cv-01257-GBW Document 1-2 Filed 09/22/22 Page 18 of 29 PageID #: 97
`
`US 7,336,814 B2
`
`5
`BRIEF DESCRIPTION OF THE SEVERAL
`VIEWS OF THE DRAWING(S)
`
`In the drawings, identical reference numbers identify
`similar elements or acts. The sizes and relative positions of
`elements in the drawings are not necessarily