`

`

`

`. '
`
`5
`
`10
`
`METHOD AND APPARATUS FOR SINGLE IMAGE 3D
`VISION GUIDED ROBOTICS
`
`Cross.,.Reference to Related Applications
`
`This application is a continuation-in-part of United States patent
`application no. 10/153680, filed May 24, 2002 which is pending,_
`
`Technical Field
`
`The invention relates to the field of vision guided robotics, and
`more particularly to a method and apparatus for single image three dimensional
`vision guided robotics.
`
`15 Background
`
`Robots have long been widely used in manufacturing processes
`for many applications. Many different types of sensors are used to guide robots
`but machine vision is increasingly being used to guide robots in their tasks. ·
`20 Typically such machine vision is used in a two-dimensional application wherein
`the target object need only be located in an x-y plane, using a single camera.
`For example see United States patent no. 4,437,114 LaRussa. However many
`robotic applications require the robot to locate and manipulate the target in
`three dimensions. In the past this has typically involved using two or more
`cameras. For example see United States patent no. 4,146,924 Birk et al.; and
`United States patent no. 5,959,425 Bieman et al. In order to reduce hardware
`costs and space requirements it is preferable to use a single camera. Prior
`single camera systems however have used laser triangulation which involves
`expensive specialized sensors, must be rigidly packaged to maintain geometric
`relationships, require sophisticated inter-tool calibration methods and tend to be
`susceptible to damage or misalignment when operating in industrial environ(cid:173)
`ments.
`
`30
`
`25
`
`35
`
`Target points on the object have also been used to assist in
`determining the location in space of the target object using single or multiple
`cameras. See United States patents no. 4,219,847 Pinkney et al. and United
`
`ABB Inc. Exhibit 1002, Page 3 of 996
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`- 2 -
`
`States patents no. 5,696,673; 5,956,417; 6,044,183 and 6,301,763 all of Pryor
`and United States patent no. 4,942,539 of McGee et al. Typically these
`methods involve computing the position of the object relative to a previous
`position, which requires knowledge of the 3D pose of the object at the starting
`point. These methods also tend to not provide the accuracy and repeatability
`required by industrial applications. There is therefore a need for a method for
`calculating the 3D pose of objects using only standard video camera equipment
`that is capable of providing the level of accuracy and repeatability required for
`vision guidance of robots as well as other applications requiring 3D pose
`information of objects.
`
`5
`
`10
`
`Summary of Invention
`
`15
`
`A method of three-dimensional object location and guidance to
`allow robotic manipulation of an object with variable position and orientation
`by a robot using a sensor array is provided. the method comprises: a) calibrat(cid:173)
`ing the sensor array to provide a Robot - Eye Calibration by finding the intrinsic
`parameters of said sensor array and the position of the sensor array relative to a
`20 preferred robot coordinate system ("Robot Frame") by placing a calibration
`model in the field of view of said sensor array; (b) training object features by:
`i) positioning the object and the sensor array such that the object is located in
`the field of view of the sensor array and acquiring and forming an image of the
`object; ii) selecting at least 5 visible object features from the image; iii)
`creating a 3D model of the object ("Object Model") by calculating the 3D
`position of each feature relative to a coordinate system rigid to the object
`("Object Space"); (c) training a robot operation path by: (i) computing the
`"Object Space-> Sensor Array Space" transformation using the "Object
`Model" and the positions of the features in the image; (ii) computing the
`"Object Space" position and orientation in "Robot Frame" using the transfor(cid:173)
`mation from "Object Space-> Sensor Array Space" and "Robot - Eye Calibra(cid:173)
`tion"; (iii) coordinating the desired robot operation path with the "Object
`Space"; (d) carrying out object location and robot guidance by:
`
`25
`
`30
`
`ABB Inc. Exhibit 1002, Page 4 of 996
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`- 3 -
`
`(i) acquiring and forming an image of the object using the sensor array, search(cid:173)
`ing for and finding said at least 5 trained features;
`ii) with the positions of features in the image and the corresponding "Object
`Model" as determined in the training step, computing the object location as the
`transformation between the "Object Space" and the "Sensor Array" and the
`transformation between the "Object Space" and "Robot Frame"; (iii)
`communicating said computed object location to the robot and modifying robot
`path points according to said computed object location.
`
`5
`
`10
`
`The invention further provides a system for carrying out the
`foregoing method.
`
`Brief Description of Drawings
`
`15
`
`In drawings which illustrate a preferred embodiment of the
`
`invention:
`
`20
`
`Fig. 1 is a perspective view of a vision-guided robot;
`Fig. 2 is a schematic diagram illustrating the relative frames of
`reference for calculating the position of the object;
`Fig. 3 is a schematic diagram illustrating the calculation of the
`intended operation path on the object;
`Fig. 4 is a representation of the operator's computer screen for
`selecting reference features on the object;
`Fig. 5 is a flow chart illustrating the calibration of the camera
`25 mounted on the robot arm;
`Fig. 6 is a flow chart illustrating the method of teaching the object
`features and handling path; and
`Fig. 7 is a flow chart illustrating the method of object positioning
`and handling.
`
`30
`
`Description
`
`Throughout the following description, specific details are set
`forth in order to provide a more thorough understanding of the invention.
`
`ABB Inc. Exhibit 1002, Page 5 of 996
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`-4-
`
`However, the invention may be practiced without these particulars. In other
`instances, well known elements have not been shown or described in detail to
`avoid unnecessarily obscuring the invention. Accordingly, the specification
`and drawings are to be regarded in an illustrative, rather than a restrictive,
`sense.
`
`Fig. 1 shows a robot 10 having a manipulating arm 12, base 22
`and attached tool 14 with camera 16 mounted thereon. Tool 14 is designed to
`manipulate a part or target object 18. Camera 16 is preferably a commercially
`available analog or digital video camera. Camera 16 and robot 10 are electri(cid:173)
`cally connected to computer control station 24 for communication of data back
`and forth.
`
`The method is performed in the main steps described as follows:
`
`a)
`b)
`c)
`
`calibration of the camera;
`teaching the features on the object; and
`finding the three-dimensional pose of the object and using
`this information to guide the robot to approach the object
`to perform any operations (e.g. handling, cutting etc.).
`
`In the following discussion the following terms have the foll(cid:173)
`owing meanings, as illustrated in Fig. 2:
`
`three-dimensional rotation & translation be(cid:173)
`Transformation:
`tween two spaces;
`Tool 14: the tool the robot is using for performing the handling,
`cutting or other robotic operations, having an operating end or
`"end-effector";
`Camera Space 26: a reference frame defined with respect to a
`point on, and therefore rigid to, the camera 16;
`Tool Space 28: a reference frame defined with respect to a point
`on, and oriented along the direction of the end-effector and there(cid:173)
`fore rigid to, the tool 14;
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`ABB Inc. Exhibit 1002, Page 6 of 996
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`- 5 -
`
`Robot base Space 30: a reference frame defined with respect to
`the physical base of the robot arm and therefore rigid to, the robot
`base 30;
`Robot Frame: a preferred robot coordinate system
`Calibration Model or Calibration Pattern or Calibration Template:
`a physical object with features of whose relative geometry is
`known
`Sensor Array: a collection of one or more sensors capable of
`forming a single image
`Training Space 32: a reference frame defined with respect to a
`point on the calibration template 40, and aligned to its main axes;
`Object Space 36: a reference frame defined with respect to, and
`therefore rigid to, the object 18;
`Object Frame 34: a reference frame defined with respect to a
`point on, and therefore rigid to, the object 18;
`Camera calibration intrinsic parameters: focal length, image
`center, real pixel size, and radial and tangential distortion for the
`lens;
`Camera calibration extrinsic parameters: rotation and translation
`of an external space relative to the Camera Space 26.
`
`Calibration
`
`The calibration process involves: i) finding the camera intrinsic
`parameters and ii) the position of the camera relative to the tool of the robot
`("hand- eye" calibration). The position of the camera in the "Training Space",
`which is a space rigid to the place where the object will be trained is also
`determined. A general explanation of the basic calibration algorithms and
`descriptions of the variables can be found in the following publications:
`
`"An Efficient and Accurate Camera Calibration Technique
`for 3D Machine Vision", Roger Y. Tsai, Proceedings of
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`ABB Inc. Exhibit 1002, Page 7 of 996
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`- 6 -
`
`IEEE Conference on Computer Vision and Pattern Recog(cid:173)
`nition, Miami Beach, FL, 1986, pages 364-374;
`
`"A Versatile Camera Calibration Technique for High(cid:173)
`Accuracy 3D Machine Vision Metrology Using Off-the(cid:173)
`Shelf TV Cameras and Lenses", Roger Y. Tsai, IEEE
`Journal of Robotics and Automation, Vol. RA-3, No. 4,
`August 1987, pages 323-344.
`
`"Tsai's Camera Calibration Method Revisited"
`http://www.ai.mit.edu/people/bkph/papers/tsaiexplain.pdf
`
`5
`
`10
`
`Tsai's camera model is based on the pin-hole model of perspec(cid:173)
`tive projection. Given the position of a point in 3D world coordinates, the
`15 model predicts the position of the point's image in 2D pixel coordinates. Tsai's
`model has 11 parameters: five internal (also called intrinsic or interior) parame(cid:173)
`ters:
`
`20
`
`25
`
`30
`
`v)
`
`f - effective focal length of the pin hole camera;
`i)
`ii)
`kappal - 1st order radial lens distortion coefficient;
`iii and iv) Cx, Cy - coordinates of center of radial lens distortion
`and the piercing point of the camera coordinate frame's
`Z axis with the camera's sensor plane;
`sx - scale factor to account for any uncertainty due to
`framegrabber horizontal scanline resampling,
`and six external (also called extrinsic or exterior) parameters:
`i)
`Rx, Ry, Rz - rotation angles of the transformation be-
`tween the world and camera coordinate frames, and
`Tx, Ty, Tz - translation components of the transfor(cid:173)
`mation between the world and camera coordinate fra(cid:173)
`mes.
`
`ii)
`
`The internal parameters describe how the camera forms an image
`while the external parameters describe the camera's pose (i.e. position and
`
`ABB Inc. Exhibit 1002, Page 8 of 996
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`- 7 -
`
`orientation) in the world coordinate frame. Calibration data for the model
`consists of 3D (x,y,z) world coordinates of a feature point (in mm. for example)
`and corresponding 2D coordinates (Xf,Yf) (typically in pixels) of the feature
`point in the image. Two forms of calibration are possible:
`coplanar - the calibration points lie in a single plane in 3D, and
`non-coplanar - the calibration points occupy a 3D volume.
`
`5
`
`As illustrated in Fig. 2 and 5, the first step in calibration is to
`position the camera 16 on the robot arm 12 so it is orthogonal to a template 40
`10 using the mirror approach, and defining the "Training Space" for the robot
`aligned with the template 40 used for calibration. A calibration template 40 is
`placed on the floor or other planar surface. It is a flat piece of paper, plastic or
`other medium having a series of fixed detectable features such as a grid of dots
`or squares. The camera 16 is positioned so that it is orthogonal to the plane of
`the calibration template 40, that is, so the camera's imaging plane is parallel to
`the template 40. The mirror approach for obtaining orthogonality comprises
`placing a flat mirror on the grid of the template 40 and centering the lens on the
`mirror in the center of the grid.
`
`15
`
`20
`
`25
`
`30
`
`Next the camera intrinsic parameters and the "Camera-> Training
`Space" transformation are computed considering the Training Space. Next,
`the "Camera-> Tool" transformation is computed using the "Camera->
`Training Space" transformation and inquiring the robot about the "Tool"
`position in "Training Space". To calculate the "Camera-> Tool" transforma-
`tion manually, the operator first touches 3 identifiable points on the grid having
`known coordinates with the tool. Next the operator stores images of the grid
`from at least 2 and preferably 4 measured heights above the grid.
`
`Alternatively, the calibration can be done automatically to
`compute both the camera intrinsic parameter and the hand-eye calibration. The
`technique requires tµe camera to observe a planar pattern shown at a plurality
`of (at least two) different orientations. The pattern can be printed on a laser
`printer and attached to a planar surf ace. The position of the tool at each station
`is acquired directly from the robot. The operator positions the calibration
`
`ABB Inc. Exhibit 1002, Page 9 of 996
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`- 8 -
`
`pattern in front of the robot and camera and starts the procedure. The automatic
`calibration takes place in less than 5 minutes. The calibration can be carried
`out in a different location from the part training or man~pulation. The calibra(cid:173)
`tion pattern can be mounted in a fixed position, out of the working space, and
`automatic calibration can take place at regular time intervals.
`
`5
`
`The following steps are carried out to perform the automatic
`
`calibration:
`
`10
`
`the calibration pattern is positioned in the field of view of
`a)
`the robot mounted camera 16;
`
`the robot is moved to a plurality of stations in a predefined
`b)
`manner so that the calibration pattern is in the field of view at each station ( see
`15 Roger Y. Tsai and Reimar K. Lenz, "A New Technique for Fully Autonomous
`and Efficient 3D Robotics Hand/Eye Calibration", IEEE Transactions on
`Robotics and Automation, Vol. 5, No. 3, June 1989 p. 345 at p. 350);
`
`20
`
`c)
`
`At each station the following operations are performed:
`the acquired image is processed to extract the calibration
`points from the image;
`the robot is asked for the tool position;
`
`25
`
`Using the calibration points information at each station,
`d)
`calibrate the camera intrinsic parameters and compute the extrinsic transforma(cid:173)
`tion from the pattern to the camera;
`
`Using the extrinsic transformation at each station and the
`e)
`corresponding tool position, the camera to tool transformation is calculated (see
`30 Tsai and Lenz reference above at p. 350).
`
`Teaching
`
`Teaching the object is the process of:
`
`ABB Inc. Exhibit 1002, Page 10 of 996
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`- 9 -
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`Selection from the object's image of a set of at least 5
`a)
`features, and determining the position of the features in the image. Features can
`be edges; holes, corners, blobs ( extracted from the image) or simply a region of
`the image which will be used in a pattern match. Preferably, one or more
`unique features may be selected to be considered the "anchor features". The
`other selected features may be small, non-unique features relative to the anchor
`features.
`
`Real world coordinates are computed for the selected
`b)
`features. The object is located in the Training Space, so that by using the fea(cid:173)
`tures' heights relative to the Training Space, the 3D position of the object
`features inside the Training Space can be computed using the position in the
`image and the Training Space to Camera Space transformation calculated at
`calibration.
`
`An Object Space is defined such that it is identical to the
`c)
`Training Space but is rigid to the object and moves with the object.
`d)
`Also an Object Frame will be defined with a constant
`relationship to the object, in a position selected by the user. Three non-co-
`linear points may be used to define the Object Frame.
`
`The Object Frame (computed in tool coordinates) is sent to
`e)
`the robot to be considered as the robot working space. To find this frame
`position, the transformation from Object Space to Camera Space is used, then
`from Camera Space to Tool Space.
`
`Relative to the Object frame, the operator can train the
`f)
`intended operation path (the tool path shown in Fig. 3) for the robot by
`moving the robot or alternatively by using an off-line robot programming
`software application to create the desired operation path with respect to the
`object.
`
`As illustrated in Fig 3, 4 and 6, the first step in teaching the
`object features is to place the object in the "Training Space " and capture an
`
`ABB Inc. Exhibit 1002, Page 11 of 996
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`- 10 -
`
`5
`
`image with the robot in the calibration position (where the "Camera Space->
`Training Space" transformation was calculated). Figure 4 illustrates the image
`50 of the part as displayed on the operator's computer screen. Next, at least 5
`visible features 54 are selected. Preferably at least one anchor feature 52 is
`selected from the image and a set of at least 4 other visible features 54 are
`selected and positioned relative to the anchor feature. The 3D position of each
`feature 54 is then calculated in "Training Space.., using the height of the feature
`relative to the Training Space and "Camera Space -> Training Space" trans(cid:173)
`formation. An "Object Space " is then defined as having a constant relationship
`10 with the "Training Space" but connected to the object and the 3D coordinates of
`the features are transferred into that space. The "Object Space-> Camera
`Space" transformation is computed using the 3D position of the features inside
`this space and the position in the image by computing an extrinsic calibration
`using the camera calibration from the previous step. Next an "Object Frame"
`inside "Object Space" is defined to be used for teaching the intended operation
`path. This Object Frame position and orientation in "Tool Frame" is computed
`using the transformation from "Object Frame-> Camera Space" and "Camera
`Space-> Tool Space". The "Object Frame" is then sent to the robot and the
`intended operation path is trained inside this space.
`
`15
`
`20
`
`The following method may also be used for computing the 3D
`position of the selected features 54 automatically, without any prior knowledge
`about the part. The following algorithm is developed using the approach
`described in Guo-Quing Wei,. "Active Self-Calibration of Robotic Eyes and
`25 Hand-Eye Relationship with Model Identification", IEEE Transactions on
`Robotics and Automation, Vol. 14, No. 1, February 1999 p. 158. The camera is
`rigidly mounted on the robot gripper. The derived method computes the world
`coordinates of the features based on robot motion parameters, known camera to
`tool transformation, image coordinate measurements and intrinsic camera
`parameters. The robot tool will undergo a set of pure translations from a base
`position P0 to a set of positions Pt
`
`30
`
`The motion equations for a point Pi in the image are:
`
`ABB Inc. Exhibit 1002, Page 12 of 996
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`- 11 -
`
`R
`
`cg(0,0)
`
`+R
`cg(l,0)
`
`+R
`cg(2,0)
`
`_ U = f
`
`O
`
`X
`
`Uij
`
`V .. _ V = f
`
`0
`
`1
`'
`
`Where:
`
`+X
`*t
`*t
`*t
`O
`g0j2
`g0jl
`g0j0
`Rcg(0,2) * t g0j0 + Rcg(l,2) * t g0jl + Rcg(2,2) * t g0j2 + Zo
`Rcg(O,l) * t g0j0 + Rcg(l,I) * t g0jl + Rcg(2,I) * t g0j2 + Yo
`*t +z
`*t
`*t
`g0j0
`g0jl
`
`y R
`
`cg(0,2)
`
`+R
`cg(l,2)
`
`+R
`cg(2,2)
`
`g0j2
`
`0
`
`5
`
`10
`
`j represents the index of the station where the measurement takes place;
`Reg are elements of the rotation matrix of the camera to tool transformation;
`tgoj represents the translation from the current station to tbe base position;
`u0, v O is the camera central point;
`uij•vij _are the undistorted image coordinates of the i-th feature in the image
`acquired at station j. To compute these values one uses the already calibrated
`cameras intrinsic parameters.
`
`At least two stations are needed for the linear system to have
`unique solutions, but a set of at least 3 stations is used in order to compensate
`for the noise in the images and other perturbation factors that may occur.
`
`15
`
`X0, Y 0,Z0 are computed for all the features in camera space, but
`the values can be transposed in any other space that is related to it, such as
`training space, tool space or eve1:1 robot base space. The space in which the
`coordinates are represented makes no difference as this space is only used to
`compute the current transformation to the camera and to transfer then the object
`frame points to the tool space.
`
`20
`
`The robot stations are located on a circle around the base to
`assure a uniform distribution.
`
`25
`
`The automatic feature position finding steps are as follows:
`
`ABB Inc. Exhibit 1002, Page 13 of 996
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`- 12 -
`
`the part is positioned in front of the camera 16;
`a)
`the features 54 that are going to be used for 3D part
`b)
`positioning are selected;
`c)
`the automatic feature position finding procedure is started
`
`5
`
`by:
`
`moving the robot to a set of stations around the base
`
`i)
`22;
`ii)
`
`10
`
`15
`
`20
`
`25
`
`30
`
`iii)
`
`memorizing the tool position relative to base at each
`station;
`at each station acquiring an image, finding the visi(cid:173)
`ble features 54 and for those features calculating the
`undistorted image position;
`for each feature solve a linear system of as many equations
`d)
`as the number of images the given feature was visible in;
`e)
`the calculated positions are transformed in a space that
`suits the application.
`
`Alternatively the location of the features can be sourced by us1ng
`a CAD model of the part.
`
`1. Object Location & Robot Guidance
`
`To carry out object location and robot guidance, the following
`steps are carried out:
`
`The tool 14 is positioned in any predefined position above
`a)
`the bin with objects 18.
`
`b)
`
`An image of the object 18 is captured.
`
`The trained features 54 are searched for. If any anchor
`c)
`features were selected, then a first search is done for the anchor features 52.
`Using the position and orientation of the anchor features 52 the rest of the
`features 54 can be found by their relative positions. This approach allows
`
`ABB Inc. Exhibit 1002, Page 14 of 996
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`- 13 -
`
`similar features to be selected as features are searched in a relatively small
`region of interest. Otherwise, each feature is searched over the entire image.
`
`The position (in the image 50 and in the Object Space) of
`d)
`the found features (at least 5) are used to calculate the transformation between
`the Object Space and the Camera Space using an extrinsic calibration algo(cid:173)
`rithm. (See the Tsai article above). The found position is used to re-orient the
`camera to "look" at the object from an orthogonal position which is the one
`used at training. This last step may be necessary if the object has major rota-
`tions, since in this case the features may be distorted and the found position
`may not be completely accurate.
`
`e)
`
`Steps c) and d) above are repeated.
`
`The previous "Object Space to Camera Space" transforma-
`f)
`tion is used in conjunction with the "Camera Space to Tool Space" transforma(cid:173)
`tion to find the position of the Object Frame in Tool Space.
`
`5
`
`IO
`
`15
`
`The Object Frame is then sent to the robot to be used as
`g)
`the reference frame for the robot's operation path.
`
`20
`
`With reference to Fig. 7, the preferred steps to carry out object
`location and robot guidance are illustrated. First the robot is positioned in a
`predefined position above the bin or container that holds the target object. If
`no object is in the field of view, the robot is moved until at least one anchor
`feature is found in the image. Next, the position and orientation of the anchor
`feature is used to compute the position of the rest of the features. The position
`of all the visible features in the image is found. With the positions of features
`from the image and their corresponding positions in "Object Space" ( calculated
`during teaching) the camera calibration is used to compute the transformation
`between the "Object Space" and "Camera Space". The camera extrinsic
`calibration is used as described in the Tsai article referenced above.
`
`25
`
`3 0
`
`ABB Inc. Exhibit 1002, Page 15 of 996
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`- 14 -
`
`5
`
`Next the transformation described above is used to calculate the
`movement of the robot to position the camera so that it "looks" orthogonally at
`the object, namely the same position as in training. In this way all the features
`will appear as similar as possible as at training. This will make the recognition
`and positioning more accurate. Next the "Object Space -> Camera Space"
`transformation is found in the same way as in the previous step (using the
`features positions). The Object Frame memorized at training is computed using
`the found transformation and "Camera Space-> Tool Space" transformation.
`Next, the commuted "Object Frame" is sent to the robot. The "Tool" position is
`10 used to define the frame in "Robot Space". The trained robot path is performed
`inside this space.
`
`15
`
`20
`
`25
`
`30
`
`Thus methods for teaching robots and handling of objects by
`robots in three dimensions using one camera mounted on the robot arm are dis-
`closed in which targets are used on objects. The targets are usually normal
`features of the object or could be created using markers, lighting, etc. It is not
`necessary to use the CAD model of the object according to this method. The
`objects need not be fixtured and can be placed anywhere within the workspace
`of the robot. While the method has been described for one trained object, the
`process can be used in the same manner to first recognize the object and then
`find its location in three dimensions. Also the method has been described for
`one visible set of features selected on one side of the object, but it can be
`extended to all the sides that can appear in a working situation.
`
`In the method described above, the calibration step can be carried
`out with just the intrinsic and hand-eye calibration, and in that case, in teaching
`step b ), the 3D position of features may be provided using the CAD model of
`the object. Similarly the 3D position of features can be calculated automati(cid:173)
`cally, without any model or measurements on the part.
`
`Further, training step c) can be accomplished by sending to the
`robot the positions of intended robot path points, with the positions computed
`using the object's current position and orientation. The robot path points can be
`sourced from an offline robot programming software application. Also, one or
`
`ABB Inc. Exhibit 1002, Page 16 of 996
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`- 15 -
`
`more features on the object can be used as grasp points and the position of
`those points in robot coordinates sent to the robot to eliminate the need for
`manually teaching grasp points using the robot teach pendant. Alternatively the
`coordinates of the object frame points or other points of interest ( e.g. robot path
`points) can be transformed using the transformation from tool to robot base and
`all the coordinates send to robot in the robot base coordinate frame instead.
`
`5
`
`10
`
`Further, the calibration and teaching steps a) and b) can be
`combined by using a self calibration method of robotic eye and hand-eye
`relationship with model identification as described in "Active self-calibration
`method of robotic eye and hand-eye relationship with model identification " by
`Guo-Qing Wei, Klaus Arbter and Gerd Hirzinger. The result of such a method
`will give camera intrinsic parameters, hand-eye calibration and position of
`selected features in camera space; The rest of the path training and run time
`15 will still remain the same and in this preferred approach;
`
`In accordance with the present invention the determination of
`object location in step c(i) and d(ii) can use any of the following algorithms:
`
`20
`
`25
`
`3D pose estimation using non linear optimization methods
`a)
`derived from the ones described in the already mentioned articles :
`"A Flexible New Technique for Camera Calibration " by
`Zhengyou Zhang;
`"An efficient and Accurate Camera Calibration Techniques
`for 3D Machine Vision" by Roger Y.Tsai;
`
`3D pose estimation from lines correspondence ( in which case
`b)
`selected features will be edges ) as described in "Determination of Camera
`Location from 2D to 3D Line and Point Correspondences" by Yucai Liu, Thomas
`30 S. Huang, Olivier D. Faugeras;
`
`pose estimation using "orthogonal iteration" described in
`c)
`"Fast and Globally Convergent Pose Estimation from Video Images" by
`Chien_Ping Lu, Gregory D. Hager, Eric Mjolsness;
`
`ABB Inc. Exhibit 1002, Page 17 of 996
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`- 16 -
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`location under weak perspective
`approximate object
`d)
`conditions as demonstrated in "Uniqueness of 3D Pose Under Weak Perspective:
`A Geometric Proof' by Thomas Huang, Alfred Bruckenstein, Robert Holt, Arnn
`Netravali;
`
`approximate object location using Direct Linear Transforma-
`e)
`tion ( DLT) as described in "An investigation on the accuracy of three-dimensional
`space reconstruction using Direct Linear Transformation techniques" by Chen,
`Armstrong, Raftopoulos.
`
`In addition to use in robotics, the described method can be applied
`to a variety of industrial and non-industrial processes lhereby knowledge of the
`~
`3D pose of an object is required.
`
`While the invention has been described using a single camera, the
`image may be formed using multiple sensors ("Sensor Array"). For instance, the
`formation of a single image may be accomplished using multiple cameras that are
`situated such that the origins of their camera spaces are coincident. In this fashion,
`each camera would view a different area of the part with some overlap with the
`areas viewed by adjacent cameras. A single image is then formed by "Mosaicing"
`the images from individual cameras similar to one described in Multiple View
`Geometry in Computer Vision by Richard Hartley and Andrew Zisserman,
`Cambridge University Press, 2000. The same image formation may be accom(cid:173)
`plished by mounting a single camera on a robot and rotating the camera about its
`origin of the camera space and capturing the multiple images needed for
`Mosaicing. The object features may be extracted from multiple images captured
`by the same sensor array located in the same position whereby each image is
`formed under a different combination of lighting and filters to highlight a group
`of object features that are not apparent in other images. The object features
`themselves may be created using markers, lighting or other means.
`
`The calibration and teaching steps may be accomplished by using
`a self calibration method of robotic eye and hand-eye relationship with model
`identification.
`
`ABB Inc. Exhibit 1002, Page 18 of 996
`ABB Inc. v. Roboticvisiontech, Inc.
` IPR2023-01426
`
`

`

`- 17 -
`
`The creation of the Object Model may be accomplished by using the
`relative heights of features and the "Robot - Eye Calibration" or by the operator
`entering the 3D position of each feature manually from a CAD model, measure-
`5 ment or other source.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`While the invention has been described with the camera mounted on
`the robot, the same approach is valid for cases whereby the camera is fixed onto
`a stationary structure. Similarly although the object has been described to be at
`rest on a surface, the same approach is valid for cases when the object is stationary
`but grasped by a robot. Also during the operation of