## Abstract

Hand–eye calibration is a typical research direction in robotics applications. The current methods can be divided into two categories according to whether the rotational and translational equations are decoupled for computation: two-step methods and one-step methods. Both one-step and two-step methods generally convert such problems to linear null space computations, which are implemented by the corresponding computational operators. Owing to the booming development of the rotation operators, the two-step methods have been more fully researched. However, due to the limitations of the research on computational operators integrating rotation and translation, the one-step methods still have much scope for research. Dual algebra, as effective mathematical entities for screws and wrenches, provides the theoretical basis for the development of the one-step methods for hand–eye calibration. In this paper, a computational operator for the dual matrices computation was first proposed, i.e., dual Kronecker product. Subsequently, a hand–eye calibration framework was proposed based on the dual Kronecker product, which allowed the screw motion to be represented as multiple dual vectors. Furthermore, the equivalence of this framework with the orthogonal-dual-tensor-based approach was derived, providing a more intuitive computational representation. The feasibility and superiority of the proposed computational framework were experimentally verified.

## 1 Introduction

Sensor calibration is a widely researched direction of study. For 6R serial robots, the sensor calibration problem has been formulated as solving for $AX=YB$ and $AX=XB$ [1–3]. Depending on whether rotational and translational parts are estimated simultaneously, the available methods can be divided into: one-step methods [4–6] and two-step methods [7–10]. This classification depends essentially on the algebraic entities of the rigid body transformations and the selection of the computational operators. The goal of this paper is to combine screw theory and dual algebra to construct a one-step calculation framework for the $AX=XB$.

Screw theory was first considered for application to hand–eye calibration as an attempt to reveal the geometric nature of the problem [11]. However, the implementation of hand–eye calibration requires algebraic entities and operators. Lie group $SE(3)$ and dual quaternion provided convenient tool for handling algebraic entities for screws [12,13]. The implementation of the one-step methods for hand–eye calibration requires two types of operators. One is the Kronecker product applied to the matrix Lie group, first proposed by Andreff et al. [8]. And another is an operator of dual quaternion, first applied to hand–eye calibration by Daniilidis [4]. Andreff et al. [8] converted $AX=XB$ into a linear system using the Kronecker product, and subsequently decoupled the rotational and translational equations to obtain the two-step solution. An important factor was that the rotation estimate obtained using the Kronecker product required an additional orthogonalization correction, and this correction process could not be reflected in the translation estimate by the one-step method. Therefore, the simultaneous estimation of rotation and translation based on Kronecker product raises an urgent question of how to implement orthogonalization in the translational estimation. In dual algebra, the orthogonality of dual matrix has its specific definition, which provides a solution to achieve orthogonality of rotation and translation simultaneously [14].

Several methods have been proposed in the last decades to solve the sensor calibration problems, and the main difference between these methods was the selection of the algebraic entities for the rigid body transformations. It is also classified into one-step and two-step methods according to whether the rotation and translation of different algebraic entities are decoupled or not. The two-step methods implemented in terms of $SE(3)$ usually start by decoupling the equations into rotational and translational equations, and then obtaining the rotational solution by the parameterization of rotation matrix. Specifically, the rotational solutions were achieved with the help of Euler angles [15], unit quaternion [16], and axis angle [17] parameterized rotation matrices. Gaussian–Newton algorithm [18,19], in which the Jacobian matrix was calculated by an exponential mapping from Lie algebra $so(3)$ to the special orthogonal group $SO(3)$. A comprehensive summary of the parameterized representation of rotation matrices was summarized by Bauchau and Trainelli [20] and Barfoot et al. [21]. In addition, Ad($SE(3)$) has been applied in hand–eye calibration as an adjoint representation of $SE(3)$ [22–24]. The dual quaternion, as an algebraic entity of the screw motion, usually implements the one-step methods of hand–eye calibration [25,26]. It is also possible to decouple rotational and translational motions according to the screw motion, thus leading to the dual-quaternion-guided two-step method [27]. The one-step methods and two-step methods have their own advantages and disadvantages. A prominent problem with two-step methods is the error transfer due to separable computation. The one-step method was first proposed precisely to eliminate the error transfer [28]. The goal of this paper is to propose a one-step method geometrically based on the screw theory to reduce the propagation error.

The advantages of dual quaternion as a part of dual algebra in robot kinematics and dynamics have been well demonstrated. However, the powers and advantages of the dual algebra are much more than that. Dual algebra has been originally conceived by Clifford [29], but its first applications to mechanics was due to study [30]. The application of dual numbers, dual vectors, and dual matrices in robot kinematics was focused on how to represent the Denavit–Hartenberg (D–H) model equivalently [31–33]. More importantly, the dual algebra provides the mathematical entities for the screws and wrenches [34,35]. And a more detailed summary of the fundamental operations involved in kinematics was provided by Pennestri and Stefanelli [14]. Condurache and Burlacu [36] have made a breakthrough in combining dual algebra with screw theory for application to robot kinematics, and have further implemented hand–eye calibration based on the orthogonal dual tensor [37].

In this paper, the dual Kronecker product was introduced as a complete computational tool for dual matrices. First, we derived the method proposed by Condurache and Burlacu based on the dual Kronecker product, providing a more visualized calculation. Subsequently, a computational framework for hand–eye calibration was constructed, which was generalized to the rigid body motion represented by the dual numbers and dual vectors as well as the dual Euler parameters. Finally, the superiority of the proposed method was experimentally verified.

## 2 Methodology

### 2.1 Dual Kronecker Product.

A dual number, usually denoted in the form $a_=a+\epsilon a\u2032$, is an ordered pair of real numbers $(a,a\u2032)$. Here, $a$ is the real part of $a_$ and $a\u2032$ the dual part of $a_$. The dual unit satisfies $\epsilon 2=\epsilon 3=\cdots =0$, $0\epsilon =\epsilon 0=0$, $1\epsilon =\epsilon 1=\epsilon $. The basic operations of dual numbers are listed in Table 1.

Dual operation | Mathematical expression |
---|---|

Sum | $a_\xb1b_=(a\xb1b)+\epsilon (a\u2032\xb1b\u2032)$ |

Product | $a_b_=ab+\epsilon (a\u2032b+ab\u2032)$ |

Division | $a_b_=ab+\epsilon a\u2032b\u2212ab\u2032b2(b\u22600)$ |

Inverse | $a_\u22121=1a_=1a\u2212\epsilon a\u2032a2(a\u22600)$ |

Square root | $a_=a+\epsilon a\u20322a(a\u22600)$ |

Dual operation | Mathematical expression |
---|---|

Sum | $a_\xb1b_=(a\xb1b)+\epsilon (a\u2032\xb1b\u2032)$ |

Product | $a_b_=ab+\epsilon (a\u2032b+ab\u2032)$ |

Division | $a_b_=ab+\epsilon a\u2032b\u2212ab\u2032b2(b\u22600)$ |

Inverse | $a_\u22121=1a_=1a\u2212\epsilon a\u2032a2(a\u22600)$ |

Square root | $a_=a+\epsilon a\u20322a(a\u22600)$ |

Dual operation | Mathematical expression |
---|---|

Sum | $a_\xb1b_=(a\xb1b)+\epsilon (a\u2032\xb1b\u2032)$ |

Vector scaling | $a_b_=ab+\epsilon (a\u2032b+ab\u2032)$ |

Scalar product | $a_\u22c5b_=a\u22c5b+\epsilon (a\u2032\u22c5b+a\u22c5b\u2032)$ |

Cross product | $a_\xd7b_=a\xd7b+\epsilon (a\u2032\xd7b+a\xd7b\u2032)$ |

Dual operation | Mathematical expression |
---|---|

Sum | $a_\xb1b_=(a\xb1b)+\epsilon (a\u2032\xb1b\u2032)$ |

Vector scaling | $a_b_=ab+\epsilon (a\u2032b+ab\u2032)$ |

Scalar product | $a_\u22c5b_=a\u22c5b+\epsilon (a\u2032\u22c5b+a\u22c5b\u2032)$ |

Cross product | $a_\xd7b_=a\xd7b+\epsilon (a\u2032\xd7b+a\xd7b\u2032)$ |

$(A_\u2297B_)(C_\u2297D_)=(A_C_)\u2297(B_D_)$ |

$vec(C_D_E_)=(C_\u2297E_T)vec(D_)$ |

$vec(A_B_)=(I\u2297B_T)vec(A_)=(A_\u2297I)vec(B_)$ |

$vec(A_b_)=(I\u2297b_T)vec(A_)$ |

$vec(a_b_T)=(I\u2297b_T)a_=a_\u2297b_$ |

$(A_\u2297B_)(C_\u2297D_)=(A_C_)\u2297(B_D_)$ |

$vec(C_D_E_)=(C_\u2297E_T)vec(D_)$ |

$vec(A_B_)=(I\u2297B_T)vec(A_)=(A_\u2297I)vec(B_)$ |

$vec(A_b_)=(I\u2297b_T)vec(A_)$ |

$vec(a_b_T)=(I\u2297b_T)a_=a_\u2297b_$ |

### 2.2 Review of Orthogonal-Dual-Tensor-Based Hand–Eye Calibration.

- Let the approximation $R~_X$ of $R_X$ be set to $R~_X=[c_1,c_2,c_3]$, and set(16)$v_1=c_1$
- $v_2$ is calculated byWhile $v_3$ is calculated by(17)$v_2=c_2\u2212c_2\u22c5v_1v_1\u22c5v_1v_1$(18)$v_3=c_3\u2212c_3\u22c5v_1v_1\u22c5v_1v_1\u2212c_3\u22c5v_2v_2\u22c5v_2v_2$
- Finally, unitize the dual vectors to ensure that $det(R_X)=1+\epsilon 0$. And so the final step isThe solution satisfying the orthogonality constraint is(19)$c_i\u2020=v_i|v_i|,i=1,2,3$(20)$R_X=[c_1\u2020,c_2\u2020,c_3\u2020]$

### 2.3 Hand–Eye Calibration Framework Based on the Dual Kronecker Product.

## 3 Experiments

### 3.1 Experimental Setup.

Condurache and Burlacu [37] concentrated on verifying sufficient and necessary conditions for the existence and uniqueness of the solution. Although the feasibility was verified by simulation data. However, the method has never been experimentally validated. The calibration results are subject to errors in real-experiments due to the presence of noise. For the one-step methods, the calibration accuracy is affected by both rotational and translational noise. In addition, Tsai and Lenz [3] pointed out that the distribution of the robot’s poses affects the calibration accuracy. Therefore, it is necessary for experimental validation to evaluate the calibration methods.

We have demonstrated that the method calculated based on dual rigid bases is fully equivalent to the results calculated based on dual Kronecker product. Therefore, the method used in this subsection is based on the dual Kronecker product implementation and is named “DS.” In addition, to further validate the generality of the proposed computational framework, a dual quaternion representation of the rigid body motion is used and named “DE.”

Three methods were selected for comparison, as shown in Table 4. The current operators implementing the one-step method were the dual quaternion and the Kronecker product. The “DQ” method was naturally considered as a reference method as a typical representative of a dual quaternion based implementation [4]. Although, Andreff et al. [8] implemented the linearization of the hand–eye equation, the real implementation of the one-step method for solving it came from the work [10]. We named it “STL.” In addition, another class of one-step methods was implemented based on an iterative process of rotation and translation. The one-step method proposed by Horaud and Dornaika [28] first obtained the solution of the rotation and translation in two steps through the quaternion and finally constructed the cost function containing the rotational and translational equations. In contrast, the “PQ” method with stronger intrinsic constraints due to the use of quaternion conjugation to couple the rotation and translation together. In summary, the “PQ” method was taken into consideration as one of the reference methods.

Method | Solve-case | Representation of screw motion | Representation of hand–eye transformation | Operator for solving | Solution |
---|---|---|---|---|---|

DQ [4] | One-step | Dual quaternion | Dual quaternion | Dual quaternion | Closed-form |

PQ [22] | One-step | Screw vector | Ad $(SE(3))$ | Quaternion | Iterative |

STL [10] | One-step | $SE(3)$ | $SE(3)$ | Kronecker product | Closed-form |

DS, this paper | One-step | Dual screw vector | $SO_(3)$ | Dual Kronecker product | Closed-form |

DE, this paper | One-step | Dual quaternion | $SO_(3)$ | Dual Kronecker product | Closed-form |

Method | Solve-case | Representation of screw motion | Representation of hand–eye transformation | Operator for solving | Solution |
---|---|---|---|---|---|

DQ [4] | One-step | Dual quaternion | Dual quaternion | Dual quaternion | Closed-form |

PQ [22] | One-step | Screw vector | Ad $(SE(3))$ | Quaternion | Iterative |

STL [10] | One-step | $SE(3)$ | $SE(3)$ | Kronecker product | Closed-form |

DS, this paper | One-step | Dual screw vector | $SO_(3)$ | Dual Kronecker product | Closed-form |

DE, this paper | One-step | Dual quaternion | $SO_(3)$ | Dual Kronecker product | Closed-form |

The hand–eye calibration problem derives from the hand–eye system. The classical hand–eye system is composed of a six degrees-of-freedom (6DOF) serial robot and the visual system, containing eye-in-hand and eye-to-hand configurations. In this paper, the benchmark experimental system, i.e., the eye-in-hand experimental system, was selected. The experimental setup was shown in Fig. 2. An industrial robot (GSK RB03) with 6DOF was used to build the robotic system with the kinematic parameters shown in Table 5. A CMOS camera (Photonfocus $MV1\u2212D2048*1088\u2212240\u2212CL$) with an $8mm$ lens installed at the end of the robot and a checkerboard with $7\xd710$ calibration grid (each calibration grid is a $25\xd725mm$ square) have been used to construct the visual system. Bright and even lighting condition which was provided by a 150 W LED light. The camera calibration toolbox of matlab [41] and the algorithm introduced in Ref. [42] were used to detect the corners of checkerboard. The intrinsic parameters of the camera were listed in Table 6.

Link no. | $\alpha i\u22121$ (deg) | $ai\u22121$ ($mm$) | $di$ ($mm$) | $\theta i$ (deg) |
---|---|---|---|---|

1 | 0 | 0 | 0 | $\theta 1$ |

2 | $\u2212$90 | 155 | 0 | $\theta 2$ |

3 | 0 | 360 | 0 | $\theta 3$ |

4 | $\u2212$90 | 100 | 365 | $\theta 4$ |

5 | 90 | 0 | 0 | $\theta 5$ |

6 | $\u2212$90 | 0 | 116 | $\theta 6$ |

Link no. | $\alpha i\u22121$ (deg) | $ai\u22121$ ($mm$) | $di$ ($mm$) | $\theta i$ (deg) |
---|---|---|---|---|

1 | 0 | 0 | 0 | $\theta 1$ |

2 | $\u2212$90 | 155 | 0 | $\theta 2$ |

3 | 0 | 360 | 0 | $\theta 3$ |

4 | $\u2212$90 | 100 | 365 | $\theta 4$ |

5 | 90 | 0 | 0 | $\theta 5$ |

6 | $\u2212$90 | 0 | 116 | $\theta 6$ |

### 3.2 Experimental Results and Discussion.

The experimental steps consist of two parts: measurement and verification. The measurement part consisted of two types of data collection, one is calibration data and the other was verification data. The collection process is performed by keeping the Checkerboard fixed and randomly taking the robot configurations within the measurement field of view. The procedure for obtaining the experimental data is as follows:

- Select ten robot configurations within the camera field of view, and the corresponding camera poses $Bj$($j=1\u202610$) were computed by the algorithm proposed by Ref. [42]. The robot poses $Aj(j=1\u202610)$ were calculated by the D–H model [43]. Finally, the data sets required to calculate the hand–eye equation were calculated according to the following equation:(35$Ai=Aj\u22121Ak$
*a*)Repeat the above process to obtain 50 sets of robot configurations to generate the validation data sets.(35$Bi=Bj\u22121Bk,j,k=1,\u2026,10,j\u2260k$*b*) - The data sets of $Ai$ and $Bi$ were first converted into screw parameters and then into the algebraic entities required by each method. The extraction process of $Ai$ is as follows [11]:(36$\theta A=arccos(trace(RA)\u221212)$
*a*)(36$kA=12sin\theta A{RA(3,2)\u2212RA(2,3)RA(1,3)\u2212RA(3,1)RA(2,1)\u2212RA(1,2)}$*b*)(36$dA=\theta AtATkA$*c*)where(36$cA=12(tA\u2212dA\theta AkA+cot\theta A2kA\xd7tA)$*d*)Then, convert $Bi$ in the same way. It is easy to construct $k_A/B$ and $f(\theta _A/B)$ for the computational framework expressed in Eq. (32).(37)$RA=[RA(1,1)RA(1,2)RA(1,3)RA(2,1)RA(2,2)RA(2,3)RA(3,1)RA(3,2)RA(3,3)]$ - Since the ground truth of the hand–eye transformation does not exist in real-experiment, the calibration results can only be verified indirectly through the validation model. The validation model was implemented by means of the homogeneous matrices, defined aswhere $X^$ was an estimate, and $Bi$ is the measured value. The calibration accuracy was quantitatively assessed through the root mean square, defined as(38)$A^i=X^BiX^\u22121$(39$ErrorofRX=1N\u2211i=1i=N[\theta (R^Ai\u22121RAi)]2$
*a*)where $RAi$ and $tAi$ were from the robot controller; and $N=50(50\u22121)2$.(39$ErroroftX=1N\u2211i=1i=N\Vert t^Ai\u2212tAi\Vert 22$*b*)

The experimental results were shown in Tables 7 and 8. The stability of the solution is assessed by the number of poses, which must be greater than three for a unique solution that exists. The experimental results demonstrated that the proposed hand–eye calibration framework outperforms the reference methods in terms of accuracy and stability. Moreover, the experimental results of “DS” and “DE” showed no significant difference, which also verified the generality of the hand–eye calibration framework.

Number of robot configurations | ||||||||
---|---|---|---|---|---|---|---|---|

Method | $n=3$ | $n=4$ | $n=5$ | $n=6$ | $n=7$ | $n=8$ | $n=9$ | $n=10$ |

DQ | 0.071 | 0.184 | 0.141 | 0.120 | 0.070 | 0.075 | 0.083 | 0.071 |

PQ | 0.068 | 0.080 | 0.081 | 0.084 | 0.068 | 0.068 | 0.073 | 0.068 |

STL | $0.066$ | 0.336 | 0.077 | 0.076 | 0.068 | 0.068 | 0.065 | $0.064$ |

DS | 0.068 | $0.067$ | $0.063$ | $0.068$ | $0.064$ | $0.064$ | $0.064$ | $0.064$ |

DE | 0.068 | $0.067$ | $0.063$ | $0.068$ | 0.065 | $0.064$ | $0.064$ | $0.064$ |

Number of robot configurations | ||||||||
---|---|---|---|---|---|---|---|---|

Method | $n=3$ | $n=4$ | $n=5$ | $n=6$ | $n=7$ | $n=8$ | $n=9$ | $n=10$ |

DQ | 0.071 | 0.184 | 0.141 | 0.120 | 0.070 | 0.075 | 0.083 | 0.071 |

PQ | 0.068 | 0.080 | 0.081 | 0.084 | 0.068 | 0.068 | 0.073 | 0.068 |

STL | $0.066$ | 0.336 | 0.077 | 0.076 | 0.068 | 0.068 | 0.065 | $0.064$ |

DS | 0.068 | $0.067$ | $0.063$ | $0.068$ | $0.064$ | $0.064$ | $0.064$ | $0.064$ |

DE | 0.068 | $0.067$ | $0.063$ | $0.068$ | 0.065 | $0.064$ | $0.064$ | $0.064$ |

Note: Bold values indicates the method with optimal experimental results.

Number of robot configurations | ||||||||
---|---|---|---|---|---|---|---|---|

Method | $n=3$ | $n=4$ | $n=5$ | $n=6$ | $n=7$ | $n=8$ | $n=9$ | $n=10$ |

DQ | $1.152$ | 1.807 | 1.526 | 1.375 | 1.173 | 1.187 | 1.217 | 1.129 |

PQ | 1.152 | 1.177 | 1.176 | 1.184 | 1.145 | 1.138 | 1.152 | 1.106 |

STL | 1.203 | 3.089 | 1.143 | 1.096 | 1.032 | 1.020 | 1.019 | 1.014 |

DS | 1.205 | $0.981$ | $0.997$ | 1.042 | $0.967$ | $0.955$ | $0.965$ | $0.964$ |

DE | 1.227 | 0.989 | 0.999 | $1.032$ | 0.968 | 0.958 | 0.968 | 0.967 |

Number of robot configurations | ||||||||
---|---|---|---|---|---|---|---|---|

Method | $n=3$ | $n=4$ | $n=5$ | $n=6$ | $n=7$ | $n=8$ | $n=9$ | $n=10$ |

DQ | $1.152$ | 1.807 | 1.526 | 1.375 | 1.173 | 1.187 | 1.217 | 1.129 |

PQ | 1.152 | 1.177 | 1.176 | 1.184 | 1.145 | 1.138 | 1.152 | 1.106 |

STL | 1.203 | 3.089 | 1.143 | 1.096 | 1.032 | 1.020 | 1.019 | 1.014 |

DS | 1.205 | $0.981$ | $0.997$ | 1.042 | $0.967$ | $0.955$ | $0.965$ | $0.964$ |

DE | 1.227 | 0.989 | 0.999 | $1.032$ | 0.968 | 0.958 | 0.968 | 0.967 |

Note: Bold values indicates the method with optimal experimental results.

We analyzed some reasons for this experimental results. In the proposed frame, the screw movement as the geometric basis leads to tighter intrinsic constraints of rotation and translation. Therefore, the one-step method can be better implemented. In addition, verification of coordinate invariants is necessary. “DQ” neglected the validation of coordinate invariants. The validation of coordinate invariants is necessary due to the noise in the experiment. “PQ” was implemented based on iterations of rotation and translation, which have weaker intrinsic constraints compared to simultaneously computed operators such as the dual quaternion and the Kronecker product. The disadvantage of “STL” was that the process of rotation orthogonalization could not be reflected in the translation calculation, as Andreff et al. [8] had already pointed out.

## 4 Conclusions

Hand–eye calibration is a typical research direction in robotics applications. Generally, published research methods were divided into two categories according to whether rotation and translation are computationally decoupled: two-step methods and one-step methods. The development of both types of methods depended on the development of the rigid body transformation operator. Therefore, two-step methods were widely implemented due to the full development of the rotation operator. Due to the development of computational operators, one-step methods still have much scope for research. The two-step methods and one-step methods have their own advantages and disadvantages. The goal of this paper was not to compare them, but to provide more options for a more comprehensive and careful comparison later.

In this paper, an operator for dual matrices computation, i.e., dual Kronecker product, was proposed. Subsequently, a hand–eye calibration framework was proposed based on the dual Kronecker product, which allowed the screw motion to be represented as multiple dual vectors. Furthermore, the equivalence of this framework with the orthogonal-dual-tensor-based approach was derived, providing a more intuitive computational representation. In the experiments, the current most representative one-step methods were used as the reference methods, and the computational framework proposed in this paper performed better in terms of accuracy and stability. Besides, the reasons for the experimental results were systematically analyzed.

## Acknowledgment

This research was supported by National Natural Science Foundation of China (Grant No. 12272266).

## Conflict of Interest

There are no conflicts of interest.

## Data Availability Statement

The datasets generated and supporting the findings of this article are obtainable from the corresponding author upon reasonable request.

## References

*Linear Dual Algebra Algorithms and Their Application to Kinematics*