Tri-level optimization-based image rectification for polydioptric cameras

Abstract Recently, as the number of cameras built into polydioptric cameras has increased, image rectification of these cameras has become complicated. However, because conventional methods cannot compensate for the calibration errors or are limited by the camera arrangement, they cannot be widely applied to various kinds of polydioptric cameras. In this work, we adopted the idea of disparity-error-minimization to overcome these limitations. We introduced the following several improvements in the optimization-based rectification. (1) We modified the objective function to include both the x and y disparity errors. (2) We added a regularization term to perform robustly for mismatched pairs. (3) We employed tri-level optimization to determine the camera pose corresponding to the rectified images. For two representative polydioptric cameras, this method reduced the average disparity error up to 66.57% compared to conventional methods. The results showed that our method exhibited significant generalization capabilities, achieving significant improvements over the existing methods.