Coarse-to-fine multiview 3d face reconstruction using multiple geometrical features

3D face reconstruction from multi-view video sequences has become a hotspot in computer vision for the last decades. Structure from Motion (SfM) methods, which have been widely used for multi-view 3D face reconstruction, have two main limitations. First, self-occlusion causes certain facial feature points (FFPs) to be invisible in the images, which will lead to missing data. The existing SfM methods could recover the missing data through iterative calculation, however, with high computational costs and long processing time. Second, the SfM methods cannot reconstruct the accurate 3D facial shapes of cheeks because there are no FFPs in this area. This paper proposes a novel “coarse-to-fine” multi-view 3D face reconstruction method by taking the advantage of the complementarity between FFPs and occluding contours, i.e., the boundary lines depicted between the facial region and the background. In this method, a block SfM algorithm is firstly proposed to reconstruct a “coarse” 3D facial shape by utilizing sparse FFPs. The block SfM algorithm does not estimate the true locations of the self-occluded FFPs iteratively. Thus, the computational cost is significantly reduced. Then, a kernel partial least squares (KPLS) algorithm is introduced to refine the “coarse” 3D facial shape. The KPLS method applies occluding contours to remedy the limitation of sparse FFPs correspondence-based SfM method. The proposed method is evaluated on the synthetic sequences generated from the BJUT-3D face database and the real-world multi-view video sequences obtained in a controlled indoor environment. The results show improvements in both accuracy and efficiency.