5D Video Stabilization through Sensor Vision Fusion

We propose a novel 5D video stabilization solution based on sensor vision fusion. Traditional gyroscope based video stabilization approaches only stabilize the 3D rotation of a camera. They often suffer in scenes with highly dynamic translational movements. In this paper, we model the residual camera motion after 3D rotation based stabilization as residual 2D translation in the image plane, which can be estimated using visual information. The proposed 5D stabilization is achieved by carefully combining sensor based 3D rotation stabilization and vision based residual 2D translation stabilization. The 5D stabilization is prototyped on a smartphone for real time recording. We demonstrate its advantages over two state-of-the-art methods. The 5D stabilization can also be used to solve the challenging foreground object stabilization problem.