Fast Inter Mode Decision Algorithm Using Spatiotemporal Characteristic of Motion Vector Field

H.264 employs variable size motion compensation with multiple reference frames, which significantly improve the coding performance compared with the previous video coding standards. However, this comes with a cost of extremely high computational complexity. Thus, one of the most challenging problems in implementing real time H.264 encoder is mode decision. In this paper, a fast inter mode decision algorithm is proposed to reduce the candidate mode set, which utilizes two basic elements, spatiotemporal characteristic of motion vector field and texture homogeneity. Spatiotemporal characteristic of motion vector field is first used to detect macroblocks (MB) in background or the interior of a motion object and early terminate mode decision. Otherwise, texture homogeneity is exploited to skip unnecessary prediction modes for MBs in the boundary region of motion objects. Simulation results show that our proposed algorithm can save 52% average computational complexity, with negligible loss of coding efficiency.