A Parallel LSTM-Based Missing Body Feature Point Completion in Video Frames

The ability to guess the posture of a body is an essential process for activity recognition. OpenPose, a deep learning-based model, detects body feature points that form the posture of a subject from moving images.OpenPose is often used for action recognition. However, there are cases where the feature points cannot be detected due to image blurring or a part of the human body being shielded, for example. Conventional interpolation methods estimate missing values, but the estimation accuracy of these methods is not high when a particular feature point is missing in a large number of consecutive frames. We propose a deep learning-based point estimation model that performs data completion of missing body feature points, which cannot be captured by OpenPose. The model is constructed based on Long Short-term Memory (LSTM) that takes consecutive frames in a video. In order to improve estimation accuracy, we revised the model by considering the following two extensions, namely, the division of body parts and the use of relative coordinates. Experimental results show that the proposed model successfully completes body feature points within an error of 8.15 pixels, which is three times better than the conventional interpolation methods and five times better than a LSTM without the proposed extensions.