Recaspia: Recognizing Carrying Actions in Single Images Using Privileged Information

Many approaches for action recognition focus on general actions, such as "running" or "walking". This work presents a method for recognizing carrying actions in single images, by utilizing privileged information, such as annotation, available only during training, following the learning using privileged information paradigm. In addition, we introduce a dataset for carrying actions, formed using images extracted from YouTube videos depicting several scenarios. We accompany the dataset with a variety of different annotation types that include human pose, object and scene attributes. The experimental results demonstrate that our method, boosted sample averaged F1 score performance by 15.4% and 4.15%, respectively, in the validation and testing partitions of our dataset, when compared to an end-to-end CNN model, trained only with the observable information.