Automated video analysis of sow nursing behavior based on fully convolutional network and oriented optical flow

Abstract This study aims to develop a method to automatically recognize nursing behaviors of sow in videos by exploiting the spatio-temporal relations. The method firstly detected spatio-temporal key cuboids which may have nursing interactions in them, and then Oriented Nursing Flows (ONuF) of the spatio-temporal key cuboids was proposed to further recognize nursing behaviors. In the first step, the temporal key episodes were first detected in video using optical flow-based features containing the distribution of nursing pixel and distance among moving pixels to estimate the distribution of motion all over the frame. Then, spatial key regions in the temporal key episodes were located by identifying the spatial position and geometric properties of the sow and her piglets using a fully convolutional network-based semantic segmentation approach. In the second step, to further recognize nursing behavior, a new feature descriptor ONuF, estimating the motion orientation change and motion magnitude of spatio-temporal key cuboids, was proposed and used to learn a SVM classifier. Testing on a set of 451 video episodes, the accuracy of classifying nursing was 94.5% with a sensitivity of 89.4% and with a specificity of 99.1%. Testing on a set of two days of continuous videos, the accuracy of recognizing nursing was 97.6% with a sensitivity of 92.1% and with a specificity of 98.6%. In addition, the nursing duration and nursing interval were counted from the recognition results of nursing behaviors in continuous videos. The results indicate that the method exploiting the spatio-temporal relations using fully convolutional network and oriented optical flow can be used for automatically recognizing nursing behaviors from daily behavioral videos of lactating sows.