Recognizing Fish Species Captured Live on Wild Sea Surface in Videos by Deep Metric Learning with a Temporal Constraint

Recognizing fish species captured live on wild sea surface in videos is a challenging task due to the deformation of fish shape, self-occlusion of body parts and similar texture between different fish classes. To address these issues, we propose a fine-grained image classification method based on a deep convolution neural network (CNN) trained by an innovative metric learning scheme with a temporal constraint. By introducing the temporal constraint in metric learning, we help the network to learn a feature embedding which implicitly takes the shape and pose changes of fish into account. Besides, for each class, we learn the representative features discriminatively by introducing an intermediate layer in the CNN before the classifier. In testing stage, we first aggregate the features of a fish from each frame into several clips in the feature space, send the clips to the classifier and then perform weighted majority vote for the final classification. The experimental results show that our approach outperforms the conventional softmax classification on our rail-fishing dataset.