Memory network for tracking with deep regression

Visual object tracking has been a fundamental topic in recent years and many tracking-by-detection approaches have achieved state-of-the-art performance. However, most of these trackers neglect the temporal information of the targets through sequences. In this paper, we develop a memory network to exploit the history of object locations as well as the distinctive visual features, which then serves as a region predictor for the new frame. In contrast to most existing trackers which use binary classification to score region candidates drawn around the target, this approach gets one specific heuristic proposal from the memory network, followed by bounding box regression for precise tracking results. Specifically, Memory and Regression based Tracker (MRT) consists of a CNN-based network for feature extraction, a stacked Long Short-Term Memory network (LSTM) for memory updating and a set of regression layers for proposal refinement. This model is fully differentiable and can be trained end-to-end. Evaluation of the proposed method is performed on the VOT2018 dataset which shows that the MRT achieves competitive performance.