Auto-Tuning Deep Stereo on the Fly

Stereo depth estimation based on learning with CNNs has become the dominant paradigm due to their significant leap in performance. However, since the inconsistency of the distribution between the training data (synthetic image) and the actual scene, the generalization ability is insufficient for deployment. Recently, the work MADNet with modular adaptation was proposed to tackle this issue on the fly. However, it still runs slowly with not-so-ideal performance. This paper introduces a “harmless training or inference” (HToI) framework to improve their work in both convergence and prediction speed. Firstly, in the actual scenario, the images arrive in sequence without ground-truth. Online adaptation to new target domains is only through self-supervised loss with errors (e.g., occluded areas). This paper introduces a confidence-weighted map (CWM) to mask the incorrect loss. The confidence map will guide the model to use the correct loss for online adaptation to obtain better accuracy. Secondly, this paper proposes a supervisor, implemented by Mann-Kendall trend detection on loss (MKL), to monitor the performance changes during the adaptation. According to the MKL, the model can autonomously determine whether to keep harmless training. If not training (i.e., only inference), the model will only predict the disparity, significantly reducing the running time. Finally, experiments on real-world stereo datasets confirm the effectiveness of CWM and MKL, which can make the model adapt to the scene more stably and accurately.