Representation Learning, Scene Understanding, and Feature Fusion for Drowsiness Detection

We propose a novel drowsiness detection method based on 3D-Deep Convolutional Neural Network (3D-DCNN). We design a learning architecture for the drowsiness detection, which consists of three building blocks for representation learning, scene understanding, and feature fusion. In this framework, the model generates a spatio-temporal representation from multiple consecutive frames and analyze the scene conditions which are defined as head, eye, and mouth movements. The result of analysis from the scene condition understanding model is used to auxiliary information for the drowsiness detection. Then the method subsequently generates fusion features using the spatio-temporal representation and the results of the classification of scene conditions. By using the fusion features, we show that the proposed method can boost the performance of drowsiness detection. The proposed method demonstrates with the NTHU Drowsy Driver Detection (NTHU-DDD) video dataset.