Real-time surveillance-video-based personalized thermal comfort recognition

Abstract The current trend in improving the energy efficiency of cooling systems in buildings focuses on utilizing the thermal comfort state of the occupants as the real-time temperature control criterion. This new trend puts forward new requirements on the accuracy and efficiency of thermal sensation recognition. This paper focuses on developing the capability to automatically evaluate and detect thermal sensations from human behavior from surveillance video. The proposed approach is based solely on the real-time visual status of humans and assumes that the thermal-adaptive behavior of people contains a variety of information that allows for inferences about the temperature comfort of a room. To this end, we develop a technique to apply thermal-adaptive behavior recognition to thermal sensation inference based on a spatial temporal graph convolutional network (ST-GCN). The approach can recognize 16 thermal-adaptive behaviors, which collected from two questionnaires were conducted at Shandong Normal University, in surveillance videos in real time. Based on the collected data, we release a video dataset of thermal-adaptive behaviors and extensively evaluate the proposed approach on the newly collected thermal-adaptive behavior video benchmark. The experimental results show that the median prediction accuracy of thermal sensation is up to 78% when all actions are considered, which demonstrates the effectiveness of the approach.