WinSet: the first multi-modal window dataset for heterogeneous window states

Windows play an important role in modern buildings. Getting to know the window states, e.g., open vs. close, is an enabler of many smart city applications, such as energy conservation and emergency response. In this work, we collect the very first multi-modal (RGB, thermal, depth, LiDAR, and ultrasound) window dataset named WinSet at various distances and angles. Multiple window types and heterogeneous window states are considered, such as openness (open vs. close), human behind (with vs. without), and lighting (on vs. off). Although our WinSet dataset has many usage scenarios, we concretize two sample ones: (i) analysis of state distinguishability using different sensor modalities and (ii) algorithms to detect open windows. We believe sharing WinSet and its collection procedure with the engineering and research communities will stimulate many creative smart city applications.