A multimodal deep fusion graph framework to detect social distancing violations and FCGs in pandemic surveillance

In pandemic surveillance, ensuring social distance has emerged as a challenging issue due to the lack of proper therapeutic agents, and this envisages the need for automated social distance monitoring to avoid the formation of social gatherings and free-standing conversation groups (FCGs). The robustness sought in detecting these groups cannot be achieved when there are illumination variation and occlusion among subjects by solely relying on video data from distributed cameras. In this paper, we propose a deep learning framework for integrating data from multiple sensor modalities taking into account the spatial properties necessary to manage illumination variation and occlusion of video data. From the fused data, social distance compliance violations are notified by the presence of social groups as graphs detected using a pre-trained deep framework and connected components in graph theory. A cost function is devised for social group graph clustering to identify FCGs by using the socio-psychological theory of Friends-formation. Experiment analysis on four benchmark datasets shows that the proposed approach excels at detecting social distance violations and FCGs, and succeeds in analyzing the potential risk of pandemic spread in an area by the calculation of violation scores and rate of violation.