RealSync: A Synchronous Multimodality Media Stream Analytic Framework for Real-Time Communications Applications

While advancements in computing algorithms and hardware have enabled real-time stream analytics in videos, the information-rich audio bonded with video is still usually dropped and wasted. Processing both audio and video stream is not trivial as synchronizing multiple data streams creates much more difficulties than processing only one stream. In this paper, we designed and implemented a lightweight multimodal stream processing system that keeps both streams synchronized in processing and tested in a typical use case profanity filter. While it is inevitable to slow down certain processors to keep the steams synchronized, by careful butter setting, the overall latency is not affected (still 400-500ms). Besides achieving real-time processing, we located a core problem causing bursty audio latency and gave directions for further latency improvements.