Scaling Up IoT Stream Processing

Users create large numbers of IoT stream queries with data streams generated from various IoT devices. Current stream processing systems such as Storm and Flink are unable to support such large numbers of IoT stream queries efficiently, as their execution models cause a flurry of cache misses while processing the events of the queries. To solve this problem, we present a new group-aware execution model, which processes the events of IoT stream queries in a way that exploits the locality of data and code references, to reduce cache misses and improve system performance. The group-aware execution model leverages the fact that users create the groups of queries according to their interests or location contexts and that queries in the same group can share the same data and codes. We realize the group-aware execution model on MIST---a new stream processing system tailored for processing many IoT stream queries efficiently---to scale up the number of IoT queries that can be processed in a machine. Our preliminary evaluation shows that our group-aware execution model increases the number of queries that can be processed within a single machine up to 3.18X compared to the Flink-based execution model.