Neon: Low-Latency Streaming Pipelines for HPC

Real time data analysis in the context of e.g. realtime monitoring or computational steering is an important tool in many fields of science, allowing scientists to make the best use of limited resources such as sensors and HPC platforms. These tools typically rely on large amounts of continuously collected data that needs to be processed in near-real time to avoid wasting compute, storage, and networking resources. Streaming pipelines are a natural fit for this use case but are inconvenient to use on high-performance computing (HPC) systems because of the diverging system software environment with big data, increasing both the cost and the complexity of the solution. In this paper we propose Neon, a clean-slate design of a streaming data processing framework for HPC systems that enables users to create arbitrarily large streaming pipelines. The experimental results on the Bebop supercomputer show significant performance improvements compared with Apache Storm, with up to 2x increased throughput and reduced latency.