Biologically-inspired distributed middleware management for stream processing systems

We present a decentralized and dynamic biologically-inspired algorithm for placing dataflow graphs composed of stream processing tasks onto a distributed network of machines, while minimizing the end-to-end latency. Our algorithm responds on-the-fly to placement requests of new flow graphs or to modifications of an already running stream processing flow graph, and dynamically adapts to changes in performance characteristics such as message rates or service times as well as to changes in processor availability or link performance during runtime. Our algorithm is derived by analogy to pheromone-based cooperation between ants to fulfill goals such as food discovery. We have conducted extensive simulation experiments to show the scalability and adaptability of our algorithm.