Towards automatic infrastructure provisioning for highly dynamic streaming applications

In recent years, there has been an increasing need for flexible stream processing platforms that focus on enabling domain experts to model data processing pipelines using graphical user interfaces. Such pipelines can then be automatically deployed and executed in a distributed big data infrastructure. However, a still remaining problem of such stream systems is the manual management of resources, which could be automated by adding an infrastructure manager. The manager uses available information about running pipelines, especially their semantics. This information combined with the real-time information of the underlying services executing pipelines can be used to automate the configuration of modern cluster management tools. With those tools, it is possible to add new resources into the infrastructure of existing systems and to allocate the running application accordingly. This paper presents a novel approach how information about the application and its usage can be used to adapt the infrastructure, which further reduces the manual configuration effort of such systems.