A Cluster Based Approach for Task Scheduling across Multiple Programming Systems

As the data processing demands have been increasing, different types of data processing systems are being developed. The new programming systems have different characteristics like types of data handled, processing technique and performance. However, multiple new systems have introduced difficulties for non-expert users like choosing the right system and usage methodology of the new systems. In order to relieve the burden of common users of conducting data processing tasks and taking relevant advantage of the systems features, we intend to integrate the popular programming systems and provide more efficient data processing services. In this paper, we propose to address the task scheduling problem for integrating multiple programming systems. We have designed a cluster based approach for task scheduling across multiple programming systems. This approach helps in minimizing the makespan of workflows and resource consumption. The simulation results show that the proposed approach can reduce the resource consumption significantly while achieving a low makespan for the workflows.