Enhancing Reliability of Compute Environments on Amazon EC2 Spot Instances

Amazon Elastic Compute Cloud (EC2) gives access to resources in the form of virtual servers, also known as instances. EC2 Spot Instances (SIs) offer spare compute capacity at steep discounts compared to reliable and fixed price on-demand instances. The drawback, however, is that waiting time until requested spots become fulfilled can be incredible high. In this paper, we propose a container migration-based solution to enhance the reliability of virtual cluster computing environments built on top of non-reserved EC2 pricing model instances. We compare the performance of our algorithm by executing different resource provisioning plans for running real-life workflow applications, constrained by user-defined deadline and budget Quality of Service (QoS) parameters. The results show that our solution is able to successfully conclude almost 98% of workflow applications and more than 99% of workflow tasks for on-demand- and spot block-based virtual compute environments. For SI-based virtual compute environments, our solution achieves similar results, completing more than 98% of workflow applications, and over 99% of workflow tasks, for a worse-case scenario.