Real-Life Experience with Private Cloud Hosting Heterogeneous Scientific Workloads

We present our experience with hosting scientific workloads in a private scientific cloud, where both the infrastructure as well as the workloads are heterogeneous. We support two major classes of workloads-classic virtual machines (VMs) and batch job computations. Furthermore, containerized applications and scientific portals such as Galaxy are also hosted in this infrastructure. Using our experience, we describe the system setup, the technologies used to run this heterogeneous system as well as some of the problems we have faced when managing this system throughout the years. Modern computing environments such as clouds, grids or HPC clusters are both complex and costly installations. Therefore, it has always been a major challenge to utilize them properly. Things like improper setup or bad scheduling policy may easily hamper overall performance of the whole system. We believe that our experience and observations may help other researchers and system administrators identify new research directions and/or potential weak spots. Importantly, the problems discussed in this paper are based on real-life data from the CERIT Scientific Cloud that we freely offer for further analysis and/or simulations.