Optimizing Multi-Cloud CDN Deployment and Scheduling Strategies Using Big Data Analysis

With the substantial incensement of broadband networks, Internet applications have shifted from simple web browsing to content-centric applications. From the perspective of the content distributor, how to reduce the cost while satisfying quality of service and how to respond timely to users are of great concern. Currently, using multi-cloud technology is a feasible solution to provide more agile and scalable services. Meanwhile, big data techniques, such as Spark and Hadoop, can help content distributors make load-direct decisions more timely and accurately. In this paper, we present a multi-cloud architecture-supported resource allocation and scheduling optimized strategy through CDN (content delivery network) operation big data analysis. We firstly analyze quantities of CDN operation log data on Spark to evaluate quality of service (QoS) between end users and multi-cloud-based CDN operator. Then we perform a long-term resource deployment algorithm to book the minimum resources to meet users' requests with higher QoS and lower cost. We apply the prediction model ARIMA on Spark to predict the short term demand through analyzing a longer time series data. When the predicted resources cannot satisfy burst demand, we design a new multi-cloud extension algorithm to schedule additional cloud resource to handle overload requests and use precopying algorithm to select media contents to be stored in the new prepared cloud. We implement and evaluate our scheme with real operation logs data provided by China's biggest CDN distributor to show the efficiency of our algorithms.