Analyzing the distribution fit for storage workload and Internet traffic traces

Abstract Understanding workloads and modeling their performance is important for optimizing systems and services. A useful first step towards understanding the characteristics of workloads is to analyze their inter-arrival times and service requirements. If these characteristics are found to follow certain probability distributions, then corresponding stochastic models can be employed to efficiently estimate the performance of workloads. Such approaches have been explored in specific domains using an assortment of distributions, including the Normal, Weibull, and Exponential. Our primary goal in this work is to understand and model storage workload performance. However, our analysis and others’ past attempts revealed that none of the commonly-employed distributions provided a good fit for storage workloads. We analyzed over 250 traces across 5 different workload families using 20 widely used distributions, including ones seldom used for storage modeling. We found that the Hyper-exponential distribution with just two phases ( H 2 ) was superior in modeling the storage traces compared to other distributions under five diverse metrics of accuracy, including metrics that assess the risk of over-fitting. Based on these results, we developed a Markov-chain-based stochastic model that accurately estimates the storage system performance across several workload traces. To assess the applicability of the Hyper-exponential for distribution fitting beyond storage traces, we evaluated distribution fitting for Internet traffic traces using over 1,600 traces from 3 different sources. We again found that the Hyper-exponential distribution provided a superior fit compared to other probability distributions. To highlight the applicability of our model, we conducted what-if analyses to investigate (i) the storage performance impact of workload variability and garbage collection under various scenarios and (ii) the impact on service response time of Internet flash crowds.