Ranking Forecasts by Stochastic Error Distance, Information, and Reliability Measures

The stochastic error distance (SED) introduced by Diebold & Shin (2017) ranks forecast models by divergence between distributions of the errors of the actual and prefect forecast models. The basic SED is defined by the variation distance and provides a representation of the mean absolute error, but by basing ranking on the entire error distribution and divergence, the SED moves beyond the traditional forecast evaluations. First, we establish connections between ranking forecast models by the SED, error entropy, and some partial orderings of distributions. Then we introduce the notion of excess error for forecast errors of magnitudes larger than a tolerance threshold and give the SED representation of the mean excess error (MEE). As a function of the threshold, the MEE is a local risk measure. With the distribution of the absolute error as a prior for the threshold, its Bayes risk is the entropy functional of the survival function, which is a known measure in the information theory and reliability. Notions and results are illustrated using various distributions for the error. The empirical versions of SED, MEE, and its Bayes risk are compared with the mean squared error in ranking regression and ARIMA models for forecasting bond risk premia.