Evaluating Density Forecasting Models

Abstract

Density forecasting in regression is gaining popularity as real world applications demand an estimate of the level of uncertainty in predictions. In this paper we describe the two goals of density forecasting1 sharpness and calibration. We review the evaluation methods available to a density forecaster to assess each of these goals and we introduce a new evaluation method that allows modelers to compare and evaluate their models across both of these goals simultaneously and identify the optimal model.