Enhanced Metrics for Prediction Interval Evaluation
Formal Definitions and Comprehensive Analysis of Key Metrics
Key Takeaways
- Coverage Error measures the discrepancy between the empirical coverage of prediction intervals and the desired nominal coverage level.
- Winkler Score combines the width of the prediction interval and penalties for observations falling outside the interval, providing a single measure of interval quality.
- Mean Interval Width (MIW) quantifies the sharpness of prediction intervals by measuring their average width, with narrower intervals being preferred if they maintain adequate coverage.
This section details the metrics used to evaluate both point forecast accuracy and the quality of prediction intervals (PIs). For point forecasts, we employ standard regression metrics computed across all spatial locations and channels, following the established practice of summing over spatial locations and channels.
Point Forecast Metrics
For a prediction \( \hat{\mathbf{Y}}_{t} \) and the corresponding ground truth \( \mathbf{Y}_t \) at time \( t \), the Mean Squared Error (MSE) is defined as:
\[ \text{MSE} = \frac{1}{BT} \sum_{b=1}^B \sum_{t=1}^T \frac{1}{HWC} \sum_{h,w,c} (\mathbf{Y}_{t}^{(b,h,w,c)} - \hat{\mathbf{Y}}_{t}^{(b,h,w,c)})^2 \]
where \( B \) is the batch size, \( T \) is the prediction horizon, \( H \) and \( W \) are the spatial dimensions, and \( C \) is the number of channels. Similarly, the Mean Absolute Error (MAE) is computed as:
\[ \text{MAE} = \frac{1}{BT} \sum_{b=1}^B \sum_{t=1}^T \frac{1}{HWC} \sum_{h,w,c} |\mathbf{Y}_{t}^{(b,h,w,c)} - \hat{\mathbf{Y}}_{t}^{(b,h,w,c)}| \]
Prediction Interval Metrics
To evaluate the quality of prediction intervals, we use several complementary metrics that assess calibration, sharpness, and overall efficacy. These metrics include Coverage Error, Mean Interval Width (MIW), and the Winkler Score.
Coverage Error (CE)
Coverage Error measures the discrepancy between the empirical coverage of the prediction intervals and the nominal coverage level. For a nominal coverage level of \( (1 - \alpha) \), the coverage error is defined as:
\[ \text{Coverage Error} = \left| \frac{1}{N} \sum_{i=1}^N \mathbb{I}\{y_i \in [L_i, U_i]\} - (1 - \alpha) \right| \]
where \( N \) is the number of observations, \( y_i \) is the true value, \( [L_i, U_i] \) is the prediction interval, and \( \mathbb{I}\{\cdot\} \) is the indicator function, which equals 1 if the condition is true and 0 otherwise. A well-calibrated interval should have a coverage error close to zero, indicating that the empirical coverage closely matches the nominal level.
Mean Interval Width (MIW)
The Mean Interval Width (MIW) quantifies the sharpness of the prediction intervals by measuring their average width. It is defined as:
\[ \text{MIW} = \frac{1}{N} \sum_{i=1}^N (U_i - L_i) \]
where \( U_i \) and \( L_i \) are the upper and lower bounds of the prediction interval, respectively. Narrower intervals are preferred, provided they maintain adequate coverage. MIW is also sometimes referred to as sharpness.
Winkler Score (WS)
The Winkler Score combines the width of the prediction interval and the proportion of observations falling outside the interval into a single score. For a prediction interval \( [L_i, U_i] \) and true value \( y_i \), the Winkler score is defined as:
\[ \text{Winkler Score} = (U_i - L_i) + \frac{2}{\alpha} (L_i - y_i) \mathbb{I}\{y_i < L_i\} + \frac{2}{\alpha} (y_i - U_i) \mathbb{I}\{y_i > U_i\} \]
where \( \alpha \) is the significance level. The Winkler score penalizes both wide intervals and observations falling outside the interval. A lower Winkler score indicates better interval quality, reflecting both sharpness and calibration.
Detailed Explanation of Metrics
To further clarify, let's delve into the specifics of each metric:
Understanding Coverage Error
The coverage error is a critical metric for assessing the reliability of prediction intervals. It directly measures how well the intervals capture the true values. For instance, if we aim for a 90% prediction interval, we expect that approximately 90% of the true values will fall within the predicted intervals. A significant deviation from this nominal level indicates a poorly calibrated model. A positive coverage error means that the intervals are under-covering (too narrow), while a negative error means that the intervals are over-covering (too wide). The goal is to minimize the absolute value of the coverage error.
Understanding Mean Interval Width
The mean interval width (MIW) is a measure of the precision of the prediction intervals. It quantifies how narrow or wide the intervals are on average. Narrower intervals are generally preferred because they provide more precise predictions. However, it is crucial to balance the width of the intervals with their coverage. If the intervals are too narrow, they may not capture the true values, leading to poor coverage. Therefore, the goal is to achieve the narrowest possible intervals while maintaining the desired coverage level. This balance is often referred to as sharpness.
Understanding Winkler Score
The Winkler score is a composite metric that combines both the width of the interval and the penalty for observations falling outside the interval. The first term, \( (U_i - L_i) \), represents the width of the interval. The second and third terms penalize observations that fall outside the interval. The penalty is proportional to the distance between the true value and the interval boundary, scaled by \( \frac{2}{\alpha} \). This scaling factor ensures that the penalty is more severe for intervals with lower nominal coverage levels. The Winkler score provides a single, comprehensive measure of interval quality, making it easier to compare different models or methods. A lower Winkler score is better, indicating narrower intervals with fewer and less severe violations.
Practical Implications
These metrics are not just theoretical constructs; they have practical implications for the use of prediction intervals in real-world applications. For example, in weather forecasting, a well-calibrated prediction interval can provide valuable information about the uncertainty associated with a forecast. In financial forecasting, accurate prediction intervals can help investors make more informed decisions. In healthcare, prediction intervals can be used to assess the uncertainty of patient outcomes. Therefore, the careful evaluation of prediction intervals using these metrics is essential for ensuring the reliability and usefulness of forecasting models.
Summary of Metrics
The following table summarizes the key metrics used in our evaluation:
| Metric | Description | Formula | Goal |
|---|---|---|---|
| Mean Squared Error (MSE) | Average of the squared differences between predicted and true values. | \( \frac{1}{BT} \sum_{b=1}^B \sum_{t=1}^T \frac{1}{HWC} \sum_{h,w,c} (\mathbf{Y}_{t}^{(b,h,w,c)} - \hat{\mathbf{Y}}_{t}^{(b,h,w,c)})^2 \) | Minimize |
| Mean Absolute Error (MAE) | Average of the absolute differences between predicted and true values. | \( \frac{1}{BT} \sum_{b=1}^B \sum_{t=1}^T \frac{1}{HWC} \sum_{h,w,c} |\mathbf{Y}_{t}^{(b,h,w,c)} - \hat{\mathbf{Y}}_{t}^{(b,h,w,c)}| \) | Minimize |
| Coverage Error (CE) | Absolute difference between empirical and nominal coverage. | \( \left| \frac{1}{N} \sum_{i=1}^N \mathbb{I}\{y_i \in [L_i, U_i]\} - (1 - \alpha) \right| \) | Minimize (close to 0) |
| Mean Interval Width (MIW) | Average width of the prediction intervals. | \( \frac{1}{N} \sum_{i=1}^N (U_i - L_i) \) | Minimize (while maintaining coverage) |
| Winkler Score (WS) | Combines interval width and penalties for observations outside the interval. | \( (U_i - L_i) + \frac{2}{\alpha} (L_i - y_i) \mathbb{I}\{y_i < L_i\} + \frac{2}{\alpha} (y_i - U_i) \mathbb{I}\{y_i > U_i\} \) | Minimize |
Conclusion
In summary, these metrics provide a comprehensive evaluation of both the accuracy of point forecasts and the quality of prediction intervals. By considering these metrics, we can ensure that our proposed methods deliver precise, reliable, and meaningful predictions. A well-calibrated model should achieve empirical coverage close to the nominal level while maintaining sharp (narrow) prediction intervals. The Winkler score provides a single, comprehensive measure of interval quality, making it easier to compare different models or methods.
References
Last updated January 20, 2025