pymc_forecast.metrics

Probabilistic forecast metrics, dim-aware.

Every metric takes forecast samples and ground truth and reduces to a float. Inputs may be labeled (xarray.DataArray) or raw numpy:

• DataArray predictions carry their sample dimensions by name — chain / draw (as produced by the forecast drivers) or an already-stacked sample dim. Labeled truth is transposed to the prediction’s dim order, so callers never think about axis positions.

• numpy predictions follow the classical convention: sample axis first.

Ported from numpyro_forecast (itself porting pyro.ops.stats.crps_empirical) with the JAX kernels replaced by NumPy.

pymc_forecast.metrics.DEFAULT_METRICS = {'coverage': <function eval_coverage>, 'crps': <function eval_crps>, 'mae': <function eval_mae>, 'rmse': <function eval_rmse>}

Default metrics used by evaluate_forecast() and backtest.

pymc_forecast.metrics.crps_empirical(pred, truth)

Elementwise empirical Continuous Ranked Probability Score.

\[\mathrm{CRPS}(F, y) = \mathbb{E}|X - y| - \tfrac{1}{2}\,\mathbb{E}|X - X'|\]

estimated from the forecast samples with the sorted-sample \(O(n \log n)\) identity.

Parameters:

• pred – Forecast samples (labeled, or numpy with the sample axis first). At least 2 samples are required.

• truth – Ground-truth values (prediction shape without the sample axis).

Returns:

numpy.ndarray – Elementwise CRPS, one value per data location.

pymc_forecast.metrics.eval_coverage(pred, truth, *, alpha=0.9)

Empirical coverage of the central alpha prediction interval.

A well-calibrated forecast has coverage close to alpha. Bind a non-default level with functools.partial(eval_coverage, alpha=0.8).

pymc_forecast.metrics.eval_crps(pred, truth)

Mean empirical CRPS over all data elements (see crps_empirical()).

pymc_forecast.metrics.eval_interval_score(pred, truth, *, alpha=0.9)

Mean Winkler interval score of the central alpha interval.

Rewards narrow intervals, penalizes truth falling outside; lower is better.

pymc_forecast.metrics.eval_mae(pred, truth)

Mean absolute error of the forecast sample median.

pymc_forecast.metrics.eval_pinball(pred, truth, *, quantile=0.5)

Mean pinball (quantile) loss of the forecast quantile.

At quantile=0.5 this is half the mean absolute error.

pymc_forecast.metrics.eval_rmse(pred, truth)

Root mean squared error of the forecast sample mean.

pymc_forecast.metrics.evaluate_forecast(pred, truth, *, metrics=None)

Apply several metrics to the same forecast samples and ground truth.

Parameters:

• pred – As accepted by the individual metrics (labeled or numpy).

• truth – As accepted by the individual metrics (labeled or numpy).

• metrics – Mapping of name to metric; defaults to DEFAULT_METRICS. Bind metric parameters with functools.partial.

Returns:

dict[str, float] – Each metric name mapped to its value.

pymc_forecast.metrics.make_mase(train_data, *, seasonality=1)

Build a Mean Absolute Scaled Error metric scaled by train_data.

MASE divides the forecast MAE (sample-median point estimate) by the in-sample MAE of the seasonal-naive forecast on train_data. The scale is computed once at factory time.

Parameters:

• train_data – Training data — a DataArray with a "time" dim, or numpy with time on axis 0.

• seasonality – Seasonal period (>= 1); 1 is the random-walk naive baseline.