pymc_forecast.metrics#

Probabilistic forecast metrics, dim-aware.

Each eval_* metric takes forecast samples and ground truth and reduces them to a single float; crps_empirical() returns elementwise scores and make_mase() builds a metric from training data.

Inputs may be labeled (xarray.DataArray) or raw numpy:

  • DataArray predictions carry their sample dimensions by name — chain / draw (as produced by the forecast drivers) or an already-stacked sample dim. Labeled truth is transposed to the prediction’s dim order and reordered to matching coordinates. Different or duplicate labels are rejected. Unlabeled inputs are positional and must match the value shape exactly; implicit broadcasting is rejected.

  • numpy predictions follow the classical convention: sample axis first.

Ported from numpyro_forecast (itself porting pyro.ops.stats.crps_empirical) with the JAX kernels replaced by NumPy.

pymc_forecast.metrics.DEFAULT_METRICS = {'coverage': <function eval_coverage>, 'crps': <function eval_crps>, 'mae': <function eval_mae>, 'rmse': <function eval_rmse>}#

Default metrics used by evaluate_forecast() and backtest.

pymc_forecast.metrics.crps_empirical(pred, truth)[source]#

Elementwise empirical Continuous Ranked Probability Score.

\[\mathrm{CRPS}(F, y) = \mathbb{E}|X - y| - \tfrac{1}{2}\,\mathbb{E}|X - X'|\]

estimated from the forecast samples with the sorted-sample \(O(n \log n)\) identity.

Parameters:

  • pred (xarray.DataArray or array_like) – Forecast samples: a DataArray with sample dims named chain, draw or sample, or an array with the sample axis first. At least 2 samples are required.

  • truth (xarray.DataArray or array_like) – Ground-truth values with the prediction’s shape without the sample axis. When pred is labeled, labeled truth is aligned by dim name and coordinates; otherwise it is used positionally.

Returns:

numpy.ndarray – Elementwise CRPS as an unlabeled float64 array, one value per data location (value dims in the prediction’s dim order).

Raises:

ValueError – If there are fewer than 2 samples, a labeled pred has no sample dim, or pred and truth disagree on dims, sizes, coordinates or shape.

pymc_forecast.metrics.eval_coverage(pred, truth, *, alpha=0.9)[source]#

Empirical coverage of the central alpha prediction interval.

A well-calibrated forecast has coverage close to alpha. Bind a non-default level with functools.partial(eval_coverage, alpha=0.8).

Parameters:

  • pred (xarray.DataArray or array_like) – Forecast samples: a DataArray with sample dims named chain, draw or sample, or an array with the sample axis first.

  • truth (xarray.DataArray or array_like) – Ground-truth values with the prediction’s shape without the sample axis. When pred is labeled, labeled truth is aligned by dim name and coordinates; otherwise it is used positionally.

  • alpha (float, default 0.9) – Nominal interval probability, in the open interval (0, 1). The bounds are the (1 - alpha) / 2 and 1 - (1 - alpha) / 2 sample quantiles and are inclusive.

Returns:

float – Fraction of value elements whose truth lies inside the interval.

Raises:

ValueError – If alpha is not in (0, 1), pred has no samples, a labeled pred has no sample dim, or pred and truth disagree on dims, sizes, coordinates or shape.

pymc_forecast.metrics.eval_crps(pred, truth)[source]#

Mean empirical CRPS over all data elements (see crps_empirical()).

Parameters:

  • pred (xarray.DataArray or array_like) – Forecast samples: a DataArray with sample dims named chain, draw or sample, or an array with the sample axis first. At least 2 samples are required.

  • truth (xarray.DataArray or array_like) – Ground-truth values with the prediction’s shape without the sample axis. When pred is labeled, labeled truth is aligned by dim name and coordinates; otherwise it is used positionally.

Returns:

float – The elementwise CRPS averaged over every value element.

Raises:

ValueError – If there are fewer than 2 samples, a labeled pred has no sample dim, or pred and truth disagree on dims, sizes, coordinates or shape.

pymc_forecast.metrics.eval_interval_score(pred, truth, *, alpha=0.9)[source]#

Mean Winkler interval score of the central alpha interval.

Rewards narrow intervals, penalizes truth falling outside; lower is better.

Parameters:

  • pred (xarray.DataArray or array_like) – Forecast samples: a DataArray with sample dims named chain, draw or sample, or an array with the sample axis first.

  • truth (xarray.DataArray or array_like) – Ground-truth values with the prediction’s shape without the sample axis. When pred is labeled, labeled truth is aligned by dim name and coordinates; otherwise it is used positionally.

  • alpha (float, default 0.9) – Nominal interval probability, in the open interval (0, 1). The bounds are the (1 - alpha) / 2 and 1 - (1 - alpha) / 2 sample quantiles.

Returns:

float – The interval score averaged over every value element.

Raises:

ValueError – If alpha is not in (0, 1), pred has no samples, a labeled pred has no sample dim, or pred and truth disagree on dims, sizes, coordinates or shape.

pymc_forecast.metrics.eval_mae(pred, truth)[source]#

Mean absolute error of the forecast sample median.

Parameters:

  • pred (xarray.DataArray or array_like) – Forecast samples: a DataArray with sample dims named chain, draw or sample, or an array with the sample axis first.

  • truth (xarray.DataArray or array_like) – Ground-truth values with the prediction’s shape without the sample axis. When pred is labeled, labeled truth is aligned by dim name and coordinates; otherwise it is used positionally.

Returns:

float – The absolute error averaged over every value element.

Raises:

ValueError – If pred has no samples, a labeled pred has no sample dim, or pred and truth disagree on dims, sizes, coordinates or shape.

pymc_forecast.metrics.eval_pinball(pred, truth, *, quantile=0.5)[source]#

Mean pinball (quantile) loss of the forecast quantile.

At quantile=0.5 this is half the mean absolute error.

Parameters:

  • pred (xarray.DataArray or array_like) – Forecast samples: a DataArray with sample dims named chain, draw or sample, or an array with the sample axis first.

  • truth (xarray.DataArray or array_like) – Ground-truth values with the prediction’s shape without the sample axis. When pred is labeled, labeled truth is aligned by dim name and coordinates; otherwise it is used positionally.

  • quantile (float, default 0.5) – Quantile level, in the open interval (0, 1); the point estimate is the corresponding sample quantile.

Returns:

float – The pinball loss averaged over every value element.

Raises:

ValueError – If quantile is not in (0, 1), pred has no samples, a labeled pred has no sample dim, or pred and truth disagree on dims, sizes, coordinates or shape.

pymc_forecast.metrics.eval_rmse(pred, truth)[source]#

Root mean squared error of the forecast sample mean.

Parameters:

  • pred (xarray.DataArray or array_like) – Forecast samples: a DataArray with sample dims named chain, draw or sample, or an array with the sample axis first.

  • truth (xarray.DataArray or array_like) – Ground-truth values with the prediction’s shape without the sample axis. When pred is labeled, labeled truth is aligned by dim name and coordinates; otherwise it is used positionally.

Returns:

float – The root of the squared error averaged over every value element.

Raises:

ValueError – If pred has no samples, a labeled pred has no sample dim, or pred and truth disagree on dims, sizes, coordinates or shape.

pymc_forecast.metrics.evaluate_forecast(pred, truth, *, metrics=None)[source]#

Apply several metrics to the same forecast samples and ground truth.

Parameters:

  • pred (xarray.DataArray or array_like) – Forecast samples, passed unchanged to every metric (labeled with sample dims, or an array with the sample axis first).

  • truth (xarray.DataArray or array_like) – Ground-truth values, passed unchanged to every metric.

  • metrics (mapping, optional) – Mapping of name to metric fn(pred, truth); None uses DEFAULT_METRICS. Bind metric parameters with functools.partial.

Returns:

dict – Maps each metric name (str) to its value (float, cast with float), in the mapping’s iteration order.

Raises:

ValueError – If a metric rejects the inputs (see the individual metrics).

pymc_forecast.metrics.make_mase(train_data, *, seasonality=1)[source]#

Build a Mean Absolute Scaled Error metric scaled by train_data.

MASE divides the forecast MAE (sample-median point estimate) by the in-sample MAE of the seasonal-naive forecast on train_data. The scale is computed once at factory time as a single scalar pooled over every element of train_data (not per series).

Parameters:

  • train_data (xarray.DataArray or array_like) – Training data — a DataArray with a "time" dim, or an array with time on axis 0.

  • seasonality (int, default 1) – Seasonal period (>= 1); 1 is the random-walk naive baseline.

Returns:

callable – A metric mase(pred, truth) -> float equal to eval_mae() divided by the training scale; it accepts the same pred/truth inputs and raises the same errors as eval_mae().

Raises:

ValueError – If seasonality < 1, train_data is not longer than seasonality along time, or the seasonal-naive scale is zero (constant training series).