pymc_forecast.evaluate#

Backtesting: score a forecasting model over moving train/test windows.

Each window is a pure refit: the forecaster is constructed on the window’s training slice and asked to forecast the test slice, and metrics are computed on the sampled predictions. Two windowing strategies are supported — "expanding" (train from the start of history) and "rolling" (fixed training length) — with window bookkeeping ported from numpyro_forecast / Pyro. Windows are positional (integer) slices along "time": results carry integer split points and the predictions keep their real time coords.

class pymc_forecast.evaluate.BacktestResult(t0, t1, t2, num_samples, train_walltime, test_walltime, metrics, train_metrics=<factory>, prediction=None)[source]#

Bases: object

Per-window result of a backtest() run.

Each parameter is available as an attribute of the same name.

Parameters:

  • t0 (int) – Inclusive start of the training slice (positional index into the data’s "time" dim); 0 for expanding windows.

  • t1 (int) – Train/test split: exclusive end of training and inclusive start of the test slice.

  • t2 (int) – Exclusive end of the test slice.

  • num_samples (int) – Requested number of forecast samples (the backtest argument).

  • train_walltime (float) – Wall-clock seconds to construct (build and fit) the forecaster.

  • test_walltime (float) – Wall-clock seconds for the forecast call.

  • metrics (dict) – Metric name → value (float) on the test window.

  • train_metrics (dict, optional) – Metric name → in-sample value (float); empty unless eval_train=True. Defaults to an empty dict.

  • prediction (xarray.DataArray, optional) – Forecast samples for the window, after transform if one was given (None unless keep_predictions=True).

pymc_forecast.evaluate.WindowType#

Allowed window_type values of backtest(), "expanding" or "rolling".

alias of Literal[‘expanding’, ‘rolling’]

pymc_forecast.evaluate.backtest(data, covariates, model_fn, *, forecaster_cls=<class 'pymc_forecast.forecaster.Forecaster'>, metrics=None, per_window_metrics=None, transform=None, window_type=None, train_window=None, min_train_window=1, test_window=None, min_test_window=1, stride=1, num_samples=100, forecaster_options=None, eval_train=False, keep_predictions=False, random_seed=None)[source]#

Backtest a forecasting model on moving (train, test) windows.

Parameters:

  • data (xarray.DataArray, pandas.Series, pandas.DataFrame or array_like) – The full series, normalized with as_dataarray() ("time" first; 2-D input gets a "series" dim).

  • covariates (xarray.DataArray, pandas.Series, pandas.DataFrame or array_like or None) – Covariates over the same span as data (normalized with as_dataarray; 2-D input gets a "covariate" dim), sliced by position alongside data. None for models without covariates.

  • model_fn (callable or pymc_forecast.model.ForecastingModel) – The model body or ForecastingModel.

  • forecaster_cls (type, default pymc_forecast.forecaster.Forecaster) – Forecaster class constructed per window as forecaster_cls(model_fn, train_data, window_covariates, random_seed=..., **options), where window_covariates are the covariate rows t0:t2 (training plus test span; zero-width covariates when covariates is None).

  • metrics (mapping, optional) – Metric name → function; defaults to DEFAULT_METRICS. Bind metric parameters with functools.partial.

  • per_window_metrics (callable, optional) – Optional (t0, t1, t2) -> Mapping producing extra metrics per window (e.g. a MASE scaled by that window’s training slice via make_mase()); entries override same-named metrics.

  • transform (callable, optional) – Optional (pred, truth) -> (pred, truth) applied before metrics (both labeled: pred has chain/draw sample dims, truth is renamed to time_future so names align). With eval_train, it is also applied to the in-sample obs samples and the training slice, which both use the "time" dim.

  • window_type ({"expanding", "rolling"}, optional) – "expanding" trains from the start of the series; "rolling" uses a fixed train_window. None infers "rolling" when train_window is set and "expanding" otherwise.

  • train_window (int, optional) – Fixed training length for rolling windows; required for "rolling" and not allowed with "expanding".

  • min_train_window (int, default 1) – Minimum training length; the first split for expanding windows.

  • test_window (int, optional) – Maximum test length; None tests to the end of the series.

  • min_test_window (int, default 1) – Minimum test length; splits stop once fewer than this many test steps would remain, so every window has at least min_test_window test steps.

  • stride (int, default 1) – Step between successive splits.

  • num_samples (int, default 100) – Forecast samples per window (also used for in-sample predictions).

  • forecaster_options (mapping or callable, optional) – Extra constructor kwargs for forecaster_cls — a mapping, or a callable (t0, t1, t2) -> mapping for per-window options.

  • eval_train (bool, default False) – Also score the in-sample posterior predictive of each window.

  • keep_predictions (bool, default False) – Keep each window’s (transformed) forecast samples on the result.

  • random_seed (int or sequence of int, optional) – Base seed (numpy.random.SeedSequence entropy); per-window fit/forecast seeds are derived deterministically.

Returns:

list of BacktestResult – One result per window, in time order.

Raises:

  • pymc_forecast.exceptions.BacktestWindowError – If window_type is unknown or conflicts with train_window, a window parameter is not an integer (or is a bool), a minimum is below 1, test_window < min_test_window, stride < 1, train_window < min_train_window, or the series is too short for any window.

  • pymc_forecast.exceptions.AlignmentError – If data or covariates cannot be normalized; errors raised by forecaster_cls (fitting/forecasting) and by the metrics propagate.

Warns:

UserWarning – From the VI convergence check of the default forecaster_cls.

pymc_forecast.evaluate.results_to_dataframe(results)[source]#

Flatten backtest results into a pandas.DataFrame.

One row per window: split points, walltimes, one column per metric, and train_-prefixed columns for in-sample metrics when present.

Parameters:

results (list of BacktestResult) – Results of backtest().

Returns:

pandas.DataFrame – Columns t0, t1, t2, num_samples, train_walltime, test_walltime, then one per metric and one train_<name> per in-sample metric; prediction is not included.