pymc_forecast.evaluate#
Backtesting: score a forecasting model over moving train/test windows.
Each window is a pure refit: the forecaster is constructed on the window’s
training slice and asked to forecast the test slice, and metrics are computed
on the sampled predictions. Two windowing strategies are supported —
"expanding" (train from the start of history) and "rolling" (fixed
training length) — with window bookkeeping ported from numpyro_forecast /
Pyro. Windows are coordinate slices: results carry integer split points and
the predictions keep their real time coords.
- class pymc_forecast.evaluate.BacktestResult(t0, t1, t2, num_samples, train_walltime, test_walltime, metrics, train_metrics=<factory>, prediction=None)[source]#
Bases:
objectPer-window result of a
backtest()run.- t0, t1, t2
Train-begin, train/test split, and test-end positions (integer offsets into the data’s
"time"dim).
- train_walltime, test_walltime
Wall-clock seconds for fitting and forecasting.
- metrics#
Metric name → value on the test window.
- Type:
dict[str,float]
- train_metrics#
Metric name → in-sample value (empty unless
eval_train=True).- Type:
dict[str,float]
- prediction#
Forecast samples for the window (
Noneunlesskeep_predictions=True).- Type:
xarray.core.dataarray.DataArray | None
- pymc_forecast.evaluate.backtest(data, covariates, model_fn, *, forecaster_cls=<class 'pymc_forecast.forecaster.Forecaster'>, metrics=None, per_window_metrics=None, transform=None, window_type=None, train_window=None, min_train_window=1, test_window=None, min_test_window=1, stride=1, num_samples=100, forecaster_options=None, eval_train=False, keep_predictions=False, random_seed=None)[source]#
Backtest a forecasting model on moving
(train, test)windows.- Parameters:
data – The full series (any input
as_dataarray()accepts).covariates – Covariates over the same span as
data(Nonefor models without covariates).model_fn – The model body or
ForecastingModel.forecaster_cls – Forecaster class constructed per window (default
Forecaster).metrics – Metric name → function; defaults to
DEFAULT_METRICS. Bind metric parameters withfunctools.partial.per_window_metrics – Optional
(t0, t1, t2) -> Mappingproducing extra metrics per window (e.g. a MASE scaled by that window’s training slice viamake_mase()).transform – Optional
(pred, truth) -> (pred, truth)applied before metrics (both labeled:predhaschain/drawsample dims,truthis renamed totime_futureso names align).window_type – Windowing controls;
window_type=Noneinfers"rolling"whentrain_windowis set and"expanding"otherwise.train_window – Windowing controls;
window_type=Noneinfers"rolling"whentrain_windowis set and"expanding"otherwise.min_train_window – Windowing controls;
window_type=Noneinfers"rolling"whentrain_windowis set and"expanding"otherwise.test_window – Windowing controls;
window_type=Noneinfers"rolling"whentrain_windowis set and"expanding"otherwise.min_test_window – Windowing controls;
window_type=Noneinfers"rolling"whentrain_windowis set and"expanding"otherwise.stride – Windowing controls;
window_type=Noneinfers"rolling"whentrain_windowis set and"expanding"otherwise.num_samples – Forecast samples per window.
forecaster_options – Extra constructor kwargs for
forecaster_cls— a mapping, or a callable(t0, t1, t2) -> mappingfor per-window options.eval_train – Also score the in-sample posterior predictive of each window.
keep_predictions – Keep each window’s forecast samples on the result.
random_seed – Base seed; per-window fit/forecast seeds are derived deterministically.
- Returns:
list[BacktestResult]– One result per window, in time order.