Agentic forecasting step-by-step¶
skforecast-ai is an AI forecasting assistant that pairs a deterministic engine, powered by skforecast, with an LLM reasoning layer. Simply provide a time series, and the assistant automatically profiles the data, selects a model using established best practices, and evaluates its performance. It returns both the final forecast and the runnable skforecast script that produced it.
This tutorial walks through the step-by-step path: the approach for users who want granular control to inspect or adjust intermediate decisions before committing to a full run. If you prefer a single-call workflow that returns results immediately, see the fast-path tutorial.
Both paths share the same deterministic engine and produce identical results and reproducible skforecast code. The step-by-step path gives you three additional capabilities:
Inspect intermediate objects: examine the
ForecastingProfileandForecastPlanbefore any model is trained, and confirm the assistant's recommendations make sense for your domain.Override any single decision: change the estimator, lags, window features, or preprocessing without re-running the entire pipeline from scratch. Pass the modified plan directly to
forecast()orbacktest().Reuse the profile and plan across branches: build the profile and plan once, then run
forecast()for future predictions andbacktest()for historical evaluation: each reusing the same profile, with no redundant profiling.
The ask() method is available at every stage. It interprets what you pass to it (a profile, a plan, a forecast result, or a backtest result), but it never executes the workflow or silently changes any recommendation.
The following example walks through the step-by-step allowing user to understand and control what happens under the hood. If you prefer the quickest way to go from raw data to a validated forecast with minimal setup, visit the fast-path tutorial.
Assistant initialization¶
The first step is to instantiate a ForecastingAssistant, which will be responsible for executing the entire workflow (profiling, planning, backtesting, and forecasting), as well as explaining the outputs and suggesting improvements.
To activate the optional LLM support, users must pass a string in the format 'provider:model_name' (for example, 'openai:gpt-5.5', 'google:gemini-3-flash-preview', 'anthropic:claude-sonnet-5', or 'ollama:qwen3:8b'). For hosted providers, the corresponding API key must be available as an environment variable or passed explicitly when creating the assistant. In this tutorial, we set send_data_to_llm=False. This ensures strict data privacy: the LLM receives only metadata and summary statistics, never the raw time series values.
In [1]:
Copied!
# Data processing
# ==============================================================================
import os
import numpy as np
import pandas as pd
from skforecast.datasets import fetch_dataset
# Plots
# ==============================================================================
import matplotlib.pyplot as plt
import plotly.graph_objects as go
import plotly.io as pio
import plotly.offline as poff
pio.templates.default = "seaborn"
poff.init_notebook_mode(connected=True)
plt.style.use('seaborn-v0_8-darkgrid')
# skforecast and skforecast-ai
# ==============================================================================
import skforecast
import skforecast_ai
from skforecast_ai import ForecastingAssistant
from skforecast.model_selection import TimeSeriesFold
from skforecast.plot import set_dark_theme
# Python utilities
# ==============================================================================
import textwrap
color = '\033[1m\033[38;5;208m'
print(f"{color}Version skforecast_ai: {skforecast_ai.__version__}")
print(f"{color}Version skforecast: {skforecast.__version__}")
# Data processing
# ==============================================================================
import os
import numpy as np
import pandas as pd
from skforecast.datasets import fetch_dataset
# Plots
# ==============================================================================
import matplotlib.pyplot as plt
import plotly.graph_objects as go
import plotly.io as pio
import plotly.offline as poff
pio.templates.default = "seaborn"
poff.init_notebook_mode(connected=True)
plt.style.use('seaborn-v0_8-darkgrid')
# skforecast and skforecast-ai
# ==============================================================================
import skforecast
import skforecast_ai
from skforecast_ai import ForecastingAssistant
from skforecast.model_selection import TimeSeriesFold
from skforecast.plot import set_dark_theme
# Python utilities
# ==============================================================================
import textwrap
color = '\033[1m\033[38;5;208m'
print(f"{color}Version skforecast_ai: {skforecast_ai.__version__}")
print(f"{color}Version skforecast: {skforecast.__version__}")
Version skforecast_ai: 0.1.0 Version skforecast: 0.23.0
✏️ Note
If you do not have access to an LLM, you can still follow the full tutorial using only the deterministic methods. Profiling, planning, backtesting, and forecasting all run without an LLM. Only the ask() explanations and the LLM-guided variants of refine_plan() and create_cv() require a configured LLM; their deterministic counterparts work without one.
In [ ]:
Copied!
# LLM-enabled assistant
# ==============================================================================
LLM_MODEL = "google:gemini-2.5-flash"
api_key = os.getenv("GOOGLE_API_KEY")
assistant = ForecastingAssistant(
llm=LLM_MODEL, api_key=api_key, send_data_to_llm=False
)
# Using AWS Bedrock
# ==============================================================================
# assistant = ForecastingAssistant(
# llm='bedrock:eu.anthropic.claude-sonnet-4-6',
# base_url="eu-west-1"
# )
# Assistant without LLM (deterministic only)
# ==============================================================================
# assistant = ForecastingAssistant()
# LLM-enabled assistant
# ==============================================================================
LLM_MODEL = "google:gemini-2.5-flash"
api_key = os.getenv("GOOGLE_API_KEY")
assistant = ForecastingAssistant(
llm=LLM_MODEL, api_key=api_key, send_data_to_llm=False
)
# Using AWS Bedrock
# ==============================================================================
# assistant = ForecastingAssistant(
# llm='bedrock:eu.anthropic.claude-sonnet-4-6',
# base_url="eu-west-1"
# )
# Assistant without LLM (deterministic only)
# ==============================================================================
# assistant = ForecastingAssistant()
⚠️ Your data stays private
By default, enabling an LLM does not send your time-series data to the model provider.
The assistant passes only summary statistics, detected frequency,
seasonality flags and the forecaster configuration, never the raw observations.
To explicitly allow it, pass send_data_to_llm=True.
Data¶
The data used in this tutorial represent the hourly usage of the bike share system in the city of Washington, D.C. during the years 2011 and 2012. In addition to the number of users per hour, information about weather conditions and holidays is available.
In [3]:
Copied!
# Downloading data
# ==============================================================================
data = fetch_dataset('bike_sharing', raw=True)
data = data[['date_time', 'users', 'holiday', 'weather', 'temp']]
data['date_time'] = pd.to_datetime(data['date_time'])
data.head()
# Downloading data
# ==============================================================================
data = fetch_dataset('bike_sharing', raw=True)
data = data[['date_time', 'users', 'holiday', 'weather', 'temp']]
data['date_time'] = pd.to_datetime(data['date_time'])
data.head()
╭───────────────────────────────── bike_sharing ──────────────────────────────────╮ │ Description: │ │ Hourly usage of the bike share system in the city of Washington D.C. during the │ │ years 2011 and 2012. In addition to the number of users per hour, information │ │ about weather conditions and holidays is available. │ │ │ │ Source: │ │ Fanaee-T,Hadi. (2013). Bike Sharing Dataset. UCI Machine Learning Repository. │ │ https://doi.org/10.24432/C5W894. │ │ │ │ URL: │ │ https://raw.githubusercontent.com/skforecast/skforecast- │ │ datasets/main/data/bike_sharing_dataset_clean.csv │ │ │ │ Shape: 17544 rows x 12 columns │ ╰─────────────────────────────────────────────────────────────────────────────────╯
Out[3]:
| date_time | users | holiday | weather | temp | |
|---|---|---|---|---|---|
| 0 | 2011-01-01 00:00:00 | 16.0 | 0.0 | clear | 9.84 |
| 1 | 2011-01-01 01:00:00 | 40.0 | 0.0 | clear | 9.02 |
| 2 | 2011-01-01 02:00:00 | 32.0 | 0.0 | clear | 9.02 |
| 3 | 2011-01-01 03:00:00 | 13.0 | 0.0 | clear | 9.84 |
| 4 | 2011-01-01 04:00:00 | 1.0 | 0.0 | clear | 9.84 |
✏️ Note
skforecast-ai is ready to preprocess the data, but it is recommended that users apply their own preprocessing steps before using the assistant. This ensures the data is in the desired format and any necessary transformations have been applied before proceeding with the forecasting workflow.
In [4]:
Copied!
# Interactive plot of time series
# ==============================================================================
fig = go.Figure()
fig.add_trace(
go.Scatter(x=data['date_time'], y=data['users'], mode='lines', name='Users')
)
fig.update_layout(
title = 'Number of users',
xaxis_title="Time",
yaxis_title="Users",
width=800,
height=400,
margin=dict(l=20, r=20, t=35, b=20),
legend=dict(orientation="h", yanchor="top", y=1, xanchor="left", x=0.001)
)
fig.show()
# Interactive plot of time series
# ==============================================================================
fig = go.Figure()
fig.add_trace(
go.Scatter(x=data['date_time'], y=data['users'], mode='lines', name='Users')
)
fig.update_layout(
title = 'Number of users',
xaxis_title="Time",
yaxis_title="Users",
width=800,
height=400,
margin=dict(l=20, r=20, t=35, b=20),
legend=dict(orientation="h", yanchor="top", y=1, xanchor="left", x=0.001)
)
fig.show()
For a deeper walkthrough of the exploratory analysis behind this dataset, see the skforecast example: Forecasting time series with skforecast, XGBoost, LightGBM and CatBoost.
Profile the data¶
The profile() method is the first stage of the step-by-step workflow. It inspects the dataset and returns a ForecastingProfile object that contains:
Data metadata: detected frequency, index type, series lengths, missing values, and exogenous column roles.
Modeling recommendations: the selected forecaster family and estimator, along with alternative candidates and the reasoning behind each choice.
Lag structure: PACF-significant lags per series, used as a baseline for the planning stage.
Window feature suggestions: rolling statistics configurations appropriate for the detected seasonality.
This is a purely deterministic step: no LLM is involved. The profile object is a prerequisite for both plan() and ask() explain mode.
| Attribute | Description |
|---|---|
data_profile |
Full dataset metadata: frequency, index type, series lengths, missing values, exog columns |
forecaster |
Recommended skforecast forecaster class name |
forecaster_candidates |
Ordered list of compatible forecaster names |
estimator |
Recommended estimator class name (None for statistical models) |
estimator_candidates |
Ordered list of compatible estimator names |
series_pacf |
Per-series PACF-significant lags (used by plan() to set default lags) |
window_features |
Suggested window feature configurations |
calendar_features |
Recommended calendar feature names based on detected seasonality |
explanation |
Human-readable explanation of why this forecaster and estimator were selected |
In [5]:
Copied!
# Profile the data
# ==============================================================================
profile = assistant.profile(
data = data,
target = 'users',
date_column = 'date_time'
)
# Profile the data
# ==============================================================================
profile = assistant.profile(
data = data,
target = 'users',
date_column = 'date_time'
)
In [6]:
Copied!
# Inspect the profile
# ==============================================================================
profile
# Inspect the profile
# ==============================================================================
profile
Out[6]:
Dataset Profile ┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Property ┃ Value ┃ ┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩ │ Format │ single │ ├────────────────┼────────────────────────┤ │ Series │ 1 │ ├────────────────┼────────────────────────┤ │ Observations │ 17544 │ ├────────────────┼────────────────────────┤ │ Frequency │ h │ ├────────────────┼────────────────────────┤ │ Target │ users │ ├────────────────┼────────────────────────┤ │ Exog columns │ holiday, weather, temp │ ├────────────────┼────────────────────────┤ │ Missing target │ none │ └────────────────┴────────────────────────┘ Recommendation ┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Property ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ Task type │ single_series │ ├───────────────────────┼────────────────────────────────────────────────────────────────┤ │ Forecaster │ ForecasterRecursive │ ├───────────────────────┼────────────────────────────────────────────────────────────────┤ │ Forecaster candidates │ ForecasterRecursive, ForecasterDirect, ForecasterFoundation, │ │ │ ForecasterStats │ ├───────────────────────┼────────────────────────────────────────────────────────────────┤ │ Estimator │ LGBMRegressor │ ├───────────────────────┼────────────────────────────────────────────────────────────────┤ │ Estimator candidates │ LGBMRegressor, XGBRegressor, Ridge │ └───────────────────────┴────────────────────────────────────────────────────────────────┘ ╭───────────────────────────────────── Explanation ──────────────────────────────────────╮ │ │ │ A single-series ML forecaster (ForecasterRecursive) is recommended. Data: 17544 │ │ observations, 'h' frequency. Alternative forecasters: ['ForecasterDirect', │ │ 'ForecasterFoundation', 'ForecasterStats']. Estimator: LGBMRegressor. A gradient │ │ boosting model is preferred for a dataset of this size (17544 observations). │ │ Alternative estimators: ['XGBRegressor', 'Ridge']. 3 exogenous variables available │ │ as predictors. │ │ │ ╰────────────────────────────────────────────────────────────────────────────────────────╯
Once you have a profile, you can pass it to ask() to get an LLM-generated explanation of the modeling decisions. Note that the pre-computed profile is passed directly, so no profiling work is repeated.
In [7]:
Copied!
# Ask the assistant to explain the profile
# ==============================================================================
answer = assistant.ask(
prompt = (
"Explain why this forecaster and estimator were recommended for my "
"hourly bike-sharing demand data, and what the exogenous variables add."
),
profile = profile,
steps = 36,
)
answer.show_explanation()
# Ask the assistant to explain the profile
# ==============================================================================
answer = assistant.ask(
prompt = (
"Explain why this forecaster and estimator were recommended for my "
"hourly bike-sharing demand data, and what the exogenous variables add."
),
profile = profile,
steps = 36,
)
answer.show_explanation()
╭───────────────────────────────────── Explanation ──────────────────────────────────────╮ │ │ │ Forecasting Plan Explanation: Hourly Bike-Sharing Demand │ │ │ │ Why ForecasterRecursive? │ │ │ │ ForecasterRecursive is the natural starting point for a single time series of this │ │ size. With 17,544 hourly observations (roughly 2 years of data), you have enough │ │ history to train a robust ML model. The recursive strategy uses one model that feeds │ │ its own predictions forward through the 36-step horizon, which is efficient and │ │ well-suited to a medium-length horizon like 36 hours. │ │ │ │ The alternative, ForecasterDirect, would train 36 separate models (one per hour │ │ ahead). That is worth exploring if you find that the relationship between past usage │ │ and future demand changes substantially depending on how far ahead you're predicting │ │ — for example, if 1-hour-ahead patterns differ sharply from 24-hour-ahead patterns. │ │ It is listed as an alternative precisely for that reason. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ Why LightGBM (LGBMRegressor)? │ │ │ │ LightGBM is the preferred estimator here for several reasons: │ │ │ │ • Dataset size: ~17,500 rows is the sweet spot where gradient boosting outperforms │ │ simpler models (Ridge) without needing deep learning. │ │ • Mixed feature types: Your feature set mixes lags (continuous), calendar features │ │ (ordinal), and exogenous variables including categoricals like holiday. LightGBM │ │ handles this natively with its histogram-based splitting. │ │ • NaN tolerance: Some lag/window features will be NaN at the start of the series │ │ (the "warm-up" period). LightGBM can train through NaN rows without requiring │ │ imputation, so dropna_from_series=False is used — no data is wasted. │ │ • Speed and accuracy: LightGBM trains fast and typically achieves lower error than │ │ Random Forest or XGBoost on tabular time series data of this kind. │ │ │ │ XGBRegressor and Ridge are listed as alternatives — Ridge is worth trying as a │ │ linear baseline, while XGBoost is a close competitor to LightGBM. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ Why These Lags? │ │ │ │ The lag selection reflects the natural cycles in hourly bike-sharing demand: │ │ │ │ │ │ Lag group What it captures │ │ ────────────────────────────────────────────────────────────────────────────────── │ │ 1, 2, 3, 5 Very short-term momentum — usage in the last few │ │ hours │ │ 8, 10, 15, 17, 19–26 Intra-day structure — the same part of the │ │ workday/commute window │ │ 32, 33 Yesterday at roughly the same time ± a couple of │ │ hours │ │ 119, 121, 135–145, 160, 166–169 ~5–7 days ago — same hour on nearby days of the │ │ week │ │ 313, 335, 337 ~2 weeks ago — capturing fortnightly or bi-weekly │ │ patterns │ │ │ │ │ │ Notably, lag 24 and lag 168 (exactly 1 day and 1 week ago) are included, which are │ │ the two most important lags for any hourly series with daily and weekly seasonality. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ What the Window Features Add │ │ │ │ The rolling statistics complement the raw lags by summarising recent trend and │ │ variability: │ │ │ │ │ │ Feature Window What it adds │ │ ────────────────────────────────────────────────────────────────────────────────── │ │ mean(window=3) 3 hours Very recent local level — smooths │ │ minute-to-minute noise │ │ std(window=3) 3 hours Recent volatility — high std may signal a │ │ surge or drop in progress │ │ mean(window=24) 24 hours Today's average demand level relative to the │ │ current hour │ │ mean(window=168) 168 hours (1 week) This week's baseline — captures longer-term │ │ level shifts │ │ │ │ │ │ Together they give the model a sense of both immediate context and recent history │ │ without needing to list every single lag between 1 and 168. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ What the Exogenous Variables Add │ │ │ │ Exogenous variables provide information the lag structure alone cannot infer: │ │ │ │ • holiday: Whether the current or forecast hour falls on a public holiday. │ │ Bike-sharing demand on holidays typically resembles a weekend leisure pattern │ │ rather than a weekday commute pattern — this is a structural shift the model │ │ cannot deduce from recent lags alone. │ │ • weather: Weather condition (rain, clear, etc.) may be associated with large, │ │ sudden drops or spikes in demand. A model without this feature would interpret │ │ bad-weather hours as unexplained noise. │ │ • temp: Temperature is likely associated with overall ride volume — comfortable │ │ temperatures tend to coincide with higher usage. It provides a continuous signal │ │ that complements the categorical weather variable. │ │ │ │ ▌ Important: These associations are correlational, not causal. The model learns │ │ ▌ that certain weather/temperature conditions tend to co-occur with particular │ │ ▌ demand levels — it does not establish that weather causes demand changes. │ │ │ │ Because these variables must be known at prediction time, you will need to supply a │ │ 36-hour forecast (or actuals) of holiday, weather, and temp when calling predict(). │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ Why MAE as the Primary Metric? │ │ │ │ Mean Absolute Error is the right choice here because: │ │ │ │ • It is in the same units as the target (number of users), making it directly │ │ interpretable. │ │ • It is robust to outliers — extreme demand spikes (e.g., a city event) won't │ │ dominate the metric the way MSE/RMSE would. │ │ • For operational planning (bike redistribution, dock capacity), knowing the │ │ average error in user counts is more actionable than a squared-error metric. │ │ │ ╰────────────────────────────────────────────────────────────────────────────────────────╯
Build the plan¶
The plan() method converts the coarse modeling decisions in the ForecastingProfile into a fully-specified, executable configuration. It determines:
- Lags: derived from the PACF-significant lags detected in the profile. You can override these explicitly.
- Window features: rolling statistics configurations appropriate for the detected seasonality.
- Preprocessing steps: ordered list of transformations (e.g., differencing, scaling, NaN handling).
- Prediction interval method:
'bootstrapping','conformal', or'native'(selected based on the estimator). - Metrics: the primary and secondary evaluation metrics.
Like profile(), this is a deterministic step. The resulting ForecastPlan object is the complete blueprint that forecast() and backtest() execute.
| Attribute | Description |
|---|---|
forecaster |
Forecaster class name |
estimator |
Estimator class name |
forecaster_kwargs |
All constructor kwargs for the forecaster, including lags and window_features |
estimator_kwargs |
Constructor kwargs for the estimator |
steps |
Forecast horizon |
interval |
Prediction interval quantiles, e.g. [0.1, 0.9] |
interval_method |
Method used to produce the interval (bootstrapping, conformal, or native) |
use_exog |
Whether exogenous variables are included |
preprocessing_steps |
Ordered list of preprocessing actions with code snippets |
explanation |
Human-readable explanation of plan decisions |
In [8]:
Copied!
# Build a plan from the profile
# ==============================================================================
plan = assistant.plan(
profile = profile,
steps = 36,
interval = [0.1, 0.9] # 80% prediction interval
)
# Build a plan from the profile
# ==============================================================================
plan = assistant.plan(
profile = profile,
steps = 36,
interval = [0.1, 0.9] # 80% prediction interval
)
In [9]:
Copied!
# Inspect the plan
# ==============================================================================
plan
# Inspect the plan
# ==============================================================================
plan
Out[9]:
Forecast Plan ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Property ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ Forecaster │ ForecasterRecursive │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Estimator │ LGBMRegressor │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Steps │ 36 │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Frequency │ h │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Lags │ [1, 2, 3, 5, 8, 10, 15, 17, 19, 20, 21, 22, 23, 24, 25, 26, 32, │ │ │ 33, 119, 121, 135, 136, 142, 143, 145, 160, 166, 167, 169, 313, │ │ │ 335, 337] │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Window features │ [{'stats': ['mean', 'std'], 'window_size': 3}, {'stats': ['mean'], │ │ │ 'window_size': 24}, {'stats': ['mean'], 'window_size': 168}] │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Calendar features │ ['hour', 'day_of_week', 'weekend', 'month'] (raw ordinal encoding) │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Use exog │ True │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Interval │ [0.1, 0.9] │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Interval method │ bootstrapping │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Primary metric │ mean_absolute_error │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Preprocessing │ none │ └───────────────────┴────────────────────────────────────────────────────────────────────┘ ╭─────────────────────────────────── Plan Explanation ───────────────────────────────────╮ │ │ │ Plan: ForecasterRecursive + LGBMRegressor. Lags: [1, 2, 3, 5, 8, 10, 15, 17, 19, 20, │ │ 21, 22, 23, 24, 25, 26, 32, 33, 119, 121, 135, 136, 142, 143, 145, 160, 166, 167, │ │ 169, 313, 335, 337]. Window features: ['mean(window=3)', 'std(window=3)', │ │ 'mean(window=24)', 'mean(window=168)']. Calendar features: ['hour', 'day_of_week', │ │ 'weekend', 'month'] (raw ordinal encoding). Prediction intervals via bootstrapping. │ │ NaN rows kept (NaN-tolerant estimator). Exogenous variables included. MAE is │ │ interpretable, robust to outliers, and works at any scale. │ │ │ ╰────────────────────────────────────────────────────────────────────────────────────────╯
Pass both the profile and the plan to ask() for a detailed explanation of the chosen configuration.
In [10]:
Copied!
# Ask the assistant to explain the plan
# ==============================================================================
answer = assistant.ask(
prompt = (
"Walk me through this plan. Why these lags and window features, "
"and how will the 80% prediction interval be produced?"
),
profile = profile,
plan = plan,
)
answer.show_explanation()
# Ask the assistant to explain the plan
# ==============================================================================
answer = assistant.ask(
prompt = (
"Walk me through this plan. Why these lags and window features, "
"and how will the 80% prediction interval be produced?"
),
profile = profile,
plan = plan,
)
answer.show_explanation()
╭───────────────────────────────────── Explanation ──────────────────────────────────────╮ │ │ │ Forecast Plan Walkthrough │ │ │ │ Overview │ │ │ │ The plan uses ForecasterRecursive + LGBMRegressor to forecast 36 hours ahead on a │ │ single hourly series (users) with three exogenous variables (holiday, weather, │ │ temp). Here is what each design choice means and why it was made. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ Lag Selection │ │ │ │ The 32 selected lags split into three distinct groups, each capturing a different │ │ temporal pattern: │ │ │ │ Short-range lags (1–33) │ │ │ │ [1, 2, 3, 5, 8, 10, 15, 17, 19, 20, 21, 22, 23, 24, 25, 26, 32, 33] │ │ │ │ These capture recent autocorrelation — how the last few hours influence the next. │ │ Lags 23–26 are particularly important because they straddle the same hour yesterday │ │ (lag 24), giving the model a direct "what was happening at this time 24 hours ago" │ │ signal. Lags 32–33 reach about 1.3 days back. │ │ │ │ Weekly-cycle lags (~120–169) │ │ │ │ [119, 121, 135, 136, 142, 143, 145, 160, 166, 167, 169] │ │ │ │ These cluster around lag 120 (5 days) and lag 168 (exactly one week). Hourly user │ │ data almost always has a strong weekly seasonality — the model learns "what was │ │ happening at this hour on similar days last week." The lags slightly above and below │ │ 168 (166, 167, 169) let the model handle weekly patterns that are not perfectly │ │ rigid. │ │ │ │ Two-week lag (~313–337) │ │ │ │ [313, 335, 337] │ │ │ │ These reach back approximately two weeks (~336 hours), reinforcing the fortnightly │ │ cycle that appears in many usage patterns (e.g., bi-weekly events or payroll │ │ cycles). │ │ │ │ The lag set was derived from partial autocorrelation analysis (PACF) — only lags │ │ where the PACF exceeds the significance threshold are retained. This avoids feeding │ │ the model redundant or noisy lag columns. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ Window Features │ │ │ │ Four rolling statistics complement the lags by summarising recent history at │ │ multiple scales: │ │ │ │ │ │ Feature Window What it captures │ │ ────────────────────────────────────────────────────────────────────────────────── │ │ roll_mean_3 3 hours Very short-term trend / momentum │ │ roll_std_3 3 hours Local volatility — is the series spiking right now? │ │ roll_mean_24 24 hours The daily average level — smoothed daily context │ │ roll_mean_168 168 hours The weekly average level — where does this week sit │ │ relative to last? │ │ │ │ │ │ The key idea is multi-scale context with very few columns. Instead of adding lags │ │ 4–168 exhaustively, these four features compress the same information. The 168-hour │ │ rolling mean is especially valuable because it gives the model a stable "baseline │ │ level" for the current week without requiring 168 explicit lag columns. │ │ │ │ ▌ One important note: because differentiation is not used here, these rolling │ │ ▌ statistics are computed on the raw series, so their interpretation is │ │ ▌ straightforward — they are genuine rolling averages and standard deviations of │ │ ▌ users. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ Calendar and Exogenous Features │ │ │ │ Calendar features (hour, day_of_week, weekend, month) are included with raw ordinal │ │ encoding. These tell the model where in time each prediction sits — critical for │ │ hourly data where patterns differ sharply between 3am and 3pm, or between Monday and │ │ Saturday. │ │ │ │ The three exogenous variables (holiday, weather, temp) are passed as additional │ │ predictors at both training and prediction time. They may contribute to explaining │ │ demand spikes or drops that the lags and calendar features alone cannot capture — │ │ for example, a public holiday might suppress users even on a normally busy weekday. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ How the 80% Prediction Interval Is Produced │ │ │ │ The interval [0.1, 0.9] means the lower bound targets the 10th percentile and the │ │ upper bound the 90th percentile of predicted demand, giving 80% nominal coverage. │ │ │ │ The method is bootstrapping with binned residuals, which works in four steps: │ │ │ │ Step 1 — Store in-sample residuals at fit time. When the forecaster is trained with │ │ store_in_sample_residuals=True, it records the errors (actual minus predicted) for │ │ every training observation. │ │ │ │ Step 2 — Bin residuals by prediction level. With use_binned_residuals=True, │ │ residuals are grouped into bins based on the magnitude of the corresponding │ │ prediction. This is important because forecast errors for users likely scale with │ │ the level — errors at peak hours behave differently from errors at quiet overnight │ │ hours. │ │ │ │ Step 3 — Bootstrap simulated futures. At prediction time, for each of the 500 │ │ bootstrap replications (n_boot=500), a path of 36 steps is simulated by drawing │ │ residuals from the appropriate bin at each step and adding them to the point │ │ forecast. Because ForecasterRecursive feeds its own predictions back as inputs, each │ │ simulated path compounds the sampled noise across all 36 steps — this propagates │ │ uncertainty over the full horizon. │ │ │ │ Step 4 — Extract quantiles. The 10th and 90th percentiles of the 500 simulated │ │ end-values at each step become the lower and upper bounds. │ │ │ │ The result is a three-column output: pred, lower_bound, upper_bound. You should │ │ check that empirical coverage (how often actual values fall inside the band) is │ │ close to 80% — the calculate_coverage metric in skforecast does this directly. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ Summary │ │ │ │ The plan is well-matched to this dataset. The lag set respects the natural hourly, │ │ daily, and weekly periodicities; the window features add multi-scale context │ │ efficiently; the exogenous variables ground predictions in real-world conditions; │ │ and the bootstrapped intervals correctly propagate 36-step-ahead uncertainty while │ │ accounting for the fact that forecast errors are not constant across demand levels. │ │ │ ╰────────────────────────────────────────────────────────────────────────────────────────╯
Refine the plan (optional)¶
The refine_plan() method lets you adjust the plan before execution. It operates in two distinct modes:
Deterministic mode (
prompt=None): pass explicit configuration overrides such aslags,estimator,estimator_kwargs,forecaster,steps,interval, orwindow_features. Only the fields you explicitly specify are updated; the rest of the configuration is deterministically re-derived from the original plan.LLM mode (
promptprovided): describe your domain knowledge in natural language. The LLM interprets this context and suggests appropriatelagsandwindow_features. Its reasoning is appended toplan.explanationand the changed fields are recorded inplan.llm_refined_fieldsfor full traceability.
⚠ Warning
A refined plan is a hypothesis, not a guaranteed improvement. The LLM may propose lags or window features that are not helpful for the series, or it may misread the domain context you provided. Always compare the refined plan against the original baseline using a proper backtest over multiple folds before adopting it.
Deterministic mode¶
In [11]:
Copied!
# Refine the plan with explicit overrides (no LLM required)
# ==============================================================================
plan_det = assistant.refine_plan(
profile = profile,
plan = plan,
lags = [1, 2, 3, 24, 48, 168],
estimator_kwargs = {'n_estimators': 200, 'max_depth': 6}
)
plan_det
# Refine the plan with explicit overrides (no LLM required)
# ==============================================================================
plan_det = assistant.refine_plan(
profile = profile,
plan = plan,
lags = [1, 2, 3, 24, 48, 168],
estimator_kwargs = {'n_estimators': 200, 'max_depth': 6}
)
plan_det
Out[11]:
Forecast Plan ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Property ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ Forecaster │ ForecasterRecursive │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Estimator │ LGBMRegressor │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Steps │ 36 │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Frequency │ h │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Lags │ [1, 2, 3, 24, 48, 168] │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Window features │ [{'stats': ['mean', 'std'], 'window_size': 3}, {'stats': ['mean'], │ │ │ 'window_size': 24}, {'stats': ['mean'], 'window_size': 168}] │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Calendar features │ ['hour', 'day_of_week', 'weekend', 'month'] (raw ordinal encoding) │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Use exog │ True │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Interval │ [0.1, 0.9] │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Interval method │ bootstrapping │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Primary metric │ mean_absolute_error │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Preprocessing │ none │ └───────────────────┴────────────────────────────────────────────────────────────────────┘ ╭─────────────────────────────────── Plan Explanation ───────────────────────────────────╮ │ │ │ Plan: ForecasterRecursive + LGBMRegressor. Lags: [1, 2, 3, 24, 48, 168]. Window │ │ features: ['mean(window=3)', 'std(window=3)', 'mean(window=24)', │ │ 'mean(window=168)']. Calendar features: ['hour', 'day_of_week', 'weekend', 'month'] │ │ (raw ordinal encoding). Prediction intervals via bootstrapping. NaN rows kept │ │ (NaN-tolerant estimator). Exogenous variables included. MAE is interpretable, robust │ │ to outliers, and works at any scale. │ │ │ ╰────────────────────────────────────────────────────────────────────────────────────────╯
LLM mode¶
In [12]:
Copied!
# Refine the plan using LLM-guided domain knowledge
# ==============================================================================
prompt = (
"I'm forecasting hourly bike rentals. Demand follows a clear daily rhythm with "
"rush-hour peaks, and it changes between weekdays and weekends. It's also usually "
"similar to what happened at the same time last week, and the last few hours give "
"a good sense of the current trend. Please pick lags and rolling features that fit this."
)
plan_refined = assistant.refine_plan(
profile = profile,
plan = plan,
prompt = prompt
)
# Refine the plan using LLM-guided domain knowledge
# ==============================================================================
prompt = (
"I'm forecasting hourly bike rentals. Demand follows a clear daily rhythm with "
"rush-hour peaks, and it changes between weekdays and weekends. It's also usually "
"similar to what happened at the same time last week, and the last few hours give "
"a good sense of the current trend. Please pick lags and rolling features that fit this."
)
plan_refined = assistant.refine_plan(
profile = profile,
plan = plan,
prompt = prompt
)
In [13]:
Copied!
# Refined plan proposed by the assistant
# ==============================================================================
plan_refined
# Refined plan proposed by the assistant
# ==============================================================================
plan_refined
Out[13]:
Forecast Plan ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Property ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ Forecaster │ ForecasterRecursive │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Estimator │ LGBMRegressor │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Steps │ 36 │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Frequency │ h │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Lags │ [1, 2, 3, 4, 5, 6, 23, 24, 25, 47, 48, 49, 167, 168, 169] │ │ │ (LLM-suggested) │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Window features │ [{'stats': ['mean', 'std'], 'window_size': 24}, {'stats': ['mean', │ │ │ 'max'], 'window_size': 168}] (LLM-suggested) │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Calendar features │ ['hour', 'day_of_week', 'weekend', 'month'] (raw ordinal encoding) │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Use exog │ True │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Interval │ [0.1, 0.9] │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Interval method │ bootstrapping │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Primary metric │ mean_absolute_error │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Preprocessing │ none │ └───────────────────┴────────────────────────────────────────────────────────────────────┘ ╭─────────────────────────────────── Plan Explanation ───────────────────────────────────╮ │ │ │ Plan: ForecasterRecursive + LGBMRegressor. Lags: [1, 2, 3, 4, 5, 6, 23, 24, 25, 47, │ │ 48, 49, 167, 168, 169]. Window features: ['mean(window=24)', 'std(window=24)', │ │ 'mean(window=168)', 'max(window=168)']. Calendar features: ['hour', 'day_of_week', │ │ 'weekend', 'month'] (raw ordinal encoding). Prediction intervals via bootstrapping. │ │ NaN rows kept (NaN-tolerant estimator). Exogenous variables included. MAE is │ │ interpretable, robust to outliers, and works at any scale. │ │ │ │ LLM Refinement Reasoning: Lags chosen: │ │ │ │ 1 Recent trend (lags 1–6): The user notes that "the last few hours give a good │ │ sense of the current trend." Lags 1–6 capture the immediate short-term momentum │ │ (e.g., a demand ramp-up or cool-down around rush hour), giving the model a window │ │ into what's happening right now. │ │ 2 Same hour yesterday (lags 23, 24, 25): The daily rhythm is the dominant signal in │ │ bike rentals. Lags 23/24/25 straddle the exact same hour from the previous day, │ │ letting the model learn the typical peak/trough structure (morning rush, lunch, │ │ evening rush) while being robust to small timing shifts. │ │ 3 Same hour two days ago (lags 47, 48, 49): A second daily anchor reinforces the │ │ daily pattern and helps distinguish weekday vs. weekend dynamics (e.g., Monday │ │ vs. Tuesday behaviour) when combined with the weekly lags. │ │ 4 Same hour last week (lags 167, 168, 169): The user explicitly states demand "is │ │ usually similar to what happened at the same time last week." Lags 167/168/169 │ │ directly encode the weekly seasonal signal — capturing the weekday/weekend │ │ contrast (e.g., a Tuesday at 8 am vs. a Saturday at 8 am). │ │ │ │ Window features chosen: │ │ │ │ 1 mean + std over 24 hours: A 24-hour rolling mean captures the recent daily-level │ │ baseline (is today busier or quieter than usual?), while the 24-hour std captures │ │ intra-day volatility — critical for distinguishing calm midday periods from │ │ volatile rush-hour windows. │ │ 2 mean + max over 168 hours (1 week): A weekly rolling mean represents the recent │ │ weekly demand level, complementing the lag-168 point estimate with a smoothed │ │ context. The weekly max captures peak demand conditions over the last 7 days, │ │ which is informative for capacity-sensitive rush-hour forecasting and helps model │ │ the weekday/weekend regime difference. │ │ │ │ Note: the LLM-suggested lags and window_features are hypotheses, not validated │ │ improvements. Confirm any expected accuracy gain before relying on them. │ │ │ ╰────────────────────────────────────────────────────────────────────────────────────────╯
Explain mode (refined plan)¶
In [14]:
Copied!
# Ask the assistant what changed and why
# ==============================================================================
answer = assistant.ask(
prompt = (
"What changed in the refined plan compared to the original, "
"and why does it matter for this dataset?"
),
profile = profile,
plan = plan_refined,
)
answer.show_explanation()
# Ask the assistant what changed and why
# ==============================================================================
answer = assistant.ask(
prompt = (
"What changed in the refined plan compared to the original, "
"and why does it matter for this dataset?"
),
profile = profile,
plan = plan_refined,
)
answer.show_explanation()
╭───────────────────────────────────── Explanation ──────────────────────────────────────╮ │ │ │ Refined Plan vs. Original: What Changed and Why It Matters │ │ │ │ Overview of Changes │ │ │ │ The refinement made two targeted adjustments to the default configuration: expanded │ │ lag selection and enriched window features. Both changes are motivated by the │ │ specific temporal structure of an hourly bike-sharing dataset. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ 1. Lag Selection │ │ │ │ What changed │ │ │ │ A generic default might use a simple contiguous block (e.g., lags 1–24). The refined │ │ plan uses a sparse, seasonality-aware set: │ │ │ │ │ │ Group Lags Rationale │ │ ──────────────────────────────────────────────────────────────────── │ │ Recent trend 1, 2, 3, 4, 5, 6 Immediate short-term momentum │ │ Yesterday (±1 h) 23, 24, 25 Same hour the prior day │ │ Two days ago (±1 h) 47, 48, 49 Second daily anchor │ │ Last week (±1 h) 167, 168, 169 Same hour last week │ │ │ │ │ │ Why it matters for this dataset │ │ │ │ Bike rentals are driven by two overlapping seasonal cycles — daily commuter patterns │ │ (rush hours, lunch) and weekly patterns (weekday vs. weekend behaviour). Using lags │ │ at 24 and 168 hours directly encodes these cycles without forcing the model to │ │ reconstruct them from a dense contiguous block. The ±1 hour buffer around each │ │ anchor (e.g., 23/24/25 rather than just 24) makes the model robust to small timing │ │ shifts in when peaks actually occur, day to day. │ │ │ │ Including lags 1–6 alongside the seasonal anchors means the model can simultaneously │ │ track where you are in a pattern right now (the current ramp-up or wind-down) and │ │ what that same moment looked like historically. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ 2. Window Features │ │ │ │ What changed │ │ │ │ Four rolling statistics are included, grouped into two windows: │ │ │ │ │ │ Window Statistics Purpose │ │ ─────────────────────────────────────────────────────────────────── │ │ 24 hours mean, std Recent daily baseline + intra-day volatility │ │ 168 hours mean, max Weekly demand level + peak demand context │ │ │ │ │ │ Why it matters for this dataset │ │ │ │ • The 24-hour mean answers "is today's demand generally higher or lower than a │ │ typical day?" — smoothing out noise from individual lag values. │ │ • The 24-hour std captures how variable the last 24 hours have been. High std │ │ signals a rush-hour window; low std signals a quiet off-peak period. This is a │ │ feature that discrete lags cannot express on their own. │ │ • The 168-hour mean provides a smoothed weekly baseline, complementing the point │ │ estimate from lag 168 with a broader trend signal. │ │ • The 168-hour max captures the recent peak demand over the last 7 days. For a │ │ capacity-sensitive series like bike rentals, knowing how extreme the recent peak │ │ was helps the model distinguish a heavy weekday-commute regime from a quieter │ │ weekend-leisure regime. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ 3. What Did Not Change (and Why That's Fine) │ │ │ │ • Estimator (LGBMRegressor): Gradient boosting handles the non-linear interactions │ │ between time-of-day, day-of-week, weather, and the lag features naturally — no │ │ change needed. │ │ • Exogenous variables (holiday, weather, temp): These are included as-is. Weather │ │ and temperature are strong drivers of outdoor activity, and holiday directly │ │ modifies the weekday/weekend pattern assumption. │ │ • Calendar features (hour, day_of_week, weekend, month): These complement the lags │ │ by giving the model an explicit, ordinal sense of where in time a prediction │ │ sits. │ │ • Prediction intervals (bootstrapping, 80%): Appropriate for a dataset of this size │ │ and consistent with the NaN-tolerant LightGBM setup. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ Key Caveat │ │ │ │ ▌ The LLM-suggested lags and window features are hypotheses, not validated │ │ ▌ improvements. The only way to confirm they improve accuracy is to compare │ │ ▌ backtesting MAE against a simpler baseline (e.g., contiguous lags 1–24 with no │ │ ▌ window features). Always verify via backtesting_forecaster before relying on the │ │ ▌ refined configuration in production. │ │ │ ╰────────────────────────────────────────────────────────────────────────────────────────╯
Forecast¶
Once you have a profile and a plan, you can call forecast() or forecast_code(). Both accept the pre-computed profile and plan so no additional profiling is performed. The forecast() method executes the generated script and returns a ForecastResult; forecast_code() generates the script only, without running it.
The forecast branch operates in two modes:
Evaluation mode (
test_sizeis set): the dataset is split into train and test sets, the model is trained on the train portion, and predictions are compared against the held-out actuals to compute metrics.Prediction mode (
test_size=None, the default): the model is trained on the entire dataset and forecasts the nextstepstime points into the future. Because there is no ground truth, no metrics are returned. If the data has exogenous variables, their future values must be supplied viaexog.
Evaluation mode¶
In [15]:
Copied!
# Forecast in evaluation mode, reusing the pre-computed profile and plan
# ==============================================================================
results_eval = assistant.forecast(
data = data,
target = 'users',
date_column = 'date_time',
steps = 36,
interval = [0.1, 0.9], # 80% prediction interval
test_size = 36, # Last 36 hours as test set
profile = profile, # Reuse the pre-computed profile
plan = plan_refined # Reuse the refined plan
)
display(results_eval.metrics)
display(results_eval.predictions.head())
# Forecast in evaluation mode, reusing the pre-computed profile and plan
# ==============================================================================
results_eval = assistant.forecast(
data = data,
target = 'users',
date_column = 'date_time',
steps = 36,
interval = [0.1, 0.9], # 80% prediction interval
test_size = 36, # Last 36 hours as test set
profile = profile, # Reuse the pre-computed profile
plan = plan_refined # Reuse the refined plan
)
display(results_eval.metrics)
display(results_eval.predictions.head())
╭─────────────────────────────── IgnoredArgumentWarning ───────────────────────────────╮ │ A pre-built `plan` was provided, so the following argument(s) are ignored: │ │ ['interval']. To change these, refine the plan with `refine_plan()` before calling. │ │ │ │ Category : skforecast.exceptions.IgnoredArgumentWarning │ │ Location : │ │ /home/ubuntu/miniconda3/envs/skforecast_ai/lib/python3.13/site-packages/skforecast_a │ │ i/_utils.py:392 │ │ Suppress : warnings.simplefilter('ignore', category=IgnoredArgumentWarning) │ ╰──────────────────────────────────────────────────────────────────────────────────────╯
| series | MAE | MSE | MASE | MAPE | |
|---|---|---|---|---|---|
| 0 | users | 40.923829 | 3810.711588 | 0.63557 | 0.488009 |
| pred | lower_bound | upper_bound | |
|---|---|---|---|
| 2012-12-30 12:00:00 | 146.245548 | 110.543227 | 181.414944 |
| 2012-12-30 13:00:00 | 136.316434 | 93.000437 | 170.402002 |
| 2012-12-30 14:00:00 | 129.139815 | 90.684704 | 164.117185 |
| 2012-12-30 15:00:00 | 132.205230 | 91.023046 | 171.324095 |
| 2012-12-30 16:00:00 | 136.448864 | 87.288046 | 173.381217 |
In [16]:
Copied!
# Plot predictions vs. actual values for the held-out test period
# ==============================================================================
set_dark_theme()
preds = results_eval.predictions
fig, ax = plt.subplots(figsize=(7, 3.5))
data.set_index('date_time').loc[preds.index, 'users'].plot(ax=ax, label='actual')
preds['pred'].plot(ax=ax, label='prediction')
if {'lower_bound', 'upper_bound'}.issubset(preds.columns):
ax.fill_between(
preds.index, preds['lower_bound'], preds['upper_bound'],
alpha=0.3, label='80% prediction interval'
)
ax.set_title('Predictions vs. actual bike demand')
ax.set_ylabel('Users')
ax.legend()
plt.tight_layout()
plt.show()
# Plot predictions vs. actual values for the held-out test period
# ==============================================================================
set_dark_theme()
preds = results_eval.predictions
fig, ax = plt.subplots(figsize=(7, 3.5))
data.set_index('date_time').loc[preds.index, 'users'].plot(ax=ax, label='actual')
preds['pred'].plot(ax=ax, label='prediction')
if {'lower_bound', 'upper_bound'}.issubset(preds.columns):
ax.fill_between(
preds.index, preds['lower_bound'], preds['upper_bound'],
alpha=0.3, label='80% prediction interval'
)
ax.set_title('Predictions vs. actual bike demand')
ax.set_ylabel('Users')
ax.legend()
plt.tight_layout()
plt.show()
In [17]:
Copied!
# Ask the assistant to interpret the forecast results
# ==============================================================================
answer = assistant.ask(
prompt = "Explain the results of this forecast, including the metrics and predictions.",
forecast_result = results_eval
)
answer.show_explanation()
# Ask the assistant to interpret the forecast results
# ==============================================================================
answer = assistant.ask(
prompt = "Explain the results of this forecast, including the metrics and predictions.",
forecast_result = results_eval
)
answer.show_explanation()
╭───────────────────────────────────── Explanation ──────────────────────────────────────╮ │ │ │ Forecast Results Explanation │ │ │ │ Overview │ │ │ │ The model forecasts 36 hours of hourly bike rental demand (users) from 2012-12-30 │ │ 12:00 through 2012-12-31 23:00, using a ForecasterRecursive with LightGBM and a rich │ │ set of lag and rolling features. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ Evaluation Metrics │ │ │ │ │ │ Metric Value Interpretation │ │ ────────────────────────────────────────────────────────────────────────────────── │ │ MAE 40.92 On average, predictions are off by ~41 users per hour │ │ MSE 3,810.7 Elevated by occasional larger errors; less interpretable directly │ │ MASE 0.636 Better than naive by ~36% — a strong result │ │ MAPE 0.488 ~49% average percentage error │ │ │ │ │ │ Key takeaways on metrics │ │ │ │ • MASE of 0.636 is the headline result. Since MASE < 1 means the model outperforms │ │ a naïve "same value as last period" baseline, a score of 0.636 confirms the model │ │ adds genuine predictive value — it is roughly 36% more accurate than simply │ │ repeating the previous observation. │ │ • MAE of ~41 users is interpretable in absolute terms. Whether this is acceptable │ │ depends on the operating context (e.g., if peak demand reaches ~175 users, an │ │ error of 41 represents about 23% of peak). │ │ • MAPE of ~49% appears high, but this is common with bike rental data. MAPE │ │ inflates during low-demand hours (late night, when the denominator is very small │ │ — e.g., 5–20 users). It is less reliable here than MAE or MASE. │ │ • MSE of 3,811 suggests the model makes some notably larger errors at certain │ │ hours, likely around demand peaks or transitions (rush hours), where the │ │ recursive multi-step propagation of small errors compounds. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ Prediction Patterns │ │ │ │ 2012-12-30 afternoon (start of horizon) │ │ │ │ • Predictions begin at ~146 users at 12:00 and gradually taper through the │ │ afternoon (132–136 range at 15:00–16:00), consistent with a typical │ │ post-lunchtime plateau before an evening peak. │ │ • The 80% prediction interval here is relatively tight (~70-unit width), reflecting │ │ higher confidence when the model has rich recent context. │ │ │ │ 2012-12-31 evening (end of horizon) │ │ │ │ • Demand drops sharply through the evening: 52 users at 19:00 → 18 users by 23:00, │ │ reflecting the expected late-night/New Year's Eve pattern where cycling activity │ │ winds down. │ │ • Interval width widens toward the end of the horizon (e.g., 23:00 interval spans │ │ ~35 units vs. ~70 units at the start), which is expected behaviour — uncertainty │ │ accumulates as the recursive forecaster steps further into the future. │ │ │ │ Overall prediction range │ │ │ │ │ │ Statistic Point forecast Lower bound Upper bound │ │ ─────────────────────────────────────────────────── │ │ Min 5.2 users 0.8 11.7 │ │ Max 175.1 users 110.5 207.1 │ │ Mean 76.5 users 41.0 108.0 │ │ │ │ │ │ The mean prediction of 76.5 users/hour is plausible for a 36-hour window spanning a │ │ Sunday afternoon into New Year's Eve. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ Prediction Intervals (80% Coverage) │ │ │ │ The intervals were generated via bootstrapping (resampling in-sample residuals). Key │ │ observations: │ │ │ │ • The 80% interval means that, if the model is well-calibrated, approximately 8 out │ │ of every 10 actual observations should fall within the lower_bound–upper_bound │ │ range. │ │ • Intervals are asymmetric (the upper bound is generally further from the point │ │ estimate than the lower bound), which is typical for count-like demand data where │ │ under-forecasting risk is higher during peaks. │ │ • The upper bound reaches 207 users at peak, which may be operationally important │ │ for capacity planning — this represents the plausible high-demand scenario within │ │ the 80% confidence range. │ │ │ │ ▌ ⚠️ Calibration should be verified. The stated 80% coverage is only guaranteed if │ │ ▌ actual coverage (proportion of true values falling inside the interval) matches │ │ ▌ 0.80 on a held-out test set. Consider computing calculate_coverage on backtest │ │ ▌ predictions to confirm. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ Feature Design Rationale │ │ │ │ The strong MASE result is supported by the lag and window feature choices: │ │ │ │ • Lags 23–25 and 167–169 directly anchor the model to the same hour yesterday and │ │ same hour last week — the dominant seasonal signals in bike rentals. │ │ • Rolling mean/std over 24 hours give the model a sense of whether today is busier │ │ or more volatile than a typical day. │ │ • Rolling mean/max over 168 hours encode the recent weekly demand regime and peak │ │ conditions. │ │ • Calendar features (hour, day_of_week, weekend, month) and exogenous variables │ │ (holiday, weather, temp) provide the contextual signals that explain demand │ │ deviations from the seasonal baseline. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ Summary │ │ │ │ ▌ The model performs well above the naïve baseline (MASE 0.64) and produces │ │ ▌ physically plausible hourly demand forecasts with widening uncertainty toward │ │ ▌ the end of the 36-hour horizon. The high MAPE is likely an artefact of low │ │ ▌ overnight demand hours and should not be the primary quality signal. Verify │ │ ▌ interval calibration against actual coverage before using these bounds for │ │ ▌ operational decisions. │ │ │ ╰────────────────────────────────────────────────────────────────────────────────────────╯
Prediction mode¶
In prediction mode, the model trains on the entire dataset and forecasts the next steps time points. Because the data includes exogenous variables (holiday, weather, temp), their future values must be supplied via the exog argument.
In [18]:
Copied!
# Forecast the next 36 hours using the entire dataset (prediction mode)
# ==============================================================================
# Simulate future values of exogenous variables for the next 36 hours
exog = data[['holiday', 'weather', 'temp']].tail(36).copy()
exog.index = pd.date_range(
start=pd.to_datetime(data['date_time'].max()) + pd.Timedelta(hours=1),
periods=36,
freq='h'
)
results_pred = assistant.forecast(
data = data,
target = 'users',
date_column = 'date_time',
steps = 36,
interval = [0.1, 0.9],
test_size = None, # Use the entire dataset (prediction mode)
exog = exog, # Future values of exogenous variables
profile = profile,
plan = plan_refined
)
display(results_pred.predictions.head())
# Forecast the next 36 hours using the entire dataset (prediction mode)
# ==============================================================================
# Simulate future values of exogenous variables for the next 36 hours
exog = data[['holiday', 'weather', 'temp']].tail(36).copy()
exog.index = pd.date_range(
start=pd.to_datetime(data['date_time'].max()) + pd.Timedelta(hours=1),
periods=36,
freq='h'
)
results_pred = assistant.forecast(
data = data,
target = 'users',
date_column = 'date_time',
steps = 36,
interval = [0.1, 0.9],
test_size = None, # Use the entire dataset (prediction mode)
exog = exog, # Future values of exogenous variables
profile = profile,
plan = plan_refined
)
display(results_pred.predictions.head())
╭─────────────────────────────── IgnoredArgumentWarning ───────────────────────────────╮ │ A pre-built `plan` was provided, so the following argument(s) are ignored: │ │ ['interval']. To change these, refine the plan with `refine_plan()` before calling. │ │ │ │ Category : skforecast.exceptions.IgnoredArgumentWarning │ │ Location : │ │ /home/ubuntu/miniconda3/envs/skforecast_ai/lib/python3.13/site-packages/skforecast_a │ │ i/_utils.py:392 │ │ Suppress : warnings.simplefilter('ignore', category=IgnoredArgumentWarning) │ ╰──────────────────────────────────────────────────────────────────────────────────────╯
| pred | lower_bound | upper_bound | |
|---|---|---|---|
| 2013-01-01 00:00:00 | 23.710329 | 14.499605 | 34.620105 |
| 2013-01-01 01:00:00 | 13.736574 | 4.459790 | 22.583204 |
| 2013-01-01 02:00:00 | 8.889456 | 2.781534 | 18.004651 |
| 2013-01-01 03:00:00 | 6.263093 | 1.728842 | 12.564202 |
| 2013-01-01 04:00:00 | 6.263093 | 1.615673 | 12.644115 |
In [19]:
Copied!
# Full results object
# ==============================================================================
results_pred
# Full results object
# ==============================================================================
results_pred
Out[19]:
Dataset Profile ┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Property ┃ Value ┃ ┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩ │ Format │ single │ ├────────────────┼────────────────────────┤ │ Series │ 1 │ ├────────────────┼────────────────────────┤ │ Observations │ 17544 │ ├────────────────┼────────────────────────┤ │ Frequency │ h │ ├────────────────┼────────────────────────┤ │ Target │ users │ ├────────────────┼────────────────────────┤ │ Exog columns │ holiday, weather, temp │ ├────────────────┼────────────────────────┤ │ Missing target │ none │ └────────────────┴────────────────────────┘ Recommendation ┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Property ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ Task type │ single_series │ ├───────────────────────┼────────────────────────────────────────────────────────────────┤ │ Forecaster │ ForecasterRecursive │ ├───────────────────────┼────────────────────────────────────────────────────────────────┤ │ Forecaster candidates │ ForecasterRecursive, ForecasterDirect, ForecasterFoundation, │ │ │ ForecasterStats │ ├───────────────────────┼────────────────────────────────────────────────────────────────┤ │ Estimator │ LGBMRegressor │ ├───────────────────────┼────────────────────────────────────────────────────────────────┤ │ Estimator candidates │ LGBMRegressor, XGBRegressor, Ridge │ └───────────────────────┴────────────────────────────────────────────────────────────────┘ ╭───────────────────────────────────── Explanation ──────────────────────────────────────╮ │ │ │ A single-series ML forecaster (ForecasterRecursive) is recommended. Data: 17544 │ │ observations, 'h' frequency. Alternative forecasters: ['ForecasterDirect', │ │ 'ForecasterFoundation', 'ForecasterStats']. Estimator: LGBMRegressor. A gradient │ │ boosting model is preferred for a dataset of this size (17544 observations). │ │ Alternative estimators: ['XGBRegressor', 'Ridge']. 3 exogenous variables available │ │ as predictors. │ │ │ ╰────────────────────────────────────────────────────────────────────────────────────────╯ Forecast Plan ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Property ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ Forecaster │ ForecasterRecursive │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Estimator │ LGBMRegressor │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Steps │ 36 │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Frequency │ h │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Lags │ [1, 2, 3, 4, 5, 6, 23, 24, 25, 47, 48, 49, 167, 168, 169] │ │ │ (LLM-suggested) │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Window features │ [{'stats': ['mean', 'std'], 'window_size': 24}, {'stats': ['mean', │ │ │ 'max'], 'window_size': 168}] (LLM-suggested) │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Calendar features │ ['hour', 'day_of_week', 'weekend', 'month'] (raw ordinal encoding) │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Use exog │ True │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Interval │ [0.1, 0.9] │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Interval method │ bootstrapping │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Primary metric │ mean_absolute_error │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Preprocessing │ none │ └───────────────────┴────────────────────────────────────────────────────────────────────┘ ╭─────────────────────────────────── Plan Explanation ───────────────────────────────────╮ │ │ │ Plan: ForecasterRecursive + LGBMRegressor. Lags: [1, 2, 3, 4, 5, 6, 23, 24, 25, 47, │ │ 48, 49, 167, 168, 169]. Window features: ['mean(window=24)', 'std(window=24)', │ │ 'mean(window=168)', 'max(window=168)']. Calendar features: ['hour', 'day_of_week', │ │ 'weekend', 'month'] (raw ordinal encoding). Prediction intervals via bootstrapping. │ │ NaN rows kept (NaN-tolerant estimator). Exogenous variables included. MAE is │ │ interpretable, robust to outliers, and works at any scale. │ │ │ │ LLM Refinement Reasoning: Lags chosen: │ │ │ │ 1 Recent trend (lags 1–6): The user notes that "the last few hours give a good │ │ sense of the current trend." Lags 1–6 capture the immediate short-term momentum │ │ (e.g., a demand ramp-up or cool-down around rush hour), giving the model a window │ │ into what's happening right now. │ │ 2 Same hour yesterday (lags 23, 24, 25): The daily rhythm is the dominant signal in │ │ bike rentals. Lags 23/24/25 straddle the exact same hour from the previous day, │ │ letting the model learn the typical peak/trough structure (morning rush, lunch, │ │ evening rush) while being robust to small timing shifts. │ │ 3 Same hour two days ago (lags 47, 48, 49): A second daily anchor reinforces the │ │ daily pattern and helps distinguish weekday vs. weekend dynamics (e.g., Monday │ │ vs. Tuesday behaviour) when combined with the weekly lags. │ │ 4 Same hour last week (lags 167, 168, 169): The user explicitly states demand "is │ │ usually similar to what happened at the same time last week." Lags 167/168/169 │ │ directly encode the weekly seasonal signal — capturing the weekday/weekend │ │ contrast (e.g., a Tuesday at 8 am vs. a Saturday at 8 am). │ │ │ │ Window features chosen: │ │ │ │ 1 mean + std over 24 hours: A 24-hour rolling mean captures the recent daily-level │ │ baseline (is today busier or quieter than usual?), while the 24-hour std captures │ │ intra-day volatility — critical for distinguishing calm midday periods from │ │ volatile rush-hour windows. │ │ 2 mean + max over 168 hours (1 week): A weekly rolling mean represents the recent │ │ weekly demand level, complementing the lag-168 point estimate with a smoothed │ │ context. The weekly max captures peak demand conditions over the last 7 days, │ │ which is informative for capacity-sensitive rush-hour forecasting and helps model │ │ the weekday/weekend regime difference. │ │ │ │ Note: the LLM-suggested lags and window_features are hypotheses, not validated │ │ improvements. Confirm any expected accuracy gain before relying on them. │ │ │ ╰────────────────────────────────────────────────────────────────────────────────────────╯ Predictions (36 rows) ┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓ ┃ Index ┃ pred ┃ lower_bound ┃ upper_bound ┃ ┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩ │ 2013-01-01 00:00:00 │ 23.7103 │ 14.4996 │ 34.6201 │ ├─────────────────────┼──────────┼─────────────┼─────────────┤ │ 2013-01-01 01:00:00 │ 13.7366 │ 4.4598 │ 22.5832 │ ├─────────────────────┼──────────┼─────────────┼─────────────┤ │ 2013-01-01 02:00:00 │ 8.8895 │ 2.7815 │ 18.0047 │ ├─────────────────────┼──────────┼─────────────┼─────────────┤ │ 2013-01-01 03:00:00 │ 6.2631 │ 1.7288 │ 12.5642 │ ├─────────────────────┼──────────┼─────────────┼─────────────┤ │ 2013-01-01 04:00:00 │ 6.2631 │ 1.6157 │ 12.6441 │ ├─────────────────────┼──────────┼─────────────┼─────────────┤ │ ... │ ... │ ... │ ... │ ├─────────────────────┼──────────┼─────────────┼─────────────┤ │ 2013-01-02 07:00:00 │ 42.3548 │ 23.3925 │ 123.7833 │ ├─────────────────────┼──────────┼─────────────┼─────────────┤ │ 2013-01-02 08:00:00 │ 82.0528 │ 46.1838 │ 221.0061 │ ├─────────────────────┼──────────┼─────────────┼─────────────┤ │ 2013-01-02 09:00:00 │ 94.2292 │ 46.0529 │ 208.8712 │ ├─────────────────────┼──────────┼─────────────┼─────────────┤ │ 2013-01-02 10:00:00 │ 105.2167 │ 43.7337 │ 185.5930 │ ├─────────────────────┼──────────┼─────────────┼─────────────┤ │ 2013-01-02 11:00:00 │ 133.5423 │ 44.3065 │ 197.8839 │ └─────────────────────┴──────────┴─────────────┴─────────────┘
Code-only mode¶
Use forecast_code() when you want to preview or export the reproducible script without executing it. This is useful for code review, auditing the generated pipeline, or running the script in a separate environment.
In [20]:
Copied!
# Generate the reproducible script without executing it
# ==============================================================================
code_result = assistant.forecast_code(
data = data,
target = 'users',
date_column = 'date_time',
steps = 36,
test_size = 36,
profile = profile,
plan = plan_refined
)
code_result.show_code()
# Generate the reproducible script without executing it
# ==============================================================================
code_result = assistant.forecast_code(
data = data,
target = 'users',
date_column = 'date_time',
steps = 36,
test_size = 36,
profile = profile,
plan = plan_refined
)
code_result.show_code()
The ForecastResult object¶
Both forecast() modes return a ForecastResult, a lightweight container that bundles everything the assistant used and produced.
| Attribute | Type | Description |
|---|---|---|
predictions |
DataFrame | Forecasted values. When intervals are requested, the bound columns are included alongside the point predictions. |
metrics |
DataFrame or None | Evaluation metrics (MAE, MSE, MASE), one row per series. None in prediction mode. |
code |
str | The exact standalone skforecast script that produced the forecast, ready to run on its own. |
profile |
ForecastingProfile |
The data profile behind the forecast. |
plan |
ForecastPlan |
The detailed configuration that was executed. |
Branch B: Backtesting¶
The backtesting branch uses the same profile and plan as the forecast branch but evaluates the model's historical performance through time series cross-validation. The key decision is how to configure the TimeSeriesFold object, which controls exactly how the historical data is partitioned into successive training and test windows.
skforecast-ai provides three distinct ways to define this validation strategy:
Explicit instantiation (recommended): manually construct a
TimeSeriesFoldand pass it directly tobacktest(). Use this when you already know your exact operational constraints.Deterministic
create_cv(): allow the assistant to derive a sensibleTimeSeriesFoldfrom the profile and plan using rule-based defaults. You can override individual parameters explicitly.LLM
create_cv()(with a prompt): describe your deployment use case in natural language. The LLM translates your description into a fully-configuredTimeSeriesFold, accompanied by an explanation you can audit.
Define the backtesting strategy¶
Manual TimeSeriesFold¶
In [21]:
Copied!
# Create your own TimeSeriesFold object
# ==============================================================================
end_train = '2012-08-31 23:59:00'
cv = TimeSeriesFold(
steps = 36,
initial_train_size = end_train,
refit = False,
verbose = False
)
cv
# Create your own TimeSeriesFold object
# ==============================================================================
end_train = '2012-08-31 23:59:00'
cv = TimeSeriesFold(
steps = 36,
initial_train_size = end_train,
refit = False,
verbose = False
)
cv
Out[21]:
TimeSeriesFold
General Information
- Initial train size: 2012-08-31 23:59:00
- Initial train size as int: None
- Steps: 36
- Fold stride: 36
- Overlapping folds: False
- Window size: None
- Differentiation: None
- Refit: False
- Fixed train size: True
- Gap: 0
- Skip folds: None
- Allow incomplete fold: True
- Return all indexes: False
Deterministic create_cv()¶
In [22]:
Copied!
# Let the assistant derive a TimeSeriesFold with rule-based defaults
# ==============================================================================
cv_det, cv_det_explanation = assistant.create_cv(
profile = profile,
plan = plan_refined,
initial_train_size = end_train,
refit = False,
)
print(cv_det_explanation)
cv_det
# Let the assistant derive a TimeSeriesFold with rule-based defaults
# ==============================================================================
cv_det, cv_det_explanation = assistant.create_cv(
profile = profile,
plan = plan_refined,
initial_train_size = end_train,
refit = False,
)
print(cv_det_explanation)
cv_det
Initial training up to 2012-08-31 23:59:00, expanding window, no refit, 36-step horizon, 82 folds.
Out[22]:
TimeSeriesFold
General Information
- Initial train size: 2012-08-31 23:59:00
- Initial train size as int: 14616
- Steps: 36
- Fold stride: 36
- Overlapping folds: False
- Window size: None
- Differentiation: None
- Refit: False
- Fixed train size: False
- Gap: 0
- Skip folds: None
- Allow incomplete fold: True
- Return all indexes: False
LLM create_cv() with a natural-language¶
Rather than manually configuring TimeSeriesFold parameters, you can describe your backtesting strategy in natural language and let the assistant translate it into a rigorous cross-validation schema.
In [23]:
Copied!
# Let the assistant create the TimeSeriesFold from a natural-language prompt
# ==============================================================================
prompt = (
"I forecast bike demand 36 hours ahead. "
"The model should be trained once on all data up to the end of August 2012, 23:59. "
"Do not refit the model as the window rolls forward."
)
cv_llm, cv_llm_explanation = assistant.create_cv(
profile = profile,
plan = plan_refined,
prompt = prompt
)
# Let the assistant create the TimeSeriesFold from a natural-language prompt
# ==============================================================================
prompt = (
"I forecast bike demand 36 hours ahead. "
"The model should be trained once on all data up to the end of August 2012, 23:59. "
"Do not refit the model as the window rolls forward."
)
cv_llm, cv_llm_explanation = assistant.create_cv(
profile = profile,
plan = plan_refined,
prompt = prompt
)
In [24]:
Copied!
# TimeSeriesFold derived from the prompt
# ==============================================================================
cv_llm
# TimeSeriesFold derived from the prompt
# ==============================================================================
cv_llm
Out[24]:
TimeSeriesFold
General Information
- Initial train size: 2012-08-31 23:59
- Initial train size as int: 14616
- Steps: 36
- Fold stride: 36
- Overlapping folds: False
- Window size: None
- Differentiation: None
- Refit: False
- Fixed train size: False
- Gap: 0
- Skip folds: None
- Allow incomplete fold: True
- Return all indexes: False
In [25]:
Copied!
# LLM reasoning behind the TimeSeriesFold configuration
# ==============================================================================
import textwrap
print(textwrap.fill(cv_llm_explanation, width=88))
# LLM reasoning behind the TimeSeriesFold configuration
# ==============================================================================
import textwrap
print(textwrap.fill(cv_llm_explanation, width=88))
The user wants to forecast 36 hours ahead (steps=36). The initial training set is explicitly defined as all data up to end of August 2012 (2012-08-31 23:59), provided as a date string cutoff. Since the user wants the model trained once and never refitted as the window rolls forward, refit=False is set. All other parameters are left at their defaults: expanding window (fixed_train_size=False, though irrelevant since refit=False means no retraining), no gap, no fold stride override, and incomplete folds allowed. Initial training up to 2012-08-31 23:59, expanding window, no refit, 36-step horizon, 82 folds.
Since the prompt correctly describes the intended use case, the cv_llm object returned by create_cv() matches the one we built manually. However, it was derived entirely from a natural-language description. The assistant also returns a cv_llm_explanation string that details the choices it made, allowing you to verify that the resulting TimeSeriesFold matches your intended strategy before executing the backtest.
Run the backtest¶
In [ ]:
Copied!
# Run backtesting, reusing the pre-computed profile and plan
# ==============================================================================
results_backtest = assistant.backtest(
data = data,
target = 'users',
date_column = 'date_time',
cv = cv, # TimeSeriesFold object
interval = [0.1, 0.9], # 80% prediction interval
profile = profile, # Reuse the pre-computed profile
plan = plan_refined # Reuse the refined plan
)
results_backtest.show_explanation()
display(results_backtest.metrics)
display(results_backtest.predictions.head())
# Run backtesting, reusing the pre-computed profile and plan
# ==============================================================================
results_backtest = assistant.backtest(
data = data,
target = 'users',
date_column = 'date_time',
cv = cv, # TimeSeriesFold object
interval = [0.1, 0.9], # 80% prediction interval
profile = profile, # Reuse the pre-computed profile
plan = plan_refined # Reuse the refined plan
)
results_backtest.show_explanation()
display(results_backtest.metrics)
display(results_backtest.predictions.head())
╭─────────────────────────────── IgnoredArgumentWarning ───────────────────────────────╮ │ A pre-built `plan` was provided, so the following argument(s) are ignored: │ │ ['interval']. To change these, refine the plan with `refine_plan()` before calling. │ │ │ │ Category : skforecast.exceptions.IgnoredArgumentWarning │ │ Location : │ │ /home/ubuntu/miniconda3/envs/skforecast_ai/lib/python3.13/site-packages/skforecast_a │ │ i/_utils.py:392 │ │ Suppress : warnings.simplefilter('ignore', category=IgnoredArgumentWarning) │ ╰──────────────────────────────────────────────────────────────────────────────────────╯
0%| | 0/82 [00:00<?, ?it/s]
╭───────────────────────────────────── Explanation ──────────────────────────────────────╮ │ │ │ Initial training up to 2012-08-31 23:59:00, fixed window, no refit, 36-step horizon, │ │ 82 folds. Results — mean_absolute_error: 50.3025, mean_squared_error: 6910.7727, │ │ mean_absolute_scaled_error: 0.8204, mean_absolute_percentage_error: 0.5551. │ │ │ ╰────────────────────────────────────────────────────────────────────────────────────────╯
| mean_absolute_error | mean_squared_error | mean_absolute_scaled_error | mean_absolute_percentage_error | |
|---|---|---|---|---|
| 0 | 50.302476 | 6910.772709 | 0.820406 | 0.555051 |
| fold | pred | lower_bound | upper_bound | |
|---|---|---|---|---|
| 2012-09-01 00:00:00 | 0 | 129.141506 | 100.578964 | 153.657475 |
| 2012-09-01 01:00:00 | 0 | 106.534982 | 73.902908 | 136.019721 |
| 2012-09-01 02:00:00 | 0 | 67.848374 | 38.599536 | 105.805277 |
| 2012-09-01 03:00:00 | 0 | 39.658499 | 19.872419 | 69.489409 |
| 2012-09-01 04:00:00 | 0 | 19.804622 | 5.966910 | 36.093576 |
In [27]:
Copied!
# Plot prediction intervals vs real value
# ==============================================================================
predictions = results_backtest.predictions
data_test = data.set_index("date_time").loc[predictions.index, :]
fig = go.Figure([
go.Scatter(name='Prediction', x=predictions.index, y=predictions['pred'], mode='lines'),
go.Scatter(
name='Real value', x=data_test.index, y=data_test['users'], mode='lines',
),
go.Scatter(
name='Upper Bound', x=predictions.index, y=predictions['upper_bound'], mode='lines',
marker=dict(color="#444"), line=dict(width=0), showlegend=False
),
go.Scatter(
name='Lower Bound', x=predictions.index, y=predictions['lower_bound'], marker=dict(color="#444"),
line=dict(width=0), mode='lines', fillcolor='rgba(68, 68, 68, 0.3)', fill='tonexty', showlegend=False
)
])
fig.update_layout(
title="Real value vs predicted in test data",
xaxis_title="Date time",
yaxis_title="users",
width=800,
height=400,
margin=dict(l=20, r=20, t=35, b=20),
hovermode="x",
legend=dict(orientation="h", yanchor="top", y=1.1, xanchor="left", x=0.001),
# Initial zoom on x axis betwee 1 oct to 10 oct
xaxis=dict(range=['2012-10-01', '2012-10-10'])
)
fig.show()
# Plot prediction intervals vs real value
# ==============================================================================
predictions = results_backtest.predictions
data_test = data.set_index("date_time").loc[predictions.index, :]
fig = go.Figure([
go.Scatter(name='Prediction', x=predictions.index, y=predictions['pred'], mode='lines'),
go.Scatter(
name='Real value', x=data_test.index, y=data_test['users'], mode='lines',
),
go.Scatter(
name='Upper Bound', x=predictions.index, y=predictions['upper_bound'], mode='lines',
marker=dict(color="#444"), line=dict(width=0), showlegend=False
),
go.Scatter(
name='Lower Bound', x=predictions.index, y=predictions['lower_bound'], marker=dict(color="#444"),
line=dict(width=0), mode='lines', fillcolor='rgba(68, 68, 68, 0.3)', fill='tonexty', showlegend=False
)
])
fig.update_layout(
title="Real value vs predicted in test data",
xaxis_title="Date time",
yaxis_title="users",
width=800,
height=400,
margin=dict(l=20, r=20, t=35, b=20),
hovermode="x",
legend=dict(orientation="h", yanchor="top", y=1.1, xanchor="left", x=0.001),
# Initial zoom on x axis betwee 1 oct to 10 oct
xaxis=dict(range=['2012-10-01', '2012-10-10'])
)
fig.show()
In [28]:
Copied!
# Ask the assistant to interpret the backtesting results
# ==============================================================================
answer = assistant.ask(
prompt = (
"Explain the results of this backtesting, including the strategy, metrics, "
"and predictions. Is the model good enough to deploy?"
),
backtest_result = results_backtest
)
answer.show_explanation()
# Ask the assistant to interpret the backtesting results
# ==============================================================================
answer = assistant.ask(
prompt = (
"Explain the results of this backtesting, including the strategy, metrics, "
"and predictions. Is the model good enough to deploy?"
),
backtest_result = results_backtest
)
answer.show_explanation()
╭───────────────────────────────────── Explanation ──────────────────────────────────────╮ │ │ │ Backtesting Results Explained │ │ │ │ Strategy Overview │ │ │ │ The backtest evaluated a ForecasterRecursive with LGBMRegressor on 17,544 hourly │ │ observations of bike-sharing demand (users). The core idea: train once on all data │ │ up to 2012-08-31, then roll forward in non-overlapping 36-hour windows through the │ │ end of 2012 (82 folds total), measuring how well the model predicts the next 36 │ │ hours at each step. │ │ │ │ Key configuration choices │ │ │ │ │ │ Parameter Value What it means │ │ ────────────────────────────────────────────────────────────────────────────────── │ │ refit=False Train once The same model serves all 82 folds — │ │ simulates deploying without retraining │ │ fixed_train_size=True Rolling window Consistent with the refit=False setup (has │ │ no effect without refit, but is the │ │ sensible default) │ │ gap=0 No delay Forecasts start immediately after the last │ │ observed hour — assumes real-time data │ │ availability │ │ fold_stride=36 Non-overlapping Each 36-hour test block is distinct, so │ │ folds never overlap │ │ │ │ │ │ The feature set is rich: 15 carefully selected lags (capturing the last 6 hours, ±1 │ │ hour around yesterday, two days ago, and last week), four rolling window statistics │ │ (24h mean/std + 168h mean/max), calendar features (hour, day of week, weekend flag, │ │ month), and three exogenous variables (holiday, weather, temperature). │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ Metric Interpretation │ │ │ │ │ │ Metric Value Interpretation │ │ ────────────────────────────────────────────────────────────────────────────────── │ │ MAE 50.3 users On average, predictions are off by ~50 users per hour │ │ MASE 0.820 ✅ Better than a naïve seasonal forecast (MASE < 1) │ │ MAPE 55.5% High — but this is largely distorted by low-demand │ │ hours (night-time, where even a small absolute error │ │ creates a large percentage) │ │ MSE 6,911 (RMSE ≈ 83) The gap between MAE (50) and RMSE (83) suggests some │ │ larger errors exist, likely at peak-demand hours │ │ │ │ │ │ The most important number: MASE = 0.82 │ │ │ │ This is the headline result. A MASE below 1.0 means the model outperforms the naïve │ │ seasonal baseline (predicting "same hour last week"). A value of 0.82 means the │ │ model reduces error by roughly 18% over the naïve approach — a meaningful but not │ │ dramatic improvement, which is honest for a complex real-world demand series. │ │ │ │ Why MAPE looks alarming but isn't │ │ │ │ A MAPE of 55% sounds poor, but the mean prediction is ~234 users while the minimum │ │ is ~3 users. Overnight hours with 3–20 users will generate enormous percentage │ │ errors even when absolute errors are small. MAPE is unreliable here — trust MAE and │ │ MASE instead. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ Prediction Quality │ │ │ │ Point predictions │ │ │ │ • Predictions range from ~3 to ~867 users (mean 234), which tracks well against the │ │ realistic demand range │ │ • The model correctly identifies low-demand overnight windows and high-demand │ │ daytime peaks across all 82 folds │ │ • No systematic collapse or explosion of predictions is visible across the │ │ September–December evaluation window │ │ │ │ Prediction intervals (80% coverage target) │ │ │ │ • Lower bounds average ~185, upper bounds average ~280, giving a typical interval │ │ width of ~95 users │ │ • One lower bound reaches −0.95 users — a minor artifact of bootstrapping on a │ │ non-negative series; worth clipping to 0 in production │ │ • The intervals appropriately widen during high-demand periods (e.g., upper bound │ │ up to 909) and narrow during quiet overnight hours │ │ │ │ ▌ Note: The actual empirical coverage of these 80% intervals is not reported here. │ │ ▌ You should compute calculate_coverage(y_true, lower_bound, upper_bound) to │ │ ▌ confirm the intervals are calibrated — ideally achieving ~80% coverage across │ │ ▌ all folds. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ Is the Model Good Enough to Deploy? │ │ │ │ ✅ Green flags │ │ │ │ • MASE < 1: The model adds genuine value over a naïve baseline │ │ • Rich feature set: The weekly/daily lag structure and rolling features are │ │ well-matched to the hourly bike-rental domain │ │ • Stable across 82 folds: The model was trained once and evaluated over four months │ │ — no obvious degradation visible from the fold range (fold 0 to 81) │ │ • Reasonable interval widths: The uncertainty bands are meaningful rather than │ │ trivially wide │ │ │ │ ⚠️ Caveats to address before deploying │ │ │ │ 1 Verify interval calibration: Run calculate_coverage — if actual coverage is well │ │ below 80%, the intervals are misleading for operational decisions │ │ 2 Negative lower bounds: Clip to zero for a non-negative demand variable before │ │ surfacing to end users │ │ 3 Peak-hour errors may be larger than the average suggests: The RMSE (83) being │ │ notably higher than MAE (50) indicates some large errors at demand peaks — │ │ inspect per-fold errors around rush hours specifically │ │ 4 refit=False assumption: The backtest assumes no retraining. In production, │ │ consider periodic retraining (e.g., weekly) to capture seasonal drift as the │ │ dataset extends beyond December 2012 │ │ 5 Exogenous variables in production: holiday, weather, and temp must be available │ │ for the full 36-hour forecast horizon at prediction time. Ensure your pipeline │ │ sources these reliably │ │ │ │ Bottom line │ │ │ │ Conditionally yes — the model is deployment-ready for use cases tolerating ~50-user │ │ average errors. It meaningfully outperforms the naïve baseline (MASE 0.82) and │ │ produces stable predictions across four months. Before going live: validate interval │ │ calibration, clip negative lower bounds, and establish a retraining cadence. If │ │ tighter accuracy is needed, the next step is hyperparameter tuning │ │ (bayesian_search_forecaster) and/or feature selection to squeeze further performance │ │ from the existing structure. │ │ │ ╰────────────────────────────────────────────────────────────────────────────────────────╯
Code-only mode¶
Use backtest_code() to generate the reproducible backtesting script without executing it.
In [29]:
Copied!
# Generate the reproducible backtest script without executing it
# ==============================================================================
code_bt = assistant.backtest_code(
data = data,
target = 'users',
date_column = 'date_time',
cv = cv,
profile = profile,
plan = plan_refined
)
code_bt.show_code()
# Generate the reproducible backtest script without executing it
# ==============================================================================
code_bt = assistant.backtest_code(
data = data,
target = 'users',
date_column = 'date_time',
cv = cv,
profile = profile,
plan = plan_refined
)
code_bt.show_code()
The BacktestResult object¶
The backtest() method returns a BacktestResult, a lightweight container that bundles all the backtesting artifacts.
| Attribute | Type | Description |
|---|---|---|
predictions |
DataFrame | Full out-of-sample backtest predictions across all folds. |
metrics |
DataFrame | Backtesting metrics (MAE, MSE, MASE), one row per series. |
cv_config |
dict | Resolved TimeSeriesFold parameters for full traceability of the validation strategy. |
code |
str | The exact standalone skforecast script that reproduces the backtesting workflow. |
explanation |
str | Human-readable summary of the backtesting configuration and results. |
profile |
ForecastingProfile |
The data profile behind the backtest. |
plan |
ForecastPlan |
The detailed configuration that was executed. |
In [30]:
Copied!
# Full results object
# ==============================================================================
results_backtest
# Full results object
# ==============================================================================
results_backtest
Out[30]:
╭───────────────────────────────────── Explanation ──────────────────────────────────────╮ │ │ │ Initial training up to 2012-08-31 23:59:00, fixed window, no refit, 36-step horizon, │ │ 82 folds. Results — mean_absolute_error: 50.3025, mean_squared_error: 6910.7727, │ │ mean_absolute_scaled_error: 0.8204, mean_absolute_percentage_error: 0.5551. │ │ │ ╰────────────────────────────────────────────────────────────────────────────────────────╯ Cross-Validation Configuration ┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┓ ┃ Parameter ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━┩ │ steps │ 36 │ ├────────────────────┼─────────────────────┤ │ initial_train_size │ 2012-08-31 23:59:00 │ ├────────────────────┼─────────────────────┤ │ refit │ False │ ├────────────────────┼─────────────────────┤ │ fixed_train_size │ True │ ├────────────────────┼─────────────────────┤ │ gap │ 0 │ ├────────────────────┼─────────────────────┤ │ fold_stride │ 36 │ ├────────────────────┼─────────────────────┤ │ differentiation │ None │ └────────────────────┴─────────────────────┘ Backtest Metrics ┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┓ ┃ mean_absolute_error ┃ mean_squared_error ┃ mean_absolute_scale… ┃ mean_absolute_perce… ┃ ┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━┩ │ 50.3025 │ 6910.7727 │ 0.8204 │ 0.5551 │ └─────────────────────┴────────────────────┴──────────────────────┴──────────────────────┘ Backtest Predictions (2928 rows) ┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓ ┃ Index ┃ fold ┃ pred ┃ lower_bound ┃ upper_bound ┃ ┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩ │ 2012-09-01 00:00:00 │ 0.0000 │ 129.1415 │ 100.5790 │ 153.6575 │ ├─────────────────────┼─────────┼──────────┼─────────────┼─────────────┤ │ 2012-09-01 01:00:00 │ 0.0000 │ 106.5350 │ 73.9029 │ 136.0197 │ ├─────────────────────┼─────────┼──────────┼─────────────┼─────────────┤ │ 2012-09-01 02:00:00 │ 0.0000 │ 67.8484 │ 38.5995 │ 105.8053 │ ├─────────────────────┼─────────┼──────────┼─────────────┼─────────────┤ │ 2012-09-01 03:00:00 │ 0.0000 │ 39.6585 │ 19.8724 │ 69.4894 │ ├─────────────────────┼─────────┼──────────┼─────────────┼─────────────┤ │ 2012-09-01 04:00:00 │ 0.0000 │ 19.8046 │ 5.9669 │ 36.0936 │ ├─────────────────────┼─────────┼──────────┼─────────────┼─────────────┤ │ ... │ ... │ ... │ ... │ ... │ ├─────────────────────┼─────────┼──────────┼─────────────┼─────────────┤ │ 2012-12-31 19:00:00 │ 81.0000 │ 67.3204 │ 31.5294 │ 108.5078 │ ├─────────────────────┼─────────┼──────────┼─────────────┼─────────────┤ │ 2012-12-31 20:00:00 │ 81.0000 │ 40.3452 │ 20.0169 │ 82.1894 │ ├─────────────────────┼─────────┼──────────┼─────────────┼─────────────┤ │ 2012-12-31 21:00:00 │ 81.0000 │ 29.3831 │ 13.5137 │ 61.1140 │ ├─────────────────────┼─────────┼──────────┼─────────────┼─────────────┤ │ 2012-12-31 22:00:00 │ 81.0000 │ 19.5810 │ 11.1901 │ 42.4820 │ ├─────────────────────┼─────────┼──────────┼─────────────┼─────────────┤ │ 2012-12-31 23:00:00 │ 81.0000 │ 16.1096 │ 8.8776 │ 31.6928 │ └─────────────────────┴─────────┴──────────┴─────────────┴─────────────┘ Dataset Profile ┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Property ┃ Value ┃ ┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━┩ │ Format │ single │ ├────────────────┼────────────────────────┤ │ Series │ 1 │ ├────────────────┼────────────────────────┤ │ Observations │ 17544 │ ├────────────────┼────────────────────────┤ │ Frequency │ h │ ├────────────────┼────────────────────────┤ │ Target │ users │ ├────────────────┼────────────────────────┤ │ Exog columns │ holiday, weather, temp │ ├────────────────┼────────────────────────┤ │ Missing target │ none │ └────────────────┴────────────────────────┘ Recommendation ┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Property ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ Task type │ single_series │ ├───────────────────────┼────────────────────────────────────────────────────────────────┤ │ Forecaster │ ForecasterRecursive │ ├───────────────────────┼────────────────────────────────────────────────────────────────┤ │ Forecaster candidates │ ForecasterRecursive, ForecasterDirect, ForecasterFoundation, │ │ │ ForecasterStats │ ├───────────────────────┼────────────────────────────────────────────────────────────────┤ │ Estimator │ LGBMRegressor │ ├───────────────────────┼────────────────────────────────────────────────────────────────┤ │ Estimator candidates │ LGBMRegressor, XGBRegressor, Ridge │ └───────────────────────┴────────────────────────────────────────────────────────────────┘ ╭───────────────────────────────────── Explanation ──────────────────────────────────────╮ │ │ │ A single-series ML forecaster (ForecasterRecursive) is recommended. Data: 17544 │ │ observations, 'h' frequency. Alternative forecasters: ['ForecasterDirect', │ │ 'ForecasterFoundation', 'ForecasterStats']. Estimator: LGBMRegressor. A gradient │ │ boosting model is preferred for a dataset of this size (17544 observations). │ │ Alternative estimators: ['XGBRegressor', 'Ridge']. 3 exogenous variables available │ │ as predictors. │ │ │ ╰────────────────────────────────────────────────────────────────────────────────────────╯ Forecast Plan ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Property ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ Forecaster │ ForecasterRecursive │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Estimator │ LGBMRegressor │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Steps │ 36 │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Frequency │ h │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Lags │ [1, 2, 3, 4, 5, 6, 23, 24, 25, 47, 48, 49, 167, 168, 169] │ │ │ (LLM-suggested) │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Window features │ [{'stats': ['mean', 'std'], 'window_size': 24}, {'stats': ['mean', │ │ │ 'max'], 'window_size': 168}] (LLM-suggested) │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Calendar features │ ['hour', 'day_of_week', 'weekend', 'month'] (raw ordinal encoding) │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Use exog │ True │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Interval │ [0.1, 0.9] │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Interval method │ bootstrapping │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Primary metric │ mean_absolute_error │ ├───────────────────┼────────────────────────────────────────────────────────────────────┤ │ Preprocessing │ none │ └───────────────────┴────────────────────────────────────────────────────────────────────┘ ╭─────────────────────────────────── Plan Explanation ───────────────────────────────────╮ │ │ │ Plan: ForecasterRecursive + LGBMRegressor. Lags: [1, 2, 3, 4, 5, 6, 23, 24, 25, 47, │ │ 48, 49, 167, 168, 169]. Window features: ['mean(window=24)', 'std(window=24)', │ │ 'mean(window=168)', 'max(window=168)']. Calendar features: ['hour', 'day_of_week', │ │ 'weekend', 'month'] (raw ordinal encoding). Prediction intervals via bootstrapping. │ │ NaN rows kept (NaN-tolerant estimator). Exogenous variables included. MAE is │ │ interpretable, robust to outliers, and works at any scale. │ │ │ │ LLM Refinement Reasoning: Lags chosen: │ │ │ │ 1 Recent trend (lags 1–6): The user notes that "the last few hours give a good │ │ sense of the current trend." Lags 1–6 capture the immediate short-term momentum │ │ (e.g., a demand ramp-up or cool-down around rush hour), giving the model a window │ │ into what's happening right now. │ │ 2 Same hour yesterday (lags 23, 24, 25): The daily rhythm is the dominant signal in │ │ bike rentals. Lags 23/24/25 straddle the exact same hour from the previous day, │ │ letting the model learn the typical peak/trough structure (morning rush, lunch, │ │ evening rush) while being robust to small timing shifts. │ │ 3 Same hour two days ago (lags 47, 48, 49): A second daily anchor reinforces the │ │ daily pattern and helps distinguish weekday vs. weekend dynamics (e.g., Monday │ │ vs. Tuesday behaviour) when combined with the weekly lags. │ │ 4 Same hour last week (lags 167, 168, 169): The user explicitly states demand "is │ │ usually similar to what happened at the same time last week." Lags 167/168/169 │ │ directly encode the weekly seasonal signal — capturing the weekday/weekend │ │ contrast (e.g., a Tuesday at 8 am vs. a Saturday at 8 am). │ │ │ │ Window features chosen: │ │ │ │ 1 mean + std over 24 hours: A 24-hour rolling mean captures the recent daily-level │ │ baseline (is today busier or quieter than usual?), while the 24-hour std captures │ │ intra-day volatility — critical for distinguishing calm midday periods from │ │ volatile rush-hour windows. │ │ 2 mean + max over 168 hours (1 week): A weekly rolling mean represents the recent │ │ weekly demand level, complementing the lag-168 point estimate with a smoothed │ │ context. The weekly max captures peak demand conditions over the last 7 days, │ │ which is informative for capacity-sensitive rush-hour forecasting and helps model │ │ the weekday/weekend regime difference. │ │ │ │ Note: the LLM-suggested lags and window_features are hypotheses, not validated │ │ improvements. Confirm any expected accuracy gain before relying on them. │ │ │ ╰────────────────────────────────────────────────────────────────────────────────────────╯
Reproducible code¶
Every forecast or backtest workflow exposes the standalone skforecast script used to produce its results. This script is strictly deterministic, ensuring identical outputs for a given set of inputs and configurations. You can access the code at any time via the show_code() method.
Free-form Q&A¶
The ask() method is not limited to interpreting workflow objects. Without any profile, plan, or result attached, it works as a general forecasting knowledge assistant, useful for clarifying methodology, choosing between approaches, or understanding metric trade-offs.
In [31]:
Copied!
# Ask a general forecasting question (no data or result required)
# ==============================================================================
answer = assistant.ask(
prompt = (
"For hourly demand with strong daily and weekly seasonality, when should "
"I prefer a direct forecasting strategy over a recursive one?"
)
)
answer.show_explanation()
# Ask a general forecasting question (no data or result required)
# ==============================================================================
answer = assistant.ask(
prompt = (
"For hourly demand with strong daily and weekly seasonality, when should "
"I prefer a direct forecasting strategy over a recursive one?"
)
)
answer.show_explanation()
╭───────────────────────────────────── Explanation ──────────────────────────────────────╮ │ │ │ Direct vs Recursive Forecasting for Hourly Demand Data │ │ │ │ When the Direct Strategy Has an Advantage │ │ │ │ For hourly demand with strong daily (24-step) and weekly (168-step) seasonality, the │ │ choice depends on several interacting factors: │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ Key Scenarios Favouring ForecasterDirect │ │ │ │ 1. Long Forecast Horizons │ │ │ │ When your horizon spans 24 hours or more, recursive forecasting accumulates │ │ prediction errors at each step — each predicted value feeds back as an input for the │ │ next. Direct trains one independent model per step, so there is no error │ │ propagation. This advantage grows with horizon length. │ │ │ │ 2. Step-Specific Seasonal Patterns │ │ │ │ Hourly demand often has fundamentally different dynamics at different points in the │ │ day: │ │ │ │ • The relationship between lag features and demand at 3 AM may be very different │ │ from that at 6 PM (peak hours) │ │ • A single recursive model must learn all these relationships simultaneously │ │ • Direct lets each step's model specialise for its own position in the cycle │ │ │ │ 3. When Your Horizon Matches Exactly 24 or 168 Steps │ │ │ │ If you need to forecast exactly 24 steps ahead (one full day), ForecasterDirect │ │ trains 24 independent models — each optimised for its specific hour offset. This is │ │ particularly powerful when hour-of-day effects are nonlinear or interact with │ │ exogenous variables (temperature, calendar events) differently per step. │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ When to Stick with ForecasterRecursive │ │ │ │ │ │ Situation Reason to prefer Recursive │ │ ────────────────────────────────────────────────────────────────────────────────── │ │ Variable or short horizons (< 12 steps) Flexible steps at predict time; one model │ │ is faster │ │ Limited training data 24–168 models risk overfitting with few │ │ observations │ │ Exogenous variables are rich One model has more data to learn exog │ │ relationships │ │ Speed is a priority Training and inference are ~N× faster │ │ You need bootstrapping intervals Both support it, but recursive has lower │ │ computational cost │ │ │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ Practical Guidance for Hourly Demand │ │ │ │ Feature Engineering Matters More Than Strategy │ │ │ │ Both strategies benefit enormously from good features. For hourly demand, always │ │ include: │ │ │ │ • Seasonal lags: lags = [1, 2, 24, 48, 168, 336] — capturing same-hour-yesterday │ │ and same-hour-last-week │ │ • Rolling features (RollingFeatures): rolling mean/std over windows of 24, 168 │ │ • Calendar features: hour-of-day, day-of-week, is_holiday — these are especially │ │ impactful for Direct since each model can weight them differently │ │ • Differentiation: consider differentiation=1 or differentiation=24 to remove │ │ trend/seasonality before modelling │ │ │ │ Recommended Starting Point │ │ │ │ │ │ Step Action │ │ ────────────────────────────────────────────────────────────────────────────────── │ │ Baseline ForecasterRecursive with LightGBM + seasonal lags — fast to iterate │ │ Compare ForecasterDirect with the same features at your target horizon │ │ Evaluate Use backtesting_forecaster + TimeSeriesFold with gap=0 and your exact │ │ steps │ │ Decide If Direct improves metrics at critical hours (e.g., morning ramp, │ │ evening peak), it is worth the training cost │ │ │ │ │ │ ------------------------------------------------------------------------------------ │ │ │ │ Bottom Line │ │ │ │ ▌ Prefer ForecasterDirect when your horizon is ≥ 24 steps, the demand pattern │ │ ▌ changes meaningfully across the day, and you have sufficient training data │ │ ▌ (typically 2+ years of hourly data for 168-step direct models). Start with │ │ ▌ ForecasterRecursive and only switch if backtesting reveals that error │ │ ▌ accumulation or step-specific patterns are hurting accuracy at the horizon that │ │ ▌ matters most to your use case. │ │ │ ╰────────────────────────────────────────────────────────────────────────────────────────╯
Summary¶
This tutorial covered the step-by-step path of skforecast-ai. Here is a recap of what each stage does and when to use it:
| Step | Method | When to use |
|---|---|---|
| 1. Profile | profile() |
Always: produces the ForecastingProfile required by all downstream methods. |
| 2. Plan | plan() |
Always: converts the profile into an executable configuration. |
| 3. Refine plan | refine_plan() |
Optional: use when you want to override specific decisions (deterministic) or inject domain knowledge (LLM). Always evaluate the result. |
| 4a. Forecast | forecast() |
When you want future predictions or a held-out evaluation in a single execution. |
| 4a. Code only | forecast_code() |
When you want to preview or export the script without running it. |
| 4b. CV strategy | create_cv() |
When you want the assistant to derive or translate a TimeSeriesFold for you. |
| 4b. Backtest | backtest() |
When you want to evaluate the model over multiple historical folds. |
| 4b. Code only | backtest_code() |
When you want to preview or export the backtesting script without running it. |
| Any time | ask() |
When you want an LLM explanation of any intermediate object or result, or a general forecasting Q&A. |
The key advantage of this path is that the profile and plan are built once and reused across both the forecast and backtest branches. This avoids redundant profiling and ensures that both branches use the same modeling configuration.
For a faster alternative that runs the entire pipeline in a single call, see the fast-path tutorial. For a comprehensive overview of backtesting mechanics, see the skforecast backtesting user guide.