Agentic forecasting step-by-stepĀ¶
skforecast-ai is an AI forecasting assistant that pairs a deterministic engine, powered by skforecast, with an LLM reasoning layer. Simply provide a time series, and the assistant automatically profiles the data, selects a model using established best practices, and evaluates its performance. It returns both the final forecast and the runnable skforecast script that produced it.
This tutorial walks through the step-by-step path: the approach for users who want granular control to inspect or adjust intermediate decisions before committing to a full run. If you prefer a single-call workflow that returns results immediately, see the fast-path tutorial.
Both paths share the same deterministic engine and produce identical results and reproducible skforecast code. The step-by-step path gives you three additional capabilities:
Inspect intermediate objects: examine the
ForecastingProfileandForecastPlanbefore any model is trained, and confirm the assistant's recommendations make sense for your domain.Override any single decision: change the forecaster, the estimator and its hyperparameters, the lags, the window features, the horizon or the interval without re-running the entire pipeline from scratch. Pass the modified plan directly to
forecast()orbacktest().Reuse the profile and plan across branches: build the profile and plan once, then run
forecast()for future predictions,backtest()for historical evaluation, andcompare()to rank several candidate configurations: each reusing the same profile, with no redundant profiling.
The ask() method is available at every stage. It interprets what you pass to it (a profile, a plan, a generated script, a cross-validation strategy, a forecast result, a backtest result, or a comparison result), but it never executes the workflow or silently changes any recommendation.
The following example walks through the step-by-step path, allowing users to understand and control what happens under the hood. If you prefer the quickest way to go from raw data to a validated forecast with minimal setup, visit the fast-path tutorial.
Assistant initializationĀ¶
The first step is to instantiate a ForecastingAssistant, which will be responsible for executing the entire workflow (profiling, planning, backtesting, and forecasting), as well as explaining the outputs and suggesting improvements.
To activate the optional LLM support, users must pass a string in the format 'provider:model_name' (for example, 'openai:gpt-5.5', 'google:gemini-3.8-flash', 'anthropic:claude-sonnet-5', or 'ollama:qwen3:8b'). For hosted providers, the corresponding API key must be available as an environment variable or passed explicitly when creating the assistant. Your dataset is never sent: the LLM receives only metadata and summary statistics, never the raw time series values. In this tutorial, we keep send_data_to_llm=False (the default), so passing a result to ask(), whose own predictions and metrics are sent, emits a DataSentToLLMWarning as a reminder.
InĀ [1]:
Copied!
# Data processing
# ==============================================================================
import os
import pandas as pd
from skforecast.datasets import fetch_dataset
# Plots
# ==============================================================================
import matplotlib.pyplot as plt
import plotly.graph_objects as go
import plotly.io as pio
import plotly.offline as poff
pio.templates.default = "seaborn"
poff.init_notebook_mode(connected=True)
plt.style.use('seaborn-v0_8-darkgrid')
# skforecast and skforecast-ai
# ==============================================================================
import skforecast
import skforecast_ai
from skforecast_ai import ForecastingAssistant
from skforecast.model_selection import TimeSeriesFold
from skforecast.plot import set_dark_theme
# Python utilities
# ==============================================================================
import textwrap
color = '\033[1m\033[38;5;208m'
print(f"{color}Version skforecast_ai: {skforecast_ai.__version__}")
print(f"{color}Version skforecast: {skforecast.__version__}")
# Data processing
# ==============================================================================
import os
import pandas as pd
from skforecast.datasets import fetch_dataset
# Plots
# ==============================================================================
import matplotlib.pyplot as plt
import plotly.graph_objects as go
import plotly.io as pio
import plotly.offline as poff
pio.templates.default = "seaborn"
poff.init_notebook_mode(connected=True)
plt.style.use('seaborn-v0_8-darkgrid')
# skforecast and skforecast-ai
# ==============================================================================
import skforecast
import skforecast_ai
from skforecast_ai import ForecastingAssistant
from skforecast.model_selection import TimeSeriesFold
from skforecast.plot import set_dark_theme
# Python utilities
# ==============================================================================
import textwrap
color = '\033[1m\033[38;5;208m'
print(f"{color}Version skforecast_ai: {skforecast_ai.__version__}")
print(f"{color}Version skforecast: {skforecast.__version__}")
Version skforecast_ai: 0.4.2 Version skforecast: 0.26.0
āļø Note
If you do not have access to an LLM, you can still follow the full tutorial using only the deterministic methods. Profiling, planning, backtesting, and forecasting all run without an LLM. Only the ask() explanations and the LLM-guided variants of refine_plan() and create_cv() require a configured LLM; their deterministic counterparts work without one.
InĀ [2]:
Copied!
# LLM-enabled assistant
# ==============================================================================
LLM_MODEL = "google:gemini-3.8-flash"
api_key = os.getenv("GOOGLE_API_KEY")
assistant = ForecastingAssistant(
llm=LLM_MODEL, api_key=api_key, send_data_to_llm=False
)
# Using AWS Bedrock
# ==============================================================================
# assistant = ForecastingAssistant(
# llm='bedrock:eu.anthropic.claude-sonnet-4-6',
# base_url="eu-west-1"
# )
# Assistant without LLM (deterministic only)
# ==============================================================================
# assistant = ForecastingAssistant()
# LLM-enabled assistant
# ==============================================================================
LLM_MODEL = "google:gemini-3.8-flash"
api_key = os.getenv("GOOGLE_API_KEY")
assistant = ForecastingAssistant(
llm=LLM_MODEL, api_key=api_key, send_data_to_llm=False
)
# Using AWS Bedrock
# ==============================================================================
# assistant = ForecastingAssistant(
# llm='bedrock:eu.anthropic.claude-sonnet-4-6',
# base_url="eu-west-1"
# )
# Assistant without LLM (deterministic only)
# ==============================================================================
# assistant = ForecastingAssistant()
Before going on, check_llm() reports how the configuration resolves (provider, credentials, endpoint, installed extras) without calling the model, so a missing key shows up here instead of in the first ask(). See Configuring the LLM for the details.
InĀ [3]:
Copied!
# Check the LLM configuration
# ==============================================================================
assistant.check_llm()
# Check the LLM configuration
# ==============================================================================
assistant.check_llm()
Out[3]:
LLM Configuration Check āāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā Check ā Value ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā Status ā ok ā āāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Provider ā google ā āāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Model ā gemini-3.8-flash ā āāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Credentials ā explicit api_key ā ā ā Explicit api_key given; the environment is not consulted. ā āāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Base URL ā None ā āāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Dependencies ā installed ā āāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Test call ā skipped ā āāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
ā ļø Your data stays private
Enabling an LLM never sends your time-series data to the model provider.
The assistant passes only a summary of the dataset (frequency, date range, target statistics,
missing values, significant lags) and the forecaster configuration, never the raw observations.
Results passed to ask() are the exception: their own predictions and metrics are
sent, because a question about a result cannot be answered from summary statistics alone, and a
DataSentToLLMWarning reminds you of it. Pass send_data_to_llm=True to
acknowledge it and silence the warning. See
What is sent to the LLM for the full list.
DataĀ¶
The data used in this tutorial represent the hourly usage of the bike share system in the city of Washington, D.C. during the years 2011 and 2012. In addition to the number of users per hour, information about weather conditions and holidays is available.
InĀ [4]:
Copied!
# Downloading data
# ==============================================================================
data = fetch_dataset('bike_sharing', raw=True)
data = data[['date_time', 'users', 'holiday', 'weather', 'temp']]
data['date_time'] = pd.to_datetime(data['date_time'])
data.head()
# Downloading data
# ==============================================================================
data = fetch_dataset('bike_sharing', raw=True)
data = data[['date_time', 'users', 'holiday', 'weather', 'temp']]
data['date_time'] = pd.to_datetime(data['date_time'])
data.head()
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā bike_sharing āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā Description: ā ā Hourly usage of the bike share system in the city of Washington D.C. during the ā ā years 2011 and 2012. In addition to the number of users per hour, information ā ā about weather conditions and holidays is available. ā ā ā ā Source: ā ā Fanaee-T,Hadi. (2013). Bike Sharing Dataset. UCI Machine Learning Repository. ā ā https://doi.org/10.24432/C5W894. ā ā ā ā URL: ā ā https://raw.githubusercontent.com/skforecast/skforecast- ā ā datasets/main/data/bike_sharing_dataset_clean.csv ā ā ā ā Shape: 17544 rows x 12 columns ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
Out[4]:
| date_time | users | holiday | weather | temp | |
|---|---|---|---|---|---|
| 0 | 2011-01-01 00:00:00 | 16.0 | 0.0 | clear | 9.84 |
| 1 | 2011-01-01 01:00:00 | 40.0 | 0.0 | clear | 9.02 |
| 2 | 2011-01-01 02:00:00 | 32.0 | 0.0 | clear | 9.02 |
| 3 | 2011-01-01 03:00:00 | 13.0 | 0.0 | clear | 9.84 |
| 4 | 2011-01-01 04:00:00 | 1.0 | 0.0 | clear | 9.84 |
āļø Note
skforecast-ai is ready to preprocess the data, but it is recommended that users apply their own preprocessing steps before using the assistant. This ensures the data is in the desired format and any necessary transformations have been applied before proceeding with the forecasting workflow.
InĀ [5]:
Copied!
# Interactive plot of time series
# ==============================================================================
fig = go.Figure()
fig.add_trace(
go.Scatter(x=data['date_time'], y=data['users'], mode='lines', name='Users')
)
fig.update_layout(
title = 'Number of users',
xaxis_title="Time",
yaxis_title="Users",
width=800,
height=400,
margin=dict(l=20, r=20, t=35, b=20),
legend=dict(orientation="h", yanchor="top", y=1, xanchor="left", x=0.001)
)
fig.show()
# Interactive plot of time series
# ==============================================================================
fig = go.Figure()
fig.add_trace(
go.Scatter(x=data['date_time'], y=data['users'], mode='lines', name='Users')
)
fig.update_layout(
title = 'Number of users',
xaxis_title="Time",
yaxis_title="Users",
width=800,
height=400,
margin=dict(l=20, r=20, t=35, b=20),
legend=dict(orientation="h", yanchor="top", y=1, xanchor="left", x=0.001)
)
fig.show()
For a deeper walkthrough of the exploratory analysis behind this dataset, see the skforecast example: Forecasting time series with skforecast, XGBoost, LightGBM and CatBoost.
Profile the dataĀ¶
The profile() method is the first stage of the step-by-step workflow. It inspects the dataset and returns a ForecastingProfile object that contains:
Data metadata: detected frequency, index type, series lengths, missing values, and exogenous column roles.
Modeling recommendations: the selected forecaster family and estimator, along with alternative candidates and the reasoning behind each choice.
Lag structure: PACF-significant lags per series, used as a baseline for the planning stage.
Window feature suggestions: rolling statistics configurations appropriate for the seasonal periods implied by the frequency.
This is a purely deterministic step: no LLM is involved. The profile object is a prerequisite for plan(), and can be passed to ask() as context to have its decisions explained.
| Attribute | Description |
|---|---|
data_profile |
Full dataset metadata: frequency, index type, series lengths, missing values, exog columns |
task_type |
Family of the recommendation: 'single_series', 'multi_series', 'multivariate', 'statistical' or 'foundation' |
forecaster |
Recommended skforecast forecaster class name |
forecaster_candidates |
Ordered list of compatible forecaster names |
estimator |
Recommended estimator: a scikit-learn compatible estimator name, 'Arima' for statistical tasks, or the Hugging Face model ID of the foundation model |
estimator_candidates |
Ordered list of compatible estimator names |
series_pacf |
Per-series PACF-significant lags (used by plan() to set default lags) |
window_features |
Suggested window feature configurations |
calendar_features |
Recommended calendar feature names derived from the frequency |
explanation |
Human-readable explanation of why this forecaster and estimator were selected |
InĀ [6]:
Copied!
# Profile the data
# ==============================================================================
profile = assistant.profile(
data = data,
target = 'users',
date_column = 'date_time'
)
# Profile the data
# ==============================================================================
profile = assistant.profile(
data = data,
target = 'users',
date_column = 'date_time'
)
InĀ [7]:
Copied!
# Inspect the profile
# ==============================================================================
profile
# Inspect the profile
# ==============================================================================
profile
Out[7]:
Dataset Profile āāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā Property ā Value ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā Format ā single ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Series ā 1 ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Observations ā 17544 ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Frequency ā h ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Target ā users ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Exog columns ā holiday, weather, temp (categorical: weather) ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Missing values ā None ā āāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Recommendation āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā Property ā Value ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā Task type ā single_series ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Forecaster ā ForecasterRecursive ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Forecaster candidates ā ForecasterRecursive, ForecasterDirect, ForecasterFoundation ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Estimator ā LGBMRegressor ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Estimator candidates ā LGBMRegressor, XGBRegressor, Ridge ā āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Profile Explanation āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā A single-series ML forecaster (ForecasterRecursive) is recommended. Data: 17544 ā ā observations, 'h' frequency. Alternative forecasters: ['ForecasterDirect', ā ā 'ForecasterFoundation']. Estimator: LGBMRegressor. A gradient boosting model is ā ā preferred for a dataset of this size (17544 observations). Alternative estimators: ā ā ['XGBRegressor', 'Ridge']. 3 exogenous variables (1 categorical) available as ā ā predictors. ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
Every result also has a describe() method. It returns a plain-text description of the object, built by rules, without calling an LLM. It is also what ask() sends to the LLM about that object, so you can read it before asking.
InĀ [8]:
Copied!
# Plain-text description of the profile (no LLM involved)
# ==============================================================================
print(profile.describe())
# Plain-text description of the profile (no LLM involved)
# ==============================================================================
print(profile.describe())
<forecast_context> <dataset> - Observations: 17544 - Series: 1 - Frequency: h - Date range: 2011-01-01 to 2012-12-31 23:00:00 - Target: users - Exogenous columns: holiday, weather, temp - Categorical exogenous columns: weather - Target statistics: min 1, max 977, mean 187.8, std 181.3 - Missing values: none - Index irregularities: none detected </dataset> <profile_decision> A single-series ML forecaster (ForecasterRecursive) is recommended. Data: 17544 observations, 'h' frequency. Alternative forecasters: ['ForecasterDirect', 'ForecasterFoundation']. Estimator: LGBMRegressor. A gradient boosting model is preferred for a dataset of this size (17544 observations). Alternative estimators: ['XGBRegressor', 'Ridge']. 3 exogenous variables (1 categorical) available as predictors. - Significant lags (partial autocorrelation, strongest first): 1, 2, 23, 22, 25, 169, 10, 145, 17, 143, 167, 19, 21, 337, 3 (first 15 of 32) - Suggested window features: mean(window=3), std(window=3), mean(window=24), mean(window=168) - Suggested calendar features: hour, day_of_week, weekend, month </profile_decision> </forecast_context>
Once you have a profile, you can pass it to ask() as context to get an LLM-generated explanation of the modeling decisions. No plan is involved yet: the LLM only sees the dataset summary and the profile decisions.
Each call briefs the model with a set of skills, curated skforecast documents chosen from the context and the wording of the question, so the answer follows the current skforecast API. See Skills for the LLM for the list and how to choose them yourself.
InĀ [9]:
Copied!
# Ask the assistant to explain the profile
# ==============================================================================
answer = assistant.ask(
prompt = (
"Explain why this forecaster and estimator were recommended for my "
"hourly bike-sharing demand data, and what the exogenous variables add."
),
context = profile,
)
answer.show_explanation()
# Ask the assistant to explain the profile
# ==============================================================================
answer = assistant.ask(
prompt = (
"Explain why this forecaster and estimator were recommended for my "
"hourly bike-sharing demand data, and what the exogenous variables add."
),
context = profile,
)
answer.show_explanation()
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Assistant Response āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā ForecasterRecursive with LGBMRegressor was recommended because the dataset consists ā ā of a single hourly series with 17544 observations, making gradient boosting well ā ā suited for learning nonlinear patterns at this scale. The exogenous variables ā ā provide external contextual signals that may be associated with hourly changes in ā ā rider demand. ā ā ā ā Forecaster Recommendation ā ā ā ā ā¢ Model type: ForecasterRecursive is recommended as the single-series machine ā ā learning forecaster for the continuous target users across 17544 observations at ā ā frequency h. ā ā ā¢ Multi-step approach: It trains a single model that recursively feeds predictions ā ā into future steps, accommodating the autoregressive structure identified across ā ā the series. ā ā ā¢ Alternatives noted: The profile decision identifies ForecasterDirect and ā ā ForecasterFoundation as alternative forecaster choices. ā ā ā ā Estimator Recommendation ā ā ā ā ā¢ Gradient boosting preference: LGBMRegressor is preferred for a dataset of 17544 ā ā observations, where gradient boosted trees handle feature interactions ā ā efficiently without requiring deep model architectures. ā ā ā¢ Categorical compatibility: LGBMRegressor natively supports categorical predictors ā ā such as the weather variable without requiring extensive manual preprocessing. ā ā ā¢ Alternatives noted: The profile decision lists XGBRegressor and Ridge as ā ā alternative estimators. ā ā ā ā Role of Exogenous Variables ā ā ā ā ā¢ Predictor set: The data provides 3 exogenous variables: holiday, weather, and ā ā temp. ā ā ā¢ Categorical variable: weather is designated as a categorical exogenous column, ā ā capturing distinct conditions. ā ā ā¢ External factors: These variables supply environmental and calendar context that ā ā may contribute to predicting demand, operating alongside the 32 identified lag ā ā values, rolling window features, and calendar features. ā ā ā¢ Attribution limit: Feature importances are not provided, so the exact predictive ā ā contribution or ranking of these exogenous variables cannot be measured. ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
Build the planĀ¶
The plan() method converts the coarse modeling decisions in the ForecastingProfile into a fully-specified, executable configuration. It determines:
- Lags: derived from the PACF-significant lags detected in the profile. You can override these explicitly.
- Window features: rolling statistics configurations appropriate for the seasonal periods implied by the frequency.
- Preprocessing steps: ordered list of data preparation actions (e.g., handling missing values, gaps or categorical exogenous variables).
- Prediction interval method:
'bootstrapping','conformal', or'native'(selected based on the forecaster). - Metrics: the primary and secondary evaluation metrics.
Like profile(), this is a deterministic step. The resulting ForecastPlan object is the complete blueprint that forecast() and backtest() execute.
| Attribute | Description |
|---|---|
task_type |
Family of the plan, as in the profile, plus 'baseline' for ForecasterEquivalentDate |
forecaster |
Forecaster class name |
estimator |
Estimator class name; the Hugging Face model ID for ForecasterFoundation; None for ForecasterEquivalentDate |
forecaster_kwargs |
All constructor kwargs for the forecaster, including lags and window_features |
estimator_kwargs |
Constructor kwargs for the estimator |
steps |
Forecast horizon |
interval |
Prediction interval quantiles, e.g. [0.1, 0.9] |
interval_method |
Method used to produce the interval (bootstrapping, conformal, or native) |
metric |
Primary metric, used to rank a comparison |
use_exog |
Whether exogenous variables are included |
overridden_fields |
Decisions you made instead of the rules; empty when every value comes from the rules |
preprocessing_steps |
Ordered list of preprocessing actions with code snippets |
explanation |
Human-readable explanation of plan decisions |
InĀ [10]:
Copied!
# Build a plan from the profile
# ==============================================================================
plan = assistant.plan(
profile = profile,
steps = 36,
interval = [0.1, 0.9] # 80% prediction interval
)
# Build a plan from the profile
# ==============================================================================
plan = assistant.plan(
profile = profile,
steps = 36,
interval = [0.1, 0.9] # 80% prediction interval
)
InĀ [11]:
Copied!
# Inspect the plan
# ==============================================================================
plan
# Inspect the plan
# ==============================================================================
plan
Out[11]:
Forecast Plan āāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā Property ā Value ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā Task type ā single_series ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Forecaster ā ForecasterRecursive ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Estimator ā LGBMRegressor ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Steps ā 36 ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Frequency ā h ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Lags ā [1, 2, 3, 5, 8, 10, 15, 17, 19, 20, 21, 22, 23, 24, 25, 26, 32, ā ā ā 33, 119, 121, 135, 136, 142, 143, 145, 160, 166, 167, 169, 313, ā ā ā 335, 337] ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Window features ā [{'stats': ['mean', 'std'], 'window_size': 3}, {'stats': ['mean'], ā ā ā 'window_size': 24}, {'stats': ['mean'], 'window_size': 168}] ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Calendar features ā ['hour', 'day_of_week', 'weekend', 'month'] (raw ordinal encoding) ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Use exog ā True ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Interval ā [0.1, 0.9] ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Interval method ā bootstrapping ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Primary metric ā mean_absolute_error ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Preprocessing ā 1 step ā āāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Preprocessing Steps āāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā Step ā Reason ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā handle_categorical_exog ā Categorical exogenous variables detected: ['weather']. These ā ā ā are handled automatically by skforecast ā ā ā (categorical_features='auto'). ā āāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Plan Explanation āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā Plan: ForecasterRecursive + LGBMRegressor. Lags: [1, 2, 3, 5, 8, 10, 15, 17, 19, 20, ā ā 21, 22, 23, 24, 25, 26, 32, 33, 119, 121, 135, 136, 142, 143, 145, 160, 166, 167, ā ā 169, 313, 335, 337]. Window features: ['mean(window=3)', 'std(window=3)', ā ā 'mean(window=24)', 'mean(window=168)']. Calendar features: ['hour', 'day_of_week', ā ā 'weekend', 'month'] (raw ordinal encoding). Prediction intervals via bootstrapping. ā ā Exogenous variables included. MAE is interpretable, robust to outliers, and works at ā ā any scale. ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
Pass the profile as context together with the plan for a detailed explanation of the chosen configuration. The two are rendered into the same script forecast_code() would produce: answer.code returns it, and the LLM receives a description of it (never the script itself).
InĀ [12]:
Copied!
# Ask the assistant to explain the plan
# ==============================================================================
answer = assistant.ask(
prompt = (
"Walk me through this plan. Why these lags and window features, "
"and how will the 80% prediction interval be produced?"
),
context = profile,
plan = plan,
)
answer.show_explanation()
# Ask the assistant to explain the plan
# ==============================================================================
answer = assistant.ask(
prompt = (
"Walk me through this plan. Why these lags and window features, "
"and how will the 80% prediction interval be produced?"
),
context = profile,
plan = plan,
)
answer.show_explanation()
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Assistant Response āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā The forecast plan configures a ForecasterRecursive model with LGBMRegressor to ā ā generate 36 hourly forecasts for the users series. The design uses 32 specific lags ā ā alongside rolling window statistics and calendar features to represent the series ā ā dynamics across multiple horizons. The 80% prediction interval is generated via ā ā residual bootstrapping using lower and upper percentiles. ā ā ā ā Plan Overview ā ā ā ā ā¢ Forecaster: ForecasterRecursive with an LGBMRegressor estimator. ā ā ā¢ Horizon: 36 steps ahead. ā ā ā¢ Exogenous inputs: holiday, weather, and temp, where weather is categorical and ā ā handled via categorical_features='auto'. ā ā ā¢ Calendar features: hour, day_of_week, weekend, and month using raw ordinal ā ā encoding. ā ā ā¢ Primary evaluation metric: mean_absolute_error. ā ā ā ā Lags and Window Features ā ā ā ā ā¢ Lag selection: The plan includes 32 lags: [1, 2, 3, 5, 8, 10, 15, 17, 19, 20, 21, ā ā 22, 23, 24, 25, 26, 32, 33, 119, 121, 135, 136, 142, 143, 145, 160, 166, 167, ā ā 169, 313, 335, 337]. ā ā ā¢ Autocorrelation basis: These lags originate from the dataset profile, which ā ā identified significant partial autocorrelation structure. The strongest partial ā ā autocorrelation values appeared at lags 1, 2, 23, 22, 25, 169, 10, 145, 17, 143, ā ā 167, 19, 21, 337, and 3. ā ā ā¢ Window features: Three sets of rolling features are configured: ā ā ā¢ Window size 3: mean and std. ā ā ā¢ Window size 24: mean. ā ā ā¢ Window size 168: mean. ā ā ā¢ Window scope: These statistics aggregate behavior over the immediate short term ā ā (3 hours), a daily cycle (24 hours), and a weekly cycle (168 hours). ā ā ā ā Prediction Interval Calculation ā ā ā ā ā¢ Target interval: An 80% prediction interval spanning the quantiles [0.1, 0.9]. ā ā ā¢ Method: Bootstrapping. ā ā ā¢ Procedure: During model fitting, residuals are collected. In recursive ā ā prediction, bootstrap samples are drawn with replacement from these residuals and ā ā applied across the 36-step trajectory, creating simulated paths from which the ā ā 10th and 90th percentiles are extracted as interval bounds. ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
Refine the plan (optional)Ā¶
The refine_plan() method lets you adjust the plan before execution. It operates in two distinct modes:
Deterministic mode (
prompt=None): pass explicit configuration overrides such aslags,estimator,estimator_kwargs,forecaster,steps,interval, orwindow_features. Only the fields you explicitly specify are updated; the rest of the configuration is deterministically re-derived from the original plan.LLM mode (
promptprovided): describe your domain knowledge in natural language. The LLM interprets this context and suggests appropriatelagsandwindow_features. Its reasoning is appended toplan.explanationand the changed fields are recorded inplan.llm_refined_fieldsfor full traceability. It applies to the machine learning forecasters:ForecasterStats,ForecasterFoundationandForecasterEquivalentDateuse no lags or window features, so for them the prompt is ignored with a warning and the LLM is not called.
The Agentic forecasting guide shows in an animation what the LLM receives in LLM mode, and how a suggestion is validated before it enters the plan.
ā Warning
A refined plan is a hypothesis, not a guaranteed improvement. The LLM may propose lags or window features that are not helpful for the series, or it may misread the domain context you provided. Always compare the refined plan against the original baseline using a proper backtest over multiple folds before adopting it.
Deterministic modeĀ¶
InĀ [13]:
Copied!
# Refine the plan with explicit overrides (no LLM required)
# ==============================================================================
plan_det = assistant.refine_plan(
profile = profile,
plan = plan,
lags = [1, 2, 3, 24, 48, 168],
estimator_kwargs = {'n_estimators': 200, 'max_depth': 6},
metric = 'mean_absolute_scaled_error'
)
plan_det
# Refine the plan with explicit overrides (no LLM required)
# ==============================================================================
plan_det = assistant.refine_plan(
profile = profile,
plan = plan,
lags = [1, 2, 3, 24, 48, 168],
estimator_kwargs = {'n_estimators': 200, 'max_depth': 6},
metric = 'mean_absolute_scaled_error'
)
plan_det
Out[13]:
Forecast Plan āāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā Property ā Value ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā Task type ā single_series ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Forecaster ā ForecasterRecursive ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Estimator ā LGBMRegressor ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Steps ā 36 ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Frequency ā h ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Lags ā [1, 2, 3, 24, 48, 168] ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Window features ā [{'stats': ['mean', 'std'], 'window_size': 3}, {'stats': ['mean'], ā ā ā 'window_size': 24}, {'stats': ['mean'], 'window_size': 168}] ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Calendar features ā ['hour', 'day_of_week', 'weekend', 'month'] (raw ordinal encoding) ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Use exog ā True ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Interval ā [0.1, 0.9] ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Interval method ā bootstrapping ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Primary metric ā mean_absolute_scaled_error ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Preprocessing ā 1 step ā āāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Preprocessing Steps āāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā Step ā Reason ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā handle_categorical_exog ā Categorical exogenous variables detected: ['weather']. These ā ā ā are handled automatically by skforecast ā ā ā (categorical_features='auto'). ā āāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Plan Explanation āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā Plan: ForecasterRecursive + LGBMRegressor. Lags: [1, 2, 3, 24, 48, 168]. Window ā ā features: ['mean(window=3)', 'std(window=3)', 'mean(window=24)', ā ā 'mean(window=168)']. Calendar features: ['hour', 'day_of_week', 'weekend', 'month'] ā ā (raw ordinal encoding). Prediction intervals via bootstrapping. Exogenous variables ā ā included. Metric: mean_absolute_scaled_error, as requested. ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
metric is one of the arguments that choose a decision instead of the rule. The others are use_exog, differentiation, calendar_features, target_transformer and dropna_from_series, and plan() accepts them too. overridden_fields names the decisions you made, so the plan always tells your choices from the ones of the rules.
InĀ [14]:
Copied!
# Decisions made by the user instead of the rules
# ==============================================================================
plan_det.overridden_fields
# Decisions made by the user instead of the rules
# ==============================================================================
plan_det.overridden_fields
Out[14]:
['estimator_kwargs', 'lags', 'metric']
LLM modeĀ¶
InĀ [15]:
Copied!
# Refine the plan using LLM-guided domain knowledge
# ==============================================================================
prompt = (
"I'm forecasting hourly bike rentals. Demand follows a clear daily rhythm with "
"rush-hour peaks, and it changes between weekdays and weekends. It's also usually "
"similar to what happened at the same time last week, and the last few hours give "
"a good sense of the current trend. Please pick lags and rolling features that fit this."
)
plan_refined = assistant.refine_plan(
profile = profile,
plan = plan,
prompt = prompt
)
# Refine the plan using LLM-guided domain knowledge
# ==============================================================================
prompt = (
"I'm forecasting hourly bike rentals. Demand follows a clear daily rhythm with "
"rush-hour peaks, and it changes between weekdays and weekends. It's also usually "
"similar to what happened at the same time last week, and the last few hours give "
"a good sense of the current trend. Please pick lags and rolling features that fit this."
)
plan_refined = assistant.refine_plan(
profile = profile,
plan = plan,
prompt = prompt
)
InĀ [16]:
Copied!
# Refined plan proposed by the assistant
# ==============================================================================
plan_refined
# Refined plan proposed by the assistant
# ==============================================================================
plan_refined
Out[16]:
Forecast Plan āāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā Property ā Value ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā Task type ā single_series ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Forecaster ā ForecasterRecursive ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Estimator ā LGBMRegressor ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Steps ā 36 ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Frequency ā h ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Lags ā [1, 2, 3, 4, 23, 24, 25, 47, 48, 167, 168, 169] (LLM-suggested) ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Window features ā [{'stats': ['mean', 'std'], 'window_size': 4}, {'stats': ['mean', ā ā ā 'max'], 'window_size': 24}, {'stats': ['mean'], 'window_size': ā ā ā 168}] (LLM-suggested) ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Calendar features ā ['hour', 'day_of_week', 'weekend', 'month'] (raw ordinal encoding) ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Use exog ā True ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Interval ā [0.1, 0.9] ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Interval method ā bootstrapping ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Primary metric ā mean_absolute_error ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Preprocessing ā 1 step ā āāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Preprocessing Steps āāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā Step ā Reason ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā handle_categorical_exog ā Categorical exogenous variables detected: ['weather']. These ā ā ā are handled automatically by skforecast ā ā ā (categorical_features='auto'). ā āāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Plan Explanation āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā Plan: ForecasterRecursive + LGBMRegressor. Lags: [1, 2, 3, 4, 23, 24, 25, 47, 48, ā ā 167, 168, 169]. Window features: ['mean(window=4)', 'std(window=4)', ā ā 'mean(window=24)', 'max(window=24)', 'mean(window=168)']. Calendar features: ā ā ['hour', 'day_of_week', 'weekend', 'month'] (raw ordinal encoding). Prediction ā ā intervals via bootstrapping. Exogenous variables included. MAE is interpretable, ā ā robust to outliers, and works at any scale. ā ā ā ā LLM Refinement Reasoning: To capture the hourly bike rental dynamics described: ā ā ā ā ā¢ Lags 1 to 4 capture short-term autoregressive momentum and the recent hourly ā ā trend. ā ā ā¢ Lags 23, 24, 25, 47, and 48 capture daily periodicity (24 hours and 48 hours ā ā prior) along with adjacent hourly shifts around daily rush hours. ā ā ā¢ Lags 167, 168, and 169 capture weekly seasonality (168 hours = 7 days), ensuring ā ā the model aligns with the corresponding hour and day of the previous week ā ā (differentiating weekday and weekend patterns). ā ā ā¢ Rolling features across a 4-hour window (mean and standard deviation) capture ā ā immediate trend and short-term demand variability. ā ā ā¢ A 24-hour rolling window (mean and max) captures daily baseline demand and peak ā ā rush-hour intensity. ā ā ā¢ A 168-hour rolling mean tracks weekly baseline level shifts. ā ā ā ā Note: the LLM-suggested lags and window_features are hypotheses, not validated ā ā improvements. Confirm any expected accuracy gain before relying on them. ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
Explain mode (refined plan)Ā¶
InĀ [17]:
Copied!
# Ask the assistant what changed and why
# ==============================================================================
answer = assistant.ask(
prompt = (
"What changed in the refined plan compared to the original, "
"and why does it matter for this dataset?"
),
context = profile,
plan = plan_refined,
)
answer.show_explanation()
# Ask the assistant what changed and why
# ==============================================================================
answer = assistant.ask(
prompt = (
"What changed in the refined plan compared to the original, "
"and why does it matter for this dataset?"
),
context = profile,
plan = plan_refined,
)
answer.show_explanation()
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Assistant Response āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā The refined plan updates the feature engineering setup by specifying a targeted lag ā ā structure and adjusting the rolling window features. These changes aim to capture ā ā multi-scale patterns in the hourly dataset, specifically short-term momentum, daily ā ā cycles, and weekly seasonality. ā ā ā ā Changes in the Refined Plan ā ā ā ā ā¢ Lags: The original profile identified significant partial autocorrelation lags ā ā (such as 1, 2, 23, 22, 25, 169, and others), whereas the refined plan selects a ā ā specific subset: [1, 2, 3, 4, 23, 24, 25, 47, 48, 167, 168, 169]. ā ā ā¢ Window features: The initial profile suggested mean(window=3), std(window=3), ā ā mean(window=24), and mean(window=168). The refined plan adjusts the short window ā ā from size 3 to size 4 with mean and std, adds max alongside mean for window size ā ā 24, and retains the 168-hour rolling mean. ā ā ā¢ Unchanged components: The forecaster type (ForecasterRecursive), estimator ā ā (LGBMRegressor), calendar features (hour, day_of_week, weekend, month), and ā ā exogenous variables (holiday, weather, temp) remain the same. ā ā ā ā Why These Adjustments Matter for the Dataset ā ā ā ā ā¢ Short-term dynamics: Lags 1 to 4 and the 4-hour rolling mean and standard ā ā deviation capture recent hourly momentum, short-term demand variability, and ā ā immediate trends. ā ā ā¢ Daily periodicity: Lags 23, 24, 25, 47, and 48 cover 24-hour and 48-hour cycles ā ā along with adjacent hours around daily rush hours. The 24-hour rolling window ā ā with mean and max captures daily baseline demand and peak rush-hour intensity. ā ā ā¢ Weekly seasonality: Lags 167, 168, and 169 (where 168 hours equals 7 days) and ā ā the 168-hour rolling mean align predictions with the corresponding day and hour ā ā of the prior week, distinguishing weekday from weekend patterns and tracking ā ā weekly baseline shifts. ā ā ā ā Note that the refined lags and window features are hypotheses rather than validated ā ā improvements. You can evaluate their performance against alternative configurations ā ā by running assistant.backtest() or comparing setups using assistant.compare(). ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
ForecastĀ¶
Once you have a profile and a plan, you can call forecast() or forecast_code(). Both accept the pre-computed profile and plan so no additional profiling is performed. The forecast() method executes the generated script and returns a ForecastResult; forecast_code() generates the script only, without running it.
The forecast branch operates in two modes:
Evaluation mode (
test_sizeis set): the dataset is split into train and test sets, the model is trained on the train portion, and predictions are compared against the held-out actuals to compute metrics.Prediction mode (
test_size=None, the default): the model is trained on the entire dataset and forecasts the nextstepstime points into the future. Because there is no ground truth, no metrics are returned. If the plan uses exogenous variables, their future values must be supplied viaexog.
Evaluation modeĀ¶
InĀ [18]:
Copied!
# Forecast in evaluation mode, reusing the pre-computed profile and plan
# ==============================================================================
results_eval = assistant.forecast(
data = data,
steps = 36,
interval = [0.1, 0.9], # 80% prediction interval
test_size = 36, # Last 36 hours as test set
profile = profile, # Reuse the pre-computed profile
plan = plan_refined # Reuse the refined plan
)
display(results_eval.metrics)
display(results_eval.predictions.head())
# Forecast in evaluation mode, reusing the pre-computed profile and plan
# ==============================================================================
results_eval = assistant.forecast(
data = data,
steps = 36,
interval = [0.1, 0.9], # 80% prediction interval
test_size = 36, # Last 36 hours as test set
profile = profile, # Reuse the pre-computed profile
plan = plan_refined # Reuse the refined plan
)
display(results_eval.metrics)
display(results_eval.predictions.head())
| series | MAE | MSE | MASE | MAPE | |
|---|---|---|---|---|---|
| 0 | users | 46.542601 | 4757.93798 | 0.722832 | 0.470479 |
| pred | lower_bound | upper_bound | |
|---|---|---|---|
| 2012-12-30 12:00:00 | 150.140531 | 119.962576 | 180.806519 |
| 2012-12-30 13:00:00 | 131.285497 | 93.529479 | 175.605520 |
| 2012-12-30 14:00:00 | 129.158744 | 92.731312 | 171.324671 |
| 2012-12-30 15:00:00 | 130.043044 | 94.448394 | 168.117601 |
| 2012-12-30 16:00:00 | 137.809651 | 92.988786 | 174.743751 |
InĀ [19]:
Copied!
# Plot predictions vs. actual values for the held-out test period
# ==============================================================================
set_dark_theme()
preds = results_eval.predictions
fig, ax = plt.subplots(figsize=(7, 3.5))
data.set_index('date_time').loc[preds.index, 'users'].plot(ax=ax, label='actual')
preds['pred'].plot(ax=ax, label='prediction')
if {'lower_bound', 'upper_bound'}.issubset(preds.columns):
ax.fill_between(
preds.index, preds['lower_bound'], preds['upper_bound'],
alpha=0.3, label='80% prediction interval'
)
ax.set_title('Predictions vs. actual bike demand')
ax.set_ylabel('Users')
ax.legend()
plt.tight_layout()
plt.show()
# Plot predictions vs. actual values for the held-out test period
# ==============================================================================
set_dark_theme()
preds = results_eval.predictions
fig, ax = plt.subplots(figsize=(7, 3.5))
data.set_index('date_time').loc[preds.index, 'users'].plot(ax=ax, label='actual')
preds['pred'].plot(ax=ax, label='prediction')
if {'lower_bound', 'upper_bound'}.issubset(preds.columns):
ax.fill_between(
preds.index, preds['lower_bound'], preds['upper_bound'],
alpha=0.3, label='80% prediction interval'
)
ax.set_title('Predictions vs. actual bike demand')
ax.set_ylabel('Users')
ax.legend()
plt.tight_layout()
plt.show()
InĀ [20]:
Copied!
# Ask the assistant to interpret the forecast results
# ==============================================================================
answer = assistant.ask(
prompt = "Explain the results of this forecast, including the metrics and predictions.",
context = results_eval
)
answer.show_explanation()
# Ask the assistant to interpret the forecast results
# ==============================================================================
answer = assistant.ask(
prompt = "Explain the results of this forecast, including the metrics and predictions.",
context = results_eval
)
answer.show_explanation()
/var/folders/wt/8tvn563d5v55nspfbydgqb9r0000gp/T/ipykernel_71015/3255811032.py:3: DataSentToLLMWarning: `send_data_to_llm=False` does not apply to `context`: the predicted values it carries are sent to the LLM, because a question about a result cannot be answered from summary statistics alone. Your input data is not sent: a result holds only the model's output, never the data it was fitted on. To keep predictions local, ask without `context`. answer = assistant.ask(
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Assistant Response āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā The 36-step forecast for the users series was generated using ForecasterRecursive ā ā with LGBMRegressor, producing point predictions alongside an 80% bootstrapping ā ā prediction interval. The model achieved a Mean Absolute Error of 46.542601 and a ā ā Mean Absolute Scaled Error of 0.722832 on evaluation. Across the 36 forecast steps, ā ā the predicted values average 67.64379275041186. ā ā ā ā Evaluation Metrics ā ā ā ā The model evaluation produced the following metrics: ā ā ā ā ā¢ MAE: 46.542601 ā ā ā¢ MSE: 4757.93798 ā ā ā¢ MASE: 0.722832. Because this value is below 1, the model beats the one-step naive ā ā forecast (last training value). ā ā ā¢ MAPE: 0.470479. MAPE represents a percentage error; note that it becomes ā ā unreliable as the target approaches zero, and the dataset target minimum is 1. ā ā ā ā Prediction Summary ā ā ā ā The forecast covers 36 hourly steps from 2012-12-30 12:00:00 to 2012-12-31 23:00:00, ā ā accompanied by lower and upper bounds corresponding to the [0.1, 0.9] interval. ā ā Across all 36 steps: ā ā ā ā ā¢ Point predictions (pred): minimum of 3.2108862522441286, maximum of ā ā 158.6940989915074, and mean of 67.64379275041186. ā ā ā¢ Lower bound: minimum of -1.021796684480627, maximum of 119.96257558258043, and ā ā mean of 40.46442222067216. ā ā ā¢ Upper bound: minimum of 9.56485040531428, maximum of 184.70997640284222, and mean ā ā of 102.6152088255848. ā ā ā ā Because intermediate rows are omitted, overall progressions or horizon-wide trends ā ā cannot be evaluated beyond these summary statistics. ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
Prediction modeĀ¶
In prediction mode, the model trains on the entire dataset and forecasts the next steps time points. Because the data includes exogenous variables (holiday, weather, temp), their future values must be supplied via the exog argument.
InĀ [21]:
Copied!
# Forecast the next 36 hours using the entire dataset (prediction mode)
# ==============================================================================
# Simulate future values of exogenous variables for the next 36 hours
exog = data[['holiday', 'weather', 'temp']].tail(36).copy()
exog.index = pd.date_range(
start=pd.to_datetime(data['date_time'].max()) + pd.Timedelta(1, unit='h'),
periods=36,
freq='h'
)
results_pred = assistant.forecast(
data = data,
steps = 36,
interval = [0.1, 0.9],
test_size = None, # Use the entire dataset (prediction mode)
exog = exog, # Future values of exogenous variables
profile = profile,
plan = plan_refined
)
display(results_pred.predictions.head())
# Forecast the next 36 hours using the entire dataset (prediction mode)
# ==============================================================================
# Simulate future values of exogenous variables for the next 36 hours
exog = data[['holiday', 'weather', 'temp']].tail(36).copy()
exog.index = pd.date_range(
start=pd.to_datetime(data['date_time'].max()) + pd.Timedelta(1, unit='h'),
periods=36,
freq='h'
)
results_pred = assistant.forecast(
data = data,
steps = 36,
interval = [0.1, 0.9],
test_size = None, # Use the entire dataset (prediction mode)
exog = exog, # Future values of exogenous variables
profile = profile,
plan = plan_refined
)
display(results_pred.predictions.head())
| pred | lower_bound | upper_bound | |
|---|---|---|---|
| 2013-01-01 00:00:00 | 21.581406 | 13.059198 | 31.121462 |
| 2013-01-01 01:00:00 | 10.182992 | 3.921566 | 19.012760 |
| 2013-01-01 02:00:00 | 6.319521 | 2.408637 | 12.343762 |
| 2013-01-01 03:00:00 | 6.074862 | 1.447885 | 10.981221 |
| 2013-01-01 04:00:00 | 6.074862 | 1.931540 | 12.771343 |
InĀ [22]:
Copied!
# Full results object
# ==============================================================================
results_pred
# Full results object
# ==============================================================================
results_pred
Out[22]:
Dataset Profile āāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā Property ā Value ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā Format ā single ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Series ā 1 ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Observations ā 17544 ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Frequency ā h ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Target ā users ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Exog columns ā holiday, weather, temp (categorical: weather) ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Missing values ā None ā āāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Recommendation āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā Property ā Value ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā Task type ā single_series ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Forecaster ā ForecasterRecursive ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Forecaster candidates ā ForecasterRecursive, ForecasterDirect, ForecasterFoundation ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Estimator ā LGBMRegressor ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Estimator candidates ā LGBMRegressor, XGBRegressor, Ridge ā āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Profile Explanation āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā A single-series ML forecaster (ForecasterRecursive) is recommended. Data: 17544 ā ā observations, 'h' frequency. Alternative forecasters: ['ForecasterDirect', ā ā 'ForecasterFoundation']. Estimator: LGBMRegressor. A gradient boosting model is ā ā preferred for a dataset of this size (17544 observations). Alternative estimators: ā ā ['XGBRegressor', 'Ridge']. 3 exogenous variables (1 categorical) available as ā ā predictors. ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ Forecast Plan āāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā Property ā Value ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā Task type ā single_series ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Forecaster ā ForecasterRecursive ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Estimator ā LGBMRegressor ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Steps ā 36 ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Frequency ā h ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Lags ā [1, 2, 3, 4, 23, 24, 25, 47, 48, 167, 168, 169] (LLM-suggested) ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Window features ā [{'stats': ['mean', 'std'], 'window_size': 4}, {'stats': ['mean', ā ā ā 'max'], 'window_size': 24}, {'stats': ['mean'], 'window_size': ā ā ā 168}] (LLM-suggested) ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Calendar features ā ['hour', 'day_of_week', 'weekend', 'month'] (raw ordinal encoding) ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Use exog ā True ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Interval ā [0.1, 0.9] ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Interval method ā bootstrapping ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Primary metric ā mean_absolute_error ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Preprocessing ā 1 step ā āāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Preprocessing Steps āāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā Step ā Reason ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā handle_categorical_exog ā Categorical exogenous variables detected: ['weather']. These ā ā ā are handled automatically by skforecast ā ā ā (categorical_features='auto'). ā āāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Plan Explanation āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā Plan: ForecasterRecursive + LGBMRegressor. Lags: [1, 2, 3, 4, 23, 24, 25, 47, 48, ā ā 167, 168, 169]. Window features: ['mean(window=4)', 'std(window=4)', ā ā 'mean(window=24)', 'max(window=24)', 'mean(window=168)']. Calendar features: ā ā ['hour', 'day_of_week', 'weekend', 'month'] (raw ordinal encoding). Prediction ā ā intervals via bootstrapping. Exogenous variables included. MAE is interpretable, ā ā robust to outliers, and works at any scale. ā ā ā ā LLM Refinement Reasoning: To capture the hourly bike rental dynamics described: ā ā ā ā ā¢ Lags 1 to 4 capture short-term autoregressive momentum and the recent hourly ā ā trend. ā ā ā¢ Lags 23, 24, 25, 47, and 48 capture daily periodicity (24 hours and 48 hours ā ā prior) along with adjacent hourly shifts around daily rush hours. ā ā ā¢ Lags 167, 168, and 169 capture weekly seasonality (168 hours = 7 days), ensuring ā ā the model aligns with the corresponding hour and day of the previous week ā ā (differentiating weekday and weekend patterns). ā ā ā¢ Rolling features across a 4-hour window (mean and standard deviation) capture ā ā immediate trend and short-term demand variability. ā ā ā¢ A 24-hour rolling window (mean and max) captures daily baseline demand and peak ā ā rush-hour intensity. ā ā ā¢ A 168-hour rolling mean tracks weekly baseline level shifts. ā ā ā ā Note: the LLM-suggested lags and window_features are hypotheses, not validated ā ā improvements. Confirm any expected accuracy gain before relying on them. ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ Predictions (36 rows) āāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāā³āāāāāāāāāāāāāā³āāāāāāāāāāāāāā ā Index ā pred ā lower_bound ā upper_bound ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā 2013-01-01 00:00:00 ā 21.5814 ā 13.0592 ā 31.1215 ā āāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¤ ā 2013-01-01 01:00:00 ā 10.1830 ā 3.9216 ā 19.0128 ā āāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¤ ā 2013-01-01 02:00:00 ā 6.3195 ā 2.4086 ā 12.3438 ā āāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¤ ā 2013-01-01 03:00:00 ā 6.0749 ā 1.4479 ā 10.9812 ā āāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¤ ā 2013-01-01 04:00:00 ā 6.0749 ā 1.9315 ā 12.7713 ā āāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¤ ā ... ā ... ā ... ā ... ā āāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¤ ā 2013-01-02 07:00:00 ā 42.6970 ā 23.3424 ā 93.2515 ā āāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¤ ā 2013-01-02 08:00:00 ā 70.6854 ā 41.8467 ā 156.6811 ā āāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¤ ā 2013-01-02 09:00:00 ā 90.7043 ā 44.4452 ā 181.0452 ā āāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¤ ā 2013-01-02 10:00:00 ā 88.7133 ā 39.4447 ā 169.7889 ā āāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¤ ā 2013-01-02 11:00:00 ā 86.6475 ā 32.0955 ā 180.7717 ā āāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāā“āāāāāāāāāāāāāā“āāāāāāāāāāāāāā
Code-only modeĀ¶
Use forecast_code() when you want to preview or export the reproducible script without executing it. This is useful for code review, auditing the generated pipeline, or running the script in a separate environment.
InĀ [23]:
Copied!
# Generate the reproducible script without executing it
# ==============================================================================
code_result = assistant.forecast_code(
data = data,
steps = 36,
test_size = 36,
profile = profile,
plan = plan_refined
)
code_result.show_code()
# Generate the reproducible script without executing it
# ==============================================================================
code_result = assistant.forecast_code(
data = data,
steps = 36,
test_size = 36,
profile = profile,
plan = plan_refined
)
code_result.show_code()
The ForecastResult objectĀ¶
Both forecast() modes return a ForecastResult, a lightweight container that bundles everything the assistant used and produced.
| Attribute | Type | Description |
|---|---|---|
predictions |
DataFrame | Forecasted values. When intervals are requested, the bound columns are included alongside the point predictions. |
metrics |
DataFrame or None | Evaluation metrics (MAE, MSE, MASE, MAPE), one row per series. None in prediction mode. |
code |
str | The exact standalone skforecast script that produced the forecast, ready to run on its own. |
profile |
ForecastingProfile |
The data profile behind the forecast. |
plan |
ForecastPlan |
The detailed configuration that was executed. |
BacktestingĀ¶
The backtesting branch uses the same profile and plan as the forecast branch but evaluates the model's historical performance through time series cross-validation. The key decision is how to configure the TimeSeriesFold object, which controls exactly how the historical data is partitioned into successive training and test windows.
skforecast-ai provides three distinct ways to define this validation strategy:
Explicit instantiation (recommended): manually construct a
TimeSeriesFoldand pass it directly tobacktest(). Use this when you already know your exact operational constraints.Deterministic
create_cv(): allow the assistant to derive a sensibleTimeSeriesFoldfrom the profile and plan using rule-based defaults. You can override individual parameters explicitly.LLM
create_cv()(with a prompt): describe your deployment use case in natural language. The LLM translates your description into a fully-configuredTimeSeriesFold, accompanied by an explanation you can audit.
Define the backtesting strategyĀ¶
Manual TimeSeriesFoldĀ¶
InĀ [24]:
Copied!
# Create your own TimeSeriesFold object
# ==============================================================================
end_train = '2012-08-31 23:59:00'
cv = TimeSeriesFold(
steps = 36,
initial_train_size = end_train,
refit = False,
verbose = False
)
cv
# Create your own TimeSeriesFold object
# ==============================================================================
end_train = '2012-08-31 23:59:00'
cv = TimeSeriesFold(
steps = 36,
initial_train_size = end_train,
refit = False,
verbose = False
)
cv
Out[24]:
TimeSeriesFold
General Information
- Initial train size: 2012-08-31 23:59:00
- Initial train size as int: None
- Steps: 36
- Fold stride: 36
- Overlapping folds: False
- Window size: None
- Differentiation: None
- Refit: False
- Fixed train size: True
- Gap: 0
- Skip folds: None
- Allow incomplete fold: True
- Return all indexes: False
Deterministic create_cv()Ā¶
Only initial_train_size is passed here; the rest of the strategy comes from the defaults. The explanation says which values you chose and why each default has its value.
InĀ [25]:
Copied!
# Let the assistant derive a TimeSeriesFold with rule-based defaults
# ==============================================================================
cv_det = assistant.create_cv(
profile = profile,
plan = plan_refined,
initial_train_size = end_train,
)
cv_det
# Let the assistant derive a TimeSeriesFold with rule-based defaults
# ==============================================================================
cv_det = assistant.create_cv(
profile = profile,
plan = plan_refined,
initial_train_size = end_train,
)
cv_det
Out[25]:
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Cross-Validation Explanation āāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā Initial training up to 2012-08-31 23:59:00, trained once (no refit), 36-step ā ā horizon, 82 folds. Trained once by default: refitting in every fold would multiply ā ā the training cost by the 82 folds. initial_train_size as requested. ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ Cross-Validation Configuration āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāā ā Parameter ā Value ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā steps ā 36 ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā initial_train_size ā 2012-08-31 23:59:00 ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā refit ā False ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā fixed_train_size ā False ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā gap ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā fold_stride ā 36 ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā skip_folds ā None ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā allow_incomplete_fold ā True ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā differentiation ā None ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā n_folds ā 82 ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā n_fits ā 1 ā āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāā
LLM create_cv() with a natural-language promptĀ¶
Rather than manually configuring TimeSeriesFold parameters, you can describe your backtesting strategy in natural language and let the assistant translate it into a rigorous cross-validation schema. The Agentic forecasting guide shows this translation in an animation.
InĀ [26]:
Copied!
# Let the assistant create the TimeSeriesFold from a natural-language prompt
# ==============================================================================
prompt = (
"I forecast bike demand 36 hours ahead. "
"The model should be trained once on all data up to the end of August 2012, 23:59. "
"Do not refit the model as the window rolls forward."
)
cv_llm = assistant.create_cv(
profile = profile,
plan = plan_refined,
prompt = prompt
)
# Let the assistant create the TimeSeriesFold from a natural-language prompt
# ==============================================================================
prompt = (
"I forecast bike demand 36 hours ahead. "
"The model should be trained once on all data up to the end of August 2012, 23:59. "
"Do not refit the model as the window rolls forward."
)
cv_llm = assistant.create_cv(
profile = profile,
plan = plan_refined,
prompt = prompt
)
InĀ [27]:
Copied!
# Cross-validation strategy derived from the prompt
# ==============================================================================
cv_llm
# Cross-validation strategy derived from the prompt
# ==============================================================================
cv_llm
Out[27]:
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Cross-Validation Explanation āāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā The model is trained once on all historical data up to the end of August 2012 ā ā ('2012-08-31 23:00:00') with refit=False as requested. This leaves 4 full months ā ā (SepāDec 2012, ~2,928 hours) for backtesting evaluation across successive 36-hour ā ā horizons, easily exceeding the 2-fold minimum requirement. Initial training up to ā ā 2012-08-31 23:00:00, trained once (no refit), 36-step horizon, 82 folds. ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ Cross-Validation Configuration āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāā ā Parameter ā Value ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā steps ā 36 ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā initial_train_size ā 2012-08-31 23:00:00 ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā refit ā False ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā fixed_train_size ā False ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā gap ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā fold_stride ā 36 ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā skip_folds ā None ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā allow_incomplete_fold ā True ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā differentiation ā None ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā n_folds ā 82 ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā n_fits ā 1 ā āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāā
InĀ [28]:
Copied!
# LLM reasoning behind the TimeSeriesFold configuration
# ==============================================================================
print(textwrap.fill(cv_llm.explanation, width=88))
# LLM reasoning behind the TimeSeriesFold configuration
# ==============================================================================
print(textwrap.fill(cv_llm.explanation, width=88))
The model is trained once on all historical data up to the end of August 2012
('2012-08-31 23:00:00') with refit=False as requested. This leaves 4 full months
(SepāDec 2012, ~2,928 hours) for backtesting evaluation across successive 36-hour
horizons, easily exceeding the 2-fold minimum requirement. Initial training up to
2012-08-31 23:00:00, trained once (no refit), 36-step horizon, 82 folds.
Since the prompt correctly describes the intended use case, the cv_llm object returned by create_cv() is equivalent to the one we built manually (the same 82 folds; the timestamp is aligned to the hourly frequency). However, it was derived entirely from a natural-language description. The returned CVResult carries the splitter as cv_llm.cv and, as cv_llm.explanation, a description of the choices the assistant made, allowing you to verify that the resulting TimeSeriesFold matches your intended strategy before executing the backtest.
Run the backtestĀ¶
InĀ [29]:
Copied!
# Run backtesting, reusing the pre-computed profile and plan
# ==============================================================================
results_backtest = assistant.backtest(
data = data,
cv = cv, # TimeSeriesFold, or a CVResult from create_cv()
interval = [0.1, 0.9], # 80% prediction interval
profile = profile, # Reuse the pre-computed profile
plan = plan_refined # Reuse the refined plan
)
results_backtest.show_explanation()
display(results_backtest.metrics)
display(results_backtest.predictions.head())
# Run backtesting, reusing the pre-computed profile and plan
# ==============================================================================
results_backtest = assistant.backtest(
data = data,
cv = cv, # TimeSeriesFold, or a CVResult from create_cv()
interval = [0.1, 0.9], # 80% prediction interval
profile = profile, # Reuse the pre-computed profile
plan = plan_refined # Reuse the refined plan
)
results_backtest.show_explanation()
display(results_backtest.metrics)
display(results_backtest.predictions.head())
0%| | 0/82 [00:00<?, ?it/s]
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Backtest Explanation āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā Initial training up to 2012-08-31 23:59:00, trained once (no refit), 36-step ā ā horizon, 82 folds. Results: mean_absolute_error: 50.1874, mean_squared_error: ā ā 6935.7326, mean_absolute_scaled_error: 0.8185, mean_absolute_percentage_error: ā ā 0.5483. ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
| mean_absolute_error | mean_squared_error | mean_absolute_scaled_error | mean_absolute_percentage_error | |
|---|---|---|---|---|
| 0 | 50.187384 | 6935.732623 | 0.818529 | 0.548273 |
| fold | pred | lower_bound | upper_bound | |
|---|---|---|---|---|
| 2012-09-01 00:00:00 | 0 | 122.935984 | 93.616739 | 153.673163 |
| 2012-09-01 01:00:00 | 0 | 115.683353 | 83.414275 | 146.211674 |
| 2012-09-01 02:00:00 | 0 | 73.660319 | 47.602204 | 104.086342 |
| 2012-09-01 03:00:00 | 0 | 45.817784 | 21.567420 | 69.256026 |
| 2012-09-01 04:00:00 | 0 | 19.884786 | 5.453657 | 37.048089 |
The plot opens on ten days of October, where the predictions, the actual values and the interval can be told apart. Double-click on it to see the whole test period, from September to December 2012.
InĀ [30]:
Copied!
# Plot prediction intervals vs real value
# ==============================================================================
predictions = results_backtest.predictions
data_test = data.set_index("date_time").loc[predictions.index, :]
fig = go.Figure([
go.Scatter(name='Prediction', x=predictions.index, y=predictions['pred'], mode='lines'),
go.Scatter(
name='Real value', x=data_test.index, y=data_test['users'], mode='lines',
),
go.Scatter(
name='Upper Bound', x=predictions.index, y=predictions['upper_bound'], mode='lines',
marker=dict(color="#444"), line=dict(width=0), showlegend=False
),
go.Scatter(
name='Lower Bound', x=predictions.index, y=predictions['lower_bound'], marker=dict(color="#444"),
line=dict(width=0), mode='lines', fillcolor='rgba(68, 68, 68, 0.3)', fill='tonexty', showlegend=False
)
])
fig.update_layout(
title="Real value vs predicted in test data",
xaxis_title="Date time",
yaxis_title="users",
width=800,
height=400,
margin=dict(l=20, r=20, t=35, b=20),
hovermode="x",
legend=dict(orientation="h", yanchor="top", y=1.1, xanchor="left", x=0.001),
# Initial zoom on x axis betwee 1 oct to 10 oct
xaxis=dict(range=['2012-10-01', '2012-10-10'])
)
fig.show()
# Plot prediction intervals vs real value
# ==============================================================================
predictions = results_backtest.predictions
data_test = data.set_index("date_time").loc[predictions.index, :]
fig = go.Figure([
go.Scatter(name='Prediction', x=predictions.index, y=predictions['pred'], mode='lines'),
go.Scatter(
name='Real value', x=data_test.index, y=data_test['users'], mode='lines',
),
go.Scatter(
name='Upper Bound', x=predictions.index, y=predictions['upper_bound'], mode='lines',
marker=dict(color="#444"), line=dict(width=0), showlegend=False
),
go.Scatter(
name='Lower Bound', x=predictions.index, y=predictions['lower_bound'], marker=dict(color="#444"),
line=dict(width=0), mode='lines', fillcolor='rgba(68, 68, 68, 0.3)', fill='tonexty', showlegend=False
)
])
fig.update_layout(
title="Real value vs predicted in test data",
xaxis_title="Date time",
yaxis_title="users",
width=800,
height=400,
margin=dict(l=20, r=20, t=35, b=20),
hovermode="x",
legend=dict(orientation="h", yanchor="top", y=1.1, xanchor="left", x=0.001),
# Initial zoom on x axis betwee 1 oct to 10 oct
xaxis=dict(range=['2012-10-01', '2012-10-10'])
)
fig.show()
InĀ [31]:
Copied!
# Ask the assistant to interpret the backtesting results
# ==============================================================================
answer = assistant.ask(
prompt = (
"Explain the results of this backtesting, including the strategy, metrics, "
"and predictions. Is the model good enough to deploy?"
),
context = results_backtest
)
answer.show_explanation()
# Ask the assistant to interpret the backtesting results
# ==============================================================================
answer = assistant.ask(
prompt = (
"Explain the results of this backtesting, including the strategy, metrics, "
"and predictions. Is the model good enough to deploy?"
),
context = results_backtest
)
answer.show_explanation()
/var/folders/wt/8tvn563d5v55nspfbydgqb9r0000gp/T/ipykernel_71015/2899763076.py:3: DataSentToLLMWarning: `send_data_to_llm=False` does not apply to `context`: the predicted values it carries are sent to the LLM, because a question about a result cannot be answered from summary statistics alone. Your input data is not sent: a result holds only the model's output, never the data it was fitted on. To keep predictions local, ask without `context`. answer = assistant.ask(
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Assistant Response āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā The backtesting evaluated a ForecasterRecursive model with LGBMRegressor over 82 ā ā non-overlapping folds of 36 steps each, fitting once on data up to 2012-08-31 ā ā 23:59:00 without refitting. Across 2928 evaluated points, the model achieved a mean ā ā absolute error of 50.187384 and a mean absolute scaled error of 0.818529. Whether ā ā the model is ready to deploy cannot be determined from the provided outputs alone, ā ā as deployment acceptance criteria and operational tolerances are not available. ā ā ā ā Backtesting Strategy ā ā ā ā ā¢ Initial training window ended at 2012-08-31 23:59:00. ā ā ā¢ Forecast horizon: 36 steps per fold. ā ā ā¢ Fold stride: 36 observations, resulting in contiguous, non-overlapping testing ā ā periods. ā ā ā¢ Refit strategy: refit=False with fixed_train_size=True, meaning the model was ā ā fitted once (n_fits: 1) and tested across all 82 folds. ā ā ā¢ Gap: 0 observations between training and forecast windows. ā ā ā ā Evaluation Metrics ā ā ā ā ā¢ mean_absolute_error: 50.187384 ā ā ā¢ mean_squared_error: 6935.732623 ā ā ā¢ mean_absolute_scaled_error: 0.818529 ā ā ā¢ mean_absolute_percentage_error: 0.548273 ā ā ā¢ Interpretation: The mean absolute scaled error is below 1, which means the model ā ā beats the one-step naive forecast (last training value). Note that mean absolute ā ā percentage error is a percentage and becomes unreliable as the target approaches ā ā zero, which is pertinent because the target series has a minimum value of 1. ā ā ā ā Predictions Summary ā ā ā ā ā¢ Total predicted steps: 2928 observations across 82 folds. ā ā ā¢ Prediction interval: 80% coverage ([0.1, 0.9]) generated via bootstrapping. ā ā ā¢ Point predictions (pred): minimum 0.9913779330166613, maximum 859.9199274681027, ā ā mean 228.7613702369822. ā ā ā¢ Lower interval bound: minimum -2.743895653237591, maximum 798.2572409634424, mean ā ā 183.41207043556466. ā ā ā¢ Upper interval bound: minimum 6.341991034967448, maximum 893.6292721994874, mean ā ā 274.2824443850359. ā ā ā ā Deployment Readiness ā ā ā ā ā¢ It cannot be validated whether the model is ready for deployment because specific ā ā business tolerances, error cost thresholds, and service-level targets are not ā ā present in the context. ā ā ā¢ The model beats the one-step naive forecast (last training value) based on its ā ā mean absolute scaled error of 0.818529, but whether an average deviation of ā ā 50.187384 users is acceptable depends on the application requirements. ā ā ā¢ If you want to benchmark this pipeline against other models before deciding, you ā ā can use assistant.compare() to evaluate candidate models on the same evaluation ā ā setup. ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
Code-only modeĀ¶
Use backtest_code() to generate the reproducible backtesting script without executing it.
InĀ [32]:
Copied!
# Generate the reproducible backtest script without executing it
# ==============================================================================
code_bt = assistant.backtest_code(
data = data,
cv = cv,
profile = profile,
plan = plan_refined
)
code_bt.show_code()
# Generate the reproducible backtest script without executing it
# ==============================================================================
code_bt = assistant.backtest_code(
data = data,
cv = cv,
profile = profile,
plan = plan_refined
)
code_bt.show_code()
The BacktestResult objectĀ¶
The backtest() method returns a BacktestResult, a lightweight container that bundles all the backtesting artifacts.
| Attribute | Type | Description |
|---|---|---|
predictions |
DataFrame | Full out-of-sample backtest predictions across all folds. |
metrics |
DataFrame | Backtesting metrics (MAE, MSE, MASE, MAPE), one row per series, computed across all folds. |
cv_config |
dict | Resolved TimeSeriesFold parameters for full traceability of the validation strategy. |
code |
str | The exact standalone skforecast script that reproduces the backtesting workflow. |
explanation |
str | Human-readable summary of the backtesting configuration and results. |
profile |
ForecastingProfile |
The data profile behind the backtest. |
plan |
ForecastPlan |
The detailed configuration that was executed. |
InĀ [33]:
Copied!
# Full results object
# ==============================================================================
results_backtest
# Full results object
# ==============================================================================
results_backtest
Out[33]:
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Backtest Explanation āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā Initial training up to 2012-08-31 23:59:00, trained once (no refit), 36-step ā ā horizon, 82 folds. Results: mean_absolute_error: 50.1874, mean_squared_error: ā ā 6935.7326, mean_absolute_scaled_error: 0.8185, mean_absolute_percentage_error: ā ā 0.5483. ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ Cross-Validation Configuration āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāā ā Parameter ā Value ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā steps ā 36 ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā initial_train_size ā 2012-08-31 23:59:00 ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā refit ā False ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā fixed_train_size ā True ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā gap ā 0 ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā fold_stride ā 36 ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā skip_folds ā None ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā allow_incomplete_fold ā True ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā differentiation ā None ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā n_folds ā 82 ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāā¤ ā n_fits ā 1 ā āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāā Backtest Metrics āāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāā ā mean_absolute_error ā mean_squared_error ā mean_absolute_scaleā¦ ā mean_absolute_perceā¦ ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā 50.1874 ā 6935.7326 ā 0.8185 ā 0.5483 ā āāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāā Backtest Predictions (2928 rows) āāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāā³āāāāāāāāāāā³āāāāāāāāāāāāāā³āāāāāāāāāāāāāā ā Index ā fold ā pred ā lower_bound ā upper_bound ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā 2012-09-01 00:00:00 ā 0.0000 ā 122.9360 ā 93.6167 ā 153.6732 ā āāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāā¼āāāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¤ ā 2012-09-01 01:00:00 ā 0.0000 ā 115.6834 ā 83.4143 ā 146.2117 ā āāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāā¼āāāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¤ ā 2012-09-01 02:00:00 ā 0.0000 ā 73.6603 ā 47.6022 ā 104.0863 ā āāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāā¼āāāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¤ ā 2012-09-01 03:00:00 ā 0.0000 ā 45.8178 ā 21.5674 ā 69.2560 ā āāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāā¼āāāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¤ ā 2012-09-01 04:00:00 ā 0.0000 ā 19.8848 ā 5.4537 ā 37.0481 ā āāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāā¼āāāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¤ ā ... ā ... ā ... ā ... ā ... ā āāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāā¼āāāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¤ ā 2012-12-31 19:00:00 ā 81.0000 ā 80.6760 ā 33.4467 ā 109.1784 ā āāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāā¼āāāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¤ ā 2012-12-31 20:00:00 ā 81.0000 ā 58.9545 ā 19.4684 ā 89.6922 ā āāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāā¼āāāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¤ ā 2012-12-31 21:00:00 ā 81.0000 ā 36.4224 ā 15.5316 ā 62.6609 ā āāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāā¼āāāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¤ ā 2012-12-31 22:00:00 ā 81.0000 ā 25.0615 ā 11.4475 ā 41.2955 ā āāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāā¼āāāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¤ ā 2012-12-31 23:00:00 ā 81.0000 ā 16.0466 ā 6.0405 ā 30.4788 ā āāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāā“āāāāāāāāāāā“āāāāāāāāāāāāāā“āāāāāāāāāāāāāā Dataset Profile āāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā Property ā Value ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā Format ā single ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Series ā 1 ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Observations ā 17544 ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Frequency ā h ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Target ā users ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Exog columns ā holiday, weather, temp (categorical: weather) ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Missing values ā None ā āāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Recommendation āāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā Property ā Value ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā Task type ā single_series ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Forecaster ā ForecasterRecursive ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Forecaster candidates ā ForecasterRecursive, ForecasterDirect, ForecasterFoundation ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Estimator ā LGBMRegressor ā āāāāāāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Estimator candidates ā LGBMRegressor, XGBRegressor, Ridge ā āāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Profile Explanation āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā A single-series ML forecaster (ForecasterRecursive) is recommended. Data: 17544 ā ā observations, 'h' frequency. Alternative forecasters: ['ForecasterDirect', ā ā 'ForecasterFoundation']. Estimator: LGBMRegressor. A gradient boosting model is ā ā preferred for a dataset of this size (17544 observations). Alternative estimators: ā ā ['XGBRegressor', 'Ridge']. 3 exogenous variables (1 categorical) available as ā ā predictors. ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ Forecast Plan āāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā Property ā Value ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā Task type ā single_series ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Forecaster ā ForecasterRecursive ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Estimator ā LGBMRegressor ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Steps ā 36 ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Frequency ā h ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Lags ā [1, 2, 3, 4, 23, 24, 25, 47, 48, 167, 168, 169] (LLM-suggested) ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Window features ā [{'stats': ['mean', 'std'], 'window_size': 4}, {'stats': ['mean', ā ā ā 'max'], 'window_size': 24}, {'stats': ['mean'], 'window_size': ā ā ā 168}] (LLM-suggested) ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Calendar features ā ['hour', 'day_of_week', 'weekend', 'month'] (raw ordinal encoding) ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Use exog ā True ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Interval ā [0.1, 0.9] ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Interval method ā bootstrapping ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Primary metric ā mean_absolute_error ā āāāāāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Preprocessing ā 1 step ā āāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Preprocessing Steps āāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā Step ā Reason ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā handle_categorical_exog ā Categorical exogenous variables detected: ['weather']. These ā ā ā are handled automatically by skforecast ā ā ā (categorical_features='auto'). ā āāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Plan Explanation āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā Plan: ForecasterRecursive + LGBMRegressor. Lags: [1, 2, 3, 4, 23, 24, 25, 47, 48, ā ā 167, 168, 169]. Window features: ['mean(window=4)', 'std(window=4)', ā ā 'mean(window=24)', 'max(window=24)', 'mean(window=168)']. Calendar features: ā ā ['hour', 'day_of_week', 'weekend', 'month'] (raw ordinal encoding). Prediction ā ā intervals via bootstrapping. Exogenous variables included. MAE is interpretable, ā ā robust to outliers, and works at any scale. ā ā ā ā LLM Refinement Reasoning: To capture the hourly bike rental dynamics described: ā ā ā ā ā¢ Lags 1 to 4 capture short-term autoregressive momentum and the recent hourly ā ā trend. ā ā ā¢ Lags 23, 24, 25, 47, and 48 capture daily periodicity (24 hours and 48 hours ā ā prior) along with adjacent hourly shifts around daily rush hours. ā ā ā¢ Lags 167, 168, and 169 capture weekly seasonality (168 hours = 7 days), ensuring ā ā the model aligns with the corresponding hour and day of the previous week ā ā (differentiating weekday and weekend patterns). ā ā ā¢ Rolling features across a 4-hour window (mean and standard deviation) capture ā ā immediate trend and short-term demand variability. ā ā ā¢ A 24-hour rolling window (mean and max) captures daily baseline demand and peak ā ā rush-hour intensity. ā ā ā¢ A 168-hour rolling mean tracks weekly baseline level shifts. ā ā ā ā Note: the LLM-suggested lags and window_features are hypotheses, not validated ā ā improvements. Confirm any expected accuracy gain before relying on them. ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
Comparing forecaster configurationsĀ¶
Choosing a forecasting model should not rely on intuition alone. Two configurations that look equally reasonable can perform very differently once evaluated on real temporal data. The most reliable approach is to test every candidate under identical conditions and compare their metrics.
The compare() method does exactly that. It receives a list of candidate configurations, backtests each one using the same TimeSeriesFold strategy, and returns a leaderboard ranked by the selected metric.
In the step-by-step path, the key argument is profile. Passing the profile computed at the beginning of this tutorial skips profiling entirely and guarantees that every candidate is evaluated against the same data profile. Note that compare() does not accept a plan: each candidate derives its own plan from the shared profile, which is precisely what makes the candidates differ.
Candidates can be provided in two ways:
Automatic candidates (
candidates=None): the assistant builds the comparison set from theprofile.forecaster_candidatesthat belong to the same family as the recommended forecaster, so every row is scored on the same thing. For multi-series data, a multi-series forecaster is never ranked against a multivariate one:ForecasterRecursiveMultiSeriesis compared withForecasterFoundation. This is useful when exploring a new dataset without a predefined shortlist. Whencvis theCVResultofcreate_cv()for a plan that is none of those candidates (a plan refined with another estimator or other lags, for example), that plan runs too, as the row'Strategy plan', and the summary names it.Explicit candidates (recommended): pass a list of
(name, config)tuples, wherenamelabels the row in the leaderboard andconfigholds the same override keys understood byplan():'forecaster','estimator','estimator_kwargs','lags','window_features','use_exog','differentiation','calendar_features','target_transformer'and'dropna_from_series'. This provides full control and makes the resulting table easier to interpret.
For a single series, compare() also adds a seasonal naive baseline (ForecasterEquivalentDate, which repeats the value of one seasonal period earlier, or the last value when the frequency has none) as one more row, ranked like the others, so the leaderboard shows whether a model beats the simplest reasonable forecast. Pass baseline=False to leave it out.
A failed candidate does not stop the comparison. Instead, a CandidateFailedWarning is issued, the row records the error and is placed last.
The Agentic forecasting guide shows a comparison in an animation: the candidates run through the same folds, and the leaderboard is a plain sort of the metric.
š” Tip
All candidates use the same cross-validation strategy, ensuring a fair comparison. However, the results are only meaningful if the cv setup reflects the real use case where the model will be deployed. For example, if the production system retrains weekly, the backtest should also refit weekly. If the model is expected to forecast 24 hours ahead, the backtest should use a 24-hour horizon.
The evaluation window must also be representative. A period that is too short or dominated by unusual events (holidays, outages, or exceptional peaks) may favor a candidate that performs poorly over time. Define the validation setup carefully before comparing models so the final ranking is reliable.
Automatic candidatesĀ¶
InĀ [34]:
Copied!
# Compare the forecaster candidates suggested by the profile
# ==============================================================================
results_compare = assistant.compare(
data = data,
cv = cv, # Same TimeSeriesFold used in the backtest above
candidates = None, # Candidates suggested by the assistant
profile = profile # Reuse the pre-computed profile
)
# Compare the forecaster candidates suggested by the profile
# ==============================================================================
results_compare = assistant.compare(
data = data,
cv = cv, # Same TimeSeriesFold used in the backtest above
candidates = None, # Candidates suggested by the assistant
profile = profile # Reuse the pre-computed profile
)
Comparing forecasters: 0%| | 0/4 [00:00<?, ?it/s]
Loading weights: 0%| | 0/92 [00:00<?, ?it/s]
InĀ [35]:
Copied!
# Ranked leaderboard
# ==============================================================================
results_compare.results
# Ranked leaderboard
# ==============================================================================
results_compare.results
Out[35]:
| rank | name | forecaster | estimator | mean_absolute_error | mean_squared_error | mean_absolute_scaled_error | mean_absolute_percentage_error | |
|---|---|---|---|---|---|---|---|---|
| 0 | 1 | ForecasterFoundation | ForecasterFoundation | autogluon/chronos-2-small | 38.436159 | 4231.325965 | 0.597467 | 0.611349 |
| 1 | 2 | ForecasterRecursive | ForecasterRecursive | LGBMRegressor | 46.312218 | 5495.881638 | 0.749598 | 0.472735 |
| 2 | 3 | ForecasterDirect | ForecasterDirect | LGBMRegressor | 49.298312 | 5936.622526 | 0.797930 | 0.499413 |
| 3 | 4 | Baseline (seasonal naive) | ForecasterEquivalentDate | None | 91.668716 | 22652.390710 | 1.503712 | 0.912133 |
InĀ [36]:
Copied!
# Deterministic summary of the comparison
# ==============================================================================
results_compare.show_explanation()
# Deterministic summary of the comparison
# ==============================================================================
results_compare.show_explanation()
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Comparison Explanation āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā Compared 4 configurations, ranked ascending by mean_absolute_error. Shared ā ā cross-validation strategy: Initial training up to 2012-08-31 23:59:00, trained once ā ā (no refit), 36-step horizon, 82 folds. ForecasterFoundation is not trained: the ā ā window and refit settings do not apply to it, each fold forecasts from the ā ā observations before it. Best: 'ForecasterFoundation' (ForecasterFoundation / ā ā autogluon/chronos-2-small) = 38.4362, 17.0% ahead of 'ForecasterRecursive' ā ā (46.3122). All 3 configurations beat the baseline 'Baseline (seasonal naive)' ā ā (91.6687). The best, 'ForecasterFoundation', beats it by 58.1%. ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
Explicit candidatesĀ¶
In practice, you will often already have a shortlist in mind: a fast baseline, a gradient boosting model, or a variant with a richer feature set. Passing explicit (name, config) tuples keeps the comparison focused and makes the resulting leaderboard easy to understand at a glance.
The config dictionary accepts the same overrides as plan(). Any omitted option falls back to the deterministic recommendation derived from the shared profile, so candidates can remain concise. For example, {'forecaster': 'ForecasterDirect'} changes only the forecaster while keeping the recommended estimator, lags, and features.
ā ļø Computational cost
Each candidate is backtested independently across all folds, so runtime increases with both the number and complexity of the configurations. Comparing four candidates will take roughly four times as long as running one backtest.
Start with a small set of clearly different options, review the results, and refine from there. Testing many near-identical variants is costly and rarely useful.
InĀ [37]:
Copied!
# Compare an explicit shortlist of configurations
# ==============================================================================
candidates = [
(
"ridge_baseline",
{
"forecaster": "ForecasterRecursive",
"estimator" : "Ridge",
"lags" : 24,
}
),
(
"lgbm_daily_lags",
{
"forecaster": "ForecasterRecursive",
"estimator" : "LGBMRegressor",
"lags" : 24,
}
),
(
"refined_plan",
{
"forecaster" : plan_refined.forecaster,
"estimator" : plan_refined.estimator,
"lags" : plan_refined.forecaster_kwargs.get("lags"),
"window_features" : plan_refined.forecaster_kwargs.get("window_features"),
}
),
(
"lgbm_direct",
{
"forecaster": "ForecasterDirect",
"estimator" : "LGBMRegressor",
"lags" : 24,
}
),
(
"foundation_model",
{
"forecaster": "ForecasterFoundation"
}
),
]
results_compare = assistant.compare(
data = data,
cv = cv,
candidates = candidates, # Specific candidates to compare
metric = ['mean_absolute_error', 'mean_absolute_scaled_error'],
profile = profile
)
# Compare an explicit shortlist of configurations
# ==============================================================================
candidates = [
(
"ridge_baseline",
{
"forecaster": "ForecasterRecursive",
"estimator" : "Ridge",
"lags" : 24,
}
),
(
"lgbm_daily_lags",
{
"forecaster": "ForecasterRecursive",
"estimator" : "LGBMRegressor",
"lags" : 24,
}
),
(
"refined_plan",
{
"forecaster" : plan_refined.forecaster,
"estimator" : plan_refined.estimator,
"lags" : plan_refined.forecaster_kwargs.get("lags"),
"window_features" : plan_refined.forecaster_kwargs.get("window_features"),
}
),
(
"lgbm_direct",
{
"forecaster": "ForecasterDirect",
"estimator" : "LGBMRegressor",
"lags" : 24,
}
),
(
"foundation_model",
{
"forecaster": "ForecasterFoundation"
}
),
]
results_compare = assistant.compare(
data = data,
cv = cv,
candidates = candidates, # Specific candidates to compare
metric = ['mean_absolute_error', 'mean_absolute_scaled_error'],
profile = profile
)
Comparing forecasters: 0%| | 0/6 [00:00<?, ?it/s]
Loading weights: 0%| | 0/92 [00:00<?, ?it/s]
The refined_plan candidate reuses the forecaster, estimator, lags and window features of the plan produced by refine_plan(). This is the recommended way to validate a refined plan: the leaderboard shows whether the extra domain knowledge actually improves the metrics compared to the deterministic baselines. Passing create_cv(profile=profile, plan=plan_refined) as cv with candidates=None does the same with the candidates of the profile: the refined plan is the row 'Strategy plan'.
When several metrics are requested, all of them are shown as columns but only the first one drives the ranking.
InĀ [38]:
Copied!
# Ranked leaderboard, sorted by the first metric requested
# ==============================================================================
results_compare.results
# Ranked leaderboard, sorted by the first metric requested
# ==============================================================================
results_compare.results
Out[38]:
| rank | name | forecaster | estimator | mean_absolute_error | mean_absolute_scaled_error | |
|---|---|---|---|---|---|---|
| 0 | 1 | foundation_model | ForecasterFoundation | autogluon/chronos-2-small | 38.436159 | 0.597467 |
| 1 | 2 | refined_plan | ForecasterRecursive | LGBMRegressor | 50.187384 | 0.818529 |
| 2 | 3 | lgbm_direct | ForecasterDirect | LGBMRegressor | 50.224908 | 0.819189 |
| 3 | 4 | lgbm_daily_lags | ForecasterRecursive | LGBMRegressor | 55.517689 | 0.905517 |
| 4 | 5 | Baseline (seasonal naive) | ForecasterEquivalentDate | None | 91.668716 | 1.503712 |
| 5 | 6 | ridge_baseline | ForecasterRecursive | Ridge | 93.145620 | 1.519244 |
Inspect individual candidatesĀ¶
Because every candidate is a full BacktestResult, the details of any individual configuration remain available, including its metrics, its predictions and the standalone script that generated them.
InĀ [39]:
Copied!
# Inspect a specific candidate
# ==============================================================================
candidate = results_compare.candidates['foundation_model']
display(candidate.metrics)
display(candidate.predictions.head())
candidate.show_code()
# Inspect a specific candidate
# ==============================================================================
candidate = results_compare.candidates['foundation_model']
display(candidate.metrics)
display(candidate.predictions.head())
candidate.show_code()
| mean_absolute_error | mean_absolute_scaled_error | |
|---|---|---|
| 0 | 38.436159 | 0.597467 |
| level | fold | pred | |
|---|---|---|---|
| 2012-09-01 00:00:00 | users | 0 | 148.059464 |
| 2012-09-01 01:00:00 | users | 0 | 102.715004 |
| 2012-09-01 02:00:00 | users | 0 | 66.084412 |
| 2012-09-01 03:00:00 | users | 0 | 40.878693 |
| 2012-09-01 04:00:00 | users | 0 | 29.577866 |
Reuse the winning configurationĀ¶
The most useful result of a comparison is often not the leaderboard, but best_candidate. It is a complete BacktestResult carrying both the winning profile and plan, so it can be fed back into the step-by-step workflow without manually rebuilding the configuration.
InĀ [40]:
Copied!
# Winning configuration
# ==============================================================================
print(f"Best candidate: {results_compare.best_name}")
results_compare.best_candidate.plan
# Winning configuration
# ==============================================================================
print(f"Best candidate: {results_compare.best_name}")
results_compare.best_candidate.plan
Best candidate: foundation_model
Out[40]:
Forecast Plan āāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāāāāā ā Property ā Value ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā Task type ā foundation ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Forecaster ā ForecasterFoundation ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Estimator ā autogluon/chronos-2-small ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Steps ā 36 ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Frequency ā h ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Use exog ā True ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Interval ā None ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Primary metric ā mean_absolute_error ā āāāāāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāāāāāāāāāāāāāāā¤ ā Preprocessing ā 1 step ā āāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāāāāā Preprocessing Steps āāāāāāāāāāāāāāāāāāāāāāāāāāā³āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā ā Step ā Reason ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā handle_categorical_exog ā Categorical exogenous variables detected: ['weather']. ā ā ā 'autogluon/chronos-2-small' consumes categorical covariates ā ā ā natively, so no encoding is needed. ā āāāāāāāāāāāāāāāāāāāāāāāāāāā“āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Plan Explanation āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā Plan: ForecasterFoundation + autogluon/chronos-2-small. No lag or window features: ā ā the foundation model forecasts directly from the raw context window. Exogenous ā ā variables included. Primary metric: mean_absolute_error, as requested; also ā ā computed: mean_absolute_scaled_error. The model reads the last 8192 observations of ā ā the series as context; the series has 17544, so older observations are not used. ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
InĀ [41]:
Copied!
# Produce the final forecast with the winning configuration
# ==============================================================================
results_pred_best = assistant.forecast(
data = data,
steps = 36,
interval = [0.1, 0.9],
test_size = None, # Prediction mode
exog = exog, # Future values of exogenous variables
profile = results_compare.profile, # Shared profile
plan = results_compare.best_candidate.plan # Winning plan
)
results_pred_best.show_code()
# Produce the final forecast with the winning configuration
# ==============================================================================
results_pred_best = assistant.forecast(
data = data,
steps = 36,
interval = [0.1, 0.9],
test_size = None, # Prediction mode
exog = exog, # Future values of exogenous variables
profile = results_compare.profile, # Shared profile
plan = results_compare.best_candidate.plan # Winning plan
)
results_pred_best.show_code()
Loading weights: 0%| | 0/92 [00:00<?, ?it/s]
Explain mode (comparison)Ā¶
Like any other result, a ComparisonResult can be passed to ask() as context to have the ranking explained. The LLM restates what the leaderboard measured, not why one candidate beat another, and it cannot change the outcome: all metrics and rankings are computed deterministically before it sees the result.
InĀ [42]:
Copied!
# Ask the assistant to interpret the comparison
# ==============================================================================
answer = assistant.ask(
prompt = (
"Explain the comparison results. Is the margin between the top candidates "
"meaningful, or are they practically equivalent?"
),
context = results_compare
)
answer.show_explanation()
# Ask the assistant to interpret the comparison
# ==============================================================================
answer = assistant.ask(
prompt = (
"Explain the comparison results. Is the margin between the top candidates "
"meaningful, or are they practically equivalent?"
),
context = results_compare
)
answer.show_explanation()
/var/folders/wt/8tvn563d5v55nspfbydgqb9r0000gp/T/ipykernel_71015/2209842622.py:3: DataSentToLLMWarning: `send_data_to_llm=False` does not apply to `context`: the predicted values it carries are sent to the LLM, because a question about a result cannot be answered from summary statistics alone. Your input data is not sent: a result holds only the model's output, never the data it was fitted on. To keep predictions local, ask without `context`. answer = assistant.ask(
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Assistant Response āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā The comparison evaluated six candidates across 82 folds on a 36-step forecast ā ā horizon, ranked in ascending order by mean_absolute_error. The top candidate is ā ā foundation_model (ForecasterFoundation using autogluon/chronos-2-small) with a ā ā mean_absolute_error of 38.436159, followed by refined_plan with 50.187384 and ā ā lgbm_direct with 50.224908. Per-fold spread is not given, so whether the margin ā ā between the top candidates is meaningful cannot be assessed. ā ā ā ā Leaderboard Summary ā ā ā ā The evaluation produced the following rankings: ā ā ā ā ā¢ Rank 1: foundation_model (ForecasterFoundation, autogluon/chronos-2-small) with ā ā mean_absolute_error of 38.436159 and mean_absolute_scaled_error of 0.597467. ā ā ā¢ Rank 2: refined_plan (ForecasterRecursive, LGBMRegressor) with ā ā mean_absolute_error of 50.187384 and mean_absolute_scaled_error of 0.818529. ā ā ā¢ Rank 3: lgbm_direct (ForecasterDirect, LGBMRegressor) with mean_absolute_error of ā ā 50.224908 and mean_absolute_scaled_error of 0.819189. ā ā ā¢ Rank 4: lgbm_daily_lags (ForecasterRecursive, LGBMRegressor) with ā ā mean_absolute_error of 55.517689 and mean_absolute_scaled_error of 0.905517. ā ā ā¢ Rank 5: Baseline (seasonal naive) (ForecasterEquivalentDate) with ā ā mean_absolute_error of 91.668716 and mean_absolute_scaled_error of 1.503712. ā ā ā¢ Rank 6: ridge_baseline (ForecasterRecursive, Ridge) with mean_absolute_error of ā ā 93.145620 and mean_absolute_scaled_error of 1.519244. ā ā ā ā Assessing Margins Between Candidates ā ā ā ā ā¢ The leaderboard reports the aggregate performance metrics across all 82 folds, ā ā but per-fold metrics and fold-to-fold variance are not provided in the results. ā ā ā¢ Because per-fold spread is not given, whether a margin is meaningful cannot be ā ā assessed, and no gap calculation can establish equivalence or significance. ā ā ā¢ The leaderboard reflects what metric values each configuration attained; it does ā ā not provide underlying causes for why one candidate outperformed another. ā ā ā ā Benchmark Interpretation ā ā ā ā ā¢ The top four models (foundation_model, refined_plan, lgbm_direct, and ā ā lgbm_daily_lags) all recorded a mean_absolute_scaled_error below 1, meaning each ā ā beat the one-step naive forecast (last training value). ā ā ā¢ Baseline (seasonal naive) and ridge_baseline each had a ā ā mean_absolute_scaled_error above 1, indicating that neither beat the one-step ā ā naive forecast (last training value). ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
The ComparisonResult objectĀ¶
The compare() method returns a ComparisonResult, which groups the shared setup, the ranked leaderboard, and the individual backtests in a single object.
| Attribute | Type | Description |
|---|---|---|
results |
DataFrame | Ranked leaderboard, one row per candidate, sorted best to worst. Columns: rank, name, forecaster, estimator, the metric columns, and error when at least one candidate failed. |
candidates |
dict | Mapping of candidate name to the full BacktestResult object. |
failures |
dict | Mapping of candidate name to a CandidateFailure describing why it failed. Empty when every candidate succeeds. |
ranking_metric |
str | Name of the metric used to sort results. |
cv_config |
dict | Resolved TimeSeriesFold parameters plus the resulting n_folds, applied identically to every candidate. |
profile |
ForecastingProfile |
The shared data profile behind every candidate. |
explanation |
str | Deterministic, human-readable summary of the comparison. |
best_name |
str | Name of the top-ranked candidate. |
best_candidate |
BacktestResult |
Top-ranked candidate as a complete BacktestResult. |
baseline_name |
str, None | Name of the seasonal naive baseline row, or None when no baseline was added (multi-series data, baseline=False, or a target with missing values or missing timestamps). |
InĀ [43]:
Copied!
# Full results object
# ==============================================================================
results_compare
# Full results object
# ==============================================================================
results_compare
Out[43]:
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Comparison Explanation āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā Compared 6 configurations, ranked ascending by mean_absolute_error. Shared ā ā cross-validation strategy: Initial training up to 2012-08-31 23:59:00, trained once ā ā (no refit), 36-step horizon, 82 folds. ForecasterFoundation is not trained: the ā ā window and refit settings do not apply to it, each fold forecasts from the ā ā observations before it. Best: 'foundation_model' (ForecasterFoundation / ā ā autogluon/chronos-2-small) = 38.4362, 23.4% ahead of 'refined_plan' (50.1874). 4 of ā ā 5 configurations beat the baseline 'Baseline (seasonal naive)' (91.6687); ā ā 'ridge_baseline' does not. The best, 'foundation_model', beats it by 58.1%. ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ Comparison Results āāāāāāāāā³āāāāāāā³āāāāāāāāāāāāāāā³āāāāāāāāāāāāāāā³āāāāāāāāāāāāāāā³āāāāāāāāāāāāāā³āāāāāāāāāāāāāāā ā Index ā rank ā name ā forecaster ā estimator ā mean_absolā¦ ā mean_absoluā¦ ā ā”āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā© ā 0 ā 1 ā foundation_ā¦ ā ForecasterFā¦ ā autogluon/cā¦ ā 38.4362 ā 0.5975 ā āāāāāāāāā¼āāāāāāā¼āāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā¤ ā 1 ā 2 ā refined_plan ā ForecasterRā¦ ā LGBMRegressā¦ ā 50.1874 ā 0.8185 ā āāāāāāāāā¼āāāāāāā¼āāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā¤ ā 2 ā 3 ā lgbm_direct ā ForecasterDā¦ ā LGBMRegressā¦ ā 50.2249 ā 0.8192 ā āāāāāāāāā¼āāāāāāā¼āāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā¤ ā 3 ā 4 ā lgbm_daily_ā¦ ā ForecasterRā¦ ā LGBMRegressā¦ ā 55.5177 ā 0.9055 ā āāāāāāāāā¼āāāāāāā¼āāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā¤ ā 4 ā 5 ā Baseline ā ForecasterEā¦ ā N/A ā 91.6687 ā 1.5037 ā ā ā ā (seasonal ā ā ā ā ā ā ā ā naive) ā ā ā ā ā āāāāāāāāā¼āāāāāāā¼āāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā¼āāāāāāāāāāāāāā¼āāāāāāāāāāāāāāā¤ ā 5 ā 6 ā ridge_baselā¦ ā ForecasterRā¦ ā Ridge ā 93.1456 ā 1.5192 ā āāāāāāāāā“āāāāāāā“āāāāāāāāāāāāāāā“āāāāāāāāāāāāāāā“āāāāāāāāāāāāāāā“āāāāāāāāāāāāāā“āāāāāāāāāāāāāāā
Reproducible codeĀ¶
Every forecast or backtest workflow exposes the standalone skforecast script used to produce its results. This script is strictly deterministic, ensuring identical outputs for a given set of inputs and configurations. You can access the code at any time via the show_code() method.
Free-form Q&AĀ¶
The ask() method is not limited to interpreting workflow objects. Without any context attached, it works as a general forecasting knowledge assistant, useful for clarifying methodology, choosing between approaches, or understanding metric trade-offs.
Without a context there is no task type to route on, so only the general choosing-a-forecaster skill is sent unless the question mentions a topic (intervals, backtesting, metrics, ...) that pulls in a specific one. Pass skills= to pick them yourself; the names sent are recorded in answer.skills. See Skills for the LLM.
InĀ [44]:
Copied!
# Ask a general forecasting question (no data or result required)
# ==============================================================================
answer = assistant.ask(
prompt = (
"For hourly demand with strong daily and weekly seasonality, when should "
"I prefer a direct forecasting strategy over a recursive one?"
)
)
answer.show_explanation()
# Ask a general forecasting question (no data or result required)
# ==============================================================================
answer = assistant.ask(
prompt = (
"For hourly demand with strong daily and weekly seasonality, when should "
"I prefer a direct forecasting strategy over a recursive one?"
)
)
answer.show_explanation()
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā Assistant Response āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā® ā ā ā You should prefer a direct forecasting strategy (ForecasterDirect) when the ā ā predictive relationship between inputs and hourly demand changes substantially ā ā across the forecast horizon, such as when the next few hours rely on recent momentum ā ā while later hours depend primarily on time-of-day or day-of-week patterns. It is ā ā also advantageous when recursive forecasting suffers from noticeable error ā ā propagation over longer multi-step horizons, provided your forecast horizon is fixed ā ā in advance. For most standard setups, however, ForecasterRecursive remains the ā ā primary starting recommendation because it trains only a single model. ā ā ā ā When to Choose Direct Forecasting ā ā ā ā ā¢ Step-specific relationships: In hourly demand with dual seasonality, predicting 1 ā ā hour ahead relies heavily on immediate autocorrelation, whereas predicting 24 or ā ā 168 hours ahead depends on structural seasonal patterns. ForecasterDirect trains ā ā an independent estimator for each step, allowing each model to learn ā ā step-specific dynamics. ā ā ā¢ Mitigating error propagation: ForecasterRecursive uses predicted values from ā ā earlier steps as lag inputs for subsequent steps, which can compound errors ā ā across long horizons. Direct models condition directly on observed historical ā ā data, eliminating this feedback loop. ā ā ā¢ Predefined forecast horizons: If your production system always generates ā ā forecasts for a fixed horizon (such as a standard 24-hour day-ahead demand ā ā forecast), the fixed steps requirement of ForecasterDirect is not a limitation. ā ā ā ā Trade-offs to Keep in Mind ā ā ā ā ā¢ Computational cost: Direct forecasting trains N separate models for an N-step ā ā horizon (for example, 24 models for 24 hours). This requires more memory and ā ā training time, though training can be parallelized using the n_jobs parameter. ā ā ā¢ Inflexible horizon: ForecasterDirect requires setting steps at initialization and ā ā cannot forecast beyond that specified horizon at prediction time, unlike ā ā ForecasterRecursive which can forecast any number of steps on demand. ā ā ā ā°āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāÆ
SummaryĀ¶
This tutorial covered the step-by-step path of skforecast-ai. Here is a recap of what each stage does and when to use it:
| Step | Method | When to use |
|---|---|---|
| 1. Profile | profile() |
Always: produces the ForecastingProfile required by all downstream methods. |
| 2. Plan | plan() |
Always: converts the profile into an executable configuration. |
| 3. Refine plan | refine_plan() |
Optional: use when you want to override specific decisions (deterministic) or inject domain knowledge (LLM). Always evaluate the result. |
| 4a. Forecast | forecast() |
When you want future predictions or a held-out evaluation in a single execution. |
| 4a. Code only | forecast_code() |
When you want to preview or export the script without running it. |
| 4b. CV strategy | create_cv() |
When you want the assistant to derive or translate a TimeSeriesFold for you. |
| 4b. Backtest | backtest() |
When you want to evaluate the model over multiple historical folds. |
| 4b. Code only | backtest_code() |
When you want to preview or export the backtesting script without running it. |
| 4c. Compare | compare() |
When you want to rank several configurations under an identical cross-validation strategy and reuse the winner. |
| Any time | ask() |
When you want an LLM explanation of any intermediate object or result, or a general forecasting Q&A. |
The key advantage of this path is that the profile and plan are built once and reused across both the forecast and backtest branches. This avoids redundant profiling and ensures that both branches use the same modeling configuration. The same profile can also be handed to compare(), so every candidate is ranked against the very same data profile.
For a faster alternative that runs the entire pipeline in a single call, see the fast-path tutorial. For a comprehensive overview of backtesting mechanics, see the skforecast backtesting user guide.