Prediction

skills/building-models

building-models

Whether a forecast or a repeated decision belongs to a model, validated out of time, and the holdout that measures its impact.

When your agent loads it

Decides whether a forecast a plan rests on, or a repeated, at-volume decision, belongs to a model, which output it needs (a forecast, a ranking, an allocation) and whether targeting needs propensity or uplift, validates the model out of time against the base rate and the naive baseline, and hands the impact claim to an experiment. Use when someone asks for a churn, propensity, LTV, scoring, forecasting, uplift, recommendation or allocation model, when a model's offline accuracy is offered as evidence that something worked, or when deciding who gets an offer, a discount or an intervention.

shell

$ git clone https://github.com/0trm/gallop
$ mkdir -p .claude/skills
$ cp -r gallop/skills/building-models .claude/skills/

Read the source on GitHub↗

The prediction position on the map: a forecast a plan rests on, or a decision made repeatedly, per user or per day, by a system rather than by a person reading a report. A forecast that sets inventory, a ranking that picks who gets the offer, an allocation that splits a budget. The bucket asks what will happen and who gets what, hands back a forecast, a ranking or an allocation with its uncertainty attached, and fails by breaking the moment you intervene, because the model learned a world in which nobody had acted on it yet.

Two rules arrive with every question routed here and leave with it. The model needs volume, history and a trusted measurement floor. And the model claims nothing about impact until an experiment says so: a churn model that predicts beautifully says nothing about whether the campaign works.

The mechanical part is a fixed set of checks on scores from an out-of-time window: lift at the top k over the base rate, calibration, a leakage screen of the features, Qini when a randomised arm exists, and MASE for a forecast. The checks score a fitted model; they do not fit one. Fit with whatever library is at hand.

0 · Is this a model's decision?

Three things have to be true. Routing checked only the first.

A forecast, or a decision made continuously. A forecast of one metric's value in a future window belongs here whether it is made once, for a plan (a hiring plan, a budget, a target), or every period. A ranking or an allocation belongs here only when it is decided continuously. A ship-or-kill call made once needs an effect size, not a forecast: send it back to the causal branch.
The floor holds. The outcome the model predicts is trusted in the metric registry. A model fits noise and reports confidence; it amplifies an instrumentation problem rather than revealing it. If the outcome is provisional or disputed, the work belongs to defining-metrics first.
Volume and history. For a classifier, a few hundred positive outcomes in the out-of-time window alone, and about a thousand in total; for a forecast, several full seasons. Below that the validation cannot separate the model from the base rate at the operating point, so nothing about the model can be believed, and the verdict is not fundable. Say so and stop. Rolling folds do not create outcomes that do not exist, and "build it thin and validate as data accrues" ships an unvalidated model for a year. The useful object at that size is a rule: a threshold on one or two fields, written down with the metric it will be judged on and the volume at which a model would become checkable. A regional squad rarely has the volume to justify anything heavier, and saying so beats shipping a model nobody can validate. A forecast, made once or every period, meets the same bar: below several full seasons, hand back the seasonal naive, or last period's value when no full season exists, with an interval from its past errors, and the history at which a model would become checkable.

1 · What will happen, or who gets what?

Three outputs, and the request names one.

A forecast. A metric's value in a future window. Hands back a point and an interval, conditional on nothing changing.
A ranking. Units ordered by an expected outcome, acted on from the top. Hands back the ordered list and the operating point: how far down the list the decision goes.
An allocation. A fixed resource split across options. Hands back the policy and its expected value, with the forecast or ranking it rests on.

Watch for the causal question in a model's clothing. "Build a churn model so we can see whether the retention emails work" asks two things. The model predicts who churns; the question is whether the emails change that. Prediction rides on correlation and does not care why. The emails were aimed at the people most likely to leave, so the model learns the campaign's targeting and reports it back as risk. Refuse to read impact from a model. Split the request: the model for targeting stays here, the impact goes to designing-experiments. The same refusal covers "the forecast says we beat plan, so the launch worked": a forecast error is not an effect size.

2 · Propensity or uplift

A propensity model ranks by the chance of the outcome. An uplift model ranks by how much the treatment changes that chance. For a decision about who gets an intervention, the second is the question and the first is the usual answer, which targets the wrong people: the sure things who would have stayed anyway and the lost causes who leave regardless sit at the top of a propensity list, and the persuadables sit in the middle. The four kinds of unit and what each model does with them: reference/propensity-vs-uplift.md.

Uplift needs randomised rows: units where the treatment was assigned at random and the outcome logged. Observational rows cannot supply it; a model fit on them learns whoever chose the targeting. If no randomised data exists, the experiment comes first: route to designing-experiments, randomise the offer across the eligible population with exposure logged, and train the uplift model on that experiment's rows. Propensity is still the right model when the question is who will do something rather than whom to treat: fraud, capacity, support routing.

3 · Validate out of time, against the base rate

The order is fixed and the checks come before the number. Mechanics and the leakage taxonomy: reference/validation.md. Every check below, run on a real pricing model: reference/worked-example.md.

Split by time. Fit on rows before a cutoff, score rows after it. Random folds on time-ordered data leak the future into the training set and report an accuracy the model will never see in production. Leave a gap the length of the label horizon.
Screen for leakage. A feature that separates the outcome near-perfectly on its own is usually the outcome, or something logged after it. Every flagged feature is explained before any number below counts.
Beat the base rate at the operating point. The decision acts on the top k. Report precision at k against the base rate, as a lift. A lift of 1.0 is the base rate wearing a model. AUC on its own is not a validation, because no decision acts on the whole ranking.
Calibration, if the scores are used as probabilities. Thresholds and expected values need calibrated scores; a ranking does not. Reliability table, Brier score against the base-rate forecast, expected calibration error.
Uplift: Qini on the randomised rows, against random targeting. Precision at k means nothing for an uplift model.

The number reported reads "at the top 10% the precision is 31% against a base rate of 8%, a lift of 3.9, out of time on July", never "AUC 0.91".

4 · Forecasts and interventions

A forecast is conditional on nothing changing. It has to beat the seasonal naive: at a MASE of 1 or above, last season's value did as well and is the forecast. Over a multi-step horizon, judge it against the naive made at the cutoff and scored on the same dates. Baselines and intervals: reference/forecast-baselines.md.

A launch inside the horizon is the failure mode in its purest form. The model has never seen one and cannot see through it. Hand back two objects: the forecast under no change, and the launch's effect routed to the causal branch as its own question, with a holdout if the plan is still open. A forecast that "includes the launch" is a guess with an interval drawn round it.

5 · The holdout that measures impact

Offline validation says the ranking beats the base rate. It does not say that acting on the ranking moves the metric. That claim belongs to an experiment: hand to designing-experiments with the model as the treatment and the current rule as the control, never no rule at all, randomised at the unit the decision acts on, exposure logged, the metric the model exists to move as the primary. reading-experiments reads it. That result enters the prior store with design set to experiment. The model's lift and AUC never do; an offline metric is not an experiment result.

When the decision runs continuously, keep a permanent holdout: a small slice never scored, so the model's contribution can be read at any time and drift shows up as the gap closing.

6 · Drift and expiry

What ships changes the data. A model in production is scored on the population it changed, and its calibration decays from the day it goes live. Compare production scores with realised outcomes at the label's horizon, on a schedule. The knowledge entry names the expiry event as a retrain trigger: calibration drift past a stated bound, the holdout gap closing, or a product change to a feature's source. A model without a stated expiry is a stale belief automated at volume.

The verdict

One of four, stated with the evidence beside it:

Not a model's decision → route back: a ship-or-kill call made once needs an effect size; an untrusted outcome needs defining-metrics first.
Not fundable at this volume → a rule, written down, with the metric it is judged on and the volume at which a model becomes checkable. Not a thinner model with a warning attached: that is the failure this verdict exists to prevent, and the rule's effect is read by a holdout like any other policy. For a forecast, the rule is the seasonal naive, or last period's value, with an interval from its past errors.
Validated, impact unmeasured → ship behind a holdout; the impact question goes to designing-experiments. A forecast made once, for a plan, ships nothing: hand back the point and the interval under no change, and writing-reports files it as a knowledge entry, with the actual added once the window closes.
Impact measured → writing-reports files the report, the belief and the prior-store record, from the experiment's numbers.

Worked requests

The request as it arrives	Verdict	Why
"Build a churn model so we can see if the retention emails work."	split	The model targets; the emails' effect is an experiment
"Score users by churn risk and send the riskiest 20% the discount."	uplift, not propensity	Sure things and lost causes top a propensity list; randomise the offer first
"Which users get the win-back discount this week?"	ranking	Continuous, at volume; validate out of time, ship behind a holdout
"Forecast Q4 signups. New pricing lands in November."	two objects	A forecast under no change, and the pricing effect as a causal question
"Our churn model has 0.94 AUC on cross-validation. Ship it?"	not validated	Random folds on time-ordered data; rerun out of time, report lift at k
"We have 300 conversions a month. Build a propensity model."	not fundable	Too few outcomes to separate a model from the base rate; hand back a rule
"Build a demand forecast for the regional warehouse."	forecast	Beat the seasonal naive or ship the naive; intervals widen with horizon
"The forecast beat plan by 8%, so the campaign worked."	refuse	A forecast error is not an effect size; route to the causal branch

reference Forecast baselines forecast-baselines.md

Forecast baselines

A forecast is a claim that the future looks like the past in a stated way. The baseline states the simplest such claim, and the model has to beat it.

The seasonal naive

Next period equals the same period one season ago. Weekly data with a yearly cycle forecasts this week from the same week last year; daily data with a weekly cycle forecasts Monday from last Monday. It costs nothing, it is explainable to anyone, and on product metrics it is hard to beat by much.

MASE scales the model's error on the test window by the seasonal naive's error on the training window (Hyndman and Koehler's MASE):

MASE below 1: the model beats the naive by that factor.
MASE at or above 1: the naive did as well. The naive is the forecast, and the model is retired or reworked. This is a verdict, not a tuning note.

Report the naive's error on the test window as well: the naive forecast made at the cutoff, scored on the same dates as the model. That is the baseline the model is actually competing with. On a multi-step horizon the two can disagree. MASE's scale is a one-step error on the training window, and when the test window sits where the series is more volatile (the last days before a sell-out, a peak season), a model can post a MASE above 1 while beating the cutoff naive by a wide margin. Read MASE as the verdict only when the horizon is one step; otherwise the verdict is the model's error divided by the cutoff naive's on the test window. A worked case: worked-example.md.

Intervals

A point forecast is a decision input only with its interval, and the interval has to widen with the horizon. A model whose 90% band is the same width at week 12 as at week 1 is not reporting uncertainty; it is reporting the residual variance of its fit. Check the band on the out-of-time window: roughly nine in ten actuals inside a 90% band, and no fewer. A band that holds 60% of actuals is a 60% band whatever the label says.

Interventions

The forecast is conditional on nothing changing. Every product change inside the horizon breaks that condition, and the model, fit on a past without that change, cannot see through it. Three cases:

The change is in the past and the model was fit across it. The training data mixes two regimes. Either fit on the post-change window only, if it is long enough, or add the change as a level shift and say so.
The change is inside the horizon. Hand back two objects: the forecast under no change, and the change's effect as a causal question routed to designing-experiments or choosing-causal-designs. If the plan is still open, ask for a holdout so the effect can be read properly. Adding the change to the forecast by hand is a guess with an interval drawn round it.
The forecast is the argument for the change. A forecast that beat plan is not evidence that something worked. It says the past was a good guide to the future; it cannot say what caused the difference. Route the impact claim to the causal branch.

Forecasts as inputs to allocation

An allocation policy (inventory, budget, staffing) is only as good as the forecast under it, and its cost is asymmetric: under-forecasting a stock out and over-forecasting a write down are different losses. State the loss on each side before choosing the point in the interval to act on; the median is rarely it. The allocation's own impact, once it runs, is again an experiment or a holdout, never the forecast's accuracy.

reference Propensity versus uplift propensity-vs-uplift.md

Propensity versus uplift

Two models that look alike on a dashboard and target different people.

The four kinds of unit

Cross whether a unit responds without the treatment with whether it responds with it:

	Responds if treated	Does not respond if treated
Responds if untreated	Sure thing	Sleeping dog
Does not respond if untreated	Persuadable	Lost cause

The treatment only earns its cost on persuadables. It is wasted on sure things, wasted on lost causes, and harmful on sleeping dogs, the users a retention email reminds to cancel.

A propensity model estimates the chance of the outcome. Its top decile is full of sure things (about to convert anyway) or, for a churn model, lost causes (leaving whatever you send). Persuadables sit in the middle of the list, where a top-k policy never reaches them.

An uplift model estimates the difference the treatment makes: P(outcome | treated) minus P(outcome | not treated), per unit. Its top decile is the persuadables, and its bottom decile, when negative, is the sleeping dogs, the people the campaign should skip.

What uplift needs

Randomised rows. Every unit in the training data was assigned the treatment or the control at random, and the outcome was logged for both arms. That is the only data in which the arm is independent of everything else about the unit, so the model can learn the difference rather than the targeting.

Observational rows cannot supply it. If marketing sent the offer to whoever looked risky, the treated and untreated units differ in exactly the ways that predict the outcome, and a model fit on them learns marketing's rule. It will confidently reproduce last quarter's targeting.

If no randomised data exists, the experiment comes first. Route to designing-experiments: randomise the offer across the eligible population, log exposure, run long enough for the outcome to mature, and train the uplift model on those rows. The experiment doubles as the impact report of the untargeted campaign, which is the baseline the model later has to beat.

Fitting it

Any of the standard constructions works:

Two models, one per arm, uplift as the difference of their predictions. Simple; the difference of two noisy estimates is noisier still.
One model with the arm as a feature and interactions, uplift as the prediction with the arm flipped.
The class-variable transformation, which turns uplift into a single classification target when the arms are balanced.

The choice matters less than the validation.

Reading the Qini curve

The Qini curve ranks the randomised rows by score, targets the top share, and counts the outcomes gained over what the control rate predicts for that many units. The curve runs from zero to the whole population's incremental outcomes; the diagonal is random targeting. Area above the diagonal is the model's value, and a curve that rises, peaks and falls says the bottom of the list contains sleeping dogs: stop targeting where the curve peaks, not at a round percentage.

Precision at k, AUC and calibration are propensity metrics. Reported for an uplift model they measure the wrong thing.

When propensity is still the right model

When the question is who will do something rather than whom to treat: fraud scoring, demand at a location, support routing, capacity. Nobody intervenes on the unit scored, so the correlation the model learned is the one that holds. The line to draw: if the score decides who receives a treatment meant to change the outcome, the model has to be uplift.

reference Validation validation.md

Validation

What has to be true before an offline number means anything, in the order the skill checks it.

Split by time

Production scores units whose outcome has not happened yet, using a model fit on units whose outcome has. Validation has to look the same: fit on rows before a cutoff, score rows on or after it.

The split takes the row dates and returns a training mask and a test mask. Two details an improvised split gets wrong:

The gap. A unit whose label window straddles the cutoff carries information from after it. Leave a gap the length of the label horizon (in days) between the last training row and the first test row: a 30-day churn label needs a 30-day gap.
Several cutoffs. One cutoff gives one estimate. Three or four rolling cutoffs give a sense of how stable the lift is across seasons, which is the thing that decides whether the model survives the year.

Random k-fold cross-validation on time-ordered data is the most common reason a model that scored 0.9 offline scores 0.6 live. It is not a validation and the skill treats a number from it as unvalidated.

The leakage taxonomy

A feature is leaking when it carries the outcome, or information from after it, into the training set. The leakage screen ranks features by how well each separates the outcome on its own and flags anything above a threshold; a single feature with an AUC of 0.97 has almost never earned it. The kinds, in the order to check:

The outcome under another name. A cancellation reason, a refund flag, a "last active" date computed after the churn window closed.
Recorded after the outcome. Fields written by the process the outcome triggers: a win-back email sent because the user churned, a support ticket opened at cancellation.
Aggregates that include the test window. A 90-day average computed over the whole table before the split, so training rows contain test outcomes.
Duplicates across the split. The same user on both sides, with a near-identical feature row.
A proxy of the label. Not leakage strictly, but a feature the decision can act on only after the outcome, such as "payment failed".

A flagged feature is removed or explained; explained means a sentence saying why it is available at scoring time and not downstream of the outcome.

The operating point

A decision acts on the top k of a ranking. Everything about the model that matters happens there, and AUC, which averages over every possible k, says little about it. Report, at the k the decision uses:

Precision at k: the outcome rate among the targeted units.
The base rate: the outcome rate among everyone.
Lift: precision over base rate. A lift of 1 is a coin toss dressed as a model; a lift of 3 at the top decile on a 8% base rate means the targeted units convert at 24%.
Recall at k: the share of all outcomes the targeted set captures, which bounds how much any policy acting on that set can move the metric.

If k is not yet chosen, report lift at two or three candidate points and let the cost of the treatment choose. Lift always falls as k grows.

Calibration

A ranking needs no calibration. A threshold, an expected value or a budget allocation does: "treat everyone above 30%" is only a decision if 30% means 30%. Check calibration with:

A reliability table: mean score against observed rate, per bin. The bins should sit on the diagonal.
The Brier score, mean squared error of the probabilities, against the Brier score of always forecasting the base rate. The skill score 1 - brier / brier_base is the share of the base-rate error the model removes; at or below zero the model is worse than the base rate.
The Murphy decomposition: reliability (calibration error, lower is better), resolution (how far the bins' observed rates sit from the base rate, higher is better) and uncertainty (the base rate's own variance, fixed). Brier = reliability − resolution + uncertainty.
Expected calibration error, the weighted mean gap between score and observed rate across bins.

Calibration drifts before ranking does. It is the first production check to schedule.

Volume

For a classifier, the out-of-time window needs enough positives that the lift at k has an interval narrower than the lift itself. A working rule: a few hundred positive outcomes in the test window; at the top decile that gives a few dozen targeted positives, the least that separates a lift of 2 from a lift of 1. Below that, the model cannot be told from the base rate and the verdict is not fundable. The hand-back at that volume is a rule on one or two fields, with the metric it is judged on and the volume at which a model becomes checkable.

reference Worked example: a ticket-pricing model worked-example.md

Worked example: a ticket-pricing model

The checks in this skill, run on dynamic-pricing: a demand forecast (one Prophet model per match and seating zone, plus an XGBoost on its residuals) feeding a grid search for the revenue-maximising price. The data is synthetic, so the true demand curve is known, and every number below comes from make evaluate in that repo.

Is this a model's decision?

Yes on all three counts. A price is set per zone, per day, for every match: a decision made continuously. The outcome (tickets sold) comes from the ticketing system, the trusted source. And there are 3,367 zone-days of history across 10 matches, enough for an out-of-time window of 518 rows.

Split by time, screen for leakage

The holdout is the last 14 days before each match; every model is refit on the days before. One feature failed the leakage screen: web_conversion_rate is sales divided by visits, so it is the outcome wearing a ratio. It is dropped. Same-day web traffic passed: replacing it with yesterday's value moves WAPE by 0.8 points.

Beat the naive, not a straw man

The repo's first baseline was the global mean, which every model beats (WAPE 79.7%). The baseline that matters is the forecast anyone could make at the cutoff without a model: each series' average over its last 7 days.

Model	WAPE	MAE vs. cutoff naive	In-sample MASE
Ensemble (Prophet + XGBoost)	26.4%	0.71	2.29
XGBoost alone, on sales	22.6%	0.60	1.96
Prophet alone	41.8%	1.12	3.63
Naive: last 7 days	37.4%	1.00	3.25

Two lessons. Prophet alone loses to the naive, so the per-series stage earns its place only through the residual model. And in-sample MASE puts every model above 1, the cutoff naive included: the test window is the run into the match, where daily sales swing far more than in the early days that set MASE's scale. The verdict comes from the right-hand comparison the forecast-baselines reference asks for, not from MASE.

Validate the decision, not only the forecast

The model's job is a price. Scoring each recommended price against the true demand curve found the optimiser searching up to 2.5x the base price while no training row was priced above 1.91x. Past the data the trees predict flat sales, revenue keeps rising with price, and 42% of recommendations landed at the cap, earning 45% of the best achievable revenue. Restricting the search to each zone's observed 5th to 95th percentile of prices lifted that to 95%. Forecast accuracy did not move; the decision did.

The case the split hides

Splitting by date inside each series never shows the model a match it has not seen, but that is how a new fixture is priced on the day it goes on sale. Leaving one match out at a time: per-series Prophet has no model for it, and the ensemble's WAPE goes to 89.8%. A pooled XGBoost fallback for series without history brings it to 12.5%, with 94% of the best achievable revenue.

What stays open

The impact claim belongs to an experiment, not to these numbers: in production the engine was judged by a holdout of seating zones on static prices. Randomising by zone inside one stadium invites interference (fans move between zones when prices diverge), which is the case the randomisation-unit reference in designing-experiments covers.