What Systematic Trading Taught Me About Trusting AI Output
Systematic trading forces a specific habit: never trust a strategy's in-sample performance, always demand out-of-sample validation, and treat a good backtest as a hypothesis, not a conclusion. Applied to AI systems, that means a model's confidence score or a strong eval result on your existing dataset is exactly as trustworthy as a backtest — informative, necessary, and not sufficient on its own to bet real usage on.
Before building AI systems, my background was in systematic trading — strategies that make decisions from rules and signals, not gut calls. The single habit that discipline leaves behind, more than any specific technique, is a deep, almost reflexive suspicion of anything that looks good only on the data you already have.
A backtest is a hypothesis, not a result
In systematic trading, a strategy that performs well on historical data is the starting point of the work, not the end of it. In-sample performance is cheap to produce and easy to fool yourself with — enough free parameters and any strategy looks great on the data it was tuned against. The real question is always how it performs on data it never saw, under conditions that weren't part of the tuning process. A strategy that only works in-sample isn't a strategy, it's an overfit curve with a good story attached.
An eval score is the same shape of claim
A model that scores well on your existing eval dataset is making the same kind of claim a backtest makes: it worked on the data you've already seen. That's necessary information and it's not nothing — but it says very little about how the model behaves on the inputs it hasn't encountered, especially the adversarial or unusual ones that real users reliably produce and curated eval sets reliably under-represent. Treating a strong eval score as "done" is the same mistake as treating a strong backtest as "ready to trade live" — both skip the step that actually matters.
What "out-of-sample" means for an AI system
- A held-out dataset the model (and the prompt engineer) never optimized against, refreshed periodically so it doesn't quietly become in-sample by accumulation.
- Shadow deployment against real traffic before a change fully ships — the AI equivalent of paper trading a strategy before risking capital on it.
- Explicit tracking of production performance against the eval-time expectation, so drift between "how it scored" and "how it actually behaves" gets caught instead of assumed away.
Confidence scores deserve the same skepticism as a Sharpe ratio
A high Sharpe ratio on a backtest tells you the strategy was profitable and relatively smooth over that specific historical window — it doesn't tell you the regime that produced those returns will persist. A model's stated confidence score has an analogous limit: it's a number the model produces about itself, shaped by its training distribution, and it degrades exactly where you need it most — on inputs unlike anything it was calibrated against. Treating a high confidence score as permission to skip verification is the same error as treating a good backtest as permission to skip risk management.
The discipline transfers cleanly
Position sizing has a direct analogue in how much autonomy an AI system gets before a human reviews its output — start small, expand as out-of-sample evidence accumulates, never all at once on a strong in-sample result alone. Signal decay has a direct analogue in model and prompt drift — the fact that something worked well six months ago is evidence about six months ago, not a guarantee about today. None of this is really about trading or about AI specifically. It's the same discipline, applied to two different domains that both punish you for confusing a good story on past data with a guarantee about the future.
Have a similar challenge?
Book a free 30-minute architecture call and we'll tell you honestly whether and how we can help.
Book Free Discovery Call →