What Systematic Trading Taught Me About Trusting AI Output

Before building AI systems, my background was in systematic trading — strategies that make decisions from rules and signals, not gut calls. The single habit that discipline leaves behind, more than any specific technique, is a deep, almost reflexive suspicion of anything that looks good only on the data you already have.

A backtest is a hypothesis, not a result

In systematic trading, a strategy that performs well on historical data is the starting point of the work, not the end of it. In-sample performance is cheap to produce and easy to fool yourself with — enough free parameters and any strategy looks great on the data it was tuned against. The real question is always how it performs on data it never saw, under conditions that weren't part of the tuning process. A strategy that only works in-sample isn't a strategy, it's an overfit curve with a good story attached.

An eval score is the same shape of claim

A model that scores well on your existing eval dataset is making the same kind of claim a backtest makes: it worked on the data you've already seen. That's necessary information and it's not nothing — but it says very little about how the model behaves on the inputs it hasn't encountered, especially the adversarial or unusual ones that real users reliably produce and curated eval sets reliably under-represent. Treating a strong eval score as "done" is the same mistake as treating a strong backtest as "ready to trade live" — both skip the step that actually matters.

What "out-of-sample" means for an AI system

• A held-out dataset the model (and the prompt engineer) never optimized against, refreshed periodically so it doesn't quietly become in-sample by accumulation.
• Shadow deployment against real traffic before a change fully ships — the AI equivalent of paper trading a strategy before risking capital on it.
• Explicit tracking of production performance against the eval-time expectation, so drift between "how it scored" and "how it actually behaves" gets caught instead of assumed away.

Confidence scores deserve the same skepticism as a Sharpe ratio

A high Sharpe ratio on a backtest tells you the strategy was profitable and relatively smooth over that specific historical window — it doesn't tell you the regime that produced those returns will persist. A model's stated confidence score has an analogous limit: it's a number the model produces about itself, shaped by its training distribution, and it degrades exactly where you need it most — on inputs unlike anything it was calibrated against. Treating a high confidence score as permission to skip verification is the same error as treating a good backtest as permission to skip risk management.

Regime change is the failure mode both domains share

A trading strategy that worked beautifully in a low-volatility market can fail the moment the regime shifts — not because the strategy got worse, but because the conditions it was built for stopped holding. An AI system has the exact same vulnerability: a model that performed well against a golden dataset built from last quarter's traffic patterns can degrade sharply when real usage shifts to a new category of question the dataset never anticipated, without anything about the model itself changing. The fix in both domains is the same — don't treat the dataset or the backtest window as a fixed, permanent ground truth. Refresh it against what's actually happening now.

This is the exact failure mode covered in why eval scores don't match production behavior, and the discipline of building a dataset that resists overfitting in the first place is covered in setting up golden datasets for LLM regression testing.

Frequently asked questions

How do you know when an eval dataset has become "in-sample" by accumulation?

If every recent tuning or prompt decision was validated against the same fixed dataset without ever holding out a genuinely untouched slice, that dataset has effectively become in-sample even though nobody explicitly trained on it — the same overfitting-by-familiarity that happens when a trading strategy gets iteratively hand-tuned against one backtest window until it fits that window perfectly and nothing else.

Is shadow deployment worth the added complexity for a small team?

For a high-stakes change — anything touching money, safety, or a core product flow — yes; the cost of a bad rollout usually exceeds the cost of the extra infrastructure. For a low-stakes, easily reversible change, a smaller gradual rollout behind a flag often gives similar protection without the full shadow-deployment setup.

The takeaway

Position sizing has a direct analogue in how much autonomy an AI system gets before a human reviews its output — start small, expand as out-of-sample evidence accumulates, never all at once on a strong in-sample result alone. Signal decay has a direct analogue in model and prompt drift — the fact that something worked well six months ago is evidence about six months ago, not a guarantee about today. None of this is really about trading or about AI specifically. It's the same discipline, applied to two different domains that both punish you for confusing a good story on past data with a guarantee about the future.

Christian Chukwuka

Founder & AI Systems Engineer