Feature Flags for Prompts: Shipping LLM Changes Without a Full Deploy

We've made the case before for versioning and testing prompts with the same discipline as code. The natural next step, and the one that's easy to skip, is shipping prompt changes the way mature teams ship code changes too — behind a flag, gradually, reversibly — instead of the more common pattern where a prompt change goes from "passed the eval suite" straight to "live for 100% of traffic" the moment it merges.

Why a passed eval suite still isn't "ship to everyone"

An eval suite, even a good one with a real regression dataset, tests against the scenarios you thought to include. Production traffic reliably includes scenarios you didn't. A prompt change that clears every check you have can still behave badly against the 2% of real traffic that looks nothing like your test set — and the first time you find out is when it's already live for everyone, at once, with no easy way back except another deploy.

What a prompt flag actually needs

• Percentage-based rollout — ship a new prompt version to 5% of traffic first, watch the eval metrics and any production monitoring on that slice specifically, and only widen the rollout once it holds up against real usage, not just the test set.
• Per-tenant or per-cohort override — a specific customer explicitly asking for an early version, or a specific segment you want more signal from before a general rollout, gets the new version without touching the percentage rollout for everyone else.
• An instant kill switch — flipping back to the previous, known-good prompt version is a config change, not a deploy, and doesn't wait on a build pipeline while a bad prompt is live.

What this looks like in practice

prompt_version = flag_client.get_variant(
    flag="support-response-prompt",
    tenant_id=tenant_id,
    default="v12",  # last known-good, used if the flag service itself is unreachable
)
prompt = load_prompt(prompt_version)

The default value matters as much as the flag logic itself — if the flag service is unreachable, falling back to the last known-good version rather than failing the request or defaulting to whatever's newest is what keeps a flagging-system outage from becoming a prompt-quality incident on top of it.

This pairs directly with the CI gate

A prompt change that clears the CI-gated regression suite is a candidate for a percentage rollout, not an automatic full release — the eval suite and the flag system are doing different, complementary jobs. The eval suite answers "does this look better than the baseline against the scenarios we know to test." The flag system answers "how do we find out about the scenarios we don't know to test, without betting the whole prompt on the answer." Skipping either one leaves a real gap the other was covering.

What should actually trigger a rollback

A percentage rollout only helps if something is actually watching the rolled-out slice and knows when to pull back — "we'll keep an eye on it" isn't a rollback trigger. The signals worth wiring to an automatic (or at minimum, immediately alerted) rollback: error and retry rate on the flagged cohort compared to the control group, latency distribution shift, and — where the volume justifies it — a lightweight eval check running specifically against the flagged cohort's real traffic, not just the offline regression suite. A rollout with no defined rollback criteria tends to either roll back too late (after a customer complains) or never gets past the initial small percentage because nobody's confident enough to widen it.

This is the same instrumentation covered in LLM observability in production — a percentage rollout is one of the more concrete reasons to have that instrumentation in place before it's needed, not after. For the CI-gate side this pairs with, see how to add LLM regression testing to a CI/CD pipeline.

Frequently asked questions

What percentage should a new prompt version start at?

There's no universal number — it depends on traffic volume and how quickly you need a meaningful signal. A common starting point is small enough to limit blast radius (single digits of a percent) but large enough that the eval and monitoring signal isn't just noise; low-traffic systems may need a higher starting percentage just to get a usable sample size.

Is a feature flag system overkill for a small team?

The core mechanics — a percentage split, a per-tenant override, and a known-good fallback default — don't require a heavyweight platform. A simple config-driven implementation covers most of the value; the discipline of shipping this way matters more than which specific tool implements it.

The takeaway

Prompt changes carry the same production risk as a meaningful code change and, in a lot of systems, more — a bad prompt can shape every response for every user of a given flow, immediately. Shipping them behind the same rollout mechanics as code — gradual, overridable, instantly reversible — closes the gap between "passed the tests" and "actually safe for 100% of real traffic," and turns a bad prompt release from an incident into a config flip.

Christian Chukwuka

Founder & AI Systems Engineer