Feature Flags for Prompts: Shipping LLM Changes Without a Full Deploy

We've made the case before for versioning and testing prompts with the same discipline as code. The natural next step, and the one that's easy to skip, is shipping prompt changes the way mature teams ship code changes too — behind a flag, gradually, reversibly — instead of the more common pattern where a prompt change goes from "passed the eval suite" straight to "live for 100% of traffic" the moment it merges.

Why a passed eval suite still isn't "ship to everyone"

An eval suite, even a good one with a real regression dataset, tests against the scenarios you thought to include. Production traffic reliably includes scenarios you didn't. A prompt change that clears every check you have can still behave badly against the 2% of real traffic that looks nothing like your test set — and the first time you find out is when it's already live for everyone, at once, with no easy way back except another deploy.

What a prompt flag actually needs

• Percentage-based rollout — ship a new prompt version to 5% of traffic first, watch the eval metrics and any production monitoring on that slice specifically, and only widen the rollout once it holds up against real usage, not just the test set.
• Per-tenant or per-cohort override — a specific customer explicitly asking for an early version, or a specific segment you want more signal from before a general rollout, gets the new version without touching the percentage rollout for everyone else.
• An instant kill switch — flipping back to the previous, known-good prompt version is a config change, not a deploy, and doesn't wait on a build pipeline while a bad prompt is live.

What this looks like in practice

prompt_version = flag_client.get_variant(
    flag="support-response-prompt",
    tenant_id=tenant_id,
    default="v12",  # last known-good, used if the flag service itself is unreachable
)
prompt = load_prompt(prompt_version)

The default value matters as much as the flag logic itself — if the flag service is unreachable, falling back to the last known-good version rather than failing the request or defaulting to whatever's newest is what keeps a flagging-system outage from becoming a prompt-quality incident on top of it.

This pairs directly with the CI gate

A prompt change that clears the CI-gated regression suite is a candidate for a percentage rollout, not an automatic full release — the eval suite and the flag system are doing different, complementary jobs. The eval suite answers "does this look better than the baseline against the scenarios we know to test." The flag system answers "how do we find out about the scenarios we don't know to test, without betting the whole prompt on the answer." Skipping either one leaves a real gap the other was covering.

The takeaway

Prompt changes carry the same production risk as a meaningful code change and, in a lot of systems, more — a bad prompt can shape every response for every user of a given flow, immediately. Shipping them behind the same rollout mechanics as code — gradual, overridable, instantly reversible — closes the gap between "passed the tests" and "actually safe for 100% of real traffic," and turns a bad prompt release from an incident into a config flip.

Christian Chukwuka

Founder & AI Systems Engineer