Prompt Engineering Isn't Engineering Until You Version and Test It

Strip away the natural-language wrapper and a prompt is an input-to-output mapping with a contract: given this context, produce output shaped like this, with these properties. That's a function signature. The fact that the implementation is a paragraph of English instead of a block of Python doesn't change what it is — but most teams don't treat it that way. Prompts live in a Notion doc or a chat window, and changes ship because they "felt better" on a handful of manual tests.

Version it like code, because it is code

The cheapest fix: prompt templates belong in the repository, reviewed in a pull request the same way any other change to product logic is reviewed. This alone catches a surprising number of problems — a reviewer reading a prompt diff notices an ambiguous instruction far more often than someone eyeballing a live response. It also gets you a real history: when a client asks why the assistant responded a certain way three weeks ago, the answer is a git blame, not an investigation.

Test it against a persistent dataset, not a vibe check

Every prompt change should run against a dataset before it ships — not the three examples the author happened to think of, but a set that includes the edge cases that have historically caused problems. Build this the way you'd build a regression suite: every bug found in production becomes a permanent test case, so a user phrasing that once broke intent classification can never silently regress again.

Decide your metrics before you look at the output

This is the step people skip most, because it's uncomfortable — it's much easier to look at a result, decide it seems fine, and move on. "Seems fine" isn't a metric, it's a rationalization that's inconsistent between two engineers or even the same engineer on two different days. Decide up front: factual accuracy against a reference, format adherence, presence of forbidden content, tone against a rubric. Some are cheap lexical checks, some need an LLM-as-judge — what matters is picking them before the change, not after.

result = run_eval(prompt_version='v14', dataset=REGRESSION_SET, baseline='v13')
if result.severity >= Severity.CRITICAL:
    sys.exit(1)  # fails the CI check, blocks the merge

Version numbers should mean something

A prompt version bump that's just an incrementing counter tells a reviewer nothing about what changed or how risky it is. Treating it more like semantic versioning — a patch-level change for wording/formatting tweaks unlikely to shift behavior, a minor bump for a new instruction or example added, a major bump for anything that changes the task itself — gives a reviewer (and a rollout system) a signal about how cautious to be before it even reads the diff.

Once it's tested, ship it the way you ship code

Passing the regression suite is the gate to merge, not the gate to 100% of production traffic — those are different bars. A prompt change that passed every offline test can still behave differently against the full variety of real traffic, which is the argument for rolling it out gradually behind a flag rather than flipping it on for everyone the moment CI goes green.

For the CI-gating mechanics this depends on, see why we run evals in CI, not notebooks and how to add LLM regression testing to a CI/CD pipeline. For the gradual-rollout step specifically, see feature flags for prompts.

Frequently asked questions

Where should prompts actually live — in the database or in the repo?

In the repo, versioned alongside the code that calls them, is the more defensible default — it gets you code review, git history, and CI integration for free. A database-backed prompt store can make sense once non-engineers need to edit prompts directly, but that convenience trades away the review and audit trail a repo gives you by default.

How is testing a prompt different from testing regular code?

The mechanics are similar — a fixed input set, an expected-property check, a pass/fail gate in CI. The real difference is the grading method: a regular unit test checks exact output, while a prompt often needs a fuzzier check (an LLM-as-judge rubric, a lexical similarity threshold) because the acceptable output space is wider than one exact string.

What this buys you

Versioned prompts, a persistent regression dataset, and pre-committed metrics together look like an ordinary CI pipeline, because that's what they are. A prompt change becomes a pull request; the pull request runs against the dataset; a regression above your severity threshold fails the check, same as a broken unit test would. None of this is exotic — it's the discipline that's been standard for code for two decades, applied to the part of an LLM product that actually determines whether it works.

Christian Chukwuka

Founder & AI Systems Engineer