Prompt Engineering Isn't Engineering Until You Version and Test It

Strip away the natural-language wrapper and a prompt is an input-to-output mapping with a contract: given this context, produce output shaped like this, with these properties. That's a function signature. The fact that the implementation is a paragraph of English instead of a block of Python doesn't change what it is — but most teams don't treat it that way. Prompts live in a Notion doc or a chat window, and changes ship because they "felt better" on a handful of manual tests.

Version it like code, because it is code

The cheapest fix: prompt templates belong in the repository, reviewed in a pull request the same way any other change to product logic is reviewed. This alone catches a surprising number of problems — a reviewer reading a prompt diff notices an ambiguous instruction far more often than someone eyeballing a live response. It also gets you a real history: when a client asks why the assistant responded a certain way three weeks ago, the answer is a git blame, not an investigation.

Test it against a persistent dataset, not a vibe check

Every prompt change should run against a dataset before it ships — not the three examples the author happened to think of, but a set that includes the edge cases that have historically caused problems. Build this the way you'd build a regression suite: every bug found in production becomes a permanent test case, so a user phrasing that once broke intent classification can never silently regress again.

Decide your metrics before you look at the output

This is the step people skip most, because it's uncomfortable — it's much easier to look at a result, decide it seems fine, and move on. "Seems fine" isn't a metric, it's a rationalization that's inconsistent between two engineers or even the same engineer on two different days. Decide up front: factual accuracy against a reference, format adherence, presence of forbidden content, tone against a rubric. Some are cheap lexical checks, some need an LLM-as-judge — what matters is picking them before the change, not after.

result = run_eval(prompt_version='v14', dataset=REGRESSION_SET, baseline='v13')
if result.severity >= Severity.CRITICAL:
    sys.exit(1)  # fails the CI check, blocks the merge

What this buys you

Versioned prompts, a persistent regression dataset, and pre-committed metrics together look like an ordinary CI pipeline, because that's what they are. A prompt change becomes a pull request; the pull request runs against the dataset; a regression above your severity threshold fails the check, same as a broken unit test would. None of this is exotic — it's the discipline that's been standard for code for two decades, applied to the part of an LLM product that actually determines whether it works.

Christian Chukwuka

Founder & AI Systems Engineer