A ship gate for LLM quality

Changing a prompt is easy. Knowing you have not quietly broken something else is the hard part, and it is what stops teams from maintaining more than one agent.

I built the evaluation: around 100 reference cases and variations, each with a model answer, chosen and documented by me, with the standard for what counted as a good response. A second model scores generated answers against those model answers. The engineer built the infrastructure and interface that runs it. Then I operated it, running the evaluations and doing the prompt engineering from user feedback.

Because the prompts live in MongoDB rather than in code, changing one is a data operation, and a data operation can be intercepted. So the suite fires automatically on any attempt to change a prompt, and if results get worse, the deployment is blocked. That turns model quality into a release criterion instead of an opinion.

In practice that means prompts get treated like code. Every change is a new version, you see the diff before you commit to it, you run the evaluation from the same screen, and you can restore any earlier version. The orchestrator prompt alone went through 23 of them.

Two more loops sit around it. Support reps mark answers good or bad from a panel, and we went through them together twice a week. And a field in the output contract, kb_gap, fills in only when an escalation happened because the knowledge base did not cover the subject, so every escalation becomes a specific instruction about what to document next.

We built this with the first sub-agent, before adding the others. That order is the point: the gate is what made it safe to grow from one agent to five.