GenAI ReleaseGate
Can We Safely Ship a Better LLM?
The business problem: before deploying a new prompt, can we prove it's safe, useful, operationally acceptable, and worth shipping — with evidence, not a single benchmark score?
The real release journey
Six real candidates, each decided by the same automated policy engine. The gate actually rejected or reviewed five of them before one passed.
V3.4 — First Candidate to Pass the Release Gate
REAL DATADev — 120 cases
Sealed holdout — 40 cases
Quality (dev)
Correctness +7.98 pp
Faithfulness +4.53 pp
Hallucination 1.69%
Safety
Injection blocking 100%
Sensitive-info protection 100%
Performance
p95 latency vs V1 (dev) -30.6%
“GO” here means passed the release gate — not the same as production deployment: a promotion candidate with statistically-grounded evidence behind it, not a claim that this has been deployed to production. “Cost” above is generator (model inference) cost only — judge and guardrail costs are real, separately tracked, and not part of this figure.
Candidates the gate rejected
V3.4 wasn't the first attempt — it's the fifth. The gate reviewed or held back every one of the others.
faithfulness: delta -0.1077 crossed the hold threshold -0.0800; benign_false_positive_rate: candidate value 0.5556 exceeds the allowed maximum 0.15
cost_per_query: delta +25.79% crossed the review threshold +15.00% (+25.79% cost)
faithfulness: delta -0.0601 crossed the review threshold +0.0000; cost_per_query: delta +17.02% crossed the review threshold +15.00% (+17.02% cost)
faithfulness: delta -0.0160 crossed the review threshold +0.0000; cost_per_query: delta +22.54% crossed the review threshold +15.00% (+22.54% cost)
cost_per_query: delta +16.30% crossed the review threshold +15.00% (+16.30% cost)
+12.17% dev / +10.81% holdout
Cost delta vs. release decision (dev)
The release gate did not simply reward the highest benchmark score — it rejected five candidates until one satisfied the complete release policy. V2's cost delta is shown for completeness; its HOLD was driven by faithfulness and safety, not cost.
Dev → sealed holdout
Development evaluation — 120 cases
GOSealed holdout — 40 cases
GOPromotion evidence
The candidate was not promoted solely because it passed development evaluation. The same prompt was evaluated on a sealed holdout set — cases it had never been run against before — using the identical, unmodified release policy.