Failure Analysis
Two distinct root causes, found by reading actual failing responses — not just aggregate numbers.
Root cause 1 — over-triggering refusal
V2's stronger insufficient-information rule caused the model to decline questions the context actually answered, especially when phrased conversationally ("Can you tell me more about it?", "However, there's another issue...").
Root cause 2 — judge sensitivity to bare refusals
Even a correct refusal scored 0.0 faithfulness under the judge's claim-extraction rubric if phrased as a generic, content-free sentence. Restating specifically what the context does/doesn't cover scored as a supported, checkable claim instead.
How V3 addressed them
Two targeted rule changes in prompts/candidate/support_agent.v3.md (see Prompts): a check-before-refusing gate targeting root cause 1, and a "restate specifically what's missing" requirement targeting root cause 2 — while preserving the literal phrase the deterministic abstention detector needs.
Root cause 2 was fully resolved — no V3 refusal, correct or not, ever scored a bare 0.0 again.
Root cause 1 was only partially fixed. The residual is exactly what the holdout regression below surfaced.
What the holdout revealed
REVIEWV3 vs V1, sealed 40-case holdout
Answer correctness delta: -7.53 pp — 95% CI [-17.7, -1.2] pp, excluding zero. Inspecting the actual failing responses traced this to the same over-refusal pattern (root cause 1) already flagged but only partially fixed in dev testing, concentrated more heavily in this particular holdout sample. It is not an unexplained anomaly — it is the residual of a known, partially-fixed cause showing up harder on cases the prompt had never been evaluated against before.