Statistics

Every comparison reports a paired mean delta and a paired bootstrap 95% confidence interval (2,000 resamples, fixed seed — identical DB state always produces an identical interval), plus a paired effect size (Cohen's d). A Holm-Bonferroni correction is applied across the metrics tested within each comparison, computed the same way, but none of it feeds back into the release decision — the gates below still read only the raw delta and CI.

Answer correctness — paired delta with 95% CI, every comparison

A bar whose interval crosses zero is statistically indistinguishable from no change.

V2 vs V1 (dev)

HOLD

delta: +0.22 pp

95% CI: [-8.3, 8.8] pp

effect size (Cohen's d): 0.006

V3 vs V1 (dev)

REVIEW

delta: +2.32 pp

95% CI: [-5.1, 10.2] pp

effect size (Cohen's d): 0.079

V3.1 vs V1 (dev)

REVIEW

delta: +7.39 pp

95% CI: [1.3, 14.5] pp

effect size (Cohen's d): n/a (zero variance)

V3.2 vs V1 (dev)

REVIEW

delta: +7.07 pp

95% CI: [0.4, 14.5] pp

effect size (Cohen's d): n/a (zero variance)

V3 vs V2 (dev)

REVIEW

delta: +2.10 pp

95% CI: [-5.4, 9.2] pp

effect size (Cohen's d): 0.072

V3.3 vs V1 (dev)

REVIEW

delta: +9.12 pp

95% CI: [2.5, 17.5] pp

effect size (Cohen's d): 0.306

V3.4 vs V1 (dev)

delta: +7.98 pp

95% CI: [0.8, 15.8] pp

effect size (Cohen's d): 0.274

V3 vs V1 (holdout, sealed)

REVIEW

delta: -7.53 pp

95% CI: [-17.7, -1.2] pp

effect size (Cohen's d): -0.363

V3.4 vs V1 (holdout, sealed)

delta: +10.11 pp

95% CI: [-3.3, 26.7] pp

effect size (Cohen's d): 0.294

Dev vs. holdout: the generalization gap

V3 vs V1, answer correctness

On the 120-case dev set, V3's correctness delta was 2.32 pp with a CI that straddles zero. On the 40-case sealed holdout — cases neither prompt had touched before that one run — the same comparison measured -7.53 pp with a CI that excludes zero. This reversal, not any single number, is the headline result this project exists to surface.

V3.4: statistical honesty, Dev vs. Holdout

Answer correctness, the metric that broke V3

Dev (n=120): delta +7.98 pp, 95% CI [+0.8, +15.8] — excludes zero, a confirmed effect.

Holdout (n=40): delta +10.11 pp, 95% CI [-3.3, +26.7] — straddles zero. The correctness improvement is therefore directionally consistent but NOT independently statistically confirmed on the smaller holdout sample.

Faithfulness: dev CI [-4.6, +13.6] pp, holdout CI [-14.7, +18.4] pp — both straddle zero. Abstention accuracy survives Holm-Bonferroni correction on both splits. Cost survives Holm correction on dev but not on the smaller holdout sample. (Holm significance is verified directly against the persisted comparison's MetricDelta.holm_significant field — not itself part of the ExperimentReport JSON schema, same as the guardrail v1/v2 figures on the Safety page.)

V3.4's quality improvements were directionally consistent across Dev and Holdout. The release decision itself replicated: GO on both datasets.

Real results vs. validation fixtures

VALIDATION FIXTURE

Every number on this page comes from results/reports/*.json — real, already-computed comparisons over actual generator/judge/guardrail responses. V2 through V3.3 were each HELD or REVIEWed for a real reason; V3.4 is the first candidate to pass all 8 gates, on both dev and the sealed holdout (see Release Gate). The synthetic GO fixture on that page predates V3.4's real result and played no part in it. No historical decision shown here was recomputed or reinterpreted to produce a different answer.