Pick baseline
Production reference v21_cr · thinking-off batch · promoted 2026-06-11 PM
The 2 standard misses (= the documented floor composition):
• avtwo_057 — governance-fenced by its own LOCK (wrong-DAU-source value bug, deliberately not relitigated);
routes fast-path since v20 (extractor label coining) — signature changed, same miss.
• avtwo_040 — known LLM-variable compile-route outer-WHERE-clamp defect (fix open by choice).
avtwo_043 (FTD-formula SQL-gen roller, the second documented roller) rolled Equal this cut —
hence floor 37–39/40 depending on the 040/043 rolls.
Why the 66.7% headline is not a correctness ceiling: the 20 non-equal rows =
9 diagnostic (narrative answers — row-set Equal is the wrong lens; tracked by Track-W fan-out 78%) +
5 refusal_expected (proxy references — refusal is the correct behavior) +
4 blocked_infra (references fail by construction — need DB-owner DDL / grants) + the 2 above.
None are wrong answers from the pipeline.
Arms: v16+ cuts are compile-route-only (_cr = production-representative;
no ablation arm). Picker is latest-only per phase (2026-06-12): the named priors v20_cr (its memo
documents the v19 dual-period catch + the 043 roller) and v18_cr (V1 closing reference), plus
v17_cr / v14_cr, all live on disk under the direct-URL retention contract.
Memo: eval/results/baseline_phase3_ev01_ds_v21.md ·
⚠ pipeline_sha does not hash src/orchestrator (carried — live :3010 probes
are the only valid verification for orchestrator-stage changes).
Top-line
By intent
By difficulty
Variance across N runs
Fingerprints
Failure taxonomy
Top-line
Gates (raw)
Pre-Phase-3 EV-03 contract. Under tiered routing (fast_path vs orchestrator), router_accuracy_pass and sql_validated_pass are stale because fast_path rows don't get validator-stamped — read the orch-routed slice in By route, and strict-AST as informational.
By route
Tiered router splits entries between fast_path (Phase-1.5 pipeline, no orchestrator findings) and orchestrator (Phase-3 specialist + Pass-2). Reading: orch-slice sql_validated is the load-bearing signal.
By expected intent
Row-set verdict distribution
Primary correctness metric since v5 (row_set_diff_v1). equal = row-sets match; *_drift = same set, ordering only (semantically equivalent); shape mismatches and exec failures are the load-bearing miss categories.
Per-row breakdown
Fingerprints
Header
Exit gate · summary_quality_v1
Per template · all-four pass-rate
Per report origin · all-four pass-rate
Production-path orchestrator_sy3 is the gated slice; harness_fast_path_synthesis is informational.
EV-04 · Coverage audit
Sub-queries that the decomposer hinted to a specialist (UG/BH) but landed on Financial as fallback (computeCoverageWarnings, SY3-03). Phase-4 spec R-04 trigger: > 15% fallback rate activates the mid-phase roster-expansion review.
Deterministic anchors show
Per-row scores
Per-row summary_quality_v1 dims: ✓ / ✗ / — (— = dim not applicable to the row's template, e.g. simple_aggregate skips actionable/driver_named).
Fingerprints
SP-04 gate
Per domain · non-adversarial slice (SP-04 gated)
SP-04 spec: specialist validator-pass-rate ≥ Financial-only on its own domain. Adversarial entries (gap-detector exercises) excluded from the gate — they're designed to fail Pass-1 validation and emit findings for Pass-2.
Adversarial slice (informational)
Stress-test entries that exercise gap-detection + Pass-2; not gated. Watching: 100% spec validator-pass under DS can mask semantic misses (clean SQL shape doesn't fire Pass-2). See GD-01 follow-up.