robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: validate regression B after normalization

Browse Source
This commit is contained in:
robbond
2026-08-09 08:25:59 +01:00
parent 36faf70a08
commit 869afee1ab
2 changed files with 107 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+4
View File
@@ -173,3 +173,7 @@ Isolated the answer-resolution step using one fixed target (preference/trade-off
### Experiment 56A Summary — Regression B Proposal Validation Enum Mismatch
The first implementation pass added proposal-level `answerMeaning` with a pre-mutation compatibility guard. Deterministic regression tests A-D passed, but live Ollama runs showed Regression B failing at `proposal_validation` before the pre-mutation guard could execute. Experiment 56A traced this to a schema mismatch: Qwen returned `supportCategory: "conditional_qualification"` while the production Zod schema only accepts `conditional_tradeoff` among five values. The value survives normalization unchanged (normalize step handles node kind aliases, not supportCategory). The failure is at Zod validation — a proposal-contract issue, not a guard failure. **Hypothesis confirmed.** No fix was attempted. Branch: `feature/reasoning-fidelity-v0.8`. First file to inspect when resuming: `lib/graph/schema.js` line 165 (Zod enum for supportCategory) or the experiment record at `docs/experiment-56a.md`. Status pending Rob's review.
### Experiment 56B Summary — Regression B Live Run After Normalisation
Commit `36faf70` added normalization for `conditional_qualification → conditional_tradeoff`, but a live Regression B run returned a *different* variant: `supportCategory: "conditional_preference"`. The existing normalisation map does not cover this value. Two independent Zod errors occurred: (1) `conditional_preference` not in the supportCategory enum, and (2) `resolutionGuidance` was free-text instead of an enum value. **Run-to-run model variation confirmed** — the same fixed input produced `conditional_qualification` in Ex 56A and `conditional_preference` in Ex 56B. The pre-mutation guard remains unreachable because proposal_validation rejects first. Failure classification: `FAIL — normalization / proposal contract`. Branch: `feature/reasoning-fidelity-v0.8`. File to inspect when resuming: `docs/experiment-56b.md`. Status pending Rob's review.