robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: isolate regression B proposal validation

Browse Source
This commit is contained in:
robbond
2026-08-09 07:55:11 +01:00
parent 0d7ad5775c
commit b2329d8608
2 changed files with 99 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+4
View File
@@ -169,3 +169,7 @@ The first implementation pass against the reasoning refinement requirements is d
### Experiment 55A Summary — Clarification Uncertainty Preservation
Isolated the answer-resolution step using one fixed target (preference/trade-off or hard constraint) and four answers of different strength: fully explicit, weak priority, conditional trade-off, non-answer. Four live Ollama calls completed at http://192.168.1.111:11434 with qwen-claude:latest (~62s total). Case 1 (explicit hard constraint) resolved correctly. Case 2 (weak priority — "Risk matters more to me.") over-resolved: the model set targetResolved=true and inferred "not a rigid, non-negotiable constraint" — meaning stronger than the user supplied. Case 3 (conditional trade-off) resolved correctly on the target but flattened conditionality into flat "preference or trade-off" language without preserving the conditional qualification ("might accept some"). Case 4 (non-answer) correctly remained unresolved with appropriate remaining uncertainty. Two of the four tested answers showed loss of nuance: one was over-resolved and one retained the correct target category while losing conditional qualification. The same over-resolution reproduced with a fixed target, so target broadening is not required for the failure to occur. Broader generalisation across other models and answers remains untested. Behaviour Selection, graph, UI, and production integration remain untouched. Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-uncertainty-preservation.test.js for the full experiment and results. Status pending Rob's review.
### Experiment 56A Summary — Regression B Proposal Validation Enum Mismatch
The first implementation pass added proposal-level `answerMeaning` with a pre-mutation compatibility guard. Deterministic regression tests A-D passed, but live Ollama runs showed Regression B failing at `proposal_validation` before the pre-mutation guard could execute. Experiment 56A traced this to a schema mismatch: Qwen returned `supportCategory: "conditional_qualification"` while the production Zod schema only accepts `conditional_tradeoff` among five values. The value survives normalization unchanged (normalize step handles node kind aliases, not supportCategory). The failure is at Zod validation — a proposal-contract issue, not a guard failure. **Hypothesis confirmed.** No fix was attempted. Branch: `feature/reasoning-fidelity-v0.8`. First file to inspect when resuming: `lib/graph/schema.js` line 165 (Zod enum for supportCategory) or the experiment record at `docs/experiment-56a.md`. Status pending Rob's review.