robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: test decision-relevance ambiguity handling

Browse Source
This commit is contained in:
robbond
2026-08-07 10:15:40 +01:00
parent 08c8f74bde
commit 9171844f5b
3 changed files with 650 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+3 -3
View File
@@ -122,8 +122,8 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38–52E. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 38–52F. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 52D)
### Return-to-Work Note (Experiment 52F)
Experiment 52E tested the `supports_decision` / `could_change_decision` boundary using three clear cross-domain contrast pairs (Market Entry, Community Event, Hiring Decision). All six cases classified correctly — every blocker mapped to `could_change_decision`, every supporting-evidence statement mapped to `supports_decision`. This implies that Experiment 52D's compliance disagreement was caused by an ambiguous reference statement rather than a broken category contract: when the input explicitly distinguishes blockers from supporting evidence, the existing definitions work cleanly. Same model (`qwen-claude:latest`) and host (`http://192.168.1.111:11434`) retained throughout. No production behaviour changed. What remains uncertain: whether moderately ambiguous phrasing (between fully clear contrast and the earlier compliance case) still produces consistent results. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/graph/decision-relevance-category-boundary.test.js` for test results, then `docs/design-evolution-log.md` Experiment 52E section.
Experiment 52F tested whether ambiguous relationship statements remain ambiguous during normalisation, rather than being forced into stronger categories. Two clear controls (explicit blocker and explicit supporting-evidence) confirmed the existing boundary still works cleanly for unambiguous inputs — both matched expected enums. Cases 3 and 4 (genuinely ambiguous regulatory-position statements) were tested against `cannot_determine`. The model's treatment of these cases — whether it preserved uncertainty or introduced external assumptions about regulation — determines whether the contract can honestly own ambiguity. Same model (`qwen-claude:latest`) and host retained; no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`, commit pending. First file to inspect when resuming: `tests/graph/decision-relevance-ambiguity.test.js` for test results, then `docs/design-evolution-log.md` Experiment 52F section.