robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: test decision-relevance category boundary

Browse Source
This commit is contained in:
robbond
2026-08-07 10:02:03 +01:00
parent 34f06f2919
commit 08c8f74bde
3 changed files with 572 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+2 -2
View File
@@ -122,8 +122,8 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38–52D. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 38–52E. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 52D)
Experiment 52D isolated enum normalisation from semantic understanding: five fixed meaning statements (no decision target or question in the input) were mapped to the existing four-category contract via one live model call each. Four of five normalised to the expected enum. The compliance boundary case persisted — the model classified a "supports" relationship as `could_change_decision`, exposing genuine ambiguity between these two categories under the current definitions. The existing contract appears clear enough for a separate normalisation step; the remaining problem lies in category definitions, not semantic understanding or normalisation mechanism. Same Qwen model (`qwen-claude:latest`) and host (`http://192.168.1.111:11434`) were retained throughout. No production behaviour changed. What remains uncertain: whether the `supports_decision` ↔ `could_change_decision` boundary can be clarified without restructuring the contract, and whether the discrepancy holds under repeated runs. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-normalisation.test.js` for results, then `docs/design-evolution-log.md` Experiment 52D section.
Experiment 52E tested the `supports_decision` / `could_change_decision` boundary using three clear cross-domain contrast pairs (Market Entry, Community Event, Hiring Decision). All six cases classified correctly — every blocker mapped to `could_change_decision`, every supporting-evidence statement mapped to `supports_decision`. This implies that Experiment 52D's compliance disagreement was caused by an ambiguous reference statement rather than a broken category contract: when the input explicitly distinguishes blockers from supporting evidence, the existing definitions work cleanly. Same model (`qwen-claude:latest`) and host (`http://192.168.1.111:11434`) retained throughout. No production behaviour changed. What remains uncertain: whether moderately ambiguous phrasing (between fully clear contrast and the earlier compliance case) still produces consistent results. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/graph/decision-relevance-category-boundary.test.js` for test results, then `docs/design-evolution-log.md` Experiment 52E section.