robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: test clarification uncertainty preservation

Browse Source
This commit is contained in:
robbond
2026-08-08 07:53:30 +01:00
parent 67f2b084a5
commit 8c12931b43
3 changed files with 599 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+6 -2
View File
@@ -152,8 +152,12 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A. Branch: `feature/user-workspace-ux-v0.7`. Status pending Rob's review.*
### Return-to-Work Note (Experiment 54Z)
54Y showed precise and broadened targets generated different questions but converged under an explicit hard-constraint answer; 54Z tested whether that convergence holds with weaker answers. For "Risk matters more to me," both variants produced materially equivalent meaning (risk not a hard constraint). For the conditional trade-off answer ("I'd normally avoid more risk, but for the right opportunity I might accept some"), the variants diverged: Variant A erased conditionality and forced flat resolution; Variant B preserved conditional structure and remaining uncertainty — unexpectedly, the broader target preserved more nuance here. Target broadening has material consequences with weaker answers, but direction is unpredictable. Broader generalisation remains untested. Behaviour Selection, graph, UI, and production integration remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-weak-answer-consequence.test.js for the full experiment and results. Status pending Rob's review.
54Y showed precise and broadened targets generated different questions but converged under an explicit hard-constraint answer; 54Z tested whether that convergence holds with weaker answers. For "Risk matters more to me," both variants produced materially equivalent meaning (risk not a hard constraint). For the conditional trade-off answer ("I'd normally avoid more risk, but for the right opportunity I might accept some"), the variants diverged: Variant A erased conditionality and forced flat resolution; Variant B preserved conditional structure and remaining uncertainty — unexpectedly, the broader target preserved more nuance here. The two tested weaker answers behaved differently: one converged across target variants and one diverged. Neither framing was consistently superior across the two tested weaker answers. Broader generalisation remains untested. Behaviour Selection, graph, UI, and production integration remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-weak-answer-consequence.test.js for the full experiment and results. Status pending Rob's review.
### Experiment 55A Summary — Clarification Uncertainty Preservation
Isolated the answer-resolution step using one fixed target (preference/trade-off or hard constraint) and four answers of different strength: fully explicit, weak priority, conditional trade-off, non-answer. Four live Ollama calls completed at http://192.168.1.111:11434 with qwen-claude:latest (~62s total). Case 1 (explicit hard constraint) resolved correctly. Case 2 (weak priority — "Risk matters more to me.") over-resolved: the model set targetResolved=true and inferred "not a rigid, non-negotiable constraint" — meaning stronger than the user supplied. Case 3 (conditional trade-off) resolved correctly on the target but flattened conditionality into flat "preference or trade-off" language without preserving the conditional qualification ("might accept some"). Case 4 (non-answer) correctly remained unresolved with appropriate remaining uncertainty. One over-resolution across four cases (Case 2). The answer-resolution step appears biased toward resolution for weak priority statements when using qwen-claude:latest with the current semantic instruction. Broader generalisation across other models and answers remains untested. Behaviour Selection, graph, UI, and production integration remain untouched. Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-uncertainty-preservation.test.js for the full experiment and results. Status pending Rob's review.