robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: test clarification broadening with weak answers

Browse Source
This commit is contained in:
robbond
2026-08-08 07:43:19 +01:00
parent 1d1dceefa3
commit 67f2b084a5
3 changed files with 601 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+6 -4
View File
@@ -38,7 +38,9 @@ Experiment 37 corrected the routing defect from Experiment 36 and tested a cross
Experiment 54X isolated target specificity using three fixed clarification cases under the exact same instruction as Experiment 54S. Case 1 (preference/trade-off versus hard constraint) returned "preferred priority between business growth and risk avoidance" — broadened from the material distinction but usable. Case 2 (upfront versus long-term affordability) preserved the definition boundary. Case 3 (user's available time next month) preserved capacity specificity. The same broadening pattern was reproduced across two tested runs under the same model and configuration, making it a repeatable candidate behaviour rather than a one-off observation. No question generation, answer resolution, Behaviour Selection, graph, or UI integration was attempted. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-target-specificity.test.js for the full experiment and results. Status pending Rob's review.
Experiment 54Y tested whether that specificity loss actually changes downstream clarification in a tested scenario. Source: "I want the business to grow, but I don't want to take on more risk." Fixed answer: "It's a hard constraint. I don't want any increase in risk." Variant A (precise target) generated question asking whether avoiding risk is a hard constraint or preference/trade-off; Variant B (broadened target) generated question asking which to prioritize when growth and risk conflict. Both resolved the same answer with materially equivalent meaning. This confirms broadening did not alter downstream resolution in this scenario; broader safety remains untested. Behaviour Selection, graph, UI, and production integration remained untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-specificity-consequence.test.js for the full experiment and results. Status pending Rob's review.
Experiment 54Y tested whether that specificity loss actually changes downstream clarification in a tested scenario. Source: "I want the business to grow, but I don't want to take on more risk." Fixed answer: "It's a hard constraint. I don't want any increase in risk." Variant A (precise target) generated question asking whether avoiding risk is a hard constraint or preference/trade-off; Variant B (broadened target) generated question asking which to prioritize when growth and risk conflict. Both resolved the same answer with materially equivalent meaning. With the explicit hard-constraint answer used in this test, both target variants converged on materially equivalent resolved meaning. The broader target changed the clarification question but not the resolved meaning for the tested explicit answer; broader safety remains untested. Behaviour Selection, graph, UI, and production integration remained untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-specificity-consequence.test.js for the full experiment and results. Status pending Rob's review.
Experiment 54Z tested whether convergence between precise and broadened targets holds with weaker answers. Source same as 54Y. Two weak answers tested against both fixed variants: (1) "Risk matters more to me" — both variants produced materially equivalent meaning (risk not a hard constraint, but stronger than growth). (2) "I'd normally avoid more risk, but for the right opportunity I might accept some" — variants diverged: Variant A collapsed conditionality into flat preference; Variant B preserved conditional structure and remaining uncertainty. Unexpectedly, the broader target preserved more nuance for the conditional answer. Target broadening has material consequences with weaker answers, but direction is unpredictable. 4 live calls completed. Behaviour Selection, graph, UI, and production integration remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-weak-answer-consequence.test.js for the full experiment and results. Status pending Rob's review.
Experiment 38 tested whether a genuinely cold session (no prior conversation context) can recover the project state from three documents alone. It recovered all capabilities, boundaries, and context-pack selection correctly without loading the full history or source code. All seven knowledge-management criteria confirmed met. One handoff update required: the open item "whether the handoff stays accurate after further advances" was resolved (handoff is accurate). The cold-start test passed.
@@ -150,8 +152,8 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Y. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 54Y)
### Return-to-Work Note (Experiment 54Z)
Experiments 54W/54X reproduced a broader priority framing for preference-versus-hard-constraint ambiguity; 54Y tested whether that specificity loss actually changes downstream clarification. The precise target generated a question asking whether avoiding risk is a hard constraint or trade-off; the broadened target asked which to prioritize when growth and risk conflict. The same fixed answer produced materially equivalent resolved meanings from both variants, so broadening did not matter in this scenario. Broader safety/generalisation remains untested. Behaviour Selection, graph, UI, and production integration remained untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-specificity-consequence.test.js for the full experiment and results. Status pending Rob's review.
54Y showed precise and broadened targets generated different questions but converged under an explicit hard-constraint answer; 54Z tested whether that convergence holds with weaker answers. For "Risk matters more to me," both variants produced materially equivalent meaning (risk not a hard constraint). For the conditional trade-off answer ("I'd normally avoid more risk, but for the right opportunity I might accept some"), the variants diverged: Variant A erased conditionality and forced flat resolution; Variant B preserved conditional structure and remaining uncertainty — unexpectedly, the broader target preserved more nuance here. Target broadening has material consequences with weaker answers, but direction is unpredictable. Broader generalisation remains untested. Behaviour Selection, graph, UI, and production integration remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-weak-answer-consequence.test.js for the full experiment and results. Status pending Rob's review.