robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: test clarification answer resolution

Browse Source
This commit is contained in:
robbond
2026-08-08 06:37:32 +01:00
parent 8b1d69279f
commit 5d1cba80cd
3 changed files with 492 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+6 -4
View File
@@ -22,7 +22,9 @@
Experiment 54R tested whether a consequential disagreement actually requires user clarification or can be resolved through evidence. Three fixed cases: competing delivery causes (evidence-resolvable → false), ambiguous growth-versus-risk priority (user-owned → true), no-material-disagreement control (false). All three correct (3/3) in one live inference call per case (~40s total). Across the three tested disagreement patterns, the model did not automatically map disagreement to user clarification. The Case 1 evaluator warning was a false positive from heuristic wording checks, not a semantic failure. No production code changed. Status pending Rob's review.
Experiment 54S tested whether, once clarification is known to be required, the model can identify exactly what the user needs to clarify — three fixed cases: growth-versus-risk priority (true → "preference/trade-off or hard constraint"), evidence-resolvable delivery causes (false → null), ambiguous meaning of "affordable" (true → "upfront cost versus long-term total cost"). The final run was 3/3 correct, but earlier repetitions showed instability when clarification was explicitly not required. Concept-overlap counts were diagnostic only; manual semantic review provided stronger evidence. Case 2 instability is an observed behaviour, not merely a test warning. Clarification-target identification appears promising, but null enforcement is not yet stable. Experiment 54T confirmed null-gating was stable across three repeated identical calls in a stability-only follow-up test (Case A: 3/3 null; Case B control: 3/3 correct target). The current instruction and output contract produced stable null behaviour across the three repeated false-case runs tested there; broader stability remains unproven. Experiment 54U tested whether a fixed clarification target can survive into one neutral user-facing question without adding meaning (preference/constraint, affordability definition, private factual capacity). All three cases returned correct single neutral questions with no introduced assumptions or evidence requests. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review.
Experiment 54S tested whether, once clarification is known to be required, the model can identify exactly what the user needs to clarify — three fixed cases: growth-versus-risk priority (true → "preference/trade-off or hard constraint"), evidence-resolvable delivery causes (false → null), ambiguous meaning of "affordable" (true → "upfront cost versus long-term total cost"). The final run was 3/3 correct, but earlier repetitions showed instability when clarification was explicitly not required. Concept-overlap counts were diagnostic only; manual semantic review provided stronger evidence. Case 2 instability is an observed behaviour, not merely a test warning. Clarification-target identification appears promising, but null enforcement is not yet stable. Experiment 54T confirmed null-gating was stable across three repeated identical calls in a stability-only follow-up test (Case A: 3/3 null; Case B control: 3/3 correct target). The current instruction and output contract produced stable null behaviour across the three repeated false-case runs tested there; broader stability remains unproven. Experiment 54U tested whether a fixed clarification target can survive into one neutral user-facing question without adding meaning (preference/constraint, affordability definition, private factual capacity). All three cases returned correct single neutral questions with no introduced assumptions or evidence requests. The clarification-target → question step worked cleanly across the three tested targets; broader wording quality and user experience remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review.
Experiment 54V tested whether the user's answer can resolve only that target without rewriting the rest of the source meaning. Three fixed cases: hard constraint resolved (true/null), affordability definition resolved (true/null), incomplete answer preserved (false/uncertainty). All three correct across boundary preservation, no forced interpretations, and no unsupported consequences or new questions generated. Clarification answers resolved only the intended target across all tested cases. Graph updates, next-question choice, Behaviour Selection, and UI remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review.
- `docs/task-context-packs.md` chooses the minimum context documents for each work type.
@@ -144,8 +146,8 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54U. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54V. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 54U)
### Return-to-Work Note (Experiment 54V)
54T found stable false/null and true/target behaviour across the tested repeats; broader stability beyond those three repetitions remains unproven. 54U tested the next user-facing step: turning a fixed clarification target into one concise neutral question. Results: preference/constraint case asked only about the boundary-vs-tradeoff distinction, affordability definition case stayed within "affordable" meaning, private factual capacity case correctly asked for self-reported availability. No question broadened the target or introduced assumptions. When-to-ask, answer handling, Behaviour Selection, and UI integration remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-clarification-question.test.js`.
54U showed a fixed clarification target can become one clean question in three tested cases; the correction clarifies the conclusion covers only those targets, not broader wording quality. 54V tested whether the user's answer resolves only that target without rewriting everything else: hard constraint resolved cleanly (true/null), affordability definition resolved correctly (true/null), and an incomplete answer ("depends on the opportunity") remained unresolved rather than being forced into a stronger category (false/uncertainty). All three cases preserved boundaries. Graph updates, next-question choice, Behaviour Selection, and UI integration remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-clarification-answer-resolution.test.js`.