robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: test clarification question wording

Browse Source
This commit is contained in:
robbond
2026-08-08 06:26:17 +01:00
parent c8ead0f690
commit 8b1d69279f
3 changed files with 417 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+4 -4
View File
@@ -22,7 +22,7 @@
Experiment 54R tested whether a consequential disagreement actually requires user clarification or can be resolved through evidence. Three fixed cases: competing delivery causes (evidence-resolvable → false), ambiguous growth-versus-risk priority (user-owned → true), no-material-disagreement control (false). All three correct (3/3) in one live inference call per case (~40s total). Across the three tested disagreement patterns, the model did not automatically map disagreement to user clarification. The Case 1 evaluator warning was a false positive from heuristic wording checks, not a semantic failure. No production code changed. Status pending Rob's review.
Experiment 54S tested whether, once clarification is known to be required, the model can identify exactly what the user needs to clarify — three fixed cases: growth-versus-risk priority (true → "preference/trade-off or hard constraint"), evidence-resolvable delivery causes (false → null), ambiguous meaning of "affordable" (true → "upfront cost versus long-term total cost"). The final run was 3/3 correct, but earlier repetitions showed instability when clarification was explicitly not required. Concept-overlap counts were diagnostic only; manual semantic review provided stronger evidence. Case 2 instability is an observed behaviour, not merely a test warning. Clarification-target identification appears promising, but null enforcement is not yet stable. Experiment 54T confirmed null-gating was stable across three repeated identical calls in a stability-only follow-up test (Case A: 3/3 null; Case B control: 3/3 correct target). Same host/model; no production code changed. Status pending Rob's review.
Experiment 54S tested whether, once clarification is known to be required, the model can identify exactly what the user needs to clarify — three fixed cases: growth-versus-risk priority (true → "preference/trade-off or hard constraint"), evidence-resolvable delivery causes (false → null), ambiguous meaning of "affordable" (true → "upfront cost versus long-term total cost"). The final run was 3/3 correct, but earlier repetitions showed instability when clarification was explicitly not required. Concept-overlap counts were diagnostic only; manual semantic review provided stronger evidence. Case 2 instability is an observed behaviour, not merely a test warning. Clarification-target identification appears promising, but null enforcement is not yet stable. Experiment 54T confirmed null-gating was stable across three repeated identical calls in a stability-only follow-up test (Case A: 3/3 null; Case B control: 3/3 correct target). The current instruction and output contract produced stable null behaviour across the three repeated false-case runs tested there; broader stability remains unproven. Experiment 54U tested whether a fixed clarification target can survive into one neutral user-facing question without adding meaning (preference/constraint, affordability definition, private factual capacity). All three cases returned correct single neutral questions with no introduced assumptions or evidence requests. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review.
- `docs/task-context-packs.md` chooses the minimum context documents for each work type.
@@ -144,8 +144,8 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54T. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54U. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 54T)
### Return-to-Work Note (Experiment 54U)
54S correctly identified clarification targets in its final run but earlier false/null behaviour was unstable across unrecorded repetitions. 54T repeated one `requiresUserClarification: false` case and one true control three times each to test stability. False consistently returned null (3/3). True consistently returned the intended semantic target (3/3). No asymmetric instability was observed in this experiment. No prompt correction or production integration was attempted. Behaviour Selection remained untouched. Same host/model (qwen-claude:latest on 192.168.1.111:11434); no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-clarification-null-stability.test.js`.
54T found stable false/null and true/target behaviour across the tested repeats; broader stability beyond those three repetitions remains unproven. 54U tested the next user-facing step: turning a fixed clarification target into one concise neutral question. Results: preference/constraint case asked only about the boundary-vs-tradeoff distinction, affordability definition case stayed within "affordable" meaning, private factual capacity case correctly asked for self-reported availability. No question broadened the target or introduced assumptions. When-to-ask, answer handling, Behaviour Selection, and UI integration remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-clarification-question.test.js`.