robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: test clarification null stability

Browse Source
This commit is contained in:
robbond
2026-08-08 06:13:43 +01:00
parent 6944f358a9
commit c8ead0f690
3 changed files with 419 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+4 -4
View File
@@ -22,7 +22,7 @@
Experiment 54R tested whether a consequential disagreement actually requires user clarification or can be resolved through evidence. Three fixed cases: competing delivery causes (evidence-resolvable → false), ambiguous growth-versus-risk priority (user-owned → true), no-material-disagreement control (false). All three correct (3/3) in one live inference call per case (~40s total). Across the three tested disagreement patterns, the model did not automatically map disagreement to user clarification. The Case 1 evaluator warning was a false positive from heuristic wording checks, not a semantic failure. No production code changed. Status pending Rob's review.
Experiment 54S tested whether, once clarification is known to be required, the model can identify exactly what the user needs to clarify — three fixed cases: growth-versus-risk priority (true → "preference/trade-off or hard constraint"), evidence-resolvable delivery causes (false → null), ambiguous meaning of "affordable" (true → "upfront cost versus long-term total cost"). 3/3 correctly classified. The model can identify the specific unresolved user-owned distinction without generating a question, choosing a winner, or changing Behaviour Selection. One behavioral note: when requiresUserClarification is false, the model sometimes defaults to producing a target rather than respecting null — but this did not affect any of the three tested cases in this run. Same host/model; no production code changed. Status pending Rob's review.
Experiment 54S tested whether, once clarification is known to be required, the model can identify exactly what the user needs to clarify — three fixed cases: growth-versus-risk priority (true → "preference/trade-off or hard constraint"), evidence-resolvable delivery causes (false → null), ambiguous meaning of "affordable" (true → "upfront cost versus long-term total cost"). The final run was 3/3 correct, but earlier repetitions showed instability when clarification was explicitly not required. Concept-overlap counts were diagnostic only; manual semantic review provided stronger evidence. Case 2 instability is an observed behaviour, not merely a test warning. Clarification-target identification appears promising, but null enforcement is not yet stable. Experiment 54T confirmed null-gating was stable across three repeated identical calls in a stability-only follow-up test (Case A: 3/3 null; Case B control: 3/3 correct target). Same host/model; no production code changed. Status pending Rob's review.
- `docs/task-context-packs.md` chooses the minimum context documents for each work type.
@@ -144,8 +144,8 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54S. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54T. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 54S)
### Return-to-Work Note (Experiment 54T)
54R showed that evidence uncertainty and user-owned ambiguity can be distinguished across three tested patterns. 54S tested whether, once clarification is known to be required, the model can identify exactly what the user needs to clarify — growth-versus-risk priority correctly identified preference/trade-off versus hard constraint; delivery causes correctly returned null (evidence-resolvable); affordable ambiguity correctly identified upfront versus long-term cost meaning. 3/3 clarified-target classification correct. No clarification question was generated. Behaviour Selection remained untouched. Same host/model (qwen-claude:latest on 192.168.1.111:11434); no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-clarification-target.test.js`.
54S correctly identified clarification targets in its final run but earlier false/null behaviour was unstable across unrecorded repetitions. 54T repeated one `requiresUserClarification: false` case and one true control three times each to test stability. False consistently returned null (3/3). True consistently returned the intended semantic target (3/3). No asymmetric instability was observed in this experiment. No prompt correction or production integration was attempted. Behaviour Selection remained untouched. Same host/model (qwen-claude:latest on 192.168.1.111:11434); no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-clarification-null-stability.test.js`.