robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: validate unresolved uncertainty after harness repair

Browse Source
This commit is contained in:
robbond
2026-08-09 12:55:10 +01:00
parent c40d8c6d49
commit 23bfe5f756
2 changed files with 115 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+3 -1
View File
@@ -30,6 +30,8 @@ Experiment 56F re-tested Regression A with the canonical live harness after Code
Experiment 56G tested Regression C (non-answer uncertainty: "I'm not really sure.") through the live production path to verify the risk-constraint distinction remains unresolved when the user expresses no position. **BLOCKED - apparatus**. The canonical helper (`tests/graph/live-update-experiment-helper.cjs`) contains a broken dynamic import path (`../lib/graph/orchestrator.js` resolves to `tests/lib/graph/orchestrator.js`, which does not exist — correct path is `../../lib/graph/orchestrator.js`). No live calls were made. Full results in `docs/experiment-56g.md`. Status pending Rob's review.
Experiment 56H re-tested Regression C after harness repair (commit c40d8c6). Result: **PASS - uncertainty preserved**. The LLM did not invent any constraint or preference position from "I'm not really sure." — `userSupportedMeaning` was null. No graph mutation occurred; `n-risk-constraint` remained unknown with value=null. One live call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed. Status pending Rob's review.
Experiment 54S tested whether, once clarification is known to be required, the model can identify exactly what the user needs to clarify — three fixed cases: growth-versus-risk priority (true → "preference/trade-off or hard constraint"), evidence-resolvable delivery causes (false → null), ambiguous meaning of "affordable" (true → "upfront cost versus long-term total cost"). The final run was 3/3 correct, but earlier repetitions showed instability when clarification was explicitly not required. Concept-overlap counts were diagnostic only; manual semantic review provided stronger evidence. Case 2 instability is an observed behaviour, not merely a test warning. Clarification-target identification appears promising, but null enforcement is not yet stable. Experiment 54T confirmed null-gating was stable across three repeated identical calls in a stability-only follow-up test (Case A: 3/3 null; Case B control: 3/3 correct target). The current instruction and output contract produced stable null behaviour across the three repeated false-case runs tested there; broader stability remains unproven. Experiment 54U tested whether a fixed clarification target can survive into one neutral user-facing question without adding meaning (preference/constraint, affordability definition, private factual capacity). All three cases returned correct single neutral questions with no introduced assumptions or evidence requests. The clarification-target → question step worked cleanly across the three tested targets; broader wording quality and user experience remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review.
Experiment 54V tested whether the user's answer can resolve only that target without rewriting the rest of the source meaning. Three fixed cases: hard constraint resolved (true/null), affordability definition resolved (true/null), incomplete answer preserved (false/uncertainty). All three correct across boundary preservation, no forced interpretations, and no unsupported consequences or new questions generated. Clarification answers resolved only the intended target across all tested cases. **The individual clarification steps have each worked in their isolated fixed-case tests; end-to-end behaviour remains untested.** Graph updates, next-question choice, Behaviour Selection, and UI remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review.
@@ -168,7 +170,7 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F, 56D–56G. Branch: `feature/reasoning-fidelity-v0.8`. Status pending Rob's review.*
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F, 56D–56H. Branch: `feature/reasoning-fidelity-v0.8`. Status pending Rob's review.*
### Return-to-Work Note (Experiment 55F)