robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: separate stated clarification meaning from inference

Browse Source
This commit is contained in:
robbond
2026-08-08 08:57:59 +01:00
parent 3af623a7d5
commit fcb7218407
3 changed files with 588 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+6 -4
View File
@@ -46,7 +46,9 @@ Experiment 55A isolated the answer-resolution step using one fixed target and fo
Experiment 55B separated answer meaning from target-resolution judgement using independent calls per case. Three fixed answers tested (weak priority, conditional trade-off, non-answer) through two modes each: Mode A (meaning-only, no resolution decision) and Mode B (resolution via the same 54V/55A instruction). Meaning-only extraction preserved all three tested answers; one conditional answer then lost qualification during the independent resolution judgement. Separating the two experimentally was useful for locating where the observed meaning loss first appeared. Additionally, Case 1 (weak priority) resolved correctly in 55B but over-resolved in 55A — this does not establish that the weak-priority problem is solved; it indicates run-to-run variation. 6 live calls completed at ~104s total. No production code changed. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-answer-meaning-vs-resolution.test.js for the full experiment and results. Status pending Rob's review.
Experiment 55C chained actual preserved meaning from Stage 1 into Stage 2 resolution, testing whether carrying semantic state forward removes the conditionality loss observed in 55B. Three cases tested (weak priority, conditional trade-off, non-answer) through two stages each = 6 live calls at ~117s total. Case 2 conditional qualification survived through both stages and resolved correctly (targetResolved=true with condition retained). Case 3 non-answer uncertainty preserved through both stages. Case 1 over-resolved in Stage 2 because Stage 1 itself strengthened "risk matters more" into language about "preference/trade-off rather than absolute constraint." Compared to 55B, the weak-priority case did not remain honestly unresolved — Stage 1 distorted it first, then Stage 2 resolved that distortion. No two-stage design is proven superior; meaning can be lost at either stage. Graph, Behaviour Selection, UI and production remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-preserved-meaning-resolution.test.js for the full experiment and results. Status pending Rob's review.
Experiment 55C chained actual preserved meaning from Stage 1 into Stage 2 resolution, testing whether carrying semantic state forward removes the conditionality loss observed in 55B. Three cases tested (weak priority, conditional trade-off, non-answer) through two stages each = 6 live calls at ~117s total. Case 2 conditional qualification survived through both stages and resolved correctly (targetResolved=true with condition retained). Case 3 non-answer uncertainty preserved through both stages. Case 1 over-resolved in Stage 2 because Stage 1 itself strengthened "risk matters more" into language about "preference/trade-off rather than absolute constraint." Compared to 55B, the weak-priority case did not remain honestly unresolved — If Stage 1 distorts the answer, Stage 2 may preserve and act on that distortion rather than correct it. No two-stage design is proven superior; meaning can be lost at either stage. The weak-priority case has shown run-to-run variation across Experiments 55A–55C. Graph, Behaviour Selection, UI and production remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-preserved-meaning-resolution.test.js for the full experiment and results. Status pending Rob's review.
Experiment 55D tested whether a first interpretation step can separate what the user established from what the model might infer, using a single-call two-field output contract (statedMeaning / possibleInference) across four fixed answers: weak priority, conditional trade-off, explicit hard constraint, and non-answer. Four live Ollama calls at http://192.168.1.111:11434 with qwen-claude:latest (~76.7s total). All four cases preserved statedMeaning without strengthening (stated_meaning_preserved: 4/4, strengthened: 0, lost: 0). Case 1's weak-priority answer stayed as relative importance only — direct improvement over 55C where the same answer was strengthened to constraint language. Conditionality survived in Case 2; explicit and uncertain controls stayed clean in Cases 3 and 4. Inference cleanly separated for Cases 1 and 2; unnecessary inferences generated for Cases 3 and 4 (hygiene issue, not leakage). No unsupported meaning leaked into statedMeaning. This does not yet prescribe production architecture. Graph, Behaviour Selection, UI and production remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-stated-vs-inferred.test.js for the full experiment and results. Status pending Rob's review.
Experiment 38 tested whether a genuinely cold session (no prior conversation context) can recover the project state from three documents alone. It recovered all capabilities, boundaries, and context-pack selection correctly without loading the full history or source code. All seven knowledge-management criteria confirmed met. One handoff update required: the open item "whether the handoff stays accurate after further advances" was resolved (handoff is accurate). The cold-start test passed.
@@ -158,11 +160,11 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55C. Branch: `feature/user-workspace-ux-v0.7`. Status pending Rob's review.*
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55D. Branch: `feature/user-workspace-ux-v0.7`. Status pending Rob's review.*
### Return-to-Work Note (Experiment 55C)
### Return-to-Work Note (Experiment 55D)
55B showed meaning-only extraction preserved all three tested answers while one conditional answer lost qualification during independent resolution. 55C chained actual preserved meaning into the resolution step to test whether carrying semantic state forward removes that loss. The weak-priority case did not remain honestly unresolved — Stage 1 strengthened it, then Stage 2 resolved the distortion. The conditional trade-off qualification survived through both stages; non-answer uncertainty survived too. Preserving meaning before resolution changed the conditionality-loss pattern (it was eliminated) but introduced a new one: if Stage 1 distorts, Stage 2 amplifies it. This does not yet justify production architecture changes. Graph, Behaviour Selection, UI and production remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7, commit 22325d5. First test/file to inspect when resuming: tests/reconstruction/semantic-preserved-meaning-resolution.test.js for the full experiment and results. Status pending Rob's review.
55C showed preserved meaning can protect later resolution, but weak-priority meaning was already strengthened in Stage 1. 55D isolated that first interpretation step using a single-call stated-vs-inferred separation with four fixed answers across risk preference cases. Weak priority stayed as relative importance only (direct improvement over 55C's constraint-language strengthening). Conditionality survived through the conditional trade-off case. Explicit and uncertain controls stayed clean — no unsupported meaning leaked into statedMeaning. Stronger implications were kept separate in possibleInference for Cases 1 and 2, though Cases 3 and 4 showed unnecessary inference generation (hygiene issue, not leakage). This does not yet prescribe production architecture. Graph, Behaviour Selection, UI and production remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-stated-vs-inferred.test.js for the full experiment and results. Status pending Rob's review.
### Experiment 55A Summary — Clarification Uncertainty Preservation