robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: test resolution from preserved answer meaning

Browse Source
This commit is contained in:
robbond
2026-08-08 08:42:28 +01:00
parent 22325d5ac3
commit 3af623a7d5
3 changed files with 732 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+6 -4
View File
@@ -44,7 +44,9 @@ Experiment 54Z tested whether convergence between precise and broadened targets
Experiment 55A isolated the answer-resolution step using one fixed target and four answers of varying strength (explicit hard constraint, weak priority, conditional trade-off, non-answer). Two of the four tested answers showed loss of nuance: one was over-resolved (weak priority set targetResolved=true with inferred "not a constraint" meaning) and one retained the correct target category while losing conditional qualification ("might accept some for the right opportunity" became "preference or trade-off rather than a hard constraint"). The same over-resolution reproduced with a fixed target, so target broadening is not required for the failure to occur. 4 live calls completed at ~62s total. The answer-resolution step appears biased toward resolution for weak priority statements. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-uncertainty-preservation.test.js for the full experiment and results. Status pending Rob's review.
Experiment 55B separated answer meaning from target-resolution judgement using independent calls per case. Three fixed answers tested (weak priority, conditional trade-off, non-answer) through two modes each: Mode A (meaning-only, no resolution decision) and Mode B (resolution via the same 54V/55A instruction). All three meanings were preserved accurately in Mode A (3/3 meaning_preserved). The only information loss appeared during resolution: Case 2's conditional qualification ("for the right opportunity I might accept some") was preserved by Mode A but flattened to a flat "preference or trade-off" statement during resolution judgement. This supports separating meaning extraction from resolution judgement in production, though further testing is needed before specific implementation changes. Additionally, Case 1 (weak priority) resolved correctly in 55B but over-resolved in 55A — suggesting the resolution step may exhibit run-to-run instability under the same configuration. 6 live calls completed at ~104s total. No production code changed. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-answer-meaning-vs-resolution.test.js for the full experiment and results. Status pending Rob's review.
Experiment 55B separated answer meaning from target-resolution judgement using independent calls per case. Three fixed answers tested (weak priority, conditional trade-off, non-answer) through two modes each: Mode A (meaning-only, no resolution decision) and Mode B (resolution via the same 54V/55A instruction). Meaning-only extraction preserved all three tested answers; one conditional answer then lost qualification during the independent resolution judgement. Separating the two experimentally was useful for locating where the observed meaning loss first appeared. Additionally, Case 1 (weak priority) resolved correctly in 55B but over-resolved in 55A — this does not establish that the weak-priority problem is solved; it indicates run-to-run variation. 6 live calls completed at ~104s total. No production code changed. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-answer-meaning-vs-resolution.test.js for the full experiment and results. Status pending Rob's review.
Experiment 55C chained actual preserved meaning from Stage 1 into Stage 2 resolution, testing whether carrying semantic state forward removes the conditionality loss observed in 55B. Three cases tested (weak priority, conditional trade-off, non-answer) through two stages each = 6 live calls at ~117s total. Case 2 conditional qualification survived through both stages and resolved correctly (targetResolved=true with condition retained). Case 3 non-answer uncertainty preserved through both stages. Case 1 over-resolved in Stage 2 because Stage 1 itself strengthened "risk matters more" into language about "preference/trade-off rather than absolute constraint." Compared to 55B, the weak-priority case did not remain honestly unresolved — Stage 1 distorted it first, then Stage 2 resolved that distortion. No two-stage design is proven superior; meaning can be lost at either stage. Graph, Behaviour Selection, UI and production remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-preserved-meaning-resolution.test.js for the full experiment and results. Status pending Rob's review.
Experiment 38 tested whether a genuinely cold session (no prior conversation context) can recover the project state from three documents alone. It recovered all capabilities, boundaries, and context-pack selection correctly without loading the full history or source code. All seven knowledge-management criteria confirmed met. One handoff update required: the open item "whether the handoff stays accurate after further advances" was resolved (handoff is accurate). The cold-start test passed.
@@ -156,11 +158,11 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55B. Branch: `feature/user-workspace-ux-v0.7`. Status pending Rob's review.*
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55C. Branch: `feature/user-workspace-ux-v0.7`. Status pending Rob's review.*
### Return-to-Work Note (Experiment 55B)
### Return-to-Work Note (Experiment 55C)
55A showed one weak answer was over-resolved and one conditional answer lost nuance; 55B separated answer meaning from target-resolution judgement using independent calls. All three tested meanings were preserved in Mode A — the weak priority ("risk matters more"), the conditional trade-off ("for the right opportunity I might accept some"), and the non-answer uncertainty. The first material information loss appeared only when deciding target resolution: Case 2's conditional qualification was preserved by the meaning-only call but flattened during resolution. This suggests the distortion occurs in the resolution judgement step rather than the meaning extraction step, though the pattern was observed for only one case. Whether other answer types show the same pattern remains unproven. Graph, Behaviour Selection, UI and production remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7, commit 8c12931. First test/file to inspect when resuming: tests/reconstruction/semantic-answer-meaning-vs-resolution.test.js for the full experiment and results. Status pending Rob's review.
55B showed meaning-only extraction preserved all three tested answers while one conditional answer lost qualification during independent resolution. 55C chained actual preserved meaning into the resolution step to test whether carrying semantic state forward removes that loss. The weak-priority case did not remain honestly unresolved — Stage 1 strengthened it, then Stage 2 resolved the distortion. The conditional trade-off qualification survived through both stages; non-answer uncertainty survived too. Preserving meaning before resolution changed the conditionality-loss pattern (it was eliminated) but introduced a new one: if Stage 1 distorts, Stage 2 amplifies it. This does not yet justify production architecture changes. Graph, Behaviour Selection, UI and production remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7, commit 22325d5. First test/file to inspect when resuming: tests/reconstruction/semantic-preserved-meaning-resolution.test.js for the full experiment and results. Status pending Rob's review.
### Experiment 55A Summary — Clarification Uncertainty Preservation
docs/design-evolution-log.md
+233 -3
View File
@@ -9697,13 +9697,13 @@ Correctly unresolved. No meaning distortion detected in either mode.
### Experiment Conclusion
**Answer meaning remained accurate while resolution judgement introduced the observed loss.**
**Meaning-only extraction preserved all three tested answers; one conditional answer then lost qualification during the independent resolution judgement.**
All three tested answers preserved their meaning correctly in Mode A (meaning_preserved: 3/3). The only information loss appeared in Case 2 when transitioning from meaning-only to resolution: the conditional qualification "for the right opportunity I might accept some" was present and preserved by Mode A, then flattened to a flat "preference or trade-off" statement during resolution.
Additionally, Case 1 produced different resolution outcomes across experiments (55A over-resolved; 55B correctly unresolved), suggesting the resolution step exhibits some run-to-run instability under the same configuration — an observation worth monitoring but not yet actionable without more data.
The evidence supports separating meaning extraction from resolution judgement as a production strategy, but does not yet establish specific implementation details.
Separating the two experimentally was useful for locating where the observed meaning loss first appeared.
### Focused Test Result
@@ -9746,6 +9746,236 @@ This experiment created one new test file only. No meaning-resolution separation
---
### Return-to-Work Note (Experiment 55B)
## Experiment 55C — Resolution From Preserved Answer Meaning (2026-08-08)
### Objective
Experiment 55B showed meaning-only extraction preserved all three tested answers while one conditional answer lost qualification during independent resolution. The remaining question: if the preserved answer meaning is explicitly carried into the resolution step, does the later judgement still flatten it?
This tests whether carrying semantic state forward across two calls eliminates the conditionality loss, introduces a new failure mode, or changes neither.
### Hypothesis
A two-step test-only chain may preserve meaning better because the resolution stage no longer needs to reinterpret the raw answer. Possible outcomes: preserved meaning survives resolution; the resolution stage still flattens it; some answers improve while others do not. Any outcome is useful.
### Configuration
Host: `http://192.168.1.111:11434` (same as all prior experiments)
Model: `qwen-claude:latest` (same as all prior experiments)
### Number of Live Inference Calls
Exactly **6** live Ollama calls — one meaning call and one resolution call per case, three cases. Stage 2 uses the actual answerMeaning from Stage 1, not a human reference.
### Fixed Source / Target / Question (identical to 55A/55B)
- **Source:** "I want the business to grow, but I don't want to take on more risk."
- **Clarification target:** whether avoiding additional risk is a preference/trade-off or a hard constraint.
- **Clarification question:** Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?
### Stage 1 instruction (unchanged from 55B Mode A)
> State only what the user's answer establishes in relation to the clarification question. Preserve uncertainty, conditionality, and qualification exactly as supplied. Do not decide whether the clarification target is resolved. Do not infer what the user did not say. Do not recommend action or generate another question.
>
> Return valid JSON only in this shape: { "answerMeaning": "short statement" }
### Stage 1 output contract
```json
{
"answerMeaning": "short statement"
}
```
### Stage 2 instruction (new — uses preserved meaning, not raw answer)
> Decide whether the supplied preserved answer meaning settles the clarification target. Treat the preserved meaning as the full extent of what has been established — do not strengthen, simplify, or reinterpret it. Mark targetResolved true only when that preserved meaning settles the target. If any part of the target remains unresolved, preserve that uncertainty. Keep qualifications and conditions intact.
>
> Return valid JSON only in this shape: { "resolvedMeaning": "short statement", "targetResolved": true, "remainingUncertainty": null }
### Stage 2 output contract
```json
{
"resolvedMeaning": "short statement",
"targetResolved": true,
"remainingUncertainty": null
}
```
---
### Case 1 — Weak Priority
**Raw answer:** "Risk matters more to me."
**Stage 1 result:**
- answerMeaning: "The user indicates that risk carries greater weight, suggesting it leans toward being a significant preference or trade-off rather than explicitly confirming an absolute hard constraint."
- Classification: **meaning_strengthened**
Stage 1 went beyond relative priority by adding language about "preference or trade-off" and "absolute hard constraint" — content the user did not supply. This was also seen in 55A's Stage 2, but now it appears at Stage 1.
**Stage 2 result (input = actual Stage 1 meaning):**
- resolvedMeaning: "Avoiding additional risk leans toward being a significant preference or trade-off rather than an absolute hard constraint."
- targetResolved: true
- remainingUncertainty: null
- Classification: **resolution_overresolved**
Because Stage 1 already strengthened the answer, Stage 2 treated that strengthened meaning as input and over-resolved with `targetResolved=true` and no remaining uncertainty. This is a chaining failure: Stage 1 created a problem that Stage 2 amplified.
---
### Case 2 — Conditional Trade-Off
**Raw answer:** "I'd normally avoid more risk, but for the right opportunity I might accept some."
**Stage 1 result:**
- answerMeaning: "Avoiding additional risk is treated as a normal preference that may be conditionally accepted for the right opportunity."
- Classification: **meaning_preserved**
Both the normal preference and conditional exception were preserved. The wording uses "treated as" rather than directly repeating the user's phrasing but captures the full semantic content.
**Stage 2 result (input = actual Stage 1 meaning):**
- resolvedMeaning: "Avoiding additional risk is a preference rather than a hard constraint, as it may be conditionally accepted for the right opportunity."
- targetResolved: true
- remainingUncertainty: null
- Classification: **resolution_correct**
Target was correctly resolved with the conditional qualification ("as it may be conditionally accepted for the right opportunity") retained in the resolved meaning. This is the key result: the two-stage chain preserved the conditionality that 55B's Mode B flattened.
---
### Case 3 — Non-Answer
**Raw answer:** "I'm not really sure."
**Stage 1 result:**
- answerMeaning: "The user states they are not really sure whether avoiding additional risk is a hard constraint or a preference/trade-off."
- Classification: **meaning_preserved**
Uncertainty preserved accurately. The model added the clarification target context ("whether...is a hard constraint or a preference/trade-off") which is reasonable contextual framing for the non-answer.
**Stage 2 result (input = actual Stage 1 meaning):**
- resolvedMeaning: "The user is uncertain whether avoiding additional risk is a preference/trade-off or a hard constraint."
- targetResolved: false
- remainingUncertainty: "Whether avoiding additional risk is a preference/trade-off or a hard constraint remains unresolved."
- Classification: **resolution_correct**
Correctly unresolved with appropriate remaining uncertainty. No meaning distortion in either stage.
---
### Evaluation Summary
| Case | Answer | Stage 1 | Stage 1 Class | Stage 2 Result | Stage 2 Class |
|------|--------|---------|--------------|----------------|---------------|
| 1 | "Risk matters more to me." | Strengthened beyond priority | meaning_strengthened | Over-resolved (targetResolved=true) | resolution_overresolved |
| 2 | "I'd normally avoid more risk, but for the right opportunity I might accept some." | Preserved conditionality | meaning_preserved | Resolved with qualification retained | resolution_correct |
| 3 | "I'm not really sure." | Preserved uncertainty | meaning_preserved | Correctly unresolved | resolution_correct |
**Meaning counts (Stage 1):**
- meaning_preserved: 2
- meaning_strengthened: 1
- meaning_lost: 0
**Resolution counts (Stage 2):**
- resolution_correct: 2
- resolution_overresolved: 1
- resolution_meaning_loss: 0
- resolution_underresolved: 0
---
### Questions Answered
1. **Did Case 1 Stage 1 preserve only relative priority?** No — it strengthened beyond relative priority by introducing "preference or trade-off" and "absolute hard constraint" language not present in the user's answer.
2. **Did Case 1 Stage 2 remain unresolved?** No — targetResolved=true because the distorted Stage 1 input led the model to conclude resolution was achieved.
3. **Did Case 2 Stage 1 preserve the conditional qualification?** Yes — "normal preference that may be conditionally accepted for the right opportunity" preserved both the default stance and the exception.
4. **Did Case 2 Stage 2 retain that qualification while resolving the target?** Yes — resolved meaning explicitly retained "as it may be conditionally accepted for the right opportunity." This is a direct improvement over 55B's Mode B (resolution_meaning_loss → resolution_correct).
5. **Did Case 3 preserve uncertainty through both stages?** Yes — Stage 1 preserved uncertainty, Stage 2 correctly returned targetResolved=false with remainingUncertainty.
6. **Did any Stage 2 output become stronger than its actual Stage 1 input?** No — manual semantic review found no case where Stage 2 strengthened beyond the actual Stage 1 meaning. The chaining check confirmed this explicitly for all three cases.
7. **Did any Stage 2 output flatten a condition present in Stage 1?** No — Case 2's condition survived both stages intact. This is the key positive finding.
8. **Compared with 55B, did carrying preserved meaning forward remove the observed conditionality loss?** Yes — in 55B Mode B, Case 2 was resolution_meaning_loss (flattened). In 55C Stage 2, Case 2 was resolution_correct with qualification retained. The two-stage chain eliminated this specific failure mode for the tested answer.
9. **Compared with 55A/55B, did the weak-priority case remain honestly unresolved?** No — in 55B, Case 1 was resolution_correct (unresolved) in that run; in 55C, it over-resolved because Stage 1 distorted the meaning first. The weak-priority problem is not solved by this approach.
10. **Does this prove a two-stage production design is required?** No — evidence from three answers is insufficient to justify specific production changes.
11. **Does this establish graph representation?** No — this experiment did not integrate with graph, Behaviour Selection, or any other engine component.
12. **Does this establish Behaviour Selection changes?** No — Behaviour Selection was not called or referenced.
---
### Limitations
- Only three answers were tested. Different answer patterns may behave differently.
- Case 1 revealed a new failure mode: if Stage 1 distorts meaning, Stage 2 amplifies it through chaining. This is not an improvement over 55B's approach for weak answers.
- The conditional trade-off improvement (Case 2) may not generalise to other conditional patterns.
- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434).
- Six live calls total — insufficient for broader generalisation.
- Case 3 Stage 1 added contextual framing ("whether...is a hard constraint or a preference/trade-off") to the non-answer, which could be questioned as mild interpretation even though it preserved uncertainty correctly.
---
### Experiment Conclusion
**Preserved meaning improved resolution but some loss remained.**
The two-stage chain successfully eliminated the conditionality loss observed in 55B: Case 2's conditional qualification survived through both stages and resolved correctly (resolution_correct). Non-answer uncertainty was also preserved through both stages (resolution_correct). These are genuine improvements.
However, the weak-priority case revealed a new failure mode: Stage 1 strengthened "Risk matters more to me." into language about "preference or trade-off rather than absolute hard constraint," and Stage 2 then over-resolved based on that distorted input. Carrying semantic state forward means distortion propagates as well as fidelity. This does not improve the weak-priority problem relative to 55B's Mode B (which correctly left Case 1 unresolved in its run).
The core finding is asymmetric: preserving meaning before resolution helps for conditional answers (eliminates flattening) and non-answers (preserves uncertainty), but does not help — and may worsen outcomes — when the meaning extraction step itself distorts. The question "does the judgement stop rewriting the meaning?" is answered partially: it stops rewriting when the input to judgement already carries the full meaning, but it amplifies rewriting when that input is itself distorted.
Does this prove a two-stage production design is required? **No.** The weak-priority case over-resolved in 55C while remaining unresolved in the 55B run — and neither result establishes which approach is better for all cases.
### Focused Test Result
All 6 live inference calls completed successfully. Three answers tested through chained meaning→resolution stages. Stage 2 eliminated the conditionality loss from 55B (Case 2: resolution_meaning_loss → resolution_correct) but did not eliminate over-resolution for weak-priority input when Stage 1 strengthened it first (Case 1: meaning_strengthened → resolution_overresolved). Non-answer uncertainty was preserved through both stages. Total: ~117s, average: ~19.5s per call, fastest: 15.0s, slowest: 23.8s.
### Historical Comparison Result
Compared to Experiment 55B's finding that "meaning was preserved in Mode A but lost during resolution," Experiment 55C shows that carrying preserved meaning forward eliminates that specific loss pattern for conditional answers (Case 2 resolved correctly with qualification retained) but introduces a different asymmetry: Stage 1 distortion propagates through Stage 2. The weak-priority case improved relative to 55A's over-resolution but degraded relative to the 55B run's correct unresolved result. Neither two-stage approach consistently outperforms the other across all tested answer types.
### Documentation Updated
- `docs/design-evolution-log.md` — added full Experiment 55C entry; applied corrected wording to Experiment 55B
- `docs/current-handoff.md` — updated with Experiment 55C summary, corrected 55B wording, and new Return-to-Work note
### Confirmation Host and Model Remained Unchanged
Host: `http://192.168.1.111:11434`. Model: `qwen-claude:latest`. Same as all prior experiments.
### Confirmation Semantic Instruction and Output Contract Remained Unchanged (for Stage 1)
Stage 1 instruction identical to Experiment 55B Mode A. No production code changed.
### Confirmation Production Prompts and Schemas Remained Unchanged
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file.
### Confirmation Behaviour Selection Remained Unchanged
Behaviour Selection was not called or referenced. No integration with the selector occurred.
### Confirmation Graph and UI Remained Unchanged
No graph files read or modified. No UI code touched. The experiment is test-only.
### Confirmation No Preserved-Meaning Resolution Logic Entered Active Runtime
This experiment created one new test file only. No preserved-meaning resolution logic entered any active runtime path, production module, or behaviour selection output. Production continues using the pre-existing combined instruction and contract.
---
### Return-to-Work Note (Experiment 55C)
55A showed one weak answer was over-resolved and one conditional answer lost nuance; 55B separated answer meaning from target-resolution judgement using independent calls. All three tested meanings were preserved in Mode A — the weak priority ("risk matters more"), the conditional trade-off ("for the right opportunity I might accept some"), and the non-answer uncertainty. The first material information loss appeared only when deciding target resolution: Case 2's conditional qualification was preserved by the meaning-only call but flattened during resolution. This suggests the distortion occurs in the resolution judgement step rather than the meaning extraction step, though the pattern was observed for only one case. Whether other answer types show the same pattern remains unproven. Graph, Behaviour Selection, UI and production remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7, commit 8c12931. First test/file to inspect when resuming: tests/reconstruction/semantic-answer-meaning-vs-resolution.test.js for the full experiment and results. Status pending Rob's review.