15 KiB
Experiment 55C — Resolution From Preserved Answer Meaning (2026-08-08)
Objective
Experiment 55B showed meaning-only extraction preserved all three tested answers while one conditional answer lost qualification during independent resolution. The remaining question: if the preserved answer meaning is explicitly carried into the resolution step, does the later judgement still flatten it?
This tests whether carrying semantic state forward across two calls eliminates the conditionality loss, introduces a new failure mode, or changes neither.
Hypothesis
A two-step test-only chain may preserve meaning better because the resolution stage no longer needs to reinterpret the raw answer. Possible outcomes: preserved meaning survives resolution; the resolution stage still flattens it; some answers improve while others do not. Any outcome is useful.
Configuration
Host: http://192.168.1.111:11434 (same as all prior experiments)
Model: qwen-claude:latest (same as all prior experiments)
Number of Live Inference Calls
Exactly 6 live Ollama calls — one meaning call and one resolution call per case, three cases. Stage 2 uses the actual answerMeaning from Stage 1, not a human reference.
Fixed Source / Target / Question (identical to 55A/55B)
- Source: "I want the business to grow, but I don't want to take on more risk."
- Clarification target: whether avoiding additional risk is a preference/trade-off or a hard constraint.
- Clarification question: Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?
Stage 1 instruction (unchanged from 55B Mode A)
State only what the user's answer establishes in relation to the clarification question. Preserve uncertainty, conditionality, and qualification exactly as supplied. Do not decide whether the clarification target is resolved. Do not infer what the user did not say. Do not recommend action or generate another question.
Return valid JSON only in this shape: { "answerMeaning": "short statement" }
Stage 1 output contract
{
"answerMeaning": "short statement"
}
Stage 2 instruction (new — uses preserved meaning, not raw answer)
Decide whether the supplied preserved answer meaning settles the clarification target. Treat the preserved meaning as the full extent of what has been established — do not strengthen, simplify, or reinterpret it. Mark targetResolved true only when that preserved meaning settles the target. If any part of the target remains unresolved, preserve that uncertainty. Keep qualifications and conditions intact.
Return valid JSON only in this shape: { "resolvedMeaning": "short statement", "targetResolved": true, "remainingUncertainty": null }
Stage 2 output contract
{
"resolvedMeaning": "short statement",
"targetResolved": true,
"remainingUncertainty": null
}
Case 1 — Weak Priority
Raw answer: "Risk matters more to me."
Stage 1 result:
- answerMeaning: "The user indicates that risk carries greater weight, suggesting it leans toward being a significant preference or trade-off rather than explicitly confirming an absolute hard constraint."
- Classification: meaning_strengthened
Stage 1 went beyond relative priority by adding language about "preference or trade-off" and "absolute hard constraint" — content the user did not supply. This was also seen in 55A's Stage 2, but now it appears at Stage 1. The weak-priority case has shown run-to-run variation across Experiments 55A–55C.
Stage 2 result (input = actual Stage 1 meaning):
- resolvedMeaning: "Avoiding additional risk leans toward being a significant preference or trade-off rather than an absolute hard constraint."
- targetResolved: true
- remainingUncertainty: null
- Classification: resolution_overresolved
Because Stage 1 already strengthened the answer, Stage 2 treated that strengthened meaning as input and over-resolved with targetResolved=true and no remaining uncertainty. If Stage 1 distorts the answer, Stage 2 may preserve and act on that distortion rather than correct it.
Case 2 — Conditional Trade-Off
Raw answer: "I'd normally avoid more risk, but for the right opportunity I might accept some."
Stage 1 result:
- answerMeaning: "Avoiding additional risk is treated as a normal preference that may be conditionally accepted for the right opportunity."
- Classification: meaning_preserved
Both the normal preference and conditional exception were preserved. The wording uses "treated as" rather than directly repeating the user's phrasing but captures the full semantic content.
Stage 2 result (input = actual Stage 1 meaning):
- resolvedMeaning: "Avoiding additional risk is a preference rather than a hard constraint, as it may be conditionally accepted for the right opportunity."
- targetResolved: true
- remainingUncertainty: null
- Classification: resolution_correct
Target was correctly resolved with the conditional qualification ("as it may be conditionally accepted for the right opportunity") retained in the resolved meaning. This is the key result: the two-stage chain preserved the conditionality that 55B's Mode B flattened.
Case 3 — Non-Answer
Raw answer: "I'm not really sure."
Stage 1 result:
- answerMeaning: "The user states they are not really sure whether avoiding additional risk is a hard constraint or a preference/trade-off."
- Classification: meaning_preserved
Uncertainty preserved accurately. The model added the clarification target context ("whether...is a hard constraint or a preference/trade-off") which is reasonable contextual framing for the non-answer.
Stage 2 result (input = actual Stage 1 meaning):
- resolvedMeaning: "The user is uncertain whether avoiding additional risk is a preference/trade-off or a hard constraint."
- targetResolved: false
- remainingUncertainty: "Whether avoiding additional risk is a preference/trade-off or a hard constraint remains unresolved."
- Classification: resolution_correct
Correctly unresolved with appropriate remaining uncertainty. No meaning distortion in either stage.
Evaluation Summary
| Case | Answer | Stage 1 | Stage 1 Class | Stage 2 Result | Stage 2 Class |
|---|---|---|---|---|---|
| 1 | "Risk matters more to me." | Strengthened beyond priority | meaning_strengthened | Over-resolved (targetResolved=true) | resolution_overresolved |
| 2 | "I'd normally avoid more risk, but for the right opportunity I might accept some." | Preserved conditionality | meaning_preserved | Resolved with qualification retained | resolution_correct |
| 3 | "I'm not really sure." | Preserved uncertainty | meaning_preserved | Correctly unresolved | resolution_correct |
Meaning counts (Stage 1):
- meaning_preserved: 2
- meaning_strengthened: 1
- meaning_lost: 0
Resolution counts (Stage 2):
- resolution_correct: 2
- resolution_overresolved: 1
- resolution_meaning_loss: 0
- resolution_underresolved: 0
Questions Answered
-
Did Case 1 Stage 1 preserve only relative priority? No — it strengthened beyond relative priority by introducing "preference or trade-off" and "absolute hard constraint" language not present in the user's answer.
-
Did Case 1 Stage 2 remain unresolved? No — targetResolved=true because the distorted Stage 1 input led the model to conclude resolution was achieved.
-
Did Case 2 Stage 1 preserve the conditional qualification? Yes — "normal preference that may be conditionally accepted for the right opportunity" preserved both the default stance and the exception.
-
Did Case 2 Stage 2 retain that qualification while resolving the target? Yes — resolved meaning explicitly retained "as it may be conditionally accepted for the right opportunity." This is a direct improvement over 55B's Mode B (resolution_meaning_loss → resolution_correct).
-
Did Case 3 preserve uncertainty through both stages? Yes — Stage 1 preserved uncertainty, Stage 2 correctly returned targetResolved=false with remainingUncertainty.
-
Did any Stage 2 output become stronger than its actual Stage 1 input? No — manual semantic review found no case where Stage 2 strengthened beyond the actual Stage 1 meaning. The chaining check confirmed this explicitly for all three cases.
-
Did any Stage 2 output flatten a condition present in Stage 1? No — Case 2's condition survived both stages intact. This is the key positive finding.
-
Compared with 55B, did carrying preserved meaning forward remove the observed conditionality loss? Yes — in 55B Mode B, Case 2 was resolution_meaning_loss (flattened). In 55C Stage 2, Case 2 was resolution_correct with qualification retained. The two-stage chain eliminated this specific failure mode for the tested answer.
-
Compared with 55A/55B, did the weak-priority case remain honestly unresolved? No — in 55B, Case 1 was resolution_correct (unresolved) in that run; in 55C, it over-resolved because Stage 1 distorted the meaning first. The weak-priority problem is not solved by this approach.
-
Does this prove a two-stage production design is required? No — evidence from three answers is insufficient to justify specific production changes.
-
Does this establish graph representation? No — this experiment did not integrate with graph, Behaviour Selection, or any other engine component.
-
Does this establish Behaviour Selection changes? No — Behaviour Selection was not called or referenced.
Limitations
- Only three answers were tested. Different answer patterns may behave differently.
- Case 1 revealed a new failure mode: if Stage 1 distorts meaning, Stage 2 amplifies it through chaining. This is not an improvement over 55B's approach for weak answers.
- The conditional trade-off improvement (Case 2) may not generalise to other conditional patterns.
- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434).
- Six live calls total — insufficient for broader generalisation.
- Case 3 Stage 1 added contextual framing ("whether...is a hard constraint or a preference/trade-off") to the non-answer, which could be questioned as mild interpretation even though it preserved uncertainty correctly.
Experiment Conclusion
Preserved meaning improved resolution but some loss remained.
The two-stage chain successfully eliminated the conditionality loss observed in 55B: Case 2's conditional qualification survived through both stages and resolved correctly (resolution_correct). Non-answer uncertainty was also preserved through both stages (resolution_correct). These are genuine improvements.
However, the weak-priority case revealed a new failure mode: Stage 1 strengthened "Risk matters more to me." into language about "preference or trade-off rather than absolute hard constraint," and Stage 2 then over-resolved based on that distorted input. Carrying semantic state forward means distortion propagates as well as fidelity. This does not improve the weak-priority problem relative to 55B's Mode B (which correctly left Case 1 unresolved in its run).
The core finding is asymmetric: preserving meaning before resolution helps for conditional answers (eliminates flattening) and non-answers (preserves uncertainty), but does not help — and may worsen outcomes — when the meaning extraction step itself distorts. The question "does the judgement stop rewriting the meaning?" is answered partially: it stops rewriting when the input to judgement already carries the full meaning, but it amplifies rewriting when that input is itself distorted.
Does this prove a two-stage production design is required? No. The weak-priority case over-resolved in 55C while remaining unresolved in the 55B run — and neither result establishes which approach is better for all cases.
Focused Test Result
All 6 live inference calls completed successfully. Three answers tested through chained meaning→resolution stages. Stage 2 eliminated the conditionality loss from 55B (Case 2: resolution_meaning_loss → resolution_correct) but did not eliminate over-resolution for weak-priority input when Stage 1 strengthened it first (Case 1: meaning_strengthened → resolution_overresolved). Non-answer uncertainty was preserved through both stages. Total: ~117s, average: ~19.5s per call, fastest: 15.0s, slowest: 23.8s.
Historical Comparison Result
Compared to Experiment 55B's finding that "meaning was preserved in Mode A but lost during resolution," Experiment 55C shows that carrying preserved meaning forward eliminates that specific loss pattern for conditional answers (Case 2 resolved correctly with qualification retained) but introduces a different asymmetry: Stage 1 distortion propagates through Stage 2. The weak-priority case improved relative to 55A's over-resolution but degraded relative to the 55B run's correct unresolved result. Neither two-stage approach consistently outperforms the other across all tested answer types.
Documentation Updated
docs/design-evolution-log.md— added full Experiment 55C entry; applied corrected wording to Experiment 55Bdocs/current-handoff.md— updated with Experiment 55C summary, corrected 55B wording, and new Return-to-Work note
Confirmation Host and Model Remained Unchanged
Host: http://192.168.1.111:11434. Model: qwen-claude:latest. Same as all prior experiments.
Confirmation Semantic Instruction and Output Contract Remained Unchanged (for Stage 1)
Stage 1 instruction identical to Experiment 55B Mode A. No production code changed.
Confirmation Production Prompts and Schemas Remained Unchanged
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file.
Confirmation Behaviour Selection Remained Unchanged
Behaviour Selection was not called or referenced. No integration with the selector occurred.
Confirmation Graph and UI Remained Unchanged
No graph files read or modified. No UI code touched. The experiment is test-only.
Confirmation No Preserved-Meaning Resolution Logic Entered Active Runtime
This experiment created one new test file only. No preserved-meaning resolution logic entered any active runtime path, production module, or behaviour selection output. Production continues using the pre-existing combined instruction and contract.
Return-to-Work Note (Experiment 55C)
55A showed one weak answer was over-resolved and one conditional answer lost nuance; 55B separated answer meaning from target-resolution judgement using independent calls. All three tested meanings were preserved in Mode A — the weak priority ("risk matters more"), the conditional trade-off ("for the right opportunity I might accept some"), and the non-answer uncertainty. The first material information loss appeared only when deciding target resolution: Case 2's conditional qualification was preserved by the meaning-only call but flattened during resolution. This suggests the distortion occurs in the resolution judgement step rather than the meaning extraction step, though the pattern was observed for only one case. Whether other answer types show the same pattern remains unproven. Graph, Behaviour Selection, UI and production remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7, commit 3af623a. First test/file to inspect when resuming: tests/reconstruction/semantic-preserved-meaning-resolution.test.js for the full experiment and results. Status pending Rob's review.