15 KiB
Experiment 55A — Can the Model Preserve Uncertainty From Weak Clarification Answers? (2026-08-08)
Objective
First, tighten Experiment 54Z so its conclusion does not over-generalise from two weaker answers.
Then isolate the strongest unresolved issue from 54Z:
When a clarification answer is weak or conditional, can the model preserve uncertainty instead of forcing the answer into a stronger resolved meaning?
This experiment is about answer interpretation only. Do not compare precise versus broadened targets. Do not test question wording. Do not integrate with Behaviour Selection, graph, or UI.
Hypothesis
The model may be too eager to convert weak clarification answers into resolved meaning. A better behaviour would preserve uncertainty when the answer does not actually settle the supplied target. If the model consistently keeps weak answers unresolved, the 54Z over-resolution may have been incidental. If it repeatedly forces resolution, that becomes a clearer limitation of the answer-resolution step itself.
Configuration
Host: http://192.168.1.111:11434 (same as all prior experiments)
Model: qwen-claude:latest (same as all prior experiments)
Number of Live Inference Calls
Exactly 4 live Ollama calls — one answer-resolution per case. Exactly 4 cases of varying strength against the same source/target/question.
Context Used
docs/current-handoff.md- Experiment 54Z in
docs/design-evolution-log.md(as basis for isolating the unresolved issue) tests/reconstruction/semantic-clarification-weak-answer-consequence.test.js(structural reference)tests/reconstruction/semantic-clarification-answer-resolution.test.js(structural reference: instruction and output contract)
Semantic Instruction Unchanged
Use the user's clarification answer only to resolve the supplied clarification target. State the meaning now established by that answer. Mark targetResolved true only when the answer settles the target. Put any uncertainty that remains specifically about that target into remainingUncertainty; otherwise return null. Do not infer wider consequences, rewrite unrelated source meaning, recommend action, or generate another question.
No examples added. No stronger coaching attempted. Same instruction as Experiment 54V.
Output Contract Unchanged
{
"resolvedMeaning": "short statement",
"targetResolved": true,
"remainingUncertainty": "short statement or null"
}
Fixed Source
I want the business to grow, but I don't want to take on more risk.
Clarification Target
whether avoiding additional risk is a preference/trade-off or a hard constraint.
Clarification Question
Do you view avoiding additional risk as a hard constraint, or as a preference or trade-off?
Case 1 — Fully Explicit Hard Constraint
User answer: "It's a hard constraint. I don't want any increase in risk."
Model output:
- resolvedMeaning: "Avoiding additional risk is a hard constraint."
- targetResolved: true
- remainingUncertainty: null
Classification: resolution_correct
The model correctly resolved the explicit hard constraint with appropriate strength and no spurious uncertainty. This is the positive control — it behaved as expected.
Case 2 — Weak Priority Statement
User answer: "Risk matters more to me."
Model output:
- resolvedMeaning: "Avoiding additional risk is prioritized as a strong preference or guiding principle rather than a rigid, non-negotiable constraint."
- targetResolved: true
- remainingUncertainty: null
Classification: uncertainty_overresolved
The model set targetResolved=true and produced a meaning stronger than the user supplied. "Risk matters more to me" establishes relative importance but does not determine whether avoiding additional risk is a hard constraint or a strong preference/trade-off. The model inferred "not a rigid, non-negotiable constraint" from a statement that only said risk was more important — which could be consistent with a strong preference that still functions as a de facto hard boundary in practice. This over-resolution matches the concern flagged in Experiment 54Z but isolated it here under a single fixed target to confirm the behaviour is real.
The model produced stronger meaning than the user supplied by declaring what risk avoidance is not (a rigid constraint) when the user only stated what it is relative to (more important than growth). This establishes that uncertainty should have been preserved with remainingUncertainty about whether avoiding additional risk is a preference/trade-off or a hard constraint.
Case 3 — Conditional Trade-Off
User answer: "I'd normally avoid more risk, but for the right opportunity I might accept some."
Model output:
- resolvedMeaning: "Avoiding additional risk is a preference or trade-off rather than a hard constraint."
- targetResolved: true
- remainingUncertainty: null
Classification: resolution_correct with conditional qualification loss
The target category was resolved correctly, but the answer's conditional qualification was lost. The user said "I might accept some for the right opportunity" — a specific conditional exception — which became "preference or trade-off rather than a hard constraint." The target (whether risk avoidance is preference/trade-off or hard constraint) was settled, but the nuance of how it functions conditionally was erased from the resolved meaning.
Case 4 — Non-Answer / Insufficient Clarification
User answer: "I'm not really sure."
Model output:
- resolvedMeaning: "The user has not yet determined if avoiding additional risk is a hard constraint or a preference/trade-off."
- targetResolved: false
- remainingUncertainty: "It remains unclear whether the user considers avoiding additional risk a hard constraint or a preference/trade-off."
Classification: uncertainty_preserved
The model correctly remained unresolved without inventing any position. The resolvedMeaning accurately describes what was supplied (nothing) and remainingUncertainty states that the distinction between hard constraint and preference/trade-off remains unresolved. This is honest behaviour.
Evaluation Summary
| Case | Answer | Resolution Status | Classification |
|---|---|---|---|
| 1 | Explicit hard constraint | Correctly resolved | resolution_correct |
| 2 | Weak priority ("Risk matters more to me.") | Over-resolved to targetResolved=true | uncertainty_overresolved |
| 3 | Conditional trade-off | Resolved, conditionality lost | resolution_correct (with note) |
| 4 | Non-answer ("I'm not really sure.") | Correctly unresolved | uncertainty_preserved |
Classification counts:
- resolution_correct: 2
- uncertainty_preserved: 1
- uncertainty_overresolved: 1
- resolution_failed: 0
Questions Answered
-
Did Case 1 correctly resolve the explicit hard constraint? Yes — targetResolved=true, appropriate meaning strength, no spurious uncertainty.
-
Did Case 2 preserve uncertainty rather than infer that risk is not a hard constraint? No — the model over-resolved, setting targetResolved=true and inferring "not a rigid, non-negotiable constraint" from a weak priority statement. This is the key finding: one tested weak answer received stronger meaning than the user supplied.
-
Did Case 3 preserve the conditional nature of the trade-off? Partially — the model correctly resolved the target (risk avoidance is not a hard constraint) but flattened the conditional qualification ("might accept some for the right opportunity") into flat "preference or trade-off" language. The conditional layer was lost even though target resolution was correct.
-
Did Case 4 correctly remain unresolved? Yes — no invented position, appropriate remainingUncertainty stating the distinction remains unresolved.
-
How many cases were:
- resolution_correct: 2
- uncertainty_preserved: 1
- uncertainty_overresolved: 1
- resolution_failed: 0
-
Did any answer get stronger meaning than the user supplied? Yes — Case 2 ("Risk matters more to me.") received a stronger interpretation than it justified. The model inferred "not a rigid, non-negotiable constraint" from a statement that only established relative priority.
-
Did any unresolved answer incorrectly return targetResolved true? No — Case 4 (the only truly unresolved case) correctly returned false. Case 2 over-resolved but did not remain unresolved.
-
Did any resolved answer incorrectly retain uncertainty? No — both resolved cases (1 and 3) correctly returned null for remainingUncertainty.
-
Does the answer-resolution step appear biased toward resolution in these tested cases? Yes — Case 2 demonstrates that a weak priority statement can be over-resolved to a definitive classification ("not a constraint") when it should have remained unresolved. One out of four cases showed this bias, but it appeared on the weakest-answer type where uncertainty preservation matters most.
-
Does this establish how answer-resolution logic should be changed? No — one weak-priority case over-resolved; this does not justify a broad change without broader testing.
-
Does this establish how the graph should represent unresolved clarification? No — the unresolved representation question is separate from whether the model should remain unresolved.
-
Does this establish how Behaviour Selection should react? No — this experiment did not integrate with Behaviour Selection.
Limitations
- Only one source scenario and four answer types were tested. Different sources may behave differently.
- Only one weak-answer pattern (priority without constraint meaning) over-resolved; other patterns not assessed.
- Only one model configuration was used (qwen-claude:latest on 192.168.1.111:11434). Different models may behave differently.
- Case 3 showed conditionality loss that is subtler than over-resolution — it resolved correctly but flattened nuance. This pattern warrants further testing with additional conditional answers.
- Four cases tested; broader generalisation remains untested.
Experiment Conclusion
Clarification-answer resolution preserved uncertainty appropriately across some cases but over-resolved one weak answer.
Case 1 (explicit hard constraint) and Case 4 (non-answer) behaved honestly — the explicit case resolved, the empty case remained unresolved. This is the expected baseline.
Case 2 (weak priority: "Risk matters more to me.") demonstrates the core limitation: the model over-resolved a weak answer, converting relative priority into a definitive negative ("not a rigid, non-negotiable constraint") when only relative importance was supplied. The model set targetResolved=true and erased uncertainty that should have been preserved.
Case 3 (conditional trade-off) resolved the target correctly but lost the conditional qualification in the process — "might accept some for the right opportunity" was flattened to "preference or trade-off rather than a hard constraint." This is a subtler form of meaning inflation: correct resolution with erasure of nuance.
Two of the four tested answers showed loss of nuance: one was over-resolved (Case 2) and one retained the correct target category while losing conditional qualification (Case 3). The remaining two cases behaved correctly (Cases 1 and 4). With qwen-claude:latest and the current semantic instruction, uncertainty preservation works for truly empty answers but is unreliable for weak-priority answers.
Focused Test Result
All 4 live inference calls completed successfully. One over-resolution (Case 2), one conditionality loss (Case 3), two honest classifications (Cases 1 and 4). Total: ~62s, average: ~15.5s per call, fastest: 8.2s, slowest: 19.3s.
Historical Comparison Result
Compared to Experiment 54Z's finding that both variants over-resolved Case 1 ("Risk matters more to me."), Experiment 55A confirms this is a persistent issue under the same semantic instruction and model — not a side-effect of target framing. When a single fixed target was used, the weak priority answer still over-resolved (targetResolved=true with inferred "not a constraint" meaning). The same over-resolution reproduced with a fixed target, so target broadening is not required for the failure to occur.
Documentation Updated
docs/design-evolution-log.md— added full Experiment 55A entry; applied 54Z wording correctionsdocs/current-handoff.md— updated with Experiment 55A summary and new Return-to-Work note
Confirmation Host and Model Remained Unchanged
Host: http://192.168.1.111:11434. Model: qwen-claude:latest. Same as all prior experiments.
Confirmation Semantic Instruction and Output Contract Remained Unchanged
Answer-resolution instruction identical to Experiment 54V. No stronger coaching, no added examples. Output contract unchanged from 54V ({ resolvedMeaning, targetResolved, remainingUncertainty }).
Confirmation Production Prompts and Schemas Remained Unchanged
No production prompts read or modified. No schemas changed. All inference calls used the experiment-specific semantic instructions defined in this test file.
Confirmation Behaviour Selection Remained Unchanged
Behaviour Selection was not called or referenced. No integration with the selector occurred.
Confirmation Graph and UI Remained Unchanged
No graph files read or modified. No UI code touched. The experiment is test-only.
Confirmation No Uncertainty-Preservation Logic Entered Active Runtime
This experiment created one new test file only. No uncertainty-preservation logic entered any active runtime path, production module, or behaviour selection output.
Return-to-Work Note (Experiment 55A)
54Z showed weaker answers can produce different downstream resolution states under different framings, suggesting target broadening matters; 55A isolated the answer-resolution step using one fixed target and four answers of varying strength (explicit hard constraint, weak priority, conditional trade-off, non-answer) to test whether uncertainty preservation holds independently of framing. The explicit case resolved correctly, the non-answer remained honestly unresolved, but the weak-priority case over-resolved by setting targetResolved=true and inferring "not a constraint" from relative importance alone. Conditional language was also flattened even when resolution was correct. Uncertainty preservation remains unreliable for weak-priority answers. Graph, Behaviour Selection, UI, and production integration remain untouched. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: tests/reconstruction/semantic-clarification-uncertainty-preservation.test.js for the full experiment and results. Status pending Rob's review.