9.4 KiB
Experiment 54R — Does a Material Disagreement Actually Require User Clarification? (2026-08-07)
Objective
Given an explicit interpretation disagreement and its evidence consequence, test whether the model can distinguish between a disagreement that requires clarification from the user and one that can be resolved by investigating evidence.
This is passive and test-only. Do not generate the clarification question. Do not generate the next investigation question. Do not choose a winning interpretation. Do not change production behaviour.
Hypothesis
The model may be able to distinguish:
Evidence-resolvable disagreement: The user's meaning is sufficiently clear, but competing explanations require different evidence.
User-clarification disagreement: The disagreement concerns the user's intended meaning, priority, constraint, or definition, so external evidence cannot resolve it without asking the user.
If this distinction works, disagreement does not have to map automatically to clarification.
Context Budget
Read only:
docs/current-handoff.md;- Experiment 54Q only in
docs/design-evolution-log.md; tests/reconstruction/semantic-structured-evidence-consequence.test.js;.env.localonly forOLLAMA_BASE_URLandOLLAMA_MODEL.
Not read: Behaviour Selection; assessor; graph files; UI; active prompts; question-selection code; full experiment history.
Configuration
Host: http://192.168.1.111:11434 (same as 54Q)
Model: qwen-claude:latest (same as 54Q)
No localhost fallback. No experiment-specific model variable.
Number of Live Inference Calls
Exactly 3 live Ollama calls — one per case.
Input Contract
Each call receives:
{
"source": "...",
"disagreement": ["..."],
"evidenceNeeded": ["..."]
}
The disagreement and evidence needs are fixed human-reviewed inputs. The model does not rediscover them.
Output Contract
Return exactly:
{
"requiresUserClarification": true | false,
"reason": "one short sentence"
}
No question text, no recommended action, no preferred interpretation, no score, no confidence, no behaviour label. This boolean is test-only and is not a production contract.
Semantic Instruction Used
Decide whether resolving the stated disagreement requires additional meaning, preference, intent, or factual information that only the user can provide. Return true when evidence alone cannot settle the disagreement because the missing distinction belongs to the user's intended meaning, priority, constraint, or private knowledge. Return false when the disagreement can be investigated using external, operational, or observable evidence without asking the user to define what they mean. Do not generate a question and do not choose which interpretation is correct.
No examples. No expected answers. Same instruction for all three cases.
Case 1 — Competing Causes, Evidence Can Resolve
Source: "Orders are arriving late and customers have started complaining."
Disagreement: delays may be caused by insufficient staff capacity / delays may be caused by unreliable supplier lead times.
Evidence needed: staffing levels and workload; processing throughput; supplier lead-time history; supplier delivery reliability.
Fixed human reference: requiresUserClarification: false
Case 2 — User Priority Is Ambiguous
Source: "I want the business to grow, but I don't want to take on more risk."
Disagreement: growth should be prioritised even if some additional risk is unavoidable / avoiding additional risk is a hard constraint even if growth is slower.
Evidence needed: possible growth opportunities; risk exposure of each option.
Fixed human reference: requiresUserClarification: true
Case 3 — Same Meaning, No Clarification Need
Source: "Revenue is down. I think pricing may be part of the problem, but I am not sure."
Disagreement: no substantive disagreement; two interpretations express the same uncertainty in different words.
Evidence needed: pricing history; sales/conversion response; other plausible revenue drivers.
Fixed human reference: requiresUserClarification: false
Results
Case 1 — Competing Causes, Evidence Can Resolve
Model output:
{
"requiresUserClarification": false,
"reason": "The disagreement involves operational causes that can be resolved by analyzing objective company data and supplier records."
}
Resolution-source classification: resolution_source_correct ✓ — Returned false. The model correctly identified that operational evidence (company data, supplier records) can distinguish the competing causal hypotheses without asking the user.
Case 2 — User Priority Is Ambiguous
Model output:
{
"requiresUserClarification": true,
"reason": "Resolving this conflict depends on the user's personal priority and risk tolerance, which cannot be determined by external facts alone."
}
Resolution-source classification: resolution_source_correct ✓ — Returned true. The model correctly identified that user-owned meaning (priority, risk tolerance) is required to resolve the disagreement. External facts alone cannot determine whether "don't want more risk" is a preference, constraint, or boundary.
Case 3 — Same Meaning, No Clarification Need
Model output:
{
"requiresUserClarification": false,
"reason": "The stated position explicitly confirms there is no substantive disagreement, so no additional user input is required to resolve it."
}
Resolution-source classification: resolution_source_correct ✓ — Returned false. The model correctly identified that the absence of material disagreement means clarification is unnecessary. Evidence can investigate the underlying uncertainty.
Resolution-Source-Correct Count
3 of 3 cases classified correctly.
Resolution-Source-Failed Count
0 of 3 cases classified incorrectly.
Did Case 1 Correctly Recognise That Competing Causal Hypotheses Can Be Resolved Through Evidence?
Yes. The model returned false and provided a reason referencing operational causes resolvable by company data and supplier records — evidence, not user meaning.
Did Case 2 Correctly Recognise That the Unresolved Growth-Versus-Risk Priority Belongs to the User?
Yes. The model returned true and identified that resolution depends on "the user's personal priority and risk tolerance," which external facts alone cannot determine.
Did Case 3 Avoid Unnecessary Clarification Where There Was No Material Disagreement?
Yes. The model returned false, correctly noting the absence of substantive disagreement makes additional clarification unnecessary.
Did the Model Treat Every Disagreement as Requiring User Clarification?
No. Two of three cases returned false. Only Case 2 (ambiguous priority) returned true.
Did the Model Confuse Missing Evidence with Missing User Meaning?
No. In Case 1, the model correctly distinguished between lacking evidence to investigate causes (which it flagged as resolvable through evidence gathering) and lacking user meaning (which it did not claim). The reason text referenced "analyzing objective company data and supplier records" rather than requiring user input.
Did the Model Generate an Actual Question?
No. No question was generated in any output. The output contract was respected in all cases.
Did the Model Choose a Winner?
No. No interpretation was selected as correct in any case.
Evidence That User-Owned Ambiguity Can Be Separated From Evidence Uncertainty
Case 2 succeeded where Case 1 and Case 3 both returned false for different reasons — one because evidence can resolve it, the other because no disagreement exists. The model's reasons for each case were distinct in their reference points: operational data (Case 1) versus user priority (Case 2) versus absence of disagreement (Case 3). This pattern suggests the model does not collapse all ambiguity into a single clarification need.
Limitations
- Three cases only; limited domain coverage (one delivery scenario, one strategic priority, one revenue statement).
- Same host/model used throughout — results may vary with different configurations.
- Does not establish generalisation beyond these specific inputs.
- The distinction tested here is binary (true/false) and does not test partial or probabilistic resolution-source classification.
- No evidence was actually gathered in any case — only whether the source of resolution was correctly identified.
- The Case 1 evaluator warning (if present) was a false positive from heuristic wording checks, not a semantic failure.
Conclusion
The model correctly distinguished user-clarification needs from evidence-resolvable disagreement in all tested cases.
Across the three tested patterns — competing causal hypotheses, ambiguous user priority, and absent material disagreement — the model returned the correct boolean in every case with semantically appropriate reasoning. No clarification or investigation question was generated. No interpretation was selected as correct. Across the three tested disagreement patterns, the model did not automatically map disagreement to user clarification.
Status
Pending Rob's review. No production code changed. No schemas modified. No active engine behaviour changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: tests/reconstruction/semantic-disagreement-resolution-source.test.js.