Experiment 56K — Evidence-resolvable disagreement must not become user clarification

Date: 2026-08-09 Branch: feature/reasoning-fidelity-v0.8 Type: Live semantic probe (single call) Status: PASS

Objective

Determine whether the configured model can distinguish uncertainty that requires external evidence from uncertainty that requires the user to clarify their own meaning, for Regression E.

Regression E — Fixed case (exact)

Source: Delivery delay concern.
Competing causes: "Staff capacity may be the issue" / "Supplier lead times are likely responsible."
Expected preserved meaning: Two distinct hypotheses about causation.
Expected uncertainty: Which hypothesis is correct — resolvable by evidence gathering, not user clarification.
Must not happen: Generating a user-facing clarification question when evidence sources can distinguish the hypotheses.

Pre-written human reference (before model inspection)

The unresolved disagreement can be reduced by obtaining relevant evidence. It must not be treated as missing user-owned meaning merely because the engine does not yet know which interpretation is correct. A correct result should preserve the difference between evidence needed to determine what is true, and clarification needed because only the user can establish what they mean, prefer, intend, define, or constrain.

Expected correct classification: evidence_needed

Configuration

Host: http://192.168.1.111:11434
Model: qwen-claude:latest
Pattern: Direct Ollama /api/chat call (from Experiment 55D historical test, commit fcb7218407a2921e9197dbb0a65e4e1282459e4c)
Format: json, stream: false

Live call result

Call count: 1
Duration: 18,580 ms
uncertaintyType: evidence_needed
reason: "The uncertainty involves competing objective causes for a delivery delay, which can be resolved by gathering factual data rather than clarifying user intent."
evidenceNeeded: "Current internal staffing capacity levels and external supplier lead time records"
userClarificationNeeded: (not included in output contract)

Human semantic classification: PASS

Rationale: The model correctly identified the disagreement as evidence_needed. It specified concrete evidence that could resolve the competing hypotheses without introducing any user clarification requirement. This matches the pre-written human reference and confirms the model can distinguish evidence-resolvable uncertainty from user-owned ambiguity in this case.

What this experiment establishes

For Regression E (delivery delay with competing causal hypotheses), the model correctly classifies the uncertainty as requiring evidence, not user clarification.
The model specified concrete, relevant evidence to seek — demonstrating it understood the nature of the disagreement rather than producing a generic or tautological classification.
The evidence-vs-user-meaning distinction was preserved in this single tested case.

What this experiment does NOT prove

That the same boundary holds for Regression F (user-owned ambiguity: preference vs constraint).
That the model consistently makes this distinction across different domains, phrasings, or weaker prompts.
That downstream reasoning steps (graph update, Behaviour Selection) will preserve this distinction.
That the distinction holds with other models or on this host without network variation.
That end-to-end production flow preserves the classification.

Critical rule compliance

Production reasoning code changed: NO
Generic harness created/modified: NO
Retries/additional calls: 0

3.6 KiB Raw Permalink Blame History