experiment: probe evidence versus clarification boundary

This commit is contained in:
2026-08-09 15:11:54 +01:00
parent 85ee4bed30
commit 11882bfaae
3 changed files with 237 additions and 0 deletions
+12
View File
@@ -211,3 +211,15 @@ One live Ollama call (19,343 ms) returned `userSupportedMeaning: "Avoiding addit
**Classification: PASS.** The model preserved the explicit hard-constraint status without weakening it into preference/trade-off language and did not add unsupported interpretation. `possibleInference` is null, which is appropriate for a direct unambiguous answer.
This experiment does not prove fidelity for other regression cases (E, F), consistency across multiple runs, or behavior in production reasoning paths. Branch: `feature/reasoning-fidelity-v0.8`. Files: `tests/reconstruction/semantic-regression-d-explicit-hard-constraint.test.js` and `docs/experiment-56j.md`. Status pending Rob's review.
### Experiment 56K Summary — Evidence-resolvable disagreement must not become user clarification
Tested whether the configured live Ollama model (`qwen-claude:latest` at `http://192.168.1.111:11434`) distinguishes evidence-resolvable uncertainty from user-owned ambiguity — Regression E from `docs/reasoning-refinement-requirements.md`.
Fixed case: Delivery delay concern with competing causes ("Staff capacity may be the issue" / "Supplier lead times are likely responsible.") — resolvable by evidence gathering, not user clarification.
One live Ollama call (18,580 ms) returned `uncertaintyType: "evidence_needed"` with specific evidence target: "Current internal staffing capacity levels and external supplier lead time records." No user clarification was introduced.
**Classification: PASS.** The model correctly identified the disagreement as requiring evidence rather than asking the user to settle an externally knowable question by clarification. It specified concrete, relevant evidence — demonstrating understanding of the causal structure rather than producing a generic classification. This confirms the model can preserve the distinction between "evidence needed to determine what is true" and "clarification needed because only the user can establish meaning/preference/intent/constraint" for this tested case.
This experiment does not prove fidelity for Regression F (user-owned ambiguity), consistency across domains/phrasings, downstream reasoning preservation, or end-to-end production flow. Branch: `feature/reasoning-fidelity-v0.8`. Files: `tests/reconstruction/semantic-regression-e-evidence-vs-clarification.test.js` and `docs/experiment-56k.md`. Status pending Rob's review.
+64
View File
@@ -0,0 +1,64 @@
# Experiment 56K — Evidence-resolvable disagreement must not become user clarification
**Date:** 2026-08-09
**Branch:** `feature/reasoning-fidelity-v0.8`
**Type:** Live semantic probe (single call)
**Status:** PASS
## Objective
Determine whether the configured model can distinguish uncertainty that requires external evidence from uncertainty that requires the user to clarify their own meaning, for **Regression E**.
## Regression E — Fixed case (exact)
- **Source:** Delivery delay concern.
- **Competing causes:** "Staff capacity may be the issue" / "Supplier lead times are likely responsible."
- **Expected preserved meaning:** Two distinct hypotheses about causation.
- **Expected uncertainty:** Which hypothesis is correct — resolvable by evidence gathering, not user clarification.
- **Must not happen:** Generating a user-facing clarification question when evidence sources can distinguish the hypotheses.
## Pre-written human reference (before model inspection)
> The unresolved disagreement can be reduced by obtaining relevant evidence. It must not be treated as missing user-owned meaning merely because the engine does not yet know which interpretation is correct. A correct result should preserve the difference between evidence needed to determine what is true, and clarification needed because only the user can establish what they mean, prefer, intend, define, or constrain.
Expected correct classification: `evidence_needed`
## Configuration
- **Host:** `http://192.168.1.111:11434`
- **Model:** `qwen-claude:latest`
- **Pattern:** Direct Ollama `/api/chat` call (from Experiment 55D historical test, commit `fcb7218407a2921e9197dbb0a65e4e1282459e4c`)
- **Format:** `json`, `stream: false`
## Live call result
- **Call count:** 1
- **Duration:** 18,580 ms
- **uncertaintyType:** `evidence_needed`
- **reason:** "The uncertainty involves competing objective causes for a delivery delay, which can be resolved by gathering factual data rather than clarifying user intent."
- **evidenceNeeded:** "Current internal staffing capacity levels and external supplier lead time records"
- **userClarificationNeeded:** (not included in output contract)
## Human semantic classification: PASS
**Rationale:** The model correctly identified the disagreement as `evidence_needed`. It specified concrete evidence that could resolve the competing hypotheses without introducing any user clarification requirement. This matches the pre-written human reference and confirms the model can distinguish evidence-resolvable uncertainty from user-owned ambiguity in this case.
## What this experiment establishes
- For Regression E (delivery delay with competing causal hypotheses), the model correctly classifies the uncertainty as requiring evidence, not user clarification.
- The model specified concrete, relevant evidence to seek — demonstrating it understood the nature of the disagreement rather than producing a generic or tautological classification.
- The evidence-vs-user-meaning distinction was preserved in this single tested case.
## What this experiment does NOT prove
- That the same boundary holds for Regression F (user-owned ambiguity: preference vs constraint).
- That the model consistently makes this distinction across different domains, phrasings, or weaker prompts.
- That downstream reasoning steps (graph update, Behaviour Selection) will preserve this distinction.
- That the distinction holds with other models or on this host without network variation.
- That end-to-end production flow preserves the classification.
## Critical rule compliance
- Production reasoning code changed: **NO**
- Generic harness created/modified: **NO**
- Retries/additional calls: **0**