robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: probe evidence versus clarification boundary

Browse Source
This commit is contained in:
robbond
2026-08-09 15:11:54 +01:00
parent 85ee4bed30
commit 11882bfaae
3 changed files with 237 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+12
View File
@@ -211,3 +211,15 @@ One live Ollama call (19,343 ms) returned `userSupportedMeaning: "Avoiding addit
**Classification: PASS.** The model preserved the explicit hard-constraint status without weakening it into preference/trade-off language and did not add unsupported interpretation. `possibleInference` is null, which is appropriate for a direct unambiguous answer.
This experiment does not prove fidelity for other regression cases (E, F), consistency across multiple runs, or behavior in production reasoning paths. Branch: `feature/reasoning-fidelity-v0.8`. Files: `tests/reconstruction/semantic-regression-d-explicit-hard-constraint.test.js` and `docs/experiment-56j.md`. Status pending Rob's review.
### Experiment 56K Summary — Evidence-resolvable disagreement must not become user clarification
Tested whether the configured live Ollama model (`qwen-claude:latest` at `http://192.168.1.111:11434`) distinguishes evidence-resolvable uncertainty from user-owned ambiguity — Regression E from `docs/reasoning-refinement-requirements.md`.
Fixed case: Delivery delay concern with competing causes ("Staff capacity may be the issue" / "Supplier lead times are likely responsible.") — resolvable by evidence gathering, not user clarification.
One live Ollama call (18,580 ms) returned `uncertaintyType: "evidence_needed"` with specific evidence target: "Current internal staffing capacity levels and external supplier lead time records." No user clarification was introduced.
**Classification: PASS.** The model correctly identified the disagreement as requiring evidence rather than asking the user to settle an externally knowable question by clarification. It specified concrete, relevant evidence — demonstrating understanding of the causal structure rather than producing a generic classification. This confirms the model can preserve the distinction between "evidence needed to determine what is true" and "clarification needed because only the user can establish meaning/preference/intent/constraint" for this tested case.
This experiment does not prove fidelity for Regression F (user-owned ambiguity), consistency across domains/phrasings, downstream reasoning preservation, or end-to-end production flow. Branch: `feature/reasoning-fidelity-v0.8`. Files: `tests/reconstruction/semantic-regression-e-evidence-vs-clarification.test.js` and `docs/experiment-56k.md`. Status pending Rob's review.