robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: probe user-owned ambiguity boundary

Browse Source
This commit is contained in:
robbond
2026-08-09 15:21:02 +01:00
parent 11882bfaae
commit e884b02e7c
3 changed files with 308 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+13 -1
View File
@@ -170,7 +170,7 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F, 56D–56H. Branch: `feature/reasoning-fidelity-v0.8`. Status pending Rob's review.*
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F, 56D–56H, 56L. Branch: `feature/reasoning-fidelity-v0.8`. Status pending Rob's review.*
### Return-to-Work Note (Experiment 55F)
@@ -223,3 +223,15 @@ One live Ollama call (18,580 ms) returned `uncertaintyType: "evidence_needed"` w
**Classification: PASS.** The model correctly identified the disagreement as requiring evidence rather than asking the user to settle an externally knowable question by clarification. It specified concrete, relevant evidence — demonstrating understanding of the causal structure rather than producing a generic classification. This confirms the model can preserve the distinction between "evidence needed to determine what is true" and "clarification needed because only the user can establish meaning/preference/intent/constraint" for this tested case.
This experiment does not prove fidelity for Regression F (user-owned ambiguity), consistency across domains/phrasings, downstream reasoning preservation, or end-to-end production flow. Branch: `feature/reasoning-fidelity-v0.8`. Files: `tests/reconstruction/semantic-regression-e-evidence-vs-clarification.test.js` and `docs/experiment-56k.md`. Status pending Rob's review.
### Experiment 56L Summary — User-owned ambiguity requires clarification, not evidence
Tested whether the configured live Ollama model (`qwen-claude:latest` at `http://192.168.1.111:11434`) recognises that a preference-vs-constraint distinction belongs to the user's own meaning and requires clarification rather than external evidence — Regression F from `docs/reasoning-refinement-requirements.md`.
Fixed case: "I want the business to grow, but I don't want to take on more risk." — user has not specified whether avoiding additional risk is a hard constraint or a strong preference/trade-off.
One live Ollama call (14,032 ms) returned `uncertaintyType: "user_clarification_needed"` with `evidenceNeeded: null` and specific `userClarificationNeeded` describing the non-negotiable-versus-trade-off distinction only the user can establish. Matches pre-written human reference exactly at category level.
**Classification: PASS.** The model correctly identified the ambiguity as user-owned, did not introduce spurious evidence gathering, and preserved the evidence-vs-user-meaning distinction cleanly.
This experiment does not prove consistency across repeated runs, fidelity for other regression cases (A–E, G+), behavior in production reasoning paths, or downstream integration with Behaviour Selection or the SituationGraph. Branch: `feature/reasoning-fidelity-v0.8`. Files: `tests/reconstruction/semantic-regression-f-user-owned-ambiguity.test.js` and `docs/experiment-56l.md`. Status pending Rob's review.