robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: validate evidence versus clarification routing

Browse Source
This commit is contained in:
robbond
2026-08-09 16:33:14 +01:00
parent f861e2cac0
commit ec398dcec9
2 changed files with 97 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+4
View File
@@ -235,3 +235,7 @@ One live Ollama call (14,032 ms) returned `uncertaintyType: "user_clarification_
**Classification: PASS.** The model correctly identified the ambiguity as user-owned, did not introduce spurious evidence gathering, and preserved the evidence-vs-user-meaning distinction cleanly.
This experiment does not prove consistency across repeated runs, fidelity for other regression cases (A–E, G+), behavior in production reasoning paths, or downstream integration with Behaviour Selection or the SituationGraph. Branch: `feature/reasoning-fidelity-v0.8`. Files: `tests/reconstruction/semantic-regression-f-user-owned-ambiguity.test.js` and `docs/experiment-56l.md`. Status pending Rob's review.
### Experiment 56M Summary — Production evidence vs clarification routing validation
Validated one production claim after Codex commit `f861e2c`: does the deterministic question-formulation boundary preserve the E/F distinction? No live Ollama calls were made (0). Deterministic `formulateQuestion()` was exercised with both regression fixtures. Regression E (competing delivery-delay causes: "Staff capacity may be the issue" / "Supplier lead times are likely responsible.") produced question: "What evidence would clarify possible causes of the delivery delay?" — reasoning pattern=diagnosis, strategy=evidence_gathering, template=diagnosis_evidence. PASS. Regression F (preference vs constraint ambiguity: "Whether avoiding additional risk is a hard constraint") produced question: "Is avoiding additional risk a hard constraint or a preference/trade-off?" — reasoning pattern=prioritisation, strategy=null, template=user_meaning_clarification, with rejected families correctly excluding all evidence-adjacent families. PASS. Both cases maintain their distinct routes: E on an evidence route and F on user clarification. All 19 existing tests continue to pass. Branch: `feature/reasoning-fidelity-v0.8`. File: `docs/experiment-56m.md`. Status pending Rob's review.