robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: test evidence needs across competing hypotheses

Browse Source
This commit is contained in:
robbond
2026-08-07 18:14:03 +01:00
parent 5daee5e911
commit 10e87d0d44
3 changed files with 497 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+3 -3
View File
@@ -141,8 +141,8 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54N. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54O. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 54N)
### Return-to-Work Note (Experiment 54O)
Experiment 54M showed two interpretations can expose their substantive disagreement without selecting a winner. Experiment 54N tested whether that disagreement materially changes what needs to be established next, using three fixed cases: pricing ambiguity (correct — model detected changed information need), paraphrase agreement (correct — model avoided false consequence), and competing causes (incorrect — model treated them as same-direction investigation). The model correctly refrained from choosing either interpretation as correct or generating an actual next question in any case. Same host/model retained; no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-disagreement-consequence.test.js`.
Experiment 54N exposed a candidate failure pattern where competing causes sharing one diagnostic goal were treated as needing the same information. 54O directly tested whether the model can distinguish evidence needs for such hypotheses. Results across delivery causes, paraphrased same-cause control, and website-sales causes all passed (3/3). Model distinguished staff-capacity from supplier evidence, recognised paraphrased same-cause as same evidence need, and distinguished pricing from technical checkout evidence. No winner or next question was generated. Same host/model retained; no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-hypothesis-evidence-needs.test.js`.