robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: test consequence of interpretation disagreement

Browse Source
This commit is contained in:
robbond
2026-08-07 18:01:47 +01:00
parent b9f737a293
commit 5daee5e911
3 changed files with 507 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+6 -4
View File
@@ -85,7 +85,9 @@ Experiment 54K tested whether the configured semantic model (`qwen-claude:latest
Experiment 54L repeated two identical grounding cases three times each to test stability across six live calls. The source-versus-added boundary was perfectly stable (zero leakage in all runs). Detection completeness appeared variable but manual analysis showed the instability came from the automated evaluator's paraphrase sensitivity, not the model itself. Case A strengthening identified in all 3 runs; Case B "other causes" and "not established as main problem" each identified in all 3 runs. Status pending Rob's review.
Experiment 54M tested whether two interpretations of one source can expose their substantive disagreement without deciding which is correct. Three live Ollama calls across three cases: real pricing attribution difference, paraphrase identity control, and competing causal explanations. All three classified as disagreement_correct by human semantic review. Paraphrase was correctly treated as agreement; shared meaning stayed separate; no invented disagreement or winner selection occurred. Status pending Rob's review.
Experiment 54M tested whether two interpretations of one source can expose their substantive disagreement without deciding which is correct. Three live Ollama calls across three cases: real pricing attribution difference, paraphrase identity control, and competing causal explanations. All three classified as disagreement_correct by human semantic review. Paraphrase was correctly treated as agreement; shared meaning stayed separate; no invented disagreement or winner selection occurred. The comparison capability worked across the three tested patterns: substantive disagreement, paraphrase agreement, and competing causal explanations. Broader generalisation remains untested. Status pending Rob's review.
Experiment 54N tested whether an interpretation disagreement can be judged for material consequence on downstream information needs without generating a next question or choosing a winner. Three fixed cases: pricing ambiguity (consequence_correct), paraphrase identity control (consequence_correct), competing causes (consequence_failed — model returned false, missing that staff-capacity vs supplier evidence represent divergent investigation directions). 2/3 correct. Model did not choose a winner or generate an actual next question in any case. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-disagreement-consequence.test.js`.
## 5. What Remains Open
@@ -139,8 +141,8 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54M. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54N. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 54M)
### Return-to-Work Note (Experiment 54N)
Experiment 54L was corrected: model grounding was materially stable across six repeated identical inputs; evaluator wording sensitivity caused the apparent classification instability. Experiment 54M tested whether two interpretations of one source can expose their substantive disagreement without deciding which is correct. Three live calls across three cases: real pricing attribution difference, paraphrase identity control, and competing causal explanations. All three classified as disagreement_correct by human semantic review. Paraphrase was correctly treated as agreement; shared meaning stayed separate; no invented disagreement or winner selection occurred. Same host/model retained; no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-interpretation-disagreement.test.js`.
Experiment 54M showed two interpretations can expose their substantive disagreement without selecting a winner. Experiment 54N tested whether that disagreement materially changes what needs to be established next, using three fixed cases: pricing ambiguity (correct — model detected changed information need), paraphrase agreement (correct — model avoided false consequence), and competing causes (incorrect — model treated them as same-direction investigation). The model correctly refrained from choosing either interpretation as correct or generating an actual next question in any case. Same host/model retained; no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-disagreement-consequence.test.js`.