robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: test semantic interpretation disagreement

Browse Source
This commit is contained in:
robbond
2026-08-07 17:30:10 +01:00
parent fb5368ec5f
commit b9f737a293
3 changed files with 630 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+7 -3
View File
@@ -83,6 +83,10 @@ Experiment 54J proved the representation can separate source-supported from inte
Experiment 54K tested whether the configured semantic model (`qwen-claude:latest` on `192.168.1.111:11434`) can perform that grounding automatically. Three live Ollama calls (total ~96s): Case 1 (strengthening detection) = grounding_correct, Case 2 (multi-addition interpretation) = partial_grounding (missed one addition), Case 3 (faithful restatement control) = grounding_correct. Interpretation-added meaning did NOT leak into source-supported meaning in any case. One source-supported content gap: model missed "alternative causes" on the added side of Case 2. Automated semantic grounding is promising but imperfect — directionally viable but needs refinement before production use. Winner selection and downstream questions remain untested. No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-interpretation-grounding.test.js`.
Experiment 54L repeated two identical grounding cases three times each to test stability across six live calls. The source-versus-added boundary was perfectly stable (zero leakage in all runs). Detection completeness appeared variable but manual analysis showed the instability came from the automated evaluator's paraphrase sensitivity, not the model itself. Case A strengthening identified in all 3 runs; Case B "other causes" and "not established as main problem" each identified in all 3 runs. Status pending Rob's review.
Experiment 54M tested whether two interpretations of one source can expose their substantive disagreement without deciding which is correct. Three live Ollama calls across three cases: real pricing attribution difference, paraphrase identity control, and competing causal explanations. All three classified as disagreement_correct by human semantic review. Paraphrase was correctly treated as agreement; shared meaning stayed separate; no invented disagreement or winner selection occurred. Status pending Rob's review.
## 5. What Remains Open
- The `too_broad` boundary sits exactly between three and four active unknowns; it is mechanically clear but conceptually uncertain — whether it aligns with genuine user confusion requires real-scenario validation;
@@ -135,8 +139,8 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54L. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54M. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 54L)
### Return-to-Work Note (Experiment 54M)
Experiment 54K showed automatic grounding was promising but incomplete — two of three cases were correct, one missed an addition. Experiment 54L repeated two identical grounding cases three times each to test stability. The source-versus-added boundary remained stable (zero leakage across all six runs). Detection completeness was variable — the model did not consistently identify every added concept on repeat calls, even when it correctly kept everything on its proper side. Correctness ranking and downstream questions remain untested. Same host/model retained; no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-grounding-stability.test.js`.
Experiment 54L was corrected: model grounding was materially stable across six repeated identical inputs; evaluator wording sensitivity caused the apparent classification instability. Experiment 54M tested whether two interpretations of one source can expose their substantive disagreement without deciding which is correct. Three live calls across three cases: real pricing attribution difference, paraphrase identity control, and competing causal explanations. All three classified as disagreement_correct by human semantic review. Paraphrase was correctly treated as agreement; shared meaning stayed separate; no invented disagreement or winner selection occurred. Same host/model retained; no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-interpretation-disagreement.test.js`.