experiment: test semantic interpretation disagreement

2026-08-07 17:30:10 +01:00
parent fb5368ec5f
commit b9f737a293
3 changed files with 630 additions and 7 deletions
@@ -83,6 +83,10 @@ Experiment 54J proved the representation can separate source-supported from inte

 Experiment 54K tested whether the configured semantic model (`qwen-claude:latest` on `192.168.1.111:11434`) can perform that grounding automatically. Three live Ollama calls (total ~96s): Case 1 (strengthening detection) = grounding_correct, Case 2 (multi-addition interpretation) = partial_grounding (missed one addition), Case 3 (faithful restatement control) = grounding_correct. Interpretation-added meaning did NOT leak into source-supported meaning in any case. One source-supported content gap: model missed "alternative causes" on the added side of Case 2. Automated semantic grounding is promising but imperfect — directionally viable but needs refinement before production use. Winner selection and downstream questions remain untested. No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-interpretation-grounding.test.js`.

+Experiment 54L repeated two identical grounding cases three times each to test stability across six live calls. The source-versus-added boundary was perfectly stable (zero leakage in all runs). Detection completeness appeared variable but manual analysis showed the instability came from the automated evaluator's paraphrase sensitivity, not the model itself. Case A strengthening identified in all 3 runs; Case B "other causes" and "not established as main problem" each identified in all 3 runs. Status pending Rob's review.
+
+Experiment 54M tested whether two interpretations of one source can expose their substantive disagreement without deciding which is correct. Three live Ollama calls across three cases: real pricing attribution difference, paraphrase identity control, and competing causal explanations. All three classified as disagreement_correct by human semantic review. Paraphrase was correctly treated as agreement; shared meaning stayed separate; no invented disagreement or winner selection occurred. Status pending Rob's review.
+
 ## 5. What Remains Open

 - The `too_broad` boundary sits exactly between three and four active unknowns; it is mechanically clear but conceptually uncertain — whether it aligns with genuine user confusion requires real-scenario validation;
@@ -135,8 +139,8 @@ Answer before continuing:

 ---

-*Created by Experiment 34. Updated by Experiments 38–53, 54A–54L. Branch: `feature/user-workspace-ux-v0.7`.*
+*Created by Experiment 34. Updated by Experiments 38–53, 54A–54M. Branch: `feature/user-workspace-ux-v0.7`.*

-### Return-to-Work Note (Experiment 54L)
+### Return-to-Work Note (Experiment 54M)

-Experiment 54K showed automatic grounding was promising but incomplete — two of three cases were correct, one missed an addition. Experiment 54L repeated two identical grounding cases three times each to test stability. The source-versus-added boundary remained stable (zero leakage across all six runs). Detection completeness was variable — the model did not consistently identify every added concept on repeat calls, even when it correctly kept everything on its proper side. Correctness ranking and downstream questions remain untested. Same host/model retained; no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-grounding-stability.test.js`.
+Experiment 54L was corrected: model grounding was materially stable across six repeated identical inputs; evaluator wording sensitivity caused the apparent classification instability. Experiment 54M tested whether two interpretations of one source can expose their substantive disagreement without deciding which is correct. Three live calls across three cases: real pricing attribution difference, paraphrase identity control, and competing causal explanations. All three classified as disagreement_correct by human semantic review. Paraphrase was correctly treated as agreement; shared meaning stayed separate; no invented disagreement or winner selection occurred. Same host/model retained; no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-interpretation-disagreement.test.js`.
@@ -6570,9 +6570,7 @@ The model showed **variable completeness**:

 ### Does Repeated Identical Input Produce Materially Identical Grounding?

-**No — not at the conceptual completeness level.** The model's additions were detected inconsistently across runs. Some runs captured all expected additions; others missed one or more. The boundary was stable but the detection was variable.
-
-At the semantic coverage level, Case B showed remarkably consistent concept capture (3/3 for both additions). But this was only evident through manual analysis — the automated evaluation flagged instability because of phrasing variation.
+**Repeated identical inputs produced materially consistent grounding across the six tested calls**, although wording varied enough to expose brittleness in the automated evaluation method. At the semantic coverage level, Case B showed perfect concept capture (3/3 for both additions) in all three runs. The automation flagged instability because it relied on paraphrase-sensitive checks — the model's output was semantically correct even when phrasing differed.

 ### Does This Establish Which Interpretation Is Better?

@@ -6603,13 +6601,25 @@ At the semantic coverage level, Case B showed remarkably consistent concept capt

 ### Evaluation Conclusion

-**Grounding boundary is stable but addition detection varies.** The model consistently preserved the source-versus-inference distinction across all six repeated identical inputs (zero leakage in either direction). However, the completeness of interpretation-added detection was variable — the model did not consistently identify every added concept on repeat calls. This suggests the grounding boundary itself is a stable capability, while addition detection is less reliable and may require multiple passes or additional prompting to achieve consistent coverage.
+**Semantic grounding was materially stable across the tested repeats.** Across all six runs, source-supported and interpretation-added meaning remained on the correct sides. The apparent classification instability came primarily from the automated evaluator being sensitive to paraphrase. The model consistently preserved the source-versus-inference distinction (zero leakage in either direction). Case B showed perfect concept coverage in all three runs when evaluated semantically; the automation flagged "instability" because of phrasing variation that was not material.

 ### Focused Test Result

 **Boundary stability:** Stable across all 6 runs (0 leaks in either direction).
 **Addition detection stability:** Variable — Case A varied between partial and correct; Case B showed variable classification despite consistent semantic coverage when evaluated manually.

+### Required Specific Findings (Preserved)
+
+- Case A strengthening was identified in all 3 runs;
+- Case B "other causes" was identified in all 3 runs;
+- Case B "not established as main problem" was identified in all 3 runs;
+- no interpretation-added meaning leaked into supportedBySource;
+- no source-supported meaning moved into addedByInterpretation.
+
+### Corrected Conclusion
+
+**Semantic grounding was materially stable in the tested repeats; evaluator brittleness caused the apparent classification instability.** The model's semantic grounding held firm across six repeated identical inputs with zero leakage. The automated test flagged partial/fail classifications where the model had correctly preserved meaning semantically — the failures were in the evaluation logic's sensitivity to phrasing, not in the model's ability to separate source-supported from interpretation-added meaning.
+
 ### Regression / Historical Comparison

 Experiment 54K was not re-run live (its test lacks timeout configuration and would trigger three additional inference calls). Committed historical results from Experiment 54K remain the comparison:
@@ -6642,3 +6652,230 @@ No engine components, no UI components, no configuration changes. This experimen
 ### Status

 **Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-grounding-stability.test.js`.
+
+## Experiment 54M — Can Two Grounded Interpretations Expose Their Actual Point of Disagreement? (2026-08-07)
+
+### Objective
+
+Correct Experiment 54L's conclusion to reflect that the model was stable and the evaluator was brittle. Then test:
+
+> **Given one exact source and two different interpretations of it, can the semantic model identify what the interpretations actually disagree about without deciding which one is correct?**
+
+Passive test-only experiment. No ranking, no confidence assignment, no next question generation.
+
+### Hypothesis
+
+The model may identify the smallest meaningful disagreement between A and B while keeping shared meaning separate. Failure modes to record: inventing disagreement absent from both interpretations, collapsing shared meaning into disagreement, or silently choosing one interpretation.
+
+### Context Used
+
+- `docs/current-handoff.md` (Experiment 54L findings);
+- Experiment 54L only in `docs/design-evolution-log.md`;
+- Existing `tests/reconstruction/semantic-grounding-stability.test.js` (structure pattern);
+- Existing `tests/reconstruction/interpretation-source-grounding.test.js` (structure pattern);
+- Existing `.env.local` configuration.
+
+### Configured Host and Model
+
+- **Ollama host:** `http://192.168.1.111:11434` (unchanged from production);
+- **Model:** `qwen-claude:latest` (unchanged from production).
+
+### Semantic Output Contract
+
+```json
+{
+  "sharedMeaning": ["..."],
+  "disagreement": ["..."]
+}
+```
+
+No explanation field. No winner. No score. No confidence.
+
+### Semantic Instruction
+
+> Compare two interpretations of the same source. Put meaning that both interpretations materially share into "sharedMeaning". Put only the smallest substantive points where the interpretations differ into "disagreement". Do not decide which interpretation is correct. Do not add facts that are absent from both interpretations.
+
+No examples provided. No expected answers mentioned.
+
+### Three Fixed Comparison Cases
+
+#### Case 1 — Real 54I disagreement
+
+- **Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
+- **Interpretation A:** "Pricing may be contributing materially to the revenue decline."
+- **Interpretation B:** "The revenue decline may have causes other than pricing, and pricing has not yet been established as the main problem."
+- **Fixed human reference — shared meaning:** revenue has declined; pricing may be related to the problem.
+- **Fixed human reference — disagreement:** A strengthens pricing toward material contribution; B keeps pricing unresolved and allows other causes.
+
+#### Case 2 — Same meaning, paraphrased
+
+- **Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
+- **Interpretation A:** "Pricing could be contributing to the revenue decline, but its importance is uncertain."
+- **Interpretation B:** "Pricing may play some role in the fall in revenue, although we do not yet know how important that role is."
+- **Fixed human reference — shared meaning:** revenue has declined; pricing may contribute; importance remains uncertain.
+- **Fixed human reference — disagreement:** none materially.
+
+#### Case 3 — Clear competing explanations
+
+- **Source:** "Orders are arriving late and customers have started complaining."
+- **Interpretation A:** "Delivery delays are probably being caused by insufficient staff capacity."
+- **Interpretation B:** "Delivery delays may instead be caused by unreliable supplier lead times."
+- **Fixed human reference — shared meaning:** orders are arriving late; there is a delivery-delay problem.
+- **Fixed human reference — disagreement:** A attributes the likely cause to staff capacity; B attributes the possible cause to supplier lead times.
+
+The source itself establishes neither cause.
+
+### Live Results
+
+#### Case 1
+
+**sharedMeaning:**
+- "Revenue is declining."
+- "Pricing may be a factor in the revenue decline."
+- "There is uncertainty about pricing's exact role."
+
+**disagreement:**
+- "Interpretation A suggests pricing may be a significant material contributor to the decline."
+- "Interpretation B emphasizes that other causes are possible and pricing has not been confirmed as the primary cause."
+
+**Human semantic review classification: disagreement_correct** ✓
+
+Shared meaning captured revenue decline, pricing's potential role, and uncertainty. Disagreement correctly identified A's stronger attribution (material contributor) versus B's unresolved framing (other causes possible, not confirmed as primary cause).
+
+#### Case 2
+
+**sharedMeaning:**
+- "Revenue has declined."
+- "Pricing may be a contributing factor to the decline."
+- "The certainty regarding pricing's specific impact is unclear."
+
+**disagreement:**
+- "No substantive disagreement exists; both interpretations convey identical core meanings with only minor phrasing variations."
+
+**Human semantic review classification: disagreement_correct** ✓
+
+Shared meaning captured all three reference items (revenue decline, pricing contribution, uncertainty about importance). Disagreement correctly identified no material disagreement and explicitly noted the output as phrasing variation rather than substantive difference. Paraphrase was NOT treated as disagreement.
+
+#### Case 3
+
+**sharedMeaning:**
+- "Orders are arriving late."
+- "Customers have started complaining."
+
+**disagreement:**
+- "Interpretation A identifies insufficient staff capacity as the cause, while Interpretation B identifies unreliable supplier lead times as the cause."
+
+**Human semantic review classification: disagreement_correct** ✓
+
+Shared meaning captured both source facts (orders late, customers complaining). Disagreement correctly identified the two competing causal interpretations (staff capacity vs. supplier lead times).
+
+### Evaluation Summary
+
+| Case | Human Classification | Notes |
+|------|---------------------|-------|
+| Case 1 — pricing attribution | disagreement_correct | Core disagreement identified |
+| Case 2 — paraphrase control | disagreement_correct | No false disagreement; phrasing correctly dismissed |
+| Case 3 — competing causes | disagreement_correct | Both causal alternatives correctly isolated |
+
+- **Disagreement-correct count: 3/3**
+- **Partial-disagreement count: 0**
+- **Disagreement-failed count: 0**
+
+### Invented Disagreement Check
+
+The automated check flagged terms from the model's own output (e.g., "interpretation", "suggests") as potentially invented because they did not appear in the raw source or interpretations. Manual semantic review confirmed: these are descriptive framing terms, not new facts. **No invented disagreement was identified.**
+
+### Did Shared Meaning Remain Separate from Disagreement?
+
+**Yes.** Across all three cases, shared meaning items appeared only in `sharedMeaning`, and disagreement items appeared only in `disagreement`. No cross-contamination observed.
+
+### Did Paraphrase Create False Disagreement?
+
+**No.** Case 2 explicitly returned "No substantive disagreement exists" — paraphrase was correctly treated as agreement, not disagreement.
+
+### Did the Model Identify the Pricing-Attribution Disagreement (Case 1)?
+
+**Yes.** The model identified that A strengthens pricing toward material contribution and B keeps pricing unresolved while allowing other causes. This matches the fixed human reference exactly in substance.
+
+### Did the Model Identify the Competing-Cause Disagreement (Case 3)?
+
+**Yes.** The model correctly isolated "insufficient staff capacity" versus "unreliable supplier lead times" as the competing causal interpretations — both absent from the source but present in the respective interpretations.
+
+### Did the Model Invent Disagreement Absent from Both Interpretations?
+
+**No.** Manual review confirmed all disagreement items trace to content present in at least one of the two interpretations. No new causes or facts were introduced.
+
+### Did the Model Choose a Winner?
+
+**No.** Neither `winner`, `score`, `confidence`, nor any equivalent field was returned. Both interpretations were treated symmetrically.
+
+### Does This Establish Which Interpretation Is Better Supported?
+
+**No.** Experiment 54M exposed disagreement without resolving it. No scoring, ranking, or preference was applied.
+
+### Does This Establish Whether Clarification Is Required?
+
+**No.** Exposure of disagreement is informational, not prescriptive. Clarification decisions remain untested.
+
+### Was Downstream Question Selection Tested?
+
+**No.** The model produced no downstream questions. That capability remains untested in this experiment.
+
+### Inference Timing
+
+| Metric | Value |
+|--------|-------|
+| Number of live inference calls | 3 |
+| Total inference time | 73,194ms (73.2s) |
+| Average | 24,398ms (24.4s) per call |
+| Fastest | 16,885ms (16.9s) — Case 3 |
+| Slowest | 29,495ms (29.5s) — Case 2 |
+
+### Limitations
+
+- Three cases only: one pricing attribution case, one paraphrase control, one competing-causes case; no cross-domain validation beyond revenue and logistics;
+- Single model (`qwen-claude:latest`) on single host; results may differ with other models or hosts;
+- Automated evaluation flagged false positives due to phrasing-sensitive keyword checks — semantic correctness was determined by human-reference review only;
+- Shared meaning in Case 3 included "customers complaining" (not in the fixed reference) but is still materially shared between both interpretations. This is semantically valid and not a defect;
+- No measurement of edge cases: near-synonymous disagreements, three-way comparisons, or ambiguous disagreement boundaries were tested.
+
+### Evaluation Conclusion
+
+**Semantic comparison cleanly exposes interpretation disagreement in the tested cases.** Across all three controlled scenarios — real disagreement, paraphrase identity, and competing causal explanations — the model correctly separated shared meaning from substantive disagreement in every case. Paraphrase was not confused with disagreement. No invented disagreement or silent winner selection occurred. Same host/model retained; no production behaviour changed. What remains untested: scalability beyond two interpretations per source, reliability across more diverse domains, and whether this capability holds under repeated runs.
+
+### Focused Test Result
+
+**Shared-meaning separation:** Clean across all 3 cases (0 cross-contamination).
+**Disagreement identification:** Correct in all 3 tested scenarios.
+**Paraphrase handling:** Correct — no false disagreement generated.
+**Invention check:** No invented disagreement detected by manual review.
+
+### Historical Comparison / Regression
+
+Experiment 54K showed grounding was promising but imperfect (one missed addition). Experiment 54L showed grounding boundary was stable but evaluator was brittle. Experiment 54M extends beyond grounding to the next reasoning step: comparing two interpretations of the same source. The semantic instruction is domain-neutral (no examples, no expected answers), yet it achieved clean results in all three cases — suggesting the capability generalises from grounding-specific tasks to a more general comparison task.
+
+### Documentation Updated
+
+- `docs/design-evolution-log.md` — Experiment 54L corrections applied; Experiment 54M section appended;
+- `docs/current-handoff.md` — Return-to-Work Note updated to reflect Experiment 54M findings.
+
+### Confirmation: Host and Model Remained Unchanged
+
+Host: `http://192.168.1.111:11434` (same as production, same as Experiment 54K, same as Experiment 54L)
+Model: `qwen-claude:latest` (same as production, same as Experiment 54K, same as Experiment 54L)
+
+### Confirmation: Production Prompts and Schemas Remained Unchanged
+
+The semantic instruction was written fresh for this experiment. No production prompts were modified. The output contract (`{ sharedMeaning, disagreement }`) is the experiment-only shape.
+
+### Confirmation: No Disagreement Logic Entered Active Runtime
+
+All inference calls were made exclusively within test code via `callDisagreementModel()`. No disagreement logic was integrated into any production module. No runtime code changed.
+
+### Confirmation: Active Engine and UI Remained Unchanged
+
+No engine components, no UI components, no configuration changes. This experiment was entirely contained within test-only code in `tests/reconstruction/semantic-interpretation-disagreement.test.js`.
+
+### Status
+
+**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-interpretation-disagreement.test.js`.