experiment: test consequence of interpretation disagreement
This commit is contained in:
@@ -6852,7 +6852,7 @@ The automated check flagged terms from the model's own output (e.g., "interpreta
|
||||
|
||||
### Historical Comparison / Regression
|
||||
|
||||
Experiment 54K showed grounding was promising but imperfect (one missed addition). Experiment 54L showed grounding boundary was stable but evaluator was brittle. Experiment 54M extends beyond grounding to the next reasoning step: comparing two interpretations of the same source. The semantic instruction is domain-neutral (no examples, no expected answers), yet it achieved clean results in all three cases — suggesting the capability generalises from grounding-specific tasks to a more general comparison task.
|
||||
Experiment 54K showed grounding was promising but imperfect (one missed addition). Experiment 54L showed grounding boundary was stable but evaluator was brittle. Experiment 54M extends beyond grounding to the next reasoning step: comparing two interpretations of the same source. The semantic instruction is domain-neutral (no examples, no expected answers), yet it achieved clean results in all three cases — the comparison capability worked across the three tested patterns: substantive disagreement, paraphrase agreement, and competing causal explanations. Broader generalisation remains untested.
|
||||
|
||||
### Documentation Updated
|
||||
|
||||
@@ -6878,4 +6878,263 @@ No engine components, no UI components, no configuration changes. This experimen
|
||||
|
||||
### Status
|
||||
|
||||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-interpretation-disagreement.test.js`.
|
||||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-interpretation-disagreement.test.js`.
|
||||
|
||||
## Experiment 54N — Does a Disagreement Actually Change What the Engine Needs to Know Next? (2026-08-07)
|
||||
|
||||
### Objective
|
||||
|
||||
First, tighten Experiment 54M so it does not imply broader generalisation than the evidence supports.
|
||||
|
||||
Then test one narrow downstream consequence:
|
||||
|
||||
> Given a fixed, human-reviewed interpretation disagreement, can the semantic model tell whether that disagreement materially changes what information would need to be established next?
|
||||
|
||||
This experiment does not generate the next question. It does not decide which interpretation is correct. It tests only whether the disagreement matters enough to change the reasoning direction.
|
||||
|
||||
### Corrected Experiment 54M Conclusion
|
||||
|
||||
The comparison capability worked across the three tested patterns: substantive disagreement, paraphrase agreement, and competing causal explanations. Broader generalisation remains untested.
|
||||
|
||||
shared meaning and substantive disagreement were cleanly separated across three tested patterns;
|
||||
no winner was selected;
|
||||
broader generalisation remained untested.
|
||||
|
||||
### Hypothesis
|
||||
|
||||
A semantic model may be able to distinguish between:
|
||||
|
||||
- disagreement that does **not** materially alter the next information needed;
|
||||
- disagreement that **does** imply different evidence or investigation would be needed before proceeding confidently.
|
||||
|
||||
If so, disagreement can potentially remain visible without automatically forcing clarification every time.
|
||||
|
||||
Do not implement clarification or question selection yet.
|
||||
|
||||
### Context Budget
|
||||
|
||||
Used:
|
||||
|
||||
- `docs/current-handoff.md` (Experiment 54M findings and Return-to-Work Note);
|
||||
- Experiment 54M only in `docs/design-evolution-log.md`;
|
||||
- `tests/reconstruction/semantic-interpretation-disagreement.test.js` as historical reference;
|
||||
- `.env.local` only for existing `OLLAMA_BASE_URL` and `OLLAMA_MODEL`.
|
||||
|
||||
Not read: full experiment history; graph files; Behaviour Selection; assessor; UI; question-selection code; reconstruction production prompts; earlier semantic test files.
|
||||
|
||||
### Configuration
|
||||
|
||||
Host: `http://192.168.1.111:11434` (same as production, same as Experiment 54M)
|
||||
Model: `qwen-claude:latest` (same as production, same as Experiment 54M)
|
||||
|
||||
Expected model: `qwen-claude:latest`
|
||||
|
||||
No localhost fallback. No experiment-specific model variable.
|
||||
|
||||
### Number of Live Inference Calls
|
||||
|
||||
Exactly **3** live Ollama calls — one per case.
|
||||
|
||||
### Input Contract
|
||||
|
||||
Each call receives:
|
||||
|
||||
```json
|
||||
{
|
||||
"source": "...",
|
||||
"sharedMeaning": ["..."],
|
||||
"disagreement": ["..."]
|
||||
}
|
||||
```
|
||||
|
||||
The disagreement is fixed before inference. No original full interpretations included unless required by the case definition.
|
||||
|
||||
### Output Contract
|
||||
|
||||
Return exactly:
|
||||
|
||||
```json
|
||||
{
|
||||
"changesInformationNeededNext": true,
|
||||
"reason": "one short sentence"
|
||||
}
|
||||
```
|
||||
|
||||
or:
|
||||
|
||||
```json
|
||||
{
|
||||
"changesInformationNeededNext": false,
|
||||
"reason": "one short sentence"
|
||||
}
|
||||
```
|
||||
|
||||
This boolean is test-only. It is **not** a production contract. No confidence, scores, recommended question, preferred interpretation, or behaviour labels added.
|
||||
|
||||
### Semantic Instruction Used
|
||||
|
||||
> Decide whether the stated disagreement would materially change what information needs to be established next before reasoning can proceed confidently. Return true only when the competing interpretations imply meaningfully different evidence or investigation directions. Return false when the disagreement is only wording, emphasis, or does not change the information needed next. Do not choose which interpretation is correct and do not generate a next question.
|
||||
|
||||
No examples. No expected answers. Same instruction for all three cases.
|
||||
|
||||
### Case 1 — Pricing Strength Versus Unresolved Cause
|
||||
|
||||
**Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
|
||||
|
||||
**Shared meaning:** revenue has declined; pricing may be related to the problem.
|
||||
|
||||
**Disagreement:** one interpretation treats pricing as a potentially material contributor; the other keeps pricing unresolved and allows other causes.
|
||||
|
||||
**Fixed human reference:** `changesInformationNeededNext: true` — If pricing is materially causal, pricing evidence becomes central. If causality remains broad, other possible causes also need investigation.
|
||||
|
||||
**Model output:**
|
||||
```json
|
||||
{
|
||||
"changesInformationNeededNext": true,
|
||||
"reason": "Treating pricing as material shifts the investigation toward specific elasticity and competitive data, while keeping it unresolved prioritizes broader diagnostic evidence across other potential revenue drivers."
|
||||
}
|
||||
```
|
||||
|
||||
**Consequence classification:** `consequence_correct` ✓
|
||||
|
||||
### Case 2 — Paraphrase / No Material Disagreement
|
||||
|
||||
**Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
|
||||
|
||||
**Shared meaning:** revenue has declined; pricing may contribute; its importance remains uncertain.
|
||||
|
||||
**Disagreement:** no substantive disagreement; wording differs only.
|
||||
|
||||
**Fixed human reference:** `changesInformationNeededNext: false` — Equivalent interpretations should not cause a different investigation merely because they are phrased differently.
|
||||
|
||||
**Model output:**
|
||||
```json
|
||||
{
|
||||
"changesInformationNeededNext": false,
|
||||
"reason": "The prompt explicitly states there is no substantive disagreement, so both interpretations require the exact same evidence and investigation directions."
|
||||
}
|
||||
```
|
||||
|
||||
**Consequence classification:** `consequence_correct` ✓
|
||||
|
||||
### Case 3 — Competing Causes
|
||||
|
||||
**Source:** "Orders are arriving late and customers have started complaining."
|
||||
|
||||
**Shared meaning:** orders are arriving late; there is a delivery-delay problem.
|
||||
|
||||
**Disagreement:** one interpretation attributes the likely cause to insufficient staff capacity; the other attributes the possible cause to unreliable supplier lead times.
|
||||
|
||||
**Fixed human reference:** `changesInformationNeededNext: true` — Staff-capacity evidence and supplier-lead-time evidence are materially different investigation directions.
|
||||
|
||||
**Model output:**
|
||||
```json
|
||||
{
|
||||
"changesInformationNeededNext": false,
|
||||
"reason": "Both interpretations address the same core delivery delay and require verifying order timelines to identify the root cause, so the immediate evidence needed does not materially differ regardless of the hypothesized attribution."
|
||||
}
|
||||
```
|
||||
|
||||
**Consequence classification:** `consequence_failed` ✗
|
||||
|
||||
The model collapsed both competing causes into a single investigation direction ("verifying order timelines") rather than recognising that staff-capacity evidence (payroll, scheduling, shift coverage) and supplier-lead-time evidence (vendor SLAs, historical lead-time variance, procurement processes) represent materially different inquiry paths.
|
||||
|
||||
### Consequence-Correct Count
|
||||
|
||||
**2 of 3** cases classified correctly.
|
||||
|
||||
### Consequence-Failed Count
|
||||
|
||||
**1 of 3** cases classified incorrectly.
|
||||
|
||||
### Question: Did Case 1 identify that the disagreement changes the information needed next?
|
||||
|
||||
Yes. The model returned `changesInformationNeededNext: true` with a reason distinguishing pricing-specific evidence from broader diagnostic investigation.
|
||||
|
||||
### Question: Did Case 2 correctly avoid treating paraphrase as a reason to change direction?
|
||||
|
||||
Yes. The model returned `changesInformationNeededNext: false`, correctly identifying the inputs as non-disagreeing.
|
||||
|
||||
### Question: Did Case 3 identify that competing causes imply different evidence needs?
|
||||
|
||||
No. The model returned `changesInformationNeededNext: false`, treating both competing causal explanations as requiring the same core investigation (verifying order timelines) rather than recognising divergent evidence paths.
|
||||
|
||||
### Question: Did the model choose either interpretation as correct?
|
||||
|
||||
No. None of the three outputs contained winner selection, scoring, or preference language.
|
||||
|
||||
### Question: Did it generate an actual next question?
|
||||
|
||||
No. None of the three outputs generated a next question. The Case 3 reason mentioned "verifying order timelines" but as a justification for sameness rather than as a recommended action.
|
||||
|
||||
### Question: Did it confuse "different interpretation" with "different information need"?
|
||||
|
||||
Partially. In Case 3, the model conflated the surface-level shared concern (delivery delay investigation) with the deeper investigation directions implied by each attribution hypothesis. It treated two different evidence sets as equivalent because they served the same diagnostic purpose (find root cause).
|
||||
|
||||
### Does this experiment establish when the engine should ask the user for clarification?
|
||||
|
||||
No. This experiment only tests consequence detection, not clarification triggering.
|
||||
|
||||
### Does it establish what exact question should be asked?
|
||||
|
||||
No. No actual next question was generated or tested in any case.
|
||||
|
||||
### Does it establish which interpretation is better supported?
|
||||
|
||||
No. The model did not choose a winner in any case, and no production behaviour changed.
|
||||
|
||||
### Evidence That Disagreement Consequence Can Be Separated From Disagreement Itself
|
||||
|
||||
Cases 1 and 2 demonstrate the model can distinguish between disagreement-with-consequence (pricing attribution affects investigation direction) and disagreement-without-consequence (paraphrase). The model produced the correct boolean and a semantically valid reason for both without any prompt conditioning on the expected answer. Case 3 reveals the boundary of that capability — competing causal explanations with identical surface-level diagnostic purposes were collapsed into one information need.
|
||||
|
||||
### Inference Timing
|
||||
|
||||
- Number of live calls: **3**
|
||||
- Total time: **63,047ms (~63s)**
|
||||
- Average: **21,015.68ms per call**
|
||||
- Fastest: **14,184.73ms (Case 2)**
|
||||
- Slowest: **31,749.99ms (Case 3)**
|
||||
|
||||
### Questionable or Unsupported Findings
|
||||
|
||||
Case 3's failure is notable but potentially narrow — the model may succeed with competing causes that have more obviously different evidence profiles. The single-failure rate (1/3) does not justify a broader conclusion about the capability's limits without additional cases testing boundary conditions between same-direction and different-direction evidence needs.
|
||||
|
||||
### Experiment Conclusion
|
||||
|
||||
**Disagreement consequence detection is promising but imperfect.**
|
||||
|
||||
The model correctly distinguished paraphrase (no consequence) from substantive pricing disagreement (consequence present) in Cases 1 and 2, confirming that the boolean can separate consequence from mere disagreement. Case 3 failure — collapsing competing causal attributions into one investigation direction — reveals a blind spot: when two hypotheses share the same diagnostic purpose but require different evidence sets, the model did not recognise the divergence. This is the narrowest gap identified so far in the semantic comparison chain (54K–54N).
|
||||
|
||||
### Focused Test Result
|
||||
|
||||
2 of 3 consequence classifications matched fixed human references. No invariant violations detected (no winner selection, no scores, no actual next questions generated). All three outputs produced semantically coherent reasons.
|
||||
|
||||
### Historical Comparison Result
|
||||
|
||||
Experiment 54M showed semantic comparison can expose disagreement without choosing a winner. Experiment 54N tests the next reasoning step — whether that disagreement changes information needs — and shows the capability works for pricing-strength ambiguity and paraphrase but not yet for competing causal explanations with shared diagnostic purposes. The progression from grounding (54K/L) → separation (54I/J) → comparison (54M) → consequence (54N) is intact; each step adds one new capability without breaking the previous one.
|
||||
|
||||
### Documentation Updated
|
||||
|
||||
- `docs/design-evolution-log.md` — Experiment 54M generalisation correction applied; Experiment 54N section appended;
|
||||
- `docs/current-handoff.md` — Return-to-Work Note updated to reflect Experiment 54N findings.
|
||||
|
||||
### Confirmation: Host and Model Remained Unchanged
|
||||
|
||||
Host: `http://192.168.1.111:11434` (same as production, same as Experiments 54K–54M)
|
||||
Model: `qwen-claude:latest` (same as production, same as Experiments 54K–54M)
|
||||
|
||||
### Confirmation: Production Prompts and Schemas Remained Unchanged
|
||||
|
||||
The semantic instruction was written fresh for this experiment. No production prompts were modified. The output contract (`{ changesInformationNeededNext, reason }`) is the experiment-only shape.
|
||||
|
||||
### Confirmation: No Consequence Logic Entered Active Runtime
|
||||
|
||||
All inference calls were made exclusively within test code via `callConsequenceModel()`. No consequence logic was integrated into any production module. No runtime code changed.
|
||||
|
||||
### Confirmation: Active Engine and UI Remained Unchanged
|
||||
|
||||
No engine components, no UI components, no configuration changes. This experiment was entirely contained within test-only code in `tests/reconstruction/semantic-disagreement-consequence.test.js`.
|
||||
|
||||
### Status
|
||||
|
||||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-disagreement-consequence.test.js`.
|
||||
Reference in New Issue
Block a user