experiment: test consequence from explicit evidence needs

docs/current-handoff.md

@@ -141,8 +141,8 @@ Answer before continuing:
 
 ---
 
-*Created by Experiment 34. Updated by Experiments 38–53, 54A–54O. Branch: `feature/user-workspace-ux-v0.7`.*
+*Created by Experiment 34. Updated by Experiments 38–53, 54A–54P. Branch: `feature/user-workspace-ux-v0.7`.*
 
-### Return-to-Work Note (Experiment 54O)
+### Return-to-Work Note (Experiment 54P)
 
-Experiment 54N exposed a candidate failure pattern where competing causes sharing one diagnostic goal were treated as needing the same information. 54O directly tested whether the model can distinguish evidence needs for such hypotheses. Results across delivery causes, paraphrased same-cause control, and website-sales causes all passed (3/3). Model distinguished staff-capacity from supplier evidence, recognised paraphrased same-cause as same evidence need, and distinguished pricing from technical checkout evidence. No winner or next question was generated. Same host/model retained; no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-hypothesis-evidence-needs.test.js`.
+Experiment 54N failed on a higher-level consequence judgement for competing causes sharing one diagnostic goal. Experiment 54O showed the underlying evidence distinction (staff vs supplier) was available when asked directly. Experiment 54P tested whether making those evidence needs explicit restores the consequence judgement; results across three cases all passed. No winner or next question was generated in either experiment. Same host/model retained; no production behaviour changed. Whether this explains the 54N failure or only narrows it remains open. Branch: `feature/user-workspace-ux-v0.7`. First test/file to inspect when resuming: `tests/reconstruction/semantic-evidence-consequence.test.js`.

docs/design-evolution-log.md

@@ -7151,7 +7151,7 @@ This is a passive test-only experiment. Do not generate a next question. Do not
 
 ### Hypothesis
 
-The model may be able to distinguish evidence needs correctly when asked directly about evidence rather than about the broader "information needed next" consequence. If it still collapses different hypotheses into one evidence need, the 54N failure pattern becomes stronger evidence. If it distinguishes them cleanly, the 54N failure may have been caused by the abstraction level of the consequence question rather than inability to understand the evidence difference.
+The model may be able to distinguish evidence needs correctly when asked directly about evidence rather than about the broader "information needed next" consequence. If it still collapses different hypotheses into one evidence need, the 54N failure pattern becomes stronger evidence. If it distinguishes them cleanly, this shows the model can represent the distinction at one level — whether the same capability exists at another remains unproven.
 
 ### Context Budget
 
@@ -7344,9 +7344,9 @@ In 54N (abstraction level: "does the disagreement change what information needs
 
 In 54O (abstraction level: "what evidence is needed to investigate each hypothesis?"), the same hypotheses now correctly return `sameEvidenceNeeded: false` with distinct evidence lists.
 
-This suggests the 54N failure was partly caused by the abstraction level of the consequence question rather than an inability to understand the evidence difference. When asked directly about evidence, the model distinguished the competing causes cleanly.
+The same model succeeded when the problem was expressed as direct evidence comparison and failed previously when expressed as a higher-level consequence judgement. The reason for that difference remains unproven.
 
-However, the 54N result still stands as a candidate failure pattern in practice: if the engine asks "does this disagreement change what information needs to be established next?" (rather than asking for evidence comparison), it may still collapse the hypotheses. The question is whether that abstraction level is what the engine actually uses downstream.
+However, the 54N result still stands as a candidate failure pattern in practice: if the engine asks "does this disagreement change what information needs to be established next?" (rather than asking for evidence comparison), it may still collapse the hypotheses.
 
 ### Inference Timing
 
@@ -7362,9 +7362,9 @@ The 3/3 pass rate is strong but comes from only three cases. The test deliberate
 
 ### Experiment Conclusion
 
-**The model distinguished same-goal hypotheses by their evidence needs in all tested cases.**
+**The model distinguished same-goal hypotheses by their evidence needs in all three tested cases; robustness beyond these cases remains untested.**
 
-All three classifications matched fixed human references. The model returned the correct boolean for same-evidence and different-evidence cases, identified materially appropriate evidence for each hypothesis, and avoided winner selection and question generation in every case. This result strengthens confidence that the underlying semantic understanding of evidence differences exists — the 54N failure may have been an artifact of asking at a higher abstraction level (information need change) rather than at the direct evidence level.
+All three classifications matched fixed human references. The model returned the correct boolean for same-evidence and different-evidence cases, identified materially appropriate evidence for each hypothesis, and avoided winner selection and question generation in every case. The same model succeeded when the problem was expressed as direct evidence comparison and failed previously when expressed as a higher-level consequence judgement. The reason for that difference remains unproven.
 
 ### Focused Test Result
 
@@ -7372,7 +7372,7 @@ All three classifications matched fixed human references. The model returned the
 
 ### Historical Comparison Result
 
-Experiment 54N showed that asking "does this disagreement change what information needs to be established next?" fails on competing causal explanations sharing a diagnostic purpose (Case 3). Experiment 54O shows that asking directly "do these hypotheses require the same evidence?" succeeds on the same hypotheses plus two additional cases. The progression from consequence detection (54N) → direct evidence comparison (54O) reveals that the capability may be present at one abstraction level but lost at another.
+Experiment 54N showed that asking "does this disagreement change what information needs to be established next?" fails on competing causal explanations sharing a diagnostic purpose (Case 3). Experiment 54O shows that asking directly "do these hypotheses require the same evidence?" succeeds on the same hypotheses plus two additional cases. The progression from consequence detection (54N) → direct evidence comparison (54O) shows what the model can do when asked about evidence directly; whether the model can use that distinction to recover a higher-level consequence judgement remains unknown.
 
 ### Documentation Updated
 
@@ -7398,4 +7398,212 @@ No engine components, no UI components, no configuration changes. This experimen
 
 ### Status
 
-**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-hypothesis-evidence-needs.test.js`.
+**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-hypothesis-evidence-needs.test.js`.
+## Experiment 54P — Can Explicit Evidence Needs Recover the Higher-Level Consequence? (2026-08-07)
+
+### Objective
+
+Experiment 54N asked "Does this disagreement change what information is needed next?" and failed on competing causal explanations sharing one diagnostic goal (Case 3: staff capacity vs supplier lead time). Experiment 54O showed the same model correctly distinguished the evidence needs when asked directly about evidence.
+
+This experiment tests one narrow bridge: if the differing evidence needs are already made explicit, can the model correctly infer whether the disagreement changes what must be established before reasoning can proceed confidently?
+
+Do not generate a next question. Do not choose a winning hypothesis. Do not change production behaviour.
+
+### Hypothesis
+
+If the model is given the evidence needs explicitly, it may correctly infer whether the disagreement changes what must be established next. If it still fails, the problem is likely not simply losing the evidence distinction. If it succeeds, that shows the higher-level consequence can be recovered when the intermediate evidence distinction is available.
+
+### Context Budget
+
+Read only:
+- `docs/current-handoff.md`;
+- Experiment 54N and 54O in `docs/design-evolution-log.md`;
+- `tests/reconstruction/semantic-hypothesis-evidence-needs.test.js` as historical reference;
+- `tests/reconstruction/semantic-disagreement-consequence.test.js` as historical reference;
+- `.env.local` only for existing `OLLAMA_BASE_URL` and `OLLAMA_MODEL`.
+
+Not read: full experiment history; graph files; assessor; Behaviour Selection; UI; question-selection code; production prompts.
+
+### Configuration
+
+Host: `http://192.168.1.111:11434` (same as production, same as Experiments 54K–54O)
+Model: `qwen-claude:latest` (same as production, same as Experiments 54K–54O)
+
+No localhost fallback. No experiment-specific model variable.
+
+### Number of Live Inference Calls
+
+Exactly **3** live Ollama calls — one per case.
+
+### Input Contract
+
+Each call receives:
+```json
+{
+  "problem": "...",
+  "evidenceForA": ["..."],
+  "evidenceForB": ["..."]
+}
+```
+
+The evidence lists are fixed human-reviewed inputs, not generated during this experiment.
+
+### Output Contract
+
+Return exactly:
+```json
+{
+  "changesInformationNeededNext": true | false,
+  "reason": "one short sentence explaining why"
+}
+```
+
+No next question, no preferred hypothesis, no confidence, no score.
+
+### Semantic Instruction Used
+
+> Decide whether these two evidence sets imply materially different information must be established before reasoning can proceed confidently. Return true when the evidence sets investigate meaningfully different things. Return false when they are materially the same despite wording differences. Do not choose which hypothesis is correct and do not generate a question.
+
+No examples. No expected answers. Same instruction for all three cases.
+
+### Case 1 — Delivery Causes / Different Evidence
+
+**Problem:** "Orders are arriving late and customers have started complaining."
+
+**Evidence for A:** staffing levels, shift coverage, workload, order-processing throughput.
+**Evidence for B:** supplier delivery records, supplier lead times, supplier reliability history.
+
+**Fixed human reference:** `changesInformationNeededNext: true`
+
+This is the exact 54N failure case (Case 3) with the 54O evidence distinction made explicit in the input.
+
+**Model output:**
+```json
+{
+  "changesInformationNeededNext": true,
+  "reason": "One evidence set focuses on internal operational capacity while the other examines external supplier logistics, requiring investigation of fundamentally different systems."
+}
+```
+
+**Consequence classification:** `consequence_correct` ✓
+
+### Case 2 — Same Staffing Cause / Paraphrased Evidence
+
+**Problem:** "Orders are arriving late and customers have started complaining."
+
+**Evidence for A:** staffing levels, shift coverage, workload, processing times.
+**Evidence for B:** team capacity, staffing levels, shift coverage, order-processing times.
+
+**Fixed human reference:** `changesInformationNeededNext: false`
+
+Purpose: ensure paraphrased evidence lists are not treated as different information needs.
+
+**Model output:**
+```json
+{
+  "changesInformationNeededNext": false,
+  "reason": "Both evidence sets investigate the same operational factors regarding staffing, scheduling, and processing throughput needed to diagnose delayed orders."
+}
+```
+
+**Consequence classification:** `consequence_correct` ✓
+
+### Case 3 — Pricing Versus Checkout / Different Evidence
+
+**Problem:** "Website sales have fallen sharply over the last month."
+
+**Evidence for A:** price changes, conversion response after price changes, customer price sensitivity, competitor pricing.
+**Evidence for B:** checkout error logs, checkout funnel drop-off, payment failures, browser/device failures.
+
+**Fixed human reference:** `changesInformationNeededNext: true`
+
+**Model output:**
+```json
+{
+  "changesInformationNeededNext": true,
+  "reason": "One set investigates pricing and market factors while the other investigates technical checkout failures and user experience barriers."
+}
+```
+
+**Consequence classification:** `consequence_correct` ✓
+
+### Consequence-Correct Count
+
+**3 of 3** cases classified correctly.
+
+### Consequence-Failed Count
+
+**0 of 3** cases classified incorrectly.
+
+### Question Answers
+
+1. Did Case 1 recover `true` once staff and supplier evidence were explicit? Yes — model returned `changesInformationNeededNext: true` with reason correctly identifying internal operational capacity versus external supplier logistics as different investigation targets.
+2. Did Case 2 correctly return `false` for materially equivalent evidence? Yes — model returned `changesInformationNeededNext: false` with reason correctly identifying that both sets investigate the same staffing/scheduling/throughput factors.
+3. Did Case 3 return `true` for pricing versus checkout evidence? Yes — model returned `changesInformationNeededNext: true` with reason correctly distinguishing pricing/market investigation from technical checkout investigation.
+4. Did the model choose a hypothesis winner? No. No preferred hypothesis language in any output.
+5. Did it generate a next question? No. No question generation in any output.
+6. Does explicit evidence distinction allow the higher-level consequence judgement in all tested cases? Yes, across all three tested cases.
+7. What does this say about the 54N failure? It narrows the candidate explanation: when evidence differences are made explicit as inputs, the model can recover the correct consequence. This suggests the 54N failure may have involved losing or failing to use the evidence distinction rather than lacking it entirely.
+8. Does this prove why 54N failed? No — only one gap between these experiments was tested (explicit vs implicit evidence presentation). Other factors in the original formulation may also contribute.
+9. Does this establish what the engine should ask next? No — this does not prescribe engine behaviour. It shows a capability boundary under controlled conditions.
+
+Questions 8 and 9 are answered No as required.
+
+### Comparison with Experiment 54N
+
+In 54N, Case 3 (identical problem: late orders; identical hypotheses: staff capacity vs supplier lead time) returned `changesInformationNeededNext: false` — collapsing the competing causes into one investigation direction.
+
+In 54P, the exact same problem and hypotheses are tested but with the evidence needs made explicit as structured input fields. The model now correctly returns `true` with a reason distinguishing internal operational capacity from external supplier logistics.
+
+This is the narrowest bridge between 54N and 54O tested so far: it controls for all inputs except whether the evidence distinction is explicit in the prompt or must be inferred by the model.
+
+### Comparison with Experiment 54O
+
+Experiment 54O showed the model can distinguish evidence needs directly (3/3). This experiment shows the model can use those explicit distinctions to recover a higher-level consequence judgement (3/3). The capability appears to be available at both levels in these tested cases.
+
+### Inference Timing
+
+- Number of live calls: **3**
+- Total time: **44,464ms (~44s)**
+- Average: **14,821.44ms per call**
+- Fastest: **13,889.83ms (Case 2)**
+- Slowest: **15,861.54ms (Case 1)**
+
+Faster than Experiment 54O's ~62s total — the evidence-consequence task may be computationally lighter than evidence-discovery.
+
+### What This Establishes
+
+Explicit evidence differences were recovered as correct consequences in all three tested cases:
+- Different evidence for different causes: correctly identified (Case 1, original 54N failure case).
+- Same evidence paraphrased differently: correctly treated as the same need (Case 2 control).
+- Different evidence across domains: correctly identified (Case 3 cross-domain test).
+
+No winner was chosen. No next question was generated. All invariant violations absent.
+
+### What This Does Not Establish
+
+- Whether the model can perform the same reasoning when evidence must be inferred rather than supplied.
+- Whether this generalises to more than three cases or different domains.
+- Why 54N failed — only one variable (explicit vs implicit evidence) was changed between the experiments.
+- What the engine should ask next.
+
+### Focused Test Result
+
+3 of 3 consequence classifications matched fixed human references. No invariant violations detected (no winner selection, no scores, no next questions). All three outputs produced semantically coherent reasons that correctly addressed the evidence/information distinction for each case.
+
+### Experiment Conclusion
+
+**Explicit evidence differences recovered the correct consequence in all tested cases.**
+
+When the model is given explicit evidence needs as structured input, it can correctly infer whether those needs imply materially different investigation directions — including on the exact case where Experiment 54N failed. The bridge from evidence distinction to consequence judgement appears functional under controlled conditions.
+
+### Limitations
+
+- Three cases only; limited domain coverage (one delivery scenario, one paraphrase control, one e-commerce scenario).
+- Evidence was fixed human input; not tested with model-generated or inferred evidence.
+- Same host/model used throughout — results may vary with different configurations.
+- Does not establish generalisation beyond these specific inputs.
+
+### Status
+
+**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-evidence-consequence.test.js`.