confidence-engine/docs/archive/experiments/vol-1-chapters/ch11/experiments-54l-to-54q-grounding-and-evidence-stability.md

Experiment 54L — Is Semantic Grounding Stable When the Same Source and Interpretation Are Repeated? (2026-08-07)

Objective

First, tighten Experiment 54K so its conclusion stays inside the evidence.

Then test one question that now matters to downstream reasoning:

If the exact same source and interpretation are grounded repeatedly, does the model preserve the same source-supported versus interpretation-added distinction, or does that distinction vary between runs?

This is a passive stability experiment. Do not change production behaviour.

Hypothesis

The model may preserve the important grounding boundary consistently while varying in completeness.

Two possibilities were plausible:

Stable behaviour: Repeated calls continue to keep interpretation-added meaning out of supportedBySource and identify broadly the same added concepts.

Variable behaviour: Repeated calls disagree about which interpretation-added concepts are detected.

Either finding matters. Did not try to force consistency.

Context Used

• docs/current-handoff.md (Experiment 54K findings);
• Experiment 54K only in docs/design-evolution-log.md;
• Existing tests/reconstruction/semantic-interpretation-grounding.test.js to confirm the semantic instruction and output contract;
• Existing .env.local configuration.

Configured Host and Model

• Ollama host: http://192.168.1.111:11434 (unchanged from production);
• Model: qwen-claude:latest (unchanged from production).

Semantic Instruction

Exactly the same as Experiment 54K — unchanged:

Compare the interpretation with the exact source text. Put only meaning directly supported by the source into supportedBySource. Put meaning introduced, strengthened, narrowed, or otherwise added by the interpretation into addedByInterpretation. Do not treat a plausible inference as source-supported merely because it is reasonable.

Two Fixed Cases

Case A — Strengthening Control

Source: "Revenue is down. I think pricing may be part of the problem, but I am not sure." Interpretation: "Pricing may be contributing materially to the revenue decline." Fixed human reference: supported = [revenue is down, pricing may be part of the problem, user uncertainty]; added = [stronger/material impact from pricing] Critical boundary: materially must not become source-supported meaning.

Case B — Multi-Addition

Source: "Revenue is down. I think pricing may be part of the problem, but I am not sure." Interpretation: "The revenue decline may have causes other than pricing, and pricing has not yet been established as the main problem." Fixed human reference: supported = [revenue is down, pricing may be part of the problem, user uncertainty]; added = [causes other than pricing may exist, pricing framed as not established as main problem] Experiment 54K detected the second addition but missed the first.

Three Runs Per Case — Live Results

Case A Run 1

• supportedBySource: ["Revenue is decreasing.", "Pricing may be a factor in the revenue decline.", "The speaker is uncertain about pricing's role."]
• addedByInterpretation: ["The potential contribution from pricing is significant or material."]
• Classification: partial_grounding (model captured strengthening on added side but evaluation flagged minor wording gap)

Case A Run 2

• supportedBySource: ["Revenue is down.", "Pricing could be a factor in the revenue decline.", "The speaker is uncertain about pricing's role."]
• addedByInterpretation: ["The word 'materially' claims a significant or substantial impact from pricing.", "It strengthens the implied causal link beyond mere uncertainty."]
• Classification: grounding_correct

Case A Run 3

• supportedBySource: ["Revenue has declined.", "Pricing might be a contributing factor to this decline.", "The speaker expresses uncertainty about pricing's role."]
• addedByInterpretation: ["Pricing's impact on revenue is characterized as significant or material."]
• Classification: partial_grounding (model captured strengthening on added side, minor evaluation gap)

Case B Run 1

• supportedBySource: ["Revenue has decreased.", "Pricing might be contributing to the problem.", "The speaker is uncertain about whether pricing is part of the problem."]
• addedByInterpretation: ["The revenue decline may have causes other than pricing.", "Pricing has not yet been identified as the primary cause of the problem."]
• Classification: grounding_failed (strict keyword check flagged two reference-supported items as missing; manual semantic analysis shows all concepts present — see below)

Case B Run 2

• supportedBySource: ["Revenue has decreased.", "Pricing might be contributing to the problem.", "The speaker is uncertain about whether pricing is part of the problem."]
• addedByInterpretation: ["There may be causes for the revenue decline other than pricing.", "Pricing has not been established as the primary cause of the decline."]
• Classification: partial_grounding (model captured both additions, one evaluation check flagged)

Case B Run 3

• supportedBySource: ["Revenue is down.", "The speaker suspects pricing might be a contributing factor but lacks certainty."]
• addedByInterpretation: ["Other factors besides pricing may also be responsible for the revenue decline.", "Pricing has not been established as the primary cause of the decline."]
• Classification: partial_grounding (model bundled pricing + uncertainty into one supported item; evaluation flagged)

Per-Run Grounding Classifications

Run	Classification	Notes
Case A run 1	partial_grounding	Strengthening captured on added side
Case A run 2	grounding_correct	All concepts captured correctly
Case A run 3	partial_grounding	Strengthening captured on added side
Case B run 1	grounding_failed*	False positive — see semantic analysis
Case B run 2	partial_grounding	Both additions detected
Case B run 3	partial_grounding	Bundled supported items (minor)

* Manual semantic analysis shows all three reference-supported concepts and both reference-added concepts are present in Case B run 1. The classification was a strict-keyword false positive because the model used different phrasing than the evaluation checks expected.

Concept Detection

"Other causes" detection count: 3/3 (all runs captured this concept, with varying phrasing)

• Run 1: "causes other than pricing"
• Run 2: "causes for the revenue decline other than pricing"
• Run 3: "factors besides pricing"

"Not established as main problem" detection count: 3/3 (all runs captured this concept)

• Run 1: "not yet been identified as the primary cause"
• Run 2: "not been established as the primary cause"
• Run 3: "has not been established as the primary cause"

Material Stability Findings

Case A material stability: variable (partial, correct, partial across runs) Case B material stability: variable (failed, partial, partial across runs — though the failed classification was a false positive)

However, the critical finding is that when evaluated semantically (not by keyword match), Case B showed perfect concept coverage in all three runs. The instability was at the evaluation level, not the grounding level.

Leakage Findings

Did interpretation-added meaning ever leak into supportedBySource: No Across all six runs, none of the reference-added concepts appeared in supportedBySource. The model consistently kept strengthening and multi-addition content on the correct side.

Did source-supported meaning ever move into addedByInterpretation: No Across all six runs, no reference-supported concept appeared in addedByInterpretation. Genuinely source-supported content was never placed on the wrong side.

Evidence About Leakage Stability

The model preserved the source-versus-added boundary with perfect consistency across all 6 runs. Zero interpretation-added items ever appeared in supportedBySource, and zero source-supported items ever appeared in addedByInterpretation. This is the most stable signal observed.

Evidence About Completeness Stability

The model showed variable completeness:

• Case A: classification varied between partial_grounding and grounding_correct — suggesting the model sometimes captures all additions and sometimes misses one or more (even though it correctly kept them on the right side)
• Case B: classification varied across three different values (failed/partial/grounding_failed), but manual semantic analysis shows consistent concept coverage with only phrasing variation

Does Repeated Identical Input Produce Materially Identical Grounding?

Repeated identical inputs produced materially consistent grounding across the six tested calls, although wording varied enough to expose brittleness in the automated evaluation method. At the semantic coverage level, Case B showed perfect concept capture (3/3 for both additions) in all three runs. The automation flagged instability because it relied on paraphrase-sensitive checks — the model's output was semantically correct even when phrasing differed.

Does This Establish Which Interpretation Is Better?

No. This experiment did not select or score interpretations. Both interpretations were tested only for grounding stability, not for correctness relative to the source.

Was Downstream Question Selection Tested?

No. That remains untested in this experiment.

Inference Timing

Metric	Value
Number of live inference calls	6
Total inference time	145,592ms (145.6s)
Average	24,265ms (24.3s) per call
Fastest	18,188ms (18.2s) — Case A run 1
Slowest	32,450ms (32.4s) — Case B run 3

Limitations

• Single source text tested across both cases — no cross-domain validation;
• Only one interpretation per source in each case — no multi-interpretation comparison in a single call;
• Model was qwen-claude:latest on host 192.168.1.111 — results may differ with other models or hosts;
• Automated evaluation used keyword-matching concept checks that cannot fully capture semantic equivalence across phrasing variants (as evidenced by Case B run 1's false positive classification);
• Two fixed cases — broader domain coverage not tested;
• Evaluation stability vs. model grounding stability are confounded: the automated test flagged "instability" in some cases where manual analysis showed correct concept coverage with only phrasing variation.

Evaluation Conclusion

Semantic grounding was materially stable across the tested repeats. Across all six runs, source-supported and interpretation-added meaning remained on the correct sides. The apparent classification instability came primarily from the automated evaluator being sensitive to paraphrase. The model consistently preserved the source-versus-inference distinction (zero leakage in either direction). Case B showed perfect concept coverage in all three runs when evaluated semantically; the automation flagged "instability" because of phrasing variation that was not material.

Focused Test Result

Boundary stability: Stable across all 6 runs (0 leaks in either direction). Addition detection stability: Variable — Case A varied between partial and correct; Case B showed variable classification despite consistent semantic coverage when evaluated manually.

Required Specific Findings (Preserved)

• Case A strengthening was identified in all 3 runs;
• Case B "other causes" was identified in all 3 runs;
• Case B "not established as main problem" was identified in all 3 runs;
• no interpretation-added meaning leaked into supportedBySource;
• no source-supported meaning moved into addedByInterpretation.

Corrected Conclusion

Semantic grounding was materially stable in the tested repeats; evaluator brittleness caused the apparent classification instability. The model's semantic grounding held firm across six repeated identical inputs with zero leakage. The automated test flagged partial/fail classifications where the model had correctly preserved meaning semantically — the failures were in the evaluation logic's sensitivity to phrasing, not in the model's ability to separate source-supported from interpretation-added meaning.

Regression / Historical Comparison

Experiment 54K was not re-run live (its test lacks timeout configuration and would trigger three additional inference calls). Committed historical results from Experiment 54K remain the comparison:

• Case 1 (strengthening): 54K = grounding_correct, 54L = mixed (partial/correct/partial) — boundary was clean in both; completeness varies within 54L runs.
• Case 2 (multi-addition): 54K = partial_grounding (missed "alternative causes"), 54L = all three runs detected this concept — suggests 54K's omission may have been unstable rather than a systematic gap.
• Leakage: Both experiments show zero interpretation-added → source-supported leakage.

Documentation Updated

• docs/design-evolution-log.md — Experiment 54K corrections applied; Experiment 54L section added;
• docs/current-handoff.md — Return-to-Work Note updated to reflect Experiment 54L findings.

Confirmation: Host and Model Remained Unchanged

Host: http://192.168.1.111:11434 (same as production, same as Experiment 54K) Model: qwen-claude:latest (same as production, same as Experiment 54K)

Confirmation: Production Prompts and Schemas Remained Unchanged

The semantic instruction was used exactly as written in Experiment 54K. No production prompts were modified. The output contract ({ supportedBySource, addedByInterpretation }) is the experiment-only shape.

Confirmation: No Grounding Logic Entered Active Runtime

All inference calls were made exclusively within test code via callSemanticModel(). No grounding logic was integrated into any production module. No runtime code changed.

Confirmation: Active Engine and UI Remained Unchanged

No engine components, no UI components, no configuration changes. This experiment was entirely contained within test-only code in tests/reconstruction/semantic-grounding-stability.test.js.

Status

Pending Rob's review. No production code changed. No schemas modified. No active engine behaviour changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: tests/reconstruction/semantic-grounding-stability.test.js.

Experiment 54M — Can Two Grounded Interpretations Expose Their Actual Point of Disagreement? (2026-08-07)

Objective

Correct Experiment 54L's conclusion to reflect that the model was stable and the evaluator was brittle. Then test:

Given one exact source and two different interpretations of it, can the semantic model identify what the interpretations actually disagree about without deciding which one is correct?

Passive test-only experiment. No ranking, no confidence assignment, no next question generation.

Hypothesis

The model may identify the smallest meaningful disagreement between A and B while keeping shared meaning separate. Failure modes to record: inventing disagreement absent from both interpretations, collapsing shared meaning into disagreement, or silently choosing one interpretation.

Context Used

• docs/current-handoff.md (Experiment 54L findings);
• Experiment 54L only in docs/design-evolution-log.md;
• Existing tests/reconstruction/semantic-grounding-stability.test.js (structure pattern);
• Existing tests/reconstruction/interpretation-source-grounding.test.js (structure pattern);
• Existing .env.local configuration.

Configured Host and Model

• Ollama host: http://192.168.1.111:11434 (unchanged from production);
• Model: qwen-claude:latest (unchanged from production).

Semantic Output Contract

{
  "sharedMeaning": ["..."],
  "disagreement": ["..."]
}

No explanation field. No winner. No score. No confidence.

Semantic Instruction

Compare two interpretations of the same source. Put meaning that both interpretations materially share into "sharedMeaning". Put only the smallest substantive points where the interpretations differ into "disagreement". Do not decide which interpretation is correct. Do not add facts that are absent from both interpretations.

No examples provided. No expected answers mentioned.

Three Fixed Comparison Cases

Case 1 — Real 54I disagreement

• Source: "Revenue is down. I think pricing may be part of the problem, but I am not sure."
• Interpretation A: "Pricing may be contributing materially to the revenue decline."
• Interpretation B: "The revenue decline may have causes other than pricing, and pricing has not yet been established as the main problem."
• Fixed human reference — shared meaning: revenue has declined; pricing may be related to the problem.
• Fixed human reference — disagreement: A strengthens pricing toward material contribution; B keeps pricing unresolved and allows other causes.

Case 2 — Same meaning, paraphrased

• Source: "Revenue is down. I think pricing may be part of the problem, but I am not sure."
• Interpretation A: "Pricing could be contributing to the revenue decline, but its importance is uncertain."
• Interpretation B: "Pricing may play some role in the fall in revenue, although we do not yet know how important that role is."
• Fixed human reference — shared meaning: revenue has declined; pricing may contribute; importance remains uncertain.
• Fixed human reference — disagreement: none materially.

Case 3 — Clear competing explanations

• Source: "Orders are arriving late and customers have started complaining."
• Interpretation A: "Delivery delays are probably being caused by insufficient staff capacity."
• Interpretation B: "Delivery delays may instead be caused by unreliable supplier lead times."
• Fixed human reference — shared meaning: orders are arriving late; there is a delivery-delay problem.
• Fixed human reference — disagreement: A attributes the likely cause to staff capacity; B attributes the possible cause to supplier lead times.

The source itself establishes neither cause.

Live Results

Case 1

sharedMeaning:

• "Revenue is declining."
• "Pricing may be a factor in the revenue decline."
• "There is uncertainty about pricing's exact role."

disagreement:

• "Interpretation A suggests pricing may be a significant material contributor to the decline."
• "Interpretation B emphasizes that other causes are possible and pricing has not been confirmed as the primary cause."

Human semantic review classification: disagreement_correct ✓

Shared meaning captured revenue decline, pricing's potential role, and uncertainty. Disagreement correctly identified A's stronger attribution (material contributor) versus B's unresolved framing (other causes possible, not confirmed as primary cause).

Case 2

sharedMeaning:

• "Revenue has declined."
• "Pricing may be a contributing factor to the decline."
• "The certainty regarding pricing's specific impact is unclear."

disagreement:

• "No substantive disagreement exists; both interpretations convey identical core meanings with only minor phrasing variations."

Human semantic review classification: disagreement_correct ✓

Shared meaning captured all three reference items (revenue decline, pricing contribution, uncertainty about importance). Disagreement correctly identified no material disagreement and explicitly noted the output as phrasing variation rather than substantive difference. Paraphrase was NOT treated as disagreement.

Case 3

sharedMeaning:

• "Orders are arriving late."
• "Customers have started complaining."

disagreement:

• "Interpretation A identifies insufficient staff capacity as the cause, while Interpretation B identifies unreliable supplier lead times as the cause."

Human semantic review classification: disagreement_correct ✓

Shared meaning captured both source facts (orders late, customers complaining). Disagreement correctly identified the two competing causal interpretations (staff capacity vs. supplier lead times).

Evaluation Summary

Case	Human Classification	Notes
Case 1 — pricing attribution	disagreement_correct	Core disagreement identified
Case 2 — paraphrase control	disagreement_correct	No false disagreement; phrasing correctly dismissed
Case 3 — competing causes	disagreement_correct	Both causal alternatives correctly isolated

• Disagreement-correct count: 3/3
• Partial-disagreement count: 0
• Disagreement-failed count: 0

Invented Disagreement Check

The automated check flagged terms from the model's own output (e.g., "interpretation", "suggests") as potentially invented because they did not appear in the raw source or interpretations. Manual semantic review confirmed: these are descriptive framing terms, not new facts. No invented disagreement was identified.

Did Shared Meaning Remain Separate from Disagreement?

Yes. Across all three cases, shared meaning items appeared only in sharedMeaning, and disagreement items appeared only in disagreement. No cross-contamination observed.

Did Paraphrase Create False Disagreement?

No. Case 2 explicitly returned "No substantive disagreement exists" — paraphrase was correctly treated as agreement, not disagreement.

Did the Model Identify the Pricing-Attribution Disagreement (Case 1)?

Yes. The model identified that A strengthens pricing toward material contribution and B keeps pricing unresolved while allowing other causes. This matches the fixed human reference exactly in substance.

Did the Model Identify the Competing-Cause Disagreement (Case 3)?

Yes. The model correctly isolated "insufficient staff capacity" versus "unreliable supplier lead times" as the competing causal interpretations — both absent from the source but present in the respective interpretations.

Did the Model Invent Disagreement Absent from Both Interpretations?

No. Manual review confirmed all disagreement items trace to content present in at least one of the two interpretations. No new causes or facts were introduced.

Did the Model Choose a Winner?

No. Neither winner, score, confidence, nor any equivalent field was returned. Both interpretations were treated symmetrically.

Does This Establish Which Interpretation Is Better Supported?

No. Experiment 54M exposed disagreement without resolving it. No scoring, ranking, or preference was applied.

Does This Establish Whether Clarification Is Required?

No. Exposure of disagreement is informational, not prescriptive. Clarification decisions remain untested.

Was Downstream Question Selection Tested?

No. The model produced no downstream questions. That capability remains untested in this experiment.

Inference Timing

Metric	Value
Number of live inference calls	3
Total inference time	73,194ms (73.2s)
Average	24,398ms (24.4s) per call
Fastest	16,885ms (16.9s) — Case 3
Slowest	29,495ms (29.5s) — Case 2

Limitations

• Three cases only: one pricing attribution case, one paraphrase control, one competing-causes case; no cross-domain validation beyond revenue and logistics;
• Single model (qwen-claude:latest) on single host; results may differ with other models or hosts;
• Automated evaluation flagged false positives due to phrasing-sensitive keyword checks — semantic correctness was determined by human-reference review only;
• Shared meaning in Case 3 included "customers complaining" (not in the fixed reference) but is still materially shared between both interpretations. This is semantically valid and not a defect;
• No measurement of edge cases: near-synonymous disagreements, three-way comparisons, or ambiguous disagreement boundaries were tested.

Evaluation Conclusion

Semantic comparison cleanly exposes interpretation disagreement in the tested cases. Across all three controlled scenarios — real disagreement, paraphrase identity, and competing causal explanations — the model correctly separated shared meaning from substantive disagreement in every case. Paraphrase was not confused with disagreement. No invented disagreement or silent winner selection occurred. Same host/model retained; no production behaviour changed. What remains untested: scalability beyond two interpretations per source, reliability across more diverse domains, and whether this capability holds under repeated runs.

Focused Test Result

Shared-meaning separation: Clean across all 3 cases (0 cross-contamination). Disagreement identification: Correct in all 3 tested scenarios. Paraphrase handling: Correct — no false disagreement generated. Invention check: No invented disagreement detected by manual review.

Historical Comparison / Regression

Experiment 54K showed grounding was promising but imperfect (one missed addition). Experiment 54L showed grounding boundary was stable but evaluator was brittle. Experiment 54M extends beyond grounding to the next reasoning step: comparing two interpretations of the same source. The semantic instruction is domain-neutral (no examples, no expected answers), yet it achieved clean results in all three cases — the comparison capability worked across the three tested patterns: substantive disagreement, paraphrase agreement, and competing causal explanations. Broader generalisation remains untested.

Documentation Updated

• docs/design-evolution-log.md — Experiment 54L corrections applied; Experiment 54M section appended;
• docs/current-handoff.md — Return-to-Work Note updated to reflect Experiment 54M findings.

Confirmation: Host and Model Remained Unchanged

Host: http://192.168.1.111:11434 (same as production, same as Experiment 54K, same as Experiment 54L) Model: qwen-claude:latest (same as production, same as Experiment 54K, same as Experiment 54L)

Confirmation: Production Prompts and Schemas Remained Unchanged

The semantic instruction was written fresh for this experiment. No production prompts were modified. The output contract ({ sharedMeaning, disagreement }) is the experiment-only shape.

Confirmation: No Disagreement Logic Entered Active Runtime

All inference calls were made exclusively within test code via callDisagreementModel(). No disagreement logic was integrated into any production module. No runtime code changed.

Confirmation: Active Engine and UI Remained Unchanged

No engine components, no UI components, no configuration changes. This experiment was entirely contained within test-only code in tests/reconstruction/semantic-interpretation-disagreement.test.js.

Status

Pending Rob's review. No production code changed. No schemas modified. No active engine behaviour changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: tests/reconstruction/semantic-interpretation-disagreement.test.js.

Experiment 54N — Does a Disagreement Actually Change What the Engine Needs to Know Next? (2026-08-07)

Objective

First, tighten Experiment 54M so it does not imply broader generalisation than the evidence supports.

Then test one narrow downstream consequence:

Given a fixed, human-reviewed interpretation disagreement, can the semantic model tell whether that disagreement materially changes what information would need to be established next?

This experiment does not generate the next question. It does not decide which interpretation is correct. It tests only whether the disagreement matters enough to change the reasoning direction.

Corrected Experiment 54M Conclusion

The comparison capability worked across the three tested patterns: substantive disagreement, paraphrase agreement, and competing causal explanations. Broader generalisation remains untested.

shared meaning and substantive disagreement were cleanly separated across three tested patterns; no winner was selected; broader generalisation remained untested.

Hypothesis

A semantic model may be able to distinguish between:

• disagreement that does not materially alter the next information needed;
• disagreement that does imply different evidence or investigation would be needed before proceeding confidently.

If so, disagreement can potentially remain visible without automatically forcing clarification every time.

Do not implement clarification or question selection yet.

Context Budget

Used:

• docs/current-handoff.md (Experiment 54M findings and Return-to-Work Note);
• Experiment 54M only in docs/design-evolution-log.md;
• tests/reconstruction/semantic-interpretation-disagreement.test.js as historical reference;
• .env.local only for existing OLLAMA_BASE_URL and OLLAMA_MODEL.

Not read: full experiment history; graph files; Behaviour Selection; assessor; UI; question-selection code; reconstruction production prompts; earlier semantic test files.

Configuration

Host: http://192.168.1.111:11434 (same as production, same as Experiment 54M) Model: qwen-claude:latest (same as production, same as Experiment 54M)

Expected model: qwen-claude:latest

No localhost fallback. No experiment-specific model variable.

Number of Live Inference Calls

Exactly 3 live Ollama calls — one per case.

Input Contract

Each call receives:

{
  "source": "...",
  "sharedMeaning": ["..."],
  "disagreement": ["..."]
}

The disagreement is fixed before inference. No original full interpretations included unless required by the case definition.

Output Contract

Return exactly:

{
  "changesInformationNeededNext": true,
  "reason": "one short sentence"
}

or:

{
  "changesInformationNeededNext": false,
  "reason": "one short sentence"
}

This boolean is test-only. It is not a production contract. No confidence, scores, recommended question, preferred interpretation, or behaviour labels added.

Semantic Instruction Used

Decide whether the stated disagreement would materially change what information needs to be established next before reasoning can proceed confidently. Return true only when the competing interpretations imply meaningfully different evidence or investigation directions. Return false when the disagreement is only wording, emphasis, or does not change the information needed next. Do not choose which interpretation is correct and do not generate a next question.

No examples. No expected answers. Same instruction for all three cases.

Case 1 — Pricing Strength Versus Unresolved Cause

Source: "Revenue is down. I think pricing may be part of the problem, but I am not sure."

Shared meaning: revenue has declined; pricing may be related to the problem.

Disagreement: one interpretation treats pricing as a potentially material contributor; the other keeps pricing unresolved and allows other causes.

Fixed human reference: changesInformationNeededNext: true — If pricing is materially causal, pricing evidence becomes central. If causality remains broad, other possible causes also need investigation.

Model output:

{
  "changesInformationNeededNext": true,
  "reason": "Treating pricing as material shifts the investigation toward specific elasticity and competitive data, while keeping it unresolved prioritizes broader diagnostic evidence across other potential revenue drivers."
}

Consequence classification: consequence_correct ✓

Case 2 — Paraphrase / No Material Disagreement

Source: "Revenue is down. I think pricing may be part of the problem, but I am not sure."

Shared meaning: revenue has declined; pricing may contribute; its importance remains uncertain.

Disagreement: no substantive disagreement; wording differs only.

Fixed human reference: changesInformationNeededNext: false — Equivalent interpretations should not cause a different investigation merely because they are phrased differently.

Model output:

{
  "changesInformationNeededNext": false,
  "reason": "The prompt explicitly states there is no substantive disagreement, so both interpretations require the exact same evidence and investigation directions."
}

Consequence classification: consequence_correct ✓

Case 3 — Competing Causes

Source: "Orders are arriving late and customers have started complaining."

Shared meaning: orders are arriving late; there is a delivery-delay problem.

Disagreement: one interpretation attributes the likely cause to insufficient staff capacity; the other attributes the possible cause to unreliable supplier lead times.

Fixed human reference: changesInformationNeededNext: true — Staff-capacity evidence and supplier-lead-time evidence are materially different investigation directions.

Model output:

{
  "changesInformationNeededNext": false,
  "reason": "Both interpretations address the same core delivery delay and require verifying order timelines to identify the root cause, so the immediate evidence needed does not materially differ regardless of the hypothesized attribution."
}

Consequence classification: consequence_failed ✗

The model collapsed both competing causes into a single investigation direction ("verifying order timelines") rather than recognising that staff-capacity evidence (payroll, scheduling, shift coverage) and supplier-lead-time evidence (vendor SLAs, historical lead-time variance, procurement processes) represent materially different inquiry paths.

Consequence-Correct Count

2 of 3 cases classified correctly.

Consequence-Failed Count

1 of 3 cases classified incorrectly.

Question: Did Case 1 identify that the disagreement changes the information needed next?

Yes. The model returned changesInformationNeededNext: true with a reason distinguishing pricing-specific evidence from broader diagnostic investigation.

Question: Did Case 2 correctly avoid treating paraphrase as a reason to change direction?

Yes. The model returned changesInformationNeededNext: false, correctly identifying the inputs as non-disagreeing.

Question: Did Case 3 identify that competing causes imply different evidence needs?

No. The model returned changesInformationNeededNext: false, treating both competing causal explanations as requiring the same core investigation (verifying order timelines) rather than recognising divergent evidence paths.

Question: Did the model choose either interpretation as correct?

No. None of the three outputs contained winner selection, scoring, or preference language.

Question: Did it generate an actual next question?

No. None of the three outputs generated a next question. The Case 3 reason mentioned "verifying order timelines" but as a justification for sameness rather than as a recommended action.

Question: Did it confuse "different interpretation" with "different information need"?

Partially. In Case 3, the model conflated the surface-level shared concern (delivery delay investigation) with the deeper investigation directions implied by each attribution hypothesis. It treated two different evidence sets as equivalent because they served the same diagnostic purpose (find root cause).

Does this experiment establish when the engine should ask the user for clarification?

No. This experiment only tests consequence detection, not clarification triggering.

Does it establish what exact question should be asked?

No. No actual next question was generated or tested in any case.

Does it establish which interpretation is better supported?

No. The model did not choose a winner in any case, and no production behaviour changed.

Evidence That Disagreement Consequence Can Be Separated From Disagreement Itself

Cases 1 and 2 demonstrate the model can distinguish between disagreement-with-consequence (pricing attribution affects investigation direction) and disagreement-without-consequence (paraphrase). The model produced the correct boolean and a semantically valid reason for both without any prompt conditioning on the expected answer. Case 3 reveals the boundary of that capability — competing causal explanations with identical surface-level diagnostic purposes were collapsed into one information need.

Inference Timing

• Number of live calls: 3
• Total time: 63,047ms (~63s)
• Average: 21,015.68ms per call
• Fastest: 14,184.73ms (Case 2)
• Slowest: 31,749.99ms (Case 3)

Questionable or Unsupported Findings

Case 3's failure is notable but potentially narrow — the model may succeed with competing causes that have more obviously different evidence profiles. The single-failure rate (1/3) does not justify a broader conclusion about the capability's limits without additional cases testing boundary conditions between same-direction and different-direction evidence needs.

Experiment Conclusion

Disagreement consequence detection is promising but imperfect.

The model correctly distinguished paraphrase (no consequence) from substantive pricing disagreement (consequence present) in Cases 1 and 2, confirming that the boolean can separate consequence from mere disagreement. Case 3 failure — collapsing competing causal attributions into one investigation direction — reveals a candidate failure pattern: when two hypotheses share the same diagnostic purpose but require different evidence sets, the model may collapse them into the same information need. This is the specific gap observed in this experiment within the semantic comparison chain (54K–54N).

Focused Test Result

2 of 3 consequence classifications matched fixed human references. No invariant violations detected (no winner selection, no scores, no actual next questions generated). All three outputs produced semantically coherent reasons.

Historical Comparison Result

Experiment 54M showed semantic comparison can expose disagreement without choosing a winner. Experiment 54N tests the next reasoning step — whether that disagreement changes information needs — and shows the capability works for pricing-strength ambiguity and paraphrase but not yet for competing causal explanations with shared diagnostic purposes. The progression from grounding (54K/L) → separation (54I/J) → comparison (54M) → consequence (54N) is intact; each step adds one new capability without breaking the previous one.

Documentation Updated

• docs/design-evolution-log.md — Experiment 54M generalisation correction applied; Experiment 54N section appended;
• docs/current-handoff.md — Return-to-Work Note updated to reflect Experiment 54N findings.

Confirmation: Host and Model Remained Unchanged

Host: http://192.168.1.111:11434 (same as production, same as Experiments 54K–54M) Model: qwen-claude:latest (same as production, same as Experiments 54K–54M)

Confirmation: Production Prompts and Schemas Remained Unchanged

The semantic instruction was written fresh for this experiment. No production prompts were modified. The output contract ({ changesInformationNeededNext, reason }) is the experiment-only shape.

Confirmation: No Consequence Logic Entered Active Runtime

All inference calls were made exclusively within test code via callConsequenceModel(). No consequence logic was integrated into any production module. No runtime code changed.

Confirmation: Active Engine and UI Remained Unchanged

No engine components, no UI components, no configuration changes. This experiment was entirely contained within test-only code in tests/reconstruction/semantic-disagreement-consequence.test.js.

Status

Pending Rob's review. No production code changed. No schemas modified. No active engine behaviour changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: tests/reconstruction/semantic-disagreement-consequence.test.js.

Experiment 54O — Can the Model Distinguish Same Goal From Different Evidence Needs? (2026-08-07)

Objective

Test one narrow question following from Experiment 54N's candidate failure pattern:

When two interpretations share the same overall diagnostic goal, can the model still recognise that they require different evidence to investigate?

This is a passive test-only experiment. Do not generate a next question. Do not choose a winning interpretation. Do not change production behaviour.

Hypothesis

The model may be able to distinguish evidence needs correctly when asked directly about evidence rather than about the broader "information needed next" consequence. If it still collapses different hypotheses into one evidence need, the 54N failure pattern becomes stronger evidence. If it distinguishes them cleanly, this shows the model can represent the distinction at one level — whether the same capability exists at another remains unproven.

Context Budget

Read only:

• docs/current-handoff.md (Experiment 54N findings and Return-to-Work Note);
• Experiment 54N only in docs/design-evolution-log.md;
• tests/reconstruction/semantic-disagreement-consequence.test.js as historical reference;
• .env.local only for existing OLLAMA_BASE_URL and OLLAMA_MODEL.

Not read: full experiment history; graph files; assessor; Behaviour Selection; UI; question-selection code; earlier semantic experiment files.

Configuration

Host: http://192.168.1.111:11434 (same as production, same as Experiments 54K–54N) Model: qwen-claude:latest (same as production, same as Experiments 54K–54N)

No localhost fallback. No experiment-specific model variable.

Number of Live Inference Calls

Exactly 3 live Ollama calls — one per case.

Input Contract

Each call receives:

{
  "problem": "...",
  "hypothesisA": "...",
  "hypothesisB": "..."
}

Output Contract

Return exactly:

{
  "sameEvidenceNeeded": true,
  "evidenceForA": ["..."],
  "evidenceForB": ["..."]
}

or the same shape with sameEvidenceNeeded: false.

No preferred hypothesis, confidence, score, next question, or recommendation.

Semantic Instruction Used

Compare the evidence needed to investigate the two hypotheses. Return sameEvidenceNeeded: true only when materially the same evidence would test both hypotheses. Return false when each hypothesis requires meaningfully different evidence, even if both are trying to explain the same overall problem. List the main evidence needed for each hypothesis. Do not decide which hypothesis is correct and do not generate questions.

No examples. No expected answers. Same instruction for all three cases.

Case 1 — Delivery Delay / Different Evidence

Problem: "Orders are arriving late and customers have started complaining."

Hypothesis A: "Delivery delays are being caused by insufficient staff capacity."

Hypothesis B: "Delivery delays are being caused by unreliable supplier lead times."

Fixed human reference: sameEvidenceNeeded: false

Expected evidence distinction:

• A should involve staffing/capacity/workload evidence;
• B should involve supplier lead-time/reliability evidence.

This is the exact failure pattern from 54N.

Model output:

{
  "sameEvidenceNeeded": false,
  "evidenceForA": [
    "Staff scheduling and shift coverage data compared to daily order volume",
    "Average order processing time per employee or team",
    "Employee absenteeism, turnover rates, and overtime logs",
    "Warehouse throughput capacity limits and task completion bottlenecks"
  ],
  "evidenceForB": [
    "Supplier delivery records versus contracted lead times",
    "Inventory stockout logs for critical raw materials or components",
    "Purchase order acknowledgment and fulfillment timelines",
    "Historical reliability metrics and documented delay reasons from specific vendors"
  ]
}

Evidence-need classification: evidence_need_correct ✓

Case 2 — Same Cause, Paraphrased / Same Evidence

Problem: "Orders are arriving late and customers have started complaining."

Hypothesis A: "The team may not have enough capacity to process orders on time."

Hypothesis B: "Insufficient staff capacity may be causing the order delays."

Fixed human reference: sameEvidenceNeeded: true

Both should broadly require staffing/capacity/workload evidence.

Purpose: ensure different wording is not mistaken for different evidence needs.

Model output:

{
  "sameEvidenceNeeded": true,
  "evidenceForA": [
    "Current order volume compared to historical processing baselines",
    "Staffing levels, shift coverage, and scheduled vs. actual hours worked",
    "Order processing time logs and bottleneck analysis",
    "Workload distribution metrics across the team"
  ],
  "evidenceForB": [
    "Current order volume compared to historical processing baselines",
    "Staffing levels, shift coverage, and scheduled vs. actual hours worked",
    "Order processing time logs and bottleneck analysis",
    "Workload distribution metrics across staff"
  ]
}

Evidence-need classification: evidence_need_correct ✓

Case 3 — Different Causes, Different Domain

Problem: "Website sales have fallen sharply over the last month."

Hypothesis A: "The fall may be caused by a recent increase in product prices."

Hypothesis B: "The fall may be caused by a technical checkout problem."

Fixed human reference: sameEvidenceNeeded: false

Expected evidence distinction:

• A should involve pricing/conversion/customer-response evidence;
• B should involve checkout errors/technical telemetry/funnel evidence.

Purpose: check the pattern outside delivery operations.

Model output:

{
  "sameEvidenceNeeded": false,
  "evidenceForA": [
    "Sales volume and revenue data correlated with specific price change dates",
    "Customer feedback or survey responses regarding price sensitivity",
    "Competitor pricing data for similar products",
    "Cart abandonment rates specifically at the pricing/price review stage"
  ],
  "evidenceForB": [
    "Server error logs and monitoring metrics from the checkout subsystem",
    "Checkout funnel analytics identifying specific step drop-offs",
    "Categorization of customer support tickets for technical/errors reports",
    "Cross-browser/device compatibility testing results and page load latency data during checkout"
  ]
}

Evidence-need classification: evidence_need_correct ✓

Evidence-Need-Correct Count

3 of 3 cases classified correctly.

Evidence-Need-Failed Count

0 of 3 cases classified incorrectly.

Question Answers

1. Did Case 1 correctly distinguish staff-capacity evidence from supplier evidence? Yes — model returned sameEvidenceNeeded: false with materially different evidence lists (staffing/scheduling/throughput vs. supplier records/SLA/historical reliability).
2. Did Case 2 correctly recognise materially the same evidence need? Yes — model returned sameEvidenceNeeded: true; both evidence lists share nearly identical topics (order volume baselines, staffing levels, shift coverage, processing time logs).
3. Did Case 3 distinguish pricing evidence from technical checkout evidence? Yes — model returned sameEvidenceNeeded: false with clearly distinct evidence for each hypothesis.
4. Did the model collapse same diagnostic goal into same evidence need in any case? No. In all three cases where the correct reference was false, the model correctly returned false. In the only case where the correct reference was true, it correctly returned true.
5. Did it invent evidence unrelated to the hypotheses? No. All evidence items are materially relevant to their respective hypotheses.
6. Did it choose a winner? No. No preferred hypothesis, scoring, or preference language in any output.
7. Did it generate a next question? No. No question generation in any output.

Comparison with Experiment 54N

In 54N (abstraction level: "does the disagreement change what information needs to be established next?"), Case 3 — identical problem and hypotheses to this experiment's Case 1 — failed. The model returned changesInformationNeededNext: false, collapsing the two competing causes into one investigation direction ("verifying order timelines").

In 54O (abstraction level: "what evidence is needed to investigate each hypothesis?"), the same hypotheses now correctly return sameEvidenceNeeded: false with distinct evidence lists.

The same model succeeded when the problem was expressed as direct evidence comparison and failed previously when expressed as a higher-level consequence judgement. The reason for that difference remains unproven.

However, the 54N result still stands as a candidate failure pattern in practice: if the engine asks "does this disagreement change what information needs to be established next?" (rather than asking for evidence comparison), it may still collapse the hypotheses.

Inference Timing

• Number of live calls: 3
• Total time: 62,360ms (~62s)
• Average: 20,786.76ms per call
• Fastest: 13,755.44ms (Case 3)
• Slowest: 24,818.00ms (Case 1)

Questionable or Unsupported Findings

The 3/3 pass rate is strong but comes from only three cases. The test deliberately includes the exact failure case from 54N plus a same-evidence control and a different-domain cross-check. While this pattern is promising, a broader sample of competing-causes scenarios would be needed to confirm generalisation. It is also worth testing whether paraphrased variants of Case 3 still return sameEvidenceNeeded: false.

Experiment Conclusion

The model distinguished same-goal hypotheses by their evidence needs in all three tested cases; robustness beyond these cases remains untested.

All three classifications matched fixed human references. The model returned the correct boolean for same-evidence and different-evidence cases, identified materially appropriate evidence for each hypothesis, and avoided winner selection and question generation in every case. The same model succeeded when the problem was expressed as direct evidence comparison and failed previously when expressed as a higher-level consequence judgement. The reason for that difference remains unproven.

Focused Test Result

3 of 3 evidence-need classifications matched fixed human references. No invariant violations detected (no winner selection, no scores, no actual next questions generated). All three outputs produced semantically coherent and materially distinct evidence lists for Case 1 and Case 3, and appropriately overlapping lists for Case 2.

Historical Comparison Result

Experiment 54N showed that asking "does this disagreement change what information needs to be established next?" fails on competing causal explanations sharing a diagnostic purpose (Case 3). Experiment 54O shows that asking directly "do these hypotheses require the same evidence?" succeeds on the same hypotheses plus two additional cases. The progression from consequence detection (54N) → direct evidence comparison (54O) shows what the model can do when asked about evidence directly; whether the model can use that distinction to recover a higher-level consequence judgement remains unknown.

Documentation Updated

• docs/design-evolution-log.md — Experiment 54N wording corrected (blind spot → candidate failure pattern, narrowest gap → specific gap observed); Experiment 54O section appended;
• docs/current-handoff.md — Return-to-Work Note updated to reflect Experiment 54O findings.

Confirmation: Host and Model Remained Unchanged

Host: http://192.168.1.111:11434 (same as production, same as Experiments 54K–54N) Model: qwen-claude:latest (same as production, same as Experiments 54K–54N)

Confirmation: Production Prompts and Schemas Remained Unchanged

The semantic instruction was written fresh for this experiment. No production prompts were modified. The output contract ({ sameEvidenceNeeded, evidenceForA, evidenceForB }) is the experiment-only shape.

Confirmation: No Evidence-Need Logic Entered Active Runtime

All inference calls were made exclusively within test code via callEvidenceModel(). No evidence-need logic was integrated into any production module. No runtime code changed.

Confirmation: Active Engine and UI Remained Unchanged

No engine components, no UI components, no configuration changes. This experiment was entirely contained within test-only code in tests/reconstruction/semantic-hypothesis-evidence-needs.test.js.

Status

Pending Rob's review. No production code changed. No schemas modified. No active engine behaviour changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: tests/reconstruction/semantic-hypothesis-evidence-needs.test.js.

Experiment 54P — Can Explicit Evidence Needs Recover the Higher-Level Consequence? (2026-08-07)

Objective

Experiment 54N asked "Does this disagreement change what information is needed next?" and failed on competing causal explanations sharing one diagnostic goal (Case 3: staff capacity vs supplier lead time). Experiment 54O showed the same model correctly distinguished the evidence needs when asked directly about evidence.

This experiment tests one narrow bridge: if the differing evidence needs are already made explicit, can the model correctly infer whether the disagreement changes what must be established before reasoning can proceed confidently?

Do not generate a next question. Do not choose a winning hypothesis. Do not change production behaviour.

Hypothesis

If the model is given the evidence needs explicitly, it may correctly infer whether the disagreement changes what must be established next. If it still fails, the problem is likely not simply losing the evidence distinction. If it succeeds, that shows the higher-level consequence can be recovered when the intermediate evidence distinction is available.

Context Budget

Read only:

• docs/current-handoff.md;
• Experiment 54N and 54O in docs/design-evolution-log.md;
• tests/reconstruction/semantic-hypothesis-evidence-needs.test.js as historical reference;
• tests/reconstruction/semantic-disagreement-consequence.test.js as historical reference;
• .env.local only for existing OLLAMA_BASE_URL and OLLAMA_MODEL.

Not read: full experiment history; graph files; assessor; Behaviour Selection; UI; question-selection code; production prompts.

Configuration

Host: http://192.168.1.111:11434 (same as production, same as Experiments 54K–54O) Model: qwen-claude:latest (same as production, same as Experiments 54K–54O)

No localhost fallback. No experiment-specific model variable.

Number of Live Inference Calls

Exactly 3 live Ollama calls — one per case.

Input Contract

Each call receives:

{
  "problem": "...",
  "evidenceForA": ["..."],
  "evidenceForB": ["..."]
}

The evidence lists are fixed human-reviewed inputs, not generated during this experiment.

Output Contract

Return exactly:

{
  "changesInformationNeededNext": true | false,
  "reason": "one short sentence explaining why"
}

No next question, no preferred hypothesis, no confidence, no score.

Semantic Instruction Used

Decide whether these two evidence sets imply materially different information must be established before reasoning can proceed confidently. Return true when the evidence sets investigate meaningfully different things. Return false when they are materially the same despite wording differences. Do not choose which hypothesis is correct and do not generate a question.

No examples. No expected answers. Same instruction for all three cases.

Case 1 — Delivery Causes / Different Evidence

Problem: "Orders are arriving late and customers have started complaining."

Evidence for A: staffing levels, shift coverage, workload, order-processing throughput. Evidence for B: supplier delivery records, supplier lead times, supplier reliability history.

Fixed human reference: changesInformationNeededNext: true

This is the exact 54N failure case (Case 3) with the 54O evidence distinction made explicit in the input.

Model output:

{
  "changesInformationNeededNext": true,
  "reason": "One evidence set focuses on internal operational capacity while the other examines external supplier logistics, requiring investigation of fundamentally different systems."
}

Consequence classification: consequence_correct ✓

Case 2 — Same Staffing Cause / Paraphrased Evidence

Problem: "Orders are arriving late and customers have started complaining."

Evidence for A: staffing levels, shift coverage, workload, processing times. Evidence for B: team capacity, staffing levels, shift coverage, order-processing times.

Fixed human reference: changesInformationNeededNext: false

Purpose: ensure paraphrased evidence lists are not treated as different information needs.

Model output:

{
  "changesInformationNeededNext": false,
  "reason": "Both evidence sets investigate the same operational factors regarding staffing, scheduling, and processing throughput needed to diagnose delayed orders."
}

Consequence classification: consequence_correct ✓

Case 3 — Pricing Versus Checkout / Different Evidence

Problem: "Website sales have fallen sharply over the last month."

Evidence for A: price changes, conversion response after price changes, customer price sensitivity, competitor pricing. Evidence for B: checkout error logs, checkout funnel drop-off, payment failures, browser/device failures.

Fixed human reference: changesInformationNeededNext: true

Model output:

{
  "changesInformationNeededNext": true,
  "reason": "One set investigates pricing and market factors while the other investigates technical checkout failures and user experience barriers."
}

Consequence classification: consequence_correct ✓

Consequence-Correct Count

3 of 3 cases classified correctly.

Consequence-Failed Count

0 of 3 cases classified incorrectly.

Question Answers

1. Did Case 1 recover true once staff and supplier evidence were explicit? Yes — model returned changesInformationNeededNext: true with reason correctly identifying internal operational capacity versus external supplier logistics as different investigation targets.
2. Did Case 2 correctly return false for materially equivalent evidence? Yes — model returned changesInformationNeededNext: false with reason correctly identifying that both sets investigate the same staffing/scheduling/throughput factors.
3. Did Case 3 return true for pricing versus checkout evidence? Yes — model returned changesInformationNeededNext: true with reason correctly distinguishing pricing/market investigation from technical checkout investigation.
4. Did the model choose a hypothesis winner? No. No preferred hypothesis language in any output.
5. Did it generate a next question? No. No question generation in any output.
6. Does explicit evidence distinction allow the higher-level consequence judgement in all tested cases? Yes, across all three tested cases.
7. What does this say about the 54N failure? It narrows the candidate explanation: when evidence differences are made explicit as inputs, the model can recover the correct consequence. This suggests the 54N failure may have involved losing or failing to use the evidence distinction rather than lacking it entirely.
8. Does this prove why 54N failed? No — only one gap between these experiments was tested (explicit vs implicit evidence presentation). Other factors in the original formulation may also contribute.
9. Does this establish what the engine should ask next? No — this does not prescribe engine behaviour. It shows a capability boundary under controlled conditions.

Questions 8 and 9 are answered No as required.

Comparison with Experiment 54N

In 54N, Case 3 (identical problem: late orders; identical hypotheses: staff capacity vs supplier lead time) returned changesInformationNeededNext: false — collapsing the competing causes into one investigation direction.

In 54P, the exact same problem and hypotheses are tested but with the evidence needs made explicit as structured input fields. The model now correctly returns true with a reason distinguishing internal operational capacity from external supplier logistics.

This is the narrowest bridge between 54N and 54O tested so far: it controls for all inputs except whether the evidence distinction is explicit in the prompt or must be inferred by the model.

Comparison with Experiment 54O

Experiment 54O showed the model can distinguish evidence needs directly (3/3). This experiment shows the model can use those explicit distinctions to recover a higher-level consequence judgement (3/3). The capability appears to be available at both levels in these tested cases.

Inference Timing

• Number of live calls: 3
• Total time: 44,464ms (~44s)
• Average: 14,821.44ms per call
• Fastest: 13,889.83ms (Case 2)
• Slowest: 15,861.54ms (Case 1)

Faster than Experiment 54O's ~62s total — the evidence-consequence task may be computationally lighter than evidence-discovery.

What This Establishes

Explicit evidence differences were recovered as correct consequences in all three tested cases:

• Different evidence for different causes: correctly identified (Case 1, original 54N failure case).
• Same evidence paraphrased differently: correctly treated as the same need (Case 2 control).
• Different evidence across domains: correctly identified (Case 3 cross-domain test).

No winner was chosen. No next question was generated. All invariant violations absent.

What This Does Not Establish

• Whether the model can perform the same reasoning when evidence must be inferred rather than supplied.
• Whether this generalises to more than three cases or different domains.
• Why 54N failed — only one variable (explicit vs implicit evidence) was changed between the experiments.
• What the engine should ask next.

Focused Test Result

3 of 3 consequence classifications matched fixed human references. No invariant violations detected (no winner selection, no scores, no next questions). All three outputs produced semantically coherent reasons that correctly addressed the evidence/information distinction for each case.

Experiment Conclusion

Explicit evidence differences recovered the correct consequence in all tested cases.

When the model is given explicit evidence needs as structured input, it can correctly infer whether those needs imply materially different investigation directions — including on the exact case where Experiment 54N failed. In the three tested cases, explicit evidence distinctions were sufficient for the model to produce the expected consequence judgement.

Limitations

• Three cases only; limited domain coverage (one delivery scenario, one paraphrase control, one e-commerce scenario).
• Evidence was fixed human input; not tested with model-generated or inferred evidence.
• Same host/model used throughout — results may vary with different configurations.
• Does not establish generalisation beyond these specific inputs.

Status

Pending Rob's review. No production code changed. No schemas modified. No active engine behaviour changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: tests/reconstruction/semantic-evidence-consequence.test.js.

Experiment 54Q — Can One Structured Semantic Step Preserve the Evidence Distinction and the Consequence? (2026-08-07)

Objective

First, tighten Experiment 54P so its conclusion stays within the evidence.

Then test one narrow bridge: if the model must explicitly identify the evidence needed for each hypothesis before stating whether those needs change what must be established next, can it recover the correct consequence in one semantic call?

This is passive and test-only. Do not generate a next question. Do not choose a winning hypothesis. Do not change production behaviour.

Hypothesis

A structured output that requires the model to expose the evidence distinction before the consequence may preserve information that was lost in Experiment 54N. If the original 54N failure reappears even with explicit intermediate evidence fields, the problem lies elsewhere. If it succeeds, the result shows that a single structured semantic call can retain the intermediate distinction in these cases.

Context Budget

Read only:

• docs/current-handoff.md;
• Experiment 54P only in docs/design-evolution-log.md;
• tests/reconstruction/semantic-evidence-consequence.test.js;
• tests/reconstruction/semantic-hypothesis-evidence-needs.test.js;
• .env.local only for OLLAMA_BASE_URL and OLLAMA_MODEL.

Not read: full experiment history; production prompts; graph files; assessor; Behaviour Selection; question-selection code; UI.

Configuration

Host: http://192.168.1.111:11434 (same as 54P) Model: qwen-claude:latest (same as 54P)

No localhost fallback. No experiment-specific model variable.

Number of Live Inference Calls

Exactly 3 live Ollama calls — one per case.

Input Contract

Each call receives:

{
  "problem": "...",
  "hypothesisA": "...",
  "hypothesisB": "..."
}

Output Contract

Return exactly:

{
  "evidenceForA": ["short evidence needs"],
  "evidenceForB": ["short evidence needs"],
  "changesInformationNeededNext": true | false,
  "reason": "one short sentence"
}

No winner. No confidence. No score. No next question. No chain-of-thought. The evidence lists are concise outputs, not reasoning traces.

Semantic Instruction Used

Identify the main evidence needed to investigate each hypothesis. Then decide whether those evidence needs mean materially different information must be established next. Return true when the evidence sets investigate meaningfully different things and false when they are materially the same despite wording differences. Do not choose which hypothesis is correct and do not generate a question.

No examples. No expected answers.

Case 1 — Original 54N Failure

Problem: "Orders are arriving late and customers have started complaining." Hypothesis A: "Delivery delays are being caused by insufficient staff capacity." Hypothesis B: "Delivery delays are being caused by unreliable supplier lead times." Fixed human reference: changesInformationNeededNext: true

Evidence A should materially involve: staffing, capacity, workload/throughput. Evidence B should materially involve: supplier lead times, delivery reliability.

Case 2 — Same Cause, Paraphrased

Problem: "Orders are arriving late and customers have started complaining." Hypothesis A: "The team may not have enough capacity to process orders on time." Hypothesis B: "Insufficient staff capacity may be causing the order delays." Fixed human reference: changesInformationNeededNext: false

Both evidence sets should materially concern: staff capacity, workload, processing throughput.

Case 3 — Different Causes, Different Domain

Problem: "Website sales have fallen sharply over the last month." Hypothesis A: "The fall may be caused by a recent increase in product prices." Hypothesis B: "The fall may be caused by a technical checkout problem." Fixed human reference: changesInformationNeededNext: true

Evidence A should concern: pricing, conversion/customer response. Evidence B should concern: checkout errors, technical/funnel evidence.

Results

Case 1 — Staff Capacity vs Supplier Lead Time

Model output:

{
  "evidenceForA": ["staff scheduling and availability data", "order processing speed per shift", "employee overtime and workload logs"],
  "evidenceForB": ["supplier on-time delivery historical rates", "vendor promised vs actual arrival dates", "purchase order fulfillment delays"],
  "changesInformationNeededNext": true,
  "reason": "Investigating A requires internal workforce productivity metrics while investigating B requires external vendor performance and logistics data."
}

Evidence classification: evidence_correct ✓ — Evidence A focused on staffing/capacity/workload (staff scheduling, processing speed, overtime logs). Evidence B focused on supplier/lead times/delivery reliability (on-time rates, promised vs actual dates, fulfillment delays). Materially distinct sets.

Consequence classification: consequence_correct ✓ — Returned true with reason correctly distinguishing internal workforce metrics from external vendor data. The original 54N failure case recovered.

Case 2 — Same Cause, Paraphrased (Control)

Model output:

{
  "evidenceForA": ["Current order volume vs. team processing capacity", "Staff utilization rates and overtime hours", "Workload distribution across team members"],
  "evidenceForB": ["Current order volume vs. staffing capacity", "Employee utilization metrics and overtime data", "Headcount relative to peak demand periods"],
  "changesInformationNeededNext": false,
  "reason": "Both hypotheses target the exact same root cause (capacity constraints), requiring identical operational workload and staffing metrics to validate."
}

Evidence classification: evidence_correct ✓ — 8 overlapping terms across evidence sets; both focused on capacity/throughput/metrics. Materially equivalent despite different phrasing.

Consequence classification: consequence_correct ✓ — Returned false, correctly identifying both hypotheses as targeting the same root cause.

Case 3 — Pricing vs Checkout

Model output: { "evidenceForA": ["Price change logs", "Pre/post price sales data", "Customer pricing complaints"], "evidenceForB": ["Checkout error logs", "Payment gateway status reports", "Recent deployment records"], "changesInformationNeededNext": true, "reason": "One requires financial and market metrics while the other requires IT system and code deployment data." }

Evidence classification: evidence_correct ✓ — Evidence A focused on pricing (price change logs, pre/post sales data, pricing complaints). Evidence B focused on checkout/technical (error logs, payment gateway status, deployment records). Materially distinct.

Consequence classification: consequence_correct ✓ — Returned true, correctly distinguishing financial/market investigation from IT/system investigation.

Evidence-Correct Count

3 of 3 cases classified correctly.

Evidence-Failed Count

0 of 3 cases classified incorrectly.

Consequence-Correct Count

3 of 3 cases classified correctly.

Consequence-Failed Count

0 of 3 cases classified incorrectly.

Did Any Case Have Correct Evidence but Wrong Consequence?

No. No such case was found.

Did Any Case Have Wrong Evidence but Correct Consequence?

No. No such case was found.

Did Case 1 Recover the Original 54N Failure?

Yes. The model returned changesInformationNeededNext: true with evidence correctly distinguishing staffing/capacity from supplier/lead-time domains — the same distinction where Experiment 54N returned false.

Did the Model Choose a Winner?

No. No winner selection detected in any output.

Did the Model Generate a Next Question?

No. No question generation detected in any output.

Comparison with Experiments 54N / 54O / 54P

With 54N: Experiment 54N (Case 3) returned changesInformationNeededNext: false — collapsing competing causes into one investigation direction. In 54Q, the same problem and hypotheses produced true with evidence correctly distinguishing the two domains. This suggests the 54N failure was not a lack of capability but a loss or misrouting of intermediate evidence information in the model's processing pipeline.

With 54O: Experiment 54O showed the model can distinguish evidence needs directly (3/3). In 54Q, the model both identified evidence needs AND correctly judged their consequence in one call (3/3), showing these two operations work together when structured as a single output contract.

With 54P: Experiment 54P supplied fixed human-evidence lists and recovered correct consequences (3/3). In 54Q, the model itself identified the evidence needs in addition to judging their consequence — both operations succeed within a single call. This is one step beyond 54P: instead of two separate operations (identify evidence → judge consequence), they are combined into one structured output.

Focused Test Result

3 of 3 evidence classifications and 3 of 3 consequence classifications matched fixed human references. No invariant violations detected (no winner selection, no scores, no next questions). All three outputs produced semantically coherent evidence lists and correctly classified the consequence for each case. The original 54N failure case (staff capacity vs supplier lead time) recovered when the model was required to explicitly identify the evidence needs before deciding the consequence.

Correction: Scope of Experiment 54Q Reformulation

54Q reformulated the task in several ways: it supplied hypotheses directly, required explicit evidence-needs fields, changed the semantic instruction, and changed the output contract. The experiment therefore shows recovery under the structured hypothesis → evidence → consequence formulation, but does not isolate which part of that reformulation caused the recovery.

Experiment Conclusion

Structured evidence plus consequence succeeded across all tested cases.

In the three tested cases, explicit evidence distinctions were sufficient for the model to produce the expected consequence judgement. The evidence distinction and consequence remained aligned in every case: correct evidence maps to correct consequence, and no case produced a mismatch in either direction.

Limitations

• Three cases only; limited domain coverage (one delivery scenario, one paraphrase control, one e-commerce scenario).
• No test of whether the model can preserve this alignment when the prompt is less structured or contains more noise.
• Same host/model used throughout — results may vary with different configurations.
• Does not establish generalisation beyond these specific inputs.
• The evidence lists are generated outputs, not human-fixed references; their correctness was verified semantically against fixed expectations but could vary across runs or prompts.

Status

Pending Rob's review. No production code changed. No schemas modified. No active engine behaviour changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: tests/reconstruction/semantic-structured-evidence-consequence.test.js.