experiment: test consequence of interpretation disagreement

docs/current-handoff.md

@@ -85,7 +85,9 @@ Experiment 54K tested whether the configured semantic model (`qwen-claude:latest
 
 Experiment 54L repeated two identical grounding cases three times each to test stability across six live calls. The source-versus-added boundary was perfectly stable (zero leakage in all runs). Detection completeness appeared variable but manual analysis showed the instability came from the automated evaluator's paraphrase sensitivity, not the model itself. Case A strengthening identified in all 3 runs; Case B "other causes" and "not established as main problem" each identified in all 3 runs. Status pending Rob's review.
 
-Experiment 54M tested whether two interpretations of one source can expose their substantive disagreement without deciding which is correct. Three live Ollama calls across three cases: real pricing attribution difference, paraphrase identity control, and competing causal explanations. All three classified as disagreement_correct by human semantic review. Paraphrase was correctly treated as agreement; shared meaning stayed separate; no invented disagreement or winner selection occurred. Status pending Rob's review.
+Experiment 54M tested whether two interpretations of one source can expose their substantive disagreement without deciding which is correct. Three live Ollama calls across three cases: real pricing attribution difference, paraphrase identity control, and competing causal explanations. All three classified as disagreement_correct by human semantic review. Paraphrase was correctly treated as agreement; shared meaning stayed separate; no invented disagreement or winner selection occurred. The comparison capability worked across the three tested patterns: substantive disagreement, paraphrase agreement, and competing causal explanations. Broader generalisation remains untested. Status pending Rob's review.
+
+Experiment 54N tested whether an interpretation disagreement can be judged for material consequence on downstream information needs without generating a next question or choosing a winner. Three fixed cases: pricing ambiguity (consequence_correct), paraphrase identity control (consequence_correct), competing causes (consequence_failed — model returned false, missing that staff-capacity vs supplier evidence represent divergent investigation directions). 2/3 correct. Model did not choose a winner or generate an actual next question in any case. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-disagreement-consequence.test.js`.
 
 ## 5. What Remains Open
 
@@ -139,8 +141,8 @@ Answer before continuing:
 
 ---
 
-*Created by Experiment 34. Updated by Experiments 38–53, 54A–54M. Branch: `feature/user-workspace-ux-v0.7`.*
+*Created by Experiment 34. Updated by Experiments 38–53, 54A–54N. Branch: `feature/user-workspace-ux-v0.7`.*
 
-### Return-to-Work Note (Experiment 54M)
+### Return-to-Work Note (Experiment 54N)
 
-Experiment 54L was corrected: model grounding was materially stable across six repeated identical inputs; evaluator wording sensitivity caused the apparent classification instability. Experiment 54M tested whether two interpretations of one source can expose their substantive disagreement without deciding which is correct. Three live calls across three cases: real pricing attribution difference, paraphrase identity control, and competing causal explanations. All three classified as disagreement_correct by human semantic review. Paraphrase was correctly treated as agreement; shared meaning stayed separate; no invented disagreement or winner selection occurred. Same host/model retained; no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-interpretation-disagreement.test.js`.
+Experiment 54M showed two interpretations can expose their substantive disagreement without selecting a winner. Experiment 54N tested whether that disagreement materially changes what needs to be established next, using three fixed cases: pricing ambiguity (correct — model detected changed information need), paraphrase agreement (correct — model avoided false consequence), and competing causes (incorrect — model treated them as same-direction investigation). The model correctly refrained from choosing either interpretation as correct or generating an actual next question in any case. Same host/model retained; no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-disagreement-consequence.test.js`.

docs/design-evolution-log.md

@@ -6852,7 +6852,7 @@ The automated check flagged terms from the model's own output (e.g., "interpreta
 
 ### Historical Comparison / Regression
 
-Experiment 54K showed grounding was promising but imperfect (one missed addition). Experiment 54L showed grounding boundary was stable but evaluator was brittle. Experiment 54M extends beyond grounding to the next reasoning step: comparing two interpretations of the same source. The semantic instruction is domain-neutral (no examples, no expected answers), yet it achieved clean results in all three cases — suggesting the capability generalises from grounding-specific tasks to a more general comparison task.
+Experiment 54K showed grounding was promising but imperfect (one missed addition). Experiment 54L showed grounding boundary was stable but evaluator was brittle. Experiment 54M extends beyond grounding to the next reasoning step: comparing two interpretations of the same source. The semantic instruction is domain-neutral (no examples, no expected answers), yet it achieved clean results in all three cases — the comparison capability worked across the three tested patterns: substantive disagreement, paraphrase agreement, and competing causal explanations. Broader generalisation remains untested.
 
 ### Documentation Updated
 
@@ -6879,3 +6879,262 @@ No engine components, no UI components, no configuration changes. This experimen
 ### Status
 
 **Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-interpretation-disagreement.test.js`.
+
+## Experiment 54N — Does a Disagreement Actually Change What the Engine Needs to Know Next? (2026-08-07)
+
+### Objective
+
+First, tighten Experiment 54M so it does not imply broader generalisation than the evidence supports.
+
+Then test one narrow downstream consequence:
+
+> Given a fixed, human-reviewed interpretation disagreement, can the semantic model tell whether that disagreement materially changes what information would need to be established next?
+
+This experiment does not generate the next question. It does not decide which interpretation is correct. It tests only whether the disagreement matters enough to change the reasoning direction.
+
+### Corrected Experiment 54M Conclusion
+
+The comparison capability worked across the three tested patterns: substantive disagreement, paraphrase agreement, and competing causal explanations. Broader generalisation remains untested.
+
+shared meaning and substantive disagreement were cleanly separated across three tested patterns;
+no winner was selected;
+broader generalisation remained untested.
+
+### Hypothesis
+
+A semantic model may be able to distinguish between:
+
+- disagreement that does **not** materially alter the next information needed;
+- disagreement that **does** imply different evidence or investigation would be needed before proceeding confidently.
+
+If so, disagreement can potentially remain visible without automatically forcing clarification every time.
+
+Do not implement clarification or question selection yet.
+
+### Context Budget
+
+Used:
+
+- `docs/current-handoff.md` (Experiment 54M findings and Return-to-Work Note);
+- Experiment 54M only in `docs/design-evolution-log.md`;
+- `tests/reconstruction/semantic-interpretation-disagreement.test.js` as historical reference;
+- `.env.local` only for existing `OLLAMA_BASE_URL` and `OLLAMA_MODEL`.
+
+Not read: full experiment history; graph files; Behaviour Selection; assessor; UI; question-selection code; reconstruction production prompts; earlier semantic test files.
+
+### Configuration
+
+Host: `http://192.168.1.111:11434` (same as production, same as Experiment 54M)
+Model: `qwen-claude:latest` (same as production, same as Experiment 54M)
+
+Expected model: `qwen-claude:latest`
+
+No localhost fallback. No experiment-specific model variable.
+
+### Number of Live Inference Calls
+
+Exactly **3** live Ollama calls — one per case.
+
+### Input Contract
+
+Each call receives:
+
+```json
+{
+  "source": "...",
+  "sharedMeaning": ["..."],
+  "disagreement": ["..."]
+}
+```
+
+The disagreement is fixed before inference. No original full interpretations included unless required by the case definition.
+
+### Output Contract
+
+Return exactly:
+
+```json
+{
+  "changesInformationNeededNext": true,
+  "reason": "one short sentence"
+}
+```
+
+or:
+
+```json
+{
+  "changesInformationNeededNext": false,
+  "reason": "one short sentence"
+}
+```
+
+This boolean is test-only. It is **not** a production contract. No confidence, scores, recommended question, preferred interpretation, or behaviour labels added.
+
+### Semantic Instruction Used
+
+> Decide whether the stated disagreement would materially change what information needs to be established next before reasoning can proceed confidently. Return true only when the competing interpretations imply meaningfully different evidence or investigation directions. Return false when the disagreement is only wording, emphasis, or does not change the information needed next. Do not choose which interpretation is correct and do not generate a next question.
+
+No examples. No expected answers. Same instruction for all three cases.
+
+### Case 1 — Pricing Strength Versus Unresolved Cause
+
+**Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
+
+**Shared meaning:** revenue has declined; pricing may be related to the problem.
+
+**Disagreement:** one interpretation treats pricing as a potentially material contributor; the other keeps pricing unresolved and allows other causes.
+
+**Fixed human reference:** `changesInformationNeededNext: true` — If pricing is materially causal, pricing evidence becomes central. If causality remains broad, other possible causes also need investigation.
+
+**Model output:**
+```json
+{
+  "changesInformationNeededNext": true,
+  "reason": "Treating pricing as material shifts the investigation toward specific elasticity and competitive data, while keeping it unresolved prioritizes broader diagnostic evidence across other potential revenue drivers."
+}
+```
+
+**Consequence classification:** `consequence_correct` ✓
+
+### Case 2 — Paraphrase / No Material Disagreement
+
+**Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
+
+**Shared meaning:** revenue has declined; pricing may contribute; its importance remains uncertain.
+
+**Disagreement:** no substantive disagreement; wording differs only.
+
+**Fixed human reference:** `changesInformationNeededNext: false` — Equivalent interpretations should not cause a different investigation merely because they are phrased differently.
+
+**Model output:**
+```json
+{
+  "changesInformationNeededNext": false,
+  "reason": "The prompt explicitly states there is no substantive disagreement, so both interpretations require the exact same evidence and investigation directions."
+}
+```
+
+**Consequence classification:** `consequence_correct` ✓
+
+### Case 3 — Competing Causes
+
+**Source:** "Orders are arriving late and customers have started complaining."
+
+**Shared meaning:** orders are arriving late; there is a delivery-delay problem.
+
+**Disagreement:** one interpretation attributes the likely cause to insufficient staff capacity; the other attributes the possible cause to unreliable supplier lead times.
+
+**Fixed human reference:** `changesInformationNeededNext: true` — Staff-capacity evidence and supplier-lead-time evidence are materially different investigation directions.
+
+**Model output:**
+```json
+{
+  "changesInformationNeededNext": false,
+  "reason": "Both interpretations address the same core delivery delay and require verifying order timelines to identify the root cause, so the immediate evidence needed does not materially differ regardless of the hypothesized attribution."
+}
+```
+
+**Consequence classification:** `consequence_failed` ✗
+
+The model collapsed both competing causes into a single investigation direction ("verifying order timelines") rather than recognising that staff-capacity evidence (payroll, scheduling, shift coverage) and supplier-lead-time evidence (vendor SLAs, historical lead-time variance, procurement processes) represent materially different inquiry paths.
+
+### Consequence-Correct Count
+
+**2 of 3** cases classified correctly.
+
+### Consequence-Failed Count
+
+**1 of 3** cases classified incorrectly.
+
+### Question: Did Case 1 identify that the disagreement changes the information needed next?
+
+Yes. The model returned `changesInformationNeededNext: true` with a reason distinguishing pricing-specific evidence from broader diagnostic investigation.
+
+### Question: Did Case 2 correctly avoid treating paraphrase as a reason to change direction?
+
+Yes. The model returned `changesInformationNeededNext: false`, correctly identifying the inputs as non-disagreeing.
+
+### Question: Did Case 3 identify that competing causes imply different evidence needs?
+
+No. The model returned `changesInformationNeededNext: false`, treating both competing causal explanations as requiring the same core investigation (verifying order timelines) rather than recognising divergent evidence paths.
+
+### Question: Did the model choose either interpretation as correct?
+
+No. None of the three outputs contained winner selection, scoring, or preference language.
+
+### Question: Did it generate an actual next question?
+
+No. None of the three outputs generated a next question. The Case 3 reason mentioned "verifying order timelines" but as a justification for sameness rather than as a recommended action.
+
+### Question: Did it confuse "different interpretation" with "different information need"?
+
+Partially. In Case 3, the model conflated the surface-level shared concern (delivery delay investigation) with the deeper investigation directions implied by each attribution hypothesis. It treated two different evidence sets as equivalent because they served the same diagnostic purpose (find root cause).
+
+### Does this experiment establish when the engine should ask the user for clarification?
+
+No. This experiment only tests consequence detection, not clarification triggering.
+
+### Does it establish what exact question should be asked?
+
+No. No actual next question was generated or tested in any case.
+
+### Does it establish which interpretation is better supported?
+
+No. The model did not choose a winner in any case, and no production behaviour changed.
+
+### Evidence That Disagreement Consequence Can Be Separated From Disagreement Itself
+
+Cases 1 and 2 demonstrate the model can distinguish between disagreement-with-consequence (pricing attribution affects investigation direction) and disagreement-without-consequence (paraphrase). The model produced the correct boolean and a semantically valid reason for both without any prompt conditioning on the expected answer. Case 3 reveals the boundary of that capability — competing causal explanations with identical surface-level diagnostic purposes were collapsed into one information need.
+
+### Inference Timing
+
+- Number of live calls: **3**
+- Total time: **63,047ms (~63s)**
+- Average: **21,015.68ms per call**
+- Fastest: **14,184.73ms (Case 2)**
+- Slowest: **31,749.99ms (Case 3)**
+
+### Questionable or Unsupported Findings
+
+Case 3's failure is notable but potentially narrow — the model may succeed with competing causes that have more obviously different evidence profiles. The single-failure rate (1/3) does not justify a broader conclusion about the capability's limits without additional cases testing boundary conditions between same-direction and different-direction evidence needs.
+
+### Experiment Conclusion
+
+**Disagreement consequence detection is promising but imperfect.**
+
+The model correctly distinguished paraphrase (no consequence) from substantive pricing disagreement (consequence present) in Cases 1 and 2, confirming that the boolean can separate consequence from mere disagreement. Case 3 failure — collapsing competing causal attributions into one investigation direction — reveals a blind spot: when two hypotheses share the same diagnostic purpose but require different evidence sets, the model did not recognise the divergence. This is the narrowest gap identified so far in the semantic comparison chain (54K–54N).
+
+### Focused Test Result
+
+2 of 3 consequence classifications matched fixed human references. No invariant violations detected (no winner selection, no scores, no actual next questions generated). All three outputs produced semantically coherent reasons.
+
+### Historical Comparison Result
+
+Experiment 54M showed semantic comparison can expose disagreement without choosing a winner. Experiment 54N tests the next reasoning step — whether that disagreement changes information needs — and shows the capability works for pricing-strength ambiguity and paraphrase but not yet for competing causal explanations with shared diagnostic purposes. The progression from grounding (54K/L) → separation (54I/J) → comparison (54M) → consequence (54N) is intact; each step adds one new capability without breaking the previous one.
+
+### Documentation Updated
+
+- `docs/design-evolution-log.md` — Experiment 54M generalisation correction applied; Experiment 54N section appended;
+- `docs/current-handoff.md` — Return-to-Work Note updated to reflect Experiment 54N findings.
+
+### Confirmation: Host and Model Remained Unchanged
+
+Host: `http://192.168.1.111:11434` (same as production, same as Experiments 54K–54M)
+Model: `qwen-claude:latest` (same as production, same as Experiments 54K–54M)
+
+### Confirmation: Production Prompts and Schemas Remained Unchanged
+
+The semantic instruction was written fresh for this experiment. No production prompts were modified. The output contract (`{ changesInformationNeededNext, reason }`) is the experiment-only shape.
+
+### Confirmation: No Consequence Logic Entered Active Runtime
+
+All inference calls were made exclusively within test code via `callConsequenceModel()`. No consequence logic was integrated into any production module. No runtime code changed.
+
+### Confirmation: Active Engine and UI Remained Unchanged
+
+No engine components, no UI components, no configuration changes. This experiment was entirely contained within test-only code in `tests/reconstruction/semantic-disagreement-consequence.test.js`.
+
+### Status
+
+**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-disagreement-consequence.test.js`.

tests/reconstruction/semantic-disagreement-consequence.test.js

@@ -0,0 +1,240 @@
+import { describe, it, expect } from "vitest";
+import { config } from "dotenv";
+import path from "path";
+import { fileURLToPath } from "url";
+
+const __filename = fileURLToPath(import.meta.url);
+const __dirname = path.dirname(__filename);
+config({ path: path.resolve(__dirname, "../../.env.local") });
+
+const OLLAMA_BASE_URL = process.env.OLLAMA_BASE_URL;
+const OLLAMA_MODEL = process.env.OLLAMA_MODEL;
+
+if (!OLLAMA_BASE_URL || !OLLAMA_MODEL) {
+  throw new Error("OLLAMA_BASE_URL and OLLAMA_MODEL must be set in .env.local");
+}
+
+/**
+ * Make one live Ollama chat call for consequence detection.
+ */
+async function callConsequenceModel(source, sharedMeaning, disagreement) {
+  const instruction = `Decide whether the stated disagreement would materially change what information needs to be established next before reasoning can proceed confidently. Return true only when the competing interpretations imply meaningfully different evidence or investigation directions. Return false when the disagreement is only wording, emphasis, or does not change the information needed next. Do not choose which interpretation is correct and do not generate a next question.
+
+Return valid JSON only in this shape:
+{
+  "changesInformationNeededNext": true | false,
+  "reason": "one short sentence explaining why"
+}`;
+
+  const messages = [
+    { role: "system", content: instruction.trim() },
+    {
+      role: "user",
+      content: `Source: "${source}"
+
+Shared meaning both interpretations carry: ${JSON.stringify(sharedMeaning)}
+
+Disagreement between the two interpretations: ${JSON.stringify(disagreement)}`,
+    },
+  ];
+
+  const res = await fetch(`${OLLAMA_BASE_URL}/api/chat`, {
+    method: "POST",
+    headers: { "Content-Type": "application/json" },
+    body: JSON.stringify({
+      model: OLLAMA_MODEL,
+      messages,
+      format: "json",
+      stream: false,
+    }),
+  });
+
+  if (!res.ok) {
+    throw new Error(`Ollama API error: ${res.status} ${res.statusText}`);
+  }
+
+  const data = await res.json();
+  const rawContent = data.message?.content ?? "";
+  const cleaned = rawContent.replace(/```(?:json)?\s*/g, "").replace(/```\s*/g, "");
+
+  return JSON.parse(cleaned.trim());
+}
+
+// ──────────────────────────────────────────────
+// Fixed human-reference ground truth (pre-written)
+// ──────────────────────────────────────────────
+
+const CASES = [
+  {
+    id: "Case 1 — Pricing Strength Versus Unresolved Cause",
+    source: "Revenue is down. I think pricing may be part of the problem, but I am not sure.",
+    sharedMeaning: ["revenue has declined", "pricing may be related to the problem"],
+    disagreement: [
+      "one interpretation treats pricing as a potentially material contributor",
+      "the other keeps pricing unresolved and allows other causes"
+    ],
+    reference: {
+      changesInformationNeededNext: true,
+      reason: "If pricing is materially causal, pricing evidence becomes central. If causality remains broad, other possible causes also need investigation."
+    }
+  },
+  {
+    id: "Case 2 — Paraphrase / No Material Disagreement",
+    source: "Revenue is down. I think pricing may be part of the problem, but I am not sure.",
+    sharedMeaning: ["revenue has declined", "pricing may contribute", "its importance remains uncertain"],
+    disagreement: ["no substantive disagreement; wording differs only"],
+    reference: {
+      changesInformationNeededNext: false,
+      reason: "Equivalent interpretations should not cause a different investigation merely because they are phrased differently."
+    }
+  },
+  {
+    id: "Case 3 — Competing Causes",
+    source: "Orders are arriving late and customers have started complaining.",
+    sharedMeaning: ["orders are arriving late", "there is a delivery-delay problem"],
+    disagreement: [
+      "one interpretation attributes the likely cause to insufficient staff capacity",
+      "the other attributes the possible cause to unreliable supplier lead times"
+    ],
+    reference: {
+      changesInformationNeededNext: true,
+      reason: "Staff-capacity evidence and supplier-lead-time evidence are materially different investigation directions."
+    }
+  }
+];
+
+// ──────────────────────────────────────────────
+// Semantic evaluation against fixed human references
+// ──────────────────────────────────────────────
+
+function evaluateConsequence(modelResult, reference) {
+  const result = modelResult;
+
+  let issues = [];
+  let notes = [];
+
+  // Boolean must match reference
+  if (result.changesInformationNeededNext !== reference.changesInformationNeededNext) {
+    issues.push("boolean_mismatch: model consequence does not match fixed human reference");
+  }
+
+  // Reason must explain information-need divergence or non-divergence
+  const reasonStr = (result.reason || "").toLowerCase();
+  if (!reasonStr || reasonStr.length < 5) {
+    issues.push("missing_reason: reason is empty or too short");
+  }
+
+  // Check for invariant violations (no winner, no scores, no questions)
+  const hasWinnerSelection = reasonStr.includes("interpretation a") && reasonStr.includes("correct") ||
+                             reasonStr.includes("interpretation b") && reasonStr.includes("correct") ||
+                             reasonStr.includes("winner");
+  if (hasWinnerSelection) {
+    issues.push("invariant_failed: model appears to have chosen a winning interpretation");
+  }
+
+  const hasNextQuestion = reasonStr.includes("ask") || reasonStr.includes("question") ||
+                          reasonStr.includes("next step") || reasonStr.includes("follow up");
+  if (hasNextQuestion) {
+    notes.push("caution: model generated a next-question suggestion alongside the consequence judgment");
+  }
+
+  const hasScores = result.score !== undefined || result.confidence !== undefined;
+  if (hasScores) {
+    issues.push("invariant_failed: output contains score or confidence fields");
+  }
+
+  if (issues.length === 0) return "consequence_correct";
+  return "consequence_failed";
+}
+
+// ──────────────────────────────────────────────
+// Describe the experiment as a single test suite
+// ──────────────────────────────────────────────
+
+describe("Experiment 54N — Consequence of Interpretation Disagreement (test-only)", () => {
+  const results = [];
+  const timings = [];
+
+  for (const testCase of CASES) {
+    it(`${testCase.id} — consequence detection`, async () => {
+      const t0 = performance.now();
+      const result = await callConsequenceModel(
+        testCase.source,
+        testCase.sharedMeaning,
+        testCase.disagreement
+      );
+      const elapsed = performance.now() - t0;
+      timings.push(elapsed);
+
+      expect(result).toHaveProperty("changesInformationNeededNext");
+      expect(typeof result.changesInformationNeededNext).toBe("boolean");
+      expect(result).toHaveProperty("reason");
+      expect(typeof result.reason).toBe("string");
+
+      const classification = evaluateConsequence(result, testCase.reference);
+
+      results.push({
+        id: testCase.id,
+        source: testCase.source,
+        sharedMeaning: testCase.sharedMeaning,
+        disagreement: testCase.disagreement,
+        reference: testCase.reference,
+        modelResult: result,
+        classification: classification,
+        timingMs: Number(elapsed.toFixed(2)),
+      });
+
+      console.log(`\n=== ${testCase.id} ===`);
+      console.log(`Model output:`);
+      console.log(`  changesInformationNeededNext:`, result.changesInformationNeededNext);
+      console.log(`  reason: "${result.reason}"`);
+      console.log(`Reference:`, testCase.reference);
+      console.log(`Classification: ${classification}`);
+    }, 120000);
+  }
+
+  it("54N — summary and required questions", () => {
+    const correct = results.filter((r) => r.classification === "consequence_correct").length;
+    const failed = results.filter((r) => r.classification === "consequence_failed").length;
+    const total = timings.reduce((a, b) => a + b, 0);
+    const avg = total / timings.length;
+    const fastest = Math.min(...timings);
+    const slowest = Math.max(...timings);
+
+    console.log("\n=== Experiment 54N Summary ===");
+    console.log(`Cases: ${results.length}`);
+    console.log(`Consequence-correct: ${correct}, Consequence-failed: ${failed}`);
+
+    results.forEach((r) => {
+      const label = r.reference.changesInformationNeededNext ? "should change" : "should NOT change";
+      const resultStr = r.modelResult.changesInformationNeededNext ? "changed" : "did not change";
+      const match = r.classification === "consequence_correct" ? "✓" : "✗";
+      console.log(`  ${match} ${r.id}: expected ${label}, model said ${resultStr}`);
+    });
+
+    // Check for invariant violations across all results
+    let winnerChosen = false;
+    let questionGenerated = false;
+    for (const r of results) {
+      const reasonStr = r.modelResult.reason?.toLowerCase() || "";
+      if (reasonStr.includes("winner") || (reasonStr.includes("interpretation a") && reasonStr.includes("correct"))) {
+        winnerChosen = true;
+      }
+      if (reasonStr.includes("ask") || reasonStr.includes("question")) {
+        questionGenerated = true;
+      }
+    }
+
+    console.log(`Total inference time: ${total.toFixed(2)}ms`);
+    console.log(`Average: ${avg.toFixed(2)}ms, Fastest: ${fastest.toFixed(2)}ms, Slowest: ${slowest.toFixed(2)}ms`);
+    console.log(`Winner chosen by model: ${winnerChosen ? "yes" : "no"}`);
+    console.log(`Next question generated: ${questionGenerated ? "yes" : "no"}`);
+
+    // Required question answers (all pass — these are questions to be answered in the report)
+    expect(false).toBe(false); // Q1-Q7 addressed in report
+    expect(false).toBe(false); // Q8-Q10 must be answered No
+
+    // Final assertion — always pass so timing/totals are recorded
+    expect(results.length).toBe(3);
+  });
+}, 600000);