diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 8607df1..ec24fe9 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -38,9 +38,15 @@ Experiment 39 resumed reasoning experiments with a passive validation of Behavio Experiment 40 diagnosed the root causes: Summarise and Pause fire their rules in real data but are always blocked by Acknowledge's priority-1 position (priority conflict, not assessor failure). Clarive's triggers never activate in tested scenarios due to the `too_broad` health condition being extremely narrow. All five behaviours confirmed independently reachable in synthetic isolation. No rules changed. +Experiment 41 compared two passive alternatives for reducing Acknowledge dominance: +- Variant A (priority reordering): evaluate Summarise/Pause before Acknowledge — introduces false-positive summarise in focusing phase +- Variant B (Acknowledge exclusions): keep priority, gate Acknowledge when phase=concluding/synthesising or progress=stalled or health=user_overloaded — recommended +- Both variants converge on the same two genuine changes: concluding→summarise and stalled→pause +- No production code changed. Branch: `feature/user-workspace-ux-v0.7`. + ## 5. What Remains Open -- Whether Acknowledge should be reordered past Summarise/Pause to resolve the priority conflict diagnosed in Exp 40; +- Implement Variant B's acknowledgment exclusion (recommended path from Exp 41); - Whether the `too_broad` health trigger needs widening so Clarify fires in more typical investigations; - Whether `user_overloaded` health should be producible by the assessor for stalled/inconsistent evidence states. @@ -88,8 +94,8 @@ Answer before continuing: --- -*Created by Experiment 34. Updated by Experiments 38, 39, 40. Branch: `feature/user-workspace-ux-v0.7`.* +*Created by Experiment 34. Updated by Experiments 38, 39, 40, 41. Branch: `feature/user-workspace-ux-v0.7`.* -### Return-to-Work Note (Experiment 39) +### Return-to-Work Note (Experiment 41) -Experiment 40 audited why Clarify, Summarise, and Pause were absent in Experiment 39. Summarise and Pause are eligible_but_blocked: their rule conditions fire but Acknowledge (priority 1) always wins when health=healthy — a priority conflict. Clarify is never_eligible_in_tested_scenarios: its `too_broad` trigger requires activeUnknownCount > 3 AND resolved < 2 simultaneously, which no fixture reaches; the `orienting+low obs` path is never produced by the assessor for tested domains. All five behaviours confirmed independently reachable in synthetic isolation. No rules changed. Status pending Rob's review. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `docs/current-handoff.md`, then `docs/design-evolution-log.md` entry for Experiment 40, and `tests/behaviour-selection.reachability.test.js` for diagnostic details. The next action is discussion about whether to reorder Acknowledge lower (past Summarise/Pause) or refine its trigger to exclude terminal phases. +Experiments 39–41 diagnosed Acknowledge dominance in Behaviour Selection and compared two passive alternatives. **Experiment 41 concluded:** Variant B (Acknowledge exclusions via phase/progress/health gates) is recommended over Variant A (priority reordering, which introduces false-positive summarise). Both variants correctly identify the same two genuine changes: `concluding → summarise` and `stalled → pause`. No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `docs/current-handoff.md`, then `docs/design-evolution-log.md` entry for Experiment 41, and `tests/behaviour-selection.counterfactual.test.js` for variant details. The next action is implementing Variant B's `isAcknowledgeExcluded()` gate as a test-only function, then evaluating against a wider range of assessment scenarios. diff --git a/docs/current-project-state.md b/docs/current-project-state.md index 7a055c6..905c581 100644 --- a/docs/current-project-state.md +++ b/docs/current-project-state.md @@ -46,7 +46,7 @@ These are what currently affect the working engine: The following were built during Experiments 18–25B. They are isolated diagnostic layers with no active integration into the user-facing investigation: - Investigation-state assessment (phase and progress classification); -- Behaviour selection from assessed state — passively evaluated in Experiment 39 against real assessment outputs; produces all valid behaviours but Acknowledge dominates when health=healthy; +- Behaviour selection from assessed state — passively evaluated in Experiments 39–41; all five behaviours reachable but Acknowledge dominates (71% on real data); Exp 41 recommends Variant B (Acknowledge exclusions via phase/progress/health gates) as the cleaner approach; - Decision condition status evaluation; - Question-to-condition relevance scoring; - Evidence direction classification (support, contradict, inform); @@ -100,7 +100,7 @@ Historical documents are retained under `docs/archive/` and should be opened onl ## 8. Return-to-Work Summary -Engine experiments advanced to Experiment 40, a reachability diagnostic confirming Summarise and Pause fire their rules but are always blocked by Acknowledge (priority-1 conflict), while Clarive's triggers never activate in tested scenarios due to narrow `too_broad` health conditions. All five behaviours confirmed independently reachable synthetically. No rules or engine behaviour changed. Knowledge-management phase is complete: all seven criteria met. No backlog items deleted or promoted. +Engine experiments advanced to Experiment 41, which compared two passive alternatives for reducing Acknowledge dominance. Variant B (Acknowledge exclusions via phase/progress/health gates) is recommended over Variant A (priority reordering with false-positive side effects). Both variants converge on the same two genuine changes: `concluding → summarise` and `stalled → pause`. No production code changed. First document to read: `docs/current-project-state.md`. Then consult `.claude/architecture-guardrails.md` before any code changes and `docs/project-knowledge-inventory.md` for task-specific references. The full experiment history remains available in `docs/design-evolution-log.md` but is no longer default reading. diff --git a/docs/design-evolution-log.md b/docs/design-evolution-log.md index 8f75e6c..4c8336e 100644 --- a/docs/design-evolution-log.md +++ b/docs/design-evolution-log.md @@ -2404,3 +2404,93 @@ This means any fix must address two distinct problems: ### Documents Updated - `docs/design-evolution-log.md` — this entry - `docs/current-handoff.md` — return-to-work note replaced + +## Experiment 41 — Compare Acknowledge Priority Alternatives (2026-08-06) + +### Purpose + +Experiment 40 confirmed Summarise and Pause are eligible_but_blocked by Acknowledge's priority-1 position. Two passive alternatives were compared without modifying production code: + +**Variant A** — Reorder rules so specific behaviours (Summarise, Pause) evaluate before Acknowledge. The idea is that if a more specific behaviour fires first, it captures the terminal/stalled states where Acknowledge should not fire. + +**Variant B** — Keep existing priority order but exclude Acknowledge from firing when phase=concluding/synthesising, progress=stalled, or health=user_overloaded. The idea is to gate Acknowledge rather than reorder everything. + +### Method + +Both variants were implemented as test-only functions in `tests/behaviour-selection.counterfactual.test.js`. Each variant was evaluated against the same 7 real assessment turns from Experiments 39/40 across 3 scenarios. All five behaviours confirmed independently reachable synthetically. No production rules changed. + +### Assessor Outputs (7 real turns) + +| # | Scenario | Turn | Phase (conf) | Progress | Health | Existing | +|---|---|---|---|---|---|---| +| 1 | long-investigation | 0 | cannot_determine(low) | cannot_determine | too_narrow | continue | +| 2 | long-investigation | 3 | focusing(high) | steady | healthy | acknowledge | +| 3 | long-investigation | 4 | concluding(high) | steady | healthy | acknowledge | +| 4 | contradictory-evidence | 0 | focusing(high) | cannot_determine | healthy | acknowledge | +| 5 | contradictory-evidence | 1 | focusing(high) | stalled | healthy | acknowledge | +| 6 | contradictory-evidence | 2 | focusing(high) | steady | healthy | acknowledge | +| 7 | short-early | 0 | exploring(low) | cannot_determine | healthy | continue | + +### Results on Real Scenarios + +| Turn | Existing | Variant A | Variant B | Change? | +|---|---|---|---|---| +| long-investigation t3 | acknowledge | summarise | acknowledge | V-A: side-effect | +| long-investigation t4 | acknowledge | **summarise** | **summarise** | **convergent ✓** | +| contradictory-evidence t1 | acknowledge | **pause** | **pause** | **convergent ✓** | +| All others | unchanged | unchanged | unchanged | — | + +### Divergence Analysis + +**Variant A diverges from Variant B at long-investigation turn 3.** Variant A produces `summarise` because its `resolvedNodeCount >= 3 && steady` rule fires at priority 1 without phase context. The assessor confirms this is a focusing-phase state (not synthesising/concluding) where the user needs acknowledgment, not compression. This is a false-positive for summarisation — a side-effect of Variant A's priority reordering. + +**Variant B correctly preserves Acknowledge** at long-investigation t3 because: +1. The exclusion list only includes `synthesising`, `concluding`, `stalled`, and `user_overloaded` — not focusing +2. SummariseV2 itself has a phase gate (`phase.value === "synthesising"`) that prevents false-fire in focusing states +3. Acknowledge at priority 1 wins because no exclusion applies + +### Key Findings + +1. **Both variants converge on the same two genuine changes:** `concluding → summarise` and `stalled → pause`. This was the experiment's primary question, and both approaches answer it correctly. + +2. **Variant A introduces a false-positive:** The `resolvedNodeCount >= 3 && steady` rule fires in focusing-phase states without phase context, causing premature summarisation when Acknowledge would be more useful. + +3. **Variant B has cleaner boundaries:** Explicit exclusion conditions prevent unwanted side-effects while preserving Acknowledge's role as the default healthy-state behaviour. + +4. **Distribution shift (both variants):** + - Existing: acknowledge 71%, continue 29% + - Variant A: acknowledge 29%, summarise 29%, pause 14%, continue 29% + - Variant B: acknowledge 43%, summarise 14%, pause 14%, continue 29% + - Variant B preserves more Acknowledge because it doesn't remove the default healthy-state behaviour entirely + +5. **Variant B is architecturally cleaner** for this problem space because it adds a targeted gate to one rule rather than reordering five priority levels — each of which would need individual review for side-effects. + +### Test Results + +- `tests/behaviour-selection.counterfactual.test.js`: 44 passed (new diagnostic file) +- `tests/behaviour-selection.reachability.test.js`: 33 passed (no regressions) +- `tests/behaviour-selection.real-assessment.test.js`: 16 passed (shared fixtures intact) +- `tests/behaviour-selection.test.js`: 51 passed (no regressions) + +### Decision Criteria + +| Criterion | Variant A | Variant B | +|---|---|---| +| Fixes concluding state | ✓ summarise | ✓ summarise | +| Fixes stalled state | ✓ pause | ✓ pause | +| No false-positive changes | ✗ long-t3 → summarise | ✓ preserved acknowledge | +| Implementation complexity | Simple reordering | Small gate function | +| Maintains Acknowledge for healthy focus states | ? (depends on future review) | ✓ explicit preservation | + +### Recommendation + +**Variant B is preferred.** Both variants correctly identify the two genuine changes needed. Variant B has no false-positives, cleaner architectural boundaries (targeted exclusion vs priority reordering), and better preserves the existing Acknowledge default for healthy focusing states where it is appropriate. A recommended implementation would: + +1. Keep existing priority order +2. Add `isAcknowledgeExcluded()` function with conditions: phase∈{synthesising, concluding}, progress=stalled, health=user_overloaded +3. Gate Acknowledge through this exclusion before selecting it at priority 1 + +### Documents Updated +- `docs/design-evolution-log.md` — this entry +- `docs/current-handoff.md` — return-to-work note replaced + diff --git a/tests/behaviour-selection.counterfactual.test.js b/tests/behaviour-selection.counterfactual.test.js new file mode 100644 index 0000000..902e6f8 --- /dev/null +++ b/tests/behaviour-selection.counterfactual.test.js @@ -0,0 +1,579 @@ +/** + * Behaviour Selection Counterfactual — Experiment 41 + * + * Compare two small, passive alternatives for reducing Acknowledge dominance: + * + * Variant A — evaluate Summarise/Pause before Acknowledge (priority reordering); + * Variant B — existing priority with Acknowledge exclusions (phase/progress/health gates). + * + * Production selector is NOT modified. Variants exist only in this test file. + */ + +import { describe, it, expect } from "vitest"; +import assessInvestigationState from "@/lib/assessment/investigation-state-assessor.js"; +import selectBehaviour from "@/lib/behaviour-selection/behaviour-selector.js"; + +/* ═══════════════════════════════════════════════════ + 1. Minimal assessment builder (not mutated) + ═══════════════════════════════════════════════════ */ + +function mkAssessment(opts = {}) { + return { + version: "v0.1", + assessedAt: new Date().toISOString(), + confidence: opts.overallConfidence || "medium", + phase: opts.phase ?? { + value: "cannot_determine", + confidence: "low", + signals: [], + evidence: { resolvedNodeCount: 0, activeUnknownCount: 0, unknownResolutionRatio: null, observationDensity: 0, evidenceDepth: "insufficient" } + }, + progress: opts.progress ?? { + value: "cannot_determine", + confidence: "low", + signals: [], + evidence: { turnCount: 0, recentResolutionsLastTurn: 0, newUnknownsPerTurn: null, repeatedNodeIds: [] } + }, + conversationHealth: opts.conversationHealth ?? { + value: "cannot_determine", + confidence: "low", + signals: [], + evidence: { questionTypeDistribution: null, activeUnknownCount: 0, resolvedNodeRatio: null, hasActiveQuestion: false, summaryLength: 0 } + } + }; +} + +/* ═══════════════════════════════════════════════════ + 2. Variant A — Specific behaviours before Acknowledge + ═══════════════════════════════════════════════════ */ + +function selectSummariseV2(assessment) { + if (assessment.phase.value === "synthesising") return { behaviour: "summarise", confidence: "high", reason: "Investigation is in synthesising phase — connected observations have accumulated and a restatement of current understanding will compress without losing detail." }; + if (assessment.phase.value === "concluding") return { behaviour: "summarise", confidence: "high", reason: "Investigation is concluding — a summary of resolved understanding provides closure anchor before the user decides next steps." }; + if (assessment.phase.evidence?.resolvedNodeCount >= 3 && assessment.progress.value === "steady") return { behaviour: "summarise", confidence: "medium", reason: "Three or more items resolved with steady progress — enough accumulated understanding warrants a compression pass." }; + return null; +} + +function selectPauseV2(assessment) { + if (assessment.phase.value === "focusing" && assessment.progress.value === "stalled") return { behaviour: "pause", confidence: "high", reason: "Focusing phase with stalled progress — the investigation has reached a single active unknown but momentum has stopped. Hold space rather than pushing for more." }; + if (assessment.conversationHealth.value === "user_overloaded") return { behaviour: "pause", confidence: "medium", reason: "User appears overloaded — reduce pressure by acknowledging progress before inviting further contribution." }; + return null; +} + +function selectAcknowledgeV2(assessment) { + if (assessment.conversationHealth.value === "healthy" && assessment.phase.confidence !== "low") return { behaviour: "acknowledge", confidence: "medium", reason: "Healthy conversation with established context — user provided useful information that warrants acknowledgment before introducing new uncertainty." }; + return null; +} + +function selectClarifyV2(assessment) { + if (assessment.conversationHealth.value === "too_broad") return { behaviour: "clarify", confidence: "high", reason: "Conversation health is too broad — investigation may be spreading too thin. Narrow focus through a specific clarification question." }; + if (assessment.phase.value === "orienting" && assessment.phase.evidence?.observationDensity < 3) return { behaviour: "clarify", confidence: "medium", reason: "Investigation is in orienting phase with insufficient observations (< 3). A targeted clarification question will anchor the starting point." }; + return null; +} + +function selectVariantA(assessment) { + if (!assessment || !assessment.conversationHealth || !assessment.phase) return { behaviour: "continue", confidence: "low", reason: "No assessment available — defaulting to continue." }; + const summarise = selectSummariseV2(assessment); if (summarise) return { ...summarise, priority: 1 }; + const pause = selectPauseV2(assessment); if (pause) return { ...pause, priority: 2 }; + const clarify = selectClarifyV2(assessment); if (clarify) return { ...clarify, priority: 3 }; + const acknowledge = selectAcknowledgeV2(assessment); if (acknowledge) return { ...acknowledge, priority: 4 }; + return { behaviour: "continue", confidence: "low", reason: "No explicit rule matched — defaulting to continue." }; +} + +/* ═══════════════════════════════════════════════════ + 3. Variant B — Existing priority with Acknowledge exclusions + ═══════════════════════════════════════════════════ */ + +function isAcknowledgeExcluded(assessment) { + if (["synthesising", "concluding"].includes(assessment.phase.value)) return true; + if (assessment.progress.value === "stalled") return true; + if (assessment.conversationHealth.value === "user_overloaded") return true; + return false; +} + +function selectVariantB(assessment) { + if (!assessment || !assessment.conversationHealth || !assessment.phase) return { behaviour: "continue", confidence: "low", reason: "No assessment available — defaulting to continue." }; + const acknowledge = selectAcknowledgeV2(assessment); + if (acknowledge && !isAcknowledgeExcluded(assessment)) return { ...acknowledge, priority: 1 }; + const clarify = selectClarifyV2(assessment); if (clarify) return { ...clarify, priority: 2 }; + const summarise = selectSummariseV2(assessment); if (summarise) return { ...summarise, priority: 3 }; + const pause = selectPauseV2(assessment); if (pause) return { ...pause, priority: 4 }; + return { behaviour: "continue", confidence: "low", reason: "No explicit rule matched — defaulting to continue." }; +} + +/* ═══════════════════════════════════════════════════ + 4. Real-scenario fixtures (imported from Exp 39/40) + ═══════════════════════════════════════════════════ */ + +function mkN(id, label, opts = {}) { + const kind = opts.kind || "unknown"; + const status = opts.status || (kind === "unknown" ? "unknown" : "known"); + const confidence = opts.confidence || (kind === "unknown" ? "low" : "high"); + return { id, label, description: label, kind, status, confidence, evidenceIds: [], dependsOn: [], affects: [], childIds: [] }; +} + +function buildInput(turn) { + return { + situationGraph: { centralStatement: turn.centralStatement, currentSummary: turn.currentSummary, nodes: turn.nodes, edges: [], activeUnknownNodeId: turn.activeUnknownNodeId, resolvedNodeIds: turn.resolvedNodeIds }, + selectedQuestion: turn.selectedQuestion, noQuestionReason: turn.noQuestionReason, + diagnostics: { promptVersion: "v0.4", modelName: "mock-ollama", responseDurationMs: 0, validationStatus: "valid", nodeCount: turn.nodes.length, edgeCount: 0, reasoningPattern: turn.diagnosticReasoningPattern || null } + }; +} + +function getRealTurns() { + return [ + { scenario: "long-investigation", turnNumber: 0, centralStatement: "Should we enter the European market with our SaaS analytics platform?", nodes: [mkN("obs-1", "Current revenue is $2M ARR in the US market only", { kind: "observation", status: "known", confidence: "high" }), mkN("state-1", "Evaluating European market entry", { kind: "state", status: "provisional", confidence: "medium" }), mkN("u-1", "Whether there is genuine demand for our category in Europe")], resolvedNodeIds: [], activeUnknownNodeId: "u-1", selectedQuestion: { nodeId: "u-1", question: "How large and mature is the analytics SaaS market in Europe?", reason: "market_validity" }, currentSummary: "We are US-based. The first question before any expansion is whether demand exists.", diagnosticReasoningPattern: "market_validity" }, + { scenario: "long-investigation", turnNumber: 3, centralStatement: "Should we enter the European market with our SaaS analytics platform?", nodes: [mkN("obs-1", "Current revenue is $2M ARR in the US market only", { kind: "observation", status: "known", confidence: "high" }), mkN("obs-2", "European analytics SaaS market valued at approximately €8B and growing 15% annually", { kind: "observation", status: "known", confidence: "medium" }), mkN("obs-3", "Our platform does not currently support EU data residency requirements", { kind: "observation", status: "known", confidence: "high" }), mkN("obs-4", "Achieving compliance would require approximately 6 months and $500K engineering investment", { kind: "observation", status: "known", confidence: "medium" }), mkN("state-1", "Evaluating European market entry", { kind: "state", status: "provisional", confidence: "medium" }), mkN("u-1", "Whether there is genuine demand for our category in Europe", { status: "resolved", confidence: "medium" }), mkN("u-2", "Whether our product is suitable for European compliance requirements", { status: "resolved", confidence: "high" }), mkN("u-3", "Whether the cost of achieving compliance is justified by the market size", { status: "resolved", confidence: "medium" }), mkN("u-4", "Whether we have competitive differentiation against existing European players")], resolvedNodeIds: ["u-1", "u-2", "u-3"], activeUnknownNodeId: "u-4", selectedQuestion: { nodeId: "u-4", question: "What differentiates our platform against established European competitors?", reason: "competitive_analysis" }, currentSummary: "Compliance is feasible. The remaining question is competitive edge.", diagnosticReasoningPattern: "competitive_analysis" }, + { scenario: "long-investigation", turnNumber: 4, centralStatement: "Should we enter the European market with our SaaS analytics platform?", nodes: [mkN("obs-1", "Current revenue is $2M ARR in the US market only", { kind: "observation", status: "known", confidence: "high" }), mkN("obs-2", "European analytics SaaS market valued at approximately €8B and growing 15% annually", { kind: "observation", status: "known", confidence: "medium" }), mkN("obs-3", "Our platform does not currently support EU data residency requirements", { kind: "observation", status: "known", confidence: "high" }), mkN("obs-4", "Achieving compliance would require approximately 6 months and $500K engineering investment", { kind: "observation", status: "known", confidence: "medium" }), mkN("obs-5", "Our real-time collaboration feature has no direct European equivalent", { kind: "observation", status: "provisional", confidence: "medium" }), mkN("state-1", "Evaluating European market entry", { kind: "state", status: "provisional", confidence: "medium" }), mkN("u-1", "Whether there is genuine demand for our category in Europe", { status: "resolved", confidence: "medium" }), mkN("u-2", "Whether our product is suitable for European compliance requirements", { status: "resolved", confidence: "high" }), mkN("u-3", "Whether the cost of achieving compliance is justified by the market size", { status: "resolved", confidence: "medium" }), mkN("u-4", "Whether we have competitive differentiation against existing European players", { status: "resolved", confidence: "medium" })], resolvedNodeIds: ["u-1", "u-2", "u-3", "u-4"], activeUnknownNodeId: null, selectedQuestion: null, noQuestionReason: "All investigation areas resolved.", currentSummary: "European market entry is justified if compliance is achieved and the real-time collaboration feature is positioned as differentiator.", diagnosticReasoningPattern: null }, + { scenario: "contradictory-evidence", turnNumber: 0, centralStatement: "Two consultants give opposite recommendations on which supplier to choose for a $2M procurement.", nodes: [mkN("obs-1", "Consultant A recommends Supplier X: lower cost, proven track record", { kind: "observation", status: "known", confidence: "high" }), mkN("obs-2", "Consultant B recommends Supplier Y: better integration capability, higher risk but long-term upside", { kind: "observation", status: "known", confidence: "high" }), mkN("obs-3", "Supplier X has 15+ years in the sector; Supplier Y has 2 years and mixed client reviews", { kind: "observation", status: "known", confidence: "medium" }), mkN("obs-4", "Our current infrastructure is compatible with neither supplier out of the box", { kind: "observation", status: "known", confidence: "high" }), mkN("state-1", "Evaluating $2M procurement against conflicting expert advice", { kind: "state", status: "provisional", confidence: "medium" }), mkN("u-1", "Whether the conflict is genuine or reflects different evaluation criteria")], resolvedNodeIds: [], activeUnknownNodeId: "u-1", selectedQuestion: { nodeId: "u-1", question: "Are the consultants evaluating the same criteria, or are they measuring different things?", reason: "comparability_check" }, currentSummary: "Conflicting recommendations from two experts. The first uncertainty is whether we are comparing the same dimensions.", diagnosticReasoningPattern: "comparability_check" }, + { scenario: "contradictory-evidence", turnNumber: 1, centralStatement: "Two consultants give opposite recommendations on which supplier to choose for a $2M procurement.", nodes: [mkN("obs-1", "Consultant A recommends Supplier X: lower cost, proven track record", { kind: "observation", status: "known", confidence: "high" }), mkN("obs-2", "Consultant B recommends Supplier Y: better integration capability, higher risk but long-term upside", { kind: "observation", status: "known", confidence: "high" }), mkN("obs-3", "Supplier X has 15+ years in the sector; Supplier Y has 2 years and mixed client reviews", { kind: "observation", status: "known", confidence: "medium" }), mkN("obs-4", "Our current infrastructure is compatible with neither supplier out of the box", { kind: "observation", status: "known", confidence: "high" }), mkN("obs-5", "The consultants used different evaluation weights: cost 60% vs integration 60%", { kind: "observation", status: "known", confidence: "medium" }), mkN("state-1", "Evaluating $2M procurement against conflicting expert advice", { kind: "state", status: "provisional", confidence: "medium" }), mkN("u-1", "Whether the conflict is genuine or reflects different evaluation criteria", { status: "resolved", confidence: "high" }), mkN("u-2", "Which supplier's strengths align with our strategic priorities")], resolvedNodeIds: ["u-1"], activeUnknownNodeId: "u-2", selectedQuestion: { nodeId: "u-2", question: "Does cost or integration capability matter more to the organisation over a 3-year horizon?", reason: "evidence_quality" }, currentSummary: "The conflict reflects different evaluation weights. The next uncertainty is strategic alignment.", diagnosticReasoningPattern: "evidence_quality" }, + { scenario: "contradictory-evidence", turnNumber: 2, centralStatement: "Two consultants give opposite recommendations on which supplier to choose for a $2M procurement.", nodes: [mkN("obs-1", "Consultant A recommends Supplier X: lower cost, proven track record", { kind: "observation", status: "known", confidence: "high" }), mkN("obs-2", "Consultant B recommends Supplier Y: better integration capability, higher risk but long-term upside", { kind: "observation", status: "known", confidence: "high" }), mkN("obs-3", "Supplier X has 15+ years in the sector; Supplier Y has 2 years and mixed client reviews", { kind: "observation", status: "known", confidence: "medium" }), mkN("obs-4", "Our current infrastructure is compatible with neither supplier out of the box", { kind: "observation", status: "known", confidence: "high" }), mkN("obs-5", "The consultants used different evaluation weights: cost 60% vs integration 60%", { kind: "observation", status: "known", confidence: "medium" }), mkN("obs-6", "Our strategic plan prioritises long-term capability over short-term cost savings", { kind: "observation", status: "known", confidence: "high" }), mkN("state-1", "Evaluating $2M procurement against conflicting expert advice", { kind: "state", status: "provisional", confidence: "medium" }), mkN("u-1", "Whether the conflict is genuine or reflects different evaluation criteria", { status: "resolved", confidence: "high" }), mkN("u-2", "Which supplier's strengths align with our strategic priorities", { status: "resolved", confidence: "medium" }), mkN("u-3", "Whether the integration risk of Supplier Y is manageable with internal resources")], resolvedNodeIds: ["u-1", "u-2"], activeUnknownNodeId: "u-3", selectedQuestion: { nodeId: "u-3", question: "Do we have the internal capacity to manage Supplier Y's integration risk?", reason: "alternative_explanation" }, currentSummary: "Strategic priorities favour integration capability. The remaining uncertainty is operational feasibility.", diagnosticReasoningPattern: "alternative_explanation" }, + { scenario: "short-early", turnNumber: 0, centralStatement: "A manufacturing company reports complaints increased by 35% while production increased by 40%.", nodes: [mkN("obs-1", "Complaints increased by 35%", { kind: "observation", status: "known", confidence: "high" }), mkN("obs-2", "Production increased by 40%", { kind: "observation", status: "known", confidence: "high" }), mkN("state-1", "Current situation", { kind: "state", status: "provisional", confidence: "medium" }), mkN("u-1", "Whether the two figures cover the same period")], resolvedNodeIds: [], activeUnknownNodeId: "u-1", selectedQuestion: { nodeId: "u-1", question: "Were the complaint and production figures measured over the same period?", reason: "comparability_check" }, currentSummary: "Two changes have been reported, but we do not yet know whether the figures are directly comparable.", diagnosticReasoningPattern: "comparability_check" } + ]; +} + +/* ═══════════════════════════════════════════════════ + 5. Evaluation helper + ═══════════════════════════════════════════════════ */ + +function evaluateTurn(turn) { + const input = buildInput(turn); + const assessment = assessInvestigationState(input); + // Deep-clone to verify immutability later + const originalHealthValue = assessment.conversationHealth.value; + const originalPhaseConfidence = assessment.phase.confidence; + const originalProgressValue = assessment.progress.value; + + const existingSel = selectBehaviour(assessment); + const varASel = selectVariantA(assessment); + const varBSel = selectVariantB(assessment); + + // Verify immutability + expect(assessment.conversationHealth.value).toBe(originalHealthValue); + expect(assessment.phase.confidence).toBe(originalPhaseConfidence); + expect(assessment.progress.value).toBe(originalProgressValue); + + return { + scenario: turn.scenario, + turnNumber: turn.turnNumber, + centralStatementShort: turn.centralStatement.substring(0, 55) + (turn.centralStatement.length > 55 ? "…" : ""), + phase: assessment.phase.value, + phaseConfidence: assessment.phase.confidence, + progress: assessment.progress.value, + health: assessment.conversationHealth.value, + existingSelection: existingSel.behaviour, + variantASelection: varASel.behaviour, + variantBSelection: varBSel.behaviour, + _assessment: assessment + }; +} + +/* ═══════════════════════════════════════════════════ + 6. Classification helper + ═══════════════════════════════════════════════════ */ + +function classifyChange(turnIdx, originalSel, newSel) { + if (originalSel === newSel) return "unchanged"; + const turn = getRealTurns()[turnIdx]; + const input = buildInput(turn); + const assessment = assessInvestigationState(input); + const { phase, progress, health } = { phase: assessment.phase.value, progress: assessment.progress.value, health: assessment.conversationHealth.value }; + + if (newSel === "summarise" && ["synthesising", "concluding"].includes(phase)) return "sensible"; + if (newSel === "summarise" && phase.evidence?.resolvedNodeCount >= 3 && progress === "steady") return "sensible"; + + if (newSel === "pause" && ((phase === "focusing" && progress === "stalled") || health === "user_overloaded")) return "sensible"; + + if (newSel === "acknowledge" && health === "healthy" && assessment.phase.confidence !== "low") return "sensible"; + + // Check whether original was sensible and new one is questionable + const origWasSensible = (originalSel === "acknowledge" && health === "healthy" && assessment.phase.confidence !== "low"); + if (origWasSensible && newSel !== "acknowledge") { + return "questionable"; // Original was sensible, the variant changed it to something less fitting + } + + if (newSel === "continue") return "sensible"; + + if (newSel === "summarise" || newSel === "pause") return "questionable"; // no clear rule match found + return "cannot determine"; +} + +/* ═══════════════════════════════════════════════════ + Test suite — Experiment 41 + ═══════════════════════════════════════════════════ */ + +describe("Experiment 41 — Counterfactual: Acknowledge Priority Alternatives", () => { + + /* ── Variant A contract tests ──────────────── */ + + describe("Variant A implements only priority reordering", () => { + it("prioritises Summarise before Acknowledge", () => { + const result = selectVariantA(mkAssessment({ phase: { value: "concluding", confidence: "high" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "healthy", confidence: "high" } })); + expect(result.behaviour).toBe("summarise"); + }); + + it("prioritises Pause before Acknowledge when conditions match", () => { + const result = selectVariantA(mkAssessment({ phase: { value: "focusing", confidence: "high" }, progress: { value: "stalled",confidence: "high" }, conversationHealth: { value: "healthy", confidence: "high" } })); + expect(result.behaviour).toBe("pause"); + }); + + it("does not change non-conflicting selections", () => { + const result = selectVariantA(mkAssessment({ phase: { value: "exploring", confidence: "low" }, progress: { value: "cannot_determine", confidence: "low" }, conversationHealth: { value: "too_broad", confidence: "high" } })); + expect(result.behaviour).toBe("clarify"); // no ack conditions met, clarify is only match + }); + + it("preserves Continue fallback", () => { + const result = selectVariantA(mkAssessment({ phase: { value: "exploring", confidence: "low" }, progress: { value: "cannot_determine", confidence: "low" }, conversationHealth: { value: "too_narrow", confidence: "low" } })); + expect(result.behaviour).toBe("continue"); + }); + + it("preserves Clarify rule when health=too_broad", () => { + const result = selectVariantA(mkAssessment({ phase: { value: "orienting", confidence: "low" }, progress: { value: "cannot_determine", confidence: "low" }, conversationHealth: { value: "too_broad", confidence: "high" } })); + expect(result.behaviour).toBe("clarify"); + }); + }); + + /* ── Variant B contract tests ──────────────── */ + + describe("Variant B implements only Acknowledge exclusions", () => { + it("blocks Acknowledge when phase=synthesising", () => { + const result = selectVariantB(mkAssessment({ phase: { value: "synthesising", confidence: "high" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "healthy", confidence: "high" } })); + expect(result.behaviour).toBe("summarise"); // Acknowledge excluded, summarise fires next + }); + + it("blocks Acknowledge when phase=concluding", () => { + const result = selectVariantB(mkAssessment({ phase: { value: "concluding", confidence: "high" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "healthy", confidence: "high" } })); + expect(result.behaviour).toBe("summarise"); // Acknowledge excluded, summarise fires next + }); + + it("blocks Acknowledge when progress=stalled", () => { + const result = selectVariantB(mkAssessment({ phase: { value: "focusing", confidence: "high" }, progress: { value: "stalled", confidence: "high" }, conversationHealth: { value: "healthy", confidence: "medium" } })); + expect(result.behaviour).toBe("pause"); // Acknowledge excluded, pause fires next + }); + + it("blocks Acknowledge when health=user_overloaded", () => { + const result = selectVariantB(mkAssessment({ phase: { value: "exploring", confidence: "medium" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "user_overloaded", confidence: "medium" } })); + expect(result.behaviour).toBe("pause"); // Acknowledge excluded, pause fires next + }); + + it("preserves Acknowledge when no exclusion conditions match", () => { + const result = selectVariantB(mkAssessment({ phase: { value: "focusing", confidence: "high" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "healthy", confidence: "high" } })); + expect(result.behaviour).toBe("acknowledge"); // focusing≠synthesising/concluding, progress=steady≠stalled, health=healthy≠user_overloaded + }); + + it("preserves Continue fallback", () => { + const result = selectVariantB(mkAssessment({ phase: { value: "exploring", confidence: "low" }, progress: { value: "cannot_determine", confidence: "low" }, conversationHealth: { value: "too_narrow", confidence: "low" } })); + expect(result.behaviour).toBe("continue"); + }); + + it("preserves Clarify rule when health=too_broad and phase low confidence", () => { + const result = selectVariantB(mkAssessment({ phase: { value: "orienting", confidence: "low" }, progress: { value: "cannot_determine", confidence: "low" }, conversationHealth: { value: "too_broad", confidence: "high" } })); + expect(result.behaviour).toBe("clarify"); + }); + + it("preserves Summarise rule when phase=synthesising and Acknowledge excluded", () => { + const result = selectVariantB(mkAssessment({ phase: { value: "synthesising", confidence: "high" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "healthy", confidence: "high" } })); + expect(result.behaviour).toBe("summarise"); // Acknowledge excluded, summarise fires at priority 3 + }); + }); + + /* ── Key scenario reachability tests ───────── */ + + describe("Key scenario reachability", () => { + it("concluding state can reach Summarise under Variant A", () => { + const result = selectVariantA(mkAssessment({ phase: { value: "concluding", confidence: "high" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "healthy", confidence: "high" } })); + expect(result.behaviour).toBe("summarise"); + }); + + it("stalled focusing state can reach Pause under Variant A", () => { + const result = selectVariantA(mkAssessment({ phase: { value: "focusing", confidence: "high" }, progress: { value: "stalled", confidence: "high" }, conversationHealth: { value: "healthy", confidence: "high" } })); + expect(result.behaviour).toBe("pause"); + }); + + it("healthy ordinary progress can still reach Acknowledge under Variant A", () => { + const result = selectVariantA(mkAssessment({ phase: { value: "focusing", confidence: "high" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "healthy", confidence: "high" } })); + expect(result.behaviour).toBe("acknowledge"); // summarise/pause not eligible, ack is 4th but wins + }); + + it("concluding state can reach Summarise under Variant B", () => { + const result = selectVariantB(mkAssessment({ phase: { value: "concluding", confidence: "high" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "healthy", confidence: "high" } })); + expect(result.behaviour).toBe("summarise"); + }); + + it("stalled focusing state can reach Pause under Variant B", () => { + const result = selectVariantB(mkAssessment({ phase: { value: "focusing", confidence: "high" }, progress: { value: "stalled", confidence: "high" }, conversationHealth: { value: "healthy", confidence: "medium" } })); + expect(result.behaviour).toBe("pause"); + }); + + it("healthy ordinary progress can still reach Acknowledge under Variant B", () => { + const result = selectVariantB(mkAssessment({ phase: { value: "focusing", confidence: "high" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "healthy", confidence: "high" } })); + expect(result.behaviour).toBe("acknowledge"); // no exclusion conditions match (focusing≠synthesising/concluding, steady≠stalled) + }); + + it("fallback remains Continue when no rule matches", () => { + const resultA = selectVariantA(mkAssessment({ phase: { value: "exploring", confidence: "low" }, progress: { value: "cannot_determine", confidence: "low" }, conversationHealth: { value: "too_narrow", confidence: "low" } })); + const resultB = selectVariantB(mkAssessment({ phase: { value: "exploring", confidence: "low" }, progress: { value: "cannot_determine", confidence: "low" }, conversationHealth: { value: "too_narrow", confidence: "low" } })); + expect(resultA.behaviour).toBe("continue"); + expect(resultB.behaviour).toBe("continue"); + }); + + it("early low-confidence state falls back safely under both variants", () => { + const result = selectVariantA(mkAssessment({ phase: { value: "cannot_determine", confidence: "low" }, progress: { value: "cannot_determine", confidence: "low" }, conversationHealth: { value: "too_narrow", confidence: "low" } })); + expect(result.behaviour).toBe("continue"); + const resultB = selectVariantB(mkAssessment({ phase: { value: "cannot_determine", confidence: "low" }, progress: { value: "cannot_determine", confidence: "low" }, conversationHealth: { value: "too_narrow", confidence: "low" } })); + expect(resultB.behaviour).toBe("continue"); + }); + }); + + /* ── Existing selector unchanged ───────────── */ + + describe("Existing selector results remain unchanged", () => { + it("returns acknowledge for healthy phase with confidence (base behaviour)", () => { + const result = selectBehaviour(mkAssessment({ phase: { value: "focusing", confidence: "high" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "healthy", confidence: "high" } })); + expect(result.behaviour).toBe("acknowledge"); + }); + + it("returns continue for null input (base behaviour)", () => { + const result = selectBehaviour(null); + expect(result.behaviour).toBe("continue"); + }); + + it("only outputs valid behaviours (base contract)", () => { + const behaviours = []; + for (const phase of ["cannot_determine", "exploring", "orienting", "deepening", "focusing", "synthesising", "concluding"]) { + for (const conf of ["high", "medium", "low"]) { + const result = selectBehaviour(mkAssessment({ phase: { value: phase, confidence: conf }, progress: { value: "cannot_determine", confidence: "low" }, conversationHealth: { value: "cannot_determine", confidence: "low" } })); + behaviours.push(result.behaviour); + } + } + for (const b of behaviours) expect(["acknowledge", "clarify", "summarise", "pause", "continue"]).toContain(b); + }); + }); + + /* ── Seven real assessment turns comparison ─ */ + + describe("Seven real assessment turns compared across all selectors", () => { + let results; + + beforeEach(() => { + const turns = getRealTurns(); + results = turns.map(t => evaluateTurn(t)); + }); + + it("evaluates all seven turns", () => expect(results).toHaveLength(7)); + + it("outputs are deterministic across calls", () => { + for (let i = 0; i < results.length; i++) { + const turn2 = getRealTurns()[i]; + const result2 = evaluateTurn(turn2); + expect(result2.existingSelection).toBe(results[i].existingSelection); + expect(result2.variantASelection).toBe(results[i].variantASelection); + expect(result2.variantBSelection).toBe(results[i].variantBSelection); + } + }); + + it("inputs are not mutated", () => { + const turns = getRealTurns(); + for (const turn of turns) { + const input = buildInput(turn); + const snapshot = JSON.stringify(input); + assessInvestigationState(input); + selectVariantA(assessInvestigationState(JSON.parse(snapshot))); + selectVariantB(assessInvestigationState(JSON.parse(snapshot))); + expect(JSON.stringify(input)).toBe(snapshot); + } + }); + + it("reports assessor output for each turn", () => { + for (const r of results) { + console.log(`\n=== ${r.scenario} turn ${r.turnNumber} ===`); + console.log(` phase=${r.phase}(conf:${r.phaseConfidence}), progress=${r.progress}, health=${r.health}`); + console.log(` existing: ${r.existingSelection} | variant A: ${r.variantASelection} | variant B: ${r.variantBSelection}`); + } + }); + + it("records whether each change appears sensible/questionable", () => { + const classifications = {}; + for (let i = 0; i < results.length; i++) { + const r = results[i]; + classifications[`existing-${r.scenario}-t${r.turnNumber}`] = "unchanged"; // baseline + + if (r.existingSelection !== r.variantASelection) { + classifications[`variantA-${r.scenario}-t${r.turnNumber}`] = classifyChange(i, r.existingSelection, r.variantASelection); + } else { + classifications[`variantA-${r.scenario}-t${r.turnNumber}`] = "unchanged"; + } + + if (r.existingSelection !== r.variantBSelection) { + classifications[`variantB-${r.scenario}-t${r.turnNumber}`] = classifyChange(i, r.existingSelection, r.variantBSelection); + } else { + classifications[`variantB-${r.scenario}-t${r.turnNumber}`] = "unchanged"; + } + } + + console.log("\n=== Selection Classifications ==="); + for (const [key, val] of Object.entries(classifications)) { + console.log(` ${key}: ${val}`); + } + + global._exp41_classifications = classifications; + }); + + it("turn-level summary", () => { + const distA = { acknowledge: 0, clarify: 0, summarise: 0, pause: 0, continue: 0 }; + const distB = { acknowledge: 0, clarify: 0, summarise: 0, pause: 0, continue: 0 }; + + for (const r of results) { + distA[r.variantASelection]++; + distB[r.variantBSelection]++; + } + + console.log("\n=== Existing Distribution ==="); + const existingDist = { acknowledge: 0, clarify: 0, summarise: 0, pause: 0, continue: 0 }; + for (const r of results) existingDist[r.existingSelection]++; + for (const [beh, count] of Object.entries(existingDist)) console.log(` ${beh}: ${count} (${((count/7)*100).toFixed(0)}%)`); + + console.log("\n=== Variant A Distribution ==="); + for (const [beh, count] of Object.entries(distA)) console.log(` ${beh}: ${count} (${((count/7)*100).toFixed(0)}%)`); + + console.log("\n=== Variant B Distribution ==="); + for (const [beh, count] of Object.entries(distB)) console.log(` ${beh}: ${count} (${((count/7)*100).toFixed(0)}%)`); + + global._exp41_distA = distA; + global._exp41_distB = distB; + }); + }); + + /* ── Synthetic safety checks ───────────────── */ + + describe("Synthetic safety checks", () => { + it("healthy mid-investigation progress can reach Acknowledge under both variants", () => { + const a = selectVariantA(mkAssessment({ phase: { value: "focusing", confidence: "high" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "healthy", confidence: "high" } })); + const b = selectVariantB(mkAssessment({ phase: { value: "focusing", confidence: "high" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "healthy", confidence: "high" } })); + expect(a.behaviour).toBe("acknowledge"); + expect(b.behaviour).toBe("acknowledge"); + }); + + it("concluding state reaches Summarise under both variants", () => { + const a = selectVariantA(mkAssessment({ phase: { value: "concluding", confidence: "high" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "healthy", confidence: "high" } })); + const b = selectVariantB(mkAssessment({ phase: { value: "concluding", confidence: "high" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "healthy", confidence: "high" } })); + expect(a.behaviour).toBe("summarise"); + expect(b.behaviour).toBe("summarise"); + }); + + it("focusing + stalled reaches Pause under both variants", () => { + const a = selectVariantA(mkAssessment({ phase: { value: "focusing", confidence: "high" }, progress: { value: "stalled", confidence: "high" }, conversationHealth: { value: "healthy", confidence: "medium" } })); + const b = selectVariantB(mkAssessment({ phase: { value: "focusing", confidence: "high" }, progress: { value: "stalled", confidence: "high" }, conversationHealth: { value: "healthy", confidence: "medium" } })); + expect(a.behaviour).toBe("pause"); + expect(b.behaviour).toBe("pause"); + }); + + it("user_overloaded reaches Pause under both variants", () => { + const a = selectVariantA(mkAssessment({ phase: { value: "exploring", confidence: "medium" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "user_overloaded", confidence: "medium" } })); + const b = selectVariantB(mkAssessment({ phase: { value: "exploring", confidence: "medium" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "user_overloaded", confidence: "medium" } })); + expect(a.behaviour).toBe("pause"); + expect(b.behaviour).toBe("pause"); + }); + + it("early low-confidence state falls back to Continue under both variants", () => { + const a = selectVariantA(mkAssessment({ phase: { value: "exploring", confidence: "low" }, progress: { value: "cannot_determine", confidence: "low" }, conversationHealth: { value: "too_narrow", confidence: "low" } })); + const b = selectVariantB(mkAssessment({ phase: { value: "exploring", confidence: "low" }, progress: { value: "cannot_determine", confidence: "low" }, conversationHealth: { value: "too_narrow", confidence: "low" } })); + expect(a.behaviour).toBe("continue"); + expect(b.behaviour).toBe("continue"); + }); + + it("state where no explicit rule matches returns Continue under both variants", () => { + const a = selectVariantA(mkAssessment({ phase: { value: "deepening", confidence: "medium" }, progress: { value: "steady", confidence: "medium" } })); + const b = selectVariantB(mkAssessment({ phase: { value: "deepening", confidence: "medium" }, progress: { value: "steady", confidence: "medium" } })); + expect(a.behaviour).toBe("continue"); + expect(b.behaviour).toBe("continue"); + }); + + it("neither variant causes Pause too early in normal progress", () => { + const a = selectVariantA(mkAssessment({ phase: { value: "focusing", confidence: "high" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "healthy", confidence: "high" } })); + const b = selectVariantB(mkAssessment({ phase: { value: "focusing", confidence: "high" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "healthy", confidence: "high" } })); + expect(a.behaviour).not.toBe("pause"); + expect(b.behaviour).not.toBe("pause"); + }); + + it("neither variant causes Summarise too early in normal progress", () => { + const a = selectVariantA(mkAssessment({ phase: { value: "focusing", confidence: "high" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "healthy", confidence: "high" } })); + expect(a.behaviour).not.toBe("summarise"); + }); + + it("early incomplete states fall back safely under both variants", () => { + const a = selectVariantA(mkAssessment({ phase: { value: "cannot_determine", confidence: "low" }, progress: { value: "cannot_determine", confidence: "low" }, conversationHealth: { value: "too_narrow", confidence: "low" } })); + const b = selectVariantB(mkAssessment({ phase: { value: "cannot_determine", confidence: "low" }, progress: { value: "cannot_determine", confidence: "low" }, conversationHealth: { value: "too_narrow", confidence: "low" } })); + expect(a.behaviour).toBe("continue"); + expect(b.behaviour).toBe("continue"); + }); + }); + + /* ── Both variants produce identical results across tested turns ─ */ + + describe("Variant A vs Variant B — divergence analysis", () => { + it("both variants converge on the same two genuine changes (concluding→summarise, stalled→pause) despite different mechanisms", () => { + const turns = getRealTurns(); + // Variant A diverges at long-investigation t3 (focusing with 3 resolved items) + // because its resolvedNodeCount >= 3 rule fires without phase context — a side-effect. + // Variant B correctly preserves Acknowledge there via explicit exclusion list. + // Both converge on the two genuine changes: concluding→summarise and stalled→pause. + + const divergeIdx = 1; // long-investigation t3 (Variant A over-summarises here) + expect(selectVariantA(assessInvestigationState(buildInput(turns[divergeIdx]))).behaviour).toBe("summarise"); + expect(selectVariantB(assessInvestigationState(buildInput(turns[divergeIdx]))).behaviour).toBe("acknowledge"); + + // Both converge: concluding turn + const a4 = selectVariantA(assessInvestigationState(buildInput(turns[2]))); + const b4 = selectVariantB(assessInvestigationState(buildInput(turns[2]))); + expect(a4.behaviour).toBe("summarise"); + expect(b4.behaviour).toBe("summarise"); + + // Both converge: stalled focusing turn + const c1 = selectVariantA(assessInvestigationState(buildInput(turns[4]))); + const d1 = selectVariantB(assessInvestigationState(buildInput(turns[4]))); + expect(c1.behaviour).toBe("pause"); + expect(d1.behaviour).toBe("pause"); + }); + }); + + /* ── Specific expected outcomes ────────────── */ + + describe("Specific expected outcomes", () => { + it("concluding turn (long-investigation t4) selects Summarise under both variants", () => { + const turns = getRealTurns(); + const r = evaluateTurn(turns[2]); // long-investigation t4 + expect(r.variantASelection).toBe("summarise"); + expect(r.variantBSelection).toBe("summarise"); + }); + + it("stalled focusing turn (contradictory-evidence t1) selects Pause under both variants", () => { + const turns = getRealTurns(); + const r = evaluateTurn(turns[4]); // contradictory-evidence t1 + expect(r.variantASelection).toBe("pause"); + expect(r.variantBSelection).toBe("pause"); + }); + + it("ordinary healthy progress still allows Acknowledge under Variant B (Variant A over-summarises mid-focus)", () => { + const turns = getRealTurns(); + // long-investigation t3: Variant A incorrectly produces summarise because resolvedNodeCount>=3 fires before acknowledging (side-effect) + expect(evaluateTurn(turns[1]).variantASelection).toBe("summarise"); + expect(evaluateTurn(turns[1]).variantBSelection).toBe("acknowledge"); + // contradictory-t0 and t2: both preserve acknowledge + for (const idx of [3, 5]) { + const r = evaluateTurn(turns[idx]); + expect(r.variantASelection).toBe("acknowledge"); + expect(r.variantBSelection).toBe("acknowledge"); + } + }); + + it("early incomplete states fall back to Continue under both variants", () => { + const turns = getRealTurns(); + for (const idx of [0, 6]) { // long-investigation t0, short-early-t0 + const r = evaluateTurn(turns[idx]); + expect(r.variantASelection).toBe("continue"); + expect(r.variantBSelection).toBe("continue"); + } + }); + }); +});