experiment: audit behaviour reachability and blocking

This commit is contained in:
2026-08-06 16:48:41 +01:00
parent da3d35f437
commit a4731d908f
4 changed files with 818 additions and 7 deletions
+7 -5
View File
@@ -36,11 +36,13 @@ Experiment 38 tested whether a genuinely cold session (no prior conversation con
Experiment 39 resumed reasoning experiments with a passive validation of Behaviour Selection against real Investigation State Assessment outputs. Seven turns across three scenarios were evaluated. Acknowledge dominated (71%) because it fires at priority 1 whenever health=healthy, even in terminal and stalled states where Summarise or Pause would be more useful. The assessor→selector contract aligns cleanly; no transformation is needed between pipeline stages. All five behaviours remain reachable but some never appear in typical scenarios (Clarify requires too_broad health which few fixtures produce). Status pending Rob's review.
Experiment 40 diagnosed the root causes: Summarise and Pause fire their rules in real data but are always blocked by Acknowledge's priority-1 position (priority conflict, not assessor failure). Clarive's triggers never activate in tested scenarios due to the `too_broad` health condition being extremely narrow. All five behaviours confirmed independently reachable in synthetic isolation. No rules changed.
## 5. What Remains Open
- Whether the Acknowledge→Summarise priority conflict can be resolved without breaking existing rule conditions;
- Whether behaviour selection produces different patterns in investigation domains other than supplier procurement, market entry, and product comparison;
- Which rules (Acknowledge, Clarify, Summarise, Pause) fire most frequently across a larger set of scenarios.
- Whether Acknowledge should be reordered past Summarise/Pause to resolve the priority conflict diagnosed in Exp 40;
- Whether the `too_broad` health trigger needs widening so Clarify fires in more typical investigations;
- Whether `user_overloaded` health should be producible by the assessor for stalled/inconsistent evidence states.
### When This Knowledge-Management Phase Is Complete
@@ -86,8 +88,8 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38, 39. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 38, 39, 40. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 39)
Experiment 39 resumed reasoning experiments with a passive validation of Behaviour Selection against real Investigation State Assessment outputs across three scenarios (long investigation, contradictory evidence, short early). No user-facing engine behaviour changed. The selector produced all valid behaviours but with skewed distribution: Acknowledge dominated at 71% because it fires first when health=healthy, even in terminal/concluding states where Summarise would be more useful — this is an acknowledged priority conflict, not a contract mismatch. The assessor→selector pipeline aligns cleanly. Clarify and Pause never fired (no too_broad health produced; other rules blocked). Status pending Rob's review. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when returning: `docs/current-handoff.md`, then `docs/design-evolution-log.md` entry for Experiment 39, and `tests/behaviour-selection.real-assessment.test.js` for evaluation details. The next action is discussion about whether to refine the selector or move on.
Experiment 40 audited why Clarify, Summarise, and Pause were absent in Experiment 39. Summarise and Pause are eligible_but_blocked: their rule conditions fire but Acknowledge (priority 1) always wins when health=healthy — a priority conflict. Clarify is never_eligible_in_tested_scenarios: its `too_broad` trigger requires activeUnknownCount > 3 AND resolved < 2 simultaneously, which no fixture reaches; the `orienting+low obs` path is never produced by the assessor for tested domains. All five behaviours confirmed independently reachable in synthetic isolation. No rules changed. Status pending Rob's review. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `docs/current-handoff.md`, then `docs/design-evolution-log.md` entry for Experiment 40, and `tests/behaviour-selection.reachability.test.js` for diagnostic details. The next action is discussion about whether to reorder Acknowledge lower (past Summarise/Pause) or refine its trigger to exclude terminal phases.
+2 -2
View File
@@ -75,7 +75,7 @@ The following were built during Experiments 1825B. They are isolated diagnost
## 6. Work Currently Paused
- Engine experiments are at Experiment 39 (passive Behaviour Selection validation, awaiting review);
- Engine experiments are at Experiment 40 (behaviour reachability diagnostic — no rule changes).
- UI experiments are paused;
- Knowledge-management experiments are complete (confirmed by Experiment 38 cold-start validation);
- Nothing historical has been deleted or archived yet.
@@ -100,7 +100,7 @@ Historical documents are retained under `docs/archive/` and should be opened onl
## 8. Return-to-Work Summary
Engine experiments advanced to Experiment 39, a passive validation of Behaviour Selection against real Investigation State Assessment outputs. The existing selector produces all valid behaviours but Acknowledge dominates (71%) because it fires first when health=healthy. No user-facing engine behaviour changed. Knowledge-management phase is complete: all seven criteria met. No backlog items deleted or promoted.
Engine experiments advanced to Experiment 40, a reachability diagnostic confirming Summarise and Pause fire their rules but are always blocked by Acknowledge (priority-1 conflict), while Clarive's triggers never activate in tested scenarios due to narrow `too_broad` health conditions. All five behaviours confirmed independently reachable synthetically. No rules or engine behaviour changed. Knowledge-management phase is complete: all seven criteria met. No backlog items deleted or promoted.
First document to read: `docs/current-project-state.md`. Then consult `.claude/architecture-guardrails.md` before any code changes and `docs/project-knowledge-inventory.md` for task-specific references. The full experiment history remains available in `docs/design-evolution-log.md` but is no longer default reading.
+80
View File
@@ -2324,3 +2324,83 @@ Pending Rob's review. Five behaviours are too narrow for this to be definitive,
- `docs/current-handoff.md` — return-to-work note replaced
---
## Experiment 40 — Audit Behaviour Reachability and Blocking (2026-08-06)
### Objective
Why did Clarify, Summarise, and Pause not appear during Experiment 39? Acknowledge: 5 (71%), Continue: 2 (29%), others: 0. This is a passive diagnostic — no rule changes, no engine modifications.
### Method
One test file (`tests/behaviour-selection.reachability.test.js`) containing:
- Diagnostic audit helper that evaluates every behaviour rule against one assessment object
- Real-scenario audits across the same Experiment 39 turns (8 turns total)
- Synthetic reachability checks for each behaviour in isolation
### Findings
#### Summarise — eligible_but_blocked
Eligible in 2 of 7 real turns:
- long-investigation turn 1 (resolvedNodeCount ≥ 3 + progress=steady triggers summarise rule)
- long-investigation turn 2 (phase=concluding triggers summarise rule)
In both cases, health=healthy simultaneously, so Acknowledge (priority 1) fires first. Summarise rules are met but its output is never returned because the selector returns early on priority ordering.
**Root cause: priority conflict, not assessor failure.** The phase evidence correctly identifies concluding/synthesising states; the problem is that Acknowledge's broader trigger condition (health=healthy is the most common state) fires first.
#### Clarify — never_eligible_in_tested_scenarios (reachable only in synthetic case)
Not eligible in any of 7 real turns because neither trigger condition is met:
- `health=too_broad`: requires activeUnknownCount > 3 AND resolvedNodeCount < 2 — no fixture reaches this state
- `phase=orienting + observationDensity < 3`: current assessor never produces phase=orienting for tested scenarios
Synthetic case confirms the rule fires correctly in isolation (with low-confidence phase to avoid Acknowledge blocking).
**Root cause: assessor health classification logic produces too few `too_broad` cases. The trigger condition is extremely narrow — needs activeUnknownCount > 3 AND resolved < 2 simultaneously.**
#### Pause — eligible_but_blocked
Eligible in 1 of 7 real turns:
- contradictory-evidence turn 1 (phase=focusing + progress=stalled triggers pause rule)
In this case, health=healthy simultaneously, so Acknowledge blocks it. The second pause trigger (`health=user_overloaded`) is never met because the assessor never produces that state.
**Root cause: same priority conflict as Summarise. One of two rules fires in real data but gets blocked by Acknowledge's earlier position.**
### Synthetic Reachability Confirmation
All five behaviours are independently reachable when isolated from Acknowledge:
- ✅ acknowledge — healthy + confident phase
- ✅ clarify — too_broad health (with low-confidence phase to avoid Acknowledge)
- ✅ summarise — synthesising/concluding phase (without healthy health)
- ✅ pause — focusing+stalled or user_overloaded (without healthy health)
- ✅ continue — no rules match
### Classifications
| Behaviour | Classification | Primary Cause |
|---|---|---|
| Summarise | eligible_but_blocked | Acknowledge priority 1 fires first when health=healthy |
| Clarify | never_eligible_in_tested_scenarios (reachable only in synthetic) | `too_broad` trigger too narrow for test scenarios; `orienting+low obs` not produced by assessor |
| Pause | eligible_but_blocked | Acknowledge priority 1 fires first when health=healthy; `user_overloaded` never produced |
### Impact on Prior Finding (Exp 39)
Experiment 39 concluded "the Acknowledge→Summarise priority conflict prevents Summarise from firing." Experiment 40 confirms this and adds that **Pause faces the same blocking** (1 eligible turn, blocked). Clarify's absence is fundamentally different: its rules are not triggered at all in tested scenarios.
This means any fix must address two distinct problems:
1. Priority conflict affecting Summarise AND Pause (same cause)
2. Narrow trigger conditions for Clarify and the `user_overloaded` health state
### Test Results
- `tests/behaviour-selection.reachability.test.js`: 33 passed (new diagnostic file)
- `tests/behaviour-selection.test.js`: 51 passed (no regressions)
- `tests/behaviour-selection.real-assessment.test.js`: 16 passed (shared fixtures intact)
- `tests/investigation-state-assessor.test.js`: 51 passed (assessor unchanged)
### Documents Updated
- `docs/design-evolution-log.md` — this entry
- `docs/current-handoff.md` — return-to-work note replaced
@@ -0,0 +1,729 @@
import { describe, it, expect } from "vitest";
import assessInvestigationState from "@/lib/assessment/investigation-state-assessor.js";
import selectBehaviour, { BEHAVIOUR_OPTIONS } from "@/lib/behaviour-selection/behaviour-selector.js";
/*
Experiment 40 Behaviour Reachability Diagnostic
*/
/* ── Helpers from the real-assessment test (shared fixtures) ─ */
function mkN(id, label, opts = {}) {
const kind = opts.kind || "unknown";
const status = opts.status || (kind === "unknown" ? "unknown" : "known");
const confidence = opts.confidence || (kind === "unknown" ? "low" : "high");
return {
id, label, description: label, kind, status, confidence,
evidenceIds: [], dependsOn: [], affects: [], childIds: []
};
}
function getEvaluationScenarios() {
return {
"long-investigation": [
{
centralStatement: "Should we enter the European market with our SaaS analytics platform?",
nodes: [
mkN("obs-1", "Current revenue is $2M ARR in the US market only", { kind: "observation", status: "known", confidence: "high" }),
mkN("state-1", "Evaluating European market entry", { kind: "state", status: "provisional", confidence: "medium" }),
mkN("u-1", "Whether there is genuine demand for our category in Europe")
],
resolvedNodeIds: [], activeUnknownNodeId: "u-1",
selectedQuestion: { nodeId: "u-1", question: "How large and mature is the analytics SaaS market in Europe?", reason: "market_validity" },
currentSummary: "We are US-based. The first question before any expansion is whether demand exists.",
diagnosticReasoningPattern: "market_validity"
},
{
centralStatement: "Should we enter the European market with our SaaS analytics platform?",
nodes: [
mkN("obs-1", "Current revenue is $2M ARR in the US market only", { kind: "observation", status: "known", confidence: "high" }),
mkN("obs-2", "European analytics SaaS market valued at approximately €8B and growing 15% annually", { kind: "observation", status: "known", confidence: "medium" }),
mkN("obs-3", "Our platform does not currently support EU data residency requirements", { kind: "observation", status: "known", confidence: "high" }),
mkN("obs-4", "Achieving compliance would require approximately 6 months and $500K engineering investment", { kind: "observation", status: "known", confidence: "medium" }),
mkN("state-1", "Evaluating European market entry", { kind: "state", status: "provisional", confidence: "medium" }),
mkN("u-1", "Whether there is genuine demand for our category in Europe", { status: "resolved", confidence: "medium" }),
mkN("u-2", "Whether our product is suitable for European compliance requirements", { status: "resolved", confidence: "high" }),
mkN("u-3", "Whether the cost of achieving compliance is justified by the market size", { status: "resolved", confidence: "medium" }),
mkN("u-4", "Whether we have competitive differentiation against existing European players")
],
resolvedNodeIds: ["u-1", "u-2", "u-3"], activeUnknownNodeId: "u-4",
selectedQuestion: { nodeId: "u-4", question: "What differentiates our platform against established European competitors?", reason: "competitive_analysis" },
currentSummary: "Compliance is feasible. The remaining question is competitive edge.",
diagnosticReasoningPattern: "competitive_analysis"
},
{
centralStatement: "Should we enter the European market with our SaaS analytics platform?",
nodes: [
mkN("obs-1", "Current revenue is $2M ARR in the US market only", { kind: "observation", status: "known", confidence: "high" }),
mkN("obs-2", "European analytics SaaS market valued at approximately €8B and growing 15% annually", { kind: "observation", status: "known", confidence: "medium" }),
mkN("obs-3", "Our platform does not currently support EU data residency requirements", { kind: "observation", status: "known", confidence: "high" }),
mkN("obs-4", "Achieving compliance would require approximately 6 months and $500K engineering investment", { kind: "observation", status: "known", confidence: "medium" }),
mkN("obs-5", "Our real-time collaboration feature has no direct European equivalent", { kind: "observation", status: "provisional", confidence: "medium" }),
mkN("state-1", "Evaluating European market entry", { kind: "state", status: "provisional", confidence: "medium" }),
mkN("u-1", "Whether there is genuine demand for our category in Europe", { status: "resolved", confidence: "medium" }),
mkN("u-2", "Whether our product is suitable for European compliance requirements", { status: "resolved", confidence: "high" }),
mkN("u-3", "Whether the cost of achieving compliance is justified by the market size", { status: "resolved", confidence: "medium" }),
mkN("u-4", "Whether we have competitive differentiation against existing European players", { status: "resolved", confidence: "medium" })
],
resolvedNodeIds: ["u-1", "u-2", "u-3", "u-4"], activeUnknownNodeId: null,
selectedQuestion: null, noQuestionReason: "All investigation areas resolved.",
currentSummary: "European market entry is justified if compliance is achieved and the real-time collaboration feature is positioned as differentiator.",
diagnosticReasoningPattern: null
}
],
"contradictory-evidence": [
{
centralStatement: "Two consultants give opposite recommendations on which supplier to choose for a $2M procurement.",
nodes: [
mkN("obs-1", "Consultant A recommends Supplier X: lower cost, proven track record", { kind: "observation", status: "known", confidence: "high" }),
mkN("obs-2", "Consultant B recommends Supplier Y: better integration capability, higher risk but long-term upside", { kind: "observation", status: "known", confidence: "high" }),
mkN("obs-3", "Supplier X has 15+ years in the sector; Supplier Y has 2 years and mixed client reviews", { kind: "observation", status: "known", confidence: "medium" }),
mkN("obs-4", "Our current infrastructure is compatible with neither supplier out of the box", { kind: "observation", status: "known", confidence: "high" }),
mkN("state-1", "Evaluating $2M procurement against conflicting expert advice", { kind: "state", status: "provisional", confidence: "medium" }),
mkN("u-1", "Whether the conflict is genuine or reflects different evaluation criteria")
],
resolvedNodeIds: [], activeUnknownNodeId: "u-1",
selectedQuestion: { nodeId: "u-1", question: "Are the consultants evaluating the same criteria, or are they measuring different things?", reason: "comparability_check" },
currentSummary: "Conflicting recommendations from two experts. The first uncertainty is whether we are comparing the same dimensions.",
diagnosticReasoningPattern: "comparability_check"
},
{
centralStatement: "Two consultants give opposite recommendations on which supplier to choose for a $2M procurement.",
nodes: [
mkN("obs-1", "Consultant A recommends Supplier X: lower cost, proven track record", { kind: "observation", status: "known", confidence: "high" }),
mkN("obs-2", "Consultant B recommends Supplier Y: better integration capability, higher risk but long-term upside", { kind: "observation", status: "known", confidence: "high" }),
mkN("obs-3", "Supplier X has 15+ years in the sector; Supplier Y has 2 years and mixed client reviews", { kind: "observation", status: "known", confidence: "medium" }),
mkN("obs-4", "Our current infrastructure is compatible with neither supplier out of the box", { kind: "observation", status: "known", confidence: "high" }),
mkN("obs-5", "The consultants used different evaluation weights: cost 60% vs integration 60%", { kind: "observation", status: "known", confidence: "medium" }),
mkN("state-1", "Evaluating $2M procurement against conflicting expert advice", { kind: "state", status: "provisional", confidence: "medium" }),
mkN("u-1", "Whether the conflict is genuine or reflects different evaluation criteria", { status: "resolved", confidence: "high" }),
mkN("u-2", "Which supplier's strengths align with our strategic priorities")
],
resolvedNodeIds: ["u-1"], activeUnknownNodeId: "u-2",
selectedQuestion: { nodeId: "u-2", question: "Does cost or integration capability matter more to the organisation over a 3-year horizon?", reason: "evidence_quality" },
currentSummary: "The conflict reflects different evaluation weights. The next uncertainty is strategic alignment.",
diagnosticReasoningPattern: "evidence_quality"
},
{
centralStatement: "Two consultants give opposite recommendations on which supplier to choose for a $2M procurement.",
nodes: [
mkN("obs-1", "Consultant A recommends Supplier X: lower cost, proven track record", { kind: "observation", status: "known", confidence: "high" }),
mkN("obs-2", "Consultant B recommends Supplier Y: better integration capability, higher risk but long-term upside", { kind: "observation", status: "known", confidence: "high" }),
mkN("obs-3", "Supplier X has 15+ years in the sector; Supplier Y has 2 years and mixed client reviews", { kind: "observation", status: "known", confidence: "medium" }),
mkN("obs-4", "Our current infrastructure is compatible with neither supplier out of the box", { kind: "observation", status: "known", confidence: "high" }),
mkN("obs-5", "The consultants used different evaluation weights: cost 60% vs integration 60%", { kind: "observation", status: "known", confidence: "medium" }),
mkN("obs-6", "Our strategic plan prioritises long-term capability over short-term cost savings", { kind: "observation", status: "known", confidence: "high" }),
mkN("state-1", "Evaluating $2M procurement against conflicting expert advice", { kind: "state", status: "provisional", confidence: "medium" }),
mkN("u-1", "Whether the conflict is genuine or reflects different evaluation criteria", { status: "resolved", confidence: "high" }),
mkN("u-2", "Which supplier's strengths align with our strategic priorities", { status: "resolved", confidence: "medium" }),
mkN("u-3", "Whether the integration risk of Supplier Y is manageable with internal resources")
],
resolvedNodeIds: ["u-1", "u-2"], activeUnknownNodeId: "u-3",
selectedQuestion: { nodeId: "u-3", question: "Do we have the internal capacity to manage Supplier Y's integration risk?", reason: "alternative_explanation" },
currentSummary: "Strategic priorities favour integration capability. The remaining uncertainty is operational feasibility.",
diagnosticReasoningPattern: "alternative_explanation"
}
],
"short-early": [
{
centralStatement: "A manufacturing company reports complaints increased by 35% while production increased by 40%.",
nodes: [
mkN("obs-1", "Complaints increased by 35%", { kind: "observation", status: "known", confidence: "high" }),
mkN("obs-2", "Production increased by 40%", { kind: "observation", status: "known", confidence: "high" }),
mkN("state-1", "Current situation", { kind: "state", status: "provisional", confidence: "medium" }),
mkN("u-1", "Whether the two figures cover the same period")
],
resolvedNodeIds: [], activeUnknownNodeId: "u-1",
selectedQuestion: { nodeId: "u-1", question: "Were the complaint and production figures measured over the same period?", reason: "comparability_check" },
currentSummary: "Two changes have been reported, but we do not yet know whether the figures are directly comparable.",
diagnosticReasoningPattern: "comparability_check"
}
]
};
}
function buildAssessmentInput(turn) {
return {
situationGraph: {
centralStatement: turn.centralStatement,
currentSummary: turn.currentSummary,
nodes: turn.nodes,
edges: turn.edges || [],
activeUnknownNodeId: turn.activeUnknownNodeId,
resolvedNodeIds: turn.resolvedNodeIds
},
selectedQuestion: turn.selectedQuestion,
noQuestionReason: turn.noQuestionReason,
diagnostics: {
promptVersion: "v0.4",
modelName: "mock-ollama",
responseDurationMs: 0,
validationStatus: "valid",
nodeCount: turn.nodes.length,
edgeCount: (turn.edges || []).length,
reasoningPattern: turn.diagnosticReasoningPattern || null
}
};
}
/*
Diagnostic audit helper checks every rule per turn
*/
const BEHAVIOUR_CHECKERS = [
{ key: "acknowledge", fn: (a) => a.conversationHealth.value === "healthy" && a.phase.confidence !== "low" },
{ key: "clarify", fn: (a) => a.conversationHealth.value === "too_broad" || (a.phase.value === "orienting" && (a.phase.evidence?.observationDensity ?? Infinity) < 3) },
{ key: "summarise", fn: (a) => a.phase.value === "synthesising" || a.phase.value === "concluding" || (a.phase.evidence?.resolvedNodeCount >= 3 && a.progress.value === "steady") },
{ key: "pause", fn: (a) => (a.phase.value === "focusing" && a.progress.value === "stalled") || a.conversationHealth.value === "user_overloaded" },
{ key: "continue", fn: () => true }, // default — always eligible as fallback
];
/**
* Audit one assessment against every behaviour rule.
* Returns { behaviour, eligible, selected, blockedBy, reason } for each behaviour.
*/
function auditBehaviour(assessment) {
const results = [];
const actualResult = selectBehaviour(assessment);
for (const { key, fn } of BEHAVIOUR_CHECKERS) {
if (key === "continue") continue; // handled separately below
const eligible = fn(assessment);
let blockedBy = null;
let selected = false;
let reason = "";
if (eligible) {
// Check whether any earlier-priority behaviour fires first
const priorityOrder = ["acknowledge", "clarify", "summarise", "pause"];
const myIndex = priorityOrder.indexOf(key);
for (let i = 0; i < myIndex; i++) {
if (BEHAVIOUR_CHECKERS[i].fn(assessment)) {
blockedBy = BEHAVIOUR_CHECKERS[i].key;
break;
}
}
selected = (blockedBy === null);
reason = eligible ? (blockedBy ? `eligible but blocked by ${blockedBy}` : "eligible and selected") : "";
} else {
reason = `${key} rule conditions not met`;
}
results.push({ behaviour: key, eligible, selected, blockedBy, reason });
}
// Continue is always the fallback when nothing else fires
const continueEligible = !results.some(r => r.eligible && r.selected);
if (continueEligible) {
results.push({ behaviour: "continue", eligible: true, selected: true, blockedBy: null, reason: `No specific rule matched — defaulting to continue` });
} else {
const blockingBehaviour = results.find(r => r.eligible && r.selected);
results.push({
behaviour: "continue",
eligible: false,
selected: false,
blockedBy: blockingBehaviour ? blockingBehaviour.behaviour : null,
reason: blockingBehaviour ? `blocked by ${blockingBehaviour.behaviour}` : "specific rule matched"
});
}
return results;
}
/*
Synthetic minimal assessment builders
*/
function mkMinimal(opts = {}) {
return {
version: "v0.1",
assessedAt: new Date().toISOString(),
confidence: opts.overallConfidence || "medium",
phase: opts.phase ?? { value: "cannot_determine", confidence: "low", signals: [], evidence: {} },
progress: opts.progress ?? { value: "cannot_determine", confidence: "low", signals: [], evidence: {} },
conversationHealth: opts.conversationHealth ?? { value: "cannot_determine", confidence: "low", signals: [], evidence: {} }
};
}
/*
Test suite Experiment 40
*/
describe("Experiment 40 — Behaviour Reachability Diagnostic", () => {
/* ── Scenario A: Real-scenario reachability audit ─────── */
describe("Real-scenario reachability (all Experiment 39 turns)", () => {
let allAudits = {};
function runAllAudits() {
if (Object.keys(allAudits).length > 0) return allAudits;
const scenarios = getEvaluationScenarios();
for (const [scenarioName, turns] of Object.entries(scenarios)) {
allAudits[scenarioName] = [];
for (let i = 0; i < turns.length; i++) {
const input = buildAssessmentInput(turns[i]);
const assessment = assessInvestigationState(input);
const behaviourResult = selectBehaviour(assessment);
allAudits[scenarioName].push({
turn: i,
centralStatement: (turns[i].centralStatement || "").substring(0, 50),
phase: assessment.phase.value,
progress: assessment.progress.value,
health: assessment.conversationHealth.value,
phaseConfidence: assessment.phase.confidence,
selectedBehaviour: behaviourResult.behaviour,
behaviours: auditBehaviour(assessment)
});
}
}
return allAudits;
}
it("audits every turn across all scenarios", () => {
const audits = runAllAudits();
expect(audits["long-investigation"]).toHaveLength(3);
expect(audits["contradictory-evidence"]).toHaveLength(3);
expect(audits["short-early"]).toHaveLength(1);
});
it("summarises reachability for each behaviour across real turns", () => {
const audits = runAllAudits();
const reachability = {};
for (const beh of ["acknowledge", "clarify", "summarise", "pause", "continue"]) {
let eligibleCount = 0;
let selectedCount = 0;
let blockedCount = 0;
const blockingSources = new Set();
const neverEligibleScenarios = new Set();
for (const [scenarioName, turns] of Object.entries(audits)) {
let scenarioEverEligible = false;
for (const turnAudit of turns) {
const behAudit = turnAudit.behaviours.find(b => b.behaviour === beh);
if (behAudit.eligible) {
eligibleCount++;
scenarioEverEligible = true;
if (behAudit.selected) selectedCount++;
else { blockedCount++; blockingSources.add(behAudit.blockedBy); }
}
}
if (!scenarioEverEligible) neverEligibleScenarios.add(scenarioName);
}
reachability[beh] = {
eligible: eligibleCount,
selected: selectedCount,
blocked: blockedCount,
blockingBy: Array.from(blockingSources),
neverEligibleInScenarios: Array.from(neverEligibleScenarios)
};
}
console.log("\n=== Experiment 40 — Reachability Summary ===");
for (const [beh, data] of Object.entries(reachability)) {
console.log(`\n${beh.toUpperCase()}:`);
console.log(` eligible: ${data.eligible}`);
console.log(` selected: ${data.selected}`);
console.log(` blocked: ${data.blocked} (by: ${data.blockingBy.join(", ") || "none"})`);
if (data.neverEligibleInScenarios.length) {
console.log(` never eligible in: ${data.neverEligibleInScenarios.join(", ")}`);
}
}
// Store for downstream assertions
global._exp40_reachability = reachability;
global._exp40_audits = audits;
return reachability;
});
it("Acknowledge is eligible in 5 of 7 turns and selected in all 5", () => {
const reachability = global._exp40_reachability || {};
expect(reachability.acknowledge?.eligible).toBe(5);
expect(reachability.acknowledge?.selected).toBe(5);
});
it("Summarise is eligible in 2 of 7 turns and always blocked", () => {
const reachability = global._exp40_reachability || {};
// Terminal state has phase=concluding, so summarise IS eligible
expect(reachability.summarise?.eligible).toBe(2);
// But acknowledge (priority 1) fires first because health=healthy
expect(reachability.summarise?.blocked).toBe(2);
expect(reachability.summarise?.blockingBy).toContain("acknowledge");
});
it("Clarify is eligible in 0 of 7 turns — never triggered", () => {
const reachability = global._exp40_reachability || {};
expect(reachability.clarify?.eligible).toBe(0);
});
it("Pause is eligible in 1 of 7 turns but always blocked", () => {
const reachability = global._exp40_reachability || {};
// Contradictory turn 1: phase=focusing, progress=stalled → pause rule fires
expect(reachability.pause?.eligible).toBe(1);
// But acknowledge (priority 1) also fires (health=healthy), blocking pause
expect(reachability.pause?.blocked).toBe(1);
expect(reachability.pause?.blockingBy).toContain("acknowledge");
});
it("Continue is eligible in 2 of 7 turns and selected in all 2", () => {
const reachability = global._exp40_reachability || {};
expect(reachability.continue?.eligible).toBe(2);
expect(reachability.continue?.selected).toBe(2);
});
it("all scenarios and turns are audited with full detail", () => {
const audits = global._exp40_audits || {};
for (const [scenarioName, turns] of Object.entries(audits)) {
for (const t of turns) {
expect(t.phase).toBeDefined();
expect(t.progress).toBeDefined();
expect(t.health).toBeDefined();
expect(["acknowledge", "clarify", "summarise", "pause", "continue"].includes(t.selectedBehaviour)).toBe(true);
expect(t.behaviours).toHaveLength(5); // all five behaviours checked
}
}
});
it("prints turn-level detail for the terminal long-investigation state", () => {
const audits = global._exp40_audits || {};
const terminalTurn = audits["long-investigation"][2];
expect(terminalTurn.phase).toBe("concluding");
expect(terminalTurn.health).toBe("healthy");
console.log("\n=== Terminal Turn Detail ===");
console.log(`Scenario: long-investigation, turn ${terminalTurn.turn}`);
console.log(` phase=${terminalTurn.phase}, progress=${terminalTurn.progress}, health=${terminalTurn.health}`);
for (const b of terminalTurn.behaviours) {
const marker = b.selected ? "▶" : b.eligible ? "⚠ blocked" : "✗";
console.log(` ${marker} ${b.behaviour}: eligible=${b.eligible}${b.blockedBy ? ` blocked by ${b.blockedBy}` : ""}`);
}
// Summarise should be eligible (concluding phase) but blocked
const summariseAudit = terminalTurn.behaviours.find(b => b.behaviour === "summarise");
expect(summariseAudit.eligible).toBe(true);
expect(summariseAudit.blockedBy).toBe("acknowledge");
// The selected behaviour is Acknowledge
expect(terminalTurn.selectedBehaviour).toBe("acknowledge");
});
it("prints turn-level detail for pause-eligible turn (contradictory turn 1)", () => {
const audits = global._exp40_audits || {};
const pauseEligibleTurn = audits["contradictory-evidence"][1];
expect(pauseEligibleTurn.phase).toBe("focusing");
expect(pauseEligibleTurn.progress).toBe("stalled");
console.log("\n=== Pause-Eligible Turn Detail ===");
console.log(`Scenario: contradictory-evidence, turn ${pauseEligibleTurn.turn}`);
console.log(` phase=${pauseEligibleTurn.phase}, progress=${pauseEligibleTurn.progress}, health=${pauseEligibleTurn.health}`);
for (const b of pauseEligibleTurn.behaviours) {
const marker = b.selected ? "▶" : b.eligible ? "⚠ blocked" : "✗";
console.log(` ${marker} ${b.behaviour}: eligible=${b.eligible}${b.blockedBy ? ` blocked by ${b.blockedBy}` : ""}`);
}
const pauseAudit = pauseEligibleTurn.behaviours.find(b => b.behaviour === "pause");
expect(pauseAudit.eligible).toBe(true);
expect(pauseAudit.blockedBy).toBe("acknowledge");
});
it("Clarify never eligible — no scenario produces too_broad health", () => {
const audits = global._exp40_audits || {};
for (const [scenarioName, turns] of Object.entries(audits)) {
for (const t of turns) {
expect(t.health).not.toBe("too_broad");
}
}
});
it("Clarify never eligible — no scenario produces orienting phase with < 3 obs", () => {
const audits = global._exp40_audits || {};
for (const [scenarioName, turns] of Object.entries(audits)) {
for (const t of turns) {
expect(t.phase).not.toBe("orienting");
}
}
});
it("Summarise eligible ONLY in the terminal long-investigation turn", () => {
const audits = global._exp40_audits || {};
let summariseTurns = [];
for (const [scenarioName, turns] of Object.entries(audits)) {
for (let i = 0; i < turns.length; i++) {
const sBeh = turns[i].behaviours.find(b => b.behaviour === "summarise");
if (sBeh.eligible) summariseTurns.push(`${scenarioName}-t${i}`);
}
}
expect(summariseTurns).toEqual(["long-investigation-t1", "long-investigation-t2"]);
});
it("Pause eligible ONLY in contradictory-evidence turn 1", () => {
const audits = global._exp40_audits || {};
let pauseTurns = [];
for (const [scenarioName, turns] of Object.entries(audits)) {
for (let i = 0; i < turns.length; i++) {
const pBeh = turns[i].behaviours.find(b => b.behaviour === "pause");
if (pBeh.eligible) pauseTurns.push(`${scenarioName}-t${i}`);
}
}
expect(pauseTurns).toEqual(["contradictory-evidence-t1"]);
});
it("terminal long-investigation turn IS included in the audit (confirms Experiment 39 coverage)", () => {
const audits = global._exp40_audits || {};
// Exp 39 tested turns: 0, 3, and 4 (complete). Turn 4 IS the terminal.
// Our getEvaluationScenarios maps to indices 0, 1, 2 — index 2 is the terminal state.
const terminalTurn = audits["long-investigation"][2];
expect(terminalTurn.selectedQuestion).toBeFalsy();
expect(terminalTurn.activeUnknownNodeId).toBeFalsy();
// Phase must be concluding for a terminal assessment
expect(terminalTurn.phase).toBe("concluding");
expect(terminalTurn.centralStatement).toContain("European market");
});
it("Acknowledge dominance caused by combination of broad eligibility, priority order, and scenario distribution", () => {
const reachability = global._exp40_reachability || {};
// Acknowledge eligible 5/7 — that's the broad eligibility factor
expect(reachability.acknowledge.eligible).toBe(5);
// It is selected every time it is eligible — that's the priority order factor
expect(reachability.acknowledge.blocked).toBe(0);
// 5/7 = 71% — scenario distribution (3 scenarios, most turns have healthy health)
const totalTurns = global._exp40_audits
? Object.values(global._exp40_audits).reduce((sum, arr) => sum + arr.length, 0)
: 7;
expect(reachability.acknowledge.eligible / totalTurns).toBeGreaterThan(0.5);
});
});
/* ── Scenario B: Synthetic isolated reachability ─────── */
describe("Synthetic reachability — each behaviour in isolation", () => {
it("Acknowledge is reachable when health=healthy and phase confidence ≠ low", () => {
const result = selectBehaviour(mkMinimal({
phase: { value: "exploring", confidence: "medium" },
conversationHealth: { value: "healthy", confidence: "high" }
}));
expect(result.behaviour).toBe("acknowledge");
});
it("Clarify is reachable when health=too_broad (no other rule fires first)", () => {
const result = selectBehaviour(mkMinimal({
phase: { value: "exploring", confidence: "low" }, // low → no acknowledge
conversationHealth: { value: "too_broad", confidence: "high" }
}));
expect(result.behaviour).toBe("clarify");
});
it("Clarify is reachable when phase=orienting and observationDensity < 3", () => {
const result = selectBehaviour(mkMinimal({
phase: { value: "orienting", confidence: "low", evidence: { observationDensity: 1 } },
conversationHealth: { value: "cannot_determine", confidence: "low" }
}));
expect(result.behaviour).toBe("clarify");
});
it("Summarise is reachable when phase=synthesising (no health trigger to block)", () => {
const result = selectBehaviour(mkMinimal({
phase: { value: "synthesising", confidence: "high" },
conversationHealth: { value: "cannot_determine", confidence: "low" } // no healthy → no acknowledge
}));
expect(result.behaviour).toBe("summarise");
});
it("Summarise is reachable when phase=concluding (no health trigger to block)", () => {
const result = selectBehaviour(mkMinimal({
phase: { value: "concluding", confidence: "high" },
conversationHealth: { value: "cannot_determine", confidence: "low" } // no healthy → no acknowledge
}));
expect(result.behaviour).toBe("summarise");
});
it("Summarise is reachable when resolvedNodeCount >= 3 and progress=steady (no health trigger)", () => {
const result = selectBehaviour(mkMinimal({
phase: { value: "focusing", confidence: "high", evidence: { resolvedNodeCount: 4 } },
progress: { value: "steady", confidence: "medium" },
conversationHealth: { value: "too_narrow", confidence: "low" } // no healthy → no acknowledge
}));
expect(result.behaviour).toBe("summarise");
});
it("Pause is reachable when phase=focusing + progress=stalled (no health trigger)", () => {
const result = selectBehaviour(mkMinimal({
phase: { value: "focusing", confidence: "high" },
progress: { value: "stalled", confidence: "high" },
conversationHealth: { value: "cannot_determine", confidence: "low" } // no healthy → no acknowledge
}));
expect(result.behaviour).toBe("pause");
});
it("Pause is reachable when health=user_overloaded", () => {
const result = selectBehaviour(mkMinimal({
phase: { value: "exploring", confidence: "medium" },
conversationHealth: { value: "user_overloaded", confidence: "medium" }
}));
expect(result.behaviour).toBe("pause");
});
it("Continue is reachable when no rule conditions match", () => {
const result = selectBehaviour(mkMinimal({
phase: { value: "exploring", confidence: "low" },
progress: { value: "cannot_determine", confidence: "low" },
conversationHealth: { value: "cannot_determine", confidence: "low" }
}));
expect(result.behaviour).toBe("continue");
});
it("each synthetic case confirms the rule is independently reachable", () => {
const tests = [
{ beh: "acknowledge", a: mkMinimal({ phase: { value: "focusing", confidence: "high" }, conversationHealth: { value: "healthy", confidence: "high" } }) },
{ beh: "clarify", a: mkMinimal({ phase: { value: "exploring", confidence: "low" }, conversationHealth: { value: "too_broad", confidence: "high" } }) },
{ beh: "summarise", a: mkMinimal({ phase: { value: "concluding", confidence: "high" }, conversationHealth: { value: "cannot_determine", confidence: "low" } }) },
{ beh: "pause", a: mkMinimal({ phase: { value: "focusing", confidence: "high" }, progress: { value: "stalled", confidence: "high" }, conversationHealth: { value: "cannot_determine", confidence: "low" } }) },
{ beh: "continue", a: mkMinimal({ phase: { value: "deepening", confidence: "medium" }, progress: { value: "steady", confidence: "medium" }, conversationHealth: { value: "cannot_determine", confidence: "low" } }) },
];
const results = {};
for (const t of tests) {
results[t.beh] = selectBehaviour(t.a).behaviour;
}
expect(results.acknowledge).toBe("acknowledge");
expect(results.clarify).toBe("clarify");
expect(results.summarise).toBe("summarise");
expect(results.pause).toBe("pause");
expect(results.continue).toBe("continue");
console.log("\n=== Synthetic Reachability ===");
for (const [beh, result] of Object.entries(results)) {
console.log(` ${beh}: ${result === beh ? "✓ reachable" : "✗ NOT reachable"}`);
}
});
});
/* ── Diagnostic correctness tests ────────────────────── */
describe("Diagnostic accuracy", () => {
it("selected behaviour matches the real selector output", () => {
const audits = global._exp40_audits || {};
let allMatch = true;
for (const [scenarioName, turns] of Object.entries(audits)) {
for (const t of turns) {
const input = buildAssessmentInput(getEvaluationScenarios()[scenarioName][t.turn]);
const assessment = assessInvestigationState(input);
const selectorResult = selectBehaviour(assessment);
if (selectorResult.behaviour !== t.selectedBehaviour) allMatch = false;
}
}
expect(allMatch).toBe(true);
});
it("identical inputs produce identical diagnostics every time", () => {
const scenarios = getEvaluationScenarios();
for (const [scenarioName, turns] of Object.entries(scenarios)) {
for (let i = 0; i < turns.length; i++) {
const input = buildAssessmentInput(turns[i]);
const assessment = assessInvestigationState(input);
const diagnostic1 = auditBehaviour(assessment);
const diagnostic2 = auditBehaviour(assessment);
expect(JSON.stringify(diagnostic1)).toBe(JSON.stringify(diagnostic2));
}
}
});
it("inputs are not mutated by audit function", () => {
const scenarios = getEvaluationScenarios();
for (const [scenarioName, turns] of Object.entries(scenarios)) {
for (let i = 0; i < turns.length; i++) {
const input = buildAssessmentInput(turns[i]);
const snapshot = JSON.stringify(input);
const assessment = assessInvestigationState(JSON.parse(snapshot));
auditBehaviour(assessment);
expect(JSON.stringify(input)).toBe(snapshot);
}
}
});
it("eligible behaviours are identified correctly for each turn", () => {
const audits = global._exp40_audits || {};
let correctCount = 0;
let totalCount = 0;
for (const [scenarioName, turns] of Object.entries(audits)) {
for (const t of turns) {
const input = buildAssessmentInput(getEvaluationScenarios()[scenarioName][t.turn]);
const assessment = assessInvestigationState(input);
for (const behaviour of ["acknowledge", "clarify", "summarise", "pause"]) {
totalCount++;
const expectedEligible = auditBehaviour(assessment).find(b => b.behaviour === behaviour).eligible;
if (expectedEligible === t.behaviours.find(b => b.behaviour === behaviour).eligible) {
correctCount++;
}
}
}
}
expect(correctCount).toBe(totalCount);
});
it("blocked behaviours name the correct earlier winning rule", () => {
const audits = global._exp40_audits || {};
for (const [scenarioName, turns] of Object.entries(audits)) {
for (const t of turns) {
const blockedBehaviours = t.behaviours.filter(b => b.eligible && !b.selected);
for (const b of blockedBehaviours) {
// The blocking behaviour must have fired BEFORE this one in priority order
const priorityOrder = ["acknowledge", "clarify", "summarise", "pause"];
const blockerIdx = priorityOrder.indexOf(b.blockedBy);
const behIdx = priorityOrder.indexOf(b.behaviour);
expect(blockerIdx).toBeLessThan(behIdx);
}
}
}
});
it("all Experiment 39 turns are audited (7 total)", () => {
const audits = global._exp40_audits || {};
let totalTurns = 0;
for (const turns of Object.values(audits)) {
totalTurns += turns.length;
}
expect(totalTurns).toBe(7);
});
});
/* ── Classification summary ──────────────────────────── */
describe("Reachability classification", () => {
it("classifies each missing behaviour for the final report", () => {
const reachability = global._exp40_reachability || {};
// Summarise: eligible in real turn but blocked → eligible_but_blocked
expect(reachability.summarise.eligible).toBeGreaterThan(0);
expect(reachability.summarise.blocked).toBeGreaterThan(0);
// Clarify: never eligible in tested scenarios, reachable only synthetically
expect(reachability.clarify.eligible).toBe(0);
// Pause: eligible in real turn but blocked
expect(reachability.pause.eligible).toBeGreaterThan(0);
expect(reachability.pause.blocked).toBeGreaterThan(0);
console.log("\n=== Experiment 40 — Final Classifications ===");
console.log(`Summarise: eligible_but_blocked (blocked by acknowledge)`);
console.log(`Clarify: never_eligible_in_tested_scenarios (reachable only in synthetic case)`);
console.log(`Pause: eligible_but_blocked (blocked by acknowledge)`);
console.log(`Acknowledge: dominant because health=healthy is the most common state`);
global._exp40_classifications = {
summarise: "eligible_but_blocked",
clarify: "never_eligible_in_tested_scenarios",
pause: "eligible_but_blocked"
};
});
});
});