From 1273861f0c90d2ded875d2bdd43e9a7ea3034236 Mon Sep 17 00:00:00 2001 From: robbond Date: Wed, 5 Aug 2026 17:52:18 +0100 Subject: [PATCH] exp(18): implement investigation state assessment layer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implement the three-dimensional assessment (phase, progress, conversation health) that sits between narrative and behaviour selection. Key changes: - lib/assessment/investigation-state-assessor.js: assessor module with countObservations, assessPhase, assessProgress, assessConversationHealth, assessInvestigationState — deterministic classifiers using known rules - tests/investigation-state-assessor.test.js: 51 tests covering phase classification (orienting→concluding), progress thresholds, health conditions, confidence aggregation, edge cases, and observation counting - lib/graph/orchestrator.js: integration calls passing correctly-shaped input to assessInvestigationState() at three call sites (~552, ~904, ~1013) Design decisions encoded in this iteration: - countObservations counts nodes with known/resolved status + high-confidence non-unknown non-state nodes (not just explicit observation-kind nodes) - Phase uses seven values including cannot_determine for insufficient data - Progress uses resolution ratio thresholds: accelerating (>0.6), steady (0.2-0.6), stalled (<0.2 with ≥1 resolved) - Overall confidence = minimum across all three dimensions (conservative) Also adds investigation-state-assessment-contract.md and updates design-evolution-log, investigation-state-assessment.md (status header), and investigation-turn-cycle.md (implementation status table). --- docs/design-evolution-log.md | 52 +- ...investigation-state-assessment-contract.md | 232 ++++++ docs/investigation-state-assessment.md | 3 +- docs/investigation-turn-cycle.md | 17 + .../investigation-state-assessor.js | 571 +++++++++++++++ lib/graph/orchestrator.js | 47 +- tests/investigation-state-assessor.test.js | 693 ++++++++++++++++++ 7 files changed, 1608 insertions(+), 7 deletions(-) create mode 100644 docs/investigation-state-assessment-contract.md create mode 100644 lib/assessment/investigation-state-assessor.js create mode 100644 tests/investigation-state-assessor.test.js diff --git a/docs/design-evolution-log.md b/docs/design-evolution-log.md index 77ad1ac..64583e1 100644 --- a/docs/design-evolution-log.md +++ b/docs/design-evolution-log.md @@ -892,13 +892,57 @@ This architecture emerged from observation, not top-down design. It may still ch A complete investigation can be described as a repeating turn cycle in which every architectural layer has a single responsibility. -Status +Result -Architectural. +Experiment validated that the investigation turn cycle is an *observation* about how existing layers interact rather than a new architectural layer. All eight stages (User Observation → Reasoning Graph → Investigation Narrative → State Assessment → Behaviour Selection → Conversation → Workspace → Wait) are supported by current architecture components, but only Stages 1–3 and 7 have working implementations. Stage 4 (State Assessment) and Stage 5 (Behaviour Selection) remain as architectural specifications without executable code. -Evaluation +What did we learn? -Pending. +- The turn cycle confirms that assessment sits between narrative and behaviour selection, not after the graph directly. +- Every layer has one responsibility: each stage's purpose maps to an existing or specified component without overlap. +- The cycle is deterministic in structure but adaptive in content — this is correct because the *sequence* of operations must be fixed while the *outputs* vary with investigation state. +- Without a working Stage 4, all downstream stages (behaviour selection, conversation, workspace projection) operate on incomplete input. Phase 5 needs an executable assessment before behaviour can be validated experimentally. + +Decision + +The turn cycle architecture is confirmed as correct but requires implementation of Stage 4 (State Assessment) to move from observation to validation. The next step is the first deterministic evaluation function — not behaviour selection, which depends on assessment output. This becomes Experiment 18: First Executable Slice. + +--- + +### Experiment 18 — First Executable Slice (Investigation State Assessment) + +#### Hypothesis + +A deterministic, conservative assessment of investigation phase and progress can be built from existing graph data without introducing new signals or modifying reasoning logic. The assessment should prefer `cannot_determine` over invented precision. + +#### Scope + +Phase detection (orienting / exploring / focusing / deepening / synthesising / concluding / cannot_determine), progress tracking (accelerating / steady / stalled / looping / spiralling / cannot_determine), and conversation health evaluation — using only data already present in the graph schema, orchestrator diagnostics, and facilitator-view outputs. + +#### Constrained By + +- Must use actual repo contracts (not assumptions about field names or structures). +- Must be pure function — no network, LLM, mutation, or side effects. +- Must handle missing fields gracefully — safe with absent data. +- Must produce versioned assessment objects for future compatibility. +- Passive integration only: add to diagnostics without changing public API or user-visible behaviour. + +#### Questions + +1. Can phase be reliably classified from node composition (kind/status ratio) alone? +2. Does progress detection require turn history, or is a single-snapshot approximation sufficient for this first slice? +3. What minimal conversation health signals can be extracted from existing graph metadata? + +#### Evaluation + +- Deterministic output across identical inputs. +- Correct `cannot_determine` when data is insufficient (no false precision). +- Handles all 11 mock scenarios at their turn points plus at least one live Ollama-shaped state. +- Unsupported signals explicitly recorded in reasoning-contract-backlog.md. + +#### Status + +Completed — see `investigation-state-assessment-contract.md` and `lib/assessment/investigation-state-assessor.js`. --- diff --git a/docs/investigation-state-assessment-contract.md b/docs/investigation-state-assessment-contract.md new file mode 100644 index 0000000..0bcfccf --- /dev/null +++ b/docs/investigation-state-assessment-contract.md @@ -0,0 +1,232 @@ +# Investigation State Assessment Contract + +> Architecture Experiment 18 — First Executable Slice +> +> This document defines the data contract for the investigation state assessment layer. It is the interface between investigation narrative (Stage 3) and behaviour selection (Stage 5). + +--- + +## Versioning + +```jsonc +{ + "version": "v0.1", // Schema version — increment only on field additions/removals + "assessedAt": "ISO-8601", // When this assessment was computed + "confidence": "high|medium|low" // Overall assessment confidence +} +``` + +All fields below are part of the v0.1 contract. Adding new fields requires a version bump to v0.2+. Removing or renaming existing fields also requires a version bump. + +--- + +## Assessment Object Shape + +### Phase + +Evaluates where the investigation is in its lifecycle. + +```json +{ + "phase": { + "value": "orienting|exploring|focusing|deepening|synthesising|concluding|cannot_determine", + "confidence": "high|medium|low", + "signals": [string], // Descriptive reasons for the classification + "evidence": { // What data supported this classification + "resolvedNodeCount": number, + "activeUnknownCount": number, + "unknownResolutionRatio": number | null, + "observationDensity": number | null, + "evidenceDepth": string | null // "shallow|moderate|deep" + } + } +} +``` + +### Progress + +Tracks whether understanding is advancing, stalled, or regressing. + +```json +{ + "progress": { + "value": "accelerating|steady|stalled|looping|spiralling|cannot_determine", + "confidence": "high|medium|low", + "signals": [string], + "evidence": { + "turnCount": number, + "recentResolutionsLastTurn": number, + "newUnknownsPerTurn": number | null, + "repeatedNodeIds": string[] // Nodes appearing across 3+ turns without resolution + } + } +} +``` + +### Conversation Health + +Evaluates whether the conversation pattern is productive. + +```json +{ + "conversationHealth": { + "value": "healthy|repetitive|too_broad|too_narrow|user_overloaded|user_under_informed|cannot_determine", + "confidence": "high|medium|low", + "signals": [string], + "evidence": { + "questionTypeDistribution": Record | null, // reasoningPattern -> count + "activeUnknownCount": number, + "resolvedNodeRatio": number | null, + "hasActiveQuestion": boolean | null, + "summaryLength": number | null + } + } +} +``` + +### Overall Assessment + +All dimensions combined: + +```json +{ + "version": "v0.1", + "assessedAt": "ISO-8601", + "confidence": "high|medium|low", + "phase": { ... }, + "progress": { ... }, + "conversationHealth": { ... } +} +``` + +--- + +## Deterministic Rules (Phase) + +The phase classifier uses conservative thresholds and defaults to `cannot_determine` when data is insufficient. + +### Values + +| Phase | Classification Rule | +|-------|---------------------| +| `concluding` | `activeUnknownCount === 0` AND `resolvedNodeCount >= 2` AND `hasActiveQuestion === false` — investigation has reached terminal state | +| `synthesising` | `activeUnknownCount <= 1` AND `resolvedNodeRatio > 0.5` — near completion, synthesis possible | +| `focusing` | `activeUnknownCount === 1` AND `knownObservations >= 3` — single remaining unknown with sufficient context | +| `exploring` | `observationCount >= 2` AND `unknownResolutionRatio < 0.4` — gathering initial evidence | +| `orienting` | `observationCount < 2` OR `totalNodeCount < 3` — insufficient structure to classify | +| `deepening` | Residual: `activeUnknownCount > 1` AND `resolvedNodeCount >= 3` — structured investigation with remaining unknowns | +| `cannot_determine` | Insufficient data for any of the above classifications | + +### Thresholds (Conservative) + +- A node is "known" only if `kind === "observation"` OR (`kind !== "unknown"` AND `status !== "provisional"`). +- An unknown is "active" only if `kind === "unknown"` AND status is not `"resolved"`. +- Resolution ratio = resolvedNodeCount / totalNonEmptyNodeCount (null if total is 0). +- Known observation density threshold: ≥3 observations before classifying as `focusing` or above. + +--- + +## Deterministic Rules (Progress) + +Progress uses a single-snapshot approximation for this first slice. Multi-turn history tracking is reserved for a future version. + +### Values + +| Progress | Classification Rule | +|----------|---------------------| +| `accelerating` | `unknownResolutionRatio > 0.6` — resolving unknowns faster than they accumulate | +| `steady` | `unknownResolutionRatio > 0.2` AND `unknownResolutionRatio <= 0.6` — balanced progress | +| `stalled` | `unknownResolutionRatio < 0.2` AND `resolvedNodeCount >= 1` — some work done but no momentum | +| `looping` | `repeatedNodeIds.length > 0` (nodes appearing across turns) AND `resolvedNodeCount === 0` for those nodes | +| `spiralling` | `newUnknownsPerTurn > resolvedNodeCount` for recent history (requires turn history) | +| `cannot_determine` | `resolvedNodeCount === 0` AND `totalNodeCount <= 2` — no sufficient data yet | + +### Thresholds (Conservative) + +- Resolution ratio = resolvedNodeCount / totalNonEmptyNodeCount. +- No progress classification is assigned below 1 resolved node. +- If resolution ratio cannot be computed (total === 0 or all nodes unresolved), return `cannot_determine`. + +--- + +## Deterministic Rules (Conversation Health) + +Conversation health evaluates the interaction pattern from available graph signals. + +### Values + +| Health | Classification Rule | +|--------|---------------------| +| `healthy` | Has active unknown AND has active question AND resolution ratio between 0 and 0.9 | +| `repetitive` | Same reasoning pattern appears in >50% of questions (requires historical data) | +| `too_broad` | Multiple active unknowns (>3) without sufficient resolved context (<2 resolved) | +| `too_narrow` | Single observation with single unknown and no edges connecting them | +| `user_overloaded` | Active unknown count > 4 (exceeds working memory limit) | +| `user_under_informed` | Total observations < 2 AND active question exists (asking without enough context) | +| `cannot_determine` | Insufficient graph signals to evaluate conversation pattern | + +### Thresholds (Conservative) + +- "Has active unknown" = `activeUnknownCount > 0`. +- "Has active question" = `selectedQuestion !== null`. +- Resolution ratio for healthy = between 0 and 0.9 (not terminal, not empty). + +--- + +## Confidence Rules + +### Overall Confidence + +The overall confidence is the *minimum* of all dimension confidences. If any dimension cannot be determined, overall confidence is `"low"`. + +### Per-Dimension Confidence + +| Confidence | Condition | +|-----------|-----------| +| `high` | Classification supported by ≥2 independent signals AND data is complete (no null evidence fields) | +| `medium` | Classification supported by 1 signal AND data is mostly complete (≤1 null evidence field) | +| `low` | Classification supported by partial data OR has null evidence fields >1 | + +--- + +## What This Contract Must Never Do + +- **Never invent signals.** Only use data actually present in the graph schema, orchestrator diagnostics, or facilitator-view outputs. +- **Never reach false precision.** When data is ambiguous, return `cannot_determine` rather than a confident but unsupported classification. +- **Never mutate input state.** This is a pure function — no side effects, no writes, no network calls. +- **Never bypass the narrative layer.** Assessment reads from narrative output and graph data, never directly from raw user input. +- **Never prescribe behaviour.** Assessment describes state only; interpretation belongs to behaviour selection. + +--- + +## Supported Data Sources (v0.1) + +| Source | Fields Available | +|--------|-----------------| +| `situationGraph.nodes[]` | `id`, `label`, `description`, `kind`, `status`, `confidence`, `evidenceIds`, `dependsOn`, `affects` | +| `situationGraph.resolvedNodeIds[]` | Array of resolved node IDs | +| `situationGraph.activeUnknownNodeId` | Nullable string — currently targeted unknown | +| `selectedQuestion` | `{ nodeId, question, reason }` nullable | +| `diagnostics.reasoningPattern` | Current turn's reasoning pattern key | +| `diagnostics.nodeCount` | Total node count | +| `diagnostics.edgeCount` | Total edge count | +| `facilitatorView._meta` | `isTerminal`, `hasUnresolvedUnknowns`, `totalObservations`, `unresolvedUnknownsCount` | + +--- + +## Unsupported Signals (Planned for Future Versions) + +These signals are specified in the investigation-state-assessment architecture document but require data not yet available in v0.1: + +- **Evidence Quality** — requires per-node evidence confidence scoring across multiple sources +- **Understanding Trajectory** — requires comparing narrative complexity across turns +- **Uncertainty Trend** — requires tracking which unknowns are resolved and by what pattern +- **Behaviour Readiness** — requires mapping assessment output to behaviour inventory + +These are tracked in `reasoning-contract-backlog.md`. + +--- + +## Recording Note + +This contract was drafted for Experiment 18's first executable slice. It captures the minimal viable assessment object shape derived from actual repository data contracts (schema.js, orchestrator.js, facilitator-view-adapter.js). Future experiments will add dimensions and signals as the graph schema and diagnostics evolve. diff --git a/docs/investigation-state-assessment.md b/docs/investigation-state-assessment.md index 0c8ca59..09be730 100644 --- a/docs/investigation-state-assessment.md +++ b/docs/investigation-state-assessment.md @@ -1,6 +1,7 @@ # Investigation State Assessment — Architectural Specification -> This is a design document only. Do not implement yet. +> **Status: Implemented (Experiment 18). First executable slice deployed.** +> Design evolved through experiments; implementation validates and adjusts the spec iteratively. --- diff --git a/docs/investigation-turn-cycle.md b/docs/investigation-turn-cycle.md index b595131..c3135b5 100644 --- a/docs/investigation-turn-cycle.md +++ b/docs/investigation-turn-cycle.md @@ -257,6 +257,23 @@ Nothing system-generated. The next observation originates entirely from the user --- +## Implementation Status + +| Stage | Description | Status | Experiment | +|-------|-------------|--------|------------| +| 1 | User Observation | Implemented (input) | — | +| 2 | Reasoning Graph Update | Implemented | Various | +| 3 | Investigation Narrative Update | Partially implemented | — | +| 4 | State Assessment | **Implemented** (v0.1) | Exp 18 | +| 5 | Behaviour Selection | Design only | **Exp 19 next** | +| 6 | Conversation Response | Design only | Post-Exp 19 | +| 7 | Workspace Projection | Implemented (UI) | Various | +| 8 | Wait for Next Observation | Implemented (state machine) | — | + +Stages 4 and 5 remain as architectural specifications without executable code. Stage 4 was completed in Experiment 18; Stage 5 is the next implementation target. + +--- + ## What This Turn Cycle Proves The investigation turn cycle is not a new layer. It is an observation about how existing layers interact during a real investigation. It confirms that: diff --git a/lib/assessment/investigation-state-assessor.js b/lib/assessment/investigation-state-assessor.js new file mode 100644 index 0000000..969efe8 --- /dev/null +++ b/lib/assessment/investigation-state-assessor.js @@ -0,0 +1,571 @@ +/** + * Investigation State Assessment — Experiment 18 First Executable Slice + * + * Pure deterministic function that evaluates investigation state across + * three dimensions: phase, progress, and conversation health. + * + * Conservative by design: prefers cannot_determine over invented precision. + * Safe with missing fields — returns cannot_determine for any dimension + * whose data is insufficient rather than guessing. + * + * Contract reference: docs/investigation-state-assessment-contract.md + */ + +/* ── Helpers ─────────────────────────────────────────────── */ + +/** + * Normalise resolvedNodeIds from the fixture format ({ resolved: [...] }) + * or from orchestrator format (resolvedNodeIds directly). + */ +function getResolvedIds(input) { + const fromGraph = input.situationGraph?.resolvedNodeIds; + if (Array.isArray(fromGraph)) return fromGraph; + + // Legacy scenario fixture shape + const fromResolved = input.resolved; + if (Array.isArray(fromResolved)) return fromResolved; + + return []; +} + +/** + * Normalise the activeUnknownNodeId across formats. + */ +function getActiveUnknownId(input) { + const fromGraph = input.situationGraph?.activeUnknownNodeId; + if (fromGraph !== undefined && fromGraph !== null) return fromGraph; + + const fromScenario = input.active; + if (fromScenario !== undefined && fromScenario !== null) return fromScenario; + + return null; +} + +/** + * Count resolved nodes — either via the explicit array or by checking + * per-node status === "resolved". + */ +function countResolved(input, nodes) { + const resolvedIds = getResolvedIds(input); + if (resolvedIds.length > 0) { + return nodes.filter(n => n && resolvedIds.includes(n.id)).length; + } + // Fallback: count nodes with status === "resolved" + return nodes.filter(n => n && n.status === "resolved").length; +} + +/** + * Classify node confidence as a normalised score for comparison. + */ +function confidenceScore(confidence) { + if (!confidence) return 0; + const map = { low: 1, medium: 2, high: 3 }; + return map[confidence] ?? 0; +} + +/** + * Normalise confidence label from score. + */ +function scoreToConfidence(score) { + if (score >= 7) return "high"; + if (score >= 3) return "medium"; + return "low"; +} + +/** + * Count observations: explicit observation kind with known/resolved status, + * or high-confidence evidence nodes. Deliberately excludes scaffolding state + * nodes and already-resolved unknowns (they have their own assessment). + */ +function countObservations(nodes, resolvedIds) { + if (!Array.isArray(nodes)) return 0; + + const resolvedSet = new Set(resolvedIds); + + let count = 0; + for (const node of nodes) { + if (!node || typeof node.kind !== "string") continue; + + // Skip already-resolved unknowns — their resolution is tracked separately + if (resolvedSet.has(node.id)) continue; + + // Include explicit observation kind with known/resolved status + if (node.kind === "observation" && (node.status === "known" || node.status === "resolved")) { + count++; + continue; + } + + // Include non-unknown nodes with high confidence that aren't scaffolding states + if (confidenceScore(node.confidence) >= 3 && node.kind !== "state") { + count++; + continue; + } + } + return count; +} + +/** + * Count total active (non-resolved) unknowns. + */ +function countActiveUnknowns(input, nodes, resolvedIds) { + const activeId = getActiveUnknownId(input); + + if (!Array.isArray(nodes)) return activeId ? 1 : 0; + + // Nodes explicitly marked as "unknown" kind that are not resolved + let count = 0; + for (const node of nodes) { + if (!node || node.kind !== "unknown") continue; + const isResolved = resolvedIds.includes(node.id) || node.status === "resolved"; + if (!isResolved) count++; + } + + // Fallback: if no unknown-kinded nodes and we have an active ID, + // the active node itself counts as an active unknown + if (count === 0 && activeId) { + const isActiveNode = nodes.find(n => n && n.id === activeId); + if (!isActiveNode || isActiveNode.status !== "resolved") count = 1; + } + + return count; +} + +/** + * Compute resolution ratio: resolved / total non-empty nodes. + */ +function computeResolutionRatio(resolvedCount, totalNodes) { + if (totalNodes <= 0 || resolvedCount === 0) return null; + return resolvedCount / totalNodes; +} + +/** + * Count distinct reasoning patterns from selected question or diagnostics. + */ +function getReasoningPatterns(input) { + const patterns = []; + + // From selectedQuestion.reason (may contain reasoning pattern keyword) + if (input.selectedQuestion?.reasoningPattern) { + patterns.push(input.selectedQuestion.reasoningPattern); + } + + // From diagnostics + const diag = input.diagnostics || {}; + if (diag.reasoningPattern && !patterns.includes(diag.reasoningPattern)) { + patterns.push(diag.reasoningPattern); + } + if (diag.investigationStrategy?.key && !patterns.includes(diag.investigationStrategy.key)) { + patterns.push(diag.investigationStrategy.key); + } + + return patterns; +} + +/** + * Count edges connected to each node for structural analysis. + */ +function countEdgeConnections(nodes, edges) { + if (!Array.isArray(edges)) return {}; + + const counts = {}; + for (const edge of edges) { + if (!edge || !edge.fromNodeId || !edge.toNodeId) continue; + counts[edge.fromNodeId] = (counts[edge.fromNodeId] ?? 0) + 1; + counts[edge.toNodeId] = (counts[edge.toNodeId] ?? 0) + 1; + } + return counts; +} + +/** + * Determine the minimum confidence across all dimensions. + */ +function minConfidence(...confidences) { + const priority = { high: 3, medium: 2, low: 1, cannot_determine: 0 }; + let minScore = 4; + let result = "high"; + + for (const c of confidences) { + const s = priority[c] ?? 4; + if (s < minScore) { + minScore = s; + result = c; + } + } + return result; +} + +/* ── Phase Classification ────────────────────────────────── */ + +function assessPhase(input) { + const resolvedIds = getResolvedIds(input); + const nodes = input.situationGraph?.nodes || []; + const totalNodes = Array.isArray(nodes) ? nodes.length : 0; + const resolvedCount = countResolved(input, nodes); + const activeUnknownCount = countActiveUnknowns(input, nodes, resolvedIds); + const observations = countObservations(nodes, resolvedIds); + const ratio = computeResolutionRatio(resolvedCount, totalNodes); + const hasQuestion = Boolean(input.selectedQuestion && input.selectedQuestion.nodeId); + const activeUnknownId = getActiveUnknownId(input); + + // Terminal: no active unknowns + sufficient history + no current question + if (activeUnknownCount === 0 && resolvedCount >= 2 && !hasQuestion) { + return { + value: "concluding", + confidence: scoreToConfidence(observations * 2 + resolvedCount), + signals: [ + `All investigation areas resolved (${resolvedCount} items)`, + `No active question — investigation complete` + ], + evidence: { + resolvedNodeCount: resolvedCount, + activeUnknownCount: 0, + unknownResolutionRatio: ratio, + observationDensity: observations, + evidenceDepth: observations >= 4 ? "deep" : observations >= 2 ? "moderate" : "shallow" + } + }; + } + + // Synthesising: near-completion with majority resolved + if (activeUnknownCount <= 1 && ratio !== null && ratio > 0.5) { + return { + value: "synthesising", + confidence: scoreToConfidence(observations * 2 + resolvedCount), + signals: [ + `Near completion: ${resolvedCount} of ${totalNodes} resolved`, + `Resolution ratio: ${(ratio * 100).toFixed(0)}%` + ], + evidence: { + resolvedNodeCount: resolvedCount, + activeUnknownCount, + unknownResolutionRatio: ratio, + observationDensity: observations, + evidenceDepth: observations >= 4 ? "deep" : observations >= 2 ? "moderate" : "shallow" + } + }; + } + + // Focusing: single remaining unknown with sufficient context + if (activeUnknownCount === 1 && observations >= 3) { + return { + value: "focusing", + confidence: scoreToConfidence(observations * 2 + resolvedCount), + signals: [ + `Single active unknown: ${activeUnknownId ?? "unspecified"}`, + `${observations} established observations provide sufficient context` + ], + evidence: { + resolvedNodeCount: resolvedCount, + activeUnknownCount, + unknownResolutionRatio: ratio, + observationDensity: observations, + evidenceDepth: observations >= 4 ? "deep" : "moderate" + } + }; + } + + // Exploring: gathering initial evidence — multiple observations but low resolution + if (observations >= 2 && (ratio === null || ratio < 0.4)) { + return { + value: "exploring", + confidence: scoreToConfidence(observations + resolvedCount), + signals: [ + `${observations} initial observations gathered`, + `Resolution progress low (${resolvedCount}/${totalNodes} or unknown)` + ], + evidence: { + resolvedNodeCount: resolvedCount, + activeUnknownCount, + unknownResolutionRatio: ratio, + observationDensity: observations, + evidenceDepth: "shallow" + } + }; + } + + // Deepening: structured investigation with remaining unknowns + if (activeUnknownCount > 1 && resolvedCount >= 3) { + return { + value: "deepening", + confidence: scoreToConfidence(resolvedCount + observations), + signals: [ + `Structured investigation in progress`, + `${resolvedCount} resolved, ${activeUnknownCount} active unknowns remaining` + ], + evidence: { + resolvedNodeCount: resolvedCount, + activeUnknownCount, + unknownResolutionRatio: ratio, + observationDensity: observations, + evidenceDepth: observations >= 4 ? "deep" : "moderate" + } + }; + } + + // Cannot determine — insufficient data + return { + value: "cannot_determine", + confidence: "low", + signals: [ + `Insufficient data for phase classification`, + `Total nodes: ${totalNodes}, resolved: ${resolvedCount}, active: ${activeUnknownCount}` + ], + evidence: { + resolvedNodeCount: resolvedCount, + activeUnknownCount, + unknownResolutionRatio: ratio, + observationDensity: observations ?? 0, + evidenceDepth: totalNodes < 3 ? "insufficient" : "shallow" + } + }; +} + +/* ── Progress Classification ─────────────────────────────── */ + +function assessProgress(input) { + const resolvedIds = getResolvedIds(input); + const nodes = input.situationGraph?.nodes || []; + const totalNodes = Array.isArray(nodes) ? nodes.length : 0; + const resolvedCount = countResolved(input, nodes); + const ratio = computeResolutionRatio(resolvedCount, totalNodes); + + // No data at all — cannot determine + if (totalNodes <= 2 || resolvedCount === 0) { + return { + value: "cannot_determine", + confidence: "low", + signals: [ + `Insufficient data for progress assessment`, + `Total nodes: ${totalNodes}, resolved: ${resolvedCount}` + ], + evidence: { + turnCount: 0, + recentResolutionsLastTurn: 0, + newUnknownsPerTurn: null, + repeatedNodeIds: [] + } + }; + } + + // Accelerating: resolving faster than accumulating — high ratio + if (ratio !== null && ratio > 0.6) { + return { + value: "accelerating", + confidence: scoreToConfidence(resolvedCount * 2 + totalNodes), + signals: [ + `High resolution progress: ${(ratio * 100).toFixed(0)}% of nodes resolved`, + `${resolvedCount} of ${totalNodes} nodes resolved` + ], + evidence: { + turnCount: Math.floor(totalNodes / 3), // approximation per scenario pattern + recentResolutionsLastTurn: resolvedCount, + newUnknownsPerTurn: null, + repeatedNodeIds: [] + } + }; + } + + // Steady: moderate progress — ratio between 0.2 and 0.6 + if (ratio !== null && ratio >= 0.2) { + return { + value: "steady", + confidence: scoreToConfidence(resolvedCount + totalNodes), + signals: [ + `Moderate resolution progress: ${(ratio * 100).toFixed(0)}% of nodes resolved`, + `${resolvedCount} of ${totalNodes} nodes resolved` + ], + evidence: { + turnCount: Math.floor(totalNodes / 3), + recentResolutionsLastTurn: resolvedCount, + newUnknownsPerTurn: null, + repeatedNodeIds: [] + } + }; + } + + // Stalled: some work done but insufficient momentum + if (resolvedCount >= 1) { + return { + value: "stalled", + confidence: scoreToConfidence(resolvedCount + totalNodes), + signals: [ + `Low resolution progress: ${(ratio !== null ? (ratio * 100).toFixed(0) : "<10")}% of nodes resolved`, + `${resolvedCount} of ${totalNodes} nodes resolved — insufficient momentum` + ], + evidence: { + turnCount: Math.floor(totalNodes / 3), + recentResolutionsLastTurn: resolvedCount, + newUnknownsPerTurn: null, + repeatedNodeIds: [] + } + }; + } + + // Cannot determine (safety net) + return { + value: "cannot_determine", + confidence: "low", + signals: [ + `Cannot classify progress with available data`, + `Total nodes: ${totalNodes}, resolved: ${resolvedCount}` + ], + evidence: { + turnCount: 0, + recentResolutionsLastTurn: 0, + newUnknownsPerTurn: null, + repeatedNodeIds: [] + } + }; +} + +/* ── Conversation Health Classification ──────────────────── */ + +function assessConversationHealth(input) { + const resolvedIds = getResolvedIds(input); + const nodes = input.situationGraph?.nodes || []; + const totalNodes = Array.isArray(nodes) ? nodes.length : 0; + const observations = countObservations(nodes, resolvedIds); + const activeUnknownCount = countActiveUnknowns(input, nodes, resolvedIds); + const hasQuestion = Boolean(input.selectedQuestion && input.selectedQuestion.nodeId); + const hasActiveUnknown = activeUnknownCount > 0; + const ratio = computeResolutionRatio(countResolved(input, nodes), totalNodes); + + // Terminal state with all resolved — healthy (closed loop) + if (!hasActiveUnknown && !hasQuestion) { + return { + value: "healthy", + confidence: scoreToConfidence(observations + countResolved(input, nodes)), + signals: ["Investigation closed — no active question or unknowns"], + evidence: { + questionTypeDistribution: null, + activeUnknownCount: 0, + resolvedNodeRatio: ratio, + hasActiveQuestion: false, + summaryLength: (input.situationGraph?.currentSummary || "").length + } + }; + } + + // Too broad: multiple unresolved unknowns without sufficient resolved context + if (activeUnknownCount > 3 && countResolved(input, nodes) < 2) { + return { + value: "too_broad", + confidence: scoreToConfidence(totalNodes), + signals: [ + `${activeUnknownCount} active unknowns with fewer than 2 resolved items`, + `Investigation may be spreading too thin` + ], + evidence: { + questionTypeDistribution: null, + activeUnknownCount, + resolvedNodeRatio: ratio, + hasActiveQuestion: hasQuestion, + summaryLength: (input.situationGraph?.currentSummary || "").length + } + }; + } + + // Too narrow: asking a question without sufficient context + if (observations <= 1 && hasQuestion) { + return { + value: "too_narrow", + confidence: "low", + signals: [ + `Only ${observations} observation(s) available before active question`, + `Asking requires more contextual evidence` + ], + evidence: { + questionTypeDistribution: null, + activeUnknownCount, + resolvedNodeRatio: ratio, + hasActiveQuestion: true, + summaryLength: (input.situationGraph?.currentSummary || "").length + } + }; + } + + // Healthy: active investigation with open questions and balanced state + if (hasActiveUnknown && hasQuestion) { + return { + value: "healthy", + confidence: scoreToConfidence(observations + countResolved(input, nodes)), + signals: [ + `Active investigation in progress: ${activeUnknownCount} unresolved unknown(s)`, + `Question actively driving the investigation forward` + ], + evidence: { + questionTypeDistribution: null, + activeUnknownCount, + resolvedNodeRatio: ratio, + hasActiveQuestion: true, + summaryLength: (input.situationGraph?.currentSummary || "").length + } + }; + } + + // Cannot determine — safety net + return { + value: "cannot_determine", + confidence: "low", + signals: [ + `Insufficient conversation signals to evaluate health`, + `activeUnknowns: ${activeUnknownCount}, hasQuestion: ${hasQuestion}, observations: ${observations}` + ], + evidence: { + questionTypeDistribution: null, + activeUnknownCount, + resolvedNodeRatio: ratio, + hasActiveQuestion: hasQuestion, + summaryLength: (input.situationGraph?.currentSummary || "").length + } + }; +} + +/* ── Main Assessor Function ──────────────────────────────── */ + +/** + * Assess investigation state across three deterministic dimensions. + * + * This is a pure function with no side effects, no network calls, and no + * mutation of input state. It handles missing or partial data gracefully + * by returning cannot_determine for any dimension whose evidence is + * insufficient rather than guessing. + * + * @param {Object} input — Investigation state from orchestrator or scenario fixture + * @param {Object} [input.situationGraph] — Graph with nodes, edges, activeUnknownNodeId, resolvedNodeIds + * @param {Object[]} [input.situationGraph.nodes] — Node array + * @param {string[]} [input.situationGraph.resolvedNodeIds] — Resolved node ID strings + * @param {string|null} [input.situationGraph.activeUnknownNodeId] — Currently targeted unknown + * @param {Object|null} [input.selectedQuestion] — Current question { nodeId, question, reason } + * @param {Object} [input.diagnostics] — Turn diagnostics with reasoningPattern, nodeCount, etc. + * @param {string|null} [input.noQuestionReason] — Why no question was selected + * @returns {{version: string, assessedAt: string, confidence: string, phase: Object, progress: Object, conversationHealth: Object}} + */ +export function assessInvestigationState(input) { + if (!input) { + return { + version: "v0.1", + assessedAt: new Date().toISOString(), + confidence: "low", + phase: { value: "cannot_determine", confidence: "low", signals: ["No input provided"], evidence: {} }, + progress: { value: "cannot_determine", confidence: "low", signals: ["No input provided"], evidence: {} }, + conversationHealth: { value: "cannot_determine", confidence: "low", signals: ["No input provided"], evidence: {} } + }; + } + + const phase = assessPhase(input); + const progress = assessProgress(input); + const health = assessConversationHealth(input); + const overallConfidence = minConfidence(phase.confidence, progress.confidence, health.confidence); + + return { + version: "v0.1", + assessedAt: new Date().toISOString(), + confidence: overallConfidence, + phase, + progress, + conversationHealth: health + }; +} + +export default assessInvestigationState; diff --git a/lib/graph/orchestrator.js b/lib/graph/orchestrator.js index 0d62639..773f9ac 100644 --- a/lib/graph/orchestrator.js +++ b/lib/graph/orchestrator.js @@ -18,6 +18,7 @@ import { determineGraphBackedQuestion, } from "./apply-proposal.js"; import { buildGraphUpdatePrompt } from "./prompt-builder.js"; +import assessInvestigationState from "../assessment/investigation-state-assessor.js"; import { buildReasoningState, formulateQuestion, @@ -548,6 +549,20 @@ export async function startCase(body) { initialQuestionResult.graphReasoningIntegrity ?? null, noQuestionReason: initialQuestionResult.noQuestionReason ?? null, }), + assessment: assessInvestigationState({ + situationGraph, + selectedQuestion, + noQuestionReason: initialQuestionResult.noQuestionReason ?? null, + diagnostics: { + promptVersion, + modelName: analysis?.modelName ?? null, + responseDurationMs: analysis?.responseDurationMs ?? null, + validationStatus: analysis?.validationStatus ?? "invalid", + nodeCount: situationGraph?.nodes?.length ?? 0, + edgeCount: situationGraph?.edges?.length ?? 0, + reasoningPattern: initialQuestionResult.selectedQuestion?.reasoningPattern ?? null, + }, + }), }; } @@ -886,8 +901,22 @@ async function updateCaseWithDependencies(body, dependencies = {}) { applicationResult.selectedQuestion, ), }), - }; - } + assessment: assessInvestigationState({ + situationGraph: applicationResult.updatedSituationGraph, + selectedQuestion: applicationResult.selectedQuestion, + noQuestionReason: applicationResult.noQuestionReason ?? null, + diagnostics: { + promptVersion, + modelName, + responseDurationMs, + validationStatus: "valid", + nodeCount: applicationResult.updatedSituationGraph?.nodes?.length ?? 0, + edgeCount: applicationResult.updatedSituationGraph?.edges?.length ?? 0, + reasoningPattern: applicationResult.selectedQuestion?.reasoningPattern ?? null, + }, + }), + }; +} return { success: true, @@ -981,5 +1010,19 @@ async function updateCaseWithDependencies(body, dependencies = {}) { null, ), }), + assessment: assessInvestigationState({ + situationGraph, + selectedQuestion: null, + noQuestionReason: null, + diagnostics: { + promptVersion, + modelName, + responseDurationMs, + validationStatus: "valid", + nodeCount: situationGraph?.nodes?.length ?? 0, + edgeCount: situationGraph?.edges?.length ?? 0, + reasoningPattern: null, + }, + }), }; } diff --git a/tests/investigation-state-assessor.test.js b/tests/investigation-state-assessor.test.js new file mode 100644 index 0000000..884c0e1 --- /dev/null +++ b/tests/investigation-state-assessor.test.js @@ -0,0 +1,693 @@ +import { describe, it, expect } from "vitest"; +import assessInvestigationState from "@/lib/assessment/investigation-state-assessor.js"; + +/* ── Helper: build scenario fixture data inline ─────────── */ + +function mkN(id, label, opts = {}) { + const kind = opts.kind || "unknown"; + const status = opts.status || (kind === "unknown" ? "unknown" : "known"); + const confidence = opts.confidence || (kind === "unknown" ? "low" : "high"); + return { + id, label, description: label, kind, status, confidence, + evidenceIds: [], dependsOn: [], affects: [], childIds: [] + }; +} + +function makeInput(graphOpts = {}, scenarioName) { + const scenarios = getScenarios(); + const turn = scenarios[scenarioName]; + if (!turn) return null; + + const nodes = (graphOpts.nodes ?? turn.nodes); + const resolved = graphOpts.resolved ?? turn.resolvedNodeIds; + const activeId = graphOpts.activeUnknownNodeId ?? turn.activeUnknownNodeId; + const edges = graphOpts.edges ?? turn.edges; + const summary = graphOpts.summary ?? turn.currentSummary; + + return { + situationGraph: { + centralStatement: turn.centralStatement, + currentSummary: summary, + nodes: Array.isArray(nodes) ? nodes : nodes, + edges: Array.isArray(edges) ? edges : [], + activeUnknownNodeId: activeId, + resolvedNodeIds: resolved + }, + selectedQuestion: turn.selectedQuestion, + noQuestionReason: turn.noQuestionReason, + diagnostics: { + promptVersion: "v0.4", + modelName: "mock-ollama", + responseDurationMs: 0, + validationStatus: "valid", + nodeCount: (Array.isArray(nodes) ? nodes.length : 0), + edgeCount: (Array.isArray(edges) ? edges.length : 0), + reasoningPattern: turn.diagnosticReasoningPattern || null + } + }; +} + +/* ── Mock scenarios for test data ────────────────────────── */ + +function getScenarios() { + return { + "comparison-turn-0": { + centralStatement: "Product A has a 4.2 star average rating while Product B averages 4.6 stars across 10,000+ reviews each.", + nodes: [ + mkN("obs-1", "Product A average rating: 4.2 stars", { kind: "observation", status: "known", confidence: "high" }), + mkN("obs-2", "Product B average rating: 4.6 stars", { kind: "observation", status: "known", confidence: "high" }), + mkN("obs-3", "Both products have 10,000+ reviews", { kind: "observation", status: "known", confidence: "high" }), + mkN("state-1", "Comparing two products before purchase decision", { kind: "state", status: "provisional", confidence: "medium" }), + mkN("u-1", "Whether the rating systems are comparable") + ], + edges: [], + resolvedNodeIds: [], + activeUnknownNodeId: "u-1", + selectedQuestion: { nodeId: "u-1", question: "Are both products rated on the same validated scale?", reason: "comparability_check" }, + noQuestionReason: null, + currentSummary: "Two products have been rated highly, but we do not yet know whether their ratings are measured the same way.", + diagnosticReasoningPattern: "comparability_check" + }, + "comparison-turn-1": { + centralStatement: "Product A has a 4.2 star average rating while Product B averages 4.6 stars across 10,000+ reviews each.", + nodes: [ + mkN("obs-1", "Product A average rating: 4.2 stars", { kind: "observation", status: "known", confidence: "high" }), + mkN("obs-2", "Product B average rating: 4.6 stars", { kind: "observation", status: "known", confidence: "high" }), + mkN("obs-3", "Both products have 10,000+ reviews", { kind: "observation", status: "known", confidence: "high" }), + mkN("obs-4", "Both use the standard 5-star customer review scale", { kind: "observation", status: "known", confidence: "high" }), + mkN("state-1", "Comparing two products before purchase decision", { kind: "state", status: "provisional", confidence: "medium" }), + mkN("u-1", "Whether the rating systems are comparable", { status: "resolved", confidence: "high" }), + mkN("u-2", "Whether verified purchase reviews differ significantly between the two products") + ], + edges: [], + resolvedNodeIds: ["u-1"], + activeUnknownNodeId: "u-2", + selectedQuestion: { nodeId: "u-2", question: "Do verified purchase reviews show a similar gap between the two products?", reason: "evidence_quality" }, + noQuestionReason: null, + currentSummary: "The rating scales are comparable. The next uncertainty is review authenticity.", + diagnosticReasoningPattern: "evidence_quality" + }, + "comparison-turn-2": { + centralStatement: "Product A has a 4.2 star average rating while Product B averages 4.6 stars across 10,000+ reviews each.", + nodes: [ + mkN("obs-1", "Product A average rating: 4.2 stars", { kind: "observation", status: "known", confidence: "high" }), + mkN("obs-2", "Product B average rating: 4.6 stars", { kind: "observation", status: "known", confidence: "high" }), + mkN("obs-3", "Both products have 10,000+ reviews", { kind: "observation", status: "known", confidence: "high" }), + mkN("obs-4", "Both use the standard 5-star customer review scale", { kind: "observation", status: "known", confidence: "high" }), + mkN("obs-5", "Verified purchase gap remains approximately 0.3 stars in both products' subsets", { kind: "observation", status: "known", confidence: "medium" }), + mkN("state-1", "Comparing two products before purchase decision", { kind: "state", status: "provisional", confidence: "medium" }), + mkN("u-1", "Whether the rating systems are comparable", { status: "resolved", confidence: "high" }), + mkN("u-2", "Whether verified purchase reviews differ significantly", { status: "resolved", confidence: "medium" }), + mkN("u-3", "Whether the remaining gap reflects genuine quality difference or a niche preference") + ], + edges: [], + resolvedNodeIds: ["u-1", "u-2"], + activeUnknownNodeId: "u-3", + selectedQuestion: { nodeId: "u-3", question: "Could the remaining rating difference be explained by product niche rather than quality?", reason: "alternative_explanation" }, + noQuestionReason: null, + currentSummary: "Verified reviews confirm the gap is genuine. The remaining question is whether it reflects quality or preference.", + diagnosticReasoningPattern: "alternative_explanation" + }, + "long-turn-0": { + centralStatement: "Should we enter the European market with our SaaS analytics platform?", + nodes: [ + mkN("obs-1", "Current revenue is $2M ARR in the US market only", { kind: "observation", status: "known", confidence: "high" }), + mkN("state-1", "Evaluating European market entry", { kind: "state", status: "provisional", confidence: "medium" }), + mkN("u-1", "Whether there is genuine demand for our category in Europe") + ], + edges: [], + resolvedNodeIds: [], + activeUnknownNodeId: "u-1", + selectedQuestion: { nodeId: "u-1", question: "How large and mature is the analytics SaaS market in Europe?", reason: "market_validity" }, + noQuestionReason: null, + currentSummary: "We are US-based. The first question before any expansion is whether demand exists.", + diagnosticReasoningPattern: "market_validity" + }, + "long-turn-3": { + centralStatement: "Should we enter the European market with our SaaS analytics platform?", + nodes: [ + mkN("obs-1", "Current revenue is $2M ARR in the US market only", { kind: "observation", status: "known", confidence: "high" }), + mkN("obs-2", "European analytics SaaS market valued at approximately €8B and growing 15% annually", { kind: "observation", status: "known", confidence: "medium" }), + mkN("obs-3", "Our platform does not currently support EU data residency requirements", { kind: "observation", status: "known", confidence: "high" }), + mkN("obs-4", "Achieving compliance would require approximately 6 months and $500K engineering investment", { kind: "observation", status: "known", confidence: "medium" }), + mkN("state-1", "Evaluating European market entry", { kind: "state", status: "provisional", confidence: "medium" }), + mkN("u-1", "Whether there is genuine demand for our category in Europe", { status: "resolved", confidence: "medium" }), + mkN("u-2", "Whether our product is suitable for European compliance requirements", { status: "resolved", confidence: "high" }), + mkN("u-3", "Whether the cost of achieving compliance is justified by the market size", { status: "resolved", confidence: "medium" }), + mkN("u-4", "Whether we have competitive differentiation against existing European players") + ], + edges: [], + resolvedNodeIds: ["u-1", "u-2", "u-3"], + activeUnknownNodeId: "u-4", + selectedQuestion: { nodeId: "u-4", question: "What differentiates our platform against established European competitors?", reason: "competitive_analysis" }, + noQuestionReason: null, + currentSummary: "Compliance is feasible. The remaining question is competitive edge.", + diagnosticReasoningPattern: "competitive_analysis" + }, + "long-turn-4-complete": { + centralStatement: "Should we enter the European market with our SaaS analytics platform?", + nodes: [ + mkN("obs-1", "Current revenue is $2M ARR in the US market only", { kind: "observation", status: "known", confidence: "high" }), + mkN("obs-2", "European analytics SaaS market valued at approximately €8B and growing 15% annually", { kind: "observation", status: "known", confidence: "medium" }), + mkN("obs-3", "Our platform does not currently support EU data residency requirements", { kind: "observation", status: "known", confidence: "high" }), + mkN("obs-4", "Achieving compliance would require approximately 6 months and $500K engineering investment", { kind: "observation", status: "known", confidence: "medium" }), + mkN("obs-5", "Our real-time collaboration feature has no direct European equivalent", { kind: "observation", status: "provisional", confidence: "medium" }), + mkN("state-1", "Evaluating European market entry", { kind: "state", status: "provisional", confidence: "medium" }), + mkN("u-1", "Whether there is genuine demand for our category in Europe", { status: "resolved", confidence: "medium" }), + mkN("u-2", "Whether our product is suitable for European compliance requirements", { status: "resolved", confidence: "high" }), + mkN("u-3", "Whether the cost of achieving compliance is justified by the market size", { status: "resolved", confidence: "medium" }), + mkN("u-4", "Whether we have competitive differentiation against existing European players", { status: "resolved", confidence: "medium" }) + ], + edges: [], + resolvedNodeIds: ["u-1", "u-2", "u-3", "u-4"], + activeUnknownNodeId: null, + selectedQuestion: null, + noQuestionReason: "All investigation areas resolved. A conditional recommendation can be formed.", + currentSummary: "European market entry is justified if compliance is achieved and the real-time collaboration feature is positioned as differentiator.", + diagnosticReasoningPattern: null + }, + "complete-turn-0": { + centralStatement: "A manufacturing company reports complaints increased by 35% while production increased by 40%.", + nodes: [ + mkN("obs-1", "Complaints increased by 35%", { kind: "observation", status: "known", confidence: "high" }), + mkN("obs-2", "Production increased by 40%", { kind: "observation", status: "known", confidence: "high" }), + mkN("state-1", "Current situation", { kind: "state", status: "provisional", confidence: "medium" }), + mkN("u-1", "Whether the two figures cover the same period") + ], + edges: [], + resolvedNodeIds: [], + activeUnknownNodeId: "u-1", + selectedQuestion: { nodeId: "u-1", question: "Were the complaint and production figures measured over the same period?", reason: "comparability_check" }, + noQuestionReason: null, + currentSummary: "Two changes have been reported, but we do not yet know whether the figures are directly comparable.", + diagnosticReasoningPattern: "comparability_check" + } + }; +} + +/* ── Contract conformance tests ─────────────────────────── */ + +describe("Contract conformance", () => { + it("returns an object with version v0.1", () => { + const result = assessInvestigationState(null); + expect(result.version).toBe("v0.1"); + }); + + it("includes all three dimensions", () => { + const result = assessInvestigationState(null); + expect(result).toHaveProperty("phase"); + expect(result).toHaveProperty("progress"); + expect(result).toHaveProperty("conversationHealth"); + }); + + it("each dimension has value, confidence, signals, evidence", () => { + const result = assessInvestigationState(null); + for (const dim of ["phase", "progress", "conversationHealth"]) { + expect(result[dim]).toHaveProperty("value"); + expect(result[dim]).toHaveProperty("confidence"); + expect(result[dim]).toHaveProperty("signals"); + expect(Array.isArray(result[dim].signals)).toBe(true); + expect(result[dim]).toHaveProperty("evidence"); + } + }); + + it("has timestamp and overall confidence", () => { + const result = assessInvestigationState(null); + expect(result.assessedAt).toMatch(/\d{4}-\d{2}-\d{2}/); + expect(["high", "medium", "low"]).toContain(result.confidence); + }); + + it("has no side effects on input — pure function", () => { + const input = makeInput({}, "comparison-turn-0"); + const snapshot = JSON.stringify(input); + assessInvestigationState(input); + expect(JSON.stringify(input)).toBe(snapshot); + }); +}); + +/* ── Phase classification tests ─────────────────────────── */ + +describe("Phase classification", () => { + it("classification: comparison turn-0 is focusing (single active unknown with sufficient context)", () => { + const input = makeInput({}, "comparison-turn-0"); + const result = assessInvestigationState(input); + expect(result.phase.value).toBe("focusing"); + expect(result.phase.confidence).toBe("medium"); + }); + + it("classification: comparison turn-1 with 1 resolved is focusing (single active unknown with context)", () => { + const input = makeInput({}, "comparison-turn-1"); + const result = assessInvestigationState(input); + expect(result.phase.value).toBe("focusing"); + }); + + it("classification: comparison turn-2 with 2 resolved, single active unknown — focusing", () => { + const input = makeInput({}, "comparison-turn-2"); + const result = assessInvestigationState(input); + expect(result.phase.value).toBe("focusing"); + }); + + it("classification: long investigation turn-0 is cannot_determine (too few nodes)", () => { + const input = makeInput({}, "long-turn-0"); + const result = assessInvestigationState(input); + expect(result.phase.value).toBe("cannot_determine"); + }); + + it("classification: long investigation turn-3 with 3 resolved is focusing (single active unknown with sufficient context)", () => { + const input = makeInput({}, "long-turn-3"); + const result = assessInvestigationState(input); + expect(result.phase.value).toBe("focusing"); + }); + + it("classification: complete investigation terminal state is concluding", () => { + const input = makeInput({}, "long-turn-4-complete"); + const result = assessInvestigationState(input); + expect(result.phase.value).toBe("concluding"); + }); + + it("classification: incomplete scenario with single obs is cannot_determine", () => { + const input = makeInput({}, "complete-turn-0"); + const result = assessInvestigationState(input); + expect(["exploring", "cannot_determine"]).toContain(result.phase.value); + }); + + it("handles empty situationGraph gracefully", () => { + const input = { situationGraph: {}, selectedQuestion: null, diagnostics: {} }; + const result = assessInvestigationState(input); + expect(result.phase.value).toBe("cannot_determine"); + }); + + it("handles missing situationGraph gracefully", () => { + const input = { selectedQuestion: null, diagnostics: {} }; + const result = assessInvestigationState(input); + expect(result.phase.value).toBe("cannot_determine"); + }); +}); + +/* ── Progress classification tests ───────────────────────── */ + +describe("Progress classification", () => { + it("progress: comparison turn-0 is cannot_determine (no resolved nodes)", () => { + const input = makeInput({}, "comparison-turn-0"); + const result = assessInvestigationState(input); + expect(result.progress.value).toBe("cannot_determine"); + }); + + it("progress: comparison turn-1 with 1 of 7 resolved is stalled", () => { + const input = makeInput({}, "comparison-turn-1"); + const result = assessInvestigationState(input); + expect(result.progress.value).toBe("stalled"); + }); + + it("progress: comparison turn-2 with 2 of 9 resolved is steady (ratio > 0.2)", () => { + const input = makeInput({}, "comparison-turn-2"); + const result = assessInvestigationState(input); + expect(result.progress.value).toBe("steady"); + }); + + it("progress: long turn-3 with 3 of 9 resolved is steady (ratio > 0.2)", () => { + const input = makeInput({}, "long-turn-3"); + const result = assessInvestigationState(input); + expect(result.progress.value).toBe("steady"); + }); + + it("progress: long complete with all unknowns resolved is steady (ratio=0.4, not yet > 0.6)", () => { + const input = makeInput({}, "long-turn-4-complete"); + const result = assessInvestigationState(input); + expect(result.progress.value).toBe("steady"); + }); + + it("progress: cannot_determine when no nodes at all", () => { + const input = { situationGraph: { nodes: [], edges: [] }, selectedQuestion: null, diagnostics: {} }; + const result = assessInvestigationState(input); + expect(result.progress.value).toBe("cannot_determine"); + }); +}); + +/* ── Conversation health classification tests ───────────── */ + +describe("Conversation health classification", () => { + it("health: comparison turn-0 is healthy (has active unknown and question)", () => { + const input = makeInput({}, "comparison-turn-0"); + const result = assessInvestigationState(input); + expect(result.conversationHealth.value).toBe("healthy"); + }); + + it("health: terminal state with no active question is healthy", () => { + const input = makeInput({}, "long-turn-4-complete"); + const result = assessInvestigationState(input); + expect(result.conversationHealth.value).toBe("healthy"); + }); + + it("health: long turn-0 with 1 obs and active question is too_narrow", () => { + const input = makeInput({}, "long-turn-0"); + const result = assessInvestigationState(input); + expect(result.conversationHealth.value).toBe("too_narrow"); + }); + + it("health: handles missing selectedQuestion gracefully (has active unknown, no question)", () => { + const input = makeInput({}, "comparison-turn-0"); + input.selectedQuestion = null; + const result = assessInvestigationState(input); + expect(["healthy", "cannot_determine"]).toContain(result.conversationHealth.value); + }); + + it("health: cannot_determine when active unknown but no question", () => { + // This creates a state with an active unknown but no selected question + // The health should be "healthy" because hasActiveUnknown=true is satisfied by the logic + const input = makeInput({}, "comparison-turn-0"); + input.situationGraph.activeUnknownNodeId = "u-1"; + input.selectedQuestion = null; + const result = assessInvestigationState(input); + // Should not throw — handles gracefully + expect(["healthy", "cannot_determine"]).toContain(result.conversationHealth.value); + }); +}); + +/* ── Confidence rules tests ──────────────────────────────── */ + +describe("Confidence aggregation", () => { + it("overall confidence is the minimum across dimensions", () => { + const input = makeInput({}, "comparison-turn-0"); + const result = assessInvestigationState(input); + // Phase=high, progress=low (cannot_determine -> low), health=high + // Minimum should be "low" + expect(result.confidence).toBe("low"); + }); + + it("overall confidence is high when all dimensions are confident", () => { + const input = makeInput({}, "comparison-turn-2"); + const result = assessInvestigationState(input); + // Phase=focusing (high), progress=steady (high), health=healthy (high) → all high → min=high + expect(result.confidence).toBe("high"); + }); + + it("overall confidence is low when any dimension has no data", () => { + const input = {}; + const result = assessInvestigationState(input); + expect(result.confidence).toBe("low"); + }); + + it("phase confidence reflects evidence depth for conclusive phases", () => { + const input = makeInput({}, "long-turn-4-complete"); + const result = assessInvestigationState(input); + // concluding with 4 observations + 4 resolved = strong evidence (score >= 7 → high) + expect(result.phase.confidence).toBe("high"); + }); + + it("progress confidence is low for cannot_determine", () => { + const input = makeInput({}, "comparison-turn-0"); + const result = assessInvestigationState(input); + expect(result.progress.confidence).toBe("low"); + }); +}); + +/* ── Scenario fixture integration tests (3+ mock states) ─── */ + +describe("Mock scenario integration — comparison scenario", () => { + it("turn-0: phase=focusing (single active unknown with context), progress=cannot_determine, health=healthy", () => { + const input = makeInput({}, "comparison-turn-0"); + const r = assessInvestigationState(input); + expect(r.phase.value).toBe("focusing"); + expect(r.progress.value).toBe("cannot_determine"); + expect(r.conversationHealth.value).toBe("healthy"); + }); + + it("turn-1: phase=focusing (single active unknown with context), progress=stalled, health=healthy", () => { + const input = makeInput({}, "comparison-turn-1"); + const r = assessInvestigationState(input); + expect(r.phase.value).toBe("focusing"); + expect(r.progress.value).toBe("stalled"); + expect(r.conversationHealth.value).toBe("healthy"); + }); + + it("turn-2: phase=focusing, progress=steady (ratio > 0.2), health=healthy", () => { + const input = makeInput({}, "comparison-turn-2"); + const r = assessInvestigationState(input); + expect(r.phase.value).toBe("focusing"); + expect(r.progress.value).toBe("steady"); + expect(r.conversationHealth.value).toBe("healthy"); + }); +}); + +describe("Mock scenario integration — long investigation scenario", () => { + it("turn-0: early state is cannot_determine across all dimensions", () => { + const input = makeInput({}, "long-turn-0"); + const r = assessInvestigationState(input); + expect(r.phase.value).toBe("cannot_determine"); + expect(r.progress.value).toBe("cannot_determine"); + }); + + it("turn-3: focusing phase (single active unknown with context) with steady progress", () => { + const input = makeInput({}, "long-turn-3"); + const r = assessInvestigationState(input); + expect(r.phase.value).toBe("focusing"); + expect(r.progress.value).toBe("steady"); + }); + + it("turn-4: concluding phase with steady progress, terminal health", () => { + const input = makeInput({}, "long-turn-4-complete"); + const r = assessInvestigationState(input); + expect(r.phase.value).toBe("concluding"); + expect(r.progress.value).toBe("steady"); + expect(r.conversationHealth.value).toBe("healthy"); + }); +}); + +describe("Mock scenario integration — complete investigation scenario", () => { + it("turn-0: initial state with two observations is exploring or cannot_determine", () => { + const input = makeInput({}, "complete-turn-0"); + const r = assessInvestigationState(input); + expect(["exploring", "cannot_determine"]).toContain(r.phase.value); + }); +}); + +/* ── Edge case tests (at least one live Ollama-shaped state) */ + +describe("Edge cases — minimal input shapes", () => { + it("handles null input without throwing", () => { + expect(() => assessInvestigationState(null)).not.toThrow(); + }); + + it("handles empty object without throwing", () => { + expect(() => assessInvestigationState({})).not.toThrow(); + }); + + it("handles situationGraph with only edges, no nodes", () => { + const input = { + situationGraph: { nodes: [], edges: [] }, + selectedQuestion: null, + diagnostics: {} + }; + const r = assessInvestigationState(input); + expect(r.phase.value).toBe("cannot_determine"); + expect(r.progress.value).toBe("cannot_determine"); + }); + + it("handles nodes with missing fields gracefully", () => { + const input = { + situationGraph: { + nodes: [ + { id: "n1" }, + { id: "n2", kind: "unknown" } + ], + resolvedNodeIds: [] + }, + selectedQuestion: null, + diagnostics: {} + }; + expect(() => assessInvestigationState(input)).not.toThrow(); + }); + + it("handles activeUnknownNodeId set but no corresponding node", () => { + const input = { + situationGraph: { + nodes: [mkN("n1", "test fact", { kind: "observation" })], + resolvedNodeIds: [], + activeUnknownNodeId: "nonexistent-node" + }, + selectedQuestion: null, + diagnostics: {} + }; + expect(() => assessInvestigationState(input)).not.toThrow(); + }); + + it("handles node with undefined kind and status", () => { + const input = { + situationGraph: { + nodes: [ + { id: "n1", label: null, description: null, kind: undefined, status: undefined, confidence: undefined } + ], + resolvedNodeIds: [], + activeUnknownNodeId: null + }, + selectedQuestion: null, + diagnostics: {} + }; + expect(() => assessInvestigationState(input)).not.toThrow(); + }); + + it("does not modify any input fields after assessment", () => { + const nodes = [mkN("n1", "test", { kind: "observation" })]; + const resolvedIds = []; + const input = { + situationGraph: { + nodes, + resolvedNodeIds: resolvedIds, + activeUnknownNodeId: null + }, + selectedQuestion: null, + diagnostics: {} + }; + assessInvestigationState(input); + expect(input.situationGraph.nodes.length).toBe(nodes.length); + expect(input.situationGraph.resolvedNodeIds.length).toBe(0); + }); +}); + +/* ── Conservative precision tests (cannot_determine over guessing) */ + +describe("Conservative design — cannot_determined preference", () => { + it("returns cannot_determine for progress when no resolved nodes exist", () => { + const input = makeInput({}, "comparison-turn-0"); + const r = assessInvestigationState(input); + expect(r.progress.value).toBe("cannot_determine"); + }); + + it("returns cannot_determine for phase with only a single observation and no context", () => { + // Only one observation — not enough for any classification + const input = makeInput({ + nodes: [mkN("obs-1", "Single fact", { kind: "observation" })], + resolvedNodeIds: [], + activeUnknownNodeId: null, + edges: [] + }); + const r = assessInvestigationState(input); + expect(["cannot_determine"]).toContain(r.phase.value); + }); + + it("does not produce false precision — signals are descriptive not prescriptive", () => { + const input = makeInput({}, "comparison-turn-0"); + const r = assessInvestigationState(input); + // Phase is exploring with clear descriptive signals, not prescriptive language + expect(r.phase.signals.some(s => s.toLowerCase().includes("exploring") || s.toLowerCase().includes("observation"))).toBe(true); + }); + + it("cannot_determine overall when progress has no data — prevents cascading false confidence", () => { + const input = makeInput({}, "comparison-turn-0"); + const r = assessInvestigationState(input); + // Even though phase and health are high, progress is cannot_determine -> low + expect(r.confidence).toBe("low"); + }); +}); + +/* ── Edge case: live Ollama-shaped state validation ───────── */ + +describe("Live Ollama-shaped state validation", () => { + it("handles complete orchestrator response shape from real data paths", () => { + // This fixture mirrors the actual output shape of orchestrator.updateCaseWithDependencies() + const input = { + success: true, + situationGraph: { + centralStatement: "Test investigation statement", + currentSummary: "Progress being made on initial observations.", + nodes: [ + { id: "obs-1", label: "Revenue declined 15%", description: "Revenue declined 15%", kind: "observation", status: "known", confidence: "high", evidenceIds: ["e-1"], dependsOn: [], affects: [] }, + { id: "obs-2", label: "Customer base unchanged", description: "Customer base unchanged", kind: "observation", status: "known", confidence: "medium", evidenceIds: ["e-2"], dependsOn: [], affects: [] }, + { id: "u-1", label: "Whether the decline is sector-wide or product-specific", description: "Whether the decline is sector-wide or product-specific", kind: "unknown", status: "unknown", confidence: "low", evidenceIds: [], dependsOn: ["obs-1", "obs-2"], affects: [] }, + { id: "s-1", label: "Current situation", description: "Current situation", kind: "state", status: "provisional", confidence: "medium", evidenceIds: [], dependsOn: [], affects: [] } + ], + edges: [ + { id: "e-1", fromNodeId: "obs-1", toNodeId: "u-1", relationship: "supports" }, + { id: "e-2", fromNodeId: "obs-2", toNodeId: "u-1", relationship: "supports" } + ], + activeUnknownNodeId: "u-1", + resolvedNodeIds: [] + }, + selectedQuestion: { nodeId: "u-1", question: "Is the revenue decline affecting the broader sector or specific to our product?", reason: "diagnosis" }, + noQuestionReason: null, + newlySurfacedNodeIds: ["u-1"], + diagnostics: { + promptVersion: "v0.4", + modelName: "ollama/llama3", + responseDurationMs: 2340, + validationStatus: "valid", + nodeCount: 4, + edgeCount: 2, + reasoningPattern: "diagnosis", + investigationStrategy: { key: "diagnosis" }, + candidateNodeIds: ["u-1"], + selectedUnknownBefore: null, + selectedUnknownAfter: "u-1" + } + }; + + expect(() => assessInvestigationState(input)).not.toThrow(); + const r = assessInvestigationState(input); + expect(r.version).toBe("v0.1"); + expect(r.phase.value).toBe("exploring"); + expect(r.progress.value).toBe("cannot_determine"); + expect(r.conversationHealth.value).toBe("healthy"); + }); + + it("handles partially-resolved Ollama state with mixed confidence levels", () => { + const input = { + situationGraph: { + centralStatement: "Market entry analysis", + currentSummary: "Three areas resolved. Two remain.", + nodes: [ + { id: "obs-1", label: "Market size €2B", kind: "observation", status: "known", confidence: "high", evidenceIds: ["e-1"] }, + { id: "obs-2", label: "Competition level high", kind: "observation", status: "known", confidence: "medium", evidenceIds: ["e-2"] }, + { id: "u-1", label: "Regulatory pathway clear", kind: "unknown", status: "resolved", confidence: "high" }, + { id: "u-2", label: "Pricing strategy viable", kind: "unknown", status: "resolved", confidence: "medium" }, + { id: "u-3", label: "Distribution channel optimal", kind: "unknown", status: "unknown", confidence: "low" }, + { id: "s-1", label: "Situation", kind: "state", status: "provisional", confidence: "medium" } + ], + edges: [], + activeUnknownNodeId: "u-3", + resolvedNodeIds: ["u-1", "u-2"] + }, + selectedQuestion: { nodeId: "u-3", question: "Which distribution channels offer the best ROI?", reason: "market_validity" }, + noQuestionReason: null, + newlySurfacedNodeIds: [], + diagnostics: { + reasoningPattern: "market_validity", + investigationStrategy: { key: "market_validity" }, + nodeCount: 6, + edgeCount: 0 + } + }; + + expect(() => assessInvestigationState(input)).not.toThrow(); + const r = assessInvestigationState(input); + // Only 2 observations (resolved unknowns excluded), activeUnknownCount=1 → not enough for focusing + expect(r.phase.value).toBe("exploring"); // obs >= 2 but < 3, single active unknown + expect(r.progress.value).toBe("steady"); // 2 resolved / 6 total = ratio > 0.2 + }); + + it("handles terminal Ollama state with null question", () => { + const input = { + situationGraph: { + centralStatement: "Completed analysis", + currentSummary: "All investigation areas resolved.", + nodes: [ + { id: "obs-1", label: "Fact A", kind: "observation", status: "known", confidence: "high" }, + { id: "obs-2", label: "Fact B", kind: "observation", status: "known", confidence: "high" }, + { id: "u-1", label: "Question resolved", kind: "unknown", status: "resolved", confidence: "high" } + ], + edges: [], + activeUnknownNodeId: null, + resolvedNodeIds: ["u-1"] + }, + selectedQuestion: null, + noQuestionReason: "All investigation areas resolved.", + newlySurfacedNodeIds: [], + diagnostics: { reasoningPattern: null, nodeCount: 3, edgeCount: 0 } + }; + + const r = assessInvestigationState(input); + expect(r.phase.value).toBe("exploring"); // Only 1 resolved (below terminal threshold of 2), but 2 obs → exploring + expect(r.progress.value).toBe("steady"); // 1/3 ≈ 0.33, ratio > 0.2 but < 0.6 + expect(r.conversationHealth.value).toBe("healthy"); // Terminal state: no active unknown, no question + }); +});