exp(18): implement investigation state assessment layer
Implement the three-dimensional assessment (phase, progress, conversation health) that sits between narrative and behaviour selection. Key changes: - lib/assessment/investigation-state-assessor.js: assessor module with countObservations, assessPhase, assessProgress, assessConversationHealth, assessInvestigationState — deterministic classifiers using known rules - tests/investigation-state-assessor.test.js: 51 tests covering phase classification (orienting→concluding), progress thresholds, health conditions, confidence aggregation, edge cases, and observation counting - lib/graph/orchestrator.js: integration calls passing correctly-shaped input to assessInvestigationState() at three call sites (~552, ~904, ~1013) Design decisions encoded in this iteration: - countObservations counts nodes with known/resolved status + high-confidence non-unknown non-state nodes (not just explicit observation-kind nodes) - Phase uses seven values including cannot_determine for insufficient data - Progress uses resolution ratio thresholds: accelerating (>0.6), steady (0.2-0.6), stalled (<0.2 with ≥1 resolved) - Overall confidence = minimum across all three dimensions (conservative) Also adds investigation-state-assessment-contract.md and updates design-evolution-log, investigation-state-assessment.md (status header), and investigation-turn-cycle.md (implementation status table).
This commit is contained in:
@@ -892,13 +892,57 @@ This architecture emerged from observation, not top-down design. It may still ch
|
||||
|
||||
A complete investigation can be described as a repeating turn cycle in which every architectural layer has a single responsibility.
|
||||
|
||||
Status
|
||||
Result
|
||||
|
||||
Architectural.
|
||||
Experiment validated that the investigation turn cycle is an *observation* about how existing layers interact rather than a new architectural layer. All eight stages (User Observation → Reasoning Graph → Investigation Narrative → State Assessment → Behaviour Selection → Conversation → Workspace → Wait) are supported by current architecture components, but only Stages 1–3 and 7 have working implementations. Stage 4 (State Assessment) and Stage 5 (Behaviour Selection) remain as architectural specifications without executable code.
|
||||
|
||||
Evaluation
|
||||
What did we learn?
|
||||
|
||||
Pending.
|
||||
- The turn cycle confirms that assessment sits between narrative and behaviour selection, not after the graph directly.
|
||||
- Every layer has one responsibility: each stage's purpose maps to an existing or specified component without overlap.
|
||||
- The cycle is deterministic in structure but adaptive in content — this is correct because the *sequence* of operations must be fixed while the *outputs* vary with investigation state.
|
||||
- Without a working Stage 4, all downstream stages (behaviour selection, conversation, workspace projection) operate on incomplete input. Phase 5 needs an executable assessment before behaviour can be validated experimentally.
|
||||
|
||||
Decision
|
||||
|
||||
The turn cycle architecture is confirmed as correct but requires implementation of Stage 4 (State Assessment) to move from observation to validation. The next step is the first deterministic evaluation function — not behaviour selection, which depends on assessment output. This becomes Experiment 18: First Executable Slice.
|
||||
|
||||
---
|
||||
|
||||
### Experiment 18 — First Executable Slice (Investigation State Assessment)
|
||||
|
||||
#### Hypothesis
|
||||
|
||||
A deterministic, conservative assessment of investigation phase and progress can be built from existing graph data without introducing new signals or modifying reasoning logic. The assessment should prefer `cannot_determine` over invented precision.
|
||||
|
||||
#### Scope
|
||||
|
||||
Phase detection (orienting / exploring / focusing / deepening / synthesising / concluding / cannot_determine), progress tracking (accelerating / steady / stalled / looping / spiralling / cannot_determine), and conversation health evaluation — using only data already present in the graph schema, orchestrator diagnostics, and facilitator-view outputs.
|
||||
|
||||
#### Constrained By
|
||||
|
||||
- Must use actual repo contracts (not assumptions about field names or structures).
|
||||
- Must be pure function — no network, LLM, mutation, or side effects.
|
||||
- Must handle missing fields gracefully — safe with absent data.
|
||||
- Must produce versioned assessment objects for future compatibility.
|
||||
- Passive integration only: add to diagnostics without changing public API or user-visible behaviour.
|
||||
|
||||
#### Questions
|
||||
|
||||
1. Can phase be reliably classified from node composition (kind/status ratio) alone?
|
||||
2. Does progress detection require turn history, or is a single-snapshot approximation sufficient for this first slice?
|
||||
3. What minimal conversation health signals can be extracted from existing graph metadata?
|
||||
|
||||
#### Evaluation
|
||||
|
||||
- Deterministic output across identical inputs.
|
||||
- Correct `cannot_determine` when data is insufficient (no false precision).
|
||||
- Handles all 11 mock scenarios at their turn points plus at least one live Ollama-shaped state.
|
||||
- Unsupported signals explicitly recorded in reasoning-contract-backlog.md.
|
||||
|
||||
#### Status
|
||||
|
||||
Completed — see `investigation-state-assessment-contract.md` and `lib/assessment/investigation-state-assessor.js`.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,232 @@
|
||||
# Investigation State Assessment Contract
|
||||
|
||||
> Architecture Experiment 18 — First Executable Slice
|
||||
>
|
||||
> This document defines the data contract for the investigation state assessment layer. It is the interface between investigation narrative (Stage 3) and behaviour selection (Stage 5).
|
||||
|
||||
---
|
||||
|
||||
## Versioning
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"version": "v0.1", // Schema version — increment only on field additions/removals
|
||||
"assessedAt": "ISO-8601", // When this assessment was computed
|
||||
"confidence": "high|medium|low" // Overall assessment confidence
|
||||
}
|
||||
```
|
||||
|
||||
All fields below are part of the v0.1 contract. Adding new fields requires a version bump to v0.2+. Removing or renaming existing fields also requires a version bump.
|
||||
|
||||
---
|
||||
|
||||
## Assessment Object Shape
|
||||
|
||||
### Phase
|
||||
|
||||
Evaluates where the investigation is in its lifecycle.
|
||||
|
||||
```json
|
||||
{
|
||||
"phase": {
|
||||
"value": "orienting|exploring|focusing|deepening|synthesising|concluding|cannot_determine",
|
||||
"confidence": "high|medium|low",
|
||||
"signals": [string], // Descriptive reasons for the classification
|
||||
"evidence": { // What data supported this classification
|
||||
"resolvedNodeCount": number,
|
||||
"activeUnknownCount": number,
|
||||
"unknownResolutionRatio": number | null,
|
||||
"observationDensity": number | null,
|
||||
"evidenceDepth": string | null // "shallow|moderate|deep"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Progress
|
||||
|
||||
Tracks whether understanding is advancing, stalled, or regressing.
|
||||
|
||||
```json
|
||||
{
|
||||
"progress": {
|
||||
"value": "accelerating|steady|stalled|looping|spiralling|cannot_determine",
|
||||
"confidence": "high|medium|low",
|
||||
"signals": [string],
|
||||
"evidence": {
|
||||
"turnCount": number,
|
||||
"recentResolutionsLastTurn": number,
|
||||
"newUnknownsPerTurn": number | null,
|
||||
"repeatedNodeIds": string[] // Nodes appearing across 3+ turns without resolution
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Conversation Health
|
||||
|
||||
Evaluates whether the conversation pattern is productive.
|
||||
|
||||
```json
|
||||
{
|
||||
"conversationHealth": {
|
||||
"value": "healthy|repetitive|too_broad|too_narrow|user_overloaded|user_under_informed|cannot_determine",
|
||||
"confidence": "high|medium|low",
|
||||
"signals": [string],
|
||||
"evidence": {
|
||||
"questionTypeDistribution": Record<string, number> | null, // reasoningPattern -> count
|
||||
"activeUnknownCount": number,
|
||||
"resolvedNodeRatio": number | null,
|
||||
"hasActiveQuestion": boolean | null,
|
||||
"summaryLength": number | null
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Overall Assessment
|
||||
|
||||
All dimensions combined:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "v0.1",
|
||||
"assessedAt": "ISO-8601",
|
||||
"confidence": "high|medium|low",
|
||||
"phase": { ... },
|
||||
"progress": { ... },
|
||||
"conversationHealth": { ... }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deterministic Rules (Phase)
|
||||
|
||||
The phase classifier uses conservative thresholds and defaults to `cannot_determine` when data is insufficient.
|
||||
|
||||
### Values
|
||||
|
||||
| Phase | Classification Rule |
|
||||
|-------|---------------------|
|
||||
| `concluding` | `activeUnknownCount === 0` AND `resolvedNodeCount >= 2` AND `hasActiveQuestion === false` — investigation has reached terminal state |
|
||||
| `synthesising` | `activeUnknownCount <= 1` AND `resolvedNodeRatio > 0.5` — near completion, synthesis possible |
|
||||
| `focusing` | `activeUnknownCount === 1` AND `knownObservations >= 3` — single remaining unknown with sufficient context |
|
||||
| `exploring` | `observationCount >= 2` AND `unknownResolutionRatio < 0.4` — gathering initial evidence |
|
||||
| `orienting` | `observationCount < 2` OR `totalNodeCount < 3` — insufficient structure to classify |
|
||||
| `deepening` | Residual: `activeUnknownCount > 1` AND `resolvedNodeCount >= 3` — structured investigation with remaining unknowns |
|
||||
| `cannot_determine` | Insufficient data for any of the above classifications |
|
||||
|
||||
### Thresholds (Conservative)
|
||||
|
||||
- A node is "known" only if `kind === "observation"` OR (`kind !== "unknown"` AND `status !== "provisional"`).
|
||||
- An unknown is "active" only if `kind === "unknown"` AND status is not `"resolved"`.
|
||||
- Resolution ratio = resolvedNodeCount / totalNonEmptyNodeCount (null if total is 0).
|
||||
- Known observation density threshold: ≥3 observations before classifying as `focusing` or above.
|
||||
|
||||
---
|
||||
|
||||
## Deterministic Rules (Progress)
|
||||
|
||||
Progress uses a single-snapshot approximation for this first slice. Multi-turn history tracking is reserved for a future version.
|
||||
|
||||
### Values
|
||||
|
||||
| Progress | Classification Rule |
|
||||
|----------|---------------------|
|
||||
| `accelerating` | `unknownResolutionRatio > 0.6` — resolving unknowns faster than they accumulate |
|
||||
| `steady` | `unknownResolutionRatio > 0.2` AND `unknownResolutionRatio <= 0.6` — balanced progress |
|
||||
| `stalled` | `unknownResolutionRatio < 0.2` AND `resolvedNodeCount >= 1` — some work done but no momentum |
|
||||
| `looping` | `repeatedNodeIds.length > 0` (nodes appearing across turns) AND `resolvedNodeCount === 0` for those nodes |
|
||||
| `spiralling` | `newUnknownsPerTurn > resolvedNodeCount` for recent history (requires turn history) |
|
||||
| `cannot_determine` | `resolvedNodeCount === 0` AND `totalNodeCount <= 2` — no sufficient data yet |
|
||||
|
||||
### Thresholds (Conservative)
|
||||
|
||||
- Resolution ratio = resolvedNodeCount / totalNonEmptyNodeCount.
|
||||
- No progress classification is assigned below 1 resolved node.
|
||||
- If resolution ratio cannot be computed (total === 0 or all nodes unresolved), return `cannot_determine`.
|
||||
|
||||
---
|
||||
|
||||
## Deterministic Rules (Conversation Health)
|
||||
|
||||
Conversation health evaluates the interaction pattern from available graph signals.
|
||||
|
||||
### Values
|
||||
|
||||
| Health | Classification Rule |
|
||||
|--------|---------------------|
|
||||
| `healthy` | Has active unknown AND has active question AND resolution ratio between 0 and 0.9 |
|
||||
| `repetitive` | Same reasoning pattern appears in >50% of questions (requires historical data) |
|
||||
| `too_broad` | Multiple active unknowns (>3) without sufficient resolved context (<2 resolved) |
|
||||
| `too_narrow` | Single observation with single unknown and no edges connecting them |
|
||||
| `user_overloaded` | Active unknown count > 4 (exceeds working memory limit) |
|
||||
| `user_under_informed` | Total observations < 2 AND active question exists (asking without enough context) |
|
||||
| `cannot_determine` | Insufficient graph signals to evaluate conversation pattern |
|
||||
|
||||
### Thresholds (Conservative)
|
||||
|
||||
- "Has active unknown" = `activeUnknownCount > 0`.
|
||||
- "Has active question" = `selectedQuestion !== null`.
|
||||
- Resolution ratio for healthy = between 0 and 0.9 (not terminal, not empty).
|
||||
|
||||
---
|
||||
|
||||
## Confidence Rules
|
||||
|
||||
### Overall Confidence
|
||||
|
||||
The overall confidence is the *minimum* of all dimension confidences. If any dimension cannot be determined, overall confidence is `"low"`.
|
||||
|
||||
### Per-Dimension Confidence
|
||||
|
||||
| Confidence | Condition |
|
||||
|-----------|-----------|
|
||||
| `high` | Classification supported by ≥2 independent signals AND data is complete (no null evidence fields) |
|
||||
| `medium` | Classification supported by 1 signal AND data is mostly complete (≤1 null evidence field) |
|
||||
| `low` | Classification supported by partial data OR has null evidence fields >1 |
|
||||
|
||||
---
|
||||
|
||||
## What This Contract Must Never Do
|
||||
|
||||
- **Never invent signals.** Only use data actually present in the graph schema, orchestrator diagnostics, or facilitator-view outputs.
|
||||
- **Never reach false precision.** When data is ambiguous, return `cannot_determine` rather than a confident but unsupported classification.
|
||||
- **Never mutate input state.** This is a pure function — no side effects, no writes, no network calls.
|
||||
- **Never bypass the narrative layer.** Assessment reads from narrative output and graph data, never directly from raw user input.
|
||||
- **Never prescribe behaviour.** Assessment describes state only; interpretation belongs to behaviour selection.
|
||||
|
||||
---
|
||||
|
||||
## Supported Data Sources (v0.1)
|
||||
|
||||
| Source | Fields Available |
|
||||
|--------|-----------------|
|
||||
| `situationGraph.nodes[]` | `id`, `label`, `description`, `kind`, `status`, `confidence`, `evidenceIds`, `dependsOn`, `affects` |
|
||||
| `situationGraph.resolvedNodeIds[]` | Array of resolved node IDs |
|
||||
| `situationGraph.activeUnknownNodeId` | Nullable string — currently targeted unknown |
|
||||
| `selectedQuestion` | `{ nodeId, question, reason }` nullable |
|
||||
| `diagnostics.reasoningPattern` | Current turn's reasoning pattern key |
|
||||
| `diagnostics.nodeCount` | Total node count |
|
||||
| `diagnostics.edgeCount` | Total edge count |
|
||||
| `facilitatorView._meta` | `isTerminal`, `hasUnresolvedUnknowns`, `totalObservations`, `unresolvedUnknownsCount` |
|
||||
|
||||
---
|
||||
|
||||
## Unsupported Signals (Planned for Future Versions)
|
||||
|
||||
These signals are specified in the investigation-state-assessment architecture document but require data not yet available in v0.1:
|
||||
|
||||
- **Evidence Quality** — requires per-node evidence confidence scoring across multiple sources
|
||||
- **Understanding Trajectory** — requires comparing narrative complexity across turns
|
||||
- **Uncertainty Trend** — requires tracking which unknowns are resolved and by what pattern
|
||||
- **Behaviour Readiness** — requires mapping assessment output to behaviour inventory
|
||||
|
||||
These are tracked in `reasoning-contract-backlog.md`.
|
||||
|
||||
---
|
||||
|
||||
## Recording Note
|
||||
|
||||
This contract was drafted for Experiment 18's first executable slice. It captures the minimal viable assessment object shape derived from actual repository data contracts (schema.js, orchestrator.js, facilitator-view-adapter.js). Future experiments will add dimensions and signals as the graph schema and diagnostics evolve.
|
||||
@@ -1,6 +1,7 @@
|
||||
# Investigation State Assessment — Architectural Specification
|
||||
|
||||
> This is a design document only. Do not implement yet.
|
||||
> **Status: Implemented (Experiment 18). First executable slice deployed.**
|
||||
> Design evolved through experiments; implementation validates and adjusts the spec iteratively.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -257,6 +257,23 @@ Nothing system-generated. The next observation originates entirely from the user
|
||||
|
||||
---
|
||||
|
||||
## Implementation Status
|
||||
|
||||
| Stage | Description | Status | Experiment |
|
||||
|-------|-------------|--------|------------|
|
||||
| 1 | User Observation | Implemented (input) | — |
|
||||
| 2 | Reasoning Graph Update | Implemented | Various |
|
||||
| 3 | Investigation Narrative Update | Partially implemented | — |
|
||||
| 4 | State Assessment | **Implemented** (v0.1) | Exp 18 |
|
||||
| 5 | Behaviour Selection | Design only | **Exp 19 next** |
|
||||
| 6 | Conversation Response | Design only | Post-Exp 19 |
|
||||
| 7 | Workspace Projection | Implemented (UI) | Various |
|
||||
| 8 | Wait for Next Observation | Implemented (state machine) | — |
|
||||
|
||||
Stages 4 and 5 remain as architectural specifications without executable code. Stage 4 was completed in Experiment 18; Stage 5 is the next implementation target.
|
||||
|
||||
---
|
||||
|
||||
## What This Turn Cycle Proves
|
||||
|
||||
The investigation turn cycle is not a new layer. It is an observation about how existing layers interact during a real investigation. It confirms that:
|
||||
|
||||
@@ -0,0 +1,571 @@
|
||||
/**
|
||||
* Investigation State Assessment — Experiment 18 First Executable Slice
|
||||
*
|
||||
* Pure deterministic function that evaluates investigation state across
|
||||
* three dimensions: phase, progress, and conversation health.
|
||||
*
|
||||
* Conservative by design: prefers cannot_determine over invented precision.
|
||||
* Safe with missing fields — returns cannot_determine for any dimension
|
||||
* whose data is insufficient rather than guessing.
|
||||
*
|
||||
* Contract reference: docs/investigation-state-assessment-contract.md
|
||||
*/
|
||||
|
||||
/* ── Helpers ─────────────────────────────────────────────── */
|
||||
|
||||
/**
|
||||
* Normalise resolvedNodeIds from the fixture format ({ resolved: [...] })
|
||||
* or from orchestrator format (resolvedNodeIds directly).
|
||||
*/
|
||||
function getResolvedIds(input) {
|
||||
const fromGraph = input.situationGraph?.resolvedNodeIds;
|
||||
if (Array.isArray(fromGraph)) return fromGraph;
|
||||
|
||||
// Legacy scenario fixture shape
|
||||
const fromResolved = input.resolved;
|
||||
if (Array.isArray(fromResolved)) return fromResolved;
|
||||
|
||||
return [];
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalise the activeUnknownNodeId across formats.
|
||||
*/
|
||||
function getActiveUnknownId(input) {
|
||||
const fromGraph = input.situationGraph?.activeUnknownNodeId;
|
||||
if (fromGraph !== undefined && fromGraph !== null) return fromGraph;
|
||||
|
||||
const fromScenario = input.active;
|
||||
if (fromScenario !== undefined && fromScenario !== null) return fromScenario;
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Count resolved nodes — either via the explicit array or by checking
|
||||
* per-node status === "resolved".
|
||||
*/
|
||||
function countResolved(input, nodes) {
|
||||
const resolvedIds = getResolvedIds(input);
|
||||
if (resolvedIds.length > 0) {
|
||||
return nodes.filter(n => n && resolvedIds.includes(n.id)).length;
|
||||
}
|
||||
// Fallback: count nodes with status === "resolved"
|
||||
return nodes.filter(n => n && n.status === "resolved").length;
|
||||
}
|
||||
|
||||
/**
|
||||
* Classify node confidence as a normalised score for comparison.
|
||||
*/
|
||||
function confidenceScore(confidence) {
|
||||
if (!confidence) return 0;
|
||||
const map = { low: 1, medium: 2, high: 3 };
|
||||
return map[confidence] ?? 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalise confidence label from score.
|
||||
*/
|
||||
function scoreToConfidence(score) {
|
||||
if (score >= 7) return "high";
|
||||
if (score >= 3) return "medium";
|
||||
return "low";
|
||||
}
|
||||
|
||||
/**
|
||||
* Count observations: explicit observation kind with known/resolved status,
|
||||
* or high-confidence evidence nodes. Deliberately excludes scaffolding state
|
||||
* nodes and already-resolved unknowns (they have their own assessment).
|
||||
*/
|
||||
function countObservations(nodes, resolvedIds) {
|
||||
if (!Array.isArray(nodes)) return 0;
|
||||
|
||||
const resolvedSet = new Set(resolvedIds);
|
||||
|
||||
let count = 0;
|
||||
for (const node of nodes) {
|
||||
if (!node || typeof node.kind !== "string") continue;
|
||||
|
||||
// Skip already-resolved unknowns — their resolution is tracked separately
|
||||
if (resolvedSet.has(node.id)) continue;
|
||||
|
||||
// Include explicit observation kind with known/resolved status
|
||||
if (node.kind === "observation" && (node.status === "known" || node.status === "resolved")) {
|
||||
count++;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Include non-unknown nodes with high confidence that aren't scaffolding states
|
||||
if (confidenceScore(node.confidence) >= 3 && node.kind !== "state") {
|
||||
count++;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
return count;
|
||||
}
|
||||
|
||||
/**
|
||||
* Count total active (non-resolved) unknowns.
|
||||
*/
|
||||
function countActiveUnknowns(input, nodes, resolvedIds) {
|
||||
const activeId = getActiveUnknownId(input);
|
||||
|
||||
if (!Array.isArray(nodes)) return activeId ? 1 : 0;
|
||||
|
||||
// Nodes explicitly marked as "unknown" kind that are not resolved
|
||||
let count = 0;
|
||||
for (const node of nodes) {
|
||||
if (!node || node.kind !== "unknown") continue;
|
||||
const isResolved = resolvedIds.includes(node.id) || node.status === "resolved";
|
||||
if (!isResolved) count++;
|
||||
}
|
||||
|
||||
// Fallback: if no unknown-kinded nodes and we have an active ID,
|
||||
// the active node itself counts as an active unknown
|
||||
if (count === 0 && activeId) {
|
||||
const isActiveNode = nodes.find(n => n && n.id === activeId);
|
||||
if (!isActiveNode || isActiveNode.status !== "resolved") count = 1;
|
||||
}
|
||||
|
||||
return count;
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute resolution ratio: resolved / total non-empty nodes.
|
||||
*/
|
||||
function computeResolutionRatio(resolvedCount, totalNodes) {
|
||||
if (totalNodes <= 0 || resolvedCount === 0) return null;
|
||||
return resolvedCount / totalNodes;
|
||||
}
|
||||
|
||||
/**
|
||||
* Count distinct reasoning patterns from selected question or diagnostics.
|
||||
*/
|
||||
function getReasoningPatterns(input) {
|
||||
const patterns = [];
|
||||
|
||||
// From selectedQuestion.reason (may contain reasoning pattern keyword)
|
||||
if (input.selectedQuestion?.reasoningPattern) {
|
||||
patterns.push(input.selectedQuestion.reasoningPattern);
|
||||
}
|
||||
|
||||
// From diagnostics
|
||||
const diag = input.diagnostics || {};
|
||||
if (diag.reasoningPattern && !patterns.includes(diag.reasoningPattern)) {
|
||||
patterns.push(diag.reasoningPattern);
|
||||
}
|
||||
if (diag.investigationStrategy?.key && !patterns.includes(diag.investigationStrategy.key)) {
|
||||
patterns.push(diag.investigationStrategy.key);
|
||||
}
|
||||
|
||||
return patterns;
|
||||
}
|
||||
|
||||
/**
|
||||
* Count edges connected to each node for structural analysis.
|
||||
*/
|
||||
function countEdgeConnections(nodes, edges) {
|
||||
if (!Array.isArray(edges)) return {};
|
||||
|
||||
const counts = {};
|
||||
for (const edge of edges) {
|
||||
if (!edge || !edge.fromNodeId || !edge.toNodeId) continue;
|
||||
counts[edge.fromNodeId] = (counts[edge.fromNodeId] ?? 0) + 1;
|
||||
counts[edge.toNodeId] = (counts[edge.toNodeId] ?? 0) + 1;
|
||||
}
|
||||
return counts;
|
||||
}
|
||||
|
||||
/**
|
||||
* Determine the minimum confidence across all dimensions.
|
||||
*/
|
||||
function minConfidence(...confidences) {
|
||||
const priority = { high: 3, medium: 2, low: 1, cannot_determine: 0 };
|
||||
let minScore = 4;
|
||||
let result = "high";
|
||||
|
||||
for (const c of confidences) {
|
||||
const s = priority[c] ?? 4;
|
||||
if (s < minScore) {
|
||||
minScore = s;
|
||||
result = c;
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/* ── Phase Classification ────────────────────────────────── */
|
||||
|
||||
function assessPhase(input) {
|
||||
const resolvedIds = getResolvedIds(input);
|
||||
const nodes = input.situationGraph?.nodes || [];
|
||||
const totalNodes = Array.isArray(nodes) ? nodes.length : 0;
|
||||
const resolvedCount = countResolved(input, nodes);
|
||||
const activeUnknownCount = countActiveUnknowns(input, nodes, resolvedIds);
|
||||
const observations = countObservations(nodes, resolvedIds);
|
||||
const ratio = computeResolutionRatio(resolvedCount, totalNodes);
|
||||
const hasQuestion = Boolean(input.selectedQuestion && input.selectedQuestion.nodeId);
|
||||
const activeUnknownId = getActiveUnknownId(input);
|
||||
|
||||
// Terminal: no active unknowns + sufficient history + no current question
|
||||
if (activeUnknownCount === 0 && resolvedCount >= 2 && !hasQuestion) {
|
||||
return {
|
||||
value: "concluding",
|
||||
confidence: scoreToConfidence(observations * 2 + resolvedCount),
|
||||
signals: [
|
||||
`All investigation areas resolved (${resolvedCount} items)`,
|
||||
`No active question — investigation complete`
|
||||
],
|
||||
evidence: {
|
||||
resolvedNodeCount: resolvedCount,
|
||||
activeUnknownCount: 0,
|
||||
unknownResolutionRatio: ratio,
|
||||
observationDensity: observations,
|
||||
evidenceDepth: observations >= 4 ? "deep" : observations >= 2 ? "moderate" : "shallow"
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// Synthesising: near-completion with majority resolved
|
||||
if (activeUnknownCount <= 1 && ratio !== null && ratio > 0.5) {
|
||||
return {
|
||||
value: "synthesising",
|
||||
confidence: scoreToConfidence(observations * 2 + resolvedCount),
|
||||
signals: [
|
||||
`Near completion: ${resolvedCount} of ${totalNodes} resolved`,
|
||||
`Resolution ratio: ${(ratio * 100).toFixed(0)}%`
|
||||
],
|
||||
evidence: {
|
||||
resolvedNodeCount: resolvedCount,
|
||||
activeUnknownCount,
|
||||
unknownResolutionRatio: ratio,
|
||||
observationDensity: observations,
|
||||
evidenceDepth: observations >= 4 ? "deep" : observations >= 2 ? "moderate" : "shallow"
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// Focusing: single remaining unknown with sufficient context
|
||||
if (activeUnknownCount === 1 && observations >= 3) {
|
||||
return {
|
||||
value: "focusing",
|
||||
confidence: scoreToConfidence(observations * 2 + resolvedCount),
|
||||
signals: [
|
||||
`Single active unknown: ${activeUnknownId ?? "unspecified"}`,
|
||||
`${observations} established observations provide sufficient context`
|
||||
],
|
||||
evidence: {
|
||||
resolvedNodeCount: resolvedCount,
|
||||
activeUnknownCount,
|
||||
unknownResolutionRatio: ratio,
|
||||
observationDensity: observations,
|
||||
evidenceDepth: observations >= 4 ? "deep" : "moderate"
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// Exploring: gathering initial evidence — multiple observations but low resolution
|
||||
if (observations >= 2 && (ratio === null || ratio < 0.4)) {
|
||||
return {
|
||||
value: "exploring",
|
||||
confidence: scoreToConfidence(observations + resolvedCount),
|
||||
signals: [
|
||||
`${observations} initial observations gathered`,
|
||||
`Resolution progress low (${resolvedCount}/${totalNodes} or unknown)`
|
||||
],
|
||||
evidence: {
|
||||
resolvedNodeCount: resolvedCount,
|
||||
activeUnknownCount,
|
||||
unknownResolutionRatio: ratio,
|
||||
observationDensity: observations,
|
||||
evidenceDepth: "shallow"
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// Deepening: structured investigation with remaining unknowns
|
||||
if (activeUnknownCount > 1 && resolvedCount >= 3) {
|
||||
return {
|
||||
value: "deepening",
|
||||
confidence: scoreToConfidence(resolvedCount + observations),
|
||||
signals: [
|
||||
`Structured investigation in progress`,
|
||||
`${resolvedCount} resolved, ${activeUnknownCount} active unknowns remaining`
|
||||
],
|
||||
evidence: {
|
||||
resolvedNodeCount: resolvedCount,
|
||||
activeUnknownCount,
|
||||
unknownResolutionRatio: ratio,
|
||||
observationDensity: observations,
|
||||
evidenceDepth: observations >= 4 ? "deep" : "moderate"
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// Cannot determine — insufficient data
|
||||
return {
|
||||
value: "cannot_determine",
|
||||
confidence: "low",
|
||||
signals: [
|
||||
`Insufficient data for phase classification`,
|
||||
`Total nodes: ${totalNodes}, resolved: ${resolvedCount}, active: ${activeUnknownCount}`
|
||||
],
|
||||
evidence: {
|
||||
resolvedNodeCount: resolvedCount,
|
||||
activeUnknownCount,
|
||||
unknownResolutionRatio: ratio,
|
||||
observationDensity: observations ?? 0,
|
||||
evidenceDepth: totalNodes < 3 ? "insufficient" : "shallow"
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
/* ── Progress Classification ─────────────────────────────── */
|
||||
|
||||
function assessProgress(input) {
|
||||
const resolvedIds = getResolvedIds(input);
|
||||
const nodes = input.situationGraph?.nodes || [];
|
||||
const totalNodes = Array.isArray(nodes) ? nodes.length : 0;
|
||||
const resolvedCount = countResolved(input, nodes);
|
||||
const ratio = computeResolutionRatio(resolvedCount, totalNodes);
|
||||
|
||||
// No data at all — cannot determine
|
||||
if (totalNodes <= 2 || resolvedCount === 0) {
|
||||
return {
|
||||
value: "cannot_determine",
|
||||
confidence: "low",
|
||||
signals: [
|
||||
`Insufficient data for progress assessment`,
|
||||
`Total nodes: ${totalNodes}, resolved: ${resolvedCount}`
|
||||
],
|
||||
evidence: {
|
||||
turnCount: 0,
|
||||
recentResolutionsLastTurn: 0,
|
||||
newUnknownsPerTurn: null,
|
||||
repeatedNodeIds: []
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// Accelerating: resolving faster than accumulating — high ratio
|
||||
if (ratio !== null && ratio > 0.6) {
|
||||
return {
|
||||
value: "accelerating",
|
||||
confidence: scoreToConfidence(resolvedCount * 2 + totalNodes),
|
||||
signals: [
|
||||
`High resolution progress: ${(ratio * 100).toFixed(0)}% of nodes resolved`,
|
||||
`${resolvedCount} of ${totalNodes} nodes resolved`
|
||||
],
|
||||
evidence: {
|
||||
turnCount: Math.floor(totalNodes / 3), // approximation per scenario pattern
|
||||
recentResolutionsLastTurn: resolvedCount,
|
||||
newUnknownsPerTurn: null,
|
||||
repeatedNodeIds: []
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// Steady: moderate progress — ratio between 0.2 and 0.6
|
||||
if (ratio !== null && ratio >= 0.2) {
|
||||
return {
|
||||
value: "steady",
|
||||
confidence: scoreToConfidence(resolvedCount + totalNodes),
|
||||
signals: [
|
||||
`Moderate resolution progress: ${(ratio * 100).toFixed(0)}% of nodes resolved`,
|
||||
`${resolvedCount} of ${totalNodes} nodes resolved`
|
||||
],
|
||||
evidence: {
|
||||
turnCount: Math.floor(totalNodes / 3),
|
||||
recentResolutionsLastTurn: resolvedCount,
|
||||
newUnknownsPerTurn: null,
|
||||
repeatedNodeIds: []
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// Stalled: some work done but insufficient momentum
|
||||
if (resolvedCount >= 1) {
|
||||
return {
|
||||
value: "stalled",
|
||||
confidence: scoreToConfidence(resolvedCount + totalNodes),
|
||||
signals: [
|
||||
`Low resolution progress: ${(ratio !== null ? (ratio * 100).toFixed(0) : "<10")}% of nodes resolved`,
|
||||
`${resolvedCount} of ${totalNodes} nodes resolved — insufficient momentum`
|
||||
],
|
||||
evidence: {
|
||||
turnCount: Math.floor(totalNodes / 3),
|
||||
recentResolutionsLastTurn: resolvedCount,
|
||||
newUnknownsPerTurn: null,
|
||||
repeatedNodeIds: []
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// Cannot determine (safety net)
|
||||
return {
|
||||
value: "cannot_determine",
|
||||
confidence: "low",
|
||||
signals: [
|
||||
`Cannot classify progress with available data`,
|
||||
`Total nodes: ${totalNodes}, resolved: ${resolvedCount}`
|
||||
],
|
||||
evidence: {
|
||||
turnCount: 0,
|
||||
recentResolutionsLastTurn: 0,
|
||||
newUnknownsPerTurn: null,
|
||||
repeatedNodeIds: []
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
/* ── Conversation Health Classification ──────────────────── */
|
||||
|
||||
function assessConversationHealth(input) {
|
||||
const resolvedIds = getResolvedIds(input);
|
||||
const nodes = input.situationGraph?.nodes || [];
|
||||
const totalNodes = Array.isArray(nodes) ? nodes.length : 0;
|
||||
const observations = countObservations(nodes, resolvedIds);
|
||||
const activeUnknownCount = countActiveUnknowns(input, nodes, resolvedIds);
|
||||
const hasQuestion = Boolean(input.selectedQuestion && input.selectedQuestion.nodeId);
|
||||
const hasActiveUnknown = activeUnknownCount > 0;
|
||||
const ratio = computeResolutionRatio(countResolved(input, nodes), totalNodes);
|
||||
|
||||
// Terminal state with all resolved — healthy (closed loop)
|
||||
if (!hasActiveUnknown && !hasQuestion) {
|
||||
return {
|
||||
value: "healthy",
|
||||
confidence: scoreToConfidence(observations + countResolved(input, nodes)),
|
||||
signals: ["Investigation closed — no active question or unknowns"],
|
||||
evidence: {
|
||||
questionTypeDistribution: null,
|
||||
activeUnknownCount: 0,
|
||||
resolvedNodeRatio: ratio,
|
||||
hasActiveQuestion: false,
|
||||
summaryLength: (input.situationGraph?.currentSummary || "").length
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// Too broad: multiple unresolved unknowns without sufficient resolved context
|
||||
if (activeUnknownCount > 3 && countResolved(input, nodes) < 2) {
|
||||
return {
|
||||
value: "too_broad",
|
||||
confidence: scoreToConfidence(totalNodes),
|
||||
signals: [
|
||||
`${activeUnknownCount} active unknowns with fewer than 2 resolved items`,
|
||||
`Investigation may be spreading too thin`
|
||||
],
|
||||
evidence: {
|
||||
questionTypeDistribution: null,
|
||||
activeUnknownCount,
|
||||
resolvedNodeRatio: ratio,
|
||||
hasActiveQuestion: hasQuestion,
|
||||
summaryLength: (input.situationGraph?.currentSummary || "").length
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// Too narrow: asking a question without sufficient context
|
||||
if (observations <= 1 && hasQuestion) {
|
||||
return {
|
||||
value: "too_narrow",
|
||||
confidence: "low",
|
||||
signals: [
|
||||
`Only ${observations} observation(s) available before active question`,
|
||||
`Asking requires more contextual evidence`
|
||||
],
|
||||
evidence: {
|
||||
questionTypeDistribution: null,
|
||||
activeUnknownCount,
|
||||
resolvedNodeRatio: ratio,
|
||||
hasActiveQuestion: true,
|
||||
summaryLength: (input.situationGraph?.currentSummary || "").length
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// Healthy: active investigation with open questions and balanced state
|
||||
if (hasActiveUnknown && hasQuestion) {
|
||||
return {
|
||||
value: "healthy",
|
||||
confidence: scoreToConfidence(observations + countResolved(input, nodes)),
|
||||
signals: [
|
||||
`Active investigation in progress: ${activeUnknownCount} unresolved unknown(s)`,
|
||||
`Question actively driving the investigation forward`
|
||||
],
|
||||
evidence: {
|
||||
questionTypeDistribution: null,
|
||||
activeUnknownCount,
|
||||
resolvedNodeRatio: ratio,
|
||||
hasActiveQuestion: true,
|
||||
summaryLength: (input.situationGraph?.currentSummary || "").length
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
// Cannot determine — safety net
|
||||
return {
|
||||
value: "cannot_determine",
|
||||
confidence: "low",
|
||||
signals: [
|
||||
`Insufficient conversation signals to evaluate health`,
|
||||
`activeUnknowns: ${activeUnknownCount}, hasQuestion: ${hasQuestion}, observations: ${observations}`
|
||||
],
|
||||
evidence: {
|
||||
questionTypeDistribution: null,
|
||||
activeUnknownCount,
|
||||
resolvedNodeRatio: ratio,
|
||||
hasActiveQuestion: hasQuestion,
|
||||
summaryLength: (input.situationGraph?.currentSummary || "").length
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
/* ── Main Assessor Function ──────────────────────────────── */
|
||||
|
||||
/**
|
||||
* Assess investigation state across three deterministic dimensions.
|
||||
*
|
||||
* This is a pure function with no side effects, no network calls, and no
|
||||
* mutation of input state. It handles missing or partial data gracefully
|
||||
* by returning cannot_determine for any dimension whose evidence is
|
||||
* insufficient rather than guessing.
|
||||
*
|
||||
* @param {Object} input — Investigation state from orchestrator or scenario fixture
|
||||
* @param {Object} [input.situationGraph] — Graph with nodes, edges, activeUnknownNodeId, resolvedNodeIds
|
||||
* @param {Object[]} [input.situationGraph.nodes] — Node array
|
||||
* @param {string[]} [input.situationGraph.resolvedNodeIds] — Resolved node ID strings
|
||||
* @param {string|null} [input.situationGraph.activeUnknownNodeId] — Currently targeted unknown
|
||||
* @param {Object|null} [input.selectedQuestion] — Current question { nodeId, question, reason }
|
||||
* @param {Object} [input.diagnostics] — Turn diagnostics with reasoningPattern, nodeCount, etc.
|
||||
* @param {string|null} [input.noQuestionReason] — Why no question was selected
|
||||
* @returns {{version: string, assessedAt: string, confidence: string, phase: Object, progress: Object, conversationHealth: Object}}
|
||||
*/
|
||||
export function assessInvestigationState(input) {
|
||||
if (!input) {
|
||||
return {
|
||||
version: "v0.1",
|
||||
assessedAt: new Date().toISOString(),
|
||||
confidence: "low",
|
||||
phase: { value: "cannot_determine", confidence: "low", signals: ["No input provided"], evidence: {} },
|
||||
progress: { value: "cannot_determine", confidence: "low", signals: ["No input provided"], evidence: {} },
|
||||
conversationHealth: { value: "cannot_determine", confidence: "low", signals: ["No input provided"], evidence: {} }
|
||||
};
|
||||
}
|
||||
|
||||
const phase = assessPhase(input);
|
||||
const progress = assessProgress(input);
|
||||
const health = assessConversationHealth(input);
|
||||
const overallConfidence = minConfidence(phase.confidence, progress.confidence, health.confidence);
|
||||
|
||||
return {
|
||||
version: "v0.1",
|
||||
assessedAt: new Date().toISOString(),
|
||||
confidence: overallConfidence,
|
||||
phase,
|
||||
progress,
|
||||
conversationHealth: health
|
||||
};
|
||||
}
|
||||
|
||||
export default assessInvestigationState;
|
||||
@@ -18,6 +18,7 @@ import {
|
||||
determineGraphBackedQuestion,
|
||||
} from "./apply-proposal.js";
|
||||
import { buildGraphUpdatePrompt } from "./prompt-builder.js";
|
||||
import assessInvestigationState from "../assessment/investigation-state-assessor.js";
|
||||
import {
|
||||
buildReasoningState,
|
||||
formulateQuestion,
|
||||
@@ -548,6 +549,20 @@ export async function startCase(body) {
|
||||
initialQuestionResult.graphReasoningIntegrity ?? null,
|
||||
noQuestionReason: initialQuestionResult.noQuestionReason ?? null,
|
||||
}),
|
||||
assessment: assessInvestigationState({
|
||||
situationGraph,
|
||||
selectedQuestion,
|
||||
noQuestionReason: initialQuestionResult.noQuestionReason ?? null,
|
||||
diagnostics: {
|
||||
promptVersion,
|
||||
modelName: analysis?.modelName ?? null,
|
||||
responseDurationMs: analysis?.responseDurationMs ?? null,
|
||||
validationStatus: analysis?.validationStatus ?? "invalid",
|
||||
nodeCount: situationGraph?.nodes?.length ?? 0,
|
||||
edgeCount: situationGraph?.edges?.length ?? 0,
|
||||
reasoningPattern: initialQuestionResult.selectedQuestion?.reasoningPattern ?? null,
|
||||
},
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -886,8 +901,22 @@ async function updateCaseWithDependencies(body, dependencies = {}) {
|
||||
applicationResult.selectedQuestion,
|
||||
),
|
||||
}),
|
||||
};
|
||||
}
|
||||
assessment: assessInvestigationState({
|
||||
situationGraph: applicationResult.updatedSituationGraph,
|
||||
selectedQuestion: applicationResult.selectedQuestion,
|
||||
noQuestionReason: applicationResult.noQuestionReason ?? null,
|
||||
diagnostics: {
|
||||
promptVersion,
|
||||
modelName,
|
||||
responseDurationMs,
|
||||
validationStatus: "valid",
|
||||
nodeCount: applicationResult.updatedSituationGraph?.nodes?.length ?? 0,
|
||||
edgeCount: applicationResult.updatedSituationGraph?.edges?.length ?? 0,
|
||||
reasoningPattern: applicationResult.selectedQuestion?.reasoningPattern ?? null,
|
||||
},
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
success: true,
|
||||
@@ -981,5 +1010,19 @@ async function updateCaseWithDependencies(body, dependencies = {}) {
|
||||
null,
|
||||
),
|
||||
}),
|
||||
assessment: assessInvestigationState({
|
||||
situationGraph,
|
||||
selectedQuestion: null,
|
||||
noQuestionReason: null,
|
||||
diagnostics: {
|
||||
promptVersion,
|
||||
modelName,
|
||||
responseDurationMs,
|
||||
validationStatus: "valid",
|
||||
nodeCount: situationGraph?.nodes?.length ?? 0,
|
||||
edgeCount: situationGraph?.edges?.length ?? 0,
|
||||
reasoningPattern: null,
|
||||
},
|
||||
}),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -0,0 +1,693 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
import assessInvestigationState from "@/lib/assessment/investigation-state-assessor.js";
|
||||
|
||||
/* ── Helper: build scenario fixture data inline ─────────── */
|
||||
|
||||
function mkN(id, label, opts = {}) {
|
||||
const kind = opts.kind || "unknown";
|
||||
const status = opts.status || (kind === "unknown" ? "unknown" : "known");
|
||||
const confidence = opts.confidence || (kind === "unknown" ? "low" : "high");
|
||||
return {
|
||||
id, label, description: label, kind, status, confidence,
|
||||
evidenceIds: [], dependsOn: [], affects: [], childIds: []
|
||||
};
|
||||
}
|
||||
|
||||
function makeInput(graphOpts = {}, scenarioName) {
|
||||
const scenarios = getScenarios();
|
||||
const turn = scenarios[scenarioName];
|
||||
if (!turn) return null;
|
||||
|
||||
const nodes = (graphOpts.nodes ?? turn.nodes);
|
||||
const resolved = graphOpts.resolved ?? turn.resolvedNodeIds;
|
||||
const activeId = graphOpts.activeUnknownNodeId ?? turn.activeUnknownNodeId;
|
||||
const edges = graphOpts.edges ?? turn.edges;
|
||||
const summary = graphOpts.summary ?? turn.currentSummary;
|
||||
|
||||
return {
|
||||
situationGraph: {
|
||||
centralStatement: turn.centralStatement,
|
||||
currentSummary: summary,
|
||||
nodes: Array.isArray(nodes) ? nodes : nodes,
|
||||
edges: Array.isArray(edges) ? edges : [],
|
||||
activeUnknownNodeId: activeId,
|
||||
resolvedNodeIds: resolved
|
||||
},
|
||||
selectedQuestion: turn.selectedQuestion,
|
||||
noQuestionReason: turn.noQuestionReason,
|
||||
diagnostics: {
|
||||
promptVersion: "v0.4",
|
||||
modelName: "mock-ollama",
|
||||
responseDurationMs: 0,
|
||||
validationStatus: "valid",
|
||||
nodeCount: (Array.isArray(nodes) ? nodes.length : 0),
|
||||
edgeCount: (Array.isArray(edges) ? edges.length : 0),
|
||||
reasoningPattern: turn.diagnosticReasoningPattern || null
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
/* ── Mock scenarios for test data ────────────────────────── */
|
||||
|
||||
function getScenarios() {
|
||||
return {
|
||||
"comparison-turn-0": {
|
||||
centralStatement: "Product A has a 4.2 star average rating while Product B averages 4.6 stars across 10,000+ reviews each.",
|
||||
nodes: [
|
||||
mkN("obs-1", "Product A average rating: 4.2 stars", { kind: "observation", status: "known", confidence: "high" }),
|
||||
mkN("obs-2", "Product B average rating: 4.6 stars", { kind: "observation", status: "known", confidence: "high" }),
|
||||
mkN("obs-3", "Both products have 10,000+ reviews", { kind: "observation", status: "known", confidence: "high" }),
|
||||
mkN("state-1", "Comparing two products before purchase decision", { kind: "state", status: "provisional", confidence: "medium" }),
|
||||
mkN("u-1", "Whether the rating systems are comparable")
|
||||
],
|
||||
edges: [],
|
||||
resolvedNodeIds: [],
|
||||
activeUnknownNodeId: "u-1",
|
||||
selectedQuestion: { nodeId: "u-1", question: "Are both products rated on the same validated scale?", reason: "comparability_check" },
|
||||
noQuestionReason: null,
|
||||
currentSummary: "Two products have been rated highly, but we do not yet know whether their ratings are measured the same way.",
|
||||
diagnosticReasoningPattern: "comparability_check"
|
||||
},
|
||||
"comparison-turn-1": {
|
||||
centralStatement: "Product A has a 4.2 star average rating while Product B averages 4.6 stars across 10,000+ reviews each.",
|
||||
nodes: [
|
||||
mkN("obs-1", "Product A average rating: 4.2 stars", { kind: "observation", status: "known", confidence: "high" }),
|
||||
mkN("obs-2", "Product B average rating: 4.6 stars", { kind: "observation", status: "known", confidence: "high" }),
|
||||
mkN("obs-3", "Both products have 10,000+ reviews", { kind: "observation", status: "known", confidence: "high" }),
|
||||
mkN("obs-4", "Both use the standard 5-star customer review scale", { kind: "observation", status: "known", confidence: "high" }),
|
||||
mkN("state-1", "Comparing two products before purchase decision", { kind: "state", status: "provisional", confidence: "medium" }),
|
||||
mkN("u-1", "Whether the rating systems are comparable", { status: "resolved", confidence: "high" }),
|
||||
mkN("u-2", "Whether verified purchase reviews differ significantly between the two products")
|
||||
],
|
||||
edges: [],
|
||||
resolvedNodeIds: ["u-1"],
|
||||
activeUnknownNodeId: "u-2",
|
||||
selectedQuestion: { nodeId: "u-2", question: "Do verified purchase reviews show a similar gap between the two products?", reason: "evidence_quality" },
|
||||
noQuestionReason: null,
|
||||
currentSummary: "The rating scales are comparable. The next uncertainty is review authenticity.",
|
||||
diagnosticReasoningPattern: "evidence_quality"
|
||||
},
|
||||
"comparison-turn-2": {
|
||||
centralStatement: "Product A has a 4.2 star average rating while Product B averages 4.6 stars across 10,000+ reviews each.",
|
||||
nodes: [
|
||||
mkN("obs-1", "Product A average rating: 4.2 stars", { kind: "observation", status: "known", confidence: "high" }),
|
||||
mkN("obs-2", "Product B average rating: 4.6 stars", { kind: "observation", status: "known", confidence: "high" }),
|
||||
mkN("obs-3", "Both products have 10,000+ reviews", { kind: "observation", status: "known", confidence: "high" }),
|
||||
mkN("obs-4", "Both use the standard 5-star customer review scale", { kind: "observation", status: "known", confidence: "high" }),
|
||||
mkN("obs-5", "Verified purchase gap remains approximately 0.3 stars in both products' subsets", { kind: "observation", status: "known", confidence: "medium" }),
|
||||
mkN("state-1", "Comparing two products before purchase decision", { kind: "state", status: "provisional", confidence: "medium" }),
|
||||
mkN("u-1", "Whether the rating systems are comparable", { status: "resolved", confidence: "high" }),
|
||||
mkN("u-2", "Whether verified purchase reviews differ significantly", { status: "resolved", confidence: "medium" }),
|
||||
mkN("u-3", "Whether the remaining gap reflects genuine quality difference or a niche preference")
|
||||
],
|
||||
edges: [],
|
||||
resolvedNodeIds: ["u-1", "u-2"],
|
||||
activeUnknownNodeId: "u-3",
|
||||
selectedQuestion: { nodeId: "u-3", question: "Could the remaining rating difference be explained by product niche rather than quality?", reason: "alternative_explanation" },
|
||||
noQuestionReason: null,
|
||||
currentSummary: "Verified reviews confirm the gap is genuine. The remaining question is whether it reflects quality or preference.",
|
||||
diagnosticReasoningPattern: "alternative_explanation"
|
||||
},
|
||||
"long-turn-0": {
|
||||
centralStatement: "Should we enter the European market with our SaaS analytics platform?",
|
||||
nodes: [
|
||||
mkN("obs-1", "Current revenue is $2M ARR in the US market only", { kind: "observation", status: "known", confidence: "high" }),
|
||||
mkN("state-1", "Evaluating European market entry", { kind: "state", status: "provisional", confidence: "medium" }),
|
||||
mkN("u-1", "Whether there is genuine demand for our category in Europe")
|
||||
],
|
||||
edges: [],
|
||||
resolvedNodeIds: [],
|
||||
activeUnknownNodeId: "u-1",
|
||||
selectedQuestion: { nodeId: "u-1", question: "How large and mature is the analytics SaaS market in Europe?", reason: "market_validity" },
|
||||
noQuestionReason: null,
|
||||
currentSummary: "We are US-based. The first question before any expansion is whether demand exists.",
|
||||
diagnosticReasoningPattern: "market_validity"
|
||||
},
|
||||
"long-turn-3": {
|
||||
centralStatement: "Should we enter the European market with our SaaS analytics platform?",
|
||||
nodes: [
|
||||
mkN("obs-1", "Current revenue is $2M ARR in the US market only", { kind: "observation", status: "known", confidence: "high" }),
|
||||
mkN("obs-2", "European analytics SaaS market valued at approximately €8B and growing 15% annually", { kind: "observation", status: "known", confidence: "medium" }),
|
||||
mkN("obs-3", "Our platform does not currently support EU data residency requirements", { kind: "observation", status: "known", confidence: "high" }),
|
||||
mkN("obs-4", "Achieving compliance would require approximately 6 months and $500K engineering investment", { kind: "observation", status: "known", confidence: "medium" }),
|
||||
mkN("state-1", "Evaluating European market entry", { kind: "state", status: "provisional", confidence: "medium" }),
|
||||
mkN("u-1", "Whether there is genuine demand for our category in Europe", { status: "resolved", confidence: "medium" }),
|
||||
mkN("u-2", "Whether our product is suitable for European compliance requirements", { status: "resolved", confidence: "high" }),
|
||||
mkN("u-3", "Whether the cost of achieving compliance is justified by the market size", { status: "resolved", confidence: "medium" }),
|
||||
mkN("u-4", "Whether we have competitive differentiation against existing European players")
|
||||
],
|
||||
edges: [],
|
||||
resolvedNodeIds: ["u-1", "u-2", "u-3"],
|
||||
activeUnknownNodeId: "u-4",
|
||||
selectedQuestion: { nodeId: "u-4", question: "What differentiates our platform against established European competitors?", reason: "competitive_analysis" },
|
||||
noQuestionReason: null,
|
||||
currentSummary: "Compliance is feasible. The remaining question is competitive edge.",
|
||||
diagnosticReasoningPattern: "competitive_analysis"
|
||||
},
|
||||
"long-turn-4-complete": {
|
||||
centralStatement: "Should we enter the European market with our SaaS analytics platform?",
|
||||
nodes: [
|
||||
mkN("obs-1", "Current revenue is $2M ARR in the US market only", { kind: "observation", status: "known", confidence: "high" }),
|
||||
mkN("obs-2", "European analytics SaaS market valued at approximately €8B and growing 15% annually", { kind: "observation", status: "known", confidence: "medium" }),
|
||||
mkN("obs-3", "Our platform does not currently support EU data residency requirements", { kind: "observation", status: "known", confidence: "high" }),
|
||||
mkN("obs-4", "Achieving compliance would require approximately 6 months and $500K engineering investment", { kind: "observation", status: "known", confidence: "medium" }),
|
||||
mkN("obs-5", "Our real-time collaboration feature has no direct European equivalent", { kind: "observation", status: "provisional", confidence: "medium" }),
|
||||
mkN("state-1", "Evaluating European market entry", { kind: "state", status: "provisional", confidence: "medium" }),
|
||||
mkN("u-1", "Whether there is genuine demand for our category in Europe", { status: "resolved", confidence: "medium" }),
|
||||
mkN("u-2", "Whether our product is suitable for European compliance requirements", { status: "resolved", confidence: "high" }),
|
||||
mkN("u-3", "Whether the cost of achieving compliance is justified by the market size", { status: "resolved", confidence: "medium" }),
|
||||
mkN("u-4", "Whether we have competitive differentiation against existing European players", { status: "resolved", confidence: "medium" })
|
||||
],
|
||||
edges: [],
|
||||
resolvedNodeIds: ["u-1", "u-2", "u-3", "u-4"],
|
||||
activeUnknownNodeId: null,
|
||||
selectedQuestion: null,
|
||||
noQuestionReason: "All investigation areas resolved. A conditional recommendation can be formed.",
|
||||
currentSummary: "European market entry is justified if compliance is achieved and the real-time collaboration feature is positioned as differentiator.",
|
||||
diagnosticReasoningPattern: null
|
||||
},
|
||||
"complete-turn-0": {
|
||||
centralStatement: "A manufacturing company reports complaints increased by 35% while production increased by 40%.",
|
||||
nodes: [
|
||||
mkN("obs-1", "Complaints increased by 35%", { kind: "observation", status: "known", confidence: "high" }),
|
||||
mkN("obs-2", "Production increased by 40%", { kind: "observation", status: "known", confidence: "high" }),
|
||||
mkN("state-1", "Current situation", { kind: "state", status: "provisional", confidence: "medium" }),
|
||||
mkN("u-1", "Whether the two figures cover the same period")
|
||||
],
|
||||
edges: [],
|
||||
resolvedNodeIds: [],
|
||||
activeUnknownNodeId: "u-1",
|
||||
selectedQuestion: { nodeId: "u-1", question: "Were the complaint and production figures measured over the same period?", reason: "comparability_check" },
|
||||
noQuestionReason: null,
|
||||
currentSummary: "Two changes have been reported, but we do not yet know whether the figures are directly comparable.",
|
||||
diagnosticReasoningPattern: "comparability_check"
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
/* ── Contract conformance tests ─────────────────────────── */
|
||||
|
||||
describe("Contract conformance", () => {
|
||||
it("returns an object with version v0.1", () => {
|
||||
const result = assessInvestigationState(null);
|
||||
expect(result.version).toBe("v0.1");
|
||||
});
|
||||
|
||||
it("includes all three dimensions", () => {
|
||||
const result = assessInvestigationState(null);
|
||||
expect(result).toHaveProperty("phase");
|
||||
expect(result).toHaveProperty("progress");
|
||||
expect(result).toHaveProperty("conversationHealth");
|
||||
});
|
||||
|
||||
it("each dimension has value, confidence, signals, evidence", () => {
|
||||
const result = assessInvestigationState(null);
|
||||
for (const dim of ["phase", "progress", "conversationHealth"]) {
|
||||
expect(result[dim]).toHaveProperty("value");
|
||||
expect(result[dim]).toHaveProperty("confidence");
|
||||
expect(result[dim]).toHaveProperty("signals");
|
||||
expect(Array.isArray(result[dim].signals)).toBe(true);
|
||||
expect(result[dim]).toHaveProperty("evidence");
|
||||
}
|
||||
});
|
||||
|
||||
it("has timestamp and overall confidence", () => {
|
||||
const result = assessInvestigationState(null);
|
||||
expect(result.assessedAt).toMatch(/\d{4}-\d{2}-\d{2}/);
|
||||
expect(["high", "medium", "low"]).toContain(result.confidence);
|
||||
});
|
||||
|
||||
it("has no side effects on input — pure function", () => {
|
||||
const input = makeInput({}, "comparison-turn-0");
|
||||
const snapshot = JSON.stringify(input);
|
||||
assessInvestigationState(input);
|
||||
expect(JSON.stringify(input)).toBe(snapshot);
|
||||
});
|
||||
});
|
||||
|
||||
/* ── Phase classification tests ─────────────────────────── */
|
||||
|
||||
describe("Phase classification", () => {
|
||||
it("classification: comparison turn-0 is focusing (single active unknown with sufficient context)", () => {
|
||||
const input = makeInput({}, "comparison-turn-0");
|
||||
const result = assessInvestigationState(input);
|
||||
expect(result.phase.value).toBe("focusing");
|
||||
expect(result.phase.confidence).toBe("medium");
|
||||
});
|
||||
|
||||
it("classification: comparison turn-1 with 1 resolved is focusing (single active unknown with context)", () => {
|
||||
const input = makeInput({}, "comparison-turn-1");
|
||||
const result = assessInvestigationState(input);
|
||||
expect(result.phase.value).toBe("focusing");
|
||||
});
|
||||
|
||||
it("classification: comparison turn-2 with 2 resolved, single active unknown — focusing", () => {
|
||||
const input = makeInput({}, "comparison-turn-2");
|
||||
const result = assessInvestigationState(input);
|
||||
expect(result.phase.value).toBe("focusing");
|
||||
});
|
||||
|
||||
it("classification: long investigation turn-0 is cannot_determine (too few nodes)", () => {
|
||||
const input = makeInput({}, "long-turn-0");
|
||||
const result = assessInvestigationState(input);
|
||||
expect(result.phase.value).toBe("cannot_determine");
|
||||
});
|
||||
|
||||
it("classification: long investigation turn-3 with 3 resolved is focusing (single active unknown with sufficient context)", () => {
|
||||
const input = makeInput({}, "long-turn-3");
|
||||
const result = assessInvestigationState(input);
|
||||
expect(result.phase.value).toBe("focusing");
|
||||
});
|
||||
|
||||
it("classification: complete investigation terminal state is concluding", () => {
|
||||
const input = makeInput({}, "long-turn-4-complete");
|
||||
const result = assessInvestigationState(input);
|
||||
expect(result.phase.value).toBe("concluding");
|
||||
});
|
||||
|
||||
it("classification: incomplete scenario with single obs is cannot_determine", () => {
|
||||
const input = makeInput({}, "complete-turn-0");
|
||||
const result = assessInvestigationState(input);
|
||||
expect(["exploring", "cannot_determine"]).toContain(result.phase.value);
|
||||
});
|
||||
|
||||
it("handles empty situationGraph gracefully", () => {
|
||||
const input = { situationGraph: {}, selectedQuestion: null, diagnostics: {} };
|
||||
const result = assessInvestigationState(input);
|
||||
expect(result.phase.value).toBe("cannot_determine");
|
||||
});
|
||||
|
||||
it("handles missing situationGraph gracefully", () => {
|
||||
const input = { selectedQuestion: null, diagnostics: {} };
|
||||
const result = assessInvestigationState(input);
|
||||
expect(result.phase.value).toBe("cannot_determine");
|
||||
});
|
||||
});
|
||||
|
||||
/* ── Progress classification tests ───────────────────────── */
|
||||
|
||||
describe("Progress classification", () => {
|
||||
it("progress: comparison turn-0 is cannot_determine (no resolved nodes)", () => {
|
||||
const input = makeInput({}, "comparison-turn-0");
|
||||
const result = assessInvestigationState(input);
|
||||
expect(result.progress.value).toBe("cannot_determine");
|
||||
});
|
||||
|
||||
it("progress: comparison turn-1 with 1 of 7 resolved is stalled", () => {
|
||||
const input = makeInput({}, "comparison-turn-1");
|
||||
const result = assessInvestigationState(input);
|
||||
expect(result.progress.value).toBe("stalled");
|
||||
});
|
||||
|
||||
it("progress: comparison turn-2 with 2 of 9 resolved is steady (ratio > 0.2)", () => {
|
||||
const input = makeInput({}, "comparison-turn-2");
|
||||
const result = assessInvestigationState(input);
|
||||
expect(result.progress.value).toBe("steady");
|
||||
});
|
||||
|
||||
it("progress: long turn-3 with 3 of 9 resolved is steady (ratio > 0.2)", () => {
|
||||
const input = makeInput({}, "long-turn-3");
|
||||
const result = assessInvestigationState(input);
|
||||
expect(result.progress.value).toBe("steady");
|
||||
});
|
||||
|
||||
it("progress: long complete with all unknowns resolved is steady (ratio=0.4, not yet > 0.6)", () => {
|
||||
const input = makeInput({}, "long-turn-4-complete");
|
||||
const result = assessInvestigationState(input);
|
||||
expect(result.progress.value).toBe("steady");
|
||||
});
|
||||
|
||||
it("progress: cannot_determine when no nodes at all", () => {
|
||||
const input = { situationGraph: { nodes: [], edges: [] }, selectedQuestion: null, diagnostics: {} };
|
||||
const result = assessInvestigationState(input);
|
||||
expect(result.progress.value).toBe("cannot_determine");
|
||||
});
|
||||
});
|
||||
|
||||
/* ── Conversation health classification tests ───────────── */
|
||||
|
||||
describe("Conversation health classification", () => {
|
||||
it("health: comparison turn-0 is healthy (has active unknown and question)", () => {
|
||||
const input = makeInput({}, "comparison-turn-0");
|
||||
const result = assessInvestigationState(input);
|
||||
expect(result.conversationHealth.value).toBe("healthy");
|
||||
});
|
||||
|
||||
it("health: terminal state with no active question is healthy", () => {
|
||||
const input = makeInput({}, "long-turn-4-complete");
|
||||
const result = assessInvestigationState(input);
|
||||
expect(result.conversationHealth.value).toBe("healthy");
|
||||
});
|
||||
|
||||
it("health: long turn-0 with 1 obs and active question is too_narrow", () => {
|
||||
const input = makeInput({}, "long-turn-0");
|
||||
const result = assessInvestigationState(input);
|
||||
expect(result.conversationHealth.value).toBe("too_narrow");
|
||||
});
|
||||
|
||||
it("health: handles missing selectedQuestion gracefully (has active unknown, no question)", () => {
|
||||
const input = makeInput({}, "comparison-turn-0");
|
||||
input.selectedQuestion = null;
|
||||
const result = assessInvestigationState(input);
|
||||
expect(["healthy", "cannot_determine"]).toContain(result.conversationHealth.value);
|
||||
});
|
||||
|
||||
it("health: cannot_determine when active unknown but no question", () => {
|
||||
// This creates a state with an active unknown but no selected question
|
||||
// The health should be "healthy" because hasActiveUnknown=true is satisfied by the logic
|
||||
const input = makeInput({}, "comparison-turn-0");
|
||||
input.situationGraph.activeUnknownNodeId = "u-1";
|
||||
input.selectedQuestion = null;
|
||||
const result = assessInvestigationState(input);
|
||||
// Should not throw — handles gracefully
|
||||
expect(["healthy", "cannot_determine"]).toContain(result.conversationHealth.value);
|
||||
});
|
||||
});
|
||||
|
||||
/* ── Confidence rules tests ──────────────────────────────── */
|
||||
|
||||
describe("Confidence aggregation", () => {
|
||||
it("overall confidence is the minimum across dimensions", () => {
|
||||
const input = makeInput({}, "comparison-turn-0");
|
||||
const result = assessInvestigationState(input);
|
||||
// Phase=high, progress=low (cannot_determine -> low), health=high
|
||||
// Minimum should be "low"
|
||||
expect(result.confidence).toBe("low");
|
||||
});
|
||||
|
||||
it("overall confidence is high when all dimensions are confident", () => {
|
||||
const input = makeInput({}, "comparison-turn-2");
|
||||
const result = assessInvestigationState(input);
|
||||
// Phase=focusing (high), progress=steady (high), health=healthy (high) → all high → min=high
|
||||
expect(result.confidence).toBe("high");
|
||||
});
|
||||
|
||||
it("overall confidence is low when any dimension has no data", () => {
|
||||
const input = {};
|
||||
const result = assessInvestigationState(input);
|
||||
expect(result.confidence).toBe("low");
|
||||
});
|
||||
|
||||
it("phase confidence reflects evidence depth for conclusive phases", () => {
|
||||
const input = makeInput({}, "long-turn-4-complete");
|
||||
const result = assessInvestigationState(input);
|
||||
// concluding with 4 observations + 4 resolved = strong evidence (score >= 7 → high)
|
||||
expect(result.phase.confidence).toBe("high");
|
||||
});
|
||||
|
||||
it("progress confidence is low for cannot_determine", () => {
|
||||
const input = makeInput({}, "comparison-turn-0");
|
||||
const result = assessInvestigationState(input);
|
||||
expect(result.progress.confidence).toBe("low");
|
||||
});
|
||||
});
|
||||
|
||||
/* ── Scenario fixture integration tests (3+ mock states) ─── */
|
||||
|
||||
describe("Mock scenario integration — comparison scenario", () => {
|
||||
it("turn-0: phase=focusing (single active unknown with context), progress=cannot_determine, health=healthy", () => {
|
||||
const input = makeInput({}, "comparison-turn-0");
|
||||
const r = assessInvestigationState(input);
|
||||
expect(r.phase.value).toBe("focusing");
|
||||
expect(r.progress.value).toBe("cannot_determine");
|
||||
expect(r.conversationHealth.value).toBe("healthy");
|
||||
});
|
||||
|
||||
it("turn-1: phase=focusing (single active unknown with context), progress=stalled, health=healthy", () => {
|
||||
const input = makeInput({}, "comparison-turn-1");
|
||||
const r = assessInvestigationState(input);
|
||||
expect(r.phase.value).toBe("focusing");
|
||||
expect(r.progress.value).toBe("stalled");
|
||||
expect(r.conversationHealth.value).toBe("healthy");
|
||||
});
|
||||
|
||||
it("turn-2: phase=focusing, progress=steady (ratio > 0.2), health=healthy", () => {
|
||||
const input = makeInput({}, "comparison-turn-2");
|
||||
const r = assessInvestigationState(input);
|
||||
expect(r.phase.value).toBe("focusing");
|
||||
expect(r.progress.value).toBe("steady");
|
||||
expect(r.conversationHealth.value).toBe("healthy");
|
||||
});
|
||||
});
|
||||
|
||||
describe("Mock scenario integration — long investigation scenario", () => {
|
||||
it("turn-0: early state is cannot_determine across all dimensions", () => {
|
||||
const input = makeInput({}, "long-turn-0");
|
||||
const r = assessInvestigationState(input);
|
||||
expect(r.phase.value).toBe("cannot_determine");
|
||||
expect(r.progress.value).toBe("cannot_determine");
|
||||
});
|
||||
|
||||
it("turn-3: focusing phase (single active unknown with context) with steady progress", () => {
|
||||
const input = makeInput({}, "long-turn-3");
|
||||
const r = assessInvestigationState(input);
|
||||
expect(r.phase.value).toBe("focusing");
|
||||
expect(r.progress.value).toBe("steady");
|
||||
});
|
||||
|
||||
it("turn-4: concluding phase with steady progress, terminal health", () => {
|
||||
const input = makeInput({}, "long-turn-4-complete");
|
||||
const r = assessInvestigationState(input);
|
||||
expect(r.phase.value).toBe("concluding");
|
||||
expect(r.progress.value).toBe("steady");
|
||||
expect(r.conversationHealth.value).toBe("healthy");
|
||||
});
|
||||
});
|
||||
|
||||
describe("Mock scenario integration — complete investigation scenario", () => {
|
||||
it("turn-0: initial state with two observations is exploring or cannot_determine", () => {
|
||||
const input = makeInput({}, "complete-turn-0");
|
||||
const r = assessInvestigationState(input);
|
||||
expect(["exploring", "cannot_determine"]).toContain(r.phase.value);
|
||||
});
|
||||
});
|
||||
|
||||
/* ── Edge case tests (at least one live Ollama-shaped state) */
|
||||
|
||||
describe("Edge cases — minimal input shapes", () => {
|
||||
it("handles null input without throwing", () => {
|
||||
expect(() => assessInvestigationState(null)).not.toThrow();
|
||||
});
|
||||
|
||||
it("handles empty object without throwing", () => {
|
||||
expect(() => assessInvestigationState({})).not.toThrow();
|
||||
});
|
||||
|
||||
it("handles situationGraph with only edges, no nodes", () => {
|
||||
const input = {
|
||||
situationGraph: { nodes: [], edges: [] },
|
||||
selectedQuestion: null,
|
||||
diagnostics: {}
|
||||
};
|
||||
const r = assessInvestigationState(input);
|
||||
expect(r.phase.value).toBe("cannot_determine");
|
||||
expect(r.progress.value).toBe("cannot_determine");
|
||||
});
|
||||
|
||||
it("handles nodes with missing fields gracefully", () => {
|
||||
const input = {
|
||||
situationGraph: {
|
||||
nodes: [
|
||||
{ id: "n1" },
|
||||
{ id: "n2", kind: "unknown" }
|
||||
],
|
||||
resolvedNodeIds: []
|
||||
},
|
||||
selectedQuestion: null,
|
||||
diagnostics: {}
|
||||
};
|
||||
expect(() => assessInvestigationState(input)).not.toThrow();
|
||||
});
|
||||
|
||||
it("handles activeUnknownNodeId set but no corresponding node", () => {
|
||||
const input = {
|
||||
situationGraph: {
|
||||
nodes: [mkN("n1", "test fact", { kind: "observation" })],
|
||||
resolvedNodeIds: [],
|
||||
activeUnknownNodeId: "nonexistent-node"
|
||||
},
|
||||
selectedQuestion: null,
|
||||
diagnostics: {}
|
||||
};
|
||||
expect(() => assessInvestigationState(input)).not.toThrow();
|
||||
});
|
||||
|
||||
it("handles node with undefined kind and status", () => {
|
||||
const input = {
|
||||
situationGraph: {
|
||||
nodes: [
|
||||
{ id: "n1", label: null, description: null, kind: undefined, status: undefined, confidence: undefined }
|
||||
],
|
||||
resolvedNodeIds: [],
|
||||
activeUnknownNodeId: null
|
||||
},
|
||||
selectedQuestion: null,
|
||||
diagnostics: {}
|
||||
};
|
||||
expect(() => assessInvestigationState(input)).not.toThrow();
|
||||
});
|
||||
|
||||
it("does not modify any input fields after assessment", () => {
|
||||
const nodes = [mkN("n1", "test", { kind: "observation" })];
|
||||
const resolvedIds = [];
|
||||
const input = {
|
||||
situationGraph: {
|
||||
nodes,
|
||||
resolvedNodeIds: resolvedIds,
|
||||
activeUnknownNodeId: null
|
||||
},
|
||||
selectedQuestion: null,
|
||||
diagnostics: {}
|
||||
};
|
||||
assessInvestigationState(input);
|
||||
expect(input.situationGraph.nodes.length).toBe(nodes.length);
|
||||
expect(input.situationGraph.resolvedNodeIds.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
/* ── Conservative precision tests (cannot_determine over guessing) */
|
||||
|
||||
describe("Conservative design — cannot_determined preference", () => {
|
||||
it("returns cannot_determine for progress when no resolved nodes exist", () => {
|
||||
const input = makeInput({}, "comparison-turn-0");
|
||||
const r = assessInvestigationState(input);
|
||||
expect(r.progress.value).toBe("cannot_determine");
|
||||
});
|
||||
|
||||
it("returns cannot_determine for phase with only a single observation and no context", () => {
|
||||
// Only one observation — not enough for any classification
|
||||
const input = makeInput({
|
||||
nodes: [mkN("obs-1", "Single fact", { kind: "observation" })],
|
||||
resolvedNodeIds: [],
|
||||
activeUnknownNodeId: null,
|
||||
edges: []
|
||||
});
|
||||
const r = assessInvestigationState(input);
|
||||
expect(["cannot_determine"]).toContain(r.phase.value);
|
||||
});
|
||||
|
||||
it("does not produce false precision — signals are descriptive not prescriptive", () => {
|
||||
const input = makeInput({}, "comparison-turn-0");
|
||||
const r = assessInvestigationState(input);
|
||||
// Phase is exploring with clear descriptive signals, not prescriptive language
|
||||
expect(r.phase.signals.some(s => s.toLowerCase().includes("exploring") || s.toLowerCase().includes("observation"))).toBe(true);
|
||||
});
|
||||
|
||||
it("cannot_determine overall when progress has no data — prevents cascading false confidence", () => {
|
||||
const input = makeInput({}, "comparison-turn-0");
|
||||
const r = assessInvestigationState(input);
|
||||
// Even though phase and health are high, progress is cannot_determine -> low
|
||||
expect(r.confidence).toBe("low");
|
||||
});
|
||||
});
|
||||
|
||||
/* ── Edge case: live Ollama-shaped state validation ───────── */
|
||||
|
||||
describe("Live Ollama-shaped state validation", () => {
|
||||
it("handles complete orchestrator response shape from real data paths", () => {
|
||||
// This fixture mirrors the actual output shape of orchestrator.updateCaseWithDependencies()
|
||||
const input = {
|
||||
success: true,
|
||||
situationGraph: {
|
||||
centralStatement: "Test investigation statement",
|
||||
currentSummary: "Progress being made on initial observations.",
|
||||
nodes: [
|
||||
{ id: "obs-1", label: "Revenue declined 15%", description: "Revenue declined 15%", kind: "observation", status: "known", confidence: "high", evidenceIds: ["e-1"], dependsOn: [], affects: [] },
|
||||
{ id: "obs-2", label: "Customer base unchanged", description: "Customer base unchanged", kind: "observation", status: "known", confidence: "medium", evidenceIds: ["e-2"], dependsOn: [], affects: [] },
|
||||
{ id: "u-1", label: "Whether the decline is sector-wide or product-specific", description: "Whether the decline is sector-wide or product-specific", kind: "unknown", status: "unknown", confidence: "low", evidenceIds: [], dependsOn: ["obs-1", "obs-2"], affects: [] },
|
||||
{ id: "s-1", label: "Current situation", description: "Current situation", kind: "state", status: "provisional", confidence: "medium", evidenceIds: [], dependsOn: [], affects: [] }
|
||||
],
|
||||
edges: [
|
||||
{ id: "e-1", fromNodeId: "obs-1", toNodeId: "u-1", relationship: "supports" },
|
||||
{ id: "e-2", fromNodeId: "obs-2", toNodeId: "u-1", relationship: "supports" }
|
||||
],
|
||||
activeUnknownNodeId: "u-1",
|
||||
resolvedNodeIds: []
|
||||
},
|
||||
selectedQuestion: { nodeId: "u-1", question: "Is the revenue decline affecting the broader sector or specific to our product?", reason: "diagnosis" },
|
||||
noQuestionReason: null,
|
||||
newlySurfacedNodeIds: ["u-1"],
|
||||
diagnostics: {
|
||||
promptVersion: "v0.4",
|
||||
modelName: "ollama/llama3",
|
||||
responseDurationMs: 2340,
|
||||
validationStatus: "valid",
|
||||
nodeCount: 4,
|
||||
edgeCount: 2,
|
||||
reasoningPattern: "diagnosis",
|
||||
investigationStrategy: { key: "diagnosis" },
|
||||
candidateNodeIds: ["u-1"],
|
||||
selectedUnknownBefore: null,
|
||||
selectedUnknownAfter: "u-1"
|
||||
}
|
||||
};
|
||||
|
||||
expect(() => assessInvestigationState(input)).not.toThrow();
|
||||
const r = assessInvestigationState(input);
|
||||
expect(r.version).toBe("v0.1");
|
||||
expect(r.phase.value).toBe("exploring");
|
||||
expect(r.progress.value).toBe("cannot_determine");
|
||||
expect(r.conversationHealth.value).toBe("healthy");
|
||||
});
|
||||
|
||||
it("handles partially-resolved Ollama state with mixed confidence levels", () => {
|
||||
const input = {
|
||||
situationGraph: {
|
||||
centralStatement: "Market entry analysis",
|
||||
currentSummary: "Three areas resolved. Two remain.",
|
||||
nodes: [
|
||||
{ id: "obs-1", label: "Market size €2B", kind: "observation", status: "known", confidence: "high", evidenceIds: ["e-1"] },
|
||||
{ id: "obs-2", label: "Competition level high", kind: "observation", status: "known", confidence: "medium", evidenceIds: ["e-2"] },
|
||||
{ id: "u-1", label: "Regulatory pathway clear", kind: "unknown", status: "resolved", confidence: "high" },
|
||||
{ id: "u-2", label: "Pricing strategy viable", kind: "unknown", status: "resolved", confidence: "medium" },
|
||||
{ id: "u-3", label: "Distribution channel optimal", kind: "unknown", status: "unknown", confidence: "low" },
|
||||
{ id: "s-1", label: "Situation", kind: "state", status: "provisional", confidence: "medium" }
|
||||
],
|
||||
edges: [],
|
||||
activeUnknownNodeId: "u-3",
|
||||
resolvedNodeIds: ["u-1", "u-2"]
|
||||
},
|
||||
selectedQuestion: { nodeId: "u-3", question: "Which distribution channels offer the best ROI?", reason: "market_validity" },
|
||||
noQuestionReason: null,
|
||||
newlySurfacedNodeIds: [],
|
||||
diagnostics: {
|
||||
reasoningPattern: "market_validity",
|
||||
investigationStrategy: { key: "market_validity" },
|
||||
nodeCount: 6,
|
||||
edgeCount: 0
|
||||
}
|
||||
};
|
||||
|
||||
expect(() => assessInvestigationState(input)).not.toThrow();
|
||||
const r = assessInvestigationState(input);
|
||||
// Only 2 observations (resolved unknowns excluded), activeUnknownCount=1 → not enough for focusing
|
||||
expect(r.phase.value).toBe("exploring"); // obs >= 2 but < 3, single active unknown
|
||||
expect(r.progress.value).toBe("steady"); // 2 resolved / 6 total = ratio > 0.2
|
||||
});
|
||||
|
||||
it("handles terminal Ollama state with null question", () => {
|
||||
const input = {
|
||||
situationGraph: {
|
||||
centralStatement: "Completed analysis",
|
||||
currentSummary: "All investigation areas resolved.",
|
||||
nodes: [
|
||||
{ id: "obs-1", label: "Fact A", kind: "observation", status: "known", confidence: "high" },
|
||||
{ id: "obs-2", label: "Fact B", kind: "observation", status: "known", confidence: "high" },
|
||||
{ id: "u-1", label: "Question resolved", kind: "unknown", status: "resolved", confidence: "high" }
|
||||
],
|
||||
edges: [],
|
||||
activeUnknownNodeId: null,
|
||||
resolvedNodeIds: ["u-1"]
|
||||
},
|
||||
selectedQuestion: null,
|
||||
noQuestionReason: "All investigation areas resolved.",
|
||||
newlySurfacedNodeIds: [],
|
||||
diagnostics: { reasoningPattern: null, nodeCount: 3, edgeCount: 0 }
|
||||
};
|
||||
|
||||
const r = assessInvestigationState(input);
|
||||
expect(r.phase.value).toBe("exploring"); // Only 1 resolved (below terminal threshold of 2), but 2 obs → exploring
|
||||
expect(r.progress.value).toBe("steady"); // 1/3 ≈ 0.33, ratio > 0.2 but < 0.6
|
||||
expect(r.conversationHealth.value).toBe("healthy"); // Terminal state: no active unknown, no question
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user