exp(18): implement investigation state assessment layer
Implement the three-dimensional assessment (phase, progress, conversation health) that sits between narrative and behaviour selection. Key changes: - lib/assessment/investigation-state-assessor.js: assessor module with countObservations, assessPhase, assessProgress, assessConversationHealth, assessInvestigationState — deterministic classifiers using known rules - tests/investigation-state-assessor.test.js: 51 tests covering phase classification (orienting→concluding), progress thresholds, health conditions, confidence aggregation, edge cases, and observation counting - lib/graph/orchestrator.js: integration calls passing correctly-shaped input to assessInvestigationState() at three call sites (~552, ~904, ~1013) Design decisions encoded in this iteration: - countObservations counts nodes with known/resolved status + high-confidence non-unknown non-state nodes (not just explicit observation-kind nodes) - Phase uses seven values including cannot_determine for insufficient data - Progress uses resolution ratio thresholds: accelerating (>0.6), steady (0.2-0.6), stalled (<0.2 with ≥1 resolved) - Overall confidence = minimum across all three dimensions (conservative) Also adds investigation-state-assessment-contract.md and updates design-evolution-log, investigation-state-assessment.md (status header), and investigation-turn-cycle.md (implementation status table).
This commit is contained in:
@@ -0,0 +1,232 @@
|
||||
# Investigation State Assessment Contract
|
||||
|
||||
> Architecture Experiment 18 — First Executable Slice
|
||||
>
|
||||
> This document defines the data contract for the investigation state assessment layer. It is the interface between investigation narrative (Stage 3) and behaviour selection (Stage 5).
|
||||
|
||||
---
|
||||
|
||||
## Versioning
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"version": "v0.1", // Schema version — increment only on field additions/removals
|
||||
"assessedAt": "ISO-8601", // When this assessment was computed
|
||||
"confidence": "high|medium|low" // Overall assessment confidence
|
||||
}
|
||||
```
|
||||
|
||||
All fields below are part of the v0.1 contract. Adding new fields requires a version bump to v0.2+. Removing or renaming existing fields also requires a version bump.
|
||||
|
||||
---
|
||||
|
||||
## Assessment Object Shape
|
||||
|
||||
### Phase
|
||||
|
||||
Evaluates where the investigation is in its lifecycle.
|
||||
|
||||
```json
|
||||
{
|
||||
"phase": {
|
||||
"value": "orienting|exploring|focusing|deepening|synthesising|concluding|cannot_determine",
|
||||
"confidence": "high|medium|low",
|
||||
"signals": [string], // Descriptive reasons for the classification
|
||||
"evidence": { // What data supported this classification
|
||||
"resolvedNodeCount": number,
|
||||
"activeUnknownCount": number,
|
||||
"unknownResolutionRatio": number | null,
|
||||
"observationDensity": number | null,
|
||||
"evidenceDepth": string | null // "shallow|moderate|deep"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Progress
|
||||
|
||||
Tracks whether understanding is advancing, stalled, or regressing.
|
||||
|
||||
```json
|
||||
{
|
||||
"progress": {
|
||||
"value": "accelerating|steady|stalled|looping|spiralling|cannot_determine",
|
||||
"confidence": "high|medium|low",
|
||||
"signals": [string],
|
||||
"evidence": {
|
||||
"turnCount": number,
|
||||
"recentResolutionsLastTurn": number,
|
||||
"newUnknownsPerTurn": number | null,
|
||||
"repeatedNodeIds": string[] // Nodes appearing across 3+ turns without resolution
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Conversation Health
|
||||
|
||||
Evaluates whether the conversation pattern is productive.
|
||||
|
||||
```json
|
||||
{
|
||||
"conversationHealth": {
|
||||
"value": "healthy|repetitive|too_broad|too_narrow|user_overloaded|user_under_informed|cannot_determine",
|
||||
"confidence": "high|medium|low",
|
||||
"signals": [string],
|
||||
"evidence": {
|
||||
"questionTypeDistribution": Record<string, number> | null, // reasoningPattern -> count
|
||||
"activeUnknownCount": number,
|
||||
"resolvedNodeRatio": number | null,
|
||||
"hasActiveQuestion": boolean | null,
|
||||
"summaryLength": number | null
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Overall Assessment
|
||||
|
||||
All dimensions combined:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "v0.1",
|
||||
"assessedAt": "ISO-8601",
|
||||
"confidence": "high|medium|low",
|
||||
"phase": { ... },
|
||||
"progress": { ... },
|
||||
"conversationHealth": { ... }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deterministic Rules (Phase)
|
||||
|
||||
The phase classifier uses conservative thresholds and defaults to `cannot_determine` when data is insufficient.
|
||||
|
||||
### Values
|
||||
|
||||
| Phase | Classification Rule |
|
||||
|-------|---------------------|
|
||||
| `concluding` | `activeUnknownCount === 0` AND `resolvedNodeCount >= 2` AND `hasActiveQuestion === false` — investigation has reached terminal state |
|
||||
| `synthesising` | `activeUnknownCount <= 1` AND `resolvedNodeRatio > 0.5` — near completion, synthesis possible |
|
||||
| `focusing` | `activeUnknownCount === 1` AND `knownObservations >= 3` — single remaining unknown with sufficient context |
|
||||
| `exploring` | `observationCount >= 2` AND `unknownResolutionRatio < 0.4` — gathering initial evidence |
|
||||
| `orienting` | `observationCount < 2` OR `totalNodeCount < 3` — insufficient structure to classify |
|
||||
| `deepening` | Residual: `activeUnknownCount > 1` AND `resolvedNodeCount >= 3` — structured investigation with remaining unknowns |
|
||||
| `cannot_determine` | Insufficient data for any of the above classifications |
|
||||
|
||||
### Thresholds (Conservative)
|
||||
|
||||
- A node is "known" only if `kind === "observation"` OR (`kind !== "unknown"` AND `status !== "provisional"`).
|
||||
- An unknown is "active" only if `kind === "unknown"` AND status is not `"resolved"`.
|
||||
- Resolution ratio = resolvedNodeCount / totalNonEmptyNodeCount (null if total is 0).
|
||||
- Known observation density threshold: ≥3 observations before classifying as `focusing` or above.
|
||||
|
||||
---
|
||||
|
||||
## Deterministic Rules (Progress)
|
||||
|
||||
Progress uses a single-snapshot approximation for this first slice. Multi-turn history tracking is reserved for a future version.
|
||||
|
||||
### Values
|
||||
|
||||
| Progress | Classification Rule |
|
||||
|----------|---------------------|
|
||||
| `accelerating` | `unknownResolutionRatio > 0.6` — resolving unknowns faster than they accumulate |
|
||||
| `steady` | `unknownResolutionRatio > 0.2` AND `unknownResolutionRatio <= 0.6` — balanced progress |
|
||||
| `stalled` | `unknownResolutionRatio < 0.2` AND `resolvedNodeCount >= 1` — some work done but no momentum |
|
||||
| `looping` | `repeatedNodeIds.length > 0` (nodes appearing across turns) AND `resolvedNodeCount === 0` for those nodes |
|
||||
| `spiralling` | `newUnknownsPerTurn > resolvedNodeCount` for recent history (requires turn history) |
|
||||
| `cannot_determine` | `resolvedNodeCount === 0` AND `totalNodeCount <= 2` — no sufficient data yet |
|
||||
|
||||
### Thresholds (Conservative)
|
||||
|
||||
- Resolution ratio = resolvedNodeCount / totalNonEmptyNodeCount.
|
||||
- No progress classification is assigned below 1 resolved node.
|
||||
- If resolution ratio cannot be computed (total === 0 or all nodes unresolved), return `cannot_determine`.
|
||||
|
||||
---
|
||||
|
||||
## Deterministic Rules (Conversation Health)
|
||||
|
||||
Conversation health evaluates the interaction pattern from available graph signals.
|
||||
|
||||
### Values
|
||||
|
||||
| Health | Classification Rule |
|
||||
|--------|---------------------|
|
||||
| `healthy` | Has active unknown AND has active question AND resolution ratio between 0 and 0.9 |
|
||||
| `repetitive` | Same reasoning pattern appears in >50% of questions (requires historical data) |
|
||||
| `too_broad` | Multiple active unknowns (>3) without sufficient resolved context (<2 resolved) |
|
||||
| `too_narrow` | Single observation with single unknown and no edges connecting them |
|
||||
| `user_overloaded` | Active unknown count > 4 (exceeds working memory limit) |
|
||||
| `user_under_informed` | Total observations < 2 AND active question exists (asking without enough context) |
|
||||
| `cannot_determine` | Insufficient graph signals to evaluate conversation pattern |
|
||||
|
||||
### Thresholds (Conservative)
|
||||
|
||||
- "Has active unknown" = `activeUnknownCount > 0`.
|
||||
- "Has active question" = `selectedQuestion !== null`.
|
||||
- Resolution ratio for healthy = between 0 and 0.9 (not terminal, not empty).
|
||||
|
||||
---
|
||||
|
||||
## Confidence Rules
|
||||
|
||||
### Overall Confidence
|
||||
|
||||
The overall confidence is the *minimum* of all dimension confidences. If any dimension cannot be determined, overall confidence is `"low"`.
|
||||
|
||||
### Per-Dimension Confidence
|
||||
|
||||
| Confidence | Condition |
|
||||
|-----------|-----------|
|
||||
| `high` | Classification supported by ≥2 independent signals AND data is complete (no null evidence fields) |
|
||||
| `medium` | Classification supported by 1 signal AND data is mostly complete (≤1 null evidence field) |
|
||||
| `low` | Classification supported by partial data OR has null evidence fields >1 |
|
||||
|
||||
---
|
||||
|
||||
## What This Contract Must Never Do
|
||||
|
||||
- **Never invent signals.** Only use data actually present in the graph schema, orchestrator diagnostics, or facilitator-view outputs.
|
||||
- **Never reach false precision.** When data is ambiguous, return `cannot_determine` rather than a confident but unsupported classification.
|
||||
- **Never mutate input state.** This is a pure function — no side effects, no writes, no network calls.
|
||||
- **Never bypass the narrative layer.** Assessment reads from narrative output and graph data, never directly from raw user input.
|
||||
- **Never prescribe behaviour.** Assessment describes state only; interpretation belongs to behaviour selection.
|
||||
|
||||
---
|
||||
|
||||
## Supported Data Sources (v0.1)
|
||||
|
||||
| Source | Fields Available |
|
||||
|--------|-----------------|
|
||||
| `situationGraph.nodes[]` | `id`, `label`, `description`, `kind`, `status`, `confidence`, `evidenceIds`, `dependsOn`, `affects` |
|
||||
| `situationGraph.resolvedNodeIds[]` | Array of resolved node IDs |
|
||||
| `situationGraph.activeUnknownNodeId` | Nullable string — currently targeted unknown |
|
||||
| `selectedQuestion` | `{ nodeId, question, reason }` nullable |
|
||||
| `diagnostics.reasoningPattern` | Current turn's reasoning pattern key |
|
||||
| `diagnostics.nodeCount` | Total node count |
|
||||
| `diagnostics.edgeCount` | Total edge count |
|
||||
| `facilitatorView._meta` | `isTerminal`, `hasUnresolvedUnknowns`, `totalObservations`, `unresolvedUnknownsCount` |
|
||||
|
||||
---
|
||||
|
||||
## Unsupported Signals (Planned for Future Versions)
|
||||
|
||||
These signals are specified in the investigation-state-assessment architecture document but require data not yet available in v0.1:
|
||||
|
||||
- **Evidence Quality** — requires per-node evidence confidence scoring across multiple sources
|
||||
- **Understanding Trajectory** — requires comparing narrative complexity across turns
|
||||
- **Uncertainty Trend** — requires tracking which unknowns are resolved and by what pattern
|
||||
- **Behaviour Readiness** — requires mapping assessment output to behaviour inventory
|
||||
|
||||
These are tracked in `reasoning-contract-backlog.md`.
|
||||
|
||||
---
|
||||
|
||||
## Recording Note
|
||||
|
||||
This contract was drafted for Experiment 18's first executable slice. It captures the minimal viable assessment object shape derived from actual repository data contracts (schema.js, orchestrator.js, facilitator-view-adapter.js). Future experiments will add dimensions and signals as the graph schema and diagnostics evolve.
|
||||
Reference in New Issue
Block a user