exp(18): implement investigation state assessment layer
Implement the three-dimensional assessment (phase, progress, conversation health) that sits between narrative and behaviour selection. Key changes: - lib/assessment/investigation-state-assessor.js: assessor module with countObservations, assessPhase, assessProgress, assessConversationHealth, assessInvestigationState — deterministic classifiers using known rules - tests/investigation-state-assessor.test.js: 51 tests covering phase classification (orienting→concluding), progress thresholds, health conditions, confidence aggregation, edge cases, and observation counting - lib/graph/orchestrator.js: integration calls passing correctly-shaped input to assessInvestigationState() at three call sites (~552, ~904, ~1013) Design decisions encoded in this iteration: - countObservations counts nodes with known/resolved status + high-confidence non-unknown non-state nodes (not just explicit observation-kind nodes) - Phase uses seven values including cannot_determine for insufficient data - Progress uses resolution ratio thresholds: accelerating (>0.6), steady (0.2-0.6), stalled (<0.2 with ≥1 resolved) - Overall confidence = minimum across all three dimensions (conservative) Also adds investigation-state-assessment-contract.md and updates design-evolution-log, investigation-state-assessment.md (status header), and investigation-turn-cycle.md (implementation status table).
This commit is contained in:
@@ -892,13 +892,57 @@ This architecture emerged from observation, not top-down design. It may still ch
|
||||
|
||||
A complete investigation can be described as a repeating turn cycle in which every architectural layer has a single responsibility.
|
||||
|
||||
Status
|
||||
Result
|
||||
|
||||
Architectural.
|
||||
Experiment validated that the investigation turn cycle is an *observation* about how existing layers interact rather than a new architectural layer. All eight stages (User Observation → Reasoning Graph → Investigation Narrative → State Assessment → Behaviour Selection → Conversation → Workspace → Wait) are supported by current architecture components, but only Stages 1–3 and 7 have working implementations. Stage 4 (State Assessment) and Stage 5 (Behaviour Selection) remain as architectural specifications without executable code.
|
||||
|
||||
Evaluation
|
||||
What did we learn?
|
||||
|
||||
Pending.
|
||||
- The turn cycle confirms that assessment sits between narrative and behaviour selection, not after the graph directly.
|
||||
- Every layer has one responsibility: each stage's purpose maps to an existing or specified component without overlap.
|
||||
- The cycle is deterministic in structure but adaptive in content — this is correct because the *sequence* of operations must be fixed while the *outputs* vary with investigation state.
|
||||
- Without a working Stage 4, all downstream stages (behaviour selection, conversation, workspace projection) operate on incomplete input. Phase 5 needs an executable assessment before behaviour can be validated experimentally.
|
||||
|
||||
Decision
|
||||
|
||||
The turn cycle architecture is confirmed as correct but requires implementation of Stage 4 (State Assessment) to move from observation to validation. The next step is the first deterministic evaluation function — not behaviour selection, which depends on assessment output. This becomes Experiment 18: First Executable Slice.
|
||||
|
||||
---
|
||||
|
||||
### Experiment 18 — First Executable Slice (Investigation State Assessment)
|
||||
|
||||
#### Hypothesis
|
||||
|
||||
A deterministic, conservative assessment of investigation phase and progress can be built from existing graph data without introducing new signals or modifying reasoning logic. The assessment should prefer `cannot_determine` over invented precision.
|
||||
|
||||
#### Scope
|
||||
|
||||
Phase detection (orienting / exploring / focusing / deepening / synthesising / concluding / cannot_determine), progress tracking (accelerating / steady / stalled / looping / spiralling / cannot_determine), and conversation health evaluation — using only data already present in the graph schema, orchestrator diagnostics, and facilitator-view outputs.
|
||||
|
||||
#### Constrained By
|
||||
|
||||
- Must use actual repo contracts (not assumptions about field names or structures).
|
||||
- Must be pure function — no network, LLM, mutation, or side effects.
|
||||
- Must handle missing fields gracefully — safe with absent data.
|
||||
- Must produce versioned assessment objects for future compatibility.
|
||||
- Passive integration only: add to diagnostics without changing public API or user-visible behaviour.
|
||||
|
||||
#### Questions
|
||||
|
||||
1. Can phase be reliably classified from node composition (kind/status ratio) alone?
|
||||
2. Does progress detection require turn history, or is a single-snapshot approximation sufficient for this first slice?
|
||||
3. What minimal conversation health signals can be extracted from existing graph metadata?
|
||||
|
||||
#### Evaluation
|
||||
|
||||
- Deterministic output across identical inputs.
|
||||
- Correct `cannot_determine` when data is insufficient (no false precision).
|
||||
- Handles all 11 mock scenarios at their turn points plus at least one live Ollama-shaped state.
|
||||
- Unsupported signals explicitly recorded in reasoning-contract-backlog.md.
|
||||
|
||||
#### Status
|
||||
|
||||
Completed — see `investigation-state-assessment-contract.md` and `lib/assessment/investigation-state-assessor.js`.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,232 @@
|
||||
# Investigation State Assessment Contract
|
||||
|
||||
> Architecture Experiment 18 — First Executable Slice
|
||||
>
|
||||
> This document defines the data contract for the investigation state assessment layer. It is the interface between investigation narrative (Stage 3) and behaviour selection (Stage 5).
|
||||
|
||||
---
|
||||
|
||||
## Versioning
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"version": "v0.1", // Schema version — increment only on field additions/removals
|
||||
"assessedAt": "ISO-8601", // When this assessment was computed
|
||||
"confidence": "high|medium|low" // Overall assessment confidence
|
||||
}
|
||||
```
|
||||
|
||||
All fields below are part of the v0.1 contract. Adding new fields requires a version bump to v0.2+. Removing or renaming existing fields also requires a version bump.
|
||||
|
||||
---
|
||||
|
||||
## Assessment Object Shape
|
||||
|
||||
### Phase
|
||||
|
||||
Evaluates where the investigation is in its lifecycle.
|
||||
|
||||
```json
|
||||
{
|
||||
"phase": {
|
||||
"value": "orienting|exploring|focusing|deepening|synthesising|concluding|cannot_determine",
|
||||
"confidence": "high|medium|low",
|
||||
"signals": [string], // Descriptive reasons for the classification
|
||||
"evidence": { // What data supported this classification
|
||||
"resolvedNodeCount": number,
|
||||
"activeUnknownCount": number,
|
||||
"unknownResolutionRatio": number | null,
|
||||
"observationDensity": number | null,
|
||||
"evidenceDepth": string | null // "shallow|moderate|deep"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Progress
|
||||
|
||||
Tracks whether understanding is advancing, stalled, or regressing.
|
||||
|
||||
```json
|
||||
{
|
||||
"progress": {
|
||||
"value": "accelerating|steady|stalled|looping|spiralling|cannot_determine",
|
||||
"confidence": "high|medium|low",
|
||||
"signals": [string],
|
||||
"evidence": {
|
||||
"turnCount": number,
|
||||
"recentResolutionsLastTurn": number,
|
||||
"newUnknownsPerTurn": number | null,
|
||||
"repeatedNodeIds": string[] // Nodes appearing across 3+ turns without resolution
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Conversation Health
|
||||
|
||||
Evaluates whether the conversation pattern is productive.
|
||||
|
||||
```json
|
||||
{
|
||||
"conversationHealth": {
|
||||
"value": "healthy|repetitive|too_broad|too_narrow|user_overloaded|user_under_informed|cannot_determine",
|
||||
"confidence": "high|medium|low",
|
||||
"signals": [string],
|
||||
"evidence": {
|
||||
"questionTypeDistribution": Record<string, number> | null, // reasoningPattern -> count
|
||||
"activeUnknownCount": number,
|
||||
"resolvedNodeRatio": number | null,
|
||||
"hasActiveQuestion": boolean | null,
|
||||
"summaryLength": number | null
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Overall Assessment
|
||||
|
||||
All dimensions combined:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "v0.1",
|
||||
"assessedAt": "ISO-8601",
|
||||
"confidence": "high|medium|low",
|
||||
"phase": { ... },
|
||||
"progress": { ... },
|
||||
"conversationHealth": { ... }
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deterministic Rules (Phase)
|
||||
|
||||
The phase classifier uses conservative thresholds and defaults to `cannot_determine` when data is insufficient.
|
||||
|
||||
### Values
|
||||
|
||||
| Phase | Classification Rule |
|
||||
|-------|---------------------|
|
||||
| `concluding` | `activeUnknownCount === 0` AND `resolvedNodeCount >= 2` AND `hasActiveQuestion === false` — investigation has reached terminal state |
|
||||
| `synthesising` | `activeUnknownCount <= 1` AND `resolvedNodeRatio > 0.5` — near completion, synthesis possible |
|
||||
| `focusing` | `activeUnknownCount === 1` AND `knownObservations >= 3` — single remaining unknown with sufficient context |
|
||||
| `exploring` | `observationCount >= 2` AND `unknownResolutionRatio < 0.4` — gathering initial evidence |
|
||||
| `orienting` | `observationCount < 2` OR `totalNodeCount < 3` — insufficient structure to classify |
|
||||
| `deepening` | Residual: `activeUnknownCount > 1` AND `resolvedNodeCount >= 3` — structured investigation with remaining unknowns |
|
||||
| `cannot_determine` | Insufficient data for any of the above classifications |
|
||||
|
||||
### Thresholds (Conservative)
|
||||
|
||||
- A node is "known" only if `kind === "observation"` OR (`kind !== "unknown"` AND `status !== "provisional"`).
|
||||
- An unknown is "active" only if `kind === "unknown"` AND status is not `"resolved"`.
|
||||
- Resolution ratio = resolvedNodeCount / totalNonEmptyNodeCount (null if total is 0).
|
||||
- Known observation density threshold: ≥3 observations before classifying as `focusing` or above.
|
||||
|
||||
---
|
||||
|
||||
## Deterministic Rules (Progress)
|
||||
|
||||
Progress uses a single-snapshot approximation for this first slice. Multi-turn history tracking is reserved for a future version.
|
||||
|
||||
### Values
|
||||
|
||||
| Progress | Classification Rule |
|
||||
|----------|---------------------|
|
||||
| `accelerating` | `unknownResolutionRatio > 0.6` — resolving unknowns faster than they accumulate |
|
||||
| `steady` | `unknownResolutionRatio > 0.2` AND `unknownResolutionRatio <= 0.6` — balanced progress |
|
||||
| `stalled` | `unknownResolutionRatio < 0.2` AND `resolvedNodeCount >= 1` — some work done but no momentum |
|
||||
| `looping` | `repeatedNodeIds.length > 0` (nodes appearing across turns) AND `resolvedNodeCount === 0` for those nodes |
|
||||
| `spiralling` | `newUnknownsPerTurn > resolvedNodeCount` for recent history (requires turn history) |
|
||||
| `cannot_determine` | `resolvedNodeCount === 0` AND `totalNodeCount <= 2` — no sufficient data yet |
|
||||
|
||||
### Thresholds (Conservative)
|
||||
|
||||
- Resolution ratio = resolvedNodeCount / totalNonEmptyNodeCount.
|
||||
- No progress classification is assigned below 1 resolved node.
|
||||
- If resolution ratio cannot be computed (total === 0 or all nodes unresolved), return `cannot_determine`.
|
||||
|
||||
---
|
||||
|
||||
## Deterministic Rules (Conversation Health)
|
||||
|
||||
Conversation health evaluates the interaction pattern from available graph signals.
|
||||
|
||||
### Values
|
||||
|
||||
| Health | Classification Rule |
|
||||
|--------|---------------------|
|
||||
| `healthy` | Has active unknown AND has active question AND resolution ratio between 0 and 0.9 |
|
||||
| `repetitive` | Same reasoning pattern appears in >50% of questions (requires historical data) |
|
||||
| `too_broad` | Multiple active unknowns (>3) without sufficient resolved context (<2 resolved) |
|
||||
| `too_narrow` | Single observation with single unknown and no edges connecting them |
|
||||
| `user_overloaded` | Active unknown count > 4 (exceeds working memory limit) |
|
||||
| `user_under_informed` | Total observations < 2 AND active question exists (asking without enough context) |
|
||||
| `cannot_determine` | Insufficient graph signals to evaluate conversation pattern |
|
||||
|
||||
### Thresholds (Conservative)
|
||||
|
||||
- "Has active unknown" = `activeUnknownCount > 0`.
|
||||
- "Has active question" = `selectedQuestion !== null`.
|
||||
- Resolution ratio for healthy = between 0 and 0.9 (not terminal, not empty).
|
||||
|
||||
---
|
||||
|
||||
## Confidence Rules
|
||||
|
||||
### Overall Confidence
|
||||
|
||||
The overall confidence is the *minimum* of all dimension confidences. If any dimension cannot be determined, overall confidence is `"low"`.
|
||||
|
||||
### Per-Dimension Confidence
|
||||
|
||||
| Confidence | Condition |
|
||||
|-----------|-----------|
|
||||
| `high` | Classification supported by ≥2 independent signals AND data is complete (no null evidence fields) |
|
||||
| `medium` | Classification supported by 1 signal AND data is mostly complete (≤1 null evidence field) |
|
||||
| `low` | Classification supported by partial data OR has null evidence fields >1 |
|
||||
|
||||
---
|
||||
|
||||
## What This Contract Must Never Do
|
||||
|
||||
- **Never invent signals.** Only use data actually present in the graph schema, orchestrator diagnostics, or facilitator-view outputs.
|
||||
- **Never reach false precision.** When data is ambiguous, return `cannot_determine` rather than a confident but unsupported classification.
|
||||
- **Never mutate input state.** This is a pure function — no side effects, no writes, no network calls.
|
||||
- **Never bypass the narrative layer.** Assessment reads from narrative output and graph data, never directly from raw user input.
|
||||
- **Never prescribe behaviour.** Assessment describes state only; interpretation belongs to behaviour selection.
|
||||
|
||||
---
|
||||
|
||||
## Supported Data Sources (v0.1)
|
||||
|
||||
| Source | Fields Available |
|
||||
|--------|-----------------|
|
||||
| `situationGraph.nodes[]` | `id`, `label`, `description`, `kind`, `status`, `confidence`, `evidenceIds`, `dependsOn`, `affects` |
|
||||
| `situationGraph.resolvedNodeIds[]` | Array of resolved node IDs |
|
||||
| `situationGraph.activeUnknownNodeId` | Nullable string — currently targeted unknown |
|
||||
| `selectedQuestion` | `{ nodeId, question, reason }` nullable |
|
||||
| `diagnostics.reasoningPattern` | Current turn's reasoning pattern key |
|
||||
| `diagnostics.nodeCount` | Total node count |
|
||||
| `diagnostics.edgeCount` | Total edge count |
|
||||
| `facilitatorView._meta` | `isTerminal`, `hasUnresolvedUnknowns`, `totalObservations`, `unresolvedUnknownsCount` |
|
||||
|
||||
---
|
||||
|
||||
## Unsupported Signals (Planned for Future Versions)
|
||||
|
||||
These signals are specified in the investigation-state-assessment architecture document but require data not yet available in v0.1:
|
||||
|
||||
- **Evidence Quality** — requires per-node evidence confidence scoring across multiple sources
|
||||
- **Understanding Trajectory** — requires comparing narrative complexity across turns
|
||||
- **Uncertainty Trend** — requires tracking which unknowns are resolved and by what pattern
|
||||
- **Behaviour Readiness** — requires mapping assessment output to behaviour inventory
|
||||
|
||||
These are tracked in `reasoning-contract-backlog.md`.
|
||||
|
||||
---
|
||||
|
||||
## Recording Note
|
||||
|
||||
This contract was drafted for Experiment 18's first executable slice. It captures the minimal viable assessment object shape derived from actual repository data contracts (schema.js, orchestrator.js, facilitator-view-adapter.js). Future experiments will add dimensions and signals as the graph schema and diagnostics evolve.
|
||||
@@ -1,6 +1,7 @@
|
||||
# Investigation State Assessment — Architectural Specification
|
||||
|
||||
> This is a design document only. Do not implement yet.
|
||||
> **Status: Implemented (Experiment 18). First executable slice deployed.**
|
||||
> Design evolved through experiments; implementation validates and adjusts the spec iteratively.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -257,6 +257,23 @@ Nothing system-generated. The next observation originates entirely from the user
|
||||
|
||||
---
|
||||
|
||||
## Implementation Status
|
||||
|
||||
| Stage | Description | Status | Experiment |
|
||||
|-------|-------------|--------|------------|
|
||||
| 1 | User Observation | Implemented (input) | — |
|
||||
| 2 | Reasoning Graph Update | Implemented | Various |
|
||||
| 3 | Investigation Narrative Update | Partially implemented | — |
|
||||
| 4 | State Assessment | **Implemented** (v0.1) | Exp 18 |
|
||||
| 5 | Behaviour Selection | Design only | **Exp 19 next** |
|
||||
| 6 | Conversation Response | Design only | Post-Exp 19 |
|
||||
| 7 | Workspace Projection | Implemented (UI) | Various |
|
||||
| 8 | Wait for Next Observation | Implemented (state machine) | — |
|
||||
|
||||
Stages 4 and 5 remain as architectural specifications without executable code. Stage 4 was completed in Experiment 18; Stage 5 is the next implementation target.
|
||||
|
||||
---
|
||||
|
||||
## What This Turn Cycle Proves
|
||||
|
||||
The investigation turn cycle is not a new layer. It is an observation about how existing layers interact during a real investigation. It confirms that:
|
||||
|
||||
Reference in New Issue
Block a user