259 lines
13 KiB
Markdown
259 lines
13 KiB
Markdown
# v0.4 Handoff — Confidence Engine (confidence-engine)
|
|
|
|
**Date:** 2026-08-01
|
|
**Branch:** `feature/reconstruction-v0.3`
|
|
**Parent branch:** `main`
|
|
|
|
---
|
|
|
|
## 1. What This Project Is
|
|
|
|
A Next.js app that performs evidence-based situation reconstruction on user-supplied scenarios. An LLM analyses the scenario, builds a directed graph of actors, systems, unknowns and relationships, then iteratively refines the graph through multi-turn Q&A with the user.
|
|
|
|
---
|
|
|
|
## 2. Recent Commit History
|
|
|
|
| Commit | Message |
|
|
|--------|---------|
|
|
| `79ea2f6` | feat: add v0.3 normalised comparison reasoning |
|
|
| `d72c7c5` | chore: establish clean v0.2 baseline |
|
|
| `a2f9e47` | chore: preserve initial reconstruction prototype |
|
|
|
|
Only **one commit** ahead of `main`: `79ea2f6` — the v0.3 normalised comparison reasoning work.
|
|
|
|
---
|
|
|
|
## 3. Current State Summary
|
|
|
|
### What's done and committed to this branch
|
|
|
|
1. **v0.3 prompt** (`prompts/reconstruct-v0.3.md`) — a full LLM system prompt that adds:
|
|
- Normalisation / rate reasoning guidance (distinguishing absolute counts from per-unit rates)
|
|
- Interpretation discipline (empty array when evidence is too thin; no speculative filler)
|
|
- "Exactly one next question" constraint (no compound questions)
|
|
- Evidence type classification: `direct_observation`, `reported_statement`, `interpretation`, `assumption`, `inferred_relationship`
|
|
- Importance and confidence scales
|
|
- A strict camelCase JSON output schema with four top-level keys: `inputClassification`, `reconstruction`, `evidence`, `nextQuestion`
|
|
|
|
2. **v0.3 prompt versioning** (`lib/reconstruction/prompt.js`) — exports `PROMPT_VERSIONS`, `DEFAULT_PROMPT_VERSION ("v0.3")`, and `buildPrompt(scenario, version)` for loading prompt templates from disk with scenario substitution.
|
|
|
|
3. **Schema validation** (`lib/reconstruction/schema.js`) — Zod schemas for v0.2 output (`reconstructionV2Schema`). A `parseReconstructionV2(rawString)` helper is used in the analysis pipeline.
|
|
|
|
4. **v0.3 reasoning tests** (`tests/v03-reasoning.test.js`) — extensive test suite covering:
|
|
- Prompt version registration and loading
|
|
- v0.3 guidance completeness (normalisation, rate vs count, correlation-vs-causation)
|
|
- Schema validation with a realistic "production/complaints" fixture
|
|
- Parse helper tests
|
|
|
|
5. **Graph library** (`lib/graph/`) — the multi-turn reconstruction pipeline:
|
|
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| `schema.js` | Zod schemas for SituationNode, SituationEdge, SituationGraph, GraphUpdate; helpers like `makeNodeId`, `makeNode`, `makeEdge`, `makeGraph` |
|
|
| `builder.js` | `buildInitialGraph(reconstruction, evidence)` — converts v0.2/v0.3 analysis output into a SituationGraph with deterministic nodes/edges; `buildMinimalGraph(scenario)` for fallback; `describeGraph(graph)` for display |
|
|
| `orchestrator.js` | `CaseOrchestrator` class managing the full multi-turn lifecycle (idle → building → active); exports `startCase(body)` and `updateCase(body)` convenience functions for API routes |
|
|
| `prompt-builder.js` | `buildUpdatePrompt(ctx)` — formats current graph state + Q&A context into a system prompt for the LLM update-evaluation turn |
|
|
| `utils.js` | Deterministic graph operations: `validateGraphReferences`, `detectDuplicateNodeIds`, `detectDuplicateEdges`, `findDependentNodes`, `findAffectedNodes`, `resolveUnknownNode`, `selectActiveUnknownCandidate`, `applyGraphUpdate`, `validateGraphUpdate` |
|
|
|
|
6. **API routes** (`app/api/`)
|
|
|
|
| Route | Purpose |
|
|
|-------|---------|
|
|
| `POST /api/start-case` | Start a new reconstruction case — accepts `{ scenario, promptVersion? }`, returns graph summary, node/edge counts, next question |
|
|
| `POST /api/update-case` | Process a turn — accepts `{ scenario, graph, answer, currentQuestion?, turnCount?, modelName? }`, returns updated graph summary, next question, changes summary |
|
|
|
|
7. **Smoke test** (`tests/smoke.test.js`) — basic integration test for the start-case API route.
|
|
|
|
### What's NOT yet committed (untracked files from git status)
|
|
|
|
| File | Description |
|
|
|------|-------------|
|
|
| `lib/graph/` (full directory) | The multi-turn graph library — built but NOT yet committed to any branch. These are the new untracked files: `builder.js`, `orchestrator.js`, `prompt-builder.js`, `schema.js`, `utils.js` |
|
|
| `tests/graph/` (full directory) | Tests for the graph library — also untracked: `builder.test.js`, `orchestrator.test.js`, `prompt-builder.test.js`, `schema.test.js`, `utils.test.js` |
|
|
| `app/api/start-case/route.js` | New API route (untracked) |
|
|
| `app/api/update-case/route.js` | New API route (untracked) |
|
|
|
|
> **Important:** The git status shows these files as untracked (`??`). They exist on disk but have never been staged or committed. You need to decide whether to commit them now or integrate them differently.
|
|
|
|
---
|
|
|
|
## 4. Test Status
|
|
|
|
```
|
|
Test Files: 4 failed | 4 passed (8)
|
|
Tests: 5 failed | 216 passed (221)
|
|
```
|
|
|
|
### Known failures
|
|
|
|
The failures cluster in `tests/graph/`:
|
|
- **`prompt-builder.test.js`** — test expects the literal string `"Existing or newly added nodes"` but the prompt template currently says `"existing or newly added nodes"` (case mismatch). The SYSTEM_PROMPT_HEADER constant uses lowercase.
|
|
- Other graph tests likely have similar fixture/reference issues.
|
|
|
|
Run `npx vitest run tests/graph/ --reporter=verbose` for full details.
|
|
|
|
---
|
|
|
|
## 5. Architecture Overview
|
|
|
|
```
|
|
User scenario
|
|
│
|
|
▼
|
|
┌──────────────┐ ┌─────────────────┐ ┌──────────────┐
|
|
│ analyseScenario│──▶│ buildPrompt │──▶│ LLM (v0.3) │
|
|
│ (lib/analysis.js) │ (reconstruction/prompt.js) │ │
|
|
└──────────────┘ └─────────────────┘ └──────┬───────┘
|
|
│
|
|
▼
|
|
┌──────────────┐
|
|
│ Parse output │
|
|
│ (Zod/parse │
|
|
│ Reconstruction│
|
|
│ V2) │
|
|
└──────┬───────┘
|
|
│
|
|
┌───────────────────────────────┤
|
|
▼ ▼
|
|
┌──────────────┐ ┌──────────────────┐
|
|
│buildInitialGraph│ │ buildMinimalGraph │
|
|
│ (graph/builder)│ │ (fallback) │
|
|
└──────┬─────────┘ └──────────────────┘
|
|
│
|
|
▼
|
|
┌──────────────┐
|
|
│SituationGraph │ ← Zod-validated graph structure
|
|
│ {nodes, edges}│ nodes: observation/metric/unknown/...
|
|
└──────┬───────┘ edges: supports/weakens/causes/...
|
|
│
|
|
(multi-turn loop via updateCase)
|
|
│
|
|
┌─────────▼─────────┐
|
|
│buildUpdatePrompt │ → LLM proposes GraphUpdate
|
|
│ │
|
|
│applyGraphUpdate │ → deterministic, validated
|
|
│validateGraphUpdate│ (no direct LLM mutation)
|
|
└───────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## 6. Key Design Decisions
|
|
|
|
### Normalisation / rate reasoning (v0.3 focus)
|
|
The v0.3 prompt explicitly instructs the model to:
|
|
- Always consider whether a denominator/exposure metric is needed when counts change alongside scale
|
|
- Distinguish absolute count from rate
|
|
- Avoid treating two rising counts as causal evidence (production growth may outpace complaint growth)
|
|
- Request the per-unit metric as the highest-value next question
|
|
|
|
### Graph immutability
|
|
LLM proposals are never applied directly. All mutations go through `applyGraphUpdate()` in `lib/graph/utils.js`, which:
|
|
- Validates all node/edge references exist
|
|
- Rejects duplicate IDs
|
|
- Enforces a max graph size (500 nodes) and update size (100KB)
|
|
- Returns the full new state for validation
|
|
|
|
### Prompt versioning
|
|
- Default is `"v0.3"` but `PROMPT_VERSIONS` includes `"v0.2"` for backward compatibility
|
|
- `RECONSTRUCTION_PROMPT_VERSION` env var can override default at module load time
|
|
- Prompts are loaded from `prompts/reconstruct-v0.{version}.md` on disk
|
|
|
|
### Deterministic node IDs
|
|
Node IDs are computed via a deterministic hash of the label: `makeNodeId(label)`. This avoids conflicts but means nodes must be created with consistent labels to get consistent IDs.
|
|
|
|
---
|
|
|
|
## 7. Open Questions / TODOs for Next Developer
|
|
|
|
1. **Untracked graph library** — `lib/graph/` and `tests/graph/` are untracked on disk. Do we commit them as part of v0.4, or keep them in a separate branch?
|
|
|
|
2. **Test failures** — 5 tests fail across the graph test suite. The prompt-builder case-sensitivity issue needs fixing. Review all failing tests before merging.
|
|
|
|
3. **Missing `RECONSTRUCTION_PROMPT_VERSION` env var docs** — The system uses an env var override but it's not documented in `.env.example`. Add it if it's intended to be configurable.
|
|
|
|
4. **Provider integration** — `lib/llm/provider.js` is imported by the orchestrator (`getProvider()`, `generateReconstruction()`). Verify the provider implementation matches what this code expects.
|
|
|
|
5. **Graph completeness heuristic** — `CaseOrchestrator.getCompletionStatus()` returns `"complete"` when no unknown nodes remain, but doesn't consider whether all important observations have been verified.
|
|
|
|
6. **Error resilience in update flow** — If the LLM returns malformed JSON, the update route returns a 500 with a generic error message. Consider retry logic or structured error parsing.
|
|
|
|
7. **`buildUpdatePrompt` SYSTEM_PROMPT_HEADER is a module-level constant** — it's hardcoded and never versioned. If v0.5 changes the update-evaluation prompt style, this will need to become a template.
|
|
|
|
8. **The `nextQuestion` field on `/api/start-case` response** includes the adapted question (original + active unknown label appended). The client may want the original and adapted separately.
|
|
|
|
---
|
|
|
|
## 8. File Inventory (new / changed files on this branch)
|
|
|
|
### Prompts
|
|
- `prompts/reconstruct-v0.3.md` — **NEW** — v0.3 system prompt (161 lines)
|
|
- `prompts/reconstruct-v0.2.md` — **existing** — baseline prompt
|
|
|
|
### Core library
|
|
- `lib/analysis.js` — **MODIFIED** — analyseScenario function (uses v0.3 prompt by default)
|
|
- `lib/reconstruction/prompt.js` — **MODIFIED** — prompt versioning exports
|
|
- `lib/reconstruction/schema.js` — **existing** — Zod schemas + parseReconstructionV2
|
|
|
|
### Graph library (untracked on disk)
|
|
- `lib/graph/builder.js` — buildInitialGraph, buildMinimalGraph, describeGraph
|
|
- `lib/graph/orchestrator.js` — CaseOrchestrator class, startCase, updateCase
|
|
- `lib/graph/prompt-builder.js` — buildUpdatePrompt + SYSTEM_PROMPT_HEADER
|
|
- `lib/graph/schema.js` — SituationNode/Edge/Graph/Update Zod schemas
|
|
- `lib/graph/utils.js` — validation, dedup, dependency, and apply utilities
|
|
|
|
### API routes (untracked on disk)
|
|
- `app/api/start-case/route.js`
|
|
- `app/api/update-case/route.js`
|
|
|
|
### Tests (untracked on disk)
|
|
- `tests/graph/builder.test.js`
|
|
- `tests/graph/orchestrator.test.js`
|
|
- `tests/graph/prompt-builder.test.js`
|
|
- `tests/graph/schema.test.js`
|
|
- `tests/graph/utils.test.js`
|
|
- `tests/v03-reasoning.test.js` — **committed** to current branch
|
|
- `tests/smoke.test.js`
|
|
|
|
### Config changes
|
|
- `package.json` — added dependency (verify which one)
|
|
- `playwright.config.js` — added/modified for integration testing
|
|
- `.env.local` — exists locally (not committed)
|
|
|
|
---
|
|
|
|
## 9. How to Run
|
|
|
|
```bash
|
|
# Install dependencies
|
|
npm install
|
|
|
|
# Unit tests
|
|
npx vitest run
|
|
|
|
# Graph library tests (has 5 failures)
|
|
npx vitest run tests/graph/ --reporter=verbose
|
|
|
|
# Start dev server
|
|
npm run dev
|
|
|
|
# API endpoints
|
|
# POST /api/start-case → { scenario: "..." }
|
|
# POST /api/update-case → { graph: {...}, answer: "...", ... }
|
|
```
|
|
|
|
---
|
|
|
|
## 10. What to Do First (Recommended Priorities)
|
|
|
|
1. **Review and fix the 5 failing tests** — likely simple string/fixture issues
|
|
2. **Decide on the untracked files** — commit them, or create a v0.4 branch from this point
|
|
3. **Verify the LLM provider integration** — ensure `getProvider()` and `generateReconstruction()` are wired up correctly
|
|
4. **Add env var documentation** for `RECONSTRUCTION_PROMPT_VERSION` to `.env.example`
|
|
5. **Smoke test end-to-end** — call `/api/start-case` with a real scenario and verify the full flow
|
|
|
|
---
|
|
|
|
*End of handoff.*
|