experiment: archive historical project documents

2026-08-06 14:24:52 +01:00
parent 354ba26aad
commit 97e4f3029e
11 changed files with 86 additions and 11 deletions
@@ -0,0 +1,26 @@
+# Archive Index — Confidence Engine
+
+> Archived means retained as historical evidence and excluded from normal context loading. It does not mean deleted, rejected or necessarily incorrect for its time.
+
+All files below were moved from `docs/` on 2026-08-06 by Experiment 29 to reduce the default reading burden while preserving full traceability.
+
+## Archived Files
+
+| Original Path | Archive Path | What It Contains | Why Archived | When to Consult |
+|---|---|---|---|---|
+| `docs/v0.4-handoff.md` (258 lines) | `docs/archive/v0.4-handoff.md` | Historical handoff document from the v0.4 transition; references CaseOrchestrator API. | Architecture has evolved since v0.4. Documented for reference only, not active guidance. | When tracing the origin of case-orchestration patterns or investigating historical API design decisions. (Also referenced in `docs/orchestrator-contract.md`.) |
+| `docs/v0.4-route-status.md` (25 lines) | `docs/archive/v0.4-route-status.md` | Historical route tracking for the v0.4 release cycle. | Current routes differ entirely from v0.4. Retained as a record of early routing assumptions. | When investigating why certain routing decisions were made in early versions. |
+| `docs/v0.5-release-notes.md` (58 lines) | `docs/archive/v0.5-release-notes.md` | Release notes documenting the state of v0.5. | Historical record only. Nothing active depends on this content. | When comparing v0.5 to later releases or verifying what was known at that release time. |
+| `docs/v0.6-ambiguity-generalisation.md` (40 lines) | `docs/archive/v0.6-ambiguity-generalisation.md` | v0.6 experiment on ambiguity generalisation. | Superseded by later reasoning architecture decisions from Experiments 15–25B. | When investigating the intellectual history of how the engine handles ambiguous inputs. |
+| `docs/v0.7-observation-report.md` (136 lines) | `docs/archive/v0.7-observation-report.md` | Experimental observation snapshot from v0.7 UX work. | Useful as a reference but not a current working document. UX work is paused. | When reviewing past UX observations that may inform future interface design decisions. |
+
+## Files Deliberately Not Archived
+
+| Document | Why Left in Place |
+|---|---|
+| `docs/architectural-principles.md` (306 lines) | 14 architectural principles derived from experiments; may be needed when re-engaging with reasoning architecture. Status: unclear how current it is — review before use but do not archive yet. |
+| `docs/backlog info.md` (390 lines) | Mock fixture backlog useful if resuming UI development. Status: verify content is current before archiving. |
+
+## Usage
+
+Load these files only when a specific experiment, version history, or past decision requires them. Use this index to locate archived material — do not read the archive directory by default.
@@ -0,0 +1,258 @@
+# v0.4 Handoff — Confidence Engine (confidence-engine)
+
+**Date:** 2026-08-01
+**Branch:** `feature/reconstruction-v0.3`
+**Parent branch:** `main`
+
+---
+
+## 1. What This Project Is
+
+A Next.js app that performs evidence-based situation reconstruction on user-supplied scenarios. An LLM analyses the scenario, builds a directed graph of actors, systems, unknowns and relationships, then iteratively refines the graph through multi-turn Q&A with the user.
+
+---
+
+## 2. Recent Commit History
+
+| Commit | Message |
+|--------|---------|
+| `79ea2f6` | feat: add v0.3 normalised comparison reasoning |
+| `d72c7c5` | chore: establish clean v0.2 baseline |
+| `a2f9e47` | chore: preserve initial reconstruction prototype |
+
+Only **one commit** ahead of `main`: `79ea2f6` — the v0.3 normalised comparison reasoning work.
+
+---
+
+## 3. Current State Summary
+
+### What's done and committed to this branch
+
+1. **v0.3 prompt** (`prompts/reconstruct-v0.3.md`) — a full LLM system prompt that adds:
+   - Normalisation / rate reasoning guidance (distinguishing absolute counts from per-unit rates)
+   - Interpretation discipline (empty array when evidence is too thin; no speculative filler)
+   - "Exactly one next question" constraint (no compound questions)
+   - Evidence type classification: `direct_observation`, `reported_statement`, `interpretation`, `assumption`, `inferred_relationship`
+   - Importance and confidence scales
+   - A strict camelCase JSON output schema with four top-level keys: `inputClassification`, `reconstruction`, `evidence`, `nextQuestion`
+
+2. **v0.3 prompt versioning** (`lib/reconstruction/prompt.js`) — exports `PROMPT_VERSIONS`, `DEFAULT_PROMPT_VERSION ("v0.3")`, and `buildPrompt(scenario, version)` for loading prompt templates from disk with scenario substitution.
+
+3. **Schema validation** (`lib/reconstruction/schema.js`) — Zod schemas for v0.2 output (`reconstructionV2Schema`). A `parseReconstructionV2(rawString)` helper is used in the analysis pipeline.
+
+4. **v0.3 reasoning tests** (`tests/v03-reasoning.test.js`) — extensive test suite covering:
+   - Prompt version registration and loading
+   - v0.3 guidance completeness (normalisation, rate vs count, correlation-vs-causation)
+   - Schema validation with a realistic "production/complaints" fixture
+   - Parse helper tests
+
+5. **Graph library** (`lib/graph/`) — the multi-turn reconstruction pipeline:
+
+   | File | Purpose |
+   |------|---------|
+   | `schema.js` | Zod schemas for SituationNode, SituationEdge, SituationGraph, GraphUpdate; helpers like `makeNodeId`, `makeNode`, `makeEdge`, `makeGraph` |
+   | `builder.js` | `buildInitialGraph(reconstruction, evidence)` — converts v0.2/v0.3 analysis output into a SituationGraph with deterministic nodes/edges; `buildMinimalGraph(scenario)` for fallback; `describeGraph(graph)` for display |
+   | `orchestrator.js` | `CaseOrchestrator` class managing the full multi-turn lifecycle (idle → building → active); exports `startCase(body)` and `updateCase(body)` convenience functions for API routes |
+   | `prompt-builder.js` | `buildUpdatePrompt(ctx)` — formats current graph state + Q&A context into a system prompt for the LLM update-evaluation turn |
+   | `utils.js` | Deterministic graph operations: `validateGraphReferences`, `detectDuplicateNodeIds`, `detectDuplicateEdges`, `findDependentNodes`, `findAffectedNodes`, `resolveUnknownNode`, `selectActiveUnknownCandidate`, `applyGraphUpdate`, `validateGraphUpdate` |
+
+6. **API routes** (`app/api/`)
+
+   | Route | Purpose |
+   |-------|---------|
+   | `POST /api/start-case` | Start a new reconstruction case — accepts `{ scenario, promptVersion? }`, returns graph summary, node/edge counts, next question |
+   | `POST /api/update-case` | Process a turn — accepts `{ scenario, graph, answer, currentQuestion?, turnCount?, modelName? }`, returns updated graph summary, next question, changes summary |
+
+7. **Smoke test** (`tests/smoke.test.js`) — basic integration test for the start-case API route.
+
+### What's NOT yet committed (untracked files from git status)
+
+| File | Description |
+|------|-------------|
+| `lib/graph/` (full directory) | The multi-turn graph library — built but NOT yet committed to any branch. These are the new untracked files: `builder.js`, `orchestrator.js`, `prompt-builder.js`, `schema.js`, `utils.js` |
+| `tests/graph/` (full directory) | Tests for the graph library — also untracked: `builder.test.js`, `orchestrator.test.js`, `prompt-builder.test.js`, `schema.test.js`, `utils.test.js` |
+| `app/api/start-case/route.js` | New API route (untracked) |
+| `app/api/update-case/route.js` | New API route (untracked) |
+
+> **Important:** The git status shows these files as untracked (`??`). They exist on disk but have never been staged or committed. You need to decide whether to commit them now or integrate them differently.
+
+---
+
+## 4. Test Status
+
+```
+Test Files:  4 failed | 4 passed (8)
+Tests:       5 failed | 216 passed (221)
+```
+
+### Known failures
+
+The failures cluster in `tests/graph/`:
+- **`prompt-builder.test.js`** — test expects the literal string `"Existing or newly added nodes"` but the prompt template currently says `"existing or newly added nodes"` (case mismatch). The SYSTEM_PROMPT_HEADER constant uses lowercase.
+- Other graph tests likely have similar fixture/reference issues.
+
+Run `npx vitest run tests/graph/ --reporter=verbose` for full details.
+
+---
+
+## 5. Architecture Overview
+
+```
+User scenario
+      │
+      ▼
+┌──────────────┐     ┌─────────────────┐     ┌──────────────┐
+│ analyseScenario│──▶│ buildPrompt     │──▶│ LLM (v0.3)   │
+│  (lib/analysis.js) │  (reconstruction/prompt.js) │              │
+└──────────────┘     └─────────────────┘     └──────┬───────┘
+                                                      │
+                                                      ▼
+                                              ┌──────────────┐
+                                              │ Parse output │
+                                              │ (Zod/parse   │
+                                              │  Reconstruction│
+                                              │  V2)          │
+                                              └──────┬───────┘
+                                                     │
+                     ┌───────────────────────────────┤
+                     ▼                               ▼
+              ┌──────────────┐           ┌──────────────────┐
+              │buildInitialGraph│         │ buildMinimalGraph │
+              │  (graph/builder)│         │  (fallback)       │
+              └──────┬─────────┘           └──────────────────┘
+                     │
+                     ▼
+              ┌──────────────┐
+              │SituationGraph │  ← Zod-validated graph structure
+              │  {nodes, edges}│     nodes: observation/metric/unknown/...
+              └──────┬───────┘     edges: supports/weakens/causes/...
+                     │
+          (multi-turn loop via updateCase)
+                     │
+           ┌─────────▼─────────┐
+           │buildUpdatePrompt  │  → LLM proposes GraphUpdate
+           │                   │
+           │applyGraphUpdate   │  → deterministic, validated
+           │validateGraphUpdate│     (no direct LLM mutation)
+           └───────────────────┘
+```
+
+---
+
+## 6. Key Design Decisions
+
+### Normalisation / rate reasoning (v0.3 focus)
+The v0.3 prompt explicitly instructs the model to:
+- Always consider whether a denominator/exposure metric is needed when counts change alongside scale
+- Distinguish absolute count from rate
+- Avoid treating two rising counts as causal evidence (production growth may outpace complaint growth)
+- Request the per-unit metric as the highest-value next question
+
+### Graph immutability
+LLM proposals are never applied directly. All mutations go through `applyGraphUpdate()` in `lib/graph/utils.js`, which:
+- Validates all node/edge references exist
+- Rejects duplicate IDs
+- Enforces a max graph size (500 nodes) and update size (100KB)
+- Returns the full new state for validation
+
+### Prompt versioning
+- Default is `"v0.3"` but `PROMPT_VERSIONS` includes `"v0.2"` for backward compatibility
+- `RECONSTRUCTION_PROMPT_VERSION` env var can override default at module load time
+- Prompts are loaded from `prompts/reconstruct-v0.{version}.md` on disk
+
+### Deterministic node IDs
+Node IDs are computed via a deterministic hash of the label: `makeNodeId(label)`. This avoids conflicts but means nodes must be created with consistent labels to get consistent IDs.
+
+---
+
+## 7. Open Questions / TODOs for Next Developer
+
+1. **Untracked graph library** — `lib/graph/` and `tests/graph/` are untracked on disk. Do we commit them as part of v0.4, or keep them in a separate branch?
+
+2. **Test failures** — 5 tests fail across the graph test suite. The prompt-builder case-sensitivity issue needs fixing. Review all failing tests before merging.
+
+3. **Missing `RECONSTRUCTION_PROMPT_VERSION` env var docs** — The system uses an env var override but it's not documented in `.env.example`. Add it if it's intended to be configurable.
+
+4. **Provider integration** — `lib/llm/provider.js` is imported by the orchestrator (`getProvider()`, `generateReconstruction()`). Verify the provider implementation matches what this code expects.
+
+5. **Graph completeness heuristic** — `CaseOrchestrator.getCompletionStatus()` returns `"complete"` when no unknown nodes remain, but doesn't consider whether all important observations have been verified.
+
+6. **Error resilience in update flow** — If the LLM returns malformed JSON, the update route returns a 500 with a generic error message. Consider retry logic or structured error parsing.
+
+7. **`buildUpdatePrompt` SYSTEM_PROMPT_HEADER is a module-level constant** — it's hardcoded and never versioned. If v0.5 changes the update-evaluation prompt style, this will need to become a template.
+
+8. **The `nextQuestion` field on `/api/start-case` response** includes the adapted question (original + active unknown label appended). The client may want the original and adapted separately.
+
+---
+
+## 8. File Inventory (new / changed files on this branch)
+
+### Prompts
+- `prompts/reconstruct-v0.3.md` — **NEW** — v0.3 system prompt (161 lines)
+- `prompts/reconstruct-v0.2.md` — **existing** — baseline prompt
+
+### Core library
+- `lib/analysis.js` — **MODIFIED** — analyseScenario function (uses v0.3 prompt by default)
+- `lib/reconstruction/prompt.js` — **MODIFIED** — prompt versioning exports
+- `lib/reconstruction/schema.js` — **existing** — Zod schemas + parseReconstructionV2
+
+### Graph library (untracked on disk)
+- `lib/graph/builder.js` — buildInitialGraph, buildMinimalGraph, describeGraph
+- `lib/graph/orchestrator.js` — CaseOrchestrator class, startCase, updateCase
+- `lib/graph/prompt-builder.js` — buildUpdatePrompt + SYSTEM_PROMPT_HEADER
+- `lib/graph/schema.js` — SituationNode/Edge/Graph/Update Zod schemas
+- `lib/graph/utils.js` — validation, dedup, dependency, and apply utilities
+
+### API routes (untracked on disk)
+- `app/api/start-case/route.js`
+- `app/api/update-case/route.js`
+
+### Tests (untracked on disk)
+- `tests/graph/builder.test.js`
+- `tests/graph/orchestrator.test.js`
+- `tests/graph/prompt-builder.test.js`
+- `tests/graph/schema.test.js`
+- `tests/graph/utils.test.js`
+- `tests/v03-reasoning.test.js` — **committed** to current branch
+- `tests/smoke.test.js`
+
+### Config changes
+- `package.json` — added dependency (verify which one)
+- `playwright.config.js` — added/modified for integration testing
+- `.env.local` — exists locally (not committed)
+
+---
+
+## 9. How to Run
+
+```bash
+# Install dependencies
+npm install
+
+# Unit tests
+npx vitest run
+
+# Graph library tests (has 5 failures)
+npx vitest run tests/graph/ --reporter=verbose
+
+# Start dev server
+npm run dev
+
+# API endpoints
+# POST /api/start-case    → { scenario: "..." }
+# POST /api/update-case   → { graph: {...}, answer: "...", ... }
+```
+
+---
+
+## 10. What to Do First (Recommended Priorities)
+
+1. **Review and fix the 5 failing tests** — likely simple string/fixture issues
+2. **Decide on the untracked files** — commit them, or create a v0.4 branch from this point
+3. **Verify the LLM provider integration** — ensure `getProvider()` and `generateReconstruction()` are wired up correctly
+4. **Add env var documentation** for `RECONSTRUCTION_PROMPT_VERSION` to `.env.example`
+5. **Smoke test end-to-end** — call `/api/start-case` with a real scenario and verify the full flow
+
+---
+
+*End of handoff.*
@@ -0,0 +1,25 @@
+# v0.4 Route Status
+
+- `app/api/cases/start/route.js`
+  - Current tracked start-case route for the v0.4 graph orchestration path.
+  - Covered by `tests/app/api/cases-start-route.test.js`.
+
+- `app/api/cases/update/route.js`
+  - Current tracked update-case route for the v0.4 graph orchestration path.
+  - Delegates to `updateCase(body, { applyProposal: true })`.
+  - Covered by `tests/app/api/cases-update-route.test.js`.
+
+- `app/api/start-case/route.js`
+  - Earlier experiment / duplicate start route.
+  - No repository UI/test references were found.
+  - Deleted from the working tree during UI connection cleanup.
+
+- `app/api/update-case/route.js`
+  - Earlier experimental duplicate update route.
+  - Removed from the working tree during route consolidation.
+
+- Current UI status
+  - `components/scenario-form.jsx` now calls `/api/cases/start` for the main experimental flow.
+  - `/api/cases/update` is the active tracked update route.
+  - `/api/analyse` remains available for legacy one-shot analysis.
+  - No UI changes were required for this route milestone.
@@ -0,0 +1,58 @@
+# v0.5 Release Notes
+
+## Purpose of v0.5
+
+v0.5 stabilises the graph-backed one-turn update flow so the engine can resolve an answered unknown, surface consequential new unknowns, prioritise the next unknown deterministically, and formulate a deterministic follow-up question without changing the UI or adding more model turns.
+
+## Capabilities proven
+
+v0.5 includes:
+
+- resolving an existing unknown
+- surfacing consequential new unknowns
+- limiting emergent unknowns
+- deterministic information-value prioritisation
+- deterministic question formulation
+- generalisation across five decision types
+- graph-backed one-turn UI update
+
+## Five-case generalisation result
+
+All five deterministic fixture scenarios passed:
+
+1. Should we hire another engineer?
+2. Should we replace the delivery vans?
+3. Should we launch in another country?
+4. Should we continue a project that is over budget?
+5. Should we introduce a paid support tier?
+
+The selector chose a foundational unknown first in each case, avoided the downstream leaf first, required no model call, and preserved graph immutability during question formulation.
+
+## Key deterministic safeguards
+
+- proposal application re-selects the active unknown deterministically after validation
+- information-value scoring penalises downstream or prerequisite-blocked unknowns
+- emergent unknown validation limits additions and requires explicit answer-derived linkage
+- final question wording is reformulated from graph context without an extra model turn
+- question validation rejects compound, awkward, or pricing-led fallback phrasing
+
+## Known limitation
+
+A correctly selected threshold node can still be phrased using an actor/customer strategy when surrounding graph context strongly references customers or value recipients.
+
+This limitation is recorded for the next experiment and is not being fixed in the v0.5 release-prep task.
+
+## Deliberately excluded work
+
+- no reasoning-logic expansion beyond the small deterministic formulation fixes already landed on the branch
+- no new features
+- no UI changes
+- no persistence
+- no additional model turn
+- no Ollama calls for validation
+- no evaluator-suite runs
+- no Playwright runs
+
+## Next experimental question
+
+Can the question formulation strategy remain aligned with the selected node's role when surrounding graph context contains competing signals?
@@ -0,0 +1,40 @@
+# v0.6 Ambiguity Generalisation
+
+## Hypothesis
+
+If the selector truly handles unjustified contradiction ties generically, it should return ambiguity across multiple domains without preferring one explanation by wording alone.
+
+## Scenarios
+
+1. Revenue increased by 18%, but cash in the bank fell over the same period.
+2. Customer satisfaction scores increased, but complaints also increased.
+3. Average delivery time decreased by 25%, but order cancellations increased.
+4. Website traffic doubled, but sales remained unchanged.
+5. Production output increased by 30%, but quality defects also increased.
+
+## Observed behaviour
+
+All five fixtures produced the same pattern:
+
+- candidate count: 2
+- selector status: `ambiguous`
+- tie reason: `No justified distinction between leading unknowns.`
+- no explanation was favoured
+- one broad investigation question was produced from the central contradiction
+- neutral label renaming did not collapse ambiguity into a winner
+
+## Repeated failure patterns
+
+None observed across two or more scenarios.
+
+The current ambiguity handling generalised cleanly across the five contradiction fixtures.
+
+## Corrections
+
+No production correction was required in this task.
+
+## Lessons learned
+
+- The current ambiguity path appears domain-agnostic when structure and semantic weights remain intentionally non-discriminating.
+- Central-statement-based tie questions are broad enough to avoid prematurely backing one branch.
+- The most useful regression signal is whether ambiguity survives neutral relabelling, not whether one label sorts ahead of another in display order.
@@ -0,0 +1,136 @@
+# v0.7 Observation Report
+
+**Date**: 2026-08-03 | **Commit**: c273209 | **Branch**: feature/reasoning-pattern-memory-v0.7
+
+## Summary Table
+
+| Scenario | Name | Start | Update | Nodes | Unknowns | Rating |
+|----------|------|-------|--------|-------|----------|--------|
+| scenario-1 | Confidence Engine commercial validation | pass | fail(400) | 9 | 3 | flow failure |
+| scenario-2 | Hiring | pass | fail(400) | 18 | 8 | flow failure |
+| scenario-3 | Vehicle replacement | pass | fail(400) | 15 | 8 | flow failure |
+| scenario-4 | Welsh Government-style programme decision | pass | fail(400) | 10 | 3 | flow failure |
+| scenario-5 | Operational contradiction | pass | fail(400) | 7 | 2 | flow failure |
+| scenario-6 | Personal decision | fail | skipped | 0 | 0 | flow failure |
+
+## Per-Scenario Findings
+
+### scenario-1: Confidence Engine commercial validation
+
+- **Overall**: Start=pass, Update=fail(400), Rating=flow failure
+- Pattern: N/A | Nodes: 9 | Edges: 0
+- Validation: valid | Duration: 63386ms
+- Unknown IDs: nirkgb4, n36c0cc, nzeyzkz
+- Error: [N/A] Invalid update-case request
+
+- **Assessment**:
+  - reasoning-pattern fit: fail
+  - one-concept simplicity: fail
+  - plain-language clarity: fail
+  - logical progression: fail (No question generated)
+  - graph-backed: fail
+  - premature-specialism avoided: fail
+
+### scenario-2: Hiring
+
+- **Overall**: Start=pass, Update=fail(400), Rating=flow failure
+- Pattern: N/A | Nodes: 18 | Edges: 5
+- Validation: valid | Duration: 146476ms
+- Unknown IDs: n7yonyv, npci7a7, nug9wj2, nz0vpey, nz8pwyc, newxmzu, nw14mjj, n25mnp3
+- Error: [N/A] Invalid update-case request
+
+- **Assessment**:
+  - reasoning-pattern fit: fail
+  - one-concept simplicity: fail
+  - plain-language clarity: fail
+  - logical progression: fail (No question generated)
+  - graph-backed: fail
+  - premature-specialism avoided: fail
+
+### scenario-3: Vehicle replacement
+
+- **Overall**: Start=pass, Update=fail(400), Rating=flow failure
+- Pattern: N/A | Nodes: 15 | Edges: 5
+- Validation: valid | Duration: 81460ms
+- Unknown IDs: ng5yr11, nogqips, n499gin, n8fbv3p, nf2f6zx, n4feiap, nvwthlt, nqrxjli
+- Error: [N/A] Invalid update-case request
+
+- **Assessment**:
+  - reasoning-pattern fit: fail
+  - one-concept simplicity: fail
+  - plain-language clarity: fail
+  - logical progression: fail (No question generated)
+  - graph-backed: fail
+  - premature-specialism avoided: fail
+
+### scenario-4: Welsh Government-style programme decision
+
+- **Overall**: Start=pass, Update=fail(400), Rating=flow failure
+- Pattern: N/A | Nodes: 10 | Edges: 0
+- Validation: valid | Duration: 129682ms
+- Unknown IDs: nrrm3qn, nefmpat, n6rtwg1
+- Error: [N/A] Invalid update-case request
+
+- **Assessment**:
+  - reasoning-pattern fit: fail
+  - one-concept simplicity: fail
+  - plain-language clarity: fail
+  - logical progression: fail (No question generated)
+  - graph-backed: fail
+  - premature-specialism avoided: fail
+
+### scenario-5: Operational contradiction
+
+- **Overall**: Start=pass, Update=fail(400), Rating=flow failure
+- Pattern: N/A | Nodes: 7 | Edges: 0
+- Validation: valid | Duration: 70579ms
+- Unknown IDs: n6gm2cv, nylhu9g
+- Error: [N/A] Invalid update-case request
+
+- **Assessment**:
+  - reasoning-pattern fit: fail
+  - one-concept simplicity: fail
+  - plain-language clarity: fail
+  - logical progression: fail (No question generated)
+  - graph-backed: fail
+  - premature-specialism avoided: fail
+
+### scenario-6: Personal decision
+
+- **Overall**: Start=fail, Update=skipped, Rating=flow failure
+- Pattern: N/A | Nodes: 0 | Edges: 0
+- Validation: invalid | Duration: 72547ms
+
+- **Assessment**:
+  - reasoning-pattern fit: fail
+  - one-concept simplicity: fail
+  - plain-language clarity: fail
+  - logical progression: fail (No question generated)
+  - graph-backed: fail
+  - premature-specialism avoided: fail
+
+## Failure Pattern Analysis
+
+### Start Phase
+- **5/6 succeeded**, 1/6 failed
+  - scenario-6: Scenario analysis failed
+
+### Update Phase
+- **0/6 succeeded**, 5/6 failed, 1/6 skipped
+
+- **N/A** (5 failures):
+  - scenario-1: Invalid update-case request
+  - scenario-2: Invalid update-case request
+  - scenario-3: Invalid update-case request
+  - scenario-4: Invalid update-case request
+  - scenario-5: Invalid update-case request
+
+## What's Stable
+
+- ✅ **Graph construction**: 5/6 start success across all scenario types (commercial, operational, personal, policy)
+
+## Recommendations
+
+1. **Fix update failures** (5/6): Primary focus area. Most failures in proposal_compatibility and delta detection.
+- Monitor reasoning pattern inference reliability across different scenario domains.
+- Consider adding timeout guards for long-running LLM calls (some exceeded 60s).