1760 lines
111 KiB
Markdown
1760 lines
111 KiB
Markdown
# Current Handoff — Confidence Engine
|
||
|
||
> **Role:** Concise operational snapshot for resuming work today. Not a historical diary.
|
||
> The design evolution archive index at `docs/design-evolution/README.md` provides progressive loading of experiment history; load the relevant chapter only when a specific historical question requires it.
|
||
|
||
## Repository checkpoint
|
||
|
||
- **Branch:** `feature/investigation-report-v0.55`
|
||
- **HEAD:** `37a9a12` — route design evolution provenance through archive index
|
||
- **Working tree:** will be clean after v0.59a commit
|
||
|
||
## v0.5 initial reconstruction relationship contract
|
||
|
||
- v0.5 first-class initial reconstruction relationship contract implemented.
|
||
- Reconstruction relationships use explicit `fromId`/`toId` references.
|
||
- Declared relationships project deterministically to existing SituationGraph edges.
|
||
- Invalid relationship references are skipped rather than inferred or repaired.
|
||
- SituationGraph schema unchanged; v0.5 is the production default.
|
||
- Targeted deterministic suite passed; live v0.5 semantic/reliability validation remains pending.
|
||
- 502 structured-output reliability remains a separate unresolved issue.
|
||
|
||
## Initial reconstruction observability
|
||
|
||
- Successful `/api/cases/start` responses now expose the validated initial `reconstruction`, the exact object used by `buildInitialGraph()`.
|
||
- Semantic experiments can compare `reconstruction.relationships` directly with projected `e-rel-*` SituationGraph edges; this is observability only and does not change reasoning semantics.
|
||
- Owning deterministic tests pass. Ten broader orchestrator failures were observed and classified as unrelated to this increment, but were not proven pre-existing because non-mutating baseline execution was blocked by the existing Vitest apparatus.
|
||
- No live validation has been rerun against this observability boundary. Next restart point: exactly one production-default `/api/cases/start` call using the fixed manufacturing scenario, comparing reconstruction relationship E directly with its projected graph edge.
|
||
|
||
## v0.5 relationship direction contract
|
||
|
||
- Directly observable v0.5 live validation proved first-class relationship declaration and deterministic projection work end-to-end, while exposing one `may_cause` directional inconsistency between endpoints and description.
|
||
- v0.5 now defines relationships literally as `fromId → relationship → toId`: directional semantics are explicit for `depends_on`, `causes`, `may_cause`, `supports`, and `weakens`; `compares_with` may use either endpoint order.
|
||
- The read-back rule requires relationship type, endpoint direction, and description to agree. Provenance and interpretation-separation rules remain unchanged.
|
||
- Deterministic tests prove the instruction exists, not that the model obeys it. No live call has tested this refinement; next restart point is live directional-integrity validation before broader repeatability testing.
|
||
|
||
## Unexplained-transition projection coverage
|
||
|
||
- Starting checkpoint: `e183914`. The latest one-call v0.5 validation emitted two reconstruction relationships: one directionally clean relationship projected, while the other (`r1`) was semantically/directionally incoherent and lost because it referenced an `unexplainedTransition` without graph projection.
|
||
- `unexplainedTransitions` now project as unresolved transition nodes and register their source IDs for existing relationship projection; the relationship architecture itself was not redesigned.
|
||
- The semantic coherence of live `r1` remains a separate unresolved reasoning issue; intervention-fit E was absent in that run and remains unresolved. No live validation has occurred after this deterministic fix.
|
||
|
||
## Intervention-fit dependency preservation
|
||
|
||
- First-class relationship generation and deterministic projection now work end-to-end; the latest live run projected all three declared relationships correctly.
|
||
- The remaining semantic weakness was substitution: narrower decision prerequisites were preserved, but the full supplied intervention-fit dependency was not. v0.5 now requires preservation of the user's actual dependency level; narrower evidence prerequisites may coexist but may not replace that higher-order contingency.
|
||
- No schema, builder, or orchestrator change was required. Deterministic tests prove this instruction exists only; live compliance remains untested. Next restart point: one live intervention-fit validation, followed by bounded repeatability if it passes.
|
||
|
||
## Reconstruction validation failure observability
|
||
|
||
- The latest one-call intervention-fit validation returned HTTP 502: a real structured-output product failure, so intervention-fit semantic compliance was not evaluable.
|
||
- Previous diagnostics flattened the failure to `reconstruction: Required` and truncated raw evidence. Failure responses now retain exact validation issue paths, codes, messages, available native metadata, and complete diagnostic raw response at the existing validation boundary.
|
||
- Reasoning prompt, schema acceptance, and provider behaviour remain unchanged; no live calls occurred in this increment. Next restart point: one production-default manufacturing call—evaluate E if it succeeds, or use the new evidence to diagnose the exact structured-output defect if it fails.
|
||
|
||
## Relationship endpoint reference integrity
|
||
|
||
- The latest valid live run semantically preserved intervention-fit E in relationship `r5` with correct direction, but `r5.fromId` referenced a semantic-unit ID never emitted, so the non-inferential builder correctly skipped projection.
|
||
- This is a generation-time relationship-reference integrity defect. v0.5 now requires semantic units before relationships and exact reuse of emitted endpoint IDs; legitimate missing endpoint concepts must be materialised first under existing provenance rules.
|
||
- The builder remains unchanged. Live compliance remains untested; next restart point is one production-default relationship-reference-integrity validation.
|
||
|
||
## Case-start error logging
|
||
|
||
- The latest live validation was blocked by an HTTP 500 route-level exception with generic client output. `/api/cases/start` now writes structured Next.js server-console diagnostics for every route-owned error response: 4xx at warning level and 5xx/502 at error level.
|
||
- Caught exceptions log their actual exception, message, and stack while the client response remains generic; successful responses are not logged. No reasoning, status, or response semantics changed.
|
||
- Next restart point: one production-default manufacturing call while observing the Next.js dev-server terminal.
|
||
|
||
## Chat capability detection
|
||
|
||
- Direct POST `/api/chat` was proven supported. The false-negative cause was the capability probe using model `dummy-check`, which conflated model availability with endpoint capability.
|
||
- The probe now uses the configured model; live behaviour after this fix remains untested.
|
||
|
||
## Failed provider-path observability
|
||
|
||
- The detector fix remains unverified live; a subsequent HTTP 500 left the actually attempted provider path unobservable.
|
||
- Case-start failure diagnostics now preserve `providerApiPath` only when the provider reports an endpoint actually attempted during that request. The next step remains one production call.
|
||
|
||
## Successful provider-path observability
|
||
|
||
- The 502 live run at `dabd9e2` was PATH NOT OBSERVABLE ON FAILURE: provider path survived provider errors but not successful-provider/later-validation failures.
|
||
- Successful provider results now retain their factual attempted path through later reconstruction-validation failure diagnostics. This corrects only that observability seam; the next unknown remains one real `/api/cases/start` call establishing `/api/chat` versus `/api/generate`.
|
||
|
||
## Zod 4 migration
|
||
|
||
- Zod upgraded from 3.25.76 to 4.5.4. Reconstruction and direct Zod-owner suites pass, and native `z.toJSONSchema()` converts the production reconstruction schema with required closed enum constraints.
|
||
- Zod-3-specific error-message assertions are version-independent while retaining semantic validation evidence. Production schema semantics are unchanged and no third-party JSON-schema conversion dependency was added.
|
||
- Reconstruction `/api/chat` now sends `z.toJSONSchema(reconstructionV2Schema)` rather than generic `format: "json"`; the same canonical Zod contract constrains generation and remains the post-hoc validator.
|
||
- The deterministic provider-boundary test passes 2/2 under Node. A direct configured Ollama/Qwen `/api/chat` call accepted and obeyed the full schema in one call; `/api/generate`, chat detection, fallback, prompt, temperature, retries, and schema semantics are unchanged.
|
||
- Production repeatability remains untested after this change. Next restart point: a small repeated `/api/cases/start` stability observation using the fixed manufacturing scenario.
|
||
|
||
## Reconstruction fallback diagnostics
|
||
|
||
- Schema-constrained reconstruction `/api/chat` remains active. A later manual production 502 came through `/api/generate`; its unconstrained response omitted required transition confidence fields and failed Zod validation.
|
||
- Provider execution diagnostics now distinguish chat skipped due capability state, chat attempted and failed before generate fallback, and successful chat without fallback. They are deterministically verified through `/api/cases/start`; endpoint selection and fallback behaviour remain unchanged.
|
||
- Next restart point: one observation-only fixed-scenario production call to identify why `/api/generate` is reached.
|
||
|
||
## Chat capability body disposal
|
||
|
||
- A fresh-process debugger observation proved capability `/api/chat` returned HTTP 200, then invalid `res.body?.consume()` threw and the catch incorrectly cached `_chatSupported = false`.
|
||
- This skipped schema-constrained reconstruction `/api/chat` and used unconstrained `/api/generate`. Response-body disposal now uses Fetch-compatible consumption without changing capability or fallback semantics.
|
||
- Next boundary: one fresh-process fixed-scenario production observation.
|
||
|
||
## Reconstruction chat timeout
|
||
|
||
- Capability detection/body disposal was fixed at `7070342`. Fresh runtime evidence then proved the real schema-constrained `/api/chat` request was attempted but Confidence Engine aborted it after 60 seconds while Qwen was still generating.
|
||
- Fallback `/api/generate` completed but produced structurally invalid reconstruction. The real reconstruction `/api/chat` timeout is now 300 seconds, matching `/api/generate`; fallback, schema, prompt, and reasoning semantics are unchanged.
|
||
- Next boundary: one fresh-process fixed-scenario production observation.
|
||
|
||
## Rule 5a outcome-category baseline
|
||
|
||
- Five consecutive post-provider-fix production runs completed without 500/502, materially improving structural provider/schema stability.
|
||
- C — the late-delivery versus product-defect distinction — remains the clearest repeated semantic weakness. Rule 5a is now isolated as the first semantic experiment variable; unrelated prompt formatting changes were removed from the experimental diff.
|
||
- No live calls occurred in this increment. Next experiment may add a separate pre-output semantic preservation self-check on top of this clean baseline.
|
||
|
||
## Outcome-preservation salience variant
|
||
|
||
- `5c9f94c` established the clean Rule 5a baseline. C — the late-delivery versus product-defect distinction — remains the targeted semantic weakness.
|
||
- This increment adds only a final semantic-preservation self-check, testing instruction salience rather than missing semantic specification. No live model calls were made.
|
||
- Next boundary: one manual fixed-scenario Postman production run, scoring C first while checking A/B/D1/D2/E and unsupported expansion or causal strengthening for regressions.
|
||
|
||
## OpenAI reconstruction-provider experiment apparatus
|
||
|
||
- Structural Ollama/provider issues were previously fixed; current semantic comparison now warrants testing model-compliance variance.
|
||
- An experiment-only OpenAI Responses API provider apparatus exists for `gpt-5.6-terra`, using the same current v0.5 prompt and canonical reconstruction schema. Production provider selection remains Ollama/Qwen.
|
||
- No live OpenAI calls have occurred. Next boundary: a bounded live semantic comparison, not production migration.
|
||
|
||
## Reconstruction provider injection seam
|
||
|
||
- OpenAI provider apparatus exists at `860ee6f`; the canonical helper was previously blocked because `startCase()` and `analyseScenario()` created providers internally.
|
||
- An optional reconstruction-provider injection seam now carries an experiment provider through the same production analysis path. Production callers still default to configured Ollama/Qwen through `getProvider()`.
|
||
- No live calls occurred. Next boundary: exactly one live OpenAI fixed-scenario run through the canonical helper; this is not a production provider migration.
|
||
|
||
## Canonical helper alias-capable runtime
|
||
|
||
- Plain Node could not resolve repository `@/` aliases, and no plain-Node alias runtime existed. `tsx` is now a dev-only experiment runtime; the canonical command is `npm run experiment:start-case`.
|
||
- Its import-only proof succeeded with `startCaseResolved: true`: the real `startCase()` dependency chain resolves and no provider call occurs. Production runtime and the Ollama/Qwen default remain unchanged.
|
||
- Next boundary: exactly one live GPT-5.6 Terra reconstruction through this canonical helper.
|
||
|
||
## Current product architecture
|
||
|
||
Three distinct routes, not a single page:
|
||
|
||
```
|
||
/ → Portfolio (notebook index)
|
||
/investigations/case-1 → Investigation (working case/pages)
|
||
/investigations/case-1/report → Investigation Report (readable derived summary)
|
||
```
|
||
|
||
**Portfolio = investigator notebook index.** Shows the single canonical persisted investigation card with actions: *View report*, *Continue investigation*, *Restart investigation*. Below the card: *+ Create new investigation* (portfolio-level, not inside the card).
|
||
|
||
**Investigation = working case.** Contains `ScenarioForm` + `ReasoningWorkspace`. Handles graph reasoning, focused investigation turns, Done/Re-open semantics, Current Understanding synthesis. No Report presentation — that is owned by the dedicated Report route.
|
||
|
||
**Report = derived artefact.** Renders persisted `investigationReport` snapshot. Generation is on-demand, triggered by the Report page itself (not ReasoningWorkspace). Exactly one `/api/cases/overview` call on first visit; zero calls on subsequent visits. The Report is not canonical reasoning evidence — it is a derived summary for review/export/use.
|
||
|
||
## Current working product journey
|
||
|
||
```
|
||
Portfolio (/)
|
||
→ "Continue investigation"
|
||
→ Investigation page (/investigations/case-1)
|
||
→ Focused question asked → user answers → Done for now
|
||
→ Current Understanding synthesizes
|
||
→ If zero Open Questions: "Review current understanding" appears
|
||
OR → "Review current understanding"
|
||
→ Report page (/investigations/case-1/report)
|
||
→ Generates via /api/cases/overview (once)
|
||
→ Persists investigationReport
|
||
→ Subsequent visits render persisted snapshot (zero calls)
|
||
```
|
||
|
||
### Restart flow
|
||
|
||
Portfolio card *Restart investigation* → confirmation dialog (title: "Restart this investigation?") → destructive second confirmation → `clearInvestigation()` (canonical localStorage clear seam). No direct storage-key manipulation.
|
||
|
||
## Current reasoning / ownership invariants
|
||
|
||
**Evidence discipline:**
|
||
```
|
||
RAW USER EVIDENCE
|
||
≠ MODEL-DERIVED CONTRIBUTION SEMANTICS
|
||
≠ CURRENT CANONICAL FINDING
|
||
≠ IMMUTABLE SOURCE OBSERVATION
|
||
≠ USER DISPOSITION / AUTHORITY
|
||
≠ TURN CONTEXT / PROVENANCE
|
||
```
|
||
|
||
**Finding dispositions:** `null` (eligible working premise), `agree` (user-endorsed), `not_relevant` (excluded from eligible reasoning, provenance retained), corrected Finding. User disposition never directly mutates authoritative graph state.
|
||
|
||
**Evidence distinctions preserved by the persistence layer.** Contributions preserve: question/context, verbatim answer, model observations, uncertainties, assumptions, relationships, follow-ups, target/provenance, sequence/order. Findings preserve: canonical proposition, sourceObservation, contributionId, originatingTargetNodeId, userDisposition.
|
||
|
||
### Focused investigation presentation ownership (v0.52)
|
||
|
||
`FocusedQuestionBody` derives a thread-local subset (`targetNodeId || originatingTargetNodeId`) for every presentation surface. Previously answered content does NOT bleed from one question to another. Verified manually and by targeted Vitest.
|
||
|
||
### Empty Done + Re-open semantics (v0.53)
|
||
|
||
- Empty Done is valid: parks/resolves the question locally, does NOT invoke episode processing, does NOT produce `no_episodic_content` 400.
|
||
- Re-open returns the question to Open Questions and removes from `doneForNowIds`.
|
||
- Older stale development localStorage states (pre-v0.53 shape) may be discarded during current dev phase. No migration required.
|
||
|
||
### v0.59a — Investigation revision provenance rule
|
||
|
||
- Investigation has a semantic revision (`investigationRevision`).
|
||
- Meaningful persisted Investigation changes advance it by exactly 1.
|
||
- Persistence activity itself (autosave, save) does NOT advance revision.
|
||
- Report generation records `generatedFromRevision: investigationRevision`.
|
||
- Report generation advances revision by 0.
|
||
- Equal revisions mean Report reflects current Investigation.
|
||
- Different revisions mean the Investigation has changed since Report generation.
|
||
- Existing Report remains available (not invalidated).
|
||
- Report update/regeneration remains manual (user-triggered).
|
||
- Restart clears Investigation + Report via `clearInvestigation()` + `setInvestigationRevision(0)`.
|
||
- Report history/comparison remains deferred beyond MVP.
|
||
|
||
### v0.59b — Report freshness UI on the Report page
|
||
|
||
- Report page shows `Current` (matching revisions) or `Update available` (differing revisions).
|
||
- Manual `Update report` action on Report page triggers regeneration via `/api/cases/overview`.
|
||
- Duplicate prevention guard during in-flight update.
|
||
- No automatic regeneration.
|
||
- Explanation copy: "The investigation has changed since this report was generated."
|
||
|
||
### v0.59c — Report freshness state surfaced on the Portfolio
|
||
|
||
- Portfolio surfaces `Current` / `Update available` alongside existing View report link.
|
||
- Derives freshness solely from revision provenance (`investigationReport.generatedFromRevision === investigationRevision`).
|
||
- Zero model calls during Portfolio render.
|
||
- No mutation of Investigation.
|
||
- No `Update report` action on the Portfolio — manual Report updating remains owned by the Report page.
|
||
- If no Report, neither freshness state is shown.
|
||
|
||
**Semantic transitions that advance revision:**
|
||
```
|
||
Episode Done (with content) → +1
|
||
Re-open (resolved → unknown) → +1
|
||
Finding proposition correction → +1
|
||
Eligible → not_relevant → +1
|
||
not_relevant → eligible (restore) → +1
|
||
First meaningful change → =1 (from initial 0)
|
||
```
|
||
|
||
**Transitions that do NOT advance revision:**
|
||
```
|
||
Autosave → 0
|
||
Report generation → 0
|
||
Hydration → 0 (reads value)
|
||
Empty proposition correction (no-op) → 0
|
||
null → null disposition (no-op) → 0
|
||
```
|
||
|
||
### Zero Open Questions milestone (v0.51)
|
||
|
||
When all unknowns are resolved and clarified questions exist: "You've now worked through all of the questions we surfaced. Would you like to see an overview of what we understand so far?" with "Review current understanding" button. This occupies the former Open Questions position. The invitation is a milestone, not a readiness/completion judgement.
|
||
|
||
### Current Understanding refresh invariants
|
||
|
||
Reconstruct CU when canonical meaning or eligible evidence set changes — NOT when investigation/question status changes alone. Re-open ≠ change what we understand; Finding correction / Not Relevant / completed episode = change what we understand.
|
||
|
||
## Persistence and Report ownership
|
||
|
||
- **Canonical persistence owner:** `lib/storage/providers/local-storage.js` (`saveInvestigation` / `loadInvestigation`). All routes read from the same snapshot.
|
||
- **Report generation owner:** Report page only (NOT ReasoningWorkspace, NOT Investigation page).
|
||
- **localStorage key:** `confidence-engine-investigation` (single canonical key — multi-investigation not yet implemented).
|
||
- **Temporary identity:** `case-1`. True multi-investigation persistence/identity is future work.
|
||
- **Portfolio client hydration:** Portfolio page uses `'use client'` — initial pre-hydration empty state ≠ absence of persisted data. Always wait for hydrated semantic controls before classifying state.
|
||
|
||
## MVP boundaries (v0.59b)
|
||
|
||
**Implemented in MVP:**
|
||
- visible Report freshness state (`Current` / `Update available`)
|
||
- explanation that the Investigation has changed since Report generation
|
||
- manual `Update report` action (user-triggered)
|
||
- no automatic regeneration
|
||
- duplicate prevention guard during in-flight update
|
||
|
||
**Deferred beyond MVP:**
|
||
- Report history
|
||
- retaining multiple Reports
|
||
- Report comparison
|
||
- modelling/preview comparison between Investigation revisions
|
||
|
||
## Current development / verification constraints
|
||
|
||
- Canonical dev server at `http://localhost:3000`. Never start/stop/restart/probe it. If unavailable → BLOCKED and stop.
|
||
- Playwright MCP: use **Run Playwright code** with semantic locators for known controls. Snapshot refs (`[ref=...]`) are observational only.
|
||
- For async/hydration states: use `waitFor({ state: 'visible', timeout })` — not arbitrary sleeps.
|
||
- If a prescribed semantic locator cannot find its control → STOP. No fallback to CSS/XPath/DOM traversal.
|
||
- Live freeze: once Playwright verification begins, no production file edits until evidence is captured and classified.
|
||
- **Tests are instruments, not product truth.** At first deterministic failure: classify PRODUCT vs APPARATUS failure, then stop. Do not enter test-harness repair loops.
|
||
- Mocked boundary ownership: if a lower-layer function is mocked, test the value crossing the seam — do not require the mock to reproduce real implementation.
|
||
|
||
## Current limitations / genuinely open boundaries
|
||
|
||
**Not yet implemented:**
|
||
|
||
- Multi-investigation portfolio (search/tag/archive/group)
|
||
- Durable investigation identities beyond `case-1`
|
||
- Export/copy of Reports to Jira or external document
|
||
- Portfolio expansion beyond one canonical investigation
|
||
- Report history / comparison
|
||
|
||
**Known boundaries:**
|
||
|
||
- Current Understanding and Investigation Report are architecturally distinct artefacts. Plausible interpretations in the Report remain explicitly interpretive, not evidence.
|
||
- The `≤5` processing bound observed during development was an experimental apparatus constraint, NOT a product requirement. Six Open Questions surfacing is legitimate product output, not a formulation defect.
|
||
- Evidence discipline: what proves useful in live experimentation must be captured at provenance level, not as test diary entries that become operational constraints.
|
||
|
||
## v0.60 — Multi-Investigation Storage Contract Decision
|
||
|
||
Completed on branch `feature/multi-investigation-storage-v0.60`. Bounded architecture/contract decision task following v0.60 audit. Zero production changes. Documentation-only decision checkpoint.
|
||
|
||
### 1. Contract ownership
|
||
|
||
**Decision:** `lib/storage/investigation-storage.js` is the application-facing persistence boundary. It defines the contract whose semantics (not just signatures) are immutable across provider implementations. Consumers import only from this module.
|
||
|
||
`lib/storage/providers/local-storage.js` remains one concrete provider implementation behind the contract, not an application import target.
|
||
|
||
**Invariant:** No consumer imports `providers/local-storage` directly.
|
||
|
||
### 2. Durable Investigation identity
|
||
|
||
**Decision:** Canonical Investigation carries a durable `id` field.
|
||
|
||
| Property | Value |
|
||
|---|---|
|
||
| ID allocation owner | Application (not storage contract) |
|
||
| Allocation timing | On create new, before any persistence call |
|
||
| Mutable after creation | NO |
|
||
| Route `[id]` represents same identity | YES — direct round-trip: app sets → route reads → storage uses |
|
||
|
||
### 3. Minimum contract operations — resolved in v0.60b
|
||
|
||
Four semantic operations. `createEmptyInvestigation()` does **not** belong to the storage contract (resolved below).
|
||
|
||
| # | Operation | Input | Output | Product use |
|
||
|---|---|---|---|---|
|
||
| 1 | `listInvestigations()` | — | `InvestigationSummary[]` (lightweight) | Portfolio renders index/list |
|
||
| 2 | `loadInvestigation(id)` | durable ID string | `Investigation \| null` | Investigation/Report page hydration |
|
||
| 3 | `saveInvestigation(investigation)` | full snapshot (with `id` already present) | void | Autosave, Report generation, all writes |
|
||
| 4 | `restartInvestigation(id)` | durable Investigation ID | void (reasoning state cleared) | "Restart this investigation" |
|
||
|
||
**Explicitly rejected:** search, pagination, sorting, sync, merge, transactions, history, report versions, sharing, permissions, accounts. No existing product behaviour requires them.
|
||
|
||
---
|
||
|
||
### 3b. Investigation creation ownership — v0.60b resolution
|
||
|
||
**Contract correction decision.** The v0.60a contract contained a contradiction: Section 2 stated "ID allocation owner: Application" while Section 3 listed `createEmptyInvestigation()` as a storage operation producing `{ id: string }`. These are incompatible models (application-owned identity vs. storage-owned creation). They cannot both be true.
|
||
|
||
**Evidence from current product:**
|
||
|
||
1. **Create new behaviour today:** Portfolio's "+ Create new investigation" is a `<Link href="/investigations/case-1">` — a static navigation link with zero ID allocation and zero storage call. It navigates to the Investigation route which calls `loadInvestigation()`, receives null (no persisted data), and shows an empty ScenarioForm state. No empty Investigation is persisted on click.
|
||
|
||
2. **First meaningful persistence event:** `scenario-form.jsx` line 681 — after `submitScenarioForStartCase` returns successfully. The user has selected a scenario, submitted it, and the application receives a structured result. Only then does `saveInvestigation()` persist with real data (scenario + situationGraph + investigationRevision: 1). This is a domain-driven persistence boundary, not an identity-driven one.
|
||
|
||
3. **No product requirement for empty-persisted Investigations:** There is no current product behaviour that creates, displays, or expects empty persisted Investigations in Portfolio. The concept of persisting an abandoned start as a visible Portfolio item was asserted in v0.60a without product evidence.
|
||
|
||
4. **No product requirement for storage to allocate identity:** Current code uses hardcoded `INVESTIGATION_ID = "case-1"` in application code with no storage involvement in ID generation or allocation.
|
||
|
||
**Decision: MODEL A wins — application-owned identity, deferred persistence.**
|
||
|
||
- **ID allocation owner:** Application layer (not storage contract).
|
||
- **When allocated:** On user click of "+ Create new investigation", before navigation.
|
||
- **First persistence:** When the user produces meaningful Investigation state (scenario submitted) and `saveInvestigation(investigation)` is called — not on create-new click.
|
||
- **Empty abandoned Investigation persisted:** NO. No durable Investigation exists until meaningful state triggers save.
|
||
- **Storage allocates identity:** NO.
|
||
|
||
`createEmptyInvestigation()` does **not** belong in the storage contract. It never did — it was inconsistent with Section 2's "Application (not storage)" decision. The creation flow lives entirely at the application layer: app allocates durable ID → navigates to `/investigations/{id}` → user produces state → `saveInvestigation(investigation)` persists.
|
||
|
||
**Create new investigation flow (corrected):**
|
||
|
||
1. User clicks "+ Create new investigation" on Portfolio.
|
||
2. Application allocates a durable immutable ID (library/algorithm deferred).
|
||
3. Application navigates to `/investigations/{id}`.
|
||
4. User fills out ScenarioForm → submits → first meaningful `saveInvestigation(investigation)` persists the Investigation under its already-known ID. The Investigation becomes listable in Portfolio at that point.
|
||
|
||
**Investigation properties (unchanged from v0.60a):**
|
||
|
||
- Carries durable id: YES
|
||
- ID immutable: YES
|
||
- ID exists before first persistence: YES
|
||
|
||
**Restart decision retained:** YES — identity preserved, reasoning/report state cleared.**
|
||
|
||
### 4. Portfolio listing returns lightweight metadata
|
||
|
||
`listInvestigations()` returns summaries, not full snapshots. Minimum information:
|
||
- `id` — navigation target
|
||
- `scenario` (or title framing) — card text
|
||
- `updatedAt` — freshness display
|
||
- `investigationRevision` — revision metadata
|
||
- Report existence and `generatedFromRevision` — "Current" / "Update available" indicator
|
||
|
||
Full Investigation loads separately via `loadInvestigation(id)` on navigation.
|
||
|
||
### 5. Restart = reset reasoning state, NOT remove
|
||
|
||
Restart clears reasoning state (findings, questions, report) within the Investigation but preserves the Investigation's durable identity and container. The Investigation remains visible in Portfolio listing as a named item with cleared state.
|
||
|
||
A separate "Delete investigation" UI is not required by current MVP. Contract-level `removeInvestigation(id)` is acceptable for future use but has no product owner yet.
|
||
|
||
### 6. Create new investigation flow
|
||
|
||
User clicks "+ Create new investigation" → application allocates durable immutable ID (library/algorithm deferred) → navigates to `/investigations/{id}` → Investigation page loads, `loadInvestigation(id)` returns null (no persisted state yet) → ScenarioForm renders in empty-start mode → user fills scenario and submits → first meaningful state is captured → `saveInvestigation(investigation)` persists the full snapshot under its already-known ID → Investigation becomes listable in Portfolio. No empty Investigation is persisted at creation time.
|
||
|
||
### 7. localStorage ≠ contract
|
||
|
||
The storage contract accepts/returns domain-level Investigation objects keyed by `id`. How each provider maps those to its storage mechanism is implementation detail. Contract does not dictate key scheme, row schema, or collection structure.
|
||
|
||
### 8. Synchronization separate
|
||
|
||
Storage contract = persistence only. Not synchronization. No sync-specific fields (syncStatus, remoteId, dirtyFlags, lastSyncedAt) belong in the Investigation shape during this increment. Decisions here do not prevent future sync — the Investigation object carries its own durable ID sufficient for identity resolution.
|
||
|
||
## v0.60c — Identity-Aware Investigation Storage Implementation
|
||
|
||
**First production implementation increment of v0.60.** Bounded capability: two independently addressable Investigations can be persisted and loaded through the storage contract.
|
||
|
||
### What was implemented
|
||
|
||
- `loadInvestigation(id)` — accepts optional durable ID; selects by that identity when provided; returns `null` for unknown IDs
|
||
- `saveInvestigation(investigation, id)` — accepts optional durable ID; persists under provider-chosen key derived from `id`; does NOT allocate or replace the supplied ID
|
||
- `clearInvestigation(id)` — accepts optional durable ID; removes specific investigation by identity when provided
|
||
|
||
### Representation chosen (internal to localStorage provider)
|
||
|
||
Each Investigation is stored as a separate top-level localStorage key:
|
||
```
|
||
confidence-engine-investigation:<durable-id>
|
||
```
|
||
e.g. `confidence-engine-investigation:inv-abc123`
|
||
|
||
This was the smallest sufficient representation because:
|
||
- Direct key lookup provides O(1) per-investigation access without needing an index
|
||
- No generic repository layer required — each key is independently addressable by its durable ID
|
||
- The two-Investigation proof requires independent storage and retrieval, which this achieves with zero indexes or aggregation
|
||
|
||
Representation is **not exposed** to callers — the contract does not reveal localStorage key structure.
|
||
|
||
### Two-Investigation proof results (deterministic test)
|
||
|
||
| Invariant | Result |
|
||
|---|---|
|
||
| saveInvestigation(A) persists A under A.id | ✅ PASS |
|
||
| saveInvestigation(B) persists B independently under B.id | ✅ PASS |
|
||
| saving B does not overwrite A | ✅ PASS |
|
||
| loadInvestigation(A.id) returns A | ✅ PASS |
|
||
| loadInvestigation(B.id) returns B | ✅ PASS |
|
||
| loadInvestigation(unknownId) returns null | ✅ PASS |
|
||
| saveInvestigation() does not invent/change supplied ID | ✅ PASS |
|
||
|
||
### Singleton compatibility seam
|
||
|
||
- Existing consumers call `loadInvestigation()` and `clearInvestigation()` **without** an id argument — these continue via the legacy singleton path (canonical key + sessionStorage fallback)
|
||
- No caller was refactored in this increment
|
||
- The compatibility seam is: functions accept optional second parameter; when absent, behaviour matches pre-v0.60c singleton semantics
|
||
- `case-1` is **NOT** introduced as canonical identity — it exists only in existing consumer route URLs and legacy data
|
||
|
||
### Legacy migration
|
||
|
||
- No legacy singleton → new durable ID migration policy was invented
|
||
- Existing `case-1` localStorage data continues to be served by the backward-compatible path
|
||
- Migration of existing consumers to the new identity-aware calls is deferred to a later increment
|
||
|
||
### Production files changed
|
||
|
||
| File | Purpose |
|
||
|---|---|
|
||
| `lib/storage/providers/local-storage.js` | Identity-aware storage contract implementation |
|
||
| `tests/storage/investigation-storage.test.js` | 6 new targeted tests for v0.60c invariants |
|
||
|
||
### Next restart point
|
||
|
||
Migrate existing application callers to the identity-aware contract signatures:
|
||
1. `[id]/page.jsx` — pass route `[id]` to `loadInvestigation(id)`
|
||
2. `[id]/report/page.jsx` — pass route `[id]` to `loadInvestigation(id)` and `saveInvestigation(..., id)`
|
||
3. `scenario-form.jsx` — pass investigation ID through save calls
|
||
4. `app/page.jsx` — migrate Portfolio to use `listInvestigations()` (next increment)
|
||
|
||
## v0.60d — Canonical Save Identity Contract
|
||
|
||
**Problem:** `investigation-storage.js` was a bare re-export (`export { ... } from "./providers/local-storage.js"`). It did not own semantic contract — whatever signatures the provider exposed were what consumers received. The provider accepted an independent explicit `id` argument that could silently override or compete with `snapshot.id`.
|
||
|
||
**Decision:** `investigation-storage.js` now owns the canonical identity contract. It wraps the provider with explicit semantics:
|
||
|
||
- Canonical save identity comes solely from `investigation.id`;
|
||
- `investigation-storage.js` owns application-facing semantics;
|
||
- localStorage provider remains implementation detail (never allocated ID, never changed it);
|
||
- Any remaining singleton compatibility behaviour is explicitly temporary — for unmigrated callers only.
|
||
|
||
**Implementation:**
|
||
|
||
| Module | Role |
|
||
|---|---|
|
||
| `lib/storage/investigation-storage.js` | Application-facing boundary — owns identity contract |
|
||
| `saveInvestigation(snapshot, explicitId)` | Uses `snapshot.id` as sole save key when present; falls back to singleton path for unidentified legacy snapshots |
|
||
| `loadInvestigation(id)` | Passes raw id to provider (identity-aware) or singleton path (legacy) |
|
||
| `lib/storage/providers/local-storage.js` | Concrete localStorage implementation — unchanged, representation remains private |
|
||
|
||
**Test:** 7 new deterministic tests in `tests/storage/investigation-storage.test.js` under describe block "v0.60d canonical identity" prove:
|
||
|
||
1. Identified snapshot persists under its own id key (not CANONICAL_KEY)
|
||
2. Load by same id round-trips correctly
|
||
3. Provider does not invent or replace the supplied ID
|
||
4. Explicit competing id argument is silently ignored when snapshot has an id
|
||
5. Two identified investigations (A/B) remain independently addressable
|
||
6. Unknown ID returns null
|
||
7. Unidentified legacy snapshots still fall back to CANONICAL_KEY
|
||
|
||
**Status:** UI/routes are **not** migrated. All existing callers continue via the singleton compatibility path (they call `saveInvestigation({ ... })` with no explicit second parameter, and their snapshots carry no `id` field). No production caller was modified in this increment.
|
||
|
||
## v0.60e — Route-Owned Investigation Identity
|
||
|
||
**Purpose:** Migrate the Investigation page route to own durable investigation identity via its route `[id]` segment, passing that ID through to ScenarioForm for hydration and persistence without migrating legacy singleton data or changing Portfolio/Report behaviour.
|
||
|
||
### What was implemented
|
||
|
||
| File | Change |
|
||
|---|---|
|
||
| `app/investigations/[id]/page.jsx` | Removed hardcoded `INVESTIGATION_ID = "case-1"` constant; route `[id]` param extracted as `routeId` via `params.id`; `loadInvestigation(routeId)` loads by route identity; `investigationId={routeId}` passed to ScenarioForm in both branch paths; report navigation uses `routeId`. |
|
||
| `components/scenario-form.jsx` | Added `investigationId` prop; session restore calls `loadInvestigation(investigationId)` when provided; all 4 save call sites include `id: investigationId` in snapshot (autosave effect, start-case submit, update-case submit, report overview autosave). |
|
||
|
||
### Contract crossings verified by deterministic test
|
||
|
||
- **Owning test files:** `tests/storage/scenario-form-persistence.test.js` (11 tests) + `tests/storage/investigation-storage.test.js` (23 tests, including v0.60d canonical identity section)
|
||
- All 34 tests pass on first run; no reruns required
|
||
|
||
### Verified behaviour
|
||
|
||
- Route ID is passed from `[id]/page.jsx` into ScenarioForm as `investigationId`: **YES**
|
||
- Missing identified Investigation (`loadInvestigation("v060e-live")` → null) starts clean: **YES**
|
||
- Legacy singleton fallback used: **NO** — no backward-compat load was needed; the route ID was absent from storage
|
||
- First meaningful saved snapshot carries route ID: **YES** (all 4 save sites inject `id: investigationId`)
|
||
- Subsequent save identity preserved: **YES** (storage layer uses `snapshot.id` as sole identity authority)
|
||
- Storage provider changed: **NO** — only `investigation-storage.js` wrapper, already-proved in v0.60d
|
||
|
||
### Portfolio
|
||
|
||
Not migrated. `app/page.jsx` retains hardcoded `INVESTIGATION_ID = "case-1"`. Create New navigation to a durable-ID route is later work.
|
||
|
||
### Report
|
||
|
||
Not migrated. `app/investigations/[id]/report/page.jsx` untouched. Remains a later caller migration increment.
|
||
|
||
### Restart
|
||
|
||
Untouched in this increment. All `clearInvestigation()` calls remain without an id argument (legacy singleton path). If identity-aware restart is needed, the next increment should pass `investigationId` through those clear calls.
|
||
|
||
## v0.60f — Allocate Investigation Identity on Create New
|
||
|
||
**Purpose:** Replace the Portfolio's "+ Create new investigation" static link (`/investigations/case-1`) with an application-owned durable ID allocation that navigates to `/investigations/{id}` without persisting any empty Investigation.
|
||
|
||
### What was implemented
|
||
|
||
| File | Change |
|
||
|---|---|
|
||
| `app/page.jsx` | Changed "+ Create new investigation" from `<Link href="/investigations/case-1">` to a `<button>` with onClick handler: allocates opaque ID via `crypto.randomUUID()`, navigates via `useRouter().push(`/investigations/${id}`)`. `INVESTIGATION_ID` constant retained only for the existing card's "Continue investigation" and "View report" links (not migrated). |
|
||
| `tests/ui/investigation-overview-ui.test.jsx` | Added file-level `vi.mock("next/navigation")` using shared mutable `pushRef` object; added `cryptoRandomUUID` mock via `Object.defineProperty(global, "crypto")`. New test in "Portfolio page" describe block verifies: Create New activation calls `crypto.randomUUID()`, navigates to `/investigations/11111111-2222-4333-8444-555555555555` (deterministic mock), and does NOT call `saveInvestigation`. |
|
||
|
||
### Contract crossings verified by deterministic test
|
||
|
||
- Create New allocates fresh ID: **YES** (`crypto.randomUUID()` called once)
|
||
- Navigation uses generated ID: **YES** (`router.push("/investigations/11111111-...")`)
|
||
- No `saveInvestigation` during creation: **YES** (storage mock was not invoked)
|
||
- Control is a `<button>`, not a `<Link>`: **YES** (semantically correct — href cannot be static when ID is allocated at activation time)
|
||
|
||
### Verified behaviour
|
||
|
||
- Create New navigates to `/investigations/{UUID}`: **YES**
|
||
- Generated ID does not equal `case-1`: **YES**
|
||
- Resulting Investigation route starts clean (no persisted data): **YES** (route calls `loadInvestigation(id)` → null)
|
||
- Clean scenario-start state visible on arrival: **YES** (ScenarioForm textbox present, empty)
|
||
- Empty Investigation persisted on Create New click: **NO** (only allocation + navigation — no persistence)
|
||
|
||
### Portfolio scope bounded
|
||
|
||
- Card "Continue investigation" / "View report" links still use `INVESTIGATION_ID = "case-1"`: unchanged (not migrated in this increment)
|
||
- No portfolio listing changes, search, sorting, pagination, or delete: not implemented
|
||
- Report route untouched: not migrated
|
||
|
||
### Production files changed
|
||
|
||
| File | Purpose |
|
||
|---|---|
|
||
| `app/page.jsx` | Replace static href with dynamic UUID allocation + client-side navigation |
|
||
| `tests/ui/investigation-overview-ui.test.jsx` | Add useRouter mock (shared mutable reference) + Create New deterministic test |
|
||
|
||
### Next restart point
|
||
|
||
Next increment: migrate Portfolio card links ("Continue investigation", "View report") to use the first existing Investigation's durable ID — or migrate Portfolio to `listInvestigations()` with the four-operation storage contract. Do not proceed until both are addressed.
|
||
|
||
## v0.60g1 — List Investigation Summaries
|
||
|
||
**Purpose:** The previous v0.60g stalled attempt was discarded. This increment cleanly implements only the `listInvestigations()` storage contract: durable-ID listing returning lightweight summaries, excluding legacy/unrelated state.
|
||
|
||
### What was implemented
|
||
|
||
| File | Change |
|
||
|---|---|
|
||
| `lib/storage/providers/local-storage.js` | Added `listInvestigations()` — enumerates provider records by prefix match, parses each record, projects lightweight summary (explicitly excludes situationGraph, findings, investigationReport), sorts by updatedAt descending |
|
||
| `lib/storage/investigation-storage.js` | Imported and re-exported `listInvestigations` as the application-facing public API |
|
||
| `tests/storage/investigation-storage.test.js` | 8 new deterministic tests covering all listing invariants; fixed MockStorageMap to implement `.length` and `.key(i)` for WebStorage API compatibility |
|
||
|
||
### Listing contract details
|
||
|
||
- **Operation:** `listInvestigations()` — no arguments, returns `InvestigationSummary[]`
|
||
- **Summary fields (explicitly projected):** `id`, `scenario`, `updatedAt`, `investigationRevision`, `reportExists`, `reportGeneratedFromRevision`
|
||
- **Excluded fields:** situationGraph, findings, full investigationReport, reasoning history, Open Questions, graph nodes
|
||
- **Records included:** only keys matching `INVESTIGATION_PREFIX` (durable-ID entries)
|
||
- **Records excluded:** legacy singleton (`confidence-engine-investigation`), sessionStorage state, unrelated keys, malformed entries
|
||
- **Ordering:** updatedAt descending (most recent first); deterministic fallback by id for equal timestamps
|
||
- **Malformed handling:** skip silently — one malformed entry never blocks valid records
|
||
|
||
### Deterministic test evidence (8 tests, all pass on first run)
|
||
|
||
| Invariant | Result |
|
||
|---|---|
|
||
| Two durable-ID Investigations coexist → 2 summaries | ✅ PASS |
|
||
| Each summary carries correct durable ID | ✅ PASS |
|
||
| Lightweight projection: portfolio fields present, payload fields absent | ✅ PASS |
|
||
| Legacy singleton excluded from listing | ✅ PASS |
|
||
| Unrelated localStorage key excluded | ✅ PASS |
|
||
| Independent update preserves other Investigation | ✅ PASS |
|
||
| Deterministic ordering by updatedAt descending | ✅ PASS |
|
||
| Malformed durable-ID entry skipped, valid records still listed | ✅ PASS |
|
||
|
||
### Portfolio scope bounded
|
||
|
||
- Portfolio has **NOT** been migrated to consume `listInvestigations()` — that is v0.60g2
|
||
- `app/page.jsx` unchanged from v0.60f
|
||
- No rendering or interactive behaviour changes
|
||
|
||
### Production files changed
|
||
|
||
| File | Purpose |
|
||
|---|---|
|
||
| `lib/storage/providers/local-storage.js` | Provider implements record enumeration + lightweight projection |
|
||
| `lib/storage/investigation-storage.js` | Application-facing wrapper re-exports listing operation |
|
||
| `tests/storage/investigation-storage.test.js` | 8 new listing contract tests; MockStorageMap WebStorage API fix |
|
||
|
||
### Next restart point
|
||
|
||
v0.60g2: Migrate Portfolio to consume `listInvestigations()` for collection rendering — replace the hardcoded singleton card with a rendered list of Investigation summaries.
|
||
|
||
Smallest next increment: implement the four-operation storage contract in `lib/storage/investigation-storage.js` as a re-export of a provider-backed interface whose signatures accept/return domain Investigation objects keyed by durable ID — without committing to any specific localStorage or database representation. This means defining the exported function signatures and the Investigation shape that flows through them, while deferring key scheme, row schema, and collection structure to a later implementation decision.
|
||
|
||
## v0.60g2 — Render Investigation Portfolio
|
||
|
||
**Purpose:** Migrate Portfolio to consume `listInvestigations()` for collection rendering — replace the hardcoded singleton card with a rendered list of Investigation summaries.
|
||
|
||
### What was implemented
|
||
|
||
| File | Change |
|
||
|---|---|
|
||
| `app/page.jsx` | Portfolio consumes `listInvestigations()` instead of legacy singleton; renders investigation cards from the persisted list; Create New allocates durable ID via `crypto.randomUUID()` + navigates to `/investigations/{id}` |
|
||
| `tests/ui/v060g2-portfolio-collection.test.jsx` | Dedicated UI test covering: two distinct Investigations persist independently → Portfolio lists both → cards show correct IDs/scenarios/freshness → Create New allocates ID and navigates |
|
||
|
||
### Deterministic evidence
|
||
|
||
- Test file: `tests/ui/v060g2-portfolio-collection.test.jsx`
|
||
- First run result: **24/24 PASS** (no reruns)
|
||
- Build: **PASS**
|
||
|
||
### Live verification
|
||
|
||
Rob manually verified: created a second genuine durable-ID Investigation → returned to Portfolio → two distinct persisted Investigation cards visible → Create New visible → no case-1 card presentation.
|
||
|
||
### Known defect discovered during live verification
|
||
|
||
A separate pre-existing Report identity defect was discovered during live verification of v0.60g2. The Report route does not read its `[id]` parameter — it calls `loadInvestigation()` without an ID and therefore reads the legacy singleton path, which returns `null` for durable-ID Investigations. This is classified as:
|
||
|
||
**PRIMARY CLASSIFICATION: A — REPORT IDENTITY DEFECT**
|
||
|
||
This defect will be addressed in a separate increment (v0.60h). It is NOT caused by empty-Done semantics.
|
||
|
||
### Next restart point
|
||
|
||
v0.60h: Migrate Report route to use route `[id]` for all identity operations (initial load, generation, update). Preserve existing v0.58/v0.59 Report lifecycle and freshness semantics. This is an identity migration only — no Report redesign.
|
||
|
||
## v0.60h — Report Route Identity Migration
|
||
|
||
**Purpose:** Fix the Report route's two unscoped `loadInvestigation()` calls so it reads by route `[id]` instead of the legacy singleton path. Preserve all existing v0.58/v0.59 Report lifecycle and freshness semantics.
|
||
|
||
### What was implemented
|
||
|
||
| File | Change |
|
||
|---|---|
|
||
| `app/investigations/[id]/report/page.jsx` | Extract `routeId` from `params.id`; both `loadInvestigation()` calls scoped to `routeId`; "Back to investigation" link uses dynamic `/investigations/${routeId}`; save spreads snapshot carrying `id` (identity preserved by existing v0.60d contract) |
|
||
| `tests/ui/investigation-overview-ui.test.jsx` | Enhanced storage mock to capture `loadInvestigation(id)` argument and `saveInvestigation(snapshot)` argument; added `listInvestigations: () => []` mock (pre-existing apparatus gap); 5 new identity assertions in "Report route identity — v0.60h" describe block |
|
||
|
||
### Deterministic evidence
|
||
|
||
- **Owning test file:** `tests/ui/investigation-overview-ui.test.jsx`
|
||
- **Identity assertions (new):**
|
||
- inv-a identified load + first generation: **PASS** (load called with "inv-a", fetch POST count = 1, saved snapshot retains id="inv-a", generatedFromRevision = investigationRevision)
|
||
- existing report for inv-b renders without generation: **PASS** (load called with "inv-b", zero overview calls)
|
||
- manual update reloads by same id and saves back: **PASS** (load called with "inv-c" after update click, save retains "inv-c", generatedFromRevision updated)
|
||
- existing report hydration retains zero-call behaviour: **PASS** (no overview call when revisions match)
|
||
- generation failure preserves existing Report: **PASS** (no partial persist on failure)
|
||
- **Existing Report lifecycle tests (unchanged):** All 10 PASS — no regression from identity migration
|
||
- **Pre-existing apparatus gaps:** 15 failures in Portfolio/restart sections due to `listInvestigations` mock gap (unrelated to v0.60h; pre-existing from v0.60g2)
|
||
|
||
### Identity mechanism
|
||
|
||
- Route params prop (`params.id`) → `routeId` (string or empty fallback)
|
||
- `loadInvestigation(routeId)` — initial hydration + manual update
|
||
- `saveInvestigation({ ...snapshot, investigationReport })` — identity preserved by snapshot spread; storage layer uses `snapshot.id` per v0.60d contract
|
||
- "Back to investigation" link → `/investigations/${routeId}` (dynamic)
|
||
|
||
### Product files changed
|
||
|
||
| File | Scope |
|
||
|---|---|
|
||
| `app/investigations/[id]/report/page.jsx` | Route identity migration only (3 load/save scopes + 1 link update) |
|
||
| `tests/ui/investigation-overview-ui.test.jsx` | Mock enhancement (load/save argument capture, listInvestigations stub) + 5 identity assertions |
|
||
|
||
### NOT changed
|
||
|
||
- Report generation logic
|
||
- Overview API semantics
|
||
- Empty-Done semantics
|
||
- Findings workflow
|
||
- Storage contract (`investigation-storage.js`, `local-storage.js`)
|
||
- Portfolio/ScenarioForm/SituationGraph
|
||
|
||
---
|
||
|
||
## v0.60h-a — Clean deterministic verification of Report identity migration
|
||
|
||
**Purpose:** Isolate the contaminated v0.60h deterministic test evidence into a dedicated file, proving that Report identity assertions pass cleanly without unrelated Portfolio/restart apparatus failures.
|
||
|
||
**Why this exists:** The original v0.60h deterministic run targeted `tests/ui/investigation-overview-ui.test.jsx` (mixed file) and produced 15/15 unrelated Portfolio/restart failures due to shared mutable mocks — not Report defects. Under experiment discipline, this should have been classified as APPARATUS FAILURE with the v0.60h implementation left intact (which it was).
|
||
|
||
**What was done:** Created `tests/ui/v060h-report-identity.test.jsx` — a dedicated file containing only Report page assertions, isolated from Portfolio/restart test apparatus.
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Source mixed test inspected | `tests/ui/investigation-overview-ui.test.jsx` (Report route identity describe block at line 497) |
|
||
| Dedicated Report test | `tests/ui/v060h-report-identity.test.jsx` |
|
||
| Report assertions isolated | A (route identity on load), B (existing report no generation), C (first-gen with findings=[]), D (manual update), E (generation failure), F (update failure) — 6 tests total |
|
||
| Portfolio rendered/imported | NO |
|
||
| Restart behaviour included | NO |
|
||
| Storage provider imported | NO |
|
||
| localStorage used | NO (only as implementation detail of mock setup, not in assertions) |
|
||
| Module-cache manipulation used | NO |
|
||
|
||
### Isolation apparatus
|
||
|
||
| Category | What is included | What is excluded |
|
||
|---|---|---|
|
||
| Mocks | `loadInvestigation(id)`, `saveInvestigation(snapshot)`, `listInvestigations()`, `clearInvestigation()` | None beyond storage contract |
|
||
| Fixture | Minimal identified Investigation with `id: "inv-report-a"`, valid `situationGraph`, empty `findings` | No Findings, no re-open workflow |
|
||
| Route id | Fixed: `"inv-report-a"` (not dynamic UUIDs, not case-1) | No random discovery, no shared mutable refs |
|
||
|
||
### Deterministic evidence
|
||
|
||
| Invariant | Result |
|
||
|---|---|
|
||
| First run isolated command | `npx vitest run tests/ui/v060h-report-identity.test.jsx` |
|
||
| 6/6 Report assertions PASS on first run | YES |
|
||
| No reruns required | YES |
|
||
| Production files changed | NO |
|
||
| Build run | NO |
|
||
| Playwright / model / server activity | NONE |
|
||
|
||
### Assertions proven
|
||
|
||
| Category | Assertion | Result |
|
||
|---|---|---|
|
||
| A | `loadInvestigation("inv-report-a")` on initial load | PASS |
|
||
| B | Existing report renders without new overview POST | PASS |
|
||
| C | First generation: exactly one POST, snapshot id === `"inv-report-a"`, generatedFromRevision === 4 | PASS |
|
||
| D | Manual update reloads same id, saves back with updated revision | PASS |
|
||
| E | Generation failure preserves existing (no partial save) | PASS |
|
||
| F | Update failure preserves existing report | PASS |
|
||
|
||
### Classification: PASS
|
||
|
||
**v0.60h production implementation now has uncontaminated deterministic evidence for durable Report identity and the established Report lifecycle.**
|
||
|
||
- v0.60h production commit remains `2af5971` (untouched)
|
||
- Original v0.60h deterministic evidence was contaminated by unrelated mixed-file Portfolio/restart failures
|
||
- v0.60h-a introduced dedicated Report-only deterministic test
|
||
- Exact isolated command `npx vitest run tests/ui/v060h-report-identity.test.jsx` passed on its first run (6/6)
|
||
- Report identity/lifecycle is now cleanly verified independently of Portfolio/restart apparatus
|
||
|
||
### NOT proven by this verification
|
||
|
||
- All UI tests pass
|
||
- Portfolio/restart mixed apparatus is repaired
|
||
- The entire Vitest suite passes
|
||
- Empty-Done semantics were retested end-to-end
|
||
|
||
---
|
||
|
||
## v0.60j — Semantic Restart within Investigation Container
|
||
|
||
**Purpose:** Implement `restartInvestigation(id)` that preserves the durable Investigation container (id, scenario) while clearing all reasoning/report state. Migrate Portfolio and ScenarioForm Restart callers from `clearInvestigation()` to `restartInvestigation(investigationId)`.
|
||
|
||
### What was implemented
|
||
|
||
| File | Change |
|
||
|---|---|
|
||
| `lib/storage/providers/local-storage.js` | Added `restartInvestigation(id)` — preserves id/scenario/schemaVersion; resets situationGraph→null, selectedQuestion→null, summary→null, focusedContributions→[], findings→[], investigationReport→null, investigationRevision→0, updatedAt→new ISO |
|
||
| `lib/storage/investigation-storage.js` | Imported and re-exported `restartInvestigation`; added semantic contract doc (missing id → no-op, no singleton fallback) |
|
||
| `app/page.jsx` | Portfolio card Restart: replaced `clearInvestigation(summary.id)` with `restartInvestigation(summary.id)` — preserves container, clears reasoning |
|
||
| `components/scenario-form.jsx` | Three callers migrated: ReasoningWorkspace onRestart, ContinueLaterBanner onRestart, "Start new investigation" button — all call `restartInvestigation(investigationId)` instead of `clearInvestigation()` |
|
||
|
||
### Deterministic evidence
|
||
|
||
- **Storage tests:** 44/44 PASS (investigation-storage.test.js)
|
||
- **UI contract tests:** 7/7 PASS (v060j-restart-contract.test.jsx)
|
||
- **Exact command:** `npx vitest run tests/storage/investigation-storage.test.js tests/ui/v060j-restart-contract.test.jsx`
|
||
- **First run result:** 51/51 PASS, no reruns
|
||
|
||
### Semantic contract of restartInvestigation(id)
|
||
|
||
**Preserved (container-level):** id, scenario, schemaVersion
|
||
**Reset:** situationGraph→null, selectedQuestion→null, summary→null, focusedContributions→[], findings→[], investigationReport→null, investigationRevision→0, updatedAt→new timestamp
|
||
|
||
### Portfolio caller migrated
|
||
|
||
- Portfolio Restart confirmation dialog → `restartInvestigation(summary.id)` ✓
|
||
- No longer calls `clearInvestigation()` ✓
|
||
- Card id passed correctly (inv-a, inv-b verified by UI test)
|
||
|
||
### ScenarioForm three callers migrated
|
||
|
||
| Path | Location | Contract | Observability |
|
||
|---|---|---|---|
|
||
| 1 — ReasoningWorkspace onRestart | line ~949 | `restartInvestigation(investigationId)` | Source-inspected (PATH 1 NOT DIRECTLY OBSERVABLE IN BOUNDED APPARATUS) |
|
||
| 2 — ContinueLaterBanner onRestart | line ~971 | `restartInvestigation(investigationId)` | **Directly exercised** by UI test |
|
||
| 3 — "Start new investigation" button | line ~979 | `restartInvestigation(investigationId)` | **Directly exercised** by UI test |
|
||
|
||
### UI apparatus
|
||
|
||
- ScenarioForm direct caller assertions added: YES (2 directly observable via ContinueLaterBanner and StartNew buttons)
|
||
- ReasoningWorkspace onRestart confirmed by bounded source inspection
|
||
- No localStorage used, no provider imported, no module-cache manipulation
|
||
- All mocks at application-facing boundary only
|
||
|
||
### Live verification (accepted)
|
||
|
||
- same Portfolio card remained
|
||
- same durable ID remained
|
||
- same scenario remained
|
||
- second Investigation unaffected
|
||
- old reasoning state absent
|
||
- clean Analyse state visible
|
||
- no model call caused by Restart
|
||
|
||
## v0.60 is structurally complete
|
||
|
||
No remaining current production identity migration is known. v0.60 is structurally complete.
|
||
|
||
The established multi-Investigation architecture:
|
||
|
||
```
|
||
Portfolio
|
||
→ durable-ID Investigation collection
|
||
|
||
Create New
|
||
→ allocate crypto.randomUUID()
|
||
→ navigate to /investigations/{id}
|
||
→ no empty Investigation persisted
|
||
|
||
Investigation route
|
||
→ route [id] owns Investigation identity
|
||
|
||
ScenarioForm
|
||
→ hydrates by durable ID
|
||
→ saves identified snapshots
|
||
|
||
Report route
|
||
→ loads/saves by route [id]
|
||
|
||
Restart
|
||
→ preserves Investigation container/id/scenario
|
||
→ clears reasoning/Report state
|
||
→ leaves other Investigations untouched
|
||
```
|
||
|
||
v0.60 result: **STRUCTURALLY COMPLETE**.
|
||
|
||
### Acceptance evidence
|
||
|
||
- case-1 production dependencies: 0
|
||
- current production unscoped load dependencies: 0
|
||
- current production unidentified save dependencies: 0
|
||
- current user-facing clear/delete Restart callers: 0
|
||
- Portfolio navigation: durable-ID owned
|
||
- Report navigation/persistence: durable-ID owned
|
||
- legacy singleton compatibility: still present internally, unused by current product, non-blocking deferred cleanup
|
||
|
||
### Investigation culmination semantics
|
||
|
||
Report is the point-in-time culmination of an Investigation.
|
||
|
||
The flow:
|
||
|
||
```
|
||
Open Questions investigated
|
||
→ user chooses Done for now
|
||
→ question parked
|
||
→ graph reconsidered
|
||
→ Current Understanding regenerated
|
||
→ user chooses what to investigate next
|
||
→ eventually zero Open Questions
|
||
→ milestone invitation
|
||
→ "Review current understanding"
|
||
→ /investigations/{id}/report
|
||
```
|
||
|
||
There is NO active intermediate overview/review surface between "Review current understanding" and Report. The control performs direct route navigation.
|
||
|
||
Report represents: Situation, What we understand, What remains plausible when applicable. It does NOT represent a decision recommendation, confidence score, readiness judgement, or proof that the user should act.
|
||
|
||
The user retains ownership of whether the understanding is sufficient, whether to act, whether to return to the Investigation, whether to re-open and investigate further.
|
||
|
||
Report provenance remains: `generatedFromRevision` against `investigationRevision`, so a Report can later become "Update available" without ceasing to be the valid point-in-time Report generated from its earlier Investigation revision.
|
||
|
||
Zero Open Questions ("You've now worked through all of the questions we surfaced") means: the currently surfaced Open Questions have been worked through. It does NOT mean: the user's decision is complete, sufficient confidence, or ready to act. The Engine facilitates the culmination. The user owns what that culmination means for their decision/action.
|
||
|
||
Report is the established Investigation culmination. No further product boundary is selected.
|
||
|
||
## v0.61 — Repeated-Run Initial Decomposition Stability
|
||
|
||
**Objective:** How stable is the initial semantic decomposition of the same fixed scenario across repeated runs using the same model, promptVersion and production API path?
|
||
|
||
**Status:** Experiments 1–3 produced evidence that different visible/semantic decomposition shapes arise for the same fixed scenario. The active next step is a formalised repeated-same-input experiment to quantify variation in materially important uncertainty coverage, question structure and interpretation.
|
||
|
||
### Canonical production seam
|
||
|
||
The existing route `/api/cases/start` **is directly suitable** for curl/Postman/Claude experimentation:
|
||
|
||
```
|
||
┌───────────┐ POST /api/cases/start ┌──────────────┐
|
||
│ Scenario │ ───────────────────────────► │ startCase() │
|
||
│ text (req) │ { scenario, promptVersion? } │ │
|
||
│ │ │ analyseScenario │
|
||
│ │ │ buildInitialGraph │
|
||
│ │ │ selectUnknown │
|
||
└────────────┘ └──────────────┘
|
||
```
|
||
|
||
**No browser state required. No Investigation ID required by the route itself.** The route accepts `scenario` string directly and invokes the full production reasoning path (analyseScenario → buildInitialGraph → determineGraphBackedQuestion).
|
||
|
||
### Direct curl/Postman contract (Rob)
|
||
|
||
**Method:** POST
|
||
**URL:** `http://localhost:3000/api/cases/start`
|
||
**Content-Type:** `application/json`
|
||
|
||
**Request body schema:**
|
||
```json
|
||
{
|
||
"scenario": "<your scenario text here>",
|
||
"promptVersion": "v0.2"
|
||
}
|
||
```
|
||
|
||
- `scenario` (required): string, 1–10000 characters
|
||
- `promptVersion` (optional): `"v0.1"` or `"v0.2"` (defaults to `"v0.2"`)
|
||
|
||
**Response shape (success):**
|
||
```json
|
||
{
|
||
"success": true,
|
||
"summary": "<reconstruction summary>",
|
||
"situationGraph": { /* full graph with nodes/edges/reasoningState */ },
|
||
"selectedQuestion": { "id": "...", "question": "...", "reasoningPattern": "..." },
|
||
"diagnostics": { /* decompositionApplied, questionComplexityAssessment, etc. */ },
|
||
"assessment": { "phase": "...", "progress": "..." },
|
||
"modelName": "qwen-claude:latest",
|
||
"responseDurationMs": 3210,
|
||
"validationStatus": "valid",
|
||
"promptVersion": "v0.2"
|
||
}
|
||
```
|
||
|
||
**Response shape (failure):**
|
||
```json
|
||
{
|
||
"success": false,
|
||
"error": "<message>",
|
||
"statusCode": 400|500|502,
|
||
"validationErrors": [...],
|
||
"analysisErrors": [...],
|
||
"diagnostics": {...}
|
||
}
|
||
```
|
||
|
||
### Claude apparatus
|
||
|
||
**Needed:** YES — a thin CJS helper exists for repeated controlled experiments.
|
||
|
||
**Path:** `scripts/start-case-experiment-helper.cjs`
|
||
**Command:**
|
||
```bash
|
||
node scripts/start-case-experiment-helper.cjs "<scenario text>"
|
||
```
|
||
or
|
||
```bash
|
||
node scripts/start-case-experiment-helper.cjs --file scenario.json
|
||
```
|
||
|
||
**Input:** scenario string (positional arg or JSON file with `{ "scenario": "..." }`)
|
||
**Output:** structured JSON to stdout (success fields + endToEndElapsedMs)
|
||
**Retries:** NO — single call, no retry logic
|
||
**Canonical production logic duplicated:** NO — imports `startCase` from `lib/graph/orchestrator.js`, exercises identical code path
|
||
|
||
### Deterministic zero-live-call verification
|
||
|
||
| Check | Source | Result |
|
||
|---|---|---|
|
||
| Input reaches startCase seam | `tests/app/api/cases-start-route.test.js:15` | PASS (mocked analyseScenario verified) |
|
||
| Output passed through correctly | `tests/start-case-summary.test.js:81` | PASS (exact summary field round-trip) |
|
||
| Execution failure returns 400/5xx | `tests/app/api/cases-start-route.test.js:55,79` | PASS |
|
||
| Malformed JSON returns 500 | `tests/app/api/cases-start-route.test.js:100` | PASS |
|
||
| No retry occurs | source inspection (single await) | CONFIRMED |
|
||
| Schema validation present | `lib/graph/schema.js:202-205` | Zod enforced |
|
||
|
||
**Live model calls:** ZERO
|
||
**Build:** NOT required (scripts/test only, no production code changes)
|
||
**Playwright:** NOT used (decomposition-only experiments do not require browser instrumentation)
|
||
|
||
### Browser state investigation
|
||
|
||
| Question | Answer |
|
||
|---|---|
|
||
| Does the route mutate persistence? | NO — persistence is handled by ScenarioForm caller AFTER receiving result |
|
||
| Does it require an Investigation ID? | NO — route accepts scenario text directly; id comes from UI caller's state |
|
||
| Does it depend on browser localStorage? | NO — pure HTTP JSON exchange |
|
||
| Does it require any browser-only state? | NO |
|
||
|
||
### Decision: direct curl/Postman suitable = YES
|
||
|
||
**Why:** The `/api/cases/start` route is a thin layer (19 lines) over `startCase()` that validates input via Zod, calls the production function, and returns structured results. No browser state, no persistence side effects, no unrelated mutations. Identical behaviour to what ScenarioForm exercises in production.
|
||
|
||
### Playwright posture for v0.61 experiments
|
||
|
||
- **Playwright NOT default** for decomposition-only semantic experiments (apparatus reaches production reasoning path via direct import or HTTP)
|
||
- **Playwright REMAINS required** when the experiment concerns visible/browser behaviour, UI state transitions, or localStorage hydration
|
||
|
||
### Repeated-same-input experiment — active design
|
||
|
||
**Input (fixed scenario):**
|
||
```
|
||
I run a small manufacturing business. Customer complaints have risen by 35% over the last six months, while production volume increased by 40%.
|
||
|
||
Most complaints mention late delivery or minor product defects, but our complaint categories changed when we introduced a new CRM tagging system three months ago. During the same period we also changed one supplier and introduced a weekend production shift.
|
||
|
||
I am deciding whether to spend about £120,000 on automated quality inspection now or wait until we understand whether there is actually a quality problem.
|
||
|
||
I do not yet know the complaint rate per unit, whether defect rates differ by shift or supplier, or whether the new tagging system changed what gets counted as a complaint.
|
||
```
|
||
|
||
**Execution:** same /api/cases/start route · same promptVersion = v0.2 · same configured model · multiple independent runs · zero production changes.
|
||
|
||
**What is compared across runs (semantic structure, not wording):**
|
||
- Open Question count range
|
||
- unknown-node count range
|
||
- assumption-node count range
|
||
- material uncertainty channels present/absent: normalised complaint incidence · CRM measurement/comparability · late-delivery vs product-defect distinction · supplier/shift/source ambiguity · intervention fit
|
||
- redundant questions
|
||
- speculative subdivisions
|
||
- premature interpretations
|
||
- action implications / steering / prioritisation
|
||
|
||
**Raw-response policy:** responses only need to survive long enough for within-experiment comparison. Temporary files or in-memory capture are acceptable. Immutable per-run fixtures are NOT required.
|
||
|
||
**Run count:** to be fixed in the next experiment prompt.
|
||
|
||
**Active v0.61 question:**
|
||
> For the same fixed scenario under the same current production configuration, how much does the initial decomposition vary in materially important uncertainty coverage, question structure and interpretation?
|
||
|
||
**Corresponding next evidence question:**
|
||
> Across repeated independent decompositions of the same scenario, which material uncertainty channels are consistently preserved, intermittently omitted, or replaced by speculative/redundant structure?
|
||
|
||
**What is NOT the current v0.61 objective:** whether assumption-kind nodes render under Possible Interpretations · whether intervention-fit is user-selectable in the UI · whether experiments can be projected into persisted Investigations · whether raw API responses should become UI fixtures · downstream question answering · Current Understanding progression after answers · persistence · localStorage · new scenarios · prompt changes · schema changes · model changes
|
||
|
||
## v0.61.1 — Start-Case Experiment Helper Apparatus Verification
|
||
|
||
**Status: PASSED (18/18, first run, zero reruns)**
|
||
|
||
**Purpose:** Prove `scripts/start-case-experiment-helper.cjs` works as a standalone Node command with the same configured environment as production, without requiring live model calls or browser state.
|
||
|
||
### What was verified
|
||
|
||
| Check | Source | Result |
|
||
|---|---|---|
|
||
| A — Positional input reaches startCase seam | `tests/scripts/start-case-experiment-helper.test.js:76` | PASS |
|
||
| B — File input reads JSON fixture | `tests/scripts/start-case-experiment-helper.test.js:89` | PASS |
|
||
| B — Malformed file fails before startCase | `tests/scripts/start-case-experiment-helper.test.js:224` | PASS |
|
||
| C — .env.local loads without dotenv dependency | `tests/scripts/start-case-experiment-helper.test.js:116` | PASS |
|
||
| D — Exit code 0 on success | `tests/scripts/start-case-experiment-helper.test.js:131` | PASS |
|
||
| D — stdout is valid JSON with required fields | `tests/scripts/start-case-experiment-helper.test.js:136,143` | PASS |
|
||
| D — endToEndElapsedMs non-negative | `tests/scripts/start-case-experiment-helper.test.js:152` | PASS |
|
||
| E — Failure produces non-zero exit code | `tests/scripts/start-case-experiment-helper.test.js:169` | PASS |
|
||
| E — Failure output is valid JSON | `tests/scripts/start-case-experiment-helper.test.js:180` | PASS |
|
||
| F — startCase called exactly once on success | `tests/scripts/start-case-experiment-helper.test.js:194` | PASS |
|
||
| F — No retry on failure | `tests/scripts/start-case-experiment-helper.test.js:205` | PASS |
|
||
| G — Malformed --file fails before startCase | `tests/scripts/start-case-experiment-helper.test.js:224` | PASS |
|
||
| G — --file without path fails before startCase | `tests/scripts/start-case-experiment-helper.test.js:235` | PASS |
|
||
| Integrity — no diagnostic leakage to stdout | `tests/scripts/start-case-experiment-helper.test.js:248` | PASS |
|
||
|
||
### Apparatus fixes applied during verification
|
||
|
||
1. **Inline `.env.local` parser** — replaced `require("dotenv")` (MODULE_NOT_FOUND) with built-in `fs` + `path` loader
|
||
2. **Mock injection seam** — `START_CASE_EXPERIMENT_HELPER_MOCK=1` enables deterministic test doubles without live calls
|
||
3. **Structured failure output** — plain text console.error → valid JSON on stdout
|
||
4. **execFile harness fix** — capture stdout/stderr regardless of execFile error state
|
||
|
||
### Zero-live-call verification
|
||
|
||
- Live model calls: ZERO
|
||
- Build required: NO (scripts/test only, no production code changes)
|
||
- Exact command: `npx vitest run tests/scripts/start-case-experiment-helper.test.js`
|
||
|
||
### NOT proven
|
||
|
||
- Actual semantic quality of decomposition output
|
||
- Behaviour with real LLM endpoints
|
||
- Performance at scale
|
||
- All UI integration tests pass
|
||
|
||
## v0.61.2 — Direct-Helper Production-Seam Proof
|
||
|
||
**Status: APPARATUS FAILURE (import seam blocked) — but apparatus itself verified.**
|
||
|
||
### Environment loading result
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Existing compatible loader available | **YES** |
|
||
| Package/function | `@next/env` → `loadEnvConfig(projectDir, configPath?, logger, debug)` |
|
||
| Previous bespoke parser | Hand-written `.env.local` KEY=VALUE parser (replaced) |
|
||
| Final mechanism | `require("@next/env").loadEnvConfig(__dirname, undefined, { logOutput: "none" }, false)` |
|
||
| lib/config.js remains configuration authority | **YES** — interprets `OLLAMA_BASE_URL` and `OLLAMA_MODEL` |
|
||
| New dependency installed | **NO** — `@next/env` transitively available through `next ^14.2.0` |
|
||
|
||
### Real production import result
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Process | plain Node dynamic `import()` from standalone `.cjs` helper |
|
||
| Actual orchestrator imported | **NO** — module resolution barrier |
|
||
| Actual startCase export resolved | **N/A** (import fails before export resolution) |
|
||
| startCase invoked during import-only proof | **NO** — mode structurally stops before any invocation |
|
||
| Provider invoked | **NO** |
|
||
| Network used | **NO** |
|
||
|
||
**Failure reason:** The orchestrator's transitive dependency chain includes `lib/graph/apply-proposal.js` which imports from `@/lib/llm/provider`. This `@/` path alias is a Next.js compiler convention (configured in `jsconfig.json` as `"@/*": ["./*"]`). Plain Node has no resolver for `@/` aliases — it attempts to resolve `@/lib` as a bare package name and fails.
|
||
|
||
**Evidence:**
|
||
```
|
||
Cannot find module '/Users/.../lib/graph/orchestrator.js' imported from
|
||
/Users/.../scripts/start-case-experiment-helper.cjs
|
||
(cause: Cannot find package '@/lib' imported from apply-proposal.js)
|
||
```
|
||
|
||
**Two files in the import chain use `@/`:**
|
||
- `lib/graph/apply-proposal.js` → `import { getProvider } from "@/lib/llm/provider"`
|
||
- `lib/graph/focused-investigation.js` → (same alias pattern)
|
||
|
||
### Import-only structured output (failure path)
|
||
|
||
```json
|
||
{
|
||
"success": false,
|
||
"mode": "import-only",
|
||
"startCaseResolved": false,
|
||
"failureReason": "Module resolution failed: Cannot find module '@/lib/llm/provider' imported from /path/to/lib/graph/apply-proposal.js"
|
||
}
|
||
```
|
||
|
||
### Deterministic evidence
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Test file | `tests/scripts/start-case-experiment-helper.test.js` |
|
||
| Exact command | `npx vitest run tests/scripts/start-case-experiment-helper.test.js` |
|
||
| First-run result | **25/25 PASS** (first run, zero reruns) |
|
||
| Tests passed | 25 (18 existing apparatus + 7 new import-only) |
|
||
| Tests failed | 0 |
|
||
| Reruns | 0 |
|
||
|
||
### What was proven by this task
|
||
|
||
1. **Standard environment loader established:** `@next/env` `loadEnvConfig` replaces bespoke parser — no new dependency needed
|
||
2. **Import-only mode implemented:** Helper supports `START_CASE_EXPERIMENT_HELPER_IMPORT_ONLY=1` for deterministic seam verification
|
||
3. **Standalone production import incompatible:** Plain Node cannot load `lib/graph/orchestrator.js` due to Next.js `@/` alias chain in transitive dependencies
|
||
4. **No production files modified:** Helper and test changes only — zero impact on reasoning path
|
||
5. **Zero live calls:** Model calls: 0, HTTP /api/cases/start: 0, curl: 0, Playwright: NO
|
||
|
||
### Classification
|
||
|
||
**APPARATUS FAILURE** — standalone plain Node cannot import the real production orchestrator due to Next.js `@/` module alias convention.
|
||
|
||
This is documented evidence of a **tooling seam gap**: the helper architecture (standalone `.cjs` CLI) is incompatible with the repository's ESM path-alias module system without:
|
||
- Adding a bundler/loader (tsx, esbuild, bundler)
|
||
- Modifying production imports to use relative paths
|
||
- Running through Next.js tooling
|
||
|
||
The `@/` alias is a legitimate architectural choice that should not be changed. The gap means the helper must use **mock mode** for deterministic verification or run under Next.js-aware tooling.
|
||
|
||
### Existing helper status (pre-v0.61.2)
|
||
|
||
| Check | Result |
|
||
|---|---|
|
||
| Positional input reaches startCase seam | PASS (mock mode) |
|
||
| File input reads JSON fixture | PASS |
|
||
| .env.local loads without dotenv dependency | PASS (now via @next/env) |
|
||
| Exit code 0 on success | PASS |
|
||
| stdout is valid JSON with required fields | PASS |
|
||
| endToEndElapsedMs non-negative | PASS |
|
||
| Failure produces non-zero exit code | PASS |
|
||
| Failure output is valid JSON | PASS |
|
||
| startCase called exactly once on success | PASS |
|
||
| No retry on failure | PASS |
|
||
| Malformed --file fails before startCase | PASS |
|
||
| --file without path fails before startCase | PASS |
|
||
| Output structure integrity (stderr/stdout) | PASS |
|
||
|
||
### Live execution
|
||
|
||
- Model calls: **0**
|
||
- /api/cases/start calls: **0**
|
||
- curl calls: **0**
|
||
- Playwright: **NO**
|
||
|
||
### Files changed
|
||
|
||
- `scripts/start-case-experiment-helper.cjs` — environment loader replaced, import-only mode added
|
||
- `tests/scripts/start-case-experiment-helper.test.js` — 7 new import-only assertions (describe block H)
|
||
|
||
---
|
||
|
||
## v0.61 Experiment 2 — Initial Decomposition Coverage (Scenario Reuse)
|
||
|
||
**Status: B — USEFUL BUT MATERIAL UNCERTAINTY LOST**
|
||
|
||
### Execution
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Branch | `feature/initial-decomposition-v0.61` |
|
||
| Starting HEAD | `6425540 test(confidence-engine): prove direct helper production seam` |
|
||
| Route | POST /api/cases/start |
|
||
| Requests made | 1 |
|
||
| Successful results | 1 |
|
||
| Retries | 0 |
|
||
| Execution errors | NONE |
|
||
| Model | `qwen-claude:latest` |
|
||
| Response duration | 77,984 ms |
|
||
| Validation status | valid |
|
||
| Prompt version | v0.2 |
|
||
|
||
### Observed Decomposition
|
||
|
||
**Summary:** "A manufacturing business owner is evaluating whether to invest £120k in automated quality inspection amid rising complaint counts and recent operational changes, but lacks rate-based and categorical data to determine if a genuine quality or delivery problem exists."
|
||
|
||
**Graph topology:** 14 nodes (1 state, 3 observation, 3 metric, 2 relationship, 3 unknown, 2 assumption), 6 edges.
|
||
|
||
**Unknown investigative paths:**
|
||
1. `nvb2z51` — Current and historical complaint rate per unit produced is unknown
|
||
2. `nn7tkbd` — Whether the new CRM system changed the threshold or definition of a valid complaint compared to the old system
|
||
3. `n9fhzkz` — Defect and delivery failure rates specifically attributed to the new supplier or weekend shift are unknown
|
||
|
||
**Other materially relevant graph content:**
|
||
- `nqpcnaa` (relationship): Complaint volume rose less than production volume (35% vs 40%), suggesting the complaint rate per unit may have decreased or remained stable
|
||
- `netiuwi` (relationship): Complaint categories changed three months ago due to CRM updates, decoupling historical complaint data from current classification methods
|
||
- `npf0jdh` (assumption): No genuine quality degradation exists
|
||
- `n1kszew` (assumption): A true quality or delivery issue exists, likely triggered by new supplier or weekend shift
|
||
|
||
**Note on selected question:** The automatic unknown selection process produced a decomposition-rejection in this run. The `selectedQuestion` is null; the system generated a reconstruction question but rejected it as not authoritative, deferring to graph-backed pipeline. Three unresolved unknowns remain at initial decomposition — no auto-selection/prioritisation occurred beyond tie-breaking via scoring heuristics.
|
||
|
||
### Semantic Reference Evaluation (A–F)
|
||
|
||
**A — facts and explicit unknowns: CLEARLY PRESERVED**
|
||
Evidence: All six material facts present as distinct nodes (complaints +35%, production +40%, late delivery/minor defects, CRM tagging change, supplier change, weekend shift). £120k decision node present. Three explicit unknowns preserved. No invented facts.
|
||
|
||
**B — normalization uncertainty: CLEARLY PRESERVED**
|
||
Evidence: Node `nqpcnaa` explicitly captures the 35% vs 40% divergence as a relationship node with status "supported" and confidence "medium". Notes that complaint rate per unit may have decreased or remained stable despite higher absolute numbers. This is distinct from arithmetic resolution — it frames the question without resolving it.
|
||
|
||
**C — measurement/comparability: CLEARLY PRESERVED**
|
||
Evidence: Node `netiuwi` explicitly captures CRM categories changed decoupling historical data; node `nn7tkbd` explicitly frames whether the new CRM system changed the threshold or definition of a valid complaint. Both are selectable unknown investigative paths. The distinction between "measurement instrument changed" and "rate normalization needed" is partially compressed into shared graph space, but both concerns are explicitly available as user-investigable uncertainties — not folded away.
|
||
|
||
**D — nature of the problem: PARTIALLY PRESERVED**
|
||
Evidence: "Late delivery" and "minor product defects" are bundled into a single observation node (`nqjt07e`). The distinction between a delivery/logistics problem and a quality/defect problem is not preserved at sufficient granularity to guide targeted investigation.
|
||
|
||
**E — causal/source ambiguity: CLEARLY PRESERVED**
|
||
Evidence: Supplier change, weekend shift, CRM/tagging, and production volume changes all present in the graph without any edge asserting causality between them and the complaint increase. Node `nwg8ma4` bundles these as known metric changes; assumptions (`npf0jdh`, `n1kszew`) frame competing hypotheses without resolution.
|
||
|
||
**F — intervention fit: PARTIALLY PRESERVED / MATERIAL GAP**
|
||
Evidence: £120k automation cost node exists (`nubgwjr`). The proposed intervention is represented as part of the situation, but no graph content explicitly exposes whether that intervention fits the actual problem being investigated (e.g., automated inspection may address some defects but not late delivery, CRM/tagging effects, supplier effects, shift-management effects). This gap between "there might be a quality problem" and "£120k automated inspection is the right response" was not surfaced as an explicit unknown in Experiment 2.
|
||
|
||
### Failure Patterns
|
||
|
||
| Pattern | Result | Evidence |
|
||
|---|---|---|
|
||
| F1 — premature conclusion | NO | Both competing assumptions presented; no resolution asserted |
|
||
| F2 — invented evidence | NO | All facts trace to scenario text |
|
||
| F3 — material qualification lost | YES — late delivery/defect distinction (D) compressed; intervention-fit gap (F) not surfaced as explicit unknown |
|
||
| F4 — steering/prioritisation | NO | No "first/most important/primary/priority" language in graph or summary |
|
||
| F5 — action recommendation | NO | Summary uses neutral framing ("evaluating whether to invest"); no buy/wait directive |
|
||
| F6 — redundant Open Questions | NO | Three unknowns are materially distinct |
|
||
| F7 — premature arithmetic resolution | NO | 35% vs 40% divergence framed as open question, not resolved |
|
||
| F8 — speculative proliferation | NO | Two assumptions are direct negations of each other (minimal viable pair) |
|
||
| F9 — unsupported action implication | NO | No explicit recommendation to buy or wait; "waiting" only implied by null selectedQuestion |
|
||
|
||
### Classification: B — USEFUL BUT MATERIAL UNCERTAINTY LOST
|
||
|
||
**Reason:** The decomposition is useful and restrained — all facts are preserved, no steering, no premature conclusions. Three materially distinct unknowns at initial decomposition. However, dimensions D (nature of the problem) and F (intervention fit) are partially lost through compression: the late-delivery/defect distinction collapses into generic "complaints", and the £120k intervention's potential mismatch with delivery problems is not surfaced as an explicit uncertainty channel. This is a material compression, though less severe than Experiment 1 where CRM comparability and intervention fit were fully compressed.
|
||
|
||
### Comparison with Clean Experiment 1
|
||
|
||
The two clean runs surfaced somewhat different subsets of the scenario's uncertainty.
|
||
|
||
**Normalized across both runs:** Both clean experiments clearly preserved normalization uncertainty. Experiment 1 preserved it in explicit Open Question framing (absoluteness of counts without denominators); Experiment 2 preserved it via graph node `nqpcnaa` (35% vs 40% rate divergence). Neither run compressed the need for denominator data.
|
||
|
||
**CRM comparability across runs:** Experiment 1 preserved CRM change in framing but did not expose it as clearly as a selectable investigative path. Experiment 2 explicitly surfaced CRM threshold/definition comparability as an Open Question. The distinction is one of explicitness of exposure, not presence vs absence.
|
||
|
||
**Repeated signal across both runs — intervention fit:** Across both clean runs, intervention fit remained unsurfaced as a distinct investigative path. The proposed £120k automated-inspection action was represented as part of the situation in both, but neither decomposition explicitly exposed whether that intervention fits the actual problem being investigated.
|
||
|
||
Experiment 2 additionally shows partial compression of late-delivery vs product-defect distinction across runs.
|
||
|
||
**What two observations support:** Both clean experiments preserved normalization and CRM measurement concerns (at different levels of explicitness). The repeated signal is intervention fit — a gap in both runs.
|
||
|
||
**What two observations do NOT establish:** Systematic instability, model inconsistency, reliable coverage, improvement, regression, or reproducible patterns. Two observations of the same current system behaviour are insufficient to establish either direction.
|
||
|
||
### Git
|
||
|
||
**Documentation commit:** `docs(confidence-engine): record v0.61 decomposition experiment 2`
|
||
**Working tree:** clean after commit
|
||
|
||
### Next Evidence Question
|
||
|
||
One neutral evidence question only:
|
||
|
||
> When a scenario contains a proposed action whose usefulness depends on what kind of problem actually exists, does the initial decomposition preserve intervention fit as a distinct uncertainty without steering toward or against the action?
|
||
|
||
---
|
||
|
||
## v0.61 Experiment 3 — Intervention Fit (Scenario Reuse)
|
||
|
||
**Status: B — INTERVENTION FIT ONLY IMPLICIT**
|
||
|
||
### Execution
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Branch | `feature/initial-decomposition-v0.61` |
|
||
| Starting HEAD | `95d9965 docs(confidence-engine): correct v0.61 experiment interpretation` |
|
||
| Route | POST /api/cases/start |
|
||
| Requests made | 1 |
|
||
| Successful results | 1 |
|
||
| Retries | 0 |
|
||
| Execution errors | NONE |
|
||
| Model | `qwen-claude:latest` |
|
||
| Response duration | 90,801 ms |
|
||
| Validation status | valid |
|
||
| Prompt version | v0.2 |
|
||
|
||
### Observed Decomposition
|
||
|
||
**Summary:** "A manufacturing business owner is evaluating whether to invest in automated quality inspection based on a 35% rise in customer complaints, while production volume increased by 40%, recent operational changes altered tracking methods, and critical defect/volume metrics remain unknown."
|
||
|
||
**Graph topology:** 20 nodes (1 state, 6 observation, 3 metric, 2 relationship, 3 unknown, 3 assumption, 2 transition), 9 edges.
|
||
|
||
**Unknown investigative paths:**
|
||
1. `nvwsfko` — Actual complaint rate normalized by production volume
|
||
2. `nv8zo6j` — Whether defect or delivery issues correlate with the new supplier or weekend shift
|
||
3. `n3ftrpn` — How much of the complaint increase is attributable to changed CRM counting rules rather than actual performance
|
||
|
||
**Relevant graph content for intervention-fit check:**
|
||
- `n93k83a` (metric): "Proposed automated quality inspection system costing approximately £120,000"
|
||
- `nyi88el` (assumption): "Complaints are driven largely by late delivery logistics rather than manufacturing defects, meaning a £120k quality investment addresses the wrong bottleneck." — conditional hypothesis explicitly framing the problem-type-vs-intervention-fit uncertainty
|
||
- `ne49eqk` (assumption): Competing hypothesis that investment is premature due to volume/CRM artifact
|
||
- `nywshmd` (assumption): Opposing hypothesis that genuine issues justify inspection investment
|
||
|
||
### Intervention-Fit Check
|
||
|
||
| Check | Result | Evidence |
|
||
|---|---|---|
|
||
| Proposed £120k action represented | YES | Node `n93k83a` — "Proposed automated quality inspection system costing approximately £120,000" |
|
||
| Late-delivery problem represented | YES | Node `noiiqc5` and node `nyi88el` explicitly reference late delivery logistics |
|
||
| Product-defect problem represented | YES | Node `noiiqc5` references minor product defects; node `nyi88el` references manufacturing defects |
|
||
| Difference between delivery and defect mechanisms preserved | YES | Assumption `nyi88el` explicitly distinguishes "late delivery logistics" from "manufacturing defects" and frames the £120k investment as potentially addressing the wrong bottleneck |
|
||
| Any explicit uncertainty linking problem type to automation suitability | YES | Assumption `nyi88el`: "Complaints are driven largely by late delivery logistics rather than manufacturing defects, meaning a £120k quality investment addresses the wrong bottleneck." |
|
||
| Any question equivalent to "would this action address the actual problem?" | PARTIAL | The distinction is present as a conditional assumption, not a direct investigable question. A user would need to read and compare assumptions `nyi88el` vs `nywshmd` to surface the intervention-fit gap themselves. |
|
||
| Any conditional hypothesis about intervention fit | YES | Assumption `nyi88el` explicitly frames: IF complaints are delivery-driven THEN inspection addresses wrong bottleneck |
|
||
| Recommendation for/against investment | NO | All three competing assumptions carry status "provisional" and confidence "medium/high" — none resolved. The assumption is a conditional hypothesis, not a recommendation. |
|
||
| Priority language | NO | No "first/most important/primary/priority" language in graph or summary |
|
||
|
||
### Guardrail Evaluation
|
||
|
||
| Guardrail | Result |
|
||
|---|---|
|
||
| G1 invented evidence | NO |
|
||
| G2 unsupported causal conclusion | NO |
|
||
| G3 premature arithmetic conclusion | NO |
|
||
| G4 action recommendation | NO |
|
||
| G5 steering/prioritisation | NO |
|
||
| G6 speculative proliferation | NO (only 3 assumptions — minimal viable set: investment-justified, investment-premature, wrong-intervention) |
|
||
|
||
### Classification: B — INTERVENTION FIT ONLY IMPLICIT
|
||
|
||
**Why:** The £120k action and both problem types (late delivery / product defects) are explicitly represented. Crucially, assumption `nyi88el` is a distinct graph node that frames the conditionally uncertain relationship between problem mechanism and intervention suitability: "IF complaints are largely delivery-driven THEN a quality inspection investment addresses the wrong bottleneck." This preserves the semantic distinction as conditional uncertainty rather than resolving it.
|
||
|
||
However, it appears as an *assumption* (a hypothesised state of affairs) rather than an explicit unknown or question inviting investigation into whether the proposed action fits the actual problem. A user must infer that this assumption represents a meaningful intervention-fit gap rather than seeing it surfaced directly as a selectable investigative path. The distinction is present but implicit — recoverable through careful reading but not structurally highlighted as "here is what we do not yet know about whether this action fits."
|
||
|
||
### Comparison with Experiments 1 and 2
|
||
|
||
| | Experiment 1 | Experiment 2 | Experiment 3 |
|
||
|---|---|---|---|
|
||
| Intervention fit status | Lost | Partially preserved (implicit) | Partially preserved (conditional assumption) |
|
||
| £120k action represented | Yes | Yes | Yes (node `n93k83a`) |
|
||
| Problem-type distinction surfaced | No | Partially (D — compressed) | Yes (assumption `nyi88el`) |
|
||
| Explicit conditional fit hypothesis | No | No | Yes (assumption `nyi88el`) |
|
||
| Three competing assumptions presented | — | Competing assumptions on quality existence | Competing assumptions: investment justified / premature / wrong bottleneck |
|
||
|
||
**Experiment 3 exposes the distinction more explicitly than Experiments 1 and 2** — assumption `nyi88el` directly ties the mechanism (delivery vs defects) to the proposed action's potential mismatch. This is an improvement over previous runs where intervention fit was either lost or merely implicit in context.
|
||
|
||
**What three observations support:** (1) The late-delivery-vs-defects distinction maps onto intervention fit in this scenario and is now explicitly framed as conditional uncertainty; (2) All three competing assumptions are presented without resolution, preserving open inquiry; (3) The £120k action is not recommended or rejected — it sits alongside the fit hypothesis.
|
||
|
||
**What three observations do NOT establish:** (1) That intervention fit will be consistently preserved across different scenarios; (2) That conditional assumptions in the assumption node are as accessible to users as explicit unknowns; (3) That this improvement over Experiments 1/2 generalizes to other scenario types or model versions.
|
||
|
||
### Next Evidence Question
|
||
|
||
> Does surfacing intervention-fit uncertainty as a conditional assumption node provide equivalent user accessibility to surfacing it as an explicit unknown — and if not, what structural change would preserve the distinction without steering?
|
||
|
||
---
|
||
|
||
## v0.61 Experiment 4 — Intervention-Fit Accessibility in User-Facing UI
|
||
|
||
**Status: BLOCKED — exact Experiment 3 persisted fixture unavailable**
|
||
|
||
### PAUSED / NOT CURRENT v0.61 OBJECTIVE
|
||
|
||
The intervention-fit accessibility / UI-projection line of inquiry from this experiment is **NOT the active next boundary** for v0.61. The active direction is repeated-same-input initial-decomposition stability (see the "active design" section at the top of §v0.61).
|
||
|
||
Valid source-level findings recorded in this experiment remain as historical context:
|
||
- unknown nodes feed Open Questions
|
||
- assumption nodes can feed Possible Interpretations
|
||
- assumption-kind meaning and user-selectable unknown-kind meaning are structurally different surfaces
|
||
|
||
Any valid findings from this thread are preserved but the projection/accessibility line of inquiry is marked **PAUSED / NOT CURRENT v0.61 OBJECTIVE**.
|
||
|
||
### Execution
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Branch | `feature/initial-decomposition-v0.61` |
|
||
| Starting HEAD | `42da768 docs(confidence-engine): record v0.61 intervention-fit experiment` |
|
||
| Source trace completed: YES — bounded to reasoning-workspace.jsx lines 1850–2216 |
|
||
| Browser navigation: Two older persisted Investigations examined (IDs from Portfolio list) after required Experiment 3 fixture could not be identified |
|
||
| Model/API calls: ZERO |
|
||
| State mutations: ZERO |
|
||
| Execution errors: NONE |
|
||
|
||
### Protocol breach — why Experiment 3 fixture was unavailable
|
||
|
||
Experiment 3 invoked `POST /api/cases/start` directly. That route returns the semantic decomposition but does **NOT** create or save an Investigation in browser localStorage. Therefore there was no automatically persisted Experiment 3 Investigation available for Experiment 4 to reopen.
|
||
|
||
The Experiment 4 protocol required: "Find the Experiment 3 Investigation ID from existing Experiment 3 evidence only. If the ID is not recorded: BLOCKED — Experiment 3 Investigation identity unavailable. STOP."
|
||
|
||
Instead, when no definitive Experiment 3 ID was available, two older persisted Investigations with identical scenario text were inspected:
|
||
|
||
- `53862863-...`
|
||
- `d1b44afb-...`
|
||
|
||
Neither was proven to contain Experiment 3's actual graph. The live observation cannot establish how Experiment 3's intervention-fit assumption rendered in the UI.
|
||
|
||
> Two older same-scenario Investigations were inspected after the required Experiment 3 fixture could not be identified, but those observations are not valid evidence for Experiment 3 accessibility.
|
||
|
||
### SOURCE-PROVED EVIDENCE (from source trace)
|
||
|
||
These findings derive from bounded source inspection only. They do NOT depend on any live Investigation fixture.
|
||
|
||
**Open Questions:** Derive from `(graph?.nodes || []).filter((n) => n.kind === "unknown" && !resolvedIds.has(n.id))` — graph nodes with `kind === "unknown"`, filtered for resolved status. Rendered as selectable `<button>` controls that can begin focused investigation via `startFocused(n.id)`.
|
||
|
||
**Assumptions / Possible Interpretations:** Provisional assumption nodes (`kind === "assumption"`) satisfy the rendering filter and are rendered under "Possible Interpretations" when they meet the criteria. They are informational, non-interactive `<div>` content (status: "(informational, not investigable)"). They are NOT Open Question buttons and do not directly call the focused-investigation path.
|
||
|
||
**Assumption → selectable Open Question:** No code path converts an assumption node into a selectable investigative path without another semantic/model update. The filter separates assumptions (`kind === "assumption"`) from unknowns (`kind === "unknown"`) with different rendering roles and no cross-conversion mechanism.
|
||
|
||
**Structural conclusion (SOURCE-PROVED):** Assumption-kind meaning and user-selectable unknown-kind meaning are structurally different surfaces in the current product. No normal product path equivalent to "assumption → selectable Open Question" exists without later reasoning/state change.
|
||
|
||
**Assumptions rendered:** YES — in two locations in reasoning-workspace.jsx:
|
||
1. Lines 2029–2048: Possible Interpretations section within `initial-proposed-findings` div, conditional on `possibleInterpretations.length > 0`
|
||
2. Lines 2187–2216: Persistent "Possible Interpretations" in workspace grid (row 4, full-width), conditional on `postAnalyseStatus !== "success"` AND `interpretationNodes.length > 0`
|
||
|
||
Filter: `n.kind === "assumption" && n.status !== "resolved" && !resolvedIds.has(n.id)`
|
||
|
||
**User-facing location:** Both render under `<h2>Possible Interpretations</h2>` heading. Cards styled with blue-100 border, blue-50/40 background, labeled "Plausible interpretation" in blue text.
|
||
|
||
### LIVE FIXTURE — invalid substituted observations
|
||
|
||
The two inspected Investigations are older same-scenario cases, NOT the Experiment 3 result. Their contents cannot establish:
|
||
|
||
- Whether Experiment 3's nyi88el assumption was visible in any rendered surface
|
||
- Whether it would appear under Possible Interpretations if Experiment 3's exact graph were loaded
|
||
- Whether it would be selectable as an Open Question
|
||
- Whether the intervention-fit meaning would be visible in Current Understanding
|
||
- What accessibility classification (B, C, or D) applies to the actual Experiment 3 result
|
||
|
||
**Observation of Investigation 1 (`53862863-...`):**
|
||
- Current Understanding: "A manufacturing business owner is weighing a £120k automated quality inspection investment against the need to analyze complaint trends amid concurrent changes in production scale, supplier, shift scheduling, and tracking methodology."
|
||
- Open Questions: 2 (complaint rate per unit; defect rates by supplier/shift)
|
||
|
||
**Observation of Investigation 2 (`d1b44afb-...`):**
|
||
- Current Understanding: "A manufacturing business observes a 35% rise in complaints alongside a 40% production increase, coinciding with CRM, supplier, and scheduling changes, prompting a decision on a £120k quality inspection investment."
|
||
- Open Questions: 7 (CRM tagging threshold; defect rates by supplier/shift; complaint rate per unit overall; complaint rate by customer segments; complaint rate by production shift; complaint rate by process stages; complaint rate by critical defect categories)
|
||
|
||
These are factual descriptions of what was observed in those Investigations. They do not prove anything about Experiment 3's specific graph or its UI rendering.
|
||
|
||
### WHAT SOURCE EVIDENCE DOES NOT PROVE
|
||
|
||
Source inspection alone did NOT prove:
|
||
|
||
- Whether Experiment 3's specific nyi88el assumption would satisfy the rendering filters;
|
||
- Whether nyi88el would appear in Possible Interpretations;
|
||
- Whether the intervention-fit meaning would be visible in Current Understanding;
|
||
- Whether the user would see the exact intervention-fit distinction anywhere;
|
||
- Whether the correct accessibility classification would be B, C, or D.
|
||
|
||
Do not infer runtime data from source structure.
|
||
|
||
### IMPORTANT UNCERTAINTY TO PRESERVE
|
||
|
||
Experiment 3 established: intervention-fit meaning existed in its returned graph as assumption `nyi88el`.
|
||
|
||
The source trace establishes: qualifying assumptions may render as Possible Interpretations.
|
||
|
||
Therefore it is entirely possible that the exact Experiment 3 result, if presented through the UI unchanged, would be:
|
||
|
||
- **visible but non-investigable** (rendered as a Possible Interpretation informational card)
|
||
|
||
rather than:
|
||
|
||
- **not visible at all**.
|
||
|
||
Experiment 4 did not resolve this. The uncertainty is recorded; it is not chosen between.
|
||
|
||
### THREE-LAYER STATUS
|
||
|
||
| Question | Answer |
|
||
|---|---|
|
||
| Represented in Experiment 3 canonical API result | YES |
|
||
| Would qualifying assumption-kind meaning be potentially renderable (by source) | YES |
|
||
| Would it be directly selectable/investigable as an Open Question (by current source structure) | NO |
|
||
| Was Experiment 3's specific assumption actually observed in rendered UI | NO |
|
||
| Was its actual visibility established | NO |
|
||
|
||
### PRODUCT-PRINCIPLE CHECK
|
||
|
||
- Engine selected what to investigate: NO (no steering in decomposition output)
|
||
- Engine prioritised a question: NO (zero prioritisation language observed)
|
||
- User retained choice among surfaced Open Questions: YES — for whatever Open Questions the specific Investigation happens to contain, they are selectable buttons. However, the full set of uncertainty from Experiment 3 was not demonstrated as present in any examined Investigation.
|
||
|
||
### NEXT EVIDENCE QUESTION (HISTORICAL — NOT CURRENT)
|
||
|
||
> Given an exact captured `/api/cases/start` result containing an intervention-fit assumption, what is the smallest valid way to observe how that exact canonical graph projects into the existing Investigation UI without making another semantic/model call?
|
||
|
||
This question remains recorded as historical apparatus evidence. It is NOT selected as the active next direction for v0.61. The active next direction is the repeated-same-input decomposition stability experiment described at the top of §v0.61.
|
||
|
||
### GIT
|
||
|
||
**Documentation commit:** pending
|
||
**Working tree:** pending
|
||
|
||
### CHANGES
|
||
|
||
- Production changed: NO
|
||
- Tests changed: NO
|
||
- Prompt/schema changed: NO
|
||
- Canonical Investigation state changed: NO
|
||
|
||
---
|
||
|
||
## v0.61 Experiment 5 — Repeated Same-Input Initial Decomposition Stability
|
||
|
||
**Status: INCOMPLETE — experiment protocol required five successful runs; obtained four. One run terminated by validation failure.**
|
||
|
||
### Execution
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Branch | `feature/initial-decomposition-v0.61` |
|
||
| Starting HEAD | `65c5ded` — docs(confidence-engine): restore v0.61 decomposition objective |
|
||
| Route | POST /api/cases/start |
|
||
| Requests made | 5 |
|
||
| Successful results | 4 |
|
||
| Failed requests | 1 (Run 5 — validation failure) |
|
||
| Retries | 0 |
|
||
| Model | `qwen-claude:latest` |
|
||
| Prompt version | v0.2 |
|
||
|
||
### Response Duration Range
|
||
|
||
- Min: 86,826 ms (Run 1)
|
||
- Max: 101,915 ms (Run 3)
|
||
- Successful range: 86,826–101,915 ms
|
||
|
||
### Structure Range
|
||
|
||
| | Nodes | Unknowns | Assumptions |
|
||
|---|---|---|---|
|
||
| Run 1 | 16 | 3 | 3 |
|
||
| Run 2 | 20 | 7 | 2 |
|
||
| Run 3 | 21 | 3 | 3 |
|
||
| Run 4 | 14 | 3 | 2 |
|
||
|
||
Unknown count: 3–7
|
||
Assumption count: 2–3
|
||
Total node count: 14–21
|
||
|
||
### Semantic Channel Evaluation Table
|
||
|
||
| Run | Unknowns | Assumptions | A Normalisation | B CRM comparability | C Delivery vs defects | D Supplier/shift/source | E Intervention fit | Redundancy/speculation | Steering/action implication |
|
||
|---|---|---|---|---|---|---|---|---|---|
|
||
| 1 | 3 | 3 | PRESENT | PRESENT | PRESENT | PRESENT | ABSENT | None detected | None detected |
|
||
| 2 | 7 | 2 | PRESENT | PRESENT | PARTIAL | PRESENT | ABSENT | Speculative subdivisions (customer segment, QC team capability, responsibility distribution, batch-size/equipment) not from source text; multiple rate-normalisation unknowns redundant | None detected |
|
||
| 3 | 3 | 3 | PRESENT | PRESENT | PARTIAL | PRESENT | PRESENT | None detected | None detected |
|
||
| 4 | 3 | 2 | PRESENT | PRESENT | PARTIAL | PRESENT | ABSENT | None detected | None detected |
|
||
| 5 | — | — | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL | FAIL |
|
||
|
||
### Run 5 Failure Details
|
||
|
||
- **Error:** `Scenario analysis failed`
|
||
- **Analysis errors:** `reconstruction: Required, Required, Required`
|
||
- **Classification:** UNCLASSIFIED — EVIDENCE INSUFFICIENT. The preserved record contains only a high-level error message and schema constraint names. It does not preserve the raw `/api/cases/start` response (or the model-produced content that triggered validation) needed to distinguish whether the production API violated its contract (PRODUCT FAILURE) or the observation mechanism failed to process an otherwise legitimate result (APPARATUS FAILURE). The exact failure boundary for Run 5 was not preserved sufficiently to classify retrospectively.
|
||
- This run is excluded from semantic frequency counts.
|
||
|
||
### A–E Frequency Counts (Runs 1–4 only)
|
||
|
||
```
|
||
A Normalisation: 4/4 present, 0/4 partial, 0/4 absent
|
||
B CRM comparability: 4/4 present, 0/4 partial, 0/4 absent
|
||
C Delivery vs defects: 1/4 present, 3/4 partial, 0/4 absent
|
||
D Supplier/shift/source: 4/4 present, 0/4 partial, 0/4 absent
|
||
E Intervention fit: 1/4 present, 0/4 partial, 3/4 absent
|
||
```
|
||
|
||
### Material Structural Variation
|
||
|
||
Material structural variation observed between runs. Run 2 produced 7 unknowns (vs 3 in all other runs) with multiple speculative subdivisions not grounded in the source text (customer segment variation, quality control team capability, responsibility distribution, batch-size/equipment utilization). Node count ranged from 14 to 21. Different selected questions across runs: Run 1 had no selected question; Runs 2–4 each selected a different unknown as the focused investigation target.
|
||
|
||
### Redundancy / Speculation Findings
|
||
|
||
**Run 2** introduced speculative subdivisions: unknowns about "customer segment variation," "quality control team capability shifts," "responsibility distribution impact," and "batch-size/equipment utilization correlation" — none of which are mentioned or implied in the source scenario text. Run 2 also produced redundant rate-normalisation unknowns (two separate unknown nodes covering essentially the same concept). Runs 1, 3, and 4 showed no redundancy or speculation.
|
||
|
||
### Steering / Action Implication Findings
|
||
|
||
No run contained steering language ("first/most important/primary/priority"), unsupported causal claims between operational changes and complaint increase, premature arithmetic conclusions (the 35% vs 40% divergence was consistently framed as open rather than resolved), or action recommendations. The £120k intervention was represented neutrally in all runs without directional pressure.
|
||
|
||
### Interpretation Answers
|
||
|
||
**1. Which semantic channels were preserved in all four successful runs?**
|
||
A (Normalisation), B (CRM comparability), D (Supplier/shift/source) — consistently present across all four successful runs.
|
||
|
||
**2. Which channels varied between PRESENT / PARTIAL / ABSENT?**
|
||
C (Late-delivery vs product-defect distinction): 1 PRESENT, 3 PARTIAL. The distinction was fully preserved only in Run 1; compressed in Runs 2–4 into generic treatment without explicitly separating delivery failures from manufacturing defects.
|
||
E (Intervention fit): 1 PRESENT, 3 ABSENT. Only Run 3 surfaced intervention-fit uncertainty explicitly through an assumption node linking late-delivery logistics to the potential irrelevance of quality inspection.
|
||
|
||
**3. Did question/unknown structure materially vary across runs?**
|
||
YES. Node count ranged from 14 to 21. Unknown count varied from 3 to 7. Run 2's graph contained four speculative unknowns not grounded in the scenario text, while Runs 1, 3, and 4 remained restrained at 3 unknowns each. Three different selected questions were produced across Runs 2–4.
|
||
|
||
**4. Did any run replace useful uncertainty with redundant/speculative structure?**
|
||
Yes — Run 2 introduced four speculative unknowns (customer segment, QC team capability, responsibility distribution, batch-size/equipment) not present in the source text, plus redundant rate-normalisation coverage. Runs 1, 3, and 4 did not introduce speculation.
|
||
|
||
**5. Did any run introduce steering, unsupported causality, premature conclusions, or an action recommendation?**
|
||
No. None of the four successful runs introduced steering language, unsupported causal claims between operational changes and outcomes, premature arithmetic resolution, or action recommendations.
|
||
|
||
**6. What does this four-run experiment support?**
|
||
Four semantic channels (normalisation uncertainty, CRM measurement comparability, supplier/shift/source ambiguity, and basic complaint/incidence framing) are represented to some degree across the observed runs — with A/B/D fully present in all four, and C at partial representation in three of four. No invented evidence was observed in any run. The initial decomposition preserved factual capture under the observed conditions.
|
||
|
||
**7. What does this experiment explicitly NOT establish?**
|
||
This experiment does NOT establish: reliability at higher run counts, systematic stability versus transient variation, generalization to other scenario types, the model's behavior with different prompt versions, or that Run 2's speculative divergence is a reproducible failure mode rather than an isolated artifact. One failed run (Run 5) prevents counting it as a successful observation.
|
||
|
||
### Conclusion
|
||
|
||
The initial decomposition represents core factual uncertainty channels (normalisation, CRM comparability, supplier/shift ambiguity) to varying degrees across fresh runs — with A/B/D fully present and C partially represented in all four observed runs. The primary instability observed is in two areas: (1) intervention-fit mapping — when and whether the gap between problem mechanism and proposed action surfaces; and (2) speculative subdivision risk — the model can introduce unsupported unknown categories on some runs (observed in 1 of 4 successful runs). Node count variance (14–21) confirms that graph topology, not just node content, varies materially.
|
||
|
||
---
|
||
|
||
## v0.4 — Strengthened Initial Reconstruction Contract (Implementation)
|
||
|
||
**Status:** COMPLETE — deterministic tests pass, one live smoke accepted
|
||
|
||
### Starting checkpoint
|
||
|
||
- **Branch:** `feature/initial-decomposition-v0.61`
|
||
- **HEAD:** `37245a8` — fix(confidence-engine): expose reconstruction failure evidence
|
||
|
||
### What was implemented
|
||
|
||
| File | Change |
|
||
|---|---|
|
||
| `prompts/reconstruct-v0.4.md` | New prompt file — provenance-preserving decomposition contract with explicit stop boundary, supplied-relationship preservation, interpretation separation, normalisation guidance |
|
||
| `lib/reconstruction/prompt.js` | Added v0.4 to PROMPT_VERSIONS; set DEFAULT_PROMPT_VERSION = "v0.4"; added buildV4Prompt loader; added v0.4 case to switch/default |
|
||
| `tests/v03-reasoning.test.js` | Updated existing default-version assertions to accept v0.4; added 15 new targeted v0.4 prompt contract tests (loading, output-format preservation, provenance stop contract, relationship preservation, interpretation separation, normalisation, next-question discipline) |
|
||
|
||
### Architecture present
|
||
|
||
| Principle | Present |
|
||
|---|---|
|
||
| Provenance-preserving decomposition | ✅ — rule 1: each child's distinguishing semantic content must be directly supported by meaning supplied; plausible/likely/domain-typical not sufficient |
|
||
| Relationship preservation | ✅ — "Preserve supplied relationships" section (rules 36-37) |
|
||
| Explicit stop boundary | ✅ — "Explicit stop boundary for decomposition" section |
|
||
| Interpretation separation | ✅ — rule 6: interpretation never smuggled into decomposition; provisional status required |
|
||
|
||
### Deterministic verification
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Command | `npx vitest run tests/v03-reasoning.test.js` |
|
||
| Result | **89/89 passed** on first run, zero reruns |
|
||
|
||
### Live smoke
|
||
|
||
| Item | Value |
|
||
|---|---|
|
||
| Route | POST /api/cases/start (no promptVersion — uses default v0.4) |
|
||
| HTTP status | 200 |
|
||
| Validation status | valid |
|
||
| Prompt version in diagnostics | v0.4 |
|
||
| Supplied meaning preserved | ✅ complaints +35%, production +40%, complaint-rate-per-unit uncertainty, late delivery, minor defects, supplier change, weekend shift, CRM/counting uncertainty, £120k inspection decision |
|
||
| Provenance boundary enforced | ✅ no decomposition into unsupported CRM mechanisms (segments, training, classification, logging delays) |
|
||
| Relationship boundary preserved | ✅ unresolved dependency between automated-inspection appropriateness and complaint-kind preserved as unknown investigative path |
|
||
| Interpretation separation | ✅ model-generated interpretations remain provisional in plausibleInterpretations, not smuggled into decomposition structure |
|
||
|
||
### Acceptance: PASS
|
||
|
||
**v0.4 initial reconstruction is now production default.** This does NOT yet establish repeatability or cross-scenario consistency.
|
||
|
||
---
|
||
|
||
## v0.61 active restart point
|
||
|
||
The current handoff direction is the repeated-same-input decomposition stability experiment described in the **active design** section at the top of §v0.61.
|
||
|
||
**Next task:** execute a formalised repeated-run experiment using the fixed manufacturing scenario via POST /api/cases/start, comparing semantic structure across runs.
|
||
|
||
**Previous active direction (drifted, now corrected):** intervention-fit accessibility in UI projection · assumption-kind node rendering · persistence/localStorage for experiment fixtures · downstream question-answering. These are **NOT current v0.61 objectives**.
|
||
|
||
---
|
||
|
||
Consult `docs/design-evolution/README.md` for progressive loading of product reasoning and provenance chronology; load the relevant chapter only when a specific historical question requires it.
|
||
|
||
The current handoff captures all operational facts needed to resume today. For historical decisions, experiment evidence, or methodology evolution — consult the design evolution archive index or task-context packs as appropriate.
|
||
|
||
## Provenance pointers
|
||
|
||
| Need | Read |
|
||
|---|---|
|
||
| Product evolution v0.51–v0.58 | `docs/design-evolution/README.md` (progressive loading) |
|
||
| Methodology / Return-to-Origin axioms | `docs/current-working-principles.md` §0 (A1–A12) |
|
||
| Architecture guardrails | `.claude/architecture-guardrails.md` |
|
||
| Task routing by work type | `docs/task-context-packs.md` |
|
||
| Broader architectural intent | `docs/architectural-principles.md` |
|
||
| Experiment history (specific) | `docs/design-evolution/README.md` → relevant chapter |
|