215 lines
17 KiB
Markdown
215 lines
17 KiB
Markdown
# Current Handoff — Confidence Engine
|
||
|
||
> **Role:** Concise operational snapshot for resuming work today. Not a historical diary.
|
||
> The design evolution archive index at `docs/design-evolution/README.md` provides progressive loading of experiment history; load the relevant chapter only when a specific historical question requires it.
|
||
|
||
## v0.62d production Docker packaging established
|
||
|
||
- `Dockerfile` — minimal multi-stage Alpine build (Node 22), Next.js standalone output mode
|
||
- `.dockerignore` — excludes dev artefacts, secrets, docs from build context
|
||
- `next.config.mjs` — added `output: 'standalone'` (required for lean production container)
|
||
- `.env.example` — reorganized: Supabase → Ollama → Mock sections; Ollama vars now tracked as deployment-relevant
|
||
- **npm production build: PASS** ✓
|
||
- **Docker image build: BLOCKED** — no Docker runtime on development machine (apparatus limitation, not product defect)
|
||
- Production container starts / `/api/health` smoke test: pending Docker runtime availability
|
||
- No persistent application volume required
|
||
- Supabase remains external, Ollama remains private and server-reachable
|
||
- Public `NEXT_PUBLIC_*` variables may require build-time injection via `--build-arg` as established by the implementation (baked into browser bundle)
|
||
- **No deployment performed yet**
|
||
|
||
## CURRENT MVP DIRECTION
|
||
|
||
Initial-decomposition hardening is frozen for the current MVP stage.
|
||
|
||
## Authenticated product boundary (v0.62a)
|
||
|
||
- Confidence Engine uses self-hosted Supabase Auth with magic-link email, `/auth/callback` code exchange, cookie-backed sessions, and protected product routes/API requests; unauthenticated API requests receive 401.
|
||
- Investigation persistence is now server-authoritative via Supabase `confidence_engine.investigations`. Browser persistence flows through authenticated Next.js API. No `confidence_engine` database schema, tables, snapshot ownership fields, or PostgREST configuration were changed in v0.62c (established in v0.62b).
|
||
|
||
## Database foundation (v0.62c)
|
||
|
||
- Server-authoritative investigation persistence via Supabase `confidence_engine.investigations` as durable authority; browser persistence flows through authenticated Next.js API (`/api/investigations`).
|
||
- `lib/storage/providers/server-http.js` replaces localStorage as the backing provider for `lib/storage/investigation-storage.js`. The storage seam now owns async load/save and per-investigation coalescing autosave (rapid concurrent saves collapse to the latest snapshot).
|
||
- Async hydration adapted across Portfolio, Investigation, Report, and ScenarioForm.
|
||
- Restart preserved via shared transformation in `lib/storage/restart-investigation.js`; server-backed restart endpoint reuses this same transformation.
|
||
- Portfolio-compatible server summary projection (`scenario`, `updatedAt`, `investigationRevision`, `reportExists`, `reportGeneratedFromRevision`).
|
||
- Missing-new-investigation 404 maps to `null` at the load boundary in `server-http.js`.
|
||
- Live save and Portfolio reload persistence proven by manual evidence on Sep 8.
|
||
- Live application-level user isolation proven: second authenticated user sees clean Portfolio; original user regains only their server-backed investigation.
|
||
- localStorage is no longer production authority. Legacy localStorage investigations remain physically present but invisible to normal product flow. No dual-write. No automatic legacy import.
|
||
- Duplicate investigation GETs observed on development reload; one database row and one Portfolio card confirmed. No data-integrity defect established. No optimisation undertaken.
|
||
|
||
**Current product checkpoint:** Read `docs/confidence-engine-product-checkpoint-2026-09-08.md` before planning new product, live-evidence, or commercial work. The core investigation loop is now sufficiently established to prioritise realistic end-to-end use, report experience, prospective-user value, repeat use, and willingness to pay—not endless isolated reasoning-mechanics experiments. Preserve user ownership and address trust-critical defects when found.
|
||
|
||
Do not resume:
|
||
- repeated-same-input reconstruction experiments
|
||
- Qwen/Terra reconstruction comparison
|
||
- initial prompt refinement
|
||
- supplier/weekend-shift decomposition experiments
|
||
- relationship-preservation experiments
|
||
- causal-fidelity experiments
|
||
|
||
unless new end-to-end user-flow evidence reopens one of those boundaries.
|
||
|
||
**Immediate strategic evidence question:**
|
||
|
||
Does the complete investigation process leave real people materially clearer about genuinely difficult situations, repeatedly enough that they will pay to use it? Use measured realistic scenarios and bounded live action budgets; the September 8 checkpoint records the current Terra cost/latency evidence.
|
||
|
||
## Server-owned UI journey provider experiment
|
||
|
||
- Server-only `CONFIDENCE_ENGINE_EXPERIMENT_PROVIDER=openai` centrally resolves the existing OpenAI provider and `gpt-5.6-terra`; without it, normal production resolution remains Ollama/Qwen. An OpenAI key alone does not switch providers.
|
||
- Browser request contracts and client state remain unchanged. Deterministic coverage includes initial start, normal update, episode reconsideration, focused deconstruction, overview synthesis, and Current Understanding synthesis.
|
||
- Zero live calls occurred. Next boundary: one real Playwright-driven Terra investigation measuring user-visible latency, actual LLM-call sequence, and OpenAI usage/cost.
|
||
|
||
## OpenAI alternate structured output
|
||
|
||
- The OpenAI provider now honors a caller-supplied structured-output schema, applying its existing strict-schema transport projection; absent an alternate schema, initial reconstruction retains its existing strict schema and transport normalization.
|
||
- The next live run exposed incomplete recursive strict projection: OpenAI rejected `relationships.items` because it lacked `additionalProperties: false`. The projector now recognizes every `type: "object"` node, including property-less objects in array items, and recursively enforces strict object schemas while preserving initial-reconstruction optionality/nullability behavior.
|
||
- The latest Terra request then exposed an inconsistent root `properties`/`required` contract. The projector now derives `required` after projection from the surviving property keys, and recursive tests verify `properties`, `required`, and `additionalProperties` consistency. Property-less object strictness and initial-reconstruction transport behavior remain preserved.
|
||
- Current deterministic final-fetch schema remains internally valid, yet the live rejection contradicts it. `CONFIDENCE_ENGINE_EXPERIMENT_TRACE_OPENAI_SCHEMA=1` now emits one safe, server-side structural summary immediately before the OpenAI fetch—no prompt, answer, request body, secret, or model output.
|
||
- The focused-deconstruction route now emits complementary safe server diagnostics for start, provider success/failure, focused validation, and end status; the OpenAI schema trace remains provider-owned. No user content or secrets are logged.
|
||
- Canonical focused relationship items are now strict `{ from, to, type }`: all required non-empty strings, free-text `type`, and no `rationale`; `relationships` remains required and may be `[]`. Schema, prompt field names, and validator align; the stale rationale-bearing test fixture was corrected.
|
||
- Live Terra focused deconstruction now passes through the real UI, but the following Current Understanding synthesis exposed a separate caller/schema mismatch: its prompt and validator require `{ currentUnderstanding }` while the provider received the default initial-reconstruction schema.
|
||
- Terra synthesis now supplies its own output schema and unwraps `providerResult.response` before validation; deterministic synthesis coverage is 62/62 PASS and the saved isolated Terra synthesis POST returned HTTP 200.
|
||
- Real Terra completed-episode reconsideration can legitimately return no additional meaningful graph change. Completed episodes now tolerate only that exact compatibility outcome; ordinary no-op updates and invalid completed-episode proposals remain rejected, without fake graph mutation or new next-question steering.
|
||
- The full apply-proposal owner suite remains known-red in independent pre-existing 60B.43 tests, so the changed Done-for-now boundary was verified through exact isolated owner tests. Zero live calls occurred during closeout. Next boundary: reuse the existing investigation and click Done for now once, observing cases/update and subsequent synthesis.
|
||
- Live UI proved Done for now briefly clarified the question, then server graph replacement reopened it: the client sends `preDoneGraph`, and the server previously had no deterministic closure owner. Episode-mode update now adds the selected `targetNodeId` to `resolvedNodeIds` only after successful semantic application; semantic no-ops and meaningful mutations remain valid, while failed episodes do not resolve the target and ordinary updates are unchanged. Zero live calls occurred during implementation. Next boundary: one live Done-for-now check on the existing investigation.
|
||
|
||
## Repository checkpoint
|
||
|
||
- **Branch:** `feature/product-platform-foundation-v0.62`
|
||
- **HEAD:** `6dd447e` — feat(confidence-engine): add authenticated investigation persistence
|
||
- **Working tree:** dirty with completed v0.62c cutover (server-authoritative persistence, async seam, 404→null correction)
|
||
|
||
## Persistence
|
||
|
||
- **Owner:** `lib/storage/providers/server-http.js` (authenticated browser HTTP provider). `lib/storage/investigation-storage.js` owns the application-facing boundary with coalescing autosave and async load/save.
|
||
- **Durable authority:** Supabase `confidence_engine.investigations` (RLS-scoped, user-owned).
|
||
- **localStorage:** legacy only — physically present but invisible to normal product flow. No dual-write. No automatic import.
|
||
- **Restart transformation:** shared in `lib/storage/restart-investigation.js`; used by both browser seam and server persistence layer.
|
||
|
||
## Canonical experiment apparatus — currently valid
|
||
|
||
```
|
||
scripts/start-case-experiment-helper.cjs — canonical helper (tsx runtime)
|
||
tsx runtime — alias-capable experiment execution
|
||
--file — file-input mode (JSON fixture)
|
||
--reconstruction-only — stop after initial reconstruction, no question generation
|
||
import-only proof — prove startCase() import chain resolves without live calls
|
||
OpenAI reconstruction experiment seam — START_CASE_EXPERIMENT_PROVIDER=openai (experiment-only)
|
||
focused-deconstruction schema fix — plumbing verified, 48/48 tests pass
|
||
```
|
||
|
||
**No other experimental apparatus is currently valid.** Obsolete setups have been removed or superseded.
|
||
|
||
## Current product architecture
|
||
|
||
Three distinct routes:
|
||
|
||
```
|
||
/ → Portfolio (notebook index)
|
||
/investigations/{id} → Investigation (working case/pages)
|
||
/investigations/{id}/report → Investigation Report (derived summary)
|
||
```
|
||
|
||
**Portfolio:** investigation collection loaded from server API (`/api/investigations`). Actions per card: View report, Continue investigation, Restart. "+ Create new investigation" allocates durable ID via `crypto.randomUUID()` + navigates.
|
||
|
||
**Investigation:** `ScenarioForm` + `ReasoningWorkspace`. Handles focused turns, Done/Re-open semantics, Current Understanding synthesis.
|
||
|
||
**Report:** rendered persisted `investigationReport` snapshot. On-demand generation (one `/api/cases/overview` on first visit; zero on subsequent). Not canonical reasoning evidence — derived artefact.
|
||
|
||
## Evidence discipline invariants
|
||
|
||
```
|
||
RAW USER EVIDENCE
|
||
≠ MODEL-DERIVED CONTRIBUTION SEMANTICS
|
||
≠ CURRENT CANONICAL FINDING
|
||
≠ IMMUTABLE SOURCE OBSERVATION
|
||
≠ USER DISPOSITION / AUTHORITY
|
||
≠ TURN CONTEXT / PROVENANCE
|
||
```
|
||
|
||
**Finding dispositions:** `null` (eligible working premise), `agree` (user-endorsed), `not_relevant` (excluded from eligible reasoning, provenance retained). User disposition never directly mutates authoritative graph state.
|
||
|
||
## Persistence
|
||
|
||
- **Owner:** `lib/storage/providers/server-http.js` (authenticated browser HTTP provider). `lib/storage/investigation-storage.js` owns the application-facing seam with coalescing autosave.
|
||
- **Durable authority:** Supabase `confidence_engine.investigations` (RLS-scoped, user-owned).
|
||
- **localStorage:** legacy only — physically present but invisible to normal product flow. No dual-write. No automatic import.
|
||
- **Identity:** durable `id` allocated by application, not storage.
|
||
- **First persistence:** when user produces meaningful state (scenario submitted), not on create-click.
|
||
- **Restart:** preserves container/id/scenario; clears reasoning/report state via shared transformation in `lib/storage/restart-investigation.js`.
|
||
|
||
## MVP boundaries
|
||
|
||
**Implemented:**
|
||
- Multi-investigation via localStorage with durable IDs
|
||
- Report freshness (`Current` / `Update available`) derived from revision comparison
|
||
- Manual `Update report` action
|
||
- Confirmation-gated restart
|
||
- No automatic regeneration
|
||
- Duplicate prevention guard
|
||
|
||
**Deferred beyond MVP:**
|
||
- Search, tag, archive, group within Portfolio
|
||
- Export/copy Reports to external document
|
||
- Report history / comparison
|
||
- Multi-provider production routing (OpenAI/Terra comparison is frozen)
|
||
|
||
## Current development constraints
|
||
|
||
- Canonical dev server at `http://localhost:3000`. Never start/stop/restart/probe it. If unavailable → BLOCKED and stop.
|
||
- Playwright MCP: use **Run Playwright code** with semantic locators. Snapshot refs are observational only.
|
||
- For async/hydration states: use `waitFor({ state: 'visible', timeout })` — not arbitrary sleeps.
|
||
- If a prescribed semantic locator cannot find its control → STOP. No fallback to CSS/XPath/DOM traversal.
|
||
- Live freeze: once Playwright verification begins, no production file edits until evidence is captured.
|
||
- **Tests are instruments, not product truth.** At first deterministic failure: classify PRODUCT vs APPARATUS, then stop.
|
||
- A failed prescribed UI step is evidence, not permission to explore: semantic `page.getByRole(...)` locators only; do not substitute actions, retry model-backed steps, or navigate away from a manually positioned persisted state. Define action budgets for live cost experiments.
|
||
|
||
## Current genuinely open boundaries
|
||
|
||
**Not yet implemented:**
|
||
- Multi-investigation portfolio (search/tag/archive/group)
|
||
- Export/copy of Reports to external document
|
||
- Production multi-provider routing (Ollama/Qwen default remains)
|
||
|
||
**Known limitations:**
|
||
- Current Understanding and Investigation Report are architecturally distinct artefacts. Report interpretations remain explicit, not evidence.
|
||
- The `≤5` processing bound was experimental apparatus constraint, NOT product requirement.
|
||
|
||
## HISTORICAL — initial-decomposition v0.61 status
|
||
|
||
Initial decomposition experiments (v0.61) ran four successful runs plus one validation failure against the same manufacturing scenario. Key findings:
|
||
|
||
- **Consistently preserved across all runs:** normalisation uncertainty, CRM comparability, supplier/shift ambiguity
|
||
- **Variable:** late-delivery vs defect distinction (1 PRESENT, 3 PARTIAL)
|
||
- **Rarely surfaced:** intervention-fit uncertainty (1 PRESENT, 3 ABSENT)
|
||
- **Run 2 introduced speculative subdivisions** not grounded in source text
|
||
- No steering language, unsupported causality, or action recommendations in any run
|
||
- Graph topology varies materially (14–21 nodes) across runs — this is expected, not a defect
|
||
|
||
Qwen/Terra matched evidence showed Terra with stronger stability on C and D1/D2 distinctions; Qwen preserved meaning but compressed more frequently. Both providers vary graph topology while preserving core dependencies.
|
||
|
||
These results are documented as **historical experiment evidence**. The v0.61 line is frozen — see CURRENT MVP DIRECTION above.
|
||
|
||
## Provenance pointers
|
||
|
||
| Need | Read |
|
||
|---|---|
|
||
| Product evolution v0.51–v0.60 | `docs/design-evolution/README.md` (progressive loading) |
|
||
| Methodology / RTO axioms | `docs/current-working-principles.md` §0 (A1–A12) |
|
||
| Architecture guardrails | `.claude/architecture-guardrails.md` |
|
||
| Task routing by work type | `docs/task-context-packs.md` |
|
||
| September 8 product/economics checkpoint | `docs/confidence-engine-product-checkpoint-2026-09-08.md` |
|
||
| Full experiment history (specific) | `docs/design-evolution/README.md` → relevant chapter |
|
||
|
||
Consult `docs/current-project-state.md` for broader project context and passive classifier status.
|
||
|
||
## Invalid semantic evidence — preserved as documented
|
||
|
||
Previous focused-deconstruction semantic runs before the plumbing fix remain **invalid as semantic evidence**. They produced all-zero semantic fields because:
|
||
1. Provider envelope buried semantic fields under wrapper (not `.response`)
|
||
2. Initial-reconstruction schema was supplied to focused-deconstruction call instead of its own contract
|
||
|
||
These are recorded as known contamination in the v0.61 archive chapter (`docs/design-evolution/ch19/initial-decomposition-v0.61.md`). The plumbing fix is complete and verified (48/48 tests).
|