206 lines
14 KiB
Markdown
206 lines
14 KiB
Markdown
# Current Handoff — Confidence Engine
|
||
|
||
> **Role:** Concise operational snapshot for resuming work today. Not a historical diary.
|
||
> The design evolution archive index at `docs/design-evolution/README.md` provides progressive loading of experiment history; load the relevant chapter only when a specific historical question requires it.
|
||
|
||
## CURRENT MVP DIRECTION
|
||
|
||
Initial-decomposition hardening is frozen for the current MVP stage.
|
||
|
||
Do not resume:
|
||
- repeated-same-input reconstruction experiments
|
||
- Qwen/Terra reconstruction comparison
|
||
- initial prompt refinement
|
||
- supplier/weekend-shift decomposition experiments
|
||
- relationship-preservation experiments
|
||
- causal-fidelity experiments
|
||
|
||
unless new end-to-end user-flow evidence reopens one of those boundaries.
|
||
|
||
**Immediate next evidence question:**
|
||
|
||
Measure one real browser investigation configured for OpenAI/Terra across every LLM stage, including user-visible latency, call sequence, and cost.
|
||
|
||
If YES, the next live experiment is one timed/costed OpenAI UI investigation measuring:
|
||
- user-visible latency
|
||
- OpenAI call count
|
||
- token usage where available
|
||
- approximate cost per investigation
|
||
|
||
## Server-owned UI journey provider experiment
|
||
|
||
- Server-only `CONFIDENCE_ENGINE_EXPERIMENT_PROVIDER=openai` centrally resolves the existing OpenAI provider and `gpt-5.6-terra`; without it, normal production resolution remains Ollama/Qwen. An OpenAI key alone does not switch providers.
|
||
- Browser request contracts and client state remain unchanged. Deterministic coverage includes initial start, normal update, episode reconsideration, focused deconstruction, overview synthesis, and Current Understanding synthesis.
|
||
- Zero live calls occurred. Next boundary: one real Playwright-driven Terra investigation measuring user-visible latency, actual LLM-call sequence, and OpenAI usage/cost.
|
||
|
||
## OpenAI alternate structured output
|
||
|
||
- The OpenAI provider now honors a caller-supplied structured-output schema, applying its existing strict-schema transport projection; absent an alternate schema, initial reconstruction retains its existing strict schema and transport normalization.
|
||
- The next live run exposed incomplete recursive strict projection: OpenAI rejected `relationships.items` because it lacked `additionalProperties: false`. The projector now recognizes every `type: "object"` node, including property-less objects in array items, and recursively enforces strict object schemas while preserving initial-reconstruction optionality/nullability behavior.
|
||
- The latest Terra request then exposed an inconsistent root `properties`/`required` contract. The projector now derives `required` after projection from the surviving property keys, and recursive tests verify `properties`, `required`, and `additionalProperties` consistency. Property-less object strictness and initial-reconstruction transport behavior remain preserved.
|
||
- Current deterministic final-fetch schema remains internally valid, yet the live rejection contradicts it. `CONFIDENCE_ENGINE_EXPERIMENT_TRACE_OPENAI_SCHEMA=1` now emits one safe, server-side structural summary immediately before the OpenAI fetch—no prompt, answer, request body, secret, or model output.
|
||
- The focused-deconstruction route now emits complementary safe server diagnostics for start, provider success/failure, focused validation, and end status; the OpenAI schema trace remains provider-owned. No user content or secrets are logged.
|
||
- Canonical focused relationship items are now strict `{ from, to, type }`: all required non-empty strings, free-text `type`, and no `rationale`; `relationships` remains required and may be `[]`. Schema, prompt field names, and validator align; the stale rationale-bearing test fixture was corrected.
|
||
- Live Terra focused deconstruction now passes through the real UI, but the following Current Understanding synthesis exposed a separate caller/schema mismatch: its prompt and validator require `{ currentUnderstanding }` while the provider received the default initial-reconstruction schema.
|
||
- Terra synthesis now supplies its own output schema and unwraps `providerResult.response` before validation; deterministic synthesis coverage is 62/62 PASS and the saved isolated Terra synthesis POST returned HTTP 200.
|
||
- Real Terra completed-episode reconsideration can legitimately return no additional meaningful graph change. Completed episodes now tolerate only that exact compatibility outcome; ordinary no-op updates and invalid completed-episode proposals remain rejected, without fake graph mutation or new next-question steering.
|
||
- The full apply-proposal owner suite remains known-red in independent pre-existing 60B.43 tests, so the changed Done-for-now boundary was verified through exact isolated owner tests. Zero live calls occurred during closeout. Next boundary: reuse the existing investigation and click Done for now once, observing cases/update and subsequent synthesis.
|
||
- Live UI proved Done for now briefly clarified the question, then server graph replacement reopened it: the client sends `preDoneGraph`, and the server previously had no deterministic closure owner. Episode-mode update now adds the selected `targetNodeId` to `resolvedNodeIds` only after successful semantic application; semantic no-ops and meaningful mutations remain valid, while failed episodes do not resolve the target and ordinary updates are unchanged. Zero live calls occurred during implementation. Next boundary: one live Done-for-now check on the existing investigation.
|
||
|
||
## Repository checkpoint
|
||
|
||
- **Branch:** `feature/initial-decomposition-v0.61`
|
||
- **HEAD:** `5878ce4` — experiment(confidence-engine): add reconstruction-only helper flag
|
||
- **Working tree:** clean after this session's commit
|
||
|
||
## Initial reconstruction — current status
|
||
|
||
**Semantically stable enough for current MVP stage.** Exact graph topology is not stable and is not treated as an invariant. Trust-critical meaning must remain stable. Some compression is acceptable when meaning survives downstream. Missing meaning cannot be faithfully recovered downstream. Causal hypotheses must remain visibly provisional.
|
||
|
||
Current production default: `reconstruct-v0.5` prompt + canonical reconstruction schema + Zod validation via `z.toJSONSchema()`.
|
||
|
||
The `/api/cases/start` route returns validated initial reconstruction, situation graph, and selected question. Observability seam exposes the exact object used by `buildInitialGraph()` for comparison.
|
||
|
||
**Frozen:** initial decomposition, prompt refinement, Qwen/Terra comparison — see CURRENT MVP DIRECTION above.
|
||
|
||
## Focused investigation — current status
|
||
|
||
Focused deconstruction plumbing fixes are complete:
|
||
- Schema mismatch resolved (focused route now supplies its own `focusedDeconstructJsonSchema`)
|
||
- Provider envelope no longer leaks into validator (inner `.response` unwrapped correctly)
|
||
- All 48 focused-investigation-boundary tests pass on first run
|
||
|
||
Focused deconstruction receives only:
|
||
- `centralStatement`
|
||
- `targetLabel`
|
||
- `targetDescription`
|
||
- `question`
|
||
- `answer`
|
||
|
||
Full SituationGraph / original scenario / previous findings are **not** supplied to that route. This is intentional epistemic separation.
|
||
|
||
Repeatability: supplier/weekend-shift epistemic separation repeated 3/3 on the fixed case after plumbing fix. Previous pre-fix semantic runs remain invalid (contaminated by provider-envelope misuse + wrong transport schema).
|
||
|
||
## Canonical experiment apparatus — currently valid
|
||
|
||
```
|
||
scripts/start-case-experiment-helper.cjs — canonical helper (tsx runtime)
|
||
tsx runtime — alias-capable experiment execution
|
||
--file — file-input mode (JSON fixture)
|
||
--reconstruction-only — stop after initial reconstruction, no question generation
|
||
import-only proof — prove startCase() import chain resolves without live calls
|
||
OpenAI reconstruction experiment seam — START_CASE_EXPERIMENT_PROVIDER=openai (experiment-only)
|
||
focused-deconstruction schema fix — plumbing verified, 48/48 tests pass
|
||
```
|
||
|
||
**No other experimental apparatus is currently valid.** Obsolete setups have been removed or superseded.
|
||
|
||
## Current product architecture
|
||
|
||
Three distinct routes:
|
||
|
||
```
|
||
/ → Portfolio (notebook index)
|
||
/investigations/{id} → Investigation (working case/pages)
|
||
/investigations/{id}/report → Investigation Report (derived summary)
|
||
```
|
||
|
||
**Portfolio:** investigation collection with actions per card (View report, Continue investigation, Restart). "+ Create new investigation" allocates durable ID via `crypto.randomUUID()` + navigates.
|
||
|
||
**Investigation:** `ScenarioForm` + `ReasoningWorkspace`. Handles focused turns, Done/Re-open semantics, Current Understanding synthesis.
|
||
|
||
**Report:** rendered persisted `investigationReport` snapshot. On-demand generation (one `/api/cases/overview` on first visit; zero on subsequent). Not canonical reasoning evidence — derived artefact.
|
||
|
||
## Evidence discipline invariants
|
||
|
||
```
|
||
RAW USER EVIDENCE
|
||
≠ MODEL-DERIVED CONTRIBUTION SEMANTICS
|
||
≠ CURRENT CANONICAL FINDING
|
||
≠ IMMUTABLE SOURCE OBSERVATION
|
||
≠ USER DISPOSITION / AUTHORITY
|
||
≠ TURN CONTEXT / PROVENANCE
|
||
```
|
||
|
||
**Finding dispositions:** `null` (eligible working premise), `agree` (user-endorsed), `not_relevant` (excluded from eligible reasoning, provenance retained). User disposition never directly mutates authoritative graph state.
|
||
|
||
## Persistence
|
||
|
||
- **Owner:** `lib/storage/providers/local-storage.js` (`saveInvestigation` / `loadInvestigation`)
|
||
- **Key prefix:** `confidence-engine-investigation:<durable-id>`
|
||
- **Storage contract:** `lib/storage/investigation-storage.js` (application-facing boundary)
|
||
- **Identity:** durable `id` allocated by application, not storage
|
||
- **First persistence:** when user produces meaningful state (scenario submitted), not on create-click
|
||
- **Restart:** preserves container/id/scenario; clears reasoning/report state
|
||
|
||
## MVP boundaries
|
||
|
||
**Implemented:**
|
||
- Multi-investigation via localStorage with durable IDs
|
||
- Report freshness (`Current` / `Update available`) derived from revision comparison
|
||
- Manual `Update report` action
|
||
- Confirmation-gated restart
|
||
- No automatic regeneration
|
||
- Duplicate prevention guard
|
||
|
||
**Deferred beyond MVP:**
|
||
- Search, tag, archive, group within Portfolio
|
||
- Export/copy Reports to external document
|
||
- Report history / comparison
|
||
- Multi-provider production routing (OpenAI/Terra comparison is frozen)
|
||
|
||
## Current development constraints
|
||
|
||
- Canonical dev server at `http://localhost:3000`. Never start/stop/restart/probe it. If unavailable → BLOCKED and stop.
|
||
- Playwright MCP: use **Run Playwright code** with semantic locators. Snapshot refs are observational only.
|
||
- For async/hydration states: use `waitFor({ state: 'visible', timeout })` — not arbitrary sleeps.
|
||
- If a prescribed semantic locator cannot find its control → STOP. No fallback to CSS/XPath/DOM traversal.
|
||
- Live freeze: once Playwright verification begins, no production file edits until evidence is captured.
|
||
- **Tests are instruments, not product truth.** At first deterministic failure: classify PRODUCT vs APPARATUS, then stop.
|
||
|
||
## Current genuinely open boundaries
|
||
|
||
**Not yet implemented:**
|
||
- Multi-investigation portfolio (search/tag/archive/group)
|
||
- Export/copy of Reports to external document
|
||
- Production multi-provider routing (Ollama/Qwen default remains)
|
||
|
||
**Known limitations:**
|
||
- Current Understanding and Investigation Report are architecturally distinct artefacts. Report interpretations remain explicit, not evidence.
|
||
- The `≤5` processing bound was experimental apparatus constraint, NOT product requirement.
|
||
|
||
## HISTORICAL — initial-decomposition v0.61 status
|
||
|
||
Initial decomposition experiments (v0.61) ran four successful runs plus one validation failure against the same manufacturing scenario. Key findings:
|
||
|
||
- **Consistently preserved across all runs:** normalisation uncertainty, CRM comparability, supplier/shift ambiguity
|
||
- **Variable:** late-delivery vs defect distinction (1 PRESENT, 3 PARTIAL)
|
||
- **Rarely surfaced:** intervention-fit uncertainty (1 PRESENT, 3 ABSENT)
|
||
- **Run 2 introduced speculative subdivisions** not grounded in source text
|
||
- No steering language, unsupported causality, or action recommendations in any run
|
||
- Graph topology varies materially (14–21 nodes) across runs — this is expected, not a defect
|
||
|
||
Qwen/Terra matched evidence showed Terra with stronger stability on C and D1/D2 distinctions; Qwen preserved meaning but compressed more frequently. Both providers vary graph topology while preserving core dependencies.
|
||
|
||
These results are documented as **historical experiment evidence**. The v0.61 line is frozen — see CURRENT MVP DIRECTION above.
|
||
|
||
## Provenance pointers
|
||
|
||
| Need | Read |
|
||
|---|---|
|
||
| Product evolution v0.51–v0.60 | `docs/design-evolution/README.md` (progressive loading) |
|
||
| Methodology / RTO axioms | `docs/current-working-principles.md` §0 (A1–A12) |
|
||
| Architecture guardrails | `.claude/architecture-guardrails.md` |
|
||
| Task routing by work type | `docs/task-context-packs.md` |
|
||
| Full experiment history (specific) | `docs/design-evolution/README.md` → relevant chapter |
|
||
|
||
Consult `docs/current-project-state.md` for broader project context and passive classifier status.
|
||
|
||
## Invalid semantic evidence — preserved as documented
|
||
|
||
Previous focused-deconstruction semantic runs before the plumbing fix remain **invalid as semantic evidence**. They produced all-zero semantic fields because:
|
||
1. Provider envelope buried semantic fields under wrapper (not `.response`)
|
||
2. Initial-reconstruction schema was supplied to focused-deconstruction call instead of its own contract
|
||
|
||
These are recorded as known contamination in the v0.61 archive chapter (`docs/design-evolution/ch19/initial-decomposition-v0.61.md`). The plumbing fix is complete and verified (48/48 tests).
|