confidence-engine/docs/confidence-engine-product-checkpoint-2026-09-08.md

# Confidence Engine Product Checkpoint — 2026-09-08

## Purpose

Record the evidence boundary at which Confidence Engine moves from proving isolated reasoning mechanics toward commercially testing the working product loop. This is a current checkpoint, not a claim of universal provider or production validation.

## PROVEN / OBSERVED

### Working product loop

```text
messy scenario → initial reconstruction / SituationGraph → Current Understanding
→ Open Questions → user selects a question → focused multi-turn investigation
→ canonical Findings and accumulated Contributions → Done for now
→ authoritative clarification → completed-episode reconsideration
→ Current Understanding resynthesis → user chooses what to investigate next
```

When all Open Questions are clarified, the direction is to surface a report derived from accumulated case understanding. It is not a recommendation engine and must not claim that the user is ready, sufficiently informed, confident, or should act.

The engine facilitates the user's reasoning; it does not steer it. The user owns question selection, depth, Done-for-now, re-opening, sufficiency, confidence, and eventual action.

### Multi-turn, closure, and persistence

- A manufacturing supplier thread retained one existing plus five additional answer turns: six learned Contributions in total.
- After the sixth contribution, Done for now clarified the supplier question, removed it from Open Questions, preserved the investigation, and resynthesised Current Understanding without retrying or selecting another question.
- A completed episode may legitimately have no semantic graph mutation. Done for now remains user-owned; the server makes the target authoritative in graph resolution state without fabricating semantic change.
- Persisted investigations resume Current Understanding, Open and clarified Questions, focused Contributions, Findings, and graph identity/provenance. Re-open preserves learned thread state.

### Re-open graph-state fix

Canonical persisted Done-for-now state is an `unknown` node with `status: "unknown"` whose ID is in `resolvedNodeIds`. The former Re-open check required `status: "resolved"`, silently no-oping across both Playwright Chrome and normal Chrome persisted state. Commit `85b9f44` uses authoritative resolution state for unknown nodes.

Deterministic evidence: `npx vitest run tests/graph/reopen-resolved-unknown.test.js --environment=node` — 1 file, 14/14 tests passed. Live evidence: supplier `resolvedNodeIds` changed from `["n9joe0e"]` to `[]`, and the supplier question visibly returned to Open Questions in both browsers.

### Provider position and measured Terra journeys

- Available routes: OpenAI / GPT-5.6 Terra and Ollama / network Qwen. Terra has shown good-enough semantic behavior on tested reasoning paths; neither route is universally validated.
- Clean shallow manufacturing journey (start → supplier question → one answer → Done for now → refreshed Current Understanding): 6 OpenAI requests and $0.12. Observed browser timings: `/start` 40.04s; formulate 0.56s; deconstruct 3.49s; synthesis 3.67s; `/update` 21.50s; synthesis 4.59s.
- Extending the same supplier thread from one to six Contributions required five additional answers and one Done-for-now completion, with no retries, other questions, or new investigation: +10 requests and +$0.05, moving the dashboard to 16 requests and $0.17. Current dashboard totals were 37.862K input, 6.822K output, 44.684K cumulative tokens. Earlier token figures lack a confirmed input/output split and are not total-token evidence.

## PROVISIONAL / EXTRAPOLATED

With five Open Questions remaining in the representative manufacturing scenario, a six-question investigation of broadly similar depth may plausibly cost $0.50–$0.80 in Terra inference, with ~$0.60 as a working midpoint. This is an extrapolation, not a validated production unit cost; replace it with a measured full-scenario run.

Inference cost is not yet the evident commercial constraint. Latency is more conspicuous: graph-level `/start` is about 40s and observed `/update` runs are about 20–40s, while focused deconstruction and synthesis are commonly low-single-digit seconds. The likely optimisation target is time-to-first-value, subject to measurement.

## KNOWN BUT NOT CURRENT WORK

- Measure the operations dominating `/start` and `/update`, and whether each must block the visible transition. Do not assume initial reconstruction alone is the issue because `/update` has comparable latency.
- Turns 3–6 revisited whether unit-level records could connect defects/materials to a supplier after evidence repeatedly established those records were unavailable. This is a future evidence-boundary-exhaustion question, not an instruction for the engine to stop or steer the user.
- Investigation-turn allowances may become a commercial entitlement mechanism, per investigation or monthly. They must never imply epistemic sufficiency or tell a user to stop.

## NEXT STRATEGIC DIRECTION

The immediate question is no longer whether the fundamental reasoning architecture can work at all. Prioritise complete realistic scenarios, trust-critical failures encountered in real use, a commercially testable report experience, a small realistic end-to-end set, prospective-user testing, repeat use, willingness to pay, and what users value. Do not use this transition to ignore genuine trust-critical defects or to return to theoretical perfection work before user-value evidence.

## Development discipline for bounded live work

A failed prescribed UI step is evidence, not permission to explore. Use exact semantic `page.getByRole(...)` locators; snapshot refs are observational only. Do not use selector fallbacks, navigate/restart/refresh without instruction, substitute questions, or retry model-backed actions. Stop at the first unexpected state and classify PRODUCT FAILURE or APPARATUS FAILURE. When a persisted browser state is manually positioned, do not navigate away. Define explicit action budgets for live cost experiments.