confidence-engine/docs/current-handoff.md

Current Return-to-Work Handoff — Confidence Engine

This file describes only the latest stopping point. Replace its current-work sections when the project moves on. Historical evidence remains in the design log and archive.

1. Where We Left It

• Engine experiments resumed with a passive validation;
• UI experiments remain paused;
• Knowledge-management experiments are complete;
• Experiment 39 tested the existing Behaviour Selection module against real Investigation State Assessment outputs across three scenarios;
• Acknowledge dominates (71% of selections) because it fires first when health=healthy, blocking Summarise/Pause/Clarify even in concluding or stalled states.

This handoff describes the latest stopping point only. When work moves on, replace stale current-work details rather than appending another historical note. Historical experiment and commit information belongs in docs/design-evolution-log.md.

2. What Is True Now

• Main active engine path: deterministic reasoning pipeline (scenario reconstruction, graph update, unknown selection, question formulation, turn orchestration).
• Passive experimental classifiers from Experiments 18–25B remain isolated diagnostic layers; none control the user-facing investigation. Behaviour Selection was passively evaluated against real assessment outputs in Experiment 39 — it produced all valid behaviours but with skewed distribution (Acknowledge 71%).
• Keyword and phrase-based scope detection remains provisional scaffolding.
• docs/current-project-state.md is the main entry point for active project state.
• Experiment 54D confirmed the production update prompt explicitly separates the user answer (## User Answer section) but the proposal schema has no provenance field — source identity at prompt level is explicit, per-node provenance at output level is absent.
• docs/task-context-packs.md chooses the minimum context documents for each work type.

3. Why Work Is Paused

Engine and UI work were deliberately paused because documentation had grown large enough to overload Claude and make returning across sessions difficult. The current phase is simplifying what a fresh session must load to understand the project, without losing evidential history. Historical material remains available under docs/archive/.

4. What Was Just Completed

Experiment 37 corrected the routing defect from Experiment 36 and tested a cross-boundary engine/UI task. It validated that two context packs can be combined deliberately while keeping working context small, explicit and accurate. All seven knowledge-management criteria are now met. No source code changed. No files moved or deleted.

Commit: 544573a (experiment: validate cross-boundary context routing)

Experiment 38 tested whether a genuinely cold session (no prior conversation context) can recover the project state from three documents alone. It recovered all capabilities, boundaries, and context-pack selection correctly without loading the full history or source code. All seven knowledge-management criteria confirmed met. One handoff update required: the open item "whether the handoff stays accurate after further advances" was resolved (handoff is accurate). The cold-start test passed.

Commit: pending (experiment: validate cold-start project recovery) — to be committed this session.

Experiment 39 resumed reasoning experiments with a passive validation of Behaviour Selection against real Investigation State Assessment outputs. Seven turns across three scenarios were evaluated. Acknowledge dominated (71%) because it fires at priority 1 whenever health=healthy, even in terminal and stalled states where Summarise or Pause would be more useful. The assessor→selector contract aligns cleanly; no transformation is needed between pipeline stages. All five behaviours remain reachable but some never appear in typical scenarios (Clarify requires too_broad health which few fixtures produce). Status pending Rob's review.

Experiment 40 diagnosed the root causes: Summarise and Pause fire their rules in real data but are always blocked by Acknowledge's priority-1 position (priority conflict, not assessor failure). Clarify's triggers never activate in tested scenarios due to the too_broad health condition being extremely narrow. All five behaviours confirmed independently reachable in synthetic isolation. No rules changed.

Experiment 41 compared two passive alternatives for reducing Acknowledge dominance:

• Variant A (priority reordering): evaluate Summarise/Pause before Acknowledge — introduces false-positive summarise in focusing phase
• Variant B (Acknowledge exclusions): keep priority, gate Acknowledge when phase=concluding/synthesising or progress=stalled or health=user_overloaded — recommended
• Both variants converge on the same two genuine changes: concluding→summarise and stalled→pause Experiment 42 implemented Variant B's narrow Acknowledge exclusion gate in the production selector (commit 05d3d96). Summarise now appears at conclusion; Pause now appears when stalled. All other tested turns remain unchanged. Behaviour Selection remains passive and isolated with no runtime caller — active user-facing engine behaviour did not change.

Experiment 43 audited Clarify readiness across all 10 real assessment turns in existing fixtures. Zero turns produced Clarify-eligible states. Two findings: (1) the orienting-based Clarify rule is dead code because the assessor never produces phase=orienting, and (2) the too_broad trigger requires conditions no fixture exercises. Branch: feature/user-workspace-ux-v0.7.

Experiment 44 created one deliberately unclear starting scenario (five competing unknowns, zero resolved evidence, vague central statement) to test whether the assessor produces a Clarify-justifying signal. The assessor returned too_broad conversation health — confirming the previously untested too_broad path works correctly with real data. Clarify became eligible via Rule A. No production code changed. Remaining open: whether orienting phase is needed for earlier-stage clarification, and whether 2–3 competing threads (below the >3 threshold) can represent genuine scope confusion. Status pending Rob's review.

Experiment 45 tested the too_broad boundary from two to five competing unknowns using identical synthetic fixtures varying only in unknown count. The assessor switched at exactly three→four active unknowns — two and three returned cannot_determine; four and five returned too_broad. Clarify eligibility followed the same boundary. Resolved-item gate works correctly: one resolved item stays too_broad, two resolves it. The boundary appears mechanically clear but conceptually uncertain — synthetic fixtures cannot confirm whether three-to-four feels right to real users. No production code changed. What remains open: whether health should default to healthy (not cannot_determine) for 2–3 unknowns with no question; whether the threshold needs widening for real-world use. Status closed.

Experiment 46 compared two four-unknown investigations with identical structural counts — one coherent (four unknowns contributing to one decision) and one scattered (four unrelated threads). Both returned too_broad with Clarify eligible, confirming the assessor cannot distinguish semantic coherence from scatter using active-unknown count alone. No production behaviour changed. Status closed.

Experiment 47 created a test-only diagnostic helper (inspectSharedUnknownAnchor) that inspects existing graph relationship fields to distinguish shared-anchor investigations from scattered ones. Three controlled fixtures (shared/separate/none anchors, all with identical structural counts) confirmed the helper correctly distinguishes all three patterns. Inspecting three real scenarios from Experiments 39-46 returned insufficient_data for all — existing data lacks populated relationship fields on unknown nodes. The assessor remains unchanged. Status pending Rob's review.

Experiment 48 audited whether real graph updates populate usable unknown relationships. Three production paths inspected: buildInitialGraph (does NOT populate dependsOn/affects/parentId), emergent reasoning via buildEmergentReasoningUnknown (DOES populate dependsOn and parentId), decomposition children (DOES populate parentId). One test file created (16 tests, all pass). Conclusion: Insufficient Data — shared-anchor detection works through the emergent-unknown path only. Status closed.

Experiment 49 tested whether any sequence of real production updates creates two or more active unknowns referencing the same populated relationship anchor. Results: no shared anchor found in production update sequences (both Cases A and B returned separate_anchors or insufficient_data). Structural capability exists but triggering logic never produces coexisting anchors. Status closed.

Experiment 50 tested whether shared edge topology from buildInitialGraph provides a usable coherence signal. Coherent and scattered inputs both produce identical edge topology — every unknown connects to the same summary node (kind=state) via depends_on edges, regardless of semantics. Initial shared edges are generic structural wiring, not coherence evidence. Closed (pending Rob's review).

Experiment 51 tested whether decision-relative relevance distinguishes coherent from scattered unknowns better than graph topology does. Within its training vocabulary, the classifier classified all four coherent unknowns as relevant and three of four scattered unknowns as irrelevant — but one scattered question was incorrectly flagged due to identical phrasing. Outside its vocabulary (different domain or paraphrased language), the classifier could not generalise: all four coherent unknowns received cannot_determine. The decision target never provided semantic context, only a binary action-keyword gate. No production code changed; no active engine behaviour changed; 70 tests pass (45 new + 25 Exp 21 regression). Status pending Rob's review.

Experiment 52 tested whether a small semantic interpretation step can judge decision relevance more reliably than keyword matching across paraphrases and domains. The semantic contract was implemented in tests/graph/decision-relevance-semantic.test.js. Live model comparison could not be completed because Ollama is not running on this machine — the test infrastructure uses the same /api/chat + format:json pattern as production. The deterministic keyword baseline continues to fail on paraphrases and new domains (confirmed via 15 passing guardrail tests). No semantic logic entered the active engine. The four-category decision-relevance contract remained unchanged. Branch: feature/user-workspace-ux-v0.7. First file to inspect: tests/graph/decision-relevance-semantic.test.js for the full experiment and results.

Experiment 52H held domain constant (market-entry / customer demand) and varied ambiguous wording across five cases. Four phrasings were strengthened beyond their supplied meaning; only "connected to" preserved cannot_determine. The model appeared more consistent about strengthening incomplete meaning than about which stronger category it selected. Experiment 52I then tested one grounding rule rather than keyword patches: three of four ambiguous cases preserved cannot_determine under grounding without harming clear classifications, but "important to" remained strengthened — the model could classify correctly while still commenting on relationship strength. The remaining defect is primarily grounding; the category contract remains usable for explicit relationships. Same host and model retained; no production behaviour changed. Status pending Rob's review.

Experiment 52A recovered the semantic test infrastructure by correcting its configuration resolution. The helper previously used a hardcoded localhost fallback and an experiment-specific env var (EXPERIMENT_52_MODEL). Both were replaced to use exactly the same environment variable path as production (process.env.OLLAMA_BASE_URL / process.env.OLLAMA_MODEL) sourced from .env.local. Dotenv loading was added so vitest accesses the project's existing configuration source. Ollama at 192.168.1.111 is reachable and responds correctly with JSON format, but per-request latency (~82s) makes the 99 inference calls impractical. Configuration path verified correct; execution requires a faster inference host. No production code changed (0 lines in provider, config, analysis, orchestrator). Branch: feature/user-workspace-ux-v0.7. First file to inspect: tests/graph/decision-relevance-semantic.test.js lines 80–85 (helper).

Experiment 52C separated free-language semantic understanding from enum normalisation into two independent calls per case across five decision/question pairs. Meaning mode captured all five intended relationships correctly (5/5). Enum classification matched expected categories on four of five cases (4/5). One meaning-correct / enum-mismatch case: Case 2 (European regulatory compliance) was correctly described as supporting in both modes but classified as could_change_decision rather than supports_decision. Same Qwen model (qwen-claude:latest) and host were retained; no production behaviour changed. What remains uncertain: whether the meaning-enum gap generalises across decision domains, stability over repeated runs, and whether normalisation mechanisms can bridge the gap without altering interpretation. Branch: feature/user-workspace-ux-v0.7. First file to inspect: tests/graph/decision-relevance-semantic-normalisation.test.js for results.

Experiment 52D isolated enum normalisation from semantic understanding: five fixed meaning statements (no decision target or question in the input) were mapped to the existing four-category contract via one live model call each. Four of five normalised to the expected enum. The compliance boundary case persisted — the model classified a "supports" relationship as could_change_decision, exposing genuine ambiguity between these two categories under the current definitions. The existing contract appears clear enough for a separate normalisation step; the remaining problem lies in category definitions, not semantic understanding or normalisation mechanism. Same Qwen model (qwen-claude:latest) and host (http://192.168.1.111:11434) were retained throughout. No production behaviour changed. What remains uncertain: whether the supports_decision ↔ could_change_decision boundary can be clarified without restructuring the contract, and whether the discrepancy holds under repeated runs. Branch: feature/user-workspace-ux-v0.7. First file to inspect: tests/graph/decision-relevance-normalisation.test.js for results.

Experiment 54H tested whether trustworthy source identity can begin deterministically from raw user input before any LLM interpretation occurs. A test-only helper createSourceRecord(rawInput) hashes the verbatim text with SHA-256 to produce a stable sourceId, preserves verbatimText unchanged, and sets sourceType: "user_input". Nine focused tests confirm identical inputs produce identical IDs (Case 1 = Case 4), paraphrases produce different IDs (Case 1 ≠ Case 2), and multi-sentence input survives intact (Case 3). No semantic interpretation, summarisation, or LLM call occurs. Trustworthy source identity is feasible before reconstruction — the remaining gap is claim/node provenance and graph linkage, not source identity. Deterministic code can assign stable identity to raw material at the application boundary without any reasoning contract. Branch: feature/user-workspace-ux-v0.7. First file to inspect: tests/reconstruction/deterministic-source-record.test.js.

Experiment 53 proved semantic separation of supplied meaning from possible inference is achievable. Experiment 54A confirmed the SituationGraph cannot recover provenance from graph state alone. Experiment 54B traced supplied-versus-inferred distinction upstream to evidenceRecordSchema but found it lost at buildInitialGraph because the node schema has no provenance field. Experiment 54C inspected the normal answer-update boundary: whole-input origin is explicit (answer = user supplied; proposal = model produced) but per-node provenance inside the proposal is not deterministically recoverable from the validated proposal alone. Experiment 54D audited the production update prompt: it clearly separates the user answer (## User Answer section) and instructions, so prompt-level source identity is explicit; however the proposed output schema has no provenance fields on nodes or edges, so per-node provenance at output level is absent — the tested prompt already preserves user-source identity clearly; the blocking gap identified here is that the validated proposal does not carry per-node provenance forward. The eventual representation remains undecided. Experiment 54E audited whether existing evidence IDs and evidence records could preserve provenance referentially without a new node field: the evidence-record schema contains vocabulary capable of distinguishing supplied-like from inferred-like material, but the reference chain breaks because (1) evidence records are consumed during startCase and never returned alongside graph state — no persistence layer retains them; and (2) no evidence records are created or retained during update cycles. Experiment 54E did not validate how those values are assigned in production. Experiment 54F audited evidenceType assignment: the reconstruction prompt instructs the LLM to classify each evidence item into one of five types based on its own judgment; no production code deterministically derives evidenceType from source origin — even reported_statement means "the model thinks this looks like a reported statement" not "production code knows this came directly from the user." Experiment 54G audited whether evidence records nevertheless retain deterministic linkage to user words: neither verbatim text nor structured location references (character offsets, turn IDs) survive in any record field; source and attribution are free-form model-generated strings that may be null; the raw user statement is available to production code while reconstruction is being performed but is not retained alongside the returned reconstruction/evidence state for later deterministic verification. Evidence records do not contain verbatim source text or deterministic source locations; evidenceType is model classification, not trustworthy provenance. Current evidence records therefore cannot independently prove source provenance.

Experiment 54I showed multiple interpretations can share one deterministic source lineage via the Experiment 54H SHA-256 method. Both branches stayed traceable to the same source while remaining distinct in their reported additions. No interpretation was selected as better and no numeric scoring occurred. Branch: feature/user-workspace-ux-v0.7. First file to inspect: tests/reconstruction/source-interpretation-lineage.test.js.

Experiment 54J proved the representation can separate source-supported from interpretation-added meaning using human-fixed references (13 tests, all pass). Grounding references were human-fixed; automated grounding remained untested. No production code or schemas changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect: tests/reconstruction/interpretation-source-grounding.test.js.

Experiment 54K tested whether the configured semantic model (qwen-claude:latest on 192.168.1.111:11434) can perform that grounding automatically. Three live Ollama calls (total ~96s): Case 1 (strengthening detection) = grounding_correct, Case 2 (multi-addition interpretation) = partial_grounding (missed one addition), Case 3 (faithful restatement control) = grounding_correct. Interpretation-added meaning did NOT leak into source-supported meaning in any case. One source-supported content gap: model missed "alternative causes" on the added side of Case 2. Automated semantic grounding is promising but imperfect — directionally viable but needs refinement before production use. Winner selection and downstream questions remain untested. No production code changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: tests/reconstruction/semantic-interpretation-grounding.test.js.

Experiment 54L repeated two identical grounding cases three times each to test stability across six live calls. The source-versus-added boundary was perfectly stable (zero leakage in all runs). Detection completeness appeared variable but manual analysis showed the instability came from the automated evaluator's paraphrase sensitivity, not the model itself. Case A strengthening identified in all 3 runs; Case B "other causes" and "not established as main problem" each identified in all 3 runs. Status pending Rob's review.

Experiment 54M tested whether two interpretations of one source can expose their substantive disagreement without deciding which is correct. Three live Ollama calls across three cases: real pricing attribution difference, paraphrase identity control, and competing causal explanations. All three classified as disagreement_correct by human semantic review. Paraphrase was correctly treated as agreement; shared meaning stayed separate; no invented disagreement or winner selection occurred. Status pending Rob's review.

5. What Remains Open

• The too_broad boundary sits exactly between three and four active unknowns; it is mechanically clear but conceptually uncertain — whether it aligns with genuine user confusion requires real-scenario validation;
• Health defaults to cannot_determine rather than healthy for 2–3 unknowns (no active question present); whether this is a bug or feature needs review;
• Whether the too_broad threshold needs widening so Clarify fires in more typical investigations;
• Whether user_overloaded health should be producible by the assessor for stalled/inconsistent evidence states;
• Existing-scenario graphs lack populated relationship fields on unknown nodes from the initial-build path; coherence detection works through the emergent-unknown path only (Populates dependsOn and parentId correctly — but requires comparable observations to trigger);

When This Knowledge-Management Phase Is Complete

Provisional criteria for review (all confirmed met by Experiment 38 cold-start test):

1. A fresh session can resume from the handoff and one context pack; — met
2. Current state has been verified against implementation; — met
3. Historical material is outside default loading; — met
4. Current principles are separated from aspirational architecture; — met
5. Task-specific routing works for engine and UI tasks; — met
6. A cross-boundary task has been tested; — met (Experiment 37)
7. Maintaining the handoff does not require reading the full history. — met

Knowledge-management structure is ready for Rob's review before engine experiments resume.

6. How to Resume

1. Read docs/current-handoff.md.
2. Read docs/current-project-state.md.
3. Choose one pack from docs/task-context-packs.md.
4. Read .claude/architecture-guardrails.md before any code change.
5. Load extra context only for a named gap — record why.
6. Check Git status before continuing.

7. First Files by Work Type

Work type	Start with
Engine experiment	Engine Experiment pack
UI or mock work	UI and Mock pack
Architecture or contract review	Architecture or Contract pack
Knowledge management	Knowledge-Management pack

8. Resume Check

Answer before continuing:

1. What work is currently active?
2. What work is paused?
3. What was the latest completed experiment?
4. Which context pack applies to the next task?
5. Is there any uncommitted work?

---

Created by Experiment 34. Updated by Experiments 38–53, 54A–54M. Branch: feature/user-workspace-ux-v0.7.

Return-to-Work Note (Experiment 54M)

Experiment 54L was corrected: model grounding was materially stable across six repeated identical inputs; evaluator wording sensitivity caused the apparent classification instability. Experiment 54M tested whether two interpretations of one source can expose their substantive disagreement without deciding which is correct. Three live calls across three cases: real pricing attribution difference, paraphrase identity control, and competing causal explanations. All three classified as disagreement_correct by human semantic review. Paraphrase was correctly treated as agreement; shared meaning stayed separate; no invented disagreement or winner selection occurred. Same host/model retained; no production behaviour changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: tests/reconstruction/semantic-interpretation-disagreement.test.js.