experiment: audit referential provenance

docs/current-handoff.md

@@ -73,7 +73,7 @@ Experiment 52C separated free-language semantic understanding from enum normalis
 
 Experiment 52D isolated enum normalisation from semantic understanding: five fixed meaning statements (no decision target or question in the input) were mapped to the existing four-category contract via one live model call each. Four of five normalised to the expected enum. The compliance boundary case persisted — the model classified a "supports" relationship as `could_change_decision`, exposing genuine ambiguity between these two categories under the current definitions. The existing contract appears clear enough for a separate normalisation step; the remaining problem lies in category definitions, not semantic understanding or normalisation mechanism. Same Qwen model (`qwen-claude:latest`) and host (`http://192.168.1.111:11434`) were retained throughout. No production behaviour changed. What remains uncertain: whether the `supports_decision` ↔ `could_change_decision` boundary can be clarified without restructuring the contract, and whether the discrepancy holds under repeated runs. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-normalisation.test.js` for results.
 
-Experiment 53 proved semantic separation of supplied meaning from possible inference is achievable. Experiment 54A confirmed the SituationGraph cannot recover provenance from graph state alone. Experiment 54B traced supplied-versus-inferred distinction upstream to evidenceRecordSchema but found it lost at buildInitialGraph because the node schema has no provenance field. Experiment 54C inspected the normal answer-update boundary: whole-input origin is explicit (answer = user supplied; proposal = model produced) but per-node provenance inside the proposal is not deterministically recoverable from the validated proposal alone. Experiment 54D audited the production update prompt: it clearly separates the user answer (## User Answer section) and instructions, so prompt-level source identity is explicit; however the proposed output schema has no provenance fields on nodes or edges, so per-node provenance at output level is absent — the gap is a schema deficiency, not a prompt-design problem.
+Experiment 53 proved semantic separation of supplied meaning from possible inference is achievable. Experiment 54A confirmed the SituationGraph cannot recover provenance from graph state alone. Experiment 54B traced supplied-versus-inferred distinction upstream to evidenceRecordSchema but found it lost at buildInitialGraph because the node schema has no provenance field. Experiment 54C inspected the normal answer-update boundary: whole-input origin is explicit (answer = user supplied; proposal = model produced) but per-node provenance inside the proposal is not deterministically recoverable from the validated proposal alone. Experiment 54D audited the production update prompt: it clearly separates the user answer (## User Answer section) and instructions, so prompt-level source identity is explicit; however the proposed output schema has no provenance fields on nodes or edges, so per-node provenance at output level is absent — the tested prompt already preserves user-source identity clearly; the blocking gap identified here is that the validated proposal does not carry per-node provenance forward. The eventual representation remains undecided. Experiment 54E audited whether existing evidence IDs and evidence records could preserve provenance referentially without a new node field: evidence records carry sufficient identity (id + evidenceType) but are consumed during startCase and never returned alongside graph state, so the reference chain breaks — existing provenance exists but referential linkage is incomplete.
 
 ## 5. What Remains Open
 
@@ -127,8 +127,8 @@ Answer before continuing:
 
 ---
 
-*Created by Experiment 34. Updated by Experiments 38–53, 54A–54D. Branch: `feature/user-workspace-ux-v0.7`.*
+*Created by Experiment 34. Updated by Experiments 38–53, 54A–54E. Branch: `feature/user-workspace-ux-v0.7`.*
 
-### Return-to-Work Note (Experiment 53 → 54A → 54B → 54C → 54D)
+### Return-to-Work Note (Experiment 53 → 54A → 54B → 54C → 54D → 54E)
 
-Experiment 54C clarified that whole-input origin remains deterministically knowable before application (answer = user supplied, proposal = model produced) but per-node provenance is lost because the validated proposal schema carries no per-node origin markers. Experiment 54D audited the production update prompt in lib/graph/prompt-builder.js and confirmed the user answer is explicitly identifiable to the model via a distinct named section (## User Answer). However, the requested proposal output schema has no provenance fields on nodes or edges, so supplied-versus-inferred origin cannot be preserved in the output. No production code changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: lib/graph/schema.js (graphUpdateSchema and situationNodeSchema) to evaluate minimal schema additions for per-node provenance fields.
+Experiment 53 separated supplied meaning from inference in semantic output. Experiments 54A–54D located where provenance disappears from durable graph representation, confirming the prompt preserves user-source identity but the proposal schema carries no per-node provenance. Experiment 54E tested whether existing evidence IDs and evidence records could preserve provenance referentially without assuming a new node field. The reference chain works only during initial reconstruction: evidence records carry sufficient identity (id + evidenceType) at that point, but they are consumed during startCase and never returned with graph state — no persistence layer retains them. No production code changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: lib/graph/orchestrator.js to evaluate where evidence records should be returned alongside case state so the referential chain does not break.

docs/design-evolution-log.md

@@ -5437,7 +5437,7 @@ However, the requested output schema (graphUpdateSchema in lib/graph/schema.js)
 This means:
 - Prompt-level source identity: explicit
 - Proposal-level provenance: absent (structural limitation of schema)
-- The gap is not a prompt-design problem; it is a schema-deficiency problem
+- The tested prompt already preserves user-source identity clearly; the blocking gap identified here is that the validated proposal does not carry per-node provenance forward. The eventual representation remains undecided.
 
 ### Limitations
 
@@ -5463,3 +5463,103 @@ This means:
 
 No test run required; Experiment 54D is a prompt-source audit.
 
+
+## Experiment 54E — Can Existing Evidence References Preserve Provenance Without Adding a Node Field? (2026-08-07)
+
+### Objective
+
+Inspect whether the current graph design already preserves referential provenance through existing evidence identities and references, without requiring a new node field. This is a source-inspection experiment only. Do not implement provenance.
+
+### Hypothesis
+
+Existing `evidenceIds` and evidence records may already provide enough referential structure to preserve provenance if: (1) evidence records survive beyond reconstruction; (2) their IDs remain resolvable later; (3) `evidenceType` remains attached to those records; (4) graph nodes can reliably link to the evidence records that justify them.
+
+### Files Inspected
+
+- `lib/reconstruction/schema.js` — evidenceRecordSchema definition, reconstructionV2Schema evidence field
+- `lib/graph/schema.js` — situationNodeSchema evidenceIds field, graphUpdateSchema structure
+- `lib/graph/builder.js` — how evidence IDs are assigned during initial graph build (buildInitialGraph)
+- `lib/graph/orchestrator.js` — startCase and updateCaseWithDependencies return values; evidence flow through the pipeline
+- `lib/reconstruction/schema.js` lines 113–127 — evidenceRecordSchema fields
+- `lib/graph/apply-proposal.js` — how evidenceIds are populated during graph updates (appendUniqueValue pattern)
+- `lib/graph/prompt-builder.js` — rule 14 "Do not invent evidence"
+
+### Evidence Record Identity
+
+Each reconstruction evidence record (v0.2 schema) has:
+- **Stable ID**: `id: z.string().min(1)` — deterministic identifier present in every record.
+- **evidenceType**: `z.enum(["direct_observation", "reported_statement", "interpretation", "assumption", "inferred_relationship"])` — five distinct values.
+- **Sufficient information to distinguish supplied from inferred**: Yes. `reported_statement` identifies user-supplied material; `direct_observation` identifies observed facts; `interpretation`, `assumption`, and `inferred_relationship` identify model-inferred or unverified material. The type field is explicitly designed for this purpose.
+
+Answer: **Yes, evidence records carry sufficient identity.**
+
+### Graph Reference Behaviour
+
+When a graph node contains an `evidenceId`:
+- During `buildInitialGraph` (builder.js:62–70), evidence records from the analysis output are placed in an `evidenceMap`. Nodes are created with `evidenceIds` populated from actual evidence record IDs (builder.js:100: `node.evidenceIds.push(obs.id)`).
+- These IDs refer to real evidence records during construction. However, after graph construction, the graph contains only string IDs — they would resolve if a lookup table existed, but they are opaque strings within the node object itself.
+
+Answer: **IDs refer to real evidence records at build time; become opaque strings in the graph post-construction.**
+
+### Evidence-Record Lifetime
+
+The critical flow is:
+1. `analyseScenario` (lib/analysis.js) produces evidence records via reconstructionV2Schema, returned as `data.evidence`.
+2. In `startCase` (orchestrator.js:378), `analysis.evidence` is passed to `buildInitialGraph` solely to populate node `evidenceIds`.
+3. **The evidence records themselves are NOT included in `startCase` return value.** The return at orchestrator.js:489-566 includes only `situationGraph`, `selectedQuestion`, `diagnostics`, and `assessment`. No `evidence` field exists in the return object.
+4. In `updateCaseWithDependencies`, no new evidence records are created anywhere in the pipeline. Rule 14 of the update prompt ("Do not invent evidence") explicitly forbids the model from creating them. The prompt-builder.js shows no mechanism for evidence generation during updates.
+5. There is **no persistence layer** in this codebase that stores case state to disk or a database. The API routes (cases/start/route.js, cases/update/route.js) return data to the client; they do not persist anything.
+
+The initial evidence records exist only during the startCase execution lifetime. They are consumed to populate graph node evidenceIds but never returned alongside the case state. Subsequent update cycles produce no evidence records at all.
+
+Answer: **not_retained_with_graph**
+
+### Provenance Through Reference
+
+If a later consumer receives normal persisted case state (which contains only `situationGraph` with nodes having string `evidenceIds`), it cannot:
+1. Read a graph node — yes, the node and its evidenceIds are present.
+2. Follow its evidenceIds — no matching evidence records exist anywhere in the persisted state.
+3. Resolve each ID to an evidence record — impossible; records do not exist.
+4. Inspect evidenceType — N/A; no records to inspect.
+
+Answer: **no** — The reference chain breaks at step 2/3 because evidence records are not part of the returned case state.
+
+### Node-to-Evidence Completeness
+
+- **Initial graph nodes (from buildInitialGraph)**: Nodes from observedStates do receive evidence references (builder.js:100). Other node types (actors, systemsOrObjects, differences, contradictions, unknowns, interpretations) are created WITHOUT evidence references — only observedStates nodes get `evidenceIds` populated.
+- **Nodes added during update**: No new evidence records are ever created during the update flow. Nodes created via `applyValidatedProposal` may have their `evidenceIds` field set (the schema allows it), but no source code in the update path populates them from actual evidence records. There is no mechanism to create or assign evidence record IDs during updates.
+
+Answer: **Incomplete by design** — Initial graph supplies evidenceRefs only for observedStates; all other nodes get none. Update-phase nodes get no evidence references at all.
+
+### Referential Recoverability Trace
+
+| Step | Result | Reason |
+|------|--------|--------|
+| Graph node → evidenceId | works | Nodes carry evidenceIds as string arrays |
+| evidenceId → evidence record | breaks | Evidence records are consumed during startCase and never returned; no persistence layer retains them |
+| Evidence record → evidenceType | N/A | Chain already broken at previous step |
+
+### Answered Questions
+
+1. **Is node identity already sufficient?** No — nodes exist and have evidenceIds, but without the referenced records they carry no provenance information.
+2. **Is evidence identity already sufficient?** The evidence records themselves (when they exist) carry sufficient identity (id + evidenceType). But they are not available in case state.
+3. **Are evidence records retained long enough to resolve references?** No — consumed during construction, not returned with graph.
+4. **Is evidenceType available after resolution?** N/A — cannot resolve the reference to get there.
+5. **Are node-to-evidence links populated consistently enough?** No — only observedStates nodes in initial build receive references; all update-phase nodes receive none.
+6. **Can supplied-versus-inferred provenance currently be recovered referentially?** No.
+7. **Is the problem primarily:** More than one of these: (a) evidence provenance exists upstream during reconstruction but is not persisted alongside graph state; (b) no new evidence records are created or retained during update cycles; (c) incomplete linkage even at construction time (only observedStates nodes get references).
+
+### Conclusion
+
+**Existing provenance exists but referential linkage is incomplete** — specifically: evidence records carry sufficient identity (id + evidenceType distinguishing supplied from inferred) when they exist, but the provenance chain breaks because (1) evidence records are consumed during startCase and never returned alongside graph state, making them unrecoverable in persisted case data; and (2) no evidence records are created or retained during update cycles at all. The problem is primarily missing persistence of existing provenance combined with a gap in evidence record creation during updates.
+
+### Limitations
+
+- Source-inspection audit only; no live execution tested
+- Client-side state management was not inspected — if the client retains evidence records alongside graph state, referential recovery may work on that layer
+- Did not evaluate whether the client could reconstruct provenance from UI-visible data
+- Conclusions apply to the server-side pipeline as currently implemented
+
+### Status
+
+**Pending Rob's review.** Source-inspection complete. No production code changed. Working tree clean before commit.