experiment: audit evidence source linkage

docs/current-handoff.md

@@ -73,7 +73,7 @@ Experiment 52C separated free-language semantic understanding from enum normalis
 
 Experiment 52D isolated enum normalisation from semantic understanding: five fixed meaning statements (no decision target or question in the input) were mapped to the existing four-category contract via one live model call each. Four of five normalised to the expected enum. The compliance boundary case persisted — the model classified a "supports" relationship as `could_change_decision`, exposing genuine ambiguity between these two categories under the current definitions. The existing contract appears clear enough for a separate normalisation step; the remaining problem lies in category definitions, not semantic understanding or normalisation mechanism. Same Qwen model (`qwen-claude:latest`) and host (`http://192.168.1.111:11434`) were retained throughout. No production behaviour changed. What remains uncertain: whether the `supports_decision` ↔ `could_change_decision` boundary can be clarified without restructuring the contract, and whether the discrepancy holds under repeated runs. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-normalisation.test.js` for results.
 
-Experiment 53 proved semantic separation of supplied meaning from possible inference is achievable. Experiment 54A confirmed the SituationGraph cannot recover provenance from graph state alone. Experiment 54B traced supplied-versus-inferred distinction upstream to evidenceRecordSchema but found it lost at buildInitialGraph because the node schema has no provenance field. Experiment 54C inspected the normal answer-update boundary: whole-input origin is explicit (answer = user supplied; proposal = model produced) but per-node provenance inside the proposal is not deterministically recoverable from the validated proposal alone. Experiment 54D audited the production update prompt: it clearly separates the user answer (## User Answer section) and instructions, so prompt-level source identity is explicit; however the proposed output schema has no provenance fields on nodes or edges, so per-node provenance at output level is absent — the tested prompt already preserves user-source identity clearly; the blocking gap identified here is that the validated proposal does not carry per-node provenance forward. The eventual representation remains undecided. Experiment 54E audited whether existing evidence IDs and evidence records could preserve provenance referentially without a new node field: the evidence-record schema contains vocabulary capable of distinguishing supplied-like from inferred-like material, but the reference chain breaks because (1) evidence records are consumed during startCase and never returned alongside graph state — no persistence layer retains them; and (2) no evidence records are created or retained during update cycles. Experiment 54E did not validate how those values are assigned in production. Experiment 54F audited evidenceType assignment: the reconstruction prompt instructs the LLM to classify each evidence item into one of five types based on its own judgment; no production code deterministically derives evidenceType from source origin — even reported_statement means "the model thinks this looks like a reported statement" not "production code knows this came directly from the user."
+Experiment 53 proved semantic separation of supplied meaning from possible inference is achievable. Experiment 54A confirmed the SituationGraph cannot recover provenance from graph state alone. Experiment 54B traced supplied-versus-inferred distinction upstream to evidenceRecordSchema but found it lost at buildInitialGraph because the node schema has no provenance field. Experiment 54C inspected the normal answer-update boundary: whole-input origin is explicit (answer = user supplied; proposal = model produced) but per-node provenance inside the proposal is not deterministically recoverable from the validated proposal alone. Experiment 54D audited the production update prompt: it clearly separates the user answer (## User Answer section) and instructions, so prompt-level source identity is explicit; however the proposed output schema has no provenance fields on nodes or edges, so per-node provenance at output level is absent — the tested prompt already preserves user-source identity clearly; the blocking gap identified here is that the validated proposal does not carry per-node provenance forward. The eventual representation remains undecided. Experiment 54E audited whether existing evidence IDs and evidence records could preserve provenance referentially without a new node field: the evidence-record schema contains vocabulary capable of distinguishing supplied-like from inferred-like material, but the reference chain breaks because (1) evidence records are consumed during startCase and never returned alongside graph state — no persistence layer retains them; and (2) no evidence records are created or retained during update cycles. Experiment 54E did not validate how those values are assigned in production. Experiment 54F audited evidenceType assignment: the reconstruction prompt instructs the LLM to classify each evidence item into one of five types based on its own judgment; no production code deterministically derives evidenceType from source origin — even reported_statement means "the model thinks this looks like a reported statement" not "production code knows this came directly from the user." Experiment 54G audited whether evidence records nevertheless retain deterministic linkage to user words: neither verbatim text nor structured location references (character offsets, turn IDs) survive in any record field; `source` and `attribution` are free-form model-generated strings that may be null; raw user statement is not returned alongside validated results. Existing evidence records do not contain deterministic source provenance.
 
 ## 5. What Remains Open
 
@@ -127,8 +127,8 @@ Answer before continuing:
 
 ---
 
-*Created by Experiment 34. Updated by Experiments 38–53, 54A–54F. Branch: `feature/user-workspace-ux-v0.7`.*
+*Created by Experiment 34. Updated by Experiments 38–53, 54A–54G. Branch: `feature/user-workspace-ux-v0.7`.*
 
-### Return-to-Work Note (Experiment 53 → 54A → 54B → 54C → 54D → 54E)
+### Return-to-Work Note (Experiment 53 → 54A → 54B → 54C → 54D → 54E → 54F → 54G)
 
-Experiment 53 separated supplied meaning from inference in semantic output. Experiments 54A–54D located where provenance disappears from durable graph representation, confirming the prompt preserves user-source identity but the proposal schema carries no per-node provenance. Experiment 54E tested whether existing evidence IDs and evidence records could preserve provenance referentially without assuming a new node field. The reference chain works only during initial reconstruction: evidence records carry sufficient identity (id + evidenceType) at that point, but they are consumed during startCase and never returned with graph state — no persistence layer retains them. Experiment 54F audited how evidenceType values are actually assigned in production: the LLM classifies each evidence item into one of five categories based on its own judgment (CRITICAL RULE 7 in reconstruct-v0.3.md), not from deterministic derivation by production code. evidenceType is semantic model output, not reliable provenance — reported_statement means "the model thinks this looks like a reported statement" not "production code knows this came from the user." The provenance gap has two independent causes: (1) evidence records are not retained alongside graph state; and (2) evidenceType itself is model-generated, not structurally derived from source origin. No production code changed. Branch: feature/user-workspace-ux-v0.7.
+Experiment 53 separated supplied meaning from inference in semantic output. Experiments 54A–54D located where provenance disappears from durable graph representation, confirming the prompt preserves user-source identity but the proposal schema carries no per-node provenance. Experiment 54E tested whether existing evidence IDs and evidence records could preserve provenance referentially without assuming a new node field — the reference chain breaks because evidence records are consumed during startCase and never returned with graph state. Experiment 54F established that `evidenceType` is model classification, not trustworthy provenance: the LLM classifies each evidence item into five categories based on its own judgment, so reported_statement means "the model thinks this looks like a reported statement" not "production code knows this came from the user." Experiment 54G tested whether evidence records nevertheless retain deterministic linkage to the user's actual words. Neither verbatim text nor structured location references (character offsets, turn IDs, source record IDs) survive in any evidence record field — `source` and `attribution` are free-form model-generated strings that may be null. The raw user statement is not returned with validated results. Deterministic code cannot independently verify evidence records against user input without trusting `evidenceType`. No production code changed. Branch: feature/user-workspace-ux-v0.7. Commit hashes: 54F closure `455d6f4`, 54G `pending`. First file to inspect when resuming: `lib/reconstruction/schema.js` (evidenceRecordSchema) or `prompts/reconstruct-v0.3.md` (evidence output format).

docs/design-evolution-log.md

@@ -5679,3 +5679,115 @@ Answer: **both**. The provenance gap has two independent causes:
 ### Status
 
 **Pending Rob's review.** Source-inspection complete. No production code changed. Working tree clean before commit.
+
+## Experiment 54G — Do Evidence Records Contain Deterministic Source Linkage Back to User Words? (2026-08-07)
+
+### Objective
+
+Answer one narrow provenance question: does the current reconstruction output contain enough source information to deterministically prove that an evidence record came from the user's actual words, without trusting the LLM's `evidenceType` label? Source-audit only. Do not implement provenance.
+
+54F established that `evidenceType` is model classification, not reliable provenance. 54G tests whether evidence records nevertheless retain deterministic linkage to the user's actual words — such as exact text, character offsets, turn ID, source path, or other structural location data.
+
+### Hypothesis
+
+An evidence record may already preserve enough source material — such as exact text, quote, source excerpt, source ID, character range, source path, turn ID, or input reference — for deterministic code to verify that a record is directly grounded in the user's supplied text. If no such information exists, then evidence-record identity alone cannot establish user provenance.
+
+### Files Inspected
+
+- `lib/reconstruction/schema.js` — `evidenceRecordSchema` definition (lines 113–127); `reconstructionV2Schema` (line 182)
+- `prompts/reconstruct-v0.3.md` — evidence record output format (lines 126–136); CRITICAL RULE 7 (line 155)
+- `lib/reconstruction/compatibility.js` — `normaliseAnalysisResponse` function (full file); source null-handling (lines 31–38)
+- `tests/reconstruction/compatibility.test.js` — real evidence-record shapes from tests (lines 36–44, 54–72, 107–115, 137–147)
+- `lib/analysis.js` — `analyseScenario` function (full file); confirms raw scenario is NOT returned alongside evidence records (lines 173–188)
+
+### Evidence-Record Source-Related Fields
+
+From `evidenceRecordSchema`:
+
+| Field | Type | Provenance potential |
+|-------|------|---------------------|
+| `id` | `z.string().min(1)` | None — free-form, LLM-generated. No structural reference to input. |
+| `description` | `z.string().min(1)` | None — model-generated summary of the evidence, not user text. |
+| `evidenceType` | enum (5 values) | None — model classification, per 54F. Not structural provenance. |
+| `source` | `z.string().optional()` | Partially — prompt says "who/where this came from". But: it is free-form text generated by the LLM, not deterministic production code output. In tests, it appears as `"report"` (free label) or `null` (removed by normalisation). No character offsets, turn IDs, span data, or source record references are structured into it. |
+| `attribution` | `z.string().nullable().optional()` | None — test fixtures show `null` by default. When populated, it is free-form model text describing who said what, not a deterministic production-code identifier back to input. |
+| `confidence` | enum | None — subjective confidence level, not source data. |
+| `importance` | enum | None — semantic weight, not source data. |
+
+### Does Raw User Statement Coexist with Reconstruction Result?
+
+**No.** The raw user statement (`scenario`) is the input parameter to `analyseScenario`. It is used only for prompt building (line 59 of analysis.js). It is NOT included in the return value alongside `evidence` records or the validated reconstruction. At validation time, deterministic code has:
+- the evidence records (with model-generated descriptions and types);
+- but NOT the raw user statement itself.
+
+Even if deterministic code had the scenario text available at validation time, the evidence records still lack any field containing verbatim user text or structural location back to it.
+
+### Does Evidence Record Preserve Exact User Wording?
+
+**No.** The `description` field is a model-generated summary/paraphrase of the evidence — not the user's actual words. The `source` field, when non-null, is free-form text like `"report"` (a label, not a quote). No evidence record contains verbatim or near-verbatim user text.
+
+### Does Evidence Record Preserve Source Location / Span / Turn Identity?
+
+**No.** None of the evidence record fields contain:
+- character offsets within user input;
+- sentence or line index;
+- turn or message ID;
+- source record ID owned by production code;
+- any structural reference to a specific location in the original input.
+
+The `source` field prompt description says "who/where this came from" but it is free-form, inconsistently populated (sometimes null), and produced by the LLM not deterministic code.
+
+### Who Creates Evidence Record IDs?
+
+**The LLM.** The prompt template (line 128 of reconstruct-v0.3.md) says `"id": "<any unique string>"`. The schema only requires `z.string().min(1)`. No production code generates or constrains the ID beyond non-emptiness. The ID is opaque and carries no provenance semantics.
+
+### Can Evidence Records Be Verified Against Raw User Input?
+
+**No.** Deterministic verification would require:
+1. Raw user text available at validation time — NOT present in result;
+2. Each evidence record containing verbatim user text or a deterministic structural reference to a location within it — neither present.
+
+The `description` field is model-generated paraphrase, not verbatim text. The `source` field is free-form model output, not a structured pointer. No field survives from the raw input to the result in a form that code can verify.
+
+### Trace: Raw User Statement → Evidence Record → Source Verification
+
+| Stage | Raw source identity available? | Exact source wording/location? | Deterministic verification possible? |
+|-------|-------------------------------|-------------------------------|-------------------------------------|
+| Raw user statement | explicit (input parameter) | explicit (raw text) | N/A — this is the ground truth |
+| Reconstruction prompt | absent (scenario pasted as {{SCENARIO}} with no structural markers) | partial (present in prompt body but unlabelled and indistinguishable from system instructions) | No |
+| LLM evidence record | absent (all source identity lost to model generation) | absent (description is model paraphrase; source/attribution are free-form model text or null) | No |
+| Validated result (analysis.js return) | absent (scenario not returned alongside evidence) | absent | No |
+
+### Trace: Evidence Record → Source Verification
+
+| Step | Status |
+|------|--------|
+| Get evidence record fields | present |
+| Check description against user text | impossible — no verbatim text to compare |
+| Check source/attribution as structured location | impossible — free-form model output, not production-code identifiers |
+| Check id for provenance semantics | impossible — opaque LLM-generated string |
+| Verify evidenceType independently | impossible — no source text to verify against |
+
+### Does Current Reconstruction Schema Contain Enough Information for Deterministic Provenance?
+
+**No.** The schema fields `source` and `attribution` exist as free-form optional strings, but neither is deterministic (they are model-generated), nor do they contain structured location data. The raw user statement is not returned alongside the evidence records. Even if it were, no evidence record field contains verbatim text or structural reference to it.
+
+### Primary Source-Linkage Gap
+
+Evidence records carry no verbatim user text and no deterministic structural reference (character offsets, turn/message IDs, source record references) back to the original input. The only candidate fields (`source`, `attribution`) are free-form model-generated strings that may be null, and which cannot be independently verified against user input. Additionally, the raw user statement itself is not returned with the validated reconstruction result at validation time.
+
+### Experiment 54G Conclusion
+
+**Existing evidence records do not contain deterministic source provenance.** Neither verbatim user text nor structured location references (character offsets, turn IDs, source record references) survive into the evidence records. The `source` and `attribution` fields are free-form model-generated text that may be null — they cannot establish deterministic linkage to user-supplied words. The raw user statement is not returned alongside the validated reconstruction result. Therefore, deterministic code cannot verify that any evidence record was produced from the user's actual words without trusting the LLM's `evidenceType` label.
+
+### Limitations
+
+- Source-inspection audit only; no live execution tested
+- Inspected only the initial reconstruction path (v0.3 prompt) and the analysis pipeline
+- Did not inspect whether evidence records are stored with graph nodes in a way that could later be recovered
+- Did not inspect the update path which uses different prompts and potentially different provenance characteristics
+- Conclusions apply to the v0.3 reconstruction output as currently implemented
+
+### Status
+
+**Pending Rob's review.** Source-inspection complete. No production code changed. Working tree clean before commit.