experiment: audit evidence type provenance

docs/design-evolution-log.md

@@ -5489,7 +5489,7 @@ Existing `evidenceIds` and evidence records may already provide enough referenti
 Each reconstruction evidence record (v0.2 schema) has:
 - **Stable ID**: `id: z.string().min(1)` — deterministic identifier present in every record.
 - **evidenceType**: `z.enum(["direct_observation", "reported_statement", "interpretation", "assumption", "inferred_relationship"])` — five distinct values.
-- **Sufficient information to distinguish supplied from inferred**: Yes. `reported_statement` identifies user-supplied material; `direct_observation` identifies observed facts; `interpretation`, `assumption`, and `inferred_relationship` identify model-inferred or unverified material. The type field is explicitly designed for this purpose.
+- **Sufficient information to distinguish supplied from inferred**: The evidence-record schema contains vocabulary capable of distinguishing supplied-like from inferred-like material, but Experiment 54E did not validate how those values are assigned in production.
 
 Answer: **Yes, evidence records carry sufficient identity.**
 
@@ -5551,7 +5551,7 @@ Answer: **Incomplete by design** — Initial graph supplies evidenceRefs only fo
 
 ### Conclusion
 
-**Existing provenance exists but referential linkage is incomplete** — specifically: evidence records carry sufficient identity (id + evidenceType distinguishing supplied from inferred) when they exist, but the provenance chain breaks because (1) evidence records are consumed during startCase and never returned alongside graph state, making them unrecoverable in persisted case data; and (2) no evidence records are created or retained during update cycles at all. The problem is primarily missing persistence of existing provenance combined with a gap in evidence record creation during updates.
+**Existing provenance exists but referential linkage is incomplete** — specifically: the evidence-record schema contains vocabulary capable of distinguishing supplied-like from inferred-like material (Experiment 54E did not validate how those values are assigned), and the reference chain breaks because (1) evidence records are consumed during startCase and never returned alongside graph state, making them unrecoverable in persisted case data; and (2) no evidence records are created or retained during update cycles at all. The problem is primarily missing persistence of existing provenance combined with a gap in evidence record creation during updates.
 
 ### Limitations
 
@@ -5563,3 +5563,119 @@ Answer: **Incomplete by design** — Initial graph supplies evidenceRefs only fo
 ### Status
 
 **Pending Rob's review.** Source-inspection complete. No production code changed. Working tree clean before commit.
+
+## Experiment 54F — Is evidenceType Actually Reliable Provenance, or Just a Model Label? (2026-08-07)
+
+### Objective
+
+Inspect whether the current reconstruction path assigns `evidenceType` from a clear provenance rule, or whether the LLM itself decides whether something is a reported statement, interpretation, assumption, or inferred relationship. Source-audit only. Do not implement provenance.
+
+### Hypothesis
+
+`evidenceType` may be semantically useful without being trustworthy provenance. If the LLM is asked to classify reconstructed content into those categories, then `reported_statement` may mean "the model thinks this looks like a reported statement" rather than "production code knows this came directly from the user."
+
+### Files Inspected
+
+- `prompts/reconstruct-v0.3.md` — full file; CRITICAL RULE 7 at line 155 specifying how evidenceType should be classified
+- `lib/reconstruction/prompt.js` — `buildV3Prompt` function (lines 67–78), which loads prompts/reconstruct-v0.3.md and substitutes the scenario via `{{SCENARIO}}`
+- `lib/reconstruction/schema.js` — `evidenceRecordSchema` definition at lines 113–127; evidence field at line 185
+- `lib/reconstruction/compatibility.js` — `normaliseAnalysisResponse` function (full file, lines 1–52); deterministic normalisation applied to parsed reconstruction results
+
+### Who Chooses evidenceType?
+
+Answer: **the LLM**.
+
+The prompt in `prompts/reconstruct-v0.3.md` line 155 states:
+
+> **evidenceType**: classify each evidence item clearly as either a direct observation, a reported statement, an interpretation, an assumption, or an inferred relationship. Do not treat raw counts as proof of causal relationships — they may be inferred relationships only when supported by explicit reasoning about denominators or rates.
+
+This is a classification instruction directed at the model. No production code determines or constrains which evidenceType value a given piece of evidence should have. The LLM receives scenario text and chooses an `evidenceType` for each evidence record it creates in its output.
+
+### Relevant Reconstruction Instruction (Summarised)
+
+The prompt instructs the model to:
+- Produce a JSON object with `inputClassification`, `reconstruction`, `evidence`, and `nextQuestion` keys;
+- For the `evidence` array, produce records each containing `id`, `description`, `evidenceType`, `source`, `attribution`, `confidence`, and `importance`;
+- **For evidenceType specifically**, instructs the model to "classify each evidence item clearly as either a direct observation, a reported statement, an interpretation, an assumption, or an inferred relationship" based on its judgment of what each piece of evidence represents;
+- No production-level rules restrict which category maps to which source origin.
+
+### Raw User Origin Status Before Reconstruction
+
+Answer: **model_classified**.
+
+The raw user scenario text is passed into the prompt via `{{SCENARIO}}` substitution at line 158 of reconstruct-v0.3.md. It is not wrapped, tagged, or structurally separated from any other content. Production code (lib/reconstruction/prompt.js) performs only a single string substitution: `content.replace("{{SCENARIO}}", scenario)`. There is no structural marker in the prompt that tells the model "this text came from the user" — it simply appears at the end of the prompt as unlabelled text.
+
+Note on Experiment 54D's finding: The update prompt (`lib/graph/prompt-builder.js`) does separate user-supplied answers via a Markdown header (`## User Answer`). However, **Experiment 54F inspects the initial reconstruction path only**, where no such structural separation exists. The update path uses a different prompt from a different builder and is not part of this audit's scope.
+
+### Raw User Origin Status Inside Reconstruction Output
+
+Answer: **model_classified**.
+
+After the LLM produces its evidence records, each record's `evidenceType` reflects the model's own judgment about what category that evidence belongs to — not any deterministic derivation from source origin. The only production-side manipulation of `evidenceType` is in `lib/reconstruction/compatibility.js`, which performs a single enum normalisation: if `evidenceType === "reported_claim"` (from an earlier schema version), it converts it to `"reported_statement"`. This does not add provenance information; it only adjusts for schema versioning.
+
+### Is evidenceType Deterministic?
+
+Answer: **no**. The model itself chooses each `evidenceType` value at generation time. No production code determines it from source structure.
+
+### Is reported_statement Trustworthy as User Provenance?
+
+Answer: **no**. A record with `evidenceType === "reported_statement"` means "the model classified this evidence item as a reported statement" — not "production code knows this came directly from the user." The model makes this classification based on its understanding of the scenario text, which may conflate what it inferred about reported claims with what was actually stated by a person.
+
+### Are interpretation / assumption / inferred_relationship Trustworthy as Model Provenance?
+
+Answer: **partially**. These values do indicate categories the model itself assigned to its own output, so they can serve as semantic labels for "model-generated content" in a loose sense. However, the boundary between these categories is defined by the LLM's judgment, not by production code — the model may classify something as an assumption when it was actually directly stated by the user, or vice versa. There is no deterministic gate separating user-supplied from model-generated content at classification time.
+
+### Trace: Raw User Statement → Reconstruction Prompt
+
+| Stage | User origin explicit? | Who assigns evidenceType? | Provenance strengthens/weakened? |
+|-------|----------------------|--------------------------|--------------------------------|
+| Raw user scenario text | Not structurally marked (just pasted as `{{SCENARIO}}`) | N/A | — |
+| Reconstruction prompt (reconstruct-v0.3.md) | Scenario appears unlabelled at end of file, no structural distinction from system instructions | Model chooses per its judgment (CRITICAL RULE 7) | Weakened — user text is indistinguishable in structure from other prompt content |
+
+### Trace: Reconstruction Prompt → LLM Evidence Record
+
+| Stage | User origin explicit? | Who assigns evidenceType? | Provenance strengthens/weakened? |
+|-------|----------------------|--------------------------|--------------------------------|
+| Model processes prompt and generates evidence records | No structural marker in prompt distinguishes user content | Model (based on CRITICAL RULE 7 classification instruction) | Unchanged — model judgment, not deterministic derivation |
+
+### Trace: LLM Evidence Record → Schema Validation
+
+| Stage | User origin explicit? | Who assigns evidenceType? | Provenance strengthens/weakened? |
+|-------|----------------------|--------------------------|--------------------------------|
+| Raw model output (JSON) | N/A — already model-generated | Model's own choice | Unchanged |
+| Compatibility normalisation (`lib/reconstruction/compatibility.js`) | No provenance added | `reported_claim` → `reported_statement` (schema versioning only) | Neutral — no provenance information added or removed; only enum compatibility |
+| Zod validation against `reconstructionV2Schema` | N/A — schema validates structure, not origin | Schema accepts whatever `evidenceType` the model chose (enum valid) | Unchanged — schema enforces enum validity but not provenance correctness |
+
+### Does Schema Validation Verify Origin or Only Enum Validity?
+
+Answer: **only enum validity**. The Zod schema (`lib/reconstruction/schema.js` line 113-127) validates that `evidenceType` is one of five allowed strings. It does not and cannot verify what produced the value — whether it came from a user, was derived deterministically by production code, or was classified by the LLM.
+
+### Can Downstream Deterministic Code Safely Treat evidenceType as Provenance?
+
+Answer: **no**. `evidenceType` values are model-generated labels reflecting semantic categories the model chose during reconstruction. They do not correspond to any deterministic derivation from source origin. Production code cannot safely interpret `evidenceType === "reported_statement"` as "this was definitely supplied by the user" without additional provenance infrastructure.
+
+### Does Experiment 54E's Referential Approach Depend on a Label That Is Itself Model-Generated?
+
+Answer: **yes**. Experiment 54E found that evidence records carry sufficient identity (id + evidenceType) when they exist. But `evidenceType` is itself model-generated, not structurally derived from source origin. The referential approach depends on labels the LLM chose, meaning provenance inference at that point already rests on model classification rather than deterministic provenance — compounding the provenance gap rather than resolving it.
+
+### Primary Provenance Gap After This Audit
+
+Answer: **both**. The provenance gap has two independent causes:
+1. **Evidence-record lifetime** (54E): Records are consumed during startCase and never returned with graph state.
+2. **Evidence-type trustworthiness** (54F): Even if records were retained, `evidenceType` is model classification, not deterministic provenance.
+
+### Experiment Conclusion
+
+**evidenceType is semantic model output, not reliable provenance.** The reconstruction prompt instructs the LLM to classify each evidence item into one of five categories based on its own judgment. No production code deterministically derives `evidenceType` from source origin. The compatibility layer adds only a single enum normalisation (`reported_claim` → `reported_statement`) for schema versioning. Schema validation enforces enum validity but not provenance correctness. Experiment 54E's referential approach depends on a label that is itself model-generated, meaning the provenance chain was already unreliable before it broke due to missing persistence. The primary gap has two independent causes: evidence records are not retained alongside graph state, and `evidenceType` values themselves are model-classified rather than deterministically derived from source origin.
+
+### Limitations
+
+- Source-inspection audit only; no live execution tested
+- Inspected only the initial reconstruction path (v0.3 prompt), not the update path which uses a different prompt structure
+- Did not evaluate whether `source` or `attribution` fields on evidence records carry any provenance value beyond what `evidenceType` does
+- Did not test how reliable the LLM's evidenceType classification is in production (this would require validation, not inspection)
+- Conclusions apply to the v0.3 reconstruction prompt as currently implemented
+
+### Status
+
+**Pending Rob's review.** Source-inspection complete. No production code changed. Working tree clean before commit.