From bb3da3d197b05a292a7d8c1e1785d04e42c59ee0 Mon Sep 17 00:00:00 2001 From: robbond Date: Thu, 3 Sep 2026 11:10:03 +0100 Subject: [PATCH] docs(confidence-engine): checkpoint design evolution archive tranche five --- .../experiments/vol-1-chapters/README.md | 18 + ...experiments-54a-to-54k-provenance-audit.md | 1443 +++++++++++++++++ 2 files changed, 1461 insertions(+) create mode 100644 docs/archive/experiments/vol-1-chapters/ch10/experiments-54a-to-54k-provenance-audit.md diff --git a/docs/archive/experiments/vol-1-chapters/README.md b/docs/archive/experiments/vol-1-chapters/README.md index dbfb768..e912c38 100644 --- a/docs/archive/experiments/vol-1-chapters/README.md +++ b/docs/archive/experiments/vol-1-chapters/README.md @@ -139,6 +139,24 @@ Experiments 52–52I and Correction to Experiment 52B Conclusion — semantic in Fidelity: Exact contiguous copy. +## Tranche 5 + +### Chapter 10 +Path: +docs/archive/experiments/vol-1-chapters/ch10/experiments-54a-to-54k-provenance-audit.md + +Original source: +docs/design-evolution-log.md lines 4981–6423 + +Contents: +Experiments 54A–54K — complete provenance audit chain including graph provenance verification, trace through pipeline, update flow determinism, prompt-level provenance analysis, evidence reference integrity, evidenceType reliability, deterministic source linkage, pre-LLM source identity establishment, multi-interpretation source anchoring, meaning separation capability, and automated semantic grounding with live model inference. + +Fidelity: +Exact contiguous copy. + +Tranches 1 through 5 have now been extracted. +The original monolithic log remains intact and authoritative while extraction is incomplete. + ## Refactor status Only tranches 1 and 2 have been extracted. diff --git a/docs/archive/experiments/vol-1-chapters/ch10/experiments-54a-to-54k-provenance-audit.md b/docs/archive/experiments/vol-1-chapters/ch10/experiments-54a-to-54k-provenance-audit.md new file mode 100644 index 0000000..5e43a3d --- /dev/null +++ b/docs/archive/experiments/vol-1-chapters/ch10/experiments-54a-to-54k-provenance-audit.md @@ -0,0 +1,1443 @@ +## Experiment 54A — Audit Existing Graph Provenance Only (2026-08-07) + +Experiment 53 showed that the semantic model can keep supplied meaning and possible inference separate in its output. The active graph compatibility question remained unknown. Experiment 54A was an inspection-only experiment to determine whether the current validated SituationGraph distinguishes information supplied by the user or evidence from information inferred by the model. + +### Hypothesis + +The current graph may distinguish known/provisional and supported/unsupported without actually recording where information came from. If true, current graph state can represent epistemic status but not reliably recover supplied-versus-inferred provenance. + +### Files Inspected + +- `docs/current-handoff.md` +- `lib/graph/schema.js` — SituationGraph and node schema definitions +- `lib/graph/builder.js` — production initial graph builder (how nodes are populated from reconstruction) +- `lib/graph/update-proposal.js` — LLM output parsing for graph updates +- `lib/reconstruction/schema.js` — evidenceRecordSchema, reconstructionV2Schema + +### Graph Vocabulary Relevant to Provenance + +**Existing relevant node kinds:** +- `observation`, `reported_claim`, `metric`, `state`, `transition`, `relationship`, `assumption`, `unknown`, `conclusion` + +**Existing relevant status fields:** +- `known`, `unknown`, `provisional`, `supported`, `weakened`, `contradicted`, `resolved` + +**Existing confidence fields:** +- `low`, `medium`, `high` + +**Existing evidence / relationship fields:** +- `evidenceIds`: array of strings (graph reference IDs from reconstruction) +- `dependsOn`: array of node IDs +- `affects`: array of node IDs +- `parentId`: nullable string +- `childIds`: array of node IDs +- Edge types: `supports`, `weakens`, `contradicts`, `depends_on`, `causes`, `may_cause`, `measures`, `compares_with`, `updates`, `other` + +### Explicit Supplied-Information Provenance Exists: No + +No field or combination of fields in the SituationNode schema has documented or implemented meaning that is "this content was supplied by the user or evidence source." The `kind` field distinguishes semantic categories (observation vs assumption vs unknown), not provenance. A node with `kind=assumption` describes what kind of claim it is, not who produced it. + +### Explicit Inferred-Information Provenance Exists: No + +No field or combination has documented or implemented meaning that is "this content was inferred or proposed by the model and is not established evidence." The LLM-inferred nodes flow through `proposal.addedNodes` into the graph with kinds determined by the LLM — but those kinds are semantic labels, not provenance markers. + +### Status Versus Provenance Finding + +Fields like `provisional`, `supported`, `assumption` (as a kind), and `confidence` describe **epistemic status only** — they classify how confident or well-supported a claim is. They do not record where the information originated. A node with `kind=unknown, status=unknown, confidence=low` could have come from user input, model inference, or evidence extraction. + +### Are evidenceIds Provenance or Graph References + +**Graph references.** In `buildInitialGraph`, `evidenceIds` are populated from `obs.id` — IDs that originate from the LLM's reconstruction output (`reconstruction.observedStates[].id`). These are internal identifiers for model-generated evidence records, not user-supplied source identifiers. The same applies during graph updates: node relationships use string IDs that are graph-internal references. + +### Are Inferred Nodes Explicitly Marked as Model-Generated + +**No.** Neither `builder.js` (initial build) nor the update-proposal path marks inferred nodes with any model-generated flag. Node kinds in the update path are set by the LLM's JSON output — there is no explicit "this was model-inferred" marker. + +### Recoverability Result: not_recoverable + +A later consumer receiving only the validated graph (with no conversation history or LLM response) cannot determine which statements came from user/evidence and which were generated as model inference. All nodes produced by different paths (initial build, emergent reasoning, decomposition children) have identical schema shape. The evidenceIds field contains IDs referencing model-generated reconstruction records, not external source identifiers. + +### Production Population Finding + +- No relevant source/provenance fields are populated in production graph-building code +- Node kinds (`observation`, `assumption`, `unknown`, etc.) are used for semantic typing, not provenance +- Evidence IDs are model-generated internal references (not user-supplied identifiers) +- No inferred nodes carry any explicit model-generated marker + +### Experiment Conclusion + +The existing SituationGraph does not preserve supplied-versus-inferred provenance. It represents epistemic status (how confident or well-supported information is) but has no mechanism to record where information originated. This confirms the hypothesis from Experiment 53's open question: while semantic output can separate supplied meaning from inference, the graph layer cannot recover that separation because it lacks provenance tracking fields entirely. + +### Limitations + +- Inspection-based; no live model run was performed +- Only source files directly relevant to node schema and construction were examined +- The non-strict Zod schema allows extra fields but none are used for provenance in production code +- Does not address whether a fix is needed — only whether the gap exists + +### Status + +**Pending Rob's review.** The audit confirms a provenance gap. No production code was changed. Working tree clean before commit. + +### Production Unchanged + +- `lib/graph/schema.js`: 0 lines changed +- `lib/graph/builder.js`: 0 lines changed +- `lib/graph/apply-proposal.js`: 0 lines changed +- `lib/graph/update-proposal.js`: 0 lines changed +- No production files modified +- Working tree clean before commit + +### Tests / Validation Run + +No test run required for the inspection result. Source inspection alone is sufficient — the schema definition in `lib/graph/schema.js` is a static contract, and no runtime execution is needed to confirm the absence of provenance fields. + +### Documentation Updated + +- `docs/current-handoff.md` — handoff line 127 and Return-to-Work Note updated +- `docs/design-evolution-log.md` — Experiment 54A section appended + +## Experiment 54B — Trace Provenance Loss Through Graph Pipeline (2026-08-07) + +Experiment 54A proved the provenance gap exists in the graph. This experiment traced both flows to find exactly where upstream provenance is lost and whether it is recoverable at any point. + +### Method + +Inspected file-by-file through the complete data flow of both paths, tracking the evidenceType field from its creation in the reconstruction layer through to the final graph state. + +**Source-traced paths:** +- Flow A (initial): analysis.js → buildInitialGraph() → SituationGraph.nodes +- Flow B (update): orchestrator.updateCase() → prompt → parseGraphUpdateProposal() → applyValidatedProposal() → SituationGraph.nodes/edges + +### Trace Table + +| Step | File | Data Present? | Provenance Status | +|------|------|---------------|-------------------| +| LLM output raw | analyseScenario() lib/analysis.js:28-134 | reconstruction + evidence with evidenceType enum | PRESENT (supplied vs inferred explicit in evidence records) | +| Validation v0.2 schema | lib/reconstruction/schema.js:129-162 | EvidenceRecordSchema includes evidenceType: ["direct_observation","reported_statement","interpretation","assumption","inferred_relationship"] | PRESENT — enum encodes the distinction upstream | +| buildSuccessResultV2 return | lib/analysis.js:173-184 | {reconstruction, evidence} returned | PRESENT in both paths | +| buildInitialGraph receives data | lib/graph/builder.js:17-18 | reconstruction + evidence map built (line 62) | UPSTREAM AVAILABLE | +| Nodes created with kind/status | lib/graph/builder.js:40-54 (ensureNode), lines 75-209 | Nodes get kind/status/confidence from semantic mapping of reconstruction fields (observedStates→observation, actors→observation, systemsOrObjects→metric, etc.) | LOST — no provenance field on nodes | +| EvidenceMap built but unused | lib/graph/builder.js:61-64 | evidenceMap populated with evidence records (ev.id → ev) | DEAD CODE — never queried after construction | +| addEvidenceToNode called | lib/graph/builder.js:66-70, 100 | Only obs.id pushed to node.evidenceIds as string reference | LOST — ID only, no type metadata transferred | +| buildMinimalGraph fallback | lib/graph/builder.js:257-280 | No evidence at all; nodes created from scenario text directly | NO UPSTREAM PROVENANCE AVAILABLE | +| Orchestrator startCase → makeGraph | lib/graph/orchestrator.js:383 | centralStatement = scenario string preserved at graph root | PARTIAL — only the raw scenario survives as centralStatement | +| updateCase receives answer | lib/graph/orchestrator.js:569-758+ | Answer parameter enters orchestrator, included in prompt to LLM | SUPPLIED ANSWER TEXT present in prompt | +| LLM proposes graph updates | based on prompt content including answer | No separation of user-supplied vs model-inferred in proposal | LOST at prompt construction boundary | +| parseGraphUpdateProposal output | lib/graph/update-proposal.js:100-157 | GraphUpdateSchema with addedNodes using situationNodeSchema | NO provenance on added nodes or edges | +| applyValidatedProposal receives answer | lib/graph/apply-proposal.js:2723 | answer parameter passed through (line 2723), used in deriveReasoningStateOverride (line 2875) | PRESENT but NOT used for provenance — only affects reasoning state derivation | +| applyGraphUpdate applies changes | lib/graph/apply-proposal.js, line 2882+ | Graph modified; no new provenance fields added | LOST — nodes/edges created without source metadata | + +### Provenance Loss Summary + +**Flow A (initial graph):** +- Upstream of graph: evidenceType enum explicitly distinguishes supplied from inferred in `evidenceRecordSchema` +- First loss point: `buildInitialGraph()` at `lib/graph/builder.js` — nodes are created with semantic kind/status but NO provenance field. The evidenceMap is built (line 62) but never used. Only `obs.id` is added to node.evidenceIds as a bare string reference without type information. +- Recovery path: NOT from graph state alone. Would require the upstream reconstruction + evidence array that was consumed during initial build. + +**Flow B (update):** +- User answer enters as `answer` parameter in orchestrator, flows through LLM prompt, becomes part of a proposal with no provenance metadata on nodes or edges +- No separation between "user supplied this text" and "model proposed these graph changes" anywhere in the update pipeline +- The `answer` reaches `applyValidatedProposal` (line 2723) and is used for reasoning state derivation (line 2875), but no provenance field is added to nodes/edges + +### EvidenceIds Clarification Conclusion + +The node-level evidenceIds does NOT represent "the list of nodes from which this information was derived." Instead: + +- `evidenceIds` is a **reference list** — each string in the array is an ID that references a specific record in the upstream evidence array +- The distinction between supplied and inferred information lives in the **evidenceType field of those evidence records**, not on the node itself +- A node's kind/status fields encode epistemic classification (what role does this node play and how confident are we), NOT provenance (where did this information come from) +- To recover whether a piece of information was user-supplied or model-inferred, you must look up each evidenceId in the original evidence array and check its evidenceType field — which means **provenance recovery depends on access to the upstream evidence data, not on the graph state alone** + +This is actually useful: it clarifies that provenance IS recoverable from validated reconstruction output (the reconstructed data includes an evidence array where each record has evidenceType), but it is NOT recoverable from graph state alone. The `evidenceIds` array is a bridge to upstream provenance, not provenance itself. + +### Key Findings + +1. **Supplied-vs-inferred provenance EXISTS upstream.** `evidenceRecordSchema` (lib/reconstruction/schema.js line 116-121) has an explicit enum: direct_observation, reported_statement (supplied categories) vs interpretation, assumption, inferred_relationship (inferred categories). This is the most important finding. + +2. **First provenance loss point is `buildInitialGraph()`.** In lib/graph/builder.js lines 61-70, the evidenceMap is built but dead-coded — never queried. Only ID strings are added to node.evidenceIds without type metadata. + +3. **Update flow has no provenance preservation.** The user's answer arrives as a parameter but becomes embedded in an LLM prompt with no traceability. No node receives source metadata during updates. + +4. **Provenance recovery requires upstream data, not graph state.** Since the SituationGraph schema has no provenance fields and nodes only carry ID references to evidence records, any provenance determination must reference the original reconstruction or update evidence array — it cannot be derived from the graph alone. + +### Experiment Conclusion + +The supplied-versus-inferred distinction is fully preserved in the upstream reconstruction/update pipeline output (the validated evidence arrays carry explicit evidenceType values for each record). However, this distinction is never encoded into the SituationGraph nodes during either initial build or update application. The evidenceIds field on nodes provides indirect access to provenance via ID references, but only if the original evidence data remains available downstream of the graph. + +The core insight: provenance is not lost from the pipeline — it is preserved in the reconstruction output that feeds the builder. It IS lost when the builder converts that output into a SituationGraph because the node schema has no field to receive it. This means any future implementation would need to add a provenance-bearing field to the node schema and propagate evidenceType through buildInitialGraph and applyValidatedProposal — though the specific implementation approach (field name, placement, propagation mechanism) remains undecided. + +### Limitations + +- Inspection-based; no live model run required +- Traced production paths only (builder.js, apply-proposal.js, update-proposal.js) +- Did not examine LLM prompt templates to determine if they preserve answer-supplied vs inference distinction in output formatting +- The analysis assumes evidence records remain accessible after graph construction — downstream usage patterns were not audited + +### Status + +**Pending Rob's review.** Tracing complete. No production code changed. Working tree clean before commit. + +### Production Unchanged + +- `lib/reconstruction/schema.js`: 0 lines changed +- `lib/graph/builder.js`: 0 lines changed +- `lib/graph/apply-proposal.js`: 0 lines changed +- `lib/graph/orchestrator.js`: 0 lines changed +- No production files modified +- Working tree clean before commit + +### Tests / Validation Run + +No test run required — this is a source-trace audit confirming data flow paths, not a behavioral test. + +### Documentation Updated + +- `docs/current-handoff.md` — Return-to-Work Note updated with 54B findings +- `docs/design-evolution-log.md` — Experiment 54B section appended + + +### Report 54B — Trace Provenance Loss Through Graph Pipeline + +#### Experiment Purpose + +Trace where the supplied-versus-inferred provenance distinction is lost in both the initial graph build and update flows, determine whether it is recoverable at any point in the pipeline, and document what future work must do to address the gap. + +#### Method + +File-by-file source inspection of the complete data flow for both paths: Flow A (analyseScenario → buildInitialGraph → SituationGraph) and Flow B (updateCase → LLM prompt → parseGraphUpdateProposal → applyValidatedProposal). + +#### Provenance Existence Upstream + +Yes. The evidenceRecordSchema in lib/reconstruction/schema.js lines 116-121 defines an explicit enum: direct_observation, reported_statement, interpretation, assumption, inferred_relationship. Supplied categories (direct_observation, reported_statement) are separate from inferred categories (interpretation, assumption, inferred_relationship). This distinction is preserved in the validated reconstruction output returned by analyseScenario and consumed by buildInitialGraph. + +#### First Provenance Loss Point + +- Flow A (initial graph): lib/graph/builder.js lines 61-70. The evidenceMap is built at line 62 from all evidence records but never queried after construction. Only the raw ID string (e.g., obs.id) is added to node.evidenceIds via addEvidenceToNode — no type metadata or provenance classification is transferred to the node schema, which has no provenance field defined in lib/graph/schema.js lines 55-70. + +- Flow B (update): lib/graph/orchestrator.js where updateCase passes the user answer into an LLM prompt without separating "user-supplied" from "model-inferred" content, and lib/graph/update-proposal.js where graphUpdateSchema builds addedNodes using situationNodeSchema which has no provenance field. The answer parameter reaches applyValidatedProposal (lib/graph/apply-proposal.js line 2723) and is used in deriveReasoningStateOverride (line 2875), but no provenance metadata is attached to nodes or edges during the update application. + +#### EvidenceIds Clarification Conclusion + +The node-level evidenceIds field does not represent "the list of nodes from which this information was derived." Instead, each string in evidenceIds is a reference ID that points to a specific record in the upstream evidence array. The supplied-versus-inferred distinction lives on those upstream records (their evidenceType enum field), not on the node itself. To recover whether a piece of information was user-supplied or model-inferred requires accessing the original reconstruction or update evidence array and checking each referenced record's evidenceType — provenance recovery therefore depends on access to upstream data, not on the graph state alone. + +#### Update Flow Answer Handling + +The user answer enters orchestrator.updateCase() as a parameter, gets embedded in an LLM prompt without source attribution markers, and the resulting proposal carries no provenance metadata onto nodes or edges. The answer survives as a raw string through to applyValidatedProposal (lib/graph/apply-proposal.js line 2723) where it influences reasoning state derivation (line 2875), but no node receives any indication that it was derived from user-supplied content versus model inference. + +#### Provenance Recovery Feasibility + +From graph alone: No — the SituationGraph schema has no provenance fields on nodes or edges, and evidenceIds only carries ID references without type metadata. From upstream data: Yes — the validated reconstruction output (returned by analyseScenario) includes a complete evidence array where each record has an explicit evidenceType enum field distinguishing supplied from inferred information. + +#### Future Work Consideration (Beyond 54B Scope) + +Any future fix would need to embed provenance in graph state rather than keeping it only in external upstream data — potentially via a node-level provenance field and evidenceType propagation through buildInitialGraph and applyValidatedProposal, while retaining the existing evidenceIds reference system as a cross-reference layer. The specific implementation approach remains undecided. 54C examines whether update provenance is deterministically knowable at the application boundary before any fix is designed. + +### Status +**Committed.** Report appended to design log. Branch: `feature/user-workspace-ux-v0.7`. Working tree clean before commit. + + +## Experiment 54C — Is Update Provenance Deterministically Knowable Before Graph Application? (2026-08-07) + +### Objective + +Determine whether production code can distinguish user-supplied material from model-proposed additions at the boundary before graph mutation occurs. This is source-trace only; no code changed, no tests run, no solution designed. + +### Hypothesis + +The active update pipeline holds the raw user answer and the validated model proposal as distinct inputs immediately before graph mutation. If so, origin may be deterministically knowable at that boundary even though the current graph does not store it. + +### Files Actually Inspected + +- `lib/graph/orchestrator.js` — lines 569–768 (updateCase / updateCaseWithDependencies) +- `lib/graph/apply-proposal.js` — lines 2719–2912 (applyValidatedProposal) +- `lib/graph/update-proposal.js` — lines 1–157 (parseGraphUpdateProposal, graphUpdateSchema field list) +- `lib/graph/schema.js` — lines 55–98 (situationNodeSchema, situationEdgeSchema), lines 155–163 (graphUpdateSchema), lines 174–179 (updateCaseRequestSchema) +- `docs/design-evolution-log.md` — Experiment 54B section (lines 5080–5178) +- `docs/current-handoff.md` — lines 129–131 (Return-to-Work Note) + +Production files actually inspected: `lib/graph/orchestrator.js`, `lib/graph/apply-proposal.js`, `lib/graph/update-proposal.js`, `lib/graph/schema.js`. +Reconstruction files NOT inspected: `lib/reconstruction/schema.js`, `lib/analysis.js`, `lib/graph/builder.js` (per budget constraints). + +### Update Flow Trace + +| Stage | What contains the user answer? | What contains model proposals? | Are they separate? | Origin deterministically knowable? | +|-------|-------------------------------|--------------------------------|--------------------|-----------------------------------| +| HTTP request body → updateCaseWithDependencies (orchestrator.js:594) | `answer` from `parsedRequest.data.answer` (schema: z.string().min(1).max(5000)) | — | Yes | N/A — no proposal yet | +| LLM prompt construction (orchestrator.js:633-638) | `answer` embedded in prompt text as context | Model generates response with additions/proposals | Separated by mechanism: answer is context, model output is the new data | Only by knowing that `answer` was supplied and `rawResponse` came from the LLM. No metadata markers separate user text from model inference within the proposal itself. | +| parseGraphUpdateProposal (update-proposal.js:100-157) | Not present in parsed output — stripped during JSON parsing | `parsedProposal.proposal` (graphUpdateSchema: addedNodes, updatedNodes, addedEdges, etc.) | N/A — user answer is gone from the parsed proposal object | Merged_or_lost — the raw user answer is no longer part of the proposal object | +| Orchestrator call to applyValidatedProposal (orchestrator.js:683-688) | `answer` passed as separate function argument | `proposal: parsedProposal.proposal` passed as separate function argument | **explicitly_separate** — two distinct named parameters in one function call | **implicitly_distinguishable** — parameter names distinguish them, but the proposal object itself contains no metadata labeling which nodes/edges came from user input vs model inference | +| Inside applyValidatedProposal (apply-proposal.js:2719-2880) | `answer` used only in `deriveReasoningStateOverride` (line 2875-2878). No provenance metadata derived from it. | `proposal` (validated against graphUpdateSchema, reconciled via reconcileResolutionSemantics) — then snapshot at line 2869 | Still **explicitly_separate** within the function scope | The two inputs are separate variables, but the proposal object carries no origin labels on its nodes/edges | +| applyGraphUpdate calls (apply-proposal.js:2882, 2912) | `answer` not passed to applyGraphUpdate | proposalSnapshot applied to graphSnapshot. Nodes created without source metadata. | N/A — applyGraphUpdate receives only the merged snapshot | **merged_or_lost** — origin information is not transmitted to the mutation function | +| Resulting SituationGraph (after line 2912) | No record of which nodes/edges came from user | All new nodes/edges carry no provenance field | N/A | The graph stores only structural data; origin is unrecoverable from graph state alone | + +### Answers to Required Questions + +1. **Is the raw user answer still available immediately before proposal application?** +Yes. In orchestrator.js line 683-688, `answer` (from parsedRequest.data.answer) is passed as a named argument to applyValidatedProposal alongside `proposal`. Both exist as separate function arguments at the call site. + +2. **Is the validated model proposal a separate object at that same point?** +Yes. `parsedProposal.proposal` is a distinct object from `answer`. It is the output of parseGraphUpdateProposal, validated against graphUpdateSchema, and passed as the `proposal` argument. The answer and proposal are different values in the JavaScript call stack. + +3. **Does applyValidatedProposal receive both, or only the proposal/graph?** +Both. The function signature (line 2719) receives `{ situationGraph, proposal, previousQuestion, answer }`. All four are separate destructured parameters. + +4. **Can deterministic code identify "user supplied" versus "model proposed" without asking the LLM?** +At the applyValidatedProposal call boundary: yes, by parameter identity. The `answer` argument contains user-supplied text; the `proposal` argument contains model-generated graph changes. These are distinguishable because they are different variables in the JavaScript runtime and come from different sources in orchestrator.js (user request vs LLM response). + +However: within the proposal object itself, there is no metadata on individual nodes or edges indicating whether a specific node originated from user-supplied information or was model-inferred. The proposal schema (graphUpdateSchema) has no provenance field on addedNodes or addedEdges — situationNodeSchema contains only structural fields (id, label, description, kind, status, confidence, value, unit, evidenceIds, dependsOn, affects, parentId, childIds). + +5. **At what exact function boundary does that distinction cease to be recoverable?** +The distinction is knowable at the `orchestrator.updateCaseWithDependencies` call site (line 683) because both `answer` and `proposal` are separate named arguments. It becomes **merged_or_lost** at two points: + +a) Inside applyValidatedProposal: the `answer` parameter is passed only to `deriveReasoningStateOverride` and never used to annotate nodes/edges with source metadata. Origin information exists in scope but is not applied to the graph mutation path. + +b) At `applyGraphUpdate` calls (lines 2882, 2912): only `graphSnapshot` and `proposalSnapshot` are passed. The `answer` parameter is discarded — it never reaches the mutation function that creates nodes/edges. + +6. **Is the loss caused by which factor?** +**More than one boundary.** Specifically: +- Proposal schema (graphUpdateSchema/situationNodeSchema): no provenance field exists on addedNodes or addedEdges — this is the primary structural cause. If a provenance field existed, it could be populated. +- Application function signature (applyGraphUpdate at lines 2882/2912): `answer` is not passed to the mutation function that actually creates graph state. Even if nodes had provenance fields, the source information would need to be carried through to reach them. +- Graph schema: the resulting situationGraph has no provenance-aware structure (follow-on effect of proposal schema gap). + +7. **Does the update path already contain enough information to assign provenance deterministically before graph storage?** +**No.** While user answer and model proposal are separate at the applyValidatedProposal call boundary, the answer text is a free-form string with no structural mapping to specific nodes in the proposal. The orchestrator does not know which parts of the LLM response were derived from user input versus independently inferred by the model. Even though both inputs exist as distinct parameters, there is no deterministic mechanism within the data flow to map user-supplied content to specific graph nodes/edges in the proposal. + +### Experiment Conclusion + +**Input origin remains explicit before application, but per-node supplied-versus-inferred provenance is not represented in the validated proposal and is lost before durable graph mutation.** + +The raw user answer and validated model proposal are explicitly separate at the `applyValidatedProposal` function call boundary (orchestrator.js:683-688). Whole-input origin is explicit: `answer` = user supplied; `proposal` = model produced. However, per-node origin inside the proposal is not deterministically recoverable from the validated proposal alone. This distinction does not translate to per-node provenance because: + +1. The graphUpdateSchema / situationNodeSchema has no provenance field on nodes or edges. +2. The raw user answer text has no structural mapping to proposal node boundaries — the LLM consumes the answer as context and generates additions independently, so there is no way to deterministically say "this node contains user information" versus "this node contains model inference." +3. The `answer` parameter is not forwarded to applyGraphUpdate, the function that actually mutates graph state. + +Provenance loss occurs at the intersection of proposal schema (no provenance field) and application logic (answer discarded before mutation). + +### Limitations + +- Source-trace audit only; no live model or parsing executed +- Did not examine prompt templates to determine if they carry answer-supplying markers +- Did not examine reconcileResolutionSemantics for any implicit origin tagging +- Did not examine applyGraphUpdate internals beyond the call signatures + +### Status + +**Pending Rob's review.** Source-trace complete. No production code changed. Working tree clean before commit. + +### Production Unchanged + +- `lib/graph/orchestrator.js`: 0 lines changed +- `lib/graph/apply-proposal.js`: 0 lines changed +- `lib/graph/update-proposal.js`: 0 lines changed +- `lib/graph/schema.js`: 0 lines changed +- No production files modified +- Working tree clean before commit + +### Tests / Validation Run + +No test run required; Experiment 54C is a source-trace audit. + +## Experiment 54D — Does the Update Prompt Already Preserve User-vs-Model Origin? (2026-08-07) + +### Objective + +Inspect the production graph-update prompt to determine whether it marks which content is the user's answer versus model-generated interpretation strongly enough that provenance could, in principle, be preserved downstream. This is a source-inspection experiment only. Do not implement provenance. + +Two distinct questions: +- **Prompt-level source identity:** Can the model tell "this text is the user's answer"? +- **Proposal-level provenance:** Can downstream code tell "this particular proposed node directly represents supplied content rather than model inference"? + +These may have different answers. The first may be explicit while the second is absent. + +### Hypothesis + +The existing update prompt may already contain clearly separated sections such as: previous graph/context; current question; user answer; instructions for graph changes. If that structure is explicit, the upstream information needed to distinguish user-supplied input from model-generated additions may already exist at prompt time. If the prompt blends everything into undifferentiated text, provenance is weaker even before proposal parsing. + +### Files Actually Inspected + +- `lib/graph/orchestrator.js` — lines 617-638 (caller passing answer to buildPrompt); line 20 (import statement) +- `lib/graph/prompt-builder.js` — full file (buildGraphUpdatePrompt function and its helpers: formatEnumValues, formatGraph, formatExampleAnswerBlock) + +Production files inspected: `lib/graph/orchestrator.js`, `lib/graph/prompt-builder.js`. + +### Prompt Builder Function Inspected + +Function: `buildGraphUpdatePrompt` in `lib/graph/prompt-builder.js`, exported as `buildGraphUpdatePrompt` (line 26), aliased as `buildUpdatePrompt` (line 134). + +### Arguments Passed Into Prompt Builder + +- `situationGraph` — full current SituationGraph object +- `previousQuestion` — string (the previously selected question) +- `answer` — string (raw user answer, min 1 char, max 5000 chars per updateCaseRequestSchema) +- `promptVersion` — string, defaults to "v0.4" + +The caller in orchestrator.js:633 passes these four arguments directly from function parameters and a config value. No provenance metadata is constructed or passed at the call site. + +### User Answer Source Identity in Prompt + +Status: **explicit** + +Section `## User Answer` (line 48-49 of prompt-builder.js) contains the raw user answer as its entire content, separated by a Markdown header from everything above and below. The section header unambiguously identifies the block as user-supplied text. No other section contains this exact string. + +### Previous Question Separation in Prompt + +Status: **explicit** + +Section `## Previous Selected Question` (line 45-46) contains only the previous question string, clearly separated by a Markdown header from both the graph above and the answer below. + +### Graph/Context Separation in Prompt + +Status: **explicit** + +Section `## Current Situation Graph` (line 42-43) contains the full situation graph as formatted JSON, clearly separated by a Markdown header from all other content. + +### Instruction Versus User-Content Separation + +Status: **explicit** + +All instruction blocks use Markdown headers (`## Proposal Rules`, `## Allowed Node Kinds`, `## Additional Guidance`, etc.). These headers create visual and structural boundaries between user-provided sections (graph, question, answer) and system instructions. The prompt does not interleave instructions within user-content blocks. + +### Does Prompt Explicitly Identify User-Supplied Content + +Status: **explicit** + +Yes. The `## User Answer` header unambiguously marks which text block is the user's contribution. Additionally, Proposal Rule 9 states "Every new unknown must be directly traceable to the user's answer," and Rule 13a references "the relevant answer-derived decision or context node" — both rules reinforce that the answer section represents the authoritative user-supplied source. + +### Does Prompt Explicitly Distinguish Supplied Meaning from Model Inference + +Status: **implicit** + +The prompt does not contain an explicit instruction telling the model to label or separate supplied meaning from inference in its output. However, implicit cues exist: Rule 9 requires traceability ("directly traceable to the user's answer"), Rule 9a requires a why-it-matters clause for new unknowns (implying the model must reason about what it derives versus what is given), and Rule 13a references "answer-derived" nodes. These create an expectation that the model should distinguish derived from supplied content, but there is no structural output mechanism to preserve that distinction in the JSON proposal. + +### Does Requested Proposal Output Contain Provenance + +Status: **absent** + +The `graphUpdateSchema` (defined in `lib/graph/schema.js`, lines 155-163) has no provenance or source fields on any of its node or edge schemas. The `addedNodes` schema uses `situationNodeSchema` which contains only structural fields (id, label, description, kind, status, confidence, value, unit, evidenceIds, dependsOn, affects, parentId, childIds). No field exists to tag content as "user-supplied," "model-inferred," or any equivalent origin marker. + +### Prompt-Level Source Identity Status + +**explicit** — The model can clearly identify which input text came from the user (the `## User Answer` section) and which sections contain context/instructions (the graph JSON, previous question, rules, guidance). + +### Proposal-Level Provenance Status + +**absent** — The proposed output schema has no provenance fields. Even if the model understands which inputs were user-supplied, it has no mechanism to annotate its output nodes/edges with origin information. + +### Trace from User Answer to Validated Proposal + +| Stage | Source Identity | Supplied-vs-Inferred Meaning Explicit? | +|-------|----------------|----------------------------------------| +| 1. `answer` parameter in orchestrator.js:636 | explicit (parameter name) | N/A — raw string | +| 2. `## User Answer` section in prompt (prompt-builder.js:49) | explicit (named Markdown section) | implicit — the answer is given; no instruction distinguishes parts of it as supplied vs inferred | +| 3. LLM processes prompt and generates proposal | explicit (model can see which text is user answer) | implicit — rules require traceability but do not provide output mechanism for provenance | +| 4. `parsedProposal.proposal` after parseGraphUpdateProposal | absent (no source metadata on nodes/edges) | absent — graphUpdateSchema has no provenance fields | + +### First Point Where Per-Node Provenance Becomes Unavailable + +The proposed output schema (`graphUpdateSchema` in `lib/graph/schema.js`) defines the JSON contract returned by the LLM. Since none of its node or edge schemas include any origin/provenance field, per-node provenance is unavailable at the **output definition** stage — i.e., the prompt's own requested format cannot carry provenance even if the model understands it internally. This is upstream of parsing and validation; even before `parseGraphUpdateProposal` runs, the schema itself forbids provenance encoding. + +### Could the Model Know Which Input Came from the User + +**Yes.** The `## User Answer` section makes the user's contribution unmistakably identifiable. Rules 9 and 13a further reinforce the distinction between answer-derived content and model-generated additions. + +### Could Downstream Deterministic Code Know Which Proposed Node Came from Supplied Meaning + +**No.** The `graphUpdateSchema` has no provenance field on addedNodes, updatedNodes, or addedEdges. The validated proposal is a plain JSON object with no origin metadata. There is no deterministic mechanism to recover per-node provenance from the proposal alone. + +### Experiment Conclusion + +**Prompt clearly preserves user-source identity but proposal schema loses per-node provenance.** + +The production update prompt (prompt-builder.js) already separates the user answer into a distinct named section (`## User Answer`) with clear visual and structural boundaries from graph context, instructions, and constraints. The model can unambiguously identify which text is user-supplied. Rules 9 and 13a reinforce traceability expectations. + +However, the requested output schema (graphUpdateSchema in lib/graph/schema.js) has no provenance fields on nodes or edges. Even if the model internally distinguishes derived from supplied content, the JSON output contract cannot encode that distinction. Provenance is lost at the output-definition stage — before any parsing or validation occurs. + +This means: +- Prompt-level source identity: explicit +- Proposal-level provenance: absent (structural limitation of schema) +- The tested prompt already preserves user-source identity clearly; the blocking gap identified here is that the validated proposal does not carry per-node provenance forward. The eventual representation remains undecided. + +### Limitations + +- Source-inspection audit only; no live model call executed +- Inspected the production prompt-builder and its immediate caller only +- Did not inspect whether evidenceType in reconstruction can carry origin information for the answer field itself +- Did not evaluate whether a schema change would be sufficient or whether additional upstream markers are needed +- Conclusions apply to the current prompt version (v0.4); earlier versions may differ + +### Status + +**Pending Rob's review.** Source-inspection complete. No production code changed. Working tree clean before commit. + +### Production Unchanged + +- `lib/graph/orchestrator.js`: 0 lines changed +- `lib/graph/prompt-builder.js`: 0 lines changed +- `lib/graph/schema.js`: 0 lines changed +- No production files modified +- Working tree clean before commit + +### Tests / Validation Run + +No test run required; Experiment 54D is a prompt-source audit. + + +## Experiment 54E — Can Existing Evidence References Preserve Provenance Without Adding a Node Field? (2026-08-07) + +### Objective + +Inspect whether the current graph design already preserves referential provenance through existing evidence identities and references, without requiring a new node field. This is a source-inspection experiment only. Do not implement provenance. + +### Hypothesis + +Existing `evidenceIds` and evidence records may already provide enough referential structure to preserve provenance if: (1) evidence records survive beyond reconstruction; (2) their IDs remain resolvable later; (3) `evidenceType` remains attached to those records; (4) graph nodes can reliably link to the evidence records that justify them. + +### Files Inspected + +- `lib/reconstruction/schema.js` — evidenceRecordSchema definition, reconstructionV2Schema evidence field +- `lib/graph/schema.js` — situationNodeSchema evidenceIds field, graphUpdateSchema structure +- `lib/graph/builder.js` — how evidence IDs are assigned during initial graph build (buildInitialGraph) +- `lib/graph/orchestrator.js` — startCase and updateCaseWithDependencies return values; evidence flow through the pipeline +- `lib/reconstruction/schema.js` lines 113–127 — evidenceRecordSchema fields +- `lib/graph/apply-proposal.js` — how evidenceIds are populated during graph updates (appendUniqueValue pattern) +- `lib/graph/prompt-builder.js` — rule 14 "Do not invent evidence" + +### Evidence Record Identity + +Each reconstruction evidence record (v0.2 schema) has: +- **Stable ID**: `id: z.string().min(1)` — deterministic identifier present in every record. +- **evidenceType**: `z.enum(["direct_observation", "reported_statement", "interpretation", "assumption", "inferred_relationship"])` — five distinct values. +- **Sufficient information to distinguish supplied from inferred**: The evidence-record schema contains vocabulary capable of distinguishing supplied-like from inferred-like material, but Experiment 54E did not validate how those values are assigned in production. + +Answer: **Yes, evidence records carry sufficient identity.** + +### Graph Reference Behaviour + +When a graph node contains an `evidenceId`: +- During `buildInitialGraph` (builder.js:62–70), evidence records from the analysis output are placed in an `evidenceMap`. Nodes are created with `evidenceIds` populated from actual evidence record IDs (builder.js:100: `node.evidenceIds.push(obs.id)`). +- These IDs refer to real evidence records during construction. However, after graph construction, the graph contains only string IDs — they would resolve if a lookup table existed, but they are opaque strings within the node object itself. + +Answer: **IDs refer to real evidence records at build time; become opaque strings in the graph post-construction.** + +### Evidence-Record Lifetime + +The critical flow is: +1. `analyseScenario` (lib/analysis.js) produces evidence records via reconstructionV2Schema, returned as `data.evidence`. +2. In `startCase` (orchestrator.js:378), `analysis.evidence` is passed to `buildInitialGraph` solely to populate node `evidenceIds`. +3. **The evidence records themselves are NOT included in `startCase` return value.** The return at orchestrator.js:489-566 includes only `situationGraph`, `selectedQuestion`, `diagnostics`, and `assessment`. No `evidence` field exists in the return object. +4. In `updateCaseWithDependencies`, no new evidence records are created anywhere in the pipeline. Rule 14 of the update prompt ("Do not invent evidence") explicitly forbids the model from creating them. The prompt-builder.js shows no mechanism for evidence generation during updates. +5. There is **no persistence layer** in this codebase that stores case state to disk or a database. The API routes (cases/start/route.js, cases/update/route.js) return data to the client; they do not persist anything. + +The initial evidence records exist only during the startCase execution lifetime. They are consumed to populate graph node evidenceIds but never returned alongside the case state. Subsequent update cycles produce no evidence records at all. + +Answer: **not_retained_with_graph** + +### Provenance Through Reference + +If a later consumer receives normal persisted case state (which contains only `situationGraph` with nodes having string `evidenceIds`), it cannot: +1. Read a graph node — yes, the node and its evidenceIds are present. +2. Follow its evidenceIds — no matching evidence records exist anywhere in the persisted state. +3. Resolve each ID to an evidence record — impossible; records do not exist. +4. Inspect evidenceType — N/A; no records to inspect. + +Answer: **no** — The reference chain breaks at step 2/3 because evidence records are not part of the returned case state. + +### Node-to-Evidence Completeness + +- **Initial graph nodes (from buildInitialGraph)**: Nodes from observedStates do receive evidence references (builder.js:100). Other node types (actors, systemsOrObjects, differences, contradictions, unknowns, interpretations) are created WITHOUT evidence references — only observedStates nodes get `evidenceIds` populated. +- **Nodes added during update**: No new evidence records are ever created during the update flow. Nodes created via `applyValidatedProposal` may have their `evidenceIds` field set (the schema allows it), but no source code in the update path populates them from actual evidence records. There is no mechanism to create or assign evidence record IDs during updates. + +Answer: **Incomplete by design** — Initial graph supplies evidenceRefs only for observedStates; all other nodes get none. Update-phase nodes get no evidence references at all. + +### Referential Recoverability Trace + +| Step | Result | Reason | +|------|--------|--------| +| Graph node → evidenceId | works | Nodes carry evidenceIds as string arrays | +| evidenceId → evidence record | breaks | Evidence records are consumed during startCase and never returned; no persistence layer retains them | +| Evidence record → evidenceType | N/A | Chain already broken at previous step | + +### Answered Questions + +1. **Is node identity already sufficient?** No — nodes exist and have evidenceIds, but without the referenced records they carry no provenance information. +2. **Is evidence identity already sufficient?** The evidence records themselves (when they exist) carry sufficient identity (id + evidenceType). But they are not available in case state. +3. **Are evidence records retained long enough to resolve references?** No — consumed during construction, not returned with graph. +4. **Is evidenceType available after resolution?** N/A — cannot resolve the reference to get there. +5. **Are node-to-evidence links populated consistently enough?** No — only observedStates nodes in initial build receive references; all update-phase nodes receive none. +6. **Can supplied-versus-inferred provenance currently be recovered referentially?** No. +7. **Is the problem primarily:** More than one of these: (a) evidence provenance exists upstream during reconstruction but is not persisted alongside graph state; (b) no new evidence records are created or retained during update cycles; (c) incomplete linkage even at construction time (only observedStates nodes get references). + +### Conclusion + +**Existing provenance exists but referential linkage is incomplete** — specifically: the evidence-record schema contains vocabulary capable of distinguishing supplied-like from inferred-like material (Experiment 54E did not validate how those values are assigned), and the reference chain breaks because (1) evidence records are consumed during startCase and never returned alongside graph state, making them unrecoverable in persisted case data; and (2) no evidence records are created or retained during update cycles at all. The problem is primarily missing persistence of existing provenance combined with a gap in evidence record creation during updates. + +### Limitations + +- Source-inspection audit only; no live execution tested +- Client-side state management was not inspected — if the client retains evidence records alongside graph state, referential recovery may work on that layer +- Did not evaluate whether the client could reconstruct provenance from UI-visible data +- Conclusions apply to the server-side pipeline as currently implemented + +### Status + +**Pending Rob's review.** Source-inspection complete. No production code changed. Working tree clean before commit. + +## Experiment 54F — Is evidenceType Actually Reliable Provenance, or Just a Model Label? (2026-08-07) + +### Objective + +Inspect whether the current reconstruction path assigns `evidenceType` from a clear provenance rule, or whether the LLM itself decides whether something is a reported statement, interpretation, assumption, or inferred relationship. Source-audit only. Do not implement provenance. + +### Hypothesis + +`evidenceType` may be semantically useful without being trustworthy provenance. If the LLM is asked to classify reconstructed content into those categories, then `reported_statement` may mean "the model thinks this looks like a reported statement" rather than "production code knows this came directly from the user." + +### Files Inspected + +- `prompts/reconstruct-v0.3.md` — full file; CRITICAL RULE 7 at line 155 specifying how evidenceType should be classified +- `lib/reconstruction/prompt.js` — `buildV3Prompt` function (lines 67–78), which loads prompts/reconstruct-v0.3.md and substitutes the scenario via `{{SCENARIO}}` +- `lib/reconstruction/schema.js` — `evidenceRecordSchema` definition at lines 113–127; evidence field at line 185 +- `lib/reconstruction/compatibility.js` — `normaliseAnalysisResponse` function (full file, lines 1–52); deterministic normalisation applied to parsed reconstruction results + +### Who Chooses evidenceType? + +Answer: **the LLM**. + +The prompt in `prompts/reconstruct-v0.3.md` line 155 states: + +> **evidenceType**: classify each evidence item clearly as either a direct observation, a reported statement, an interpretation, an assumption, or an inferred relationship. Do not treat raw counts as proof of causal relationships — they may be inferred relationships only when supported by explicit reasoning about denominators or rates. + +This is a classification instruction directed at the model. No production code determines or constrains which evidenceType value a given piece of evidence should have. The LLM receives scenario text and chooses an `evidenceType` for each evidence record it creates in its output. + +### Relevant Reconstruction Instruction (Summarised) + +The prompt instructs the model to: +- Produce a JSON object with `inputClassification`, `reconstruction`, `evidence`, and `nextQuestion` keys; +- For the `evidence` array, produce records each containing `id`, `description`, `evidenceType`, `source`, `attribution`, `confidence`, and `importance`; +- **For evidenceType specifically**, instructs the model to "classify each evidence item clearly as either a direct observation, a reported statement, an interpretation, an assumption, or an inferred relationship" based on its judgment of what each piece of evidence represents; +- No production-level rules restrict which category maps to which source origin. + +### Raw User Origin Status Before Reconstruction + +Answer: **model_classified**. + +The raw user scenario text is passed into the prompt via `{{SCENARIO}}` substitution at line 158 of reconstruct-v0.3.md. It is not wrapped, tagged, or structurally separated from any other content. Production code (lib/reconstruction/prompt.js) performs only a single string substitution: `content.replace("{{SCENARIO}}", scenario)`. There is no structural marker in the prompt that tells the model "this text came from the user" — it simply appears at the end of the prompt as unlabelled text. + +Note on Experiment 54D's finding: The update prompt (`lib/graph/prompt-builder.js`) does separate user-supplied answers via a Markdown header (`## User Answer`). However, **Experiment 54F inspects the initial reconstruction path only**, where no such structural separation exists. The update path uses a different prompt from a different builder and is not part of this audit's scope. + +### Raw User Origin Status Inside Reconstruction Output + +Answer: **model_classified**. + +After the LLM produces its evidence records, each record's `evidenceType` reflects the model's own judgment about what category that evidence belongs to — not any deterministic derivation from source origin. The only production-side manipulation of `evidenceType` is in `lib/reconstruction/compatibility.js`, which performs a single enum normalisation: if `evidenceType === "reported_claim"` (from an earlier schema version), it converts it to `"reported_statement"`. This does not add provenance information; it only adjusts for schema versioning. + +### Is evidenceType Deterministic? + +Answer: **no**. The model itself chooses each `evidenceType` value at generation time. No production code determines it from source structure. + +### Is reported_statement Trustworthy as User Provenance? + +Answer: **no**. A record with `evidenceType === "reported_statement"` means "the model classified this evidence item as a reported statement" — not "production code knows this came directly from the user." The model makes this classification based on its understanding of the scenario text, which may conflate what it inferred about reported claims with what was actually stated by a person. + +### Are interpretation / assumption / inferred_relationship Trustworthy as Model Provenance? + +Answer: **partially**. These values do indicate categories the model itself assigned to its own output, so they can serve as semantic labels for "model-generated content" in a loose sense. However, the boundary between these categories is defined by the LLM's judgment, not by production code — the model may classify something as an assumption when it was actually directly stated by the user, or vice versa. There is no deterministic gate separating user-supplied from model-generated content at classification time. + +### Trace: Raw User Statement → Reconstruction Prompt + +| Stage | User origin explicit? | Who assigns evidenceType? | Provenance strengthens/weakened? | +|-------|----------------------|--------------------------|--------------------------------| +| Raw user scenario text | Not structurally marked (just pasted as `{{SCENARIO}}`) | N/A | — | +| Reconstruction prompt (reconstruct-v0.3.md) | Scenario appears unlabelled at end of file, no structural distinction from system instructions | Model chooses per its judgment (CRITICAL RULE 7) | Weakened — user text is indistinguishable in structure from other prompt content | + +### Trace: Reconstruction Prompt → LLM Evidence Record + +| Stage | User origin explicit? | Who assigns evidenceType? | Provenance strengthens/weakened? | +|-------|----------------------|--------------------------|--------------------------------| +| Model processes prompt and generates evidence records | No structural marker in prompt distinguishes user content | Model (based on CRITICAL RULE 7 classification instruction) | Unchanged — model judgment, not deterministic derivation | + +### Trace: LLM Evidence Record → Schema Validation + +| Stage | User origin explicit? | Who assigns evidenceType? | Provenance strengthens/weakened? | +|-------|----------------------|--------------------------|--------------------------------| +| Raw model output (JSON) | N/A — already model-generated | Model's own choice | Unchanged | +| Compatibility normalisation (`lib/reconstruction/compatibility.js`) | No provenance added | `reported_claim` → `reported_statement` (schema versioning only) | Neutral — no provenance information added or removed; only enum compatibility | +| Zod validation against `reconstructionV2Schema` | N/A — schema validates structure, not origin | Schema accepts whatever `evidenceType` the model chose (enum valid) | Unchanged — schema enforces enum validity but not provenance correctness | + +### Does Schema Validation Verify Origin or Only Enum Validity? + +Answer: **only enum validity**. The Zod schema (`lib/reconstruction/schema.js` line 113-127) validates that `evidenceType` is one of five allowed strings. It does not and cannot verify what produced the value — whether it came from a user, was derived deterministically by production code, or was classified by the LLM. + +### Can Downstream Deterministic Code Safely Treat evidenceType as Provenance? + +Answer: **no**. `evidenceType` values are model-generated labels reflecting semantic categories the model chose during reconstruction. They do not correspond to any deterministic derivation from source origin. Production code cannot safely interpret `evidenceType === "reported_statement"` as "this was definitely supplied by the user" without additional provenance infrastructure. + +### Does Experiment 54E's Referential Approach Depend on a Label That Is Itself Model-Generated? + +Answer: **yes**. Experiment 54E found that evidence records carry sufficient identity (id + evidenceType) when they exist. But `evidenceType` is itself model-generated, not structurally derived from source origin. The referential approach depends on labels the LLM chose, meaning provenance inference at that point already rests on model classification rather than deterministic provenance — compounding the provenance gap rather than resolving it. + +### Primary Provenance Gap After This Audit + +Answer: **both**. The provenance gap has two independent causes: +1. **Evidence-record lifetime** (54E): Records are consumed during startCase and never returned with graph state. +2. **Evidence-type trustworthiness** (54F): Even if records were retained, `evidenceType` is model classification, not deterministic provenance. + +### Experiment Conclusion + +**evidenceType is semantic model output, not reliable provenance.** The reconstruction prompt instructs the LLM to classify each evidence item into one of five categories based on its own judgment. No production code deterministically derives `evidenceType` from source origin. The compatibility layer adds only a single enum normalisation (`reported_claim` → `reported_statement`) for schema versioning. Schema validation enforces enum validity but not provenance correctness. Experiment 54E's referential approach depends on a label that is itself model-generated, meaning the provenance chain was already unreliable before it broke due to missing persistence. The primary gap has two independent causes: evidence records are not retained alongside graph state, and `evidenceType` values themselves are model-classified rather than deterministically derived from source origin. + +### Limitations + +- Source-inspection audit only; no live execution tested +- Inspected only the initial reconstruction path (v0.3 prompt), not the update path which uses a different prompt structure +- Did not evaluate whether `source` or `attribution` fields on evidence records carry any provenance value beyond what `evidenceType` does +- Did not test how reliable the LLM's evidenceType classification is in production (this would require validation, not inspection) +- Conclusions apply to the v0.3 reconstruction prompt as currently implemented + +### Status + +**Pending Rob's review.** Source-inspection complete. No production code changed. Working tree clean before commit. + +## Experiment 54G — Do Evidence Records Contain Deterministic Source Linkage Back to User Words? (2026-08-07) + +### Objective + +Answer one narrow provenance question: does the current reconstruction output contain enough source information to deterministically prove that an evidence record came from the user's actual words, without trusting the LLM's `evidenceType` label? Source-audit only. Do not implement provenance. + +54F established that `evidenceType` is model classification, not reliable provenance. 54G tests whether evidence records nevertheless retain deterministic linkage to the user's actual words — such as exact text, character offsets, turn ID, source path, or other structural location data. + +### Hypothesis + +An evidence record may already preserve enough source material — such as exact text, quote, source excerpt, source ID, character range, source path, turn ID, or input reference — for deterministic code to verify that a record is directly grounded in the user's supplied text. If no such information exists, then evidence-record identity alone cannot establish user provenance. + +### Files Inspected + +- `lib/reconstruction/schema.js` — `evidenceRecordSchema` definition (lines 113–127); `reconstructionV2Schema` (line 182) +- `prompts/reconstruct-v0.3.md` — evidence record output format (lines 126–136); CRITICAL RULE 7 (line 155) +- `lib/reconstruction/compatibility.js` — `normaliseAnalysisResponse` function (full file); source null-handling (lines 31–38) +- `tests/reconstruction/compatibility.test.js` — real evidence-record shapes from tests (lines 36–44, 54–72, 107–115, 137–147) +- `lib/analysis.js` — `analyseScenario` function (full file); confirms raw scenario is NOT returned alongside evidence records (lines 173–188) + +### Evidence-Record Source-Related Fields + +From `evidenceRecordSchema`: + +| Field | Type | Provenance potential | +|-------|------|---------------------| +| `id` | `z.string().min(1)` | None — free-form, LLM-generated. No structural reference to input. | +| `description` | `z.string().min(1)` | None — model-generated summary of the evidence, not user text. | +| `evidenceType` | enum (5 values) | None — model classification, per 54F. Not structural provenance. | +| `source` | `z.string().optional()` | Partially — prompt says "who/where this came from". But: it is free-form text generated by the LLM, not deterministic production code output. In tests, it appears as `"report"` (free label) or `null` (removed by normalisation). No character offsets, turn IDs, span data, or source record references are structured into it. | +| `attribution` | `z.string().nullable().optional()` | None — test fixtures show `null` by default. When populated, it is free-form model text describing who said what, not a deterministic production-code identifier back to input. | +| `confidence` | enum | None — subjective confidence level, not source data. | +| `importance` | enum | None — semantic weight, not source data. | + +### Does Raw User Statement Coexist with Reconstruction Result? + +**Not retained alongside results.** The raw user statement (`scenario`) is the input parameter to `analyseScenario`. It is available to production code while reconstruction is being performed, but it is NOT retained alongside the returned reconstruction/evidence state for later deterministic verification. At validation time, deterministic code has: +- the evidence records (with model-generated descriptions and types); +- but NOT the raw user statement itself. + +Even if deterministic code had the scenario text available at validation time, the evidence records still lack any field containing verbatim user text or structural location back to it. + +### Does Evidence Record Preserve Exact User Wording? + +**No.** The `description` field is a model-generated summary/paraphrase of the evidence — not the user's actual words. The `source` field, when non-null, is free-form text like `"report"` (a label, not a quote). No evidence record contains verbatim or near-verbatim user text. + +### Does Evidence Record Preserve Source Location / Span / Turn Identity? + +**No.** None of the evidence record fields contain: +- character offsets within user input; +- sentence or line index; +- turn or message ID; +- source record ID owned by production code; +- any structural reference to a specific location in the original input. + +The `source` field prompt description says "who/where this came from" but it is free-form, inconsistently populated (sometimes null), and produced by the LLM not deterministic code. + +### Who Creates Evidence Record IDs? + +**The LLM.** The prompt template (line 128 of reconstruct-v0.3.md) says `"id": ""`. The schema only requires `z.string().min(1)`. No production code generates or constrains the ID beyond non-emptiness. The ID is opaque and carries no provenance semantics. + +### Can Evidence Records Be Verified Against Raw User Input? + +**No.** Deterministic verification would require: +1. Raw user text available at validation time — NOT present in result; +2. Each evidence record containing verbatim user text or a deterministic structural reference to a location within it — neither present. + +The `description` field is model-generated paraphrase, not verbatim text. The `source` field is free-form model output, not a structured pointer. No field survives from the raw input to the result in a form that code can verify. + +### Trace: Raw User Statement → Evidence Record → Source Verification + +| Stage | Raw source identity available? | Exact source wording/location? | Deterministic verification possible? | +|-------|-------------------------------|-------------------------------|-------------------------------------| +| Raw user statement | explicit (input parameter) | explicit (raw text) | N/A — this is the ground truth | +| Reconstruction prompt | absent (scenario pasted as {{SCENARIO}} with no structural markers) | partial (present in prompt body but unlabelled and indistinguishable from system instructions) | No | +| LLM evidence record | absent (all source identity lost to model generation) | absent (description is model paraphrase; source/attribution are free-form model text or null) | No | +| Validated result (analysis.js return) | absent (scenario not returned alongside evidence) | absent | No | + +### Trace: Evidence Record → Source Verification + +| Step | Status | +|------|--------| +| Get evidence record fields | present | +| Check description against user text | impossible — no verbatim text to compare | +| Check source/attribution as structured location | impossible — free-form model output, not production-code identifiers | +| Check id for provenance semantics | impossible — opaque LLM-generated string | +| Verify evidenceType independently | impossible — no source text to verify against | + +### Does Current Reconstruction Schema Contain Enough Information for Deterministic Provenance? + +**No.** The schema fields `source` and `attribution` exist as free-form optional strings, but neither is deterministic (they are model-generated), nor do they contain structured location data. The raw user statement is not returned alongside the evidence records. Even if it were, no evidence record field contains verbatim text or structural reference to it. + +### Primary Source-Linkage Gap + +Evidence records carry no verbatim user text and no deterministic structural reference (character offsets, turn/message IDs, source record references) back to the original input. The only candidate fields (`source`, `attribution`) are free-form model-generated strings that may be null, and which cannot be independently verified against user input. Additionally, the raw user statement itself is not returned with the validated reconstruction result at validation time. + +### Experiment 54G Conclusion + +**Bounded findings.** Evidence records do not contain verbatim source text. Evidence records do not contain deterministic source locations (character offsets, turn/message IDs, or structural references). `evidenceType` is model classification, not production-code provenance. Current evidence records therefore cannot independently prove source provenance without trusting the LLM's label. + +### Limitations + +- Source-inspection audit only; no live execution tested +- Inspected only the initial reconstruction path (v0.3 prompt) and the analysis pipeline +- Did not inspect whether evidence records are stored with graph nodes in a way that could later be recovered +- Did not inspect the update path which uses different prompts and potentially different provenance characteristics +- Conclusions apply to the v0.3 reconstruction output as currently implemented + +### Status + +**Pending Rob's review.** Source-inspection complete. No production code changed. Working tree clean before commit. + +## Experiment 54H — Can Trustworthy Source Identity Be Established Before LLM Interpretation? (2026-08-07) + +### Objective + +First, tighten Experiment 54G so it does not imply the raw user statement is unavailable during reconstruction. + +Then test one small capability outside the active engine: + +> **Can production code create a deterministic source record from the user's raw input before any LLM interpretation occurs, so later reasoning has something trustworthy to refer back to?** + +This is a passive test-only experiment. Do not integrate provenance into the graph or runtime. + +### Hypothesis + +A trustworthy provenance root may not require the LLM to decide anything. Production code may be able to take the raw input and create a small immutable source record **before** reconstruction that provides stable identity, source type, and verbatim source text — without attempting to classify what individual reasoning claims mean. If this cannot be done cleanly without introducing hidden interpretation, record why. + +### Context Used + +- `docs/current-handoff.md` (54G findings) +- Experiment 54G only in `docs/design-evolution-log.md` +- `lib/reconstruction/schema.js` — evidenceRecordSchema to avoid naming collisions +- Node.js built-in `crypto.createHash('sha256')` — standard library, deterministic + +### Test-Only Source Record Shape + +```json +{ + "sourceId": "...", + "sourceType": "user_input", + "verbatimText": "..." +} +``` + +This is deliberately small and does not include: evidence type, confidence, interpretation, node kind, relevance, inferred meaning, or summary. + +### Deterministic ID Method Used + +SHA-256 hash of the verbatim input text using Node.js built-in `crypto.createHash('sha256')`. + +### Why the ID Method Is Deterministic + +- SHA-256 is a deterministic cryptographic hash function: same input always produces the same output; different inputs produce different outputs (with negligible collision probability). +- Uses Node.js standard library — no LLM, no random UUID, no clock/time dependency, no mutable global state. +- Input is the verbatim string in UTF-8 encoding with no transformation or normalisation. + +### Four Input Cases + +| Case | Text | Purpose | +|------|------|---------| +| 1 | "The business has lost three major customers this year." | Baseline simple statement | +| 2 | "Three major customers have left the business this year." | Paraphrase — confirm identity tracks actual text, not meaning | +| 3 | "Revenue is down. I think pricing may be part of the problem, but I am not sure." | Multi-sentence — confirm full verbatim input survives unchanged | +| 4 | Same as Case 1 | Confirm deterministic identity | + +### Focused Test Results (9 tests, all pass) + +| # | Test | Result | +|---|------|--------| +| 1 | Case 1: verbatimText equals the exact input string | Pass | +| 2 | Case 2 (paraphrase): sourceType is always user_input | Pass | +| 3 | Case 1 and Case 4 produce identical sourceId | Pass | +| 4 | Case 1 and Case 2 produce different sourceIds despite similar meaning | Pass | +| 5 | Case 3: multi-sentence verbatim text preserved unchanged | Pass | +| 6 | Input string is not mutated by createSourceRecord | Pass | +| 7 | Helper output is deterministic across repeated calls | Pass | +| 8 | No interpretation or summarisation occurs in the source record | Pass | +| 9 | Different inputs produce different IDs (distinct text) | Pass | + +### Case Results Summary + +- **Case 1 result:** verbatimText preserved exactly ✓; deterministic SHA-256 ID produced. +- **Case 2 paraphrase result:** Same meaning, different text → different sourceId ✓; verbatimText preserved exactly. +- **Case 3 multi-sentence result:** Full multi-sentence text preserved unchanged ✓; single deterministic ID. +- **Case 4 repeated-input result:** Identical to Case 1 text → identical sourceId ✓. + +### Same-Input Identity Result + +Confirmed: Cases 1 and 4 produce exactly the same sourceId (sha256 of identical string = identical hash). + +### Different-Text Identity Result + +Confirmed: Cases 1, 2, and 3 produce different sourceIds despite Case 1≈Case 2 in meaning. + +### Verbatim-Text Preservation Result + +All three unique inputs preserved exactly as-is. No summarisation, normalisation, or transformation applied to verbatimText. + +### Input Immutability Result + +Confirmed: the original input string is never mutated. createSourceRecord reads via `String(rawInput)` (copy) before hashing. + +### LLM / Network Dependency + +None. The helper uses only Node.js built-in `crypto` and JavaScript primitives. Zero inference calls. Zero network calls. + +### Does the Helper Perform Semantic Interpretation + +No. The helper outputs exactly: `{ sourceId, sourceType, verbatimText }`. No meaning is extracted, classified, summarised, or inferred. + +### Distinction Between Source Identity and Semantic Provenance + +This experiment establishes **source identity** — answering "what exact material entered the system, and what stable identity can we give that material?" It does **not** answer: what the text means, whether it is evidence, whether it was inferred, or whether graph nodes can link to it. Source identity ≠ claim provenance. + +### What Remains Explicitly Untested + +- Linking evidence records to source records +- Linking graph nodes to evidence or source +- Update-turn source identity (not initial reconstruction) +- Persistence of source records alongside graph state +- Whether production code can reliably invoke this helper at the right boundary +- How downstream consumers discover and use the source record +- Multiple-input scenarios (multi-turn conversations) + +### Experiment Conclusion + +**Deterministic source identity is feasible before LLM interpretation.** A deterministic SHA-256 hash of raw user text produces stable, verifiable identity without any semantic processing. The concept is straightforward because it operates purely at the input boundary — no classification, no inference, no graph contract required. + +### Limitations + +- Test-only implementation; not integrated into any production path +- Single-input only (does not address multi-turn or batched input scenarios) +- Does not test whether production code can reliably invoke this at the correct reconstruction boundary +- Does not test persistence, retrieval, or downstream linking +- SHA-256 collision resistance is sufficient for source identity but does not guarantee uniqueness across all possible inputs in practice + +### Status + +**Closed.** Bounded conclusion: deterministic source identity is feasible before LLM interpretation; identical tested text produces stable identity; source identity does not establish claim or graph provenance. No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/deterministic-source-record.test.js`. + +## Experiment 54I — Can Two Different Interpretations Stay Anchored to the Same User Source? (2026-08-07) + +### Objective + +Take the deterministic source identity proved in Experiment 54H and test the next smallest step: + +> **Can two different interpretations of the exact same user input remain separate while both retaining an explicit link back to that same source?** + +This experiment deliberately does not decide which interpretation is better. It tests whether the reasoning system could preserve one source, multiple interpretations, and clear lineage back to that source without silently replacing the original input or collapsing the interpretations together. This is a passive, test-only experiment. + +### Hypothesis + +A very small lineage structure may be enough to demonstrate that: +- one source identity can anchor multiple interpretations; +- interpretations remain separately identifiable; +- neither interpretation changes or replaces the source; +- a later consumer can deterministically see that both interpretations came from the same exact input. + +If even this cannot be represented cleanly without semantic ambiguity, record that honestly. + +### Context Used + +- `docs/current-handoff.md` (Experiment 54H findings and Return-to-Work Note) +- Experiment 54H only in `docs/design-evolution-log.md` +- `tests/reconstruction/deterministic-source-record.test.js` (to verify the source-identity helper remains intact) +- Node.js built-in `crypto.createHash('sha256')` — deterministic, no LLM + +### Fixed Source + +Raw user input: +> Revenue is down. I think pricing may be part of the problem, but I am not sure. + +One deterministic source record created from that exact text using the Experiment 54H method (SHA-256 of verbatim text). + +### Two Fixed Interpretations + +**Interpretation A:** +> Pricing may be contributing materially to the revenue decline. + +**Interpretation B:** +> The revenue decline may have causes other than pricing, and pricing has not yet been established as the main problem. + +Both are plausible readings of the source. The experiment does not claim either is correct. + +### Interpretation Identity Method + +Each `interpretationId` is generated deterministically from: +- the shared `sourceId`; +- plus the exact interpretation text (combined as concatenation of `sourceId + "|" + interpretationText`). + +Same source + same interpretation → same interpretation ID. +Same source + different interpretation → different interpretation ID. +Interpretation ID changes if interpretation text changes. +No random UUID, no clock, no LLM, no mutable global state. + +### Interpretation Record Shape (Test-Only) + +```json +{ + "interpretationId": "...", + "sourceId": "...", + "interpretationText": "..." +} +``` + +This is **not** a proposed production schema. It does not include: confidence, evidence type, status, scores, timestamps, model names, node kinds, or next-question information. The purpose is only identity and lineage. + +### Focused Test Results (15 tests, all pass) + +| # | Test | Result | +|---|------|--------| +| 1 | Source retains exact verbatim user input | Pass | +| 2 | Interpretation A references the source's exact sourceId | Pass | +| 3 | Interpretation B references the same exact sourceId | Pass | +| 4 | A and B have different interpretationId values | Pass | +| 5 | Recreating A produces exactly the same interpretationId | Pass | +| 6 | Recreating B produces exactly the same interpretationId | Pass | +| 7 | Changing interpretation text changes the interpretation ID | Pass | +| 8 | Neither interpretation mutates the source record | Pass | +| 9 | Creating one interpretation does not mutate the other | Pass | +| 10 | Deterministic consumer identifies one shared source and two distinct interpretations | Pass | +| 11 | No semantic judgement chooses A over B | Pass | +| 12 | No LLM or network call occurs | Pass | +| Cross-check | Different source + same text → different interpretationId | Pass | +| Immutability | Source unchanged after multiple interpretation creations | Pass | +| Lineage traceability | Consumer traces both interpretations to exact verbatim source | Pass | + +### Shared Source Lineage Result + +Confirmed: both Interpretation A and Interpretation B reference the same `sourceId`. A deterministic consumer can recover that exactly one source anchors both interpretations. + +### Distinct Interpretation Identity Result + +Confirmed: Interpretation A and Interpretation B have different `interpretationId` values despite sharing a source. The IDs remain stable across recreation and diverge when text changes. + +### Source Immutability Result + +Confirmed: creating one or multiple interpretations from a source does not mutate the source record in any way (fields, structure, or content remain identical). + +### Interpretation Independence Result + +Confirmed: creating an interpretation for A does not affect B's identity, and vice versa. Each interpretation is independently derived from the shared sourceId + its own text. + +### Deterministic Consumer Recovery + +Confirmed: a consumer given the source record and both interpretation records can deterministically identify (1) one shared source, (2) two distinct interpretations, and (3) that both interpretations trace back to the same exact verbatim input. + +### Was Either Interpretation Selected as More Correct? + +No. The experiment does not select, score, or prefer either interpretation. + +### Was Downstream Question Selection Tested? + +No. The experiment explicitly excludes next-question derivation. + +### What This Experiment Establishes + +- One source can deterministically anchor multiple interpretations. +- Those interpretations remain independently identifiable via stable interpretationId values. +- A later consumer can determine that two interpretations came from the same exact source. +- Preserving multiple interpretations does not require changing the original source record. +- The distinction between "the user supplied this" and "interpretation A/B says this may mean X/Y" survives representation cleanly in code. + +### What This Experiment Does Not Establish + +- Which interpretation is more justified or better grounded. +- Whether different interpretations would lead to different next questions. +- Any semantic correctness claim about either interpretation. +- Graph integration, persistence, multi-turn history, or production architecture. + +### Explicitly Untested + +- Deciding which interpretation is better grounded. +- Comparing interpretations against verbatim source wording. +- Comparing interpretations against other evidence. +- Contradictions between interpretations. +- Whether disagreement should reduce confidence. +- Whether disagreement should trigger clarification. +- Downstream question selection. +- Graph integration. +- Persistence. +- Multi-turn interpretation history. + +### Limitations + +- Test-only implementation; not integrated into any production path. +- Only two fixed interpretations tested (not a general multi-interpretation protocol). +- No semantic analysis of whether either interpretation faithfully represents the source. +- The `|` separator in interpretationId derivation assumes the separator does not appear in user text; this is adequate for identity stability but would need review if adopted as production code. + +### Experiment Conclusion + +**Multiple interpretations can retain deterministic lineage to one source.** One source identity successfully anchors two distinct interpretations while both remain separately identifiable and neither mutates the original source. The distinction between unchanged user evidence and changing model interpretation remains visible in the data structure. + +### Status + +**Closed.** Bounded conclusion: multiple interpretations can retain deterministic lineage to one source. One source identity successfully anchors two distinct interpretations while both remain separately identifiable and neither mutates the original source. The distinction between unchanged user evidence and changing model interpretation remains visible in the data structure. No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/source-interpretation-lineage.test.js`. + +## Experiment 54J — Can Source-Grounded Meaning Be Separated From Interpretation-Added Meaning? (2026-08-07) + +### Objective + +Take the lineage proved in Experiment 54I and test the next smallest reasoning capability: + +> **Given one exact user source and two different interpretations linked to it, can deterministic test logic identify which parts of each interpretation are directly grounded in the source and which parts go beyond what the source actually states?** + +This experiment must **not** decide which interpretation is ultimately correct. It tests source-grounding comparison only. + +### Hypothesis + +A small test-only grounding representation may be enough to distinguish: + +- meaning directly supported by the source; +- meaning introduced by the interpretation. + +If that distinction can be represented cleanly, later reasoning could compare interpretations without immediately selecting one. If deterministic comparison requires semantic judgement that cannot be justified from text alone, record that boundary honestly. + +### Context Used + +- `docs/current-handoff.md` (Experiment 54I findings and Return-to-Work Note) +- Experiment 54I only in `docs/design-evolution-log.md` +- `tests/reconstruction/source-interpretation-lineage.test.js` (to confirm 54I remains intact — 15 tests pass) +- `tests/reconstruction/deterministic-source-record.test.js` (to confirm source identity helper remains intact — 9 tests pass) +- Node.js built-in `crypto.createHash('sha256')` — deterministic, no LLM + +### Fixed Source + +Raw user input: +> Revenue is down. I think pricing may be part of the problem, but I am not sure. + +### Two Fixed Interpretations + +**Interpretation A:** +> Pricing may be contributing materially to the revenue decline. + +**Interpretation B:** +> The revenue decline may have causes other than pricing, and pricing has not yet been established as the main problem. + +Both are plausible readings of the source. The experiment does not claim either is correct. + +### Grounding Record Shape (Test-Only) + +```json +{ + "sourceId": "...", + "interpretationId": "...", + "supportedBySource": ["..."], + "addedByInterpretation": ["..."] +} +``` + +This is **not** a proposed production schema. It does not include: confidence, evidence type, status, scores, timestamps, model names, node kinds, or next-question information. + +### Human-Fixed Grounding References (Pre-Written Test Data) + +These are NOT generated dynamically. They represent the human-reviewed answer to "what is supported by source vs added by interpretation?" + +**Interpretation A reference:** + +`Supported by source` captures only: +- revenue is down; +- pricing may be part of the problem. + +`Added by interpretation` captures that: +- pricing may be contributing **materially** to the decline. + +The source does not establish material impact. + +**Interpretation B reference:** + +`Supported by source` captures only: +- revenue is down; +- pricing may be part of the problem; +- the user is unsure. + +`Added by interpretation` captures that: +- there may be causes other than pricing; +- pricing is not established as the main problem. + +These are plausible interpretations of uncertainty, but they are not directly stated as facts in the source. + +### Focused Test Results (13 tests, all pass) + +| # | Test | Result | +|---|------|--------| +| 1 | Both grounding records reference the same exact sourceId | Pass | +| 2 | Each grounding record references its distinct interpretationId | Pass | +| 3 | Source-supported statements remain separate from interpretation-added | Pass | +| 4 | Interpretation A records 'materially' as added meaning | Pass | +| 5 | Interpretation B records alternative causes as added meaning | Pass | +| 6 | B's "not established as main problem" is interpretation-added, not user-stated fact | Pass | +| 7 | Neither grounding record mutates the source | Pass | +| 8 | Grounding creation does not affect interpretationId values | Pass | +| 9 | Recreating the same grounding record produces identical results | Pass | +| 10 | Consumer can inspect A and B: shared source / distinct interpretation / supported vs added | Pass | +| 11 | Neither interpretation is selected as a winner | Pass | +| 12 | No numeric scoring is present in grounding records | Pass | +| 13 | No LLM or network call occurs | Pass | + +### Shared Source Lineage Result + +Confirmed: both Interpretation A and Interpretation B grounding records reference the same `sourceId`. A deterministic consumer can recover that exactly one source anchors both interpretations' grounding. + +### Distinct Grounding Identity Result + +Confirmed: Interpretation A and Interpretation B grounding records have different `interpretationId` values and distinct sets of `supportedBySource` and `addedByInterpretation` entries despite sharing a source. + +### Source-Added Meaning Comparison + +**Interpretation A adds:** stronger causal/importance language ("materially") that the source does not contain. The source says pricing "may be part of the problem" without quantifying impact. Interpretation A strengthens this to material contribution. + +**Interpretation B adds:** alternative explanations ("causes other than pricing") and framing around what has not yet been established, rather than what has been confirmed. This is a plausible interpretation of uncertainty but not directly stated as fact in the source. + +### Source Immutability Result + +Confirmed: creating grounding records does not mutate the source or interpretation records in any way (fields, structure, or content remain identical). + +### Deterministic Consumer Recovery + +Confirmed: a consumer given the grounding records can deterministically identify (1) one shared source, (2) two distinct interpretations with distinct grounding profiles, and (3) what each interpretation adds beyond the source. + +### Was Either Interpretation Selected as More Correct? + +No. The experiment does not select, score, or prefer either interpretation. + +### Was Downstream Question Selection Tested? + +No. The experiment explicitly excludes next-question derivation. + +### What This Experiment Establishes + +- Source-supported meaning and interpretation-added meaning can be represented separately within a deterministic data structure. +- Two competing interpretations of the same source can each expose their own grounding profile while sharing the same source lineage. +- Interpretation A introduces stronger causal/importance language than the source. +- Interpretation B introduces alternative explanations not directly stated by the source. +- The representation preserves disagreement without treating either branch as fact. +- A later consumer can distinguish shared source content from distinct interpretation additions. + +### What This Experiment Does Not Establish + +- Which interpretation is more justified or better grounded. +- Automated semantic extraction of grounding (grounding references are human-fixed test data, not dynamically extracted). +- Whether different interpretations would lead to different downstream questions. +- Any semantic correctness claim about either interpretation. +- Graph integration, persistence, multi-turn history, or production architecture. + +### Explicitly Untested + +- Automated semantic extraction of grounding; +- choosing the better interpretation; +- evaluating interpretation correctness; +- comparing against other evidence; +- contradiction handling; +- confidence reduction; +- clarification behaviour; +- downstream question selection; +- graph integration; +- persistence; +- multi-turn reasoning. + +### Limitations + +- Test-only implementation with human-fixed grounding references (not automated semantic extraction); +- Only two fixed interpretations tested (not a general multi-interpretation protocol); +- Grounding distinctions were written by human review, not derived algorithmically; +- The `|` separator in interpretationId derivation assumes the separator does not appear in user text; this is adequate for identity stability but would need review if adopted as production code; +- Does not test whether a consumer can use these grounding records to make downstream decisions about confidence or question selection. + +### Evaluation Questions + +1. Can both interpretations retain the same source lineage while exposing different added meaning? **Yes.** +2. Can a later deterministic consumer see which claims were source-supported and which came from interpretation? **Yes.** +3. Does Interpretation A introduce stronger causal/importance language than the source? **Yes — "materially" is not in the source.** +4. Does Interpretation B introduce alternative explanations not explicitly stated by the source? **Yes — "causes other than pricing" is plausible but not stated.** +5. Does this representation preserve disagreement without treating either branch as fact? **Yes — no correctness or status fields are assigned.** +6. Does this experiment establish which interpretation is more justified? **No.** +7. Does this experiment establish which downstream question should be asked? **No.** + +### Evaluation Conclusion + +**Interpretation-added meaning can remain separate from source-supported meaning.** The test primitive of fixed reference lists successfully preserves the distinction without collapsing either interpretation into the source or into each other. + +### Regression Tests + +- `source-interpretation-lineage.test.js`: 15 tests, all pass +- `deterministic-source-record.test.js`: 9 tests, all pass + +### Status + +**Pending Rob's review.** No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/interpretation-source-grounding.test.js`. + + +## Experiment 54K — Can the Model Automatically Separate Source-Supported Meaning From Interpretation-Added Meaning? (2026-08-07) + +### Objective + +Take the representation proved in Experiment 54J and test the smallest missing capability: + +> **Given an exact user source and one interpretation of it, can the configured semantic model identify which meaning is supported by the source and which meaning was added by the interpretation?** + +Experiment 54J used human-fixed grounding references. +Experiment 54K tests whether that grounding distinction can be produced semantically without changing production behaviour. + +This is a passive test-only experiment. + +### Hypothesis + +Given an exact verbatim source and one interpretation, the model may be able to separate: +1. what the source actually supports; +2. what the interpretation adds beyond the source. + +If it can do this without strengthening, weakening, or rewriting the source meaning, then automated grounding is plausible enough for further investigation. + +### Context Used + +- `docs/current-handoff.md` (Experiment 54J findings and Return-to-Work Note); +- Experiment 54J only in `docs/design-evolution-log.md`; +- Existing `tests/reconstruction/interpretation-source-grounding.test.js` to confirm the grounding record shape remains valid; +- Existing `.env.local` configuration (same Ollama host and model). + +### Configured Host and Model + +- **Ollama host:** `http://192.168.1.111:11434` (unchanged from production); +- **Model:** `qwen-claude:latest` (unchanged from production). + +### Semantic Output Contract + +Each call receives `{ source, interpretation }` and returns exactly: +```json +{ + "supportedBySource": ["short factual statements"], + "addedByInterpretation": ["short factual statements"] +} +``` +No confidence. No scores. No explanation field. No chain-of-thought. + +### Semantic Instruction (identical for all cases) + +> Compare the interpretation with the exact source text. Put only meaning directly supported by the source into `supportedBySource`. Put meaning introduced, strengthened, narrowed, or otherwise added by the interpretation into `addedByInterpretation`. Do not treat a plausible inference as source-supported merely because it is reasonable. + +### Three Fixed Cases — Inputs and Semantic Outputs + +#### Case 1 — Interpretation A + +**Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure." +**Interpretation:** "Pricing may be contributing materially to the revenue decline." + +**Semantic output (live model):** +- `supportedBySource`: ["Revenue has declined.", "Pricing might be a factor in the problem.", "The speaker expresses uncertainty about pricing's role."] +- `addedByInterpretation`: ["Pricing has a significant or substantial impact on the revenue decline.", "The explicit expression of uncertainty is removed, strengthening the claim."] + +**Human-fixed reference:** supported = [revenue is down, pricing may be part of the problem]; added = [pricing may be contributing materially to the decline] + +**Classification:** grounding_correct + +#### Case 2 — Interpretation B + +**Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure." +**Interpretation:** "The revenue decline may have causes other than pricing, and pricing has not yet been established as the main problem." + +**Semantic output (live model):** +- `supportedBySource`: ["Revenue has declined.", "Pricing might be one contributing factor to the decline.", "The speaker is unsure whether pricing is a factor."] +- `addedByInterpretation`: ["Pricing has not yet been established as the main problem of the revenue decline."] + +**Human-fixed reference:** supported = [revenue is down, pricing may be part of the problem, the user is unsure]; added = [there may be causes other than pricing, pricing has not yet been established as the main problem] + +**Classification:** partial_grounding +The model correctly captured "not established as main problem" on the added side. However, it did not capture "alternative causes besides pricing" on the added side — this is a genuine missed addition for Interpretation B, since alternative causality is a core element of that interpretation. The supportedBySource content was all correct including uncertainty. + +#### Case 3 — Fully Grounded Control + +**Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure." +**Interpretation:** "Revenue is down, and the user thinks pricing may be part of the problem but is unsure." + +**Semantic output (live model):** +- `supportedBySource`: ["Revenue is down.", "Pricing may be part of the problem.", "The speaker expresses uncertainty about whether pricing is contributing to the issue."] +- `addedByInterpretation`: [] + +**Human-fixed reference:** supported = [revenue is down, pricing may be part of the problem, the user is unsure]; added = [] + +**Classification:** grounding_correct +The model produced no material additions for a faithful restatement. The output is an empty array on the added side, confirming it did not invent distinctions where none exist. + +### Required Questions — Answers + +1. **Did Case 1 correctly keep `materially` on the interpretation-added side?** Yes. The model placed "significant or substantial" strengthening in `addedByInterpretation` and kept it out of `supportedBySource`. +2. **Did Case 2 distinguish user uncertainty from the more specific interpretation layered onto it?** Partially. It correctly kept uncertainty in supportedBySource but failed to capture one of two additions (alternative causes) on the added side. +3. **Did Case 3 correctly produce no material added meaning?** Yes. Empty `addedByInterpretation` array for a faithful restatement. +4. **Did any plausible inference get incorrectly promoted into `supportedBySource`?** No. The model did not promote any interpretation-specific content into the supported side across any case. +5. **Did any genuinely source-supported meaning get incorrectly treated as interpretation-added?** No. All three cases retained their core source-supported content in the supported side. The most notable was uncertainty — the model correctly identified it as source-supported (even in Case 1 where my reference didn't include it). +6. **How many of three cases were:** grounding_correct = 2, partial_grounding = 1, grounding_failed = 0. +7. **Does the result suggest automated semantic grounding is plausible enough for further testing?** Yes, with caution. The model kept strengthening (materially) on the correct side in every case and did not promote interpretation content into source-supported territory. One missed addition (alternative causes) suggests occasional under-detection of additions but no false positives on the critical dimension. +8. **Does this experiment establish which interpretation is better?** No. It does not select or score interpretations. +9. **Does it establish what question should be asked next?** No. That remains untested in this experiment. + +### Grounding Summary + +| Metric | Value | +|--------|-------| +| Grounding-correct count | 2 | +| Partial-grounding count | 1 | +| Grounding-failed count | 0 | +| Interpretation-added meaning leaked into supportedBySource | No | +| Source-supported meaning pushed into addedByInterpretation | No | +| Fully grounded control avoided invented additions | Yes | + +### Evidence for Automated Semantic Grounding + +The model correctly separated strengthening ("materially" → "significant or substantial") from source meaning in every case. No interpretation-specific content leaked into supportedBySource. The fully grounded control produced an empty added array. One partial result (Case 2) missed one of two expected additions but preserved all three pieces of source-supported content. + +### Inference Timing + +| Metric | Value | +|--------|-------| +| Number of live inference calls | 3 | +| Total inference time | 96,372.37ms | +| Average | 32,124.12ms per call | +| Fastest | 25,575.87ms (Case 3) | +| Slowest | 41,532.40ms (Case 2) | + +Timing is observational only. All three cases required ~25–42 seconds of model inference time on this host/model. + +### Limitations + +- Single source text tested across all three cases — no cross-domain validation; +- Only one interpretation per source tested in each case — no multi-interpretation comparison in a single call; +- Model was `qwen-claude:latest` on host `192.168.1.111` — results may differ with other models or hosts; +- The partially correct Case 2 still captured the core supported content — the gap was in added-content completeness, not source-meaning accuracy; +- Evaluation used structured meaning checks (not keyword matching) but remains a heuristic approximation of semantic comparison; +- No repeated sampling — results reflect a single run per case. + +### Evaluation Conclusion + +**No interpretation-added meaning was observed in `supportedBySource` across the three tested cases.** One missed addition in Case 2 and occasional paraphrase mismatches mean the result is not yet sufficient for production use because completeness, stability, and broader-domain behaviour remain untested. The direction is viable pending further testing. + +### Status + +**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-interpretation-grounding.test.js`. + +---