From 7e74ad86f27b9f2c6e2542955b087c74712b61e4 Mon Sep 17 00:00:00 2001 From: robbond Date: Fri, 7 Aug 2026 13:28:22 +0100 Subject: [PATCH] experiment: trace provenance loss through graph pipeline --- docs/current-handoff.md | 4 +- docs/design-evolution-log.md | 140 +++++++++++++++++++++++++++++++++++ 2 files changed, 142 insertions(+), 2 deletions(-) diff --git a/docs/current-handoff.md b/docs/current-handoff.md index cbcd47f..1008018 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -126,6 +126,6 @@ Answer before continuing: *Created by Experiment 34. Updated by Experiments 38–53, 54A. Branch: `feature/user-workspace-ux-v0.7`.* -### Return-to-Work Note (Experiment 53 → 54A) +### Return-to-Work Note (Experiment 53 → 54A → 54B) -Experiment 53 proved semantic separation was possible; 54A inspected whether the current SituationGraph can preserve that distinction from source code alone. The audit confirms the graph's kind/status/confidence fields describe epistemic status only — there is no provenance field recording where information came from. Supplied-versus-inferred provenance is not recoverable from validated graph state. No production code changed; branch: `feature/user-workspace-ux-v0.7`. What remains open: a compatibility layer to bridge the semantic separation proved in 53 with the graph's provenance gap identified in 54A. First file to inspect when resuming: `lib/graph/schema.js` for current node fields, then `lib/graph/apply-proposal.js` for the production update path that populates nodes during inference turns. +Experiment 53 proved semantic separation was possible; 54A inspected whether the current SituationGraph preserves that distinction from source code alone (confirmed: it does not); 54B traced both flows to find where provenance is lost and whether it is recoverable. The key finding: supplied-versus-inferred provenance EXISTS upstream in evidenceRecordSchema (evidenceType enum) but is LOST at buildInitialGraph because the node schema has no provenance field and only IDs are added to evidenceIds as bare string references without type metadata. For updates, the answer survives through to applyValidatedProposal but carries no provenance onto nodes/edges. Provenance recovery from graph state alone is NOT possible — it requires access to upstream evidence arrays. What remains open: creating (not merely preserving) a node-level provenance field and propagating evidenceType through both buildInitialGraph and applyValidatedProposal. First file to inspect when resuming: `lib/reconstruction/schema.js` line 116-121 for the upstream evidenceType enum, then `lib/graph/builder.js` lines 61-70 (dead code), then `lib/graph/apply-proposal.js` lines 2719-2882 (update path with no provenance attachment). diff --git a/docs/design-evolution-log.md b/docs/design-evolution-log.md index 3aa7e7a..38bdf81 100644 --- a/docs/design-evolution-log.md +++ b/docs/design-evolution-log.md @@ -5076,3 +5076,143 @@ No test run required for the inspection result. Source inspection alone is suffi - `docs/current-handoff.md` — handoff line 127 and Return-to-Work Note updated - `docs/design-evolution-log.md` — Experiment 54A section appended + +## Experiment 54B — Trace Provenance Loss Through Graph Pipeline (2026-08-07) + +Experiment 54A proved the provenance gap exists in the graph. This experiment traced both flows to find exactly where upstream provenance is lost and whether it is recoverable at any point. + +### Method + +Inspected file-by-file through the complete data flow of both paths, tracking the evidenceType field from its creation in the reconstruction layer through to the final graph state. + +**Source-traced paths:** +- Flow A (initial): analysis.js → buildInitialGraph() → SituationGraph.nodes +- Flow B (update): orchestrator.updateCase() → prompt → parseGraphUpdateProposal() → applyValidatedProposal() → SituationGraph.nodes/edges + +### Trace Table + +| Step | File | Data Present? | Provenance Status | +|------|------|---------------|-------------------| +| LLM output raw | analyseScenario() lib/analysis.js:28-134 | reconstruction + evidence with evidenceType enum | PRESENT (supplied vs inferred explicit in evidence records) | +| Validation v0.2 schema | lib/reconstruction/schema.js:129-162 | EvidenceRecordSchema includes evidenceType: ["direct_observation","reported_statement","interpretation","assumption","inferred_relationship"] | PRESENT — enum encodes the distinction upstream | +| buildSuccessResultV2 return | lib/analysis.js:173-184 | {reconstruction, evidence} returned | PRESENT in both paths | +| buildInitialGraph receives data | lib/graph/builder.js:17-18 | reconstruction + evidence map built (line 62) | UPSTREAM AVAILABLE | +| Nodes created with kind/status | lib/graph/builder.js:40-54 (ensureNode), lines 75-209 | Nodes get kind/status/confidence from semantic mapping of reconstruction fields (observedStates→observation, actors→observation, systemsOrObjects→metric, etc.) | LOST — no provenance field on nodes | +| EvidenceMap built but unused | lib/graph/builder.js:61-64 | evidenceMap populated with evidence records (ev.id → ev) | DEAD CODE — never queried after construction | +| addEvidenceToNode called | lib/graph/builder.js:66-70, 100 | Only obs.id pushed to node.evidenceIds as string reference | LOST — ID only, no type metadata transferred | +| buildMinimalGraph fallback | lib/graph/builder.js:257-280 | No evidence at all; nodes created from scenario text directly | NO UPSTREAM PROVENANCE AVAILABLE | +| Orchestrator startCase → makeGraph | lib/graph/orchestrator.js:383 | centralStatement = scenario string preserved at graph root | PARTIAL — only the raw scenario survives as centralStatement | +| updateCase receives answer | lib/graph/orchestrator.js:569-758+ | Answer parameter enters orchestrator, included in prompt to LLM | SUPPLIED ANSWER TEXT present in prompt | +| LLM proposes graph updates | based on prompt content including answer | No separation of user-supplied vs model-inferred in proposal | LOST at prompt construction boundary | +| parseGraphUpdateProposal output | lib/graph/update-proposal.js:100-157 | GraphUpdateSchema with addedNodes using situationNodeSchema | NO provenance on added nodes or edges | +| applyValidatedProposal receives answer | lib/graph/apply-proposal.js:2723 | answer parameter passed through (line 2723), used in deriveReasoningStateOverride (line 2875) | PRESENT but NOT used for provenance — only affects reasoning state derivation | +| applyGraphUpdate applies changes | lib/graph/apply-proposal.js, line 2882+ | Graph modified; no new provenance fields added | LOST — nodes/edges created without source metadata | + +### Provenance Loss Summary + +**Flow A (initial graph):** +- Upstream of graph: evidenceType enum explicitly distinguishes supplied from inferred in `evidenceRecordSchema` +- First loss point: `buildInitialGraph()` at `lib/graph/builder.js` — nodes are created with semantic kind/status but NO provenance field. The evidenceMap is built (line 62) but never used. Only `obs.id` is added to node.evidenceIds as a bare string reference without type information. +- Recovery path: NOT from graph state alone. Would require the upstream reconstruction + evidence array that was consumed during initial build. + +**Flow B (update):** +- User answer enters as `answer` parameter in orchestrator, flows through LLM prompt, becomes part of a proposal with no provenance metadata on nodes or edges +- No separation between "user supplied this text" and "model proposed these graph changes" anywhere in the update pipeline +- The `answer` reaches `applyValidatedProposal` (line 2723) and is used for reasoning state derivation (line 2875), but no provenance field is added to nodes/edges + +### EvidenceIds Clarification Conclusion + +The node-level evidenceIds does NOT represent "the list of nodes from which this information was derived." Instead: + +- `evidenceIds` is a **reference list** — each string in the array is an ID that references a specific record in the upstream evidence array +- The distinction between supplied and inferred information lives in the **evidenceType field of those evidence records**, not on the node itself +- A node's kind/status fields encode epistemic classification (what role does this node play and how confident are we), NOT provenance (where did this information come from) +- To recover whether a piece of information was user-supplied or model-inferred, you must look up each evidenceId in the original evidence array and check its evidenceType field — which means **provenance recovery depends on access to the upstream evidence data, not on the graph state alone** + +This is actually useful: it clarifies that provenance IS recoverable from validated reconstruction output (the reconstructed data includes an evidence array where each record has evidenceType), but it is NOT recoverable from graph state alone. The `evidenceIds` array is a bridge to upstream provenance, not provenance itself. + +### Key Findings + +1. **Supplied-vs-inferred provenance EXISTS upstream.** `evidenceRecordSchema` (lib/reconstruction/schema.js line 116-121) has an explicit enum: direct_observation, reported_statement (supplied categories) vs interpretation, assumption, inferred_relationship (inferred categories). This is the most important finding. + +2. **First provenance loss point is `buildInitialGraph()`.** In lib/graph/builder.js lines 61-70, the evidenceMap is built but dead-coded — never queried. Only ID strings are added to node.evidenceIds without type metadata. + +3. **Update flow has no provenance preservation.** The user's answer arrives as a parameter but becomes embedded in an LLM prompt with no traceability. No node receives source metadata during updates. + +4. **Provenance recovery requires upstream data, not graph state.** Since the SituationGraph schema has no provenance fields and nodes only carry ID references to evidence records, any provenance determination must reference the original reconstruction or update evidence array — it cannot be derived from the graph alone. + +### Experiment Conclusion + +The supplied-versus-inferred distinction is fully preserved in the upstream reconstruction/update pipeline output (the validated evidence arrays carry explicit evidenceType values for each record). However, this distinction is never encoded into the SituationGraph nodes during either initial build or update application. The evidenceIds field on nodes provides indirect access to provenance via ID references, but only if the original evidence data remains available downstream of the graph. + +The core insight: provenance is not lost from the pipeline — it is preserved in the reconstruction output that feeds the builder. It IS lost when the builder converts that output into a SituationGraph because the node schema has no field to receive it. This means a fix would add a provenance field to the node schema and propagate evidenceType through buildInitialGraph and applyValidatedProposal. + +### Limitations + +- Inspection-based; no live model run required +- Traced production paths only (builder.js, apply-proposal.js, update-proposal.js) +- Did not examine LLM prompt templates to determine if they preserve answer-supplied vs inference distinction in output formatting +- The analysis assumes evidence records remain accessible after graph construction — downstream usage patterns were not audited + +### Status + +**Pending Rob's review.** Tracing complete. No production code changed. Working tree clean before commit. + +### Production Unchanged + +- `lib/reconstruction/schema.js`: 0 lines changed +- `lib/graph/builder.js`: 0 lines changed +- `lib/graph/apply-proposal.js`: 0 lines changed +- `lib/graph/orchestrator.js`: 0 lines changed +- No production files modified +- Working tree clean before commit + +### Tests / Validation Run + +No test run required — this is a source-trace audit confirming data flow paths, not a behavioral test. + +### Documentation Updated + +- `docs/current-handoff.md` — Return-to-Work Note updated with 54B findings +- `docs/design-evolution-log.md` — Experiment 54B section appended + + +### Report 54B — Trace Provenance Loss Through Graph Pipeline + +#### Experiment Purpose + +Trace where the supplied-versus-inferred provenance distinction is lost in both the initial graph build and update flows, determine whether it is recoverable at any point in the pipeline, and document what future work must do to address the gap. + +#### Method + +File-by-file source inspection of the complete data flow for both paths: Flow A (analyseScenario → buildInitialGraph → SituationGraph) and Flow B (updateCase → LLM prompt → parseGraphUpdateProposal → applyValidatedProposal). + +#### Provenance Existence Upstream + +Yes. The evidenceRecordSchema in lib/reconstruction/schema.js lines 116-121 defines an explicit enum: direct_observation, reported_statement, interpretation, assumption, inferred_relationship. Supplied categories (direct_observation, reported_statement) are separate from inferred categories (interpretation, assumption, inferred_relationship). This distinction is preserved in the validated reconstruction output returned by analyseScenario and consumed by buildInitialGraph. + +#### First Provenance Loss Point + +- Flow A (initial graph): lib/graph/builder.js lines 61-70. The evidenceMap is built at line 62 from all evidence records but never queried after construction. Only the raw ID string (e.g., obs.id) is added to node.evidenceIds via addEvidenceToNode — no type metadata or provenance classification is transferred to the node schema, which has no provenance field defined in lib/graph/schema.js lines 55-70. + +- Flow B (update): lib/graph/orchestrator.js where updateCase passes the user answer into an LLM prompt without separating "user-supplied" from "model-inferred" content, and lib/graph/update-proposal.js where graphUpdateSchema builds addedNodes using situationNodeSchema which has no provenance field. The answer parameter reaches applyValidatedProposal (lib/graph/apply-proposal.js line 2723) and is used in deriveReasoningStateOverride (line 2875), but no provenance metadata is attached to nodes or edges during the update application. + +#### EvidenceIds Clarification Conclusion + +The node-level evidenceIds field does not represent "the list of nodes from which this information was derived." Instead, each string in evidenceIds is a reference ID that points to a specific record in the upstream evidence array. The supplied-versus-inferred distinction lives on those upstream records (their evidenceType enum field), not on the node itself. To recover whether a piece of information was user-supplied or model-inferred requires accessing the original reconstruction or update evidence array and checking each referenced record's evidenceType — provenance recovery therefore depends on access to upstream data, not on the graph state alone. + +#### Update Flow Answer Handling + +The user answer enters orchestrator.updateCase() as a parameter, gets embedded in an LLM prompt without source attribution markers, and the resulting proposal carries no provenance metadata onto nodes or edges. The answer survives as a raw string through to applyValidatedProposal (lib/graph/apply-proposal.js line 2723) where it influences reasoning state derivation (line 2875), but no node receives any indication that it was derived from user-supplied content versus model inference. + +#### Provenance Recovery Feasibility + +From graph alone: No — the SituationGraph schema has no provenance fields on nodes or edges, and evidenceIds only carries ID references without type metadata. From upstream data: Yes — the validated reconstruction output (returned by analyseScenario) includes a complete evidence array where each record has an explicit evidenceType enum field distinguishing supplied from inferred information. + +#### Future Work Recommendation + +Create — not merely preserve. Adding a provenance field to the SituationNode schema and propagating evidenceType through both buildInitialGraph (lib/graph/builder.js) and applyValidatedProposal (lib/graph/apply-proposal.js) is required so that provenance is embedded in the graph state itself rather than remaining recoverable only from external upstream data. The existing evidenceIds reference system should be retained as a cross-reference layer, but the critical gap is the absence of a node-level provenance field. + +### Status +**Committed.** Report appended to design log. Branch: `feature/user-workspace-ux-v0.7`. Working tree clean before commit. +