diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 02198fe..cbcd47f 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -124,8 +124,8 @@ Answer before continuing: --- -*Created by Experiment 34. Updated by Experiments 38–52I and 53. Branch: `feature/user-workspace-ux-v0.7`.* +*Created by Experiment 34. Updated by Experiments 38–53, 54A. Branch: `feature/user-workspace-ux-v0.7`.* -### Return-to-Work Note (Experiment 52I) +### Return-to-Work Note (Experiment 53 → 54A) -Experiments 52F–52H established that the model strengthens vague relevance phrasing beyond what was supplied: four of five ambiguous statements became `could_change_decision`, only "connected to" preserved `cannot_determine`. Corrections were applied to 52H's documentation so its conclusions match its evidence. Experiment 52I tested one grounding rule (rather than keyword patches): two clear controls remained correct, three of four ambiguous cases returned `cannot_determine` under grounding, but case 3 ("important to") still became `could_change_decision`. The remaining defect is primarily grounding — the model can still add relationship strength that the supplied meaning did not establish. Same host/model retained; no production behaviour changed; branch: `feature/user-workspace-ux-v0.7` (commit pending). First file to inspect when resuming: `docs/design-evolution-log.md` Experiment 52I section, then `tests/graph/decision-relevance-grounding.test.js`. +Experiment 53 proved semantic separation was possible; 54A inspected whether the current SituationGraph can preserve that distinction from source code alone. The audit confirms the graph's kind/status/confidence fields describe epistemic status only — there is no provenance field recording where information came from. Supplied-versus-inferred provenance is not recoverable from validated graph state. No production code changed; branch: `feature/user-workspace-ux-v0.7`. What remains open: a compatibility layer to bridge the semantic separation proved in 53 with the graph's provenance gap identified in 54A. First file to inspect when resuming: `lib/graph/schema.js` for current node fields, then `lib/graph/apply-proposal.js` for the production update path that populates nodes during inference turns. diff --git a/docs/design-evolution-log.md b/docs/design-evolution-log.md index 5c5ba21..3aa7e7a 100644 --- a/docs/design-evolution-log.md +++ b/docs/design-evolution-log.md @@ -4977,3 +4977,102 @@ Experiment 21 deterministic classifier: zero regressions across all 25 tests. No - `tests/graph/decision-relevance-grounding.test.js` — Exp 52I probe (49 tests, 12 live calls) + +## Experiment 54A — Audit Existing Graph Provenance Only (2026-08-07) + +Experiment 53 showed that the semantic model can keep supplied meaning and possible inference separate in its output. The active graph compatibility question remained unknown. Experiment 54A was an inspection-only experiment to determine whether the current validated SituationGraph distinguishes information supplied by the user or evidence from information inferred by the model. + +### Hypothesis + +The current graph may distinguish known/provisional and supported/unsupported without actually recording where information came from. If true, current graph state can represent epistemic status but not reliably recover supplied-versus-inferred provenance. + +### Files Inspected + +- `docs/current-handoff.md` +- `lib/graph/schema.js` — SituationGraph and node schema definitions +- `lib/graph/builder.js` — production initial graph builder (how nodes are populated from reconstruction) +- `lib/graph/update-proposal.js` — LLM output parsing for graph updates +- `lib/reconstruction/schema.js` — evidenceRecordSchema, reconstructionV2Schema + +### Graph Vocabulary Relevant to Provenance + +**Existing relevant node kinds:** +- `observation`, `reported_claim`, `metric`, `state`, `transition`, `relationship`, `assumption`, `unknown`, `conclusion` + +**Existing relevant status fields:** +- `known`, `unknown`, `provisional`, `supported`, `weakened`, `contradicted`, `resolved` + +**Existing confidence fields:** +- `low`, `medium`, `high` + +**Existing evidence / relationship fields:** +- `evidenceIds`: array of strings (graph reference IDs from reconstruction) +- `dependsOn`: array of node IDs +- `affects`: array of node IDs +- `parentId`: nullable string +- `childIds`: array of node IDs +- Edge types: `supports`, `weakens`, `contradicts`, `depends_on`, `causes`, `may_cause`, `measures`, `compares_with`, `updates`, `other` + +### Explicit Supplied-Information Provenance Exists: No + +No field or combination of fields in the SituationNode schema has documented or implemented meaning that is "this content was supplied by the user or evidence source." The `kind` field distinguishes semantic categories (observation vs assumption vs unknown), not provenance. A node with `kind=assumption` describes what kind of claim it is, not who produced it. + +### Explicit Inferred-Information Provenance Exists: No + +No field or combination has documented or implemented meaning that is "this content was inferred or proposed by the model and is not established evidence." The LLM-inferred nodes flow through `proposal.addedNodes` into the graph with kinds determined by the LLM — but those kinds are semantic labels, not provenance markers. + +### Status Versus Provenance Finding + +Fields like `provisional`, `supported`, `assumption` (as a kind), and `confidence` describe **epistemic status only** — they classify how confident or well-supported a claim is. They do not record where the information originated. A node with `kind=unknown, status=unknown, confidence=low` could have come from user input, model inference, or evidence extraction. + +### Are evidenceIds Provenance or Graph References + +**Graph references.** In `buildInitialGraph`, `evidenceIds` are populated from `obs.id` — IDs that originate from the LLM's reconstruction output (`reconstruction.observedStates[].id`). These are internal identifiers for model-generated evidence records, not user-supplied source identifiers. The same applies during graph updates: node relationships use string IDs that are graph-internal references. + +### Are Inferred Nodes Explicitly Marked as Model-Generated + +**No.** Neither `builder.js` (initial build) nor the update-proposal path marks inferred nodes with any model-generated flag. Node kinds in the update path are set by the LLM's JSON output — there is no explicit "this was model-inferred" marker. + +### Recoverability Result: not_recoverable + +A later consumer receiving only the validated graph (with no conversation history or LLM response) cannot determine which statements came from user/evidence and which were generated as model inference. All nodes produced by different paths (initial build, emergent reasoning, decomposition children) have identical schema shape. The evidenceIds field contains IDs referencing model-generated reconstruction records, not external source identifiers. + +### Production Population Finding + +- No relevant source/provenance fields are populated in production graph-building code +- Node kinds (`observation`, `assumption`, `unknown`, etc.) are used for semantic typing, not provenance +- Evidence IDs are model-generated internal references (not user-supplied identifiers) +- No inferred nodes carry any explicit model-generated marker + +### Experiment Conclusion + +The existing SituationGraph does not preserve supplied-versus-inferred provenance. It represents epistemic status (how confident or well-supported information is) but has no mechanism to record where information originated. This confirms the hypothesis from Experiment 53's open question: while semantic output can separate supplied meaning from inference, the graph layer cannot recover that separation because it lacks provenance tracking fields entirely. + +### Limitations + +- Inspection-based; no live model run was performed +- Only source files directly relevant to node schema and construction were examined +- The non-strict Zod schema allows extra fields but none are used for provenance in production code +- Does not address whether a fix is needed — only whether the gap exists + +### Status + +**Pending Rob's review.** The audit confirms a provenance gap. No production code was changed. Working tree clean before commit. + +### Production Unchanged + +- `lib/graph/schema.js`: 0 lines changed +- `lib/graph/builder.js`: 0 lines changed +- `lib/graph/apply-proposal.js`: 0 lines changed +- `lib/graph/update-proposal.js`: 0 lines changed +- No production files modified +- Working tree clean before commit + +### Tests / Validation Run + +No test run required for the inspection result. Source inspection alone is sufficient — the schema definition in `lib/graph/schema.js` is a static contract, and no runtime execution is needed to confirm the absence of provenance fields. + +### Documentation Updated + +- `docs/current-handoff.md` — handoff line 127 and Return-to-Work Note updated +- `docs/design-evolution-log.md` — Experiment 54A section appended