experiment: audit referential provenance

This commit is contained in:
2026-08-07 14:39:11 +01:00
parent 90472de766
commit 23f02d6169
2 changed files with 105 additions and 5 deletions
+4 -4
View File
@@ -73,7 +73,7 @@ Experiment 52C separated free-language semantic understanding from enum normalis
Experiment 52D isolated enum normalisation from semantic understanding: five fixed meaning statements (no decision target or question in the input) were mapped to the existing four-category contract via one live model call each. Four of five normalised to the expected enum. The compliance boundary case persisted — the model classified a "supports" relationship as `could_change_decision`, exposing genuine ambiguity between these two categories under the current definitions. The existing contract appears clear enough for a separate normalisation step; the remaining problem lies in category definitions, not semantic understanding or normalisation mechanism. Same Qwen model (`qwen-claude:latest`) and host (`http://192.168.1.111:11434`) were retained throughout. No production behaviour changed. What remains uncertain: whether the `supports_decision``could_change_decision` boundary can be clarified without restructuring the contract, and whether the discrepancy holds under repeated runs. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-normalisation.test.js` for results.
Experiment 53 proved semantic separation of supplied meaning from possible inference is achievable. Experiment 54A confirmed the SituationGraph cannot recover provenance from graph state alone. Experiment 54B traced supplied-versus-inferred distinction upstream to evidenceRecordSchema but found it lost at buildInitialGraph because the node schema has no provenance field. Experiment 54C inspected the normal answer-update boundary: whole-input origin is explicit (answer = user supplied; proposal = model produced) but per-node provenance inside the proposal is not deterministically recoverable from the validated proposal alone. Experiment 54D audited the production update prompt: it clearly separates the user answer (## User Answer section) and instructions, so prompt-level source identity is explicit; however the proposed output schema has no provenance fields on nodes or edges, so per-node provenance at output level is absent — the gap is a schema deficiency, not a prompt-design problem.
Experiment 53 proved semantic separation of supplied meaning from possible inference is achievable. Experiment 54A confirmed the SituationGraph cannot recover provenance from graph state alone. Experiment 54B traced supplied-versus-inferred distinction upstream to evidenceRecordSchema but found it lost at buildInitialGraph because the node schema has no provenance field. Experiment 54C inspected the normal answer-update boundary: whole-input origin is explicit (answer = user supplied; proposal = model produced) but per-node provenance inside the proposal is not deterministically recoverable from the validated proposal alone. Experiment 54D audited the production update prompt: it clearly separates the user answer (## User Answer section) and instructions, so prompt-level source identity is explicit; however the proposed output schema has no provenance fields on nodes or edges, so per-node provenance at output level is absent — the tested prompt already preserves user-source identity clearly; the blocking gap identified here is that the validated proposal does not carry per-node provenance forward. The eventual representation remains undecided. Experiment 54E audited whether existing evidence IDs and evidence records could preserve provenance referentially without a new node field: evidence records carry sufficient identity (id + evidenceType) but are consumed during startCase and never returned alongside graph state, so the reference chain breaks — existing provenance exists but referential linkage is incomplete.
## 5. What Remains Open
@@ -127,8 +127,8 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 3853, 54A54D. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 3853, 54A54E. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 53 → 54A → 54B → 54C → 54D)
### Return-to-Work Note (Experiment 53 → 54A → 54B → 54C → 54D → 54E)
Experiment 54C clarified that whole-input origin remains deterministically knowable before application (answer = user supplied, proposal = model produced) but per-node provenance is lost because the validated proposal schema carries no per-node origin markers. Experiment 54D audited the production update prompt in lib/graph/prompt-builder.js and confirmed the user answer is explicitly identifiable to the model via a distinct named section (## User Answer). However, the requested proposal output schema has no provenance fields on nodes or edges, so supplied-versus-inferred origin cannot be preserved in the output. No production code changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: lib/graph/schema.js (graphUpdateSchema and situationNodeSchema) to evaluate minimal schema additions for per-node provenance fields.
Experiment 53 separated supplied meaning from inference in semantic output. Experiments 54A54D located where provenance disappears from durable graph representation, confirming the prompt preserves user-source identity but the proposal schema carries no per-node provenance. Experiment 54E tested whether existing evidence IDs and evidence records could preserve provenance referentially without assuming a new node field. The reference chain works only during initial reconstruction: evidence records carry sufficient identity (id + evidenceType) at that point, but they are consumed during startCase and never returned with graph state — no persistence layer retains them. No production code changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: lib/graph/orchestrator.js to evaluate where evidence records should be returned alongside case state so the referential chain does not break.