experiment: audit referential provenance
This commit is contained in:
@@ -73,7 +73,7 @@ Experiment 52C separated free-language semantic understanding from enum normalis
|
||||
|
||||
Experiment 52D isolated enum normalisation from semantic understanding: five fixed meaning statements (no decision target or question in the input) were mapped to the existing four-category contract via one live model call each. Four of five normalised to the expected enum. The compliance boundary case persisted — the model classified a "supports" relationship as `could_change_decision`, exposing genuine ambiguity between these two categories under the current definitions. The existing contract appears clear enough for a separate normalisation step; the remaining problem lies in category definitions, not semantic understanding or normalisation mechanism. Same Qwen model (`qwen-claude:latest`) and host (`http://192.168.1.111:11434`) were retained throughout. No production behaviour changed. What remains uncertain: whether the `supports_decision` ↔ `could_change_decision` boundary can be clarified without restructuring the contract, and whether the discrepancy holds under repeated runs. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-normalisation.test.js` for results.
|
||||
|
||||
Experiment 53 proved semantic separation of supplied meaning from possible inference is achievable. Experiment 54A confirmed the SituationGraph cannot recover provenance from graph state alone. Experiment 54B traced supplied-versus-inferred distinction upstream to evidenceRecordSchema but found it lost at buildInitialGraph because the node schema has no provenance field. Experiment 54C inspected the normal answer-update boundary: whole-input origin is explicit (answer = user supplied; proposal = model produced) but per-node provenance inside the proposal is not deterministically recoverable from the validated proposal alone. Experiment 54D audited the production update prompt: it clearly separates the user answer (## User Answer section) and instructions, so prompt-level source identity is explicit; however the proposed output schema has no provenance fields on nodes or edges, so per-node provenance at output level is absent — the gap is a schema deficiency, not a prompt-design problem.
|
||||
Experiment 53 proved semantic separation of supplied meaning from possible inference is achievable. Experiment 54A confirmed the SituationGraph cannot recover provenance from graph state alone. Experiment 54B traced supplied-versus-inferred distinction upstream to evidenceRecordSchema but found it lost at buildInitialGraph because the node schema has no provenance field. Experiment 54C inspected the normal answer-update boundary: whole-input origin is explicit (answer = user supplied; proposal = model produced) but per-node provenance inside the proposal is not deterministically recoverable from the validated proposal alone. Experiment 54D audited the production update prompt: it clearly separates the user answer (## User Answer section) and instructions, so prompt-level source identity is explicit; however the proposed output schema has no provenance fields on nodes or edges, so per-node provenance at output level is absent — the tested prompt already preserves user-source identity clearly; the blocking gap identified here is that the validated proposal does not carry per-node provenance forward. The eventual representation remains undecided. Experiment 54E audited whether existing evidence IDs and evidence records could preserve provenance referentially without a new node field: evidence records carry sufficient identity (id + evidenceType) but are consumed during startCase and never returned alongside graph state, so the reference chain breaks — existing provenance exists but referential linkage is incomplete.
|
||||
|
||||
## 5. What Remains Open
|
||||
|
||||
@@ -127,8 +127,8 @@ Answer before continuing:
|
||||
|
||||
---
|
||||
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54D. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54E. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
|
||||
### Return-to-Work Note (Experiment 53 → 54A → 54B → 54C → 54D)
|
||||
### Return-to-Work Note (Experiment 53 → 54A → 54B → 54C → 54D → 54E)
|
||||
|
||||
Experiment 54C clarified that whole-input origin remains deterministically knowable before application (answer = user supplied, proposal = model produced) but per-node provenance is lost because the validated proposal schema carries no per-node origin markers. Experiment 54D audited the production update prompt in lib/graph/prompt-builder.js and confirmed the user answer is explicitly identifiable to the model via a distinct named section (## User Answer). However, the requested proposal output schema has no provenance fields on nodes or edges, so supplied-versus-inferred origin cannot be preserved in the output. No production code changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: lib/graph/schema.js (graphUpdateSchema and situationNodeSchema) to evaluate minimal schema additions for per-node provenance fields.
|
||||
Experiment 53 separated supplied meaning from inference in semantic output. Experiments 54A–54D located where provenance disappears from durable graph representation, confirming the prompt preserves user-source identity but the proposal schema carries no per-node provenance. Experiment 54E tested whether existing evidence IDs and evidence records could preserve provenance referentially without assuming a new node field. The reference chain works only during initial reconstruction: evidence records carry sufficient identity (id + evidenceType) at that point, but they are consumed during startCase and never returned with graph state — no persistence layer retains them. No production code changed. Branch: feature/user-workspace-ux-v0.7. First file to inspect when resuming: lib/graph/orchestrator.js to evaluate where evidence records should be returned alongside case state so the referential chain does not break.
|
||||
|
||||
Reference in New Issue
Block a user