experiment: test deterministic user source identity
This commit is contained in:
@@ -73,7 +73,9 @@ Experiment 52C separated free-language semantic understanding from enum normalis
|
||||
|
||||
Experiment 52D isolated enum normalisation from semantic understanding: five fixed meaning statements (no decision target or question in the input) were mapped to the existing four-category contract via one live model call each. Four of five normalised to the expected enum. The compliance boundary case persisted — the model classified a "supports" relationship as `could_change_decision`, exposing genuine ambiguity between these two categories under the current definitions. The existing contract appears clear enough for a separate normalisation step; the remaining problem lies in category definitions, not semantic understanding or normalisation mechanism. Same Qwen model (`qwen-claude:latest`) and host (`http://192.168.1.111:11434`) were retained throughout. No production behaviour changed. What remains uncertain: whether the `supports_decision` ↔ `could_change_decision` boundary can be clarified without restructuring the contract, and whether the discrepancy holds under repeated runs. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/graph/decision-relevance-normalisation.test.js` for results.
|
||||
|
||||
Experiment 53 proved semantic separation of supplied meaning from possible inference is achievable. Experiment 54A confirmed the SituationGraph cannot recover provenance from graph state alone. Experiment 54B traced supplied-versus-inferred distinction upstream to evidenceRecordSchema but found it lost at buildInitialGraph because the node schema has no provenance field. Experiment 54C inspected the normal answer-update boundary: whole-input origin is explicit (answer = user supplied; proposal = model produced) but per-node provenance inside the proposal is not deterministically recoverable from the validated proposal alone. Experiment 54D audited the production update prompt: it clearly separates the user answer (## User Answer section) and instructions, so prompt-level source identity is explicit; however the proposed output schema has no provenance fields on nodes or edges, so per-node provenance at output level is absent — the tested prompt already preserves user-source identity clearly; the blocking gap identified here is that the validated proposal does not carry per-node provenance forward. The eventual representation remains undecided. Experiment 54E audited whether existing evidence IDs and evidence records could preserve provenance referentially without a new node field: the evidence-record schema contains vocabulary capable of distinguishing supplied-like from inferred-like material, but the reference chain breaks because (1) evidence records are consumed during startCase and never returned alongside graph state — no persistence layer retains them; and (2) no evidence records are created or retained during update cycles. Experiment 54E did not validate how those values are assigned in production. Experiment 54F audited evidenceType assignment: the reconstruction prompt instructs the LLM to classify each evidence item into one of five types based on its own judgment; no production code deterministically derives evidenceType from source origin — even reported_statement means "the model thinks this looks like a reported statement" not "production code knows this came directly from the user." Experiment 54G audited whether evidence records nevertheless retain deterministic linkage to user words: neither verbatim text nor structured location references (character offsets, turn IDs) survive in any record field; `source` and `attribution` are free-form model-generated strings that may be null; raw user statement is not returned alongside validated results. Existing evidence records do not contain deterministic source provenance.
|
||||
Experiment 54H tested whether trustworthy source identity can begin deterministically from raw user input before any LLM interpretation occurs. A test-only helper `createSourceRecord(rawInput)` hashes the verbatim text with SHA-256 to produce a stable `sourceId`, preserves `verbatimText` unchanged, and sets `sourceType: "user_input"`. Nine focused tests confirm identical inputs produce identical IDs (Case 1 = Case 4), paraphrases produce different IDs (Case 1 ≠ Case 2), and multi-sentence input survives intact (Case 3). No semantic interpretation, summarisation, or LLM call occurs. Trustworthy source identity is feasible before reconstruction — the remaining gap is claim/node provenance and graph linkage, not source identity. Deterministic code can assign stable identity to raw material at the application boundary without any reasoning contract. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/deterministic-source-record.test.js`.
|
||||
|
||||
Experiment 53 proved semantic separation of supplied meaning from possible inference is achievable. Experiment 54A confirmed the SituationGraph cannot recover provenance from graph state alone. Experiment 54B traced supplied-versus-inferred distinction upstream to evidenceRecordSchema but found it lost at buildInitialGraph because the node schema has no provenance field. Experiment 54C inspected the normal answer-update boundary: whole-input origin is explicit (answer = user supplied; proposal = model produced) but per-node provenance inside the proposal is not deterministically recoverable from the validated proposal alone. Experiment 54D audited the production update prompt: it clearly separates the user answer (## User Answer section) and instructions, so prompt-level source identity is explicit; however the proposed output schema has no provenance fields on nodes or edges, so per-node provenance at output level is absent — the tested prompt already preserves user-source identity clearly; the blocking gap identified here is that the validated proposal does not carry per-node provenance forward. The eventual representation remains undecided. Experiment 54E audited whether existing evidence IDs and evidence records could preserve provenance referentially without a new node field: the evidence-record schema contains vocabulary capable of distinguishing supplied-like from inferred-like material, but the reference chain breaks because (1) evidence records are consumed during startCase and never returned alongside graph state — no persistence layer retains them; and (2) no evidence records are created or retained during update cycles. Experiment 54E did not validate how those values are assigned in production. Experiment 54F audited evidenceType assignment: the reconstruction prompt instructs the LLM to classify each evidence item into one of five types based on its own judgment; no production code deterministically derives evidenceType from source origin — even reported_statement means "the model thinks this looks like a reported statement" not "production code knows this came directly from the user." Experiment 54G audited whether evidence records nevertheless retain deterministic linkage to user words: neither verbatim text nor structured location references (character offsets, turn IDs) survive in any record field; `source` and `attribution` are free-form model-generated strings that may be null; the raw user statement is available to production code while reconstruction is being performed but is not retained alongside the returned reconstruction/evidence state for later deterministic verification. Evidence records do not contain verbatim source text or deterministic source locations; `evidenceType` is model classification, not trustworthy provenance. Current evidence records therefore cannot independently prove source provenance.
|
||||
|
||||
## 5. What Remains Open
|
||||
|
||||
@@ -127,8 +129,8 @@ Answer before continuing:
|
||||
|
||||
---
|
||||
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54G. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54H. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
|
||||
### Return-to-Work Note (Experiment 53 → 54A → 54B → 54C → 54D → 54E → 54F → 54G)
|
||||
### Return-to-Work Note (Experiment 54H)
|
||||
|
||||
Experiment 53 separated supplied meaning from inference in semantic output. Experiments 54A–54D located where provenance disappears from durable graph representation, confirming the prompt preserves user-source identity but the proposal schema carries no per-node provenance. Experiment 54E tested whether existing evidence IDs and evidence records could preserve provenance referentially without assuming a new node field — the reference chain breaks because evidence records are consumed during startCase and never returned with graph state. Experiment 54F established that `evidenceType` is model classification, not trustworthy provenance: the LLM classifies each evidence item into five categories based on its own judgment, so reported_statement means "the model thinks this looks like a reported statement" not "production code knows this came from the user." Experiment 54G tested whether evidence records nevertheless retain deterministic linkage to the user's actual words. Neither verbatim text nor structured location references (character offsets, turn IDs, source record IDs) survive in any evidence record field — `source` and `attribution` are free-form model-generated strings that may be null. The raw user statement is not returned with validated results. Deterministic code cannot independently verify evidence records against user input without trusting `evidenceType`. No production code changed. Branch: feature/user-workspace-ux-v0.7. Commit hashes: 54F closure `455d6f4`, 54G `pending`. First file to inspect when resuming: `lib/reconstruction/schema.js` (evidenceRecordSchema) or `prompts/reconstruct-v0.3.md` (evidence output format).
|
||||
Experiments 54F and 54G established that current evidence metadata cannot prove provenance: `evidenceType` is model classification, not production-code origin. Experiment 54H tested whether trustworthy source identity can instead begin deterministically from the raw user input before any LLM interpretation occurs. Verbatim text preservation and stable identity both work — SHA-256 of raw input produces deterministic `sourceId` without semantic processing. Source identity is established; claim provenance and graph-node linkage remain explicitly untested. No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/deterministic-source-record.test.js`.
|
||||
|
||||
Reference in New Issue
Block a user