experiment: test semantic grounding of interpretations
This commit is contained in:
@@ -77,6 +77,12 @@ Experiment 54H tested whether trustworthy source identity can begin deterministi
|
||||
|
||||
Experiment 53 proved semantic separation of supplied meaning from possible inference is achievable. Experiment 54A confirmed the SituationGraph cannot recover provenance from graph state alone. Experiment 54B traced supplied-versus-inferred distinction upstream to evidenceRecordSchema but found it lost at buildInitialGraph because the node schema has no provenance field. Experiment 54C inspected the normal answer-update boundary: whole-input origin is explicit (answer = user supplied; proposal = model produced) but per-node provenance inside the proposal is not deterministically recoverable from the validated proposal alone. Experiment 54D audited the production update prompt: it clearly separates the user answer (## User Answer section) and instructions, so prompt-level source identity is explicit; however the proposed output schema has no provenance fields on nodes or edges, so per-node provenance at output level is absent — the tested prompt already preserves user-source identity clearly; the blocking gap identified here is that the validated proposal does not carry per-node provenance forward. The eventual representation remains undecided. Experiment 54E audited whether existing evidence IDs and evidence records could preserve provenance referentially without a new node field: the evidence-record schema contains vocabulary capable of distinguishing supplied-like from inferred-like material, but the reference chain breaks because (1) evidence records are consumed during startCase and never returned alongside graph state — no persistence layer retains them; and (2) no evidence records are created or retained during update cycles. Experiment 54E did not validate how those values are assigned in production. Experiment 54F audited evidenceType assignment: the reconstruction prompt instructs the LLM to classify each evidence item into one of five types based on its own judgment; no production code deterministically derives evidenceType from source origin — even reported_statement means "the model thinks this looks like a reported statement" not "production code knows this came directly from the user." Experiment 54G audited whether evidence records nevertheless retain deterministic linkage to user words: neither verbatim text nor structured location references (character offsets, turn IDs) survive in any record field; `source` and `attribution` are free-form model-generated strings that may be null; the raw user statement is available to production code while reconstruction is being performed but is not retained alongside the returned reconstruction/evidence state for later deterministic verification. Evidence records do not contain verbatim source text or deterministic source locations; `evidenceType` is model classification, not trustworthy provenance. Current evidence records therefore cannot independently prove source provenance.
|
||||
|
||||
Experiment 54I showed multiple interpretations can share one deterministic source lineage via the Experiment 54H SHA-256 method. Both branches stayed traceable to the same source while remaining distinct in their reported additions. No interpretation was selected as better and no numeric scoring occurred. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/source-interpretation-lineage.test.js`.
|
||||
|
||||
Experiment 54J proved the representation can separate source-supported from interpretation-added meaning using human-fixed references (13 tests, all pass). Grounding references were human-fixed; automated grounding remained untested. No production code or schemas changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/interpretation-source-grounding.test.js`.
|
||||
|
||||
Experiment 54K tested whether the configured semantic model (`qwen-claude:latest` on `192.168.1.111:11434`) can perform that grounding automatically. Three live Ollama calls (total ~96s): Case 1 (strengthening detection) = grounding_correct, Case 2 (multi-addition interpretation) = partial_grounding (missed one addition), Case 3 (faithful restatement control) = grounding_correct. Interpretation-added meaning did NOT leak into source-supported meaning in any case. One source-supported content gap: model missed "alternative causes" on the added side of Case 2. Automated semantic grounding is promising but imperfect — directionally viable but needs refinement before production use. Winner selection and downstream questions remain untested. No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-interpretation-grounding.test.js`.
|
||||
|
||||
## 5. What Remains Open
|
||||
|
||||
- The `too_broad` boundary sits exactly between three and four active unknowns; it is mechanically clear but conceptually uncertain — whether it aligns with genuine user confusion requires real-scenario validation;
|
||||
@@ -129,8 +135,8 @@ Answer before continuing:
|
||||
|
||||
---
|
||||
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54J. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54K. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
|
||||
### Return-to-Work Note (Experiment 54J)
|
||||
### Return-to-Work Note (Experiment 54K)
|
||||
|
||||
Experiment 54I showed multiple interpretations can share one deterministic source lineage; Experiment 54J tested whether each interpretation can separately expose what is source-supported versus added by interpretation using a fixed grounding-record shape. Both branches stayed traceable to the same source while remaining distinct in their reported additions. No interpretation was selected as better and no numeric scoring occurred. Automated grounding and downstream-question effects remain untested. No production code or schemas changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/interpretation-source-grounding.test.js`.
|
||||
Experiment 54J proved the representation can separate source-supported from interpretation-added meaning using human-fixed references. Experiment 54K tested whether the semantic model can perform that grounding automatically: two of three cases were grounding_correct, one was partial_grounding (model missed one addition — "alternative causes" — on the added side). Interpretation-added meaning did not leak into source-supported meaning in any case. Winner selection and downstream questions remain untested. Same Ollama host (`192.168.1.111:11434`) and model (`qwen-claude:latest`) retained; no production behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-interpretation-grounding.test.js`.
|
||||
|
||||
Reference in New Issue
Block a user