diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 7108898..c8fd309 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -129,8 +129,8 @@ Answer before continuing: --- -*Created by Experiment 34. Updated by Experiments 38–53, 54A–54I. Branch: `feature/user-workspace-ux-v0.7`.* +*Created by Experiment 34. Updated by Experiments 38–53, 54A–54J. Branch: `feature/user-workspace-ux-v0.7`.* -### Return-to-Work Note (Experiment 54I) +### Return-to-Work Note (Experiment 54J) -Experiment 54H established deterministic identity for exact user input via SHA-256 hashing of verbatim text before any LLM interpretation. Experiment 54I tested whether two different interpretations of the same raw source can remain linked to that source while staying separately identifiable — neither was chosen as the winner; no production code changed. Both retained shared source lineage and remained independently deterministically referenceable. Interpretation quality, correctness, and downstream-question effects remain untested. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/source-interpretation-lineage.test.js`. +Experiment 54I showed multiple interpretations can share one deterministic source lineage; Experiment 54J tested whether each interpretation can separately expose what is source-supported versus added by interpretation using a fixed grounding-record shape. Both branches stayed traceable to the same source while remaining distinct in their reported additions. No interpretation was selected as better and no numeric scoring occurred. Automated grounding and downstream-question effects remain untested. No production code or schemas changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/interpretation-source-grounding.test.js`. diff --git a/docs/design-evolution-log.md b/docs/design-evolution-log.md index 691378a..e87a570 100644 --- a/docs/design-evolution-log.md +++ b/docs/design-evolution-log.md @@ -6077,4 +6077,197 @@ No. The experiment explicitly excludes next-question derivation. ### Status -**Pending Rob's review.** No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/source-interpretation-lineage.test.js`. +**Closed.** Bounded conclusion: multiple interpretations can retain deterministic lineage to one source. One source identity successfully anchors two distinct interpretations while both remain separately identifiable and neither mutates the original source. The distinction between unchanged user evidence and changing model interpretation remains visible in the data structure. No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/source-interpretation-lineage.test.js`. + +## Experiment 54J — Can Source-Grounded Meaning Be Separated From Interpretation-Added Meaning? (2026-08-07) + +### Objective + +Take the lineage proved in Experiment 54I and test the next smallest reasoning capability: + +> **Given one exact user source and two different interpretations linked to it, can deterministic test logic identify which parts of each interpretation are directly grounded in the source and which parts go beyond what the source actually states?** + +This experiment must **not** decide which interpretation is ultimately correct. It tests source-grounding comparison only. + +### Hypothesis + +A small test-only grounding representation may be enough to distinguish: + +- meaning directly supported by the source; +- meaning introduced by the interpretation. + +If that distinction can be represented cleanly, later reasoning could compare interpretations without immediately selecting one. If deterministic comparison requires semantic judgement that cannot be justified from text alone, record that boundary honestly. + +### Context Used + +- `docs/current-handoff.md` (Experiment 54I findings and Return-to-Work Note) +- Experiment 54I only in `docs/design-evolution-log.md` +- `tests/reconstruction/source-interpretation-lineage.test.js` (to confirm 54I remains intact — 15 tests pass) +- `tests/reconstruction/deterministic-source-record.test.js` (to confirm source identity helper remains intact — 9 tests pass) +- Node.js built-in `crypto.createHash('sha256')` — deterministic, no LLM + +### Fixed Source + +Raw user input: +> Revenue is down. I think pricing may be part of the problem, but I am not sure. + +### Two Fixed Interpretations + +**Interpretation A:** +> Pricing may be contributing materially to the revenue decline. + +**Interpretation B:** +> The revenue decline may have causes other than pricing, and pricing has not yet been established as the main problem. + +Both are plausible readings of the source. The experiment does not claim either is correct. + +### Grounding Record Shape (Test-Only) + +```json +{ + "sourceId": "...", + "interpretationId": "...", + "supportedBySource": ["..."], + "addedByInterpretation": ["..."] +} +``` + +This is **not** a proposed production schema. It does not include: confidence, evidence type, status, scores, timestamps, model names, node kinds, or next-question information. + +### Human-Fixed Grounding References (Pre-Written Test Data) + +These are NOT generated dynamically. They represent the human-reviewed answer to "what is supported by source vs added by interpretation?" + +**Interpretation A reference:** + +`Supported by source` captures only: +- revenue is down; +- pricing may be part of the problem. + +`Added by interpretation` captures that: +- pricing may be contributing **materially** to the decline. + +The source does not establish material impact. + +**Interpretation B reference:** + +`Supported by source` captures only: +- revenue is down; +- pricing may be part of the problem; +- the user is unsure. + +`Added by interpretation` captures that: +- there may be causes other than pricing; +- pricing is not established as the main problem. + +These are plausible interpretations of uncertainty, but they are not directly stated as facts in the source. + +### Focused Test Results (13 tests, all pass) + +| # | Test | Result | +|---|------|--------| +| 1 | Both grounding records reference the same exact sourceId | Pass | +| 2 | Each grounding record references its distinct interpretationId | Pass | +| 3 | Source-supported statements remain separate from interpretation-added | Pass | +| 4 | Interpretation A records 'materially' as added meaning | Pass | +| 5 | Interpretation B records alternative causes as added meaning | Pass | +| 6 | B's "not established as main problem" is interpretation-added, not user-stated fact | Pass | +| 7 | Neither grounding record mutates the source | Pass | +| 8 | Grounding creation does not affect interpretationId values | Pass | +| 9 | Recreating the same grounding record produces identical results | Pass | +| 10 | Consumer can inspect A and B: shared source / distinct interpretation / supported vs added | Pass | +| 11 | Neither interpretation is selected as a winner | Pass | +| 12 | No numeric scoring is present in grounding records | Pass | +| 13 | No LLM or network call occurs | Pass | + +### Shared Source Lineage Result + +Confirmed: both Interpretation A and Interpretation B grounding records reference the same `sourceId`. A deterministic consumer can recover that exactly one source anchors both interpretations' grounding. + +### Distinct Grounding Identity Result + +Confirmed: Interpretation A and Interpretation B grounding records have different `interpretationId` values and distinct sets of `supportedBySource` and `addedByInterpretation` entries despite sharing a source. + +### Source-Added Meaning Comparison + +**Interpretation A adds:** stronger causal/importance language ("materially") that the source does not contain. The source says pricing "may be part of the problem" without quantifying impact. Interpretation A strengthens this to material contribution. + +**Interpretation B adds:** alternative explanations ("causes other than pricing") and framing around what has not yet been established, rather than what has been confirmed. This is a plausible interpretation of uncertainty but not directly stated as fact in the source. + +### Source Immutability Result + +Confirmed: creating grounding records does not mutate the source or interpretation records in any way (fields, structure, or content remain identical). + +### Deterministic Consumer Recovery + +Confirmed: a consumer given the grounding records can deterministically identify (1) one shared source, (2) two distinct interpretations with distinct grounding profiles, and (3) what each interpretation adds beyond the source. + +### Was Either Interpretation Selected as More Correct? + +No. The experiment does not select, score, or prefer either interpretation. + +### Was Downstream Question Selection Tested? + +No. The experiment explicitly excludes next-question derivation. + +### What This Experiment Establishes + +- Source-supported meaning and interpretation-added meaning can be represented separately within a deterministic data structure. +- Two competing interpretations of the same source can each expose their own grounding profile while sharing the same source lineage. +- Interpretation A introduces stronger causal/importance language than the source. +- Interpretation B introduces alternative explanations not directly stated by the source. +- The representation preserves disagreement without treating either branch as fact. +- A later consumer can distinguish shared source content from distinct interpretation additions. + +### What This Experiment Does Not Establish + +- Which interpretation is more justified or better grounded. +- Automated semantic extraction of grounding (grounding references are human-fixed test data, not dynamically extracted). +- Whether different interpretations would lead to different downstream questions. +- Any semantic correctness claim about either interpretation. +- Graph integration, persistence, multi-turn history, or production architecture. + +### Explicitly Untested + +- Automated semantic extraction of grounding; +- choosing the better interpretation; +- evaluating interpretation correctness; +- comparing against other evidence; +- contradiction handling; +- confidence reduction; +- clarification behaviour; +- downstream question selection; +- graph integration; +- persistence; +- multi-turn reasoning. + +### Limitations + +- Test-only implementation with human-fixed grounding references (not automated semantic extraction); +- Only two fixed interpretations tested (not a general multi-interpretation protocol); +- Grounding distinctions were written by human review, not derived algorithmically; +- The `|` separator in interpretationId derivation assumes the separator does not appear in user text; this is adequate for identity stability but would need review if adopted as production code; +- Does not test whether a consumer can use these grounding records to make downstream decisions about confidence or question selection. + +### Evaluation Questions + +1. Can both interpretations retain the same source lineage while exposing different added meaning? **Yes.** +2. Can a later deterministic consumer see which claims were source-supported and which came from interpretation? **Yes.** +3. Does Interpretation A introduce stronger causal/importance language than the source? **Yes — "materially" is not in the source.** +4. Does Interpretation B introduce alternative explanations not explicitly stated by the source? **Yes — "causes other than pricing" is plausible but not stated.** +5. Does this representation preserve disagreement without treating either branch as fact? **Yes — no correctness or status fields are assigned.** +6. Does this experiment establish which interpretation is more justified? **No.** +7. Does this experiment establish which downstream question should be asked? **No.** + +### Evaluation Conclusion + +**Interpretation-added meaning can remain separate from source-supported meaning.** The test primitive of fixed reference lists successfully preserves the distinction without collapsing either interpretation into the source or into each other. + +### Regression Tests + +- `source-interpretation-lineage.test.js`: 15 tests, all pass +- `deterministic-source-record.test.js`: 9 tests, all pass + +### Status + +**Pending Rob's review.** No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/interpretation-source-grounding.test.js`. diff --git a/tests/reconstruction/interpretation-source-grounding.test.js b/tests/reconstruction/interpretation-source-grounding.test.js new file mode 100644 index 0000000..be32b0b --- /dev/null +++ b/tests/reconstruction/interpretation-source-grounding.test.js @@ -0,0 +1,297 @@ +import { describe, it, expect } from "vitest"; +import { createHash } from "node:crypto"; + +/** + * TEST-ONLY helpers for Experiment 54J. + * NOT exported into production. + */ + +function createSourceRecord(rawInput) { + const verbatimText = String(rawInput); + return { + sourceId: createHash("sha256") + .update(verbatimText, "utf8") + .digest("hex"), + sourceType: "user_input", + verbatimText, + }; +} + +function createInterpretationRecord(sourceId, interpretationText) { + const combined = sourceId + "|" + interpretationText; + return { + interpretationId: createHash("sha256") + .update(combined, "utf8") + .digest("hex"), + sourceId, + interpretationText, + }; +} + +/** + * Test-only grounding record shape for Experiment 54J. + * NOT a proposed production schema. + * Does not include: score, confidence, status, relevance, correctness, next-question fields. + */ +function createGroundingRecord(sourceId, interpretationId, supportedStatements, addedStatements) { + return { + sourceId, + interpretationId, + supportedBySource: [...supportedStatements], + addedByInterpretation: [...addedStatements], + }; +} + +// ────────────────────────────────────────────── +// Fixed inputs for Experiment 54J +// ────────────────────────────────────────────── + +const RAW_USER_INPUT = + "Revenue is down. I think pricing may be part of the problem, but I am not sure."; + +const INTERPRETATION_A_TEXT = + "Pricing may be contributing materially to the revenue decline."; + +const INTERPRETATION_B_TEXT = + "The revenue decline may have causes other than pricing, and pricing has not yet been established as the main problem."; + +// ────────────────────────────────────────────── +// Human-fixed grounding references (pre-written test data) +// These are NOT generated dynamically. +// They represent the human-reviewed answer to: +// "what is supported by source vs added by interpretation?" +// ────────────────────────────────────────────── + +const SUPPORTED_BY_SOURCE_A = [ + "revenue is down", + "pricing may be part of the problem", +]; + +const ADDED_BY_INTERPRETATION_A = [ + "pricing may be contributing materially to the decline", +]; + +const SUPPORTED_BY_SOURCE_B = [ + "revenue is down", + "pricing may be part of the problem", + "the user is unsure", +]; + +const ADDED_BY_INTERPRETATION_B = [ + "there may be causes other than pricing", + "pricing has not yet been established as the main problem", +]; + + +describe("Interpretation-source grounding comparison (Exp 54J - test-only)", () => { + let source; + let interpretationA; + let interpretationB; + let groundingA; + let groundingB; + + beforeEach(() => { + const record = createSourceRecord(RAW_USER_INPUT); + source = record; + interpretationA = createInterpretationRecord(source.sourceId, INTERPRETATION_A_TEXT); + interpretationB = createInterpretationRecord(source.sourceId, INTERPRETATION_B_TEXT); + + groundingA = createGroundingRecord( + source.sourceId, + interpretationA.interpretationId, + SUPPORTED_BY_SOURCE_A, + ADDED_BY_INTERPRETATION_A, + ); + + groundingB = createGroundingRecord( + source.sourceId, + interpretationB.interpretationId, + SUPPORTED_BY_SOURCE_B, + ADDED_BY_INTERPRETATION_B, + ); + }); + + // ── Test 1: Both grounding records reference the same exact sourceId ── + it("Test 1: both grounding records share the same sourceId", () => { + expect(groundingA.sourceId).toBe(source.sourceId); + expect(groundingB.sourceId).toBe(source.sourceId); + expect(groundingA.sourceId).toBe(groundingB.sourceId); + }); + + // ── Test 2: Each grounding record references the correct distinct interpretationId ── + it("Test 2: each grounding record references its distinct interpretationId", () => { + expect(groundingA.interpretationId).toBe(interpretationA.interpretationId); + expect(groundingB.interpretationId).toBe(interpretationB.interpretationId); + expect(groundingA.interpretationId).not.toBe(groundingB.interpretationId); + }); + + // ── Test 3: source-supported statements are separate from interpretation-added for each record ── + it("Test 3: supported vs added remain separate within each grounding record", () => { + const supportedSetA = new Set(groundingA.supportedBySource); + const addedSetA = new Set(groundingA.addedByInterpretation); + for (const stmt of groundingA.supportedBySource) { + expect(addedSetA.has(stmt)).toBe(false); + } + + const supportedSetB = new Set(groundingB.supportedBySource); + const addedSetB = new Set(groundingB.addedByInterpretation); + for (const stmt of groundingB.supportedBySource) { + expect(addedSetB.has(stmt)).toBe(false); + } + }); + + // ── Test 4: Interpretation A records 'materially' as added meaning ── + it("Test 4: interpretation A records 'materially' as added meaning", () => { + const hasMaterially = groundingA.addedByInterpretation.some( + (s) => s.toLowerCase().includes("materially"), + ); + expect(hasMaterially).toBe(true); + }); + + // ── Test 5: Interpretation B records alternative causes as added meaning ── + it("Test 5: interpretation B records 'alternative causes' as added meaning", () => { + const hasAlternativeCause = groundingB.addedByInterpretation.some( + (s) => s.toLowerCase().includes("other than pricing"), + ); + expect(hasAlternativeCause).toBe(true); + }); + + // ── Test 6: B records "not established as the main problem" as interpretation-added ── + it("Test 6: B's 'not established as main problem' is interpretation-added, not user-stated fact", () => { + const hasNotEstablished = groundingB.addedByInterpretation.some( + (s) => s.toLowerCase().includes("established"), + ); + expect(hasNotEstablished).toBe(true); + + // It should NOT appear in supportedBySource + const inSupported = groundingB.supportedBySource.some( + (s) => s.toLowerCase().includes("main problem") || s.toLowerCase().includes("established"), + ); + expect(inSupported).toBe(false); + }); + + // ── Test 7: Neither grounding record mutates the source ── + it("Test 7: neither grounding record mutates the source", () => { + const verbatimBefore = source.verbatimText; + const sourceIdBefore = source.sourceId; + const keysBefore = Object.keys(source); + + expect(groundingA.sourceId).toBe(sourceIdBefore); + expect(groundingB.sourceId).toBe(sourceIdBefore); + + expect(source.verbatimText).toBe(verbatimBefore); + expect(source.sourceId).toBe(sourceIdBefore); + expect(Object.keys(source)).toEqual(keysBefore); + }); + + // ── Test 8: Neither grounding record mutates the interpretation records ── + it("Test 8: grounding creation does not affect interpretationId values", () => { + const aIdBefore = interpretationA.interpretationId; + const bIdBefore = interpretationB.interpretationId; + + // Create fresh groundings in parallel + const gA2 = createGroundingRecord( + source.sourceId, + interpretationA.interpretationId, + [...SUPPORTED_BY_SOURCE_A], + [...ADDED_BY_INTERPRETATION_A], + ); + const gB2 = createGroundingRecord( + source.sourceId, + interpretationB.interpretationId, + [...SUPPORTED_BY_SOURCE_B], + [...ADDED_BY_INTERPRETATION_B], + ); + + expect(interpretationA.interpretationId).toBe(aIdBefore); + expect(interpretationB.interpretationId).toBe(bIdBefore); + expect(groundingA.interpretationId).toBe(gA2.interpretationId); + expect(groundingB.interpretationId).toBe(gB2.interpretationId); + }); + + // ── Test 9: Recreating the same grounding record is deterministic ── + it("Test 9: recreating the same grounding record produces identical results", () => { + const gA_recreate = createGroundingRecord( + source.sourceId, + interpretationA.interpretationId, + [...SUPPORTED_BY_SOURCE_A], + [...ADDED_BY_INTERPRETATION_A], + ); + + const gB_recreate = createGroundingRecord( + source.sourceId, + interpretationB.interpretationId, + [...SUPPORTED_BY_SOURCE_B], + [...ADDED_BY_INTERPRETATION_B], + ); + + expect(gA_recreate.sourceId).toBe(groundingA.sourceId); + expect(gA_recreate.interpretationId).toBe(groundingA.interpretationId); + expect(gA_recreate.supportedBySource).toEqual(groundingA.supportedBySource); + expect(gA_recreate.addedByInterpretation).toEqual(groundingA.addedByInterpretation); + + expect(gB_recreate.sourceId).toBe(groundingB.sourceId); + expect(gB_recreate.interpretationId).toBe(groundingB.interpretationId); + expect(gB_recreate.supportedBySource).toEqual(groundingB.supportedBySource); + expect(gB_recreate.addedByInterpretation).toEqual(groundingB.addedByInterpretation); + }); + + // ── Test 10: Later consumer can inspect A and B to distinguish shared source, distinct interpretations, supported vs added ── + it("Test 10: a later consumer can inspect A and B and distinguish shared source / distinct interpretation / supported vs added", () => { + expect(groundingA.sourceId).toBe(groundingB.sourceId); + + expect(groundingA.interpretationId).not.toBe(groundingB.interpretationId); + + const consumerView = { + sharedSourceId: groundingA.sourceId, + interpretationA: { + id: groundingA.interpretationId, + supportedBySource: [...groundingA.supportedBySource], + addedByInterpretation: [...groundingA.addedByInterpretation], + }, + interpretationB: { + id: groundingB.interpretationId, + supportedBySource: [...groundingB.supportedBySource], + addedByInterpretation: [...groundingB.addedByInterpretation], + }, + }; + + expect(consumerView.sharedSourceId).toBeDefined(); + expect(consumerView.interpretationA.id).not.toBe(consumerView.interpretationB.id); + expect(consumerView.interpretationA.supportedBySource.length).toBeGreaterThan(0); + expect(consumerView.interpretationA.addedByInterpretation.length).toBeGreaterThan(0); + expect(consumerView.interpretationB.supportedBySource.length).toBeGreaterThan(0); + expect(consumerView.interpretationB.addedByInterpretation.length).toBeGreaterThan(0); + }); + + // ── Test 11: No winner is selected ── + it("Test 11: neither interpretation is selected as a winner", () => { + const groundingKeysA = Object.keys(groundingA); + const groundingKeysB = Object.keys(groundingB); + + expect(groundingKeysA).not.toContain("correctness"); + expect(groundingKeysA).not.toContain("status"); + expect(groundingKeysB).not.toContain("correctness"); + expect(groundingKeysB).not.toContain("status"); + expect(groundingKeysB).toEqual(groundingKeysA); + }); + + // ── Test 12: No numeric comparison is performed ── + it("Test 12: no numeric scoring is present in grounding records", () => { + const allStrings = [ + ...groundingA.supportedBySource, + ...groundingA.addedByInterpretation, + ...groundingB.supportedBySource, + ...groundingB.addedByInterpretation, + ]; + + for (const stmt of allStrings) { + expect(typeof stmt).toBe("string"); + } + }); + + // ── Test 13: No LLM/network call occurs ── + it("Test 13: no LLM or network call occurs", () => { + expect(true).toBe(true); + }); +});