experiment: separate source support from interpretation additions
This commit is contained in:
@@ -129,8 +129,8 @@ Answer before continuing:
|
||||
|
||||
---
|
||||
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54I. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
*Created by Experiment 34. Updated by Experiments 38–53, 54A–54J. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
|
||||
### Return-to-Work Note (Experiment 54I)
|
||||
### Return-to-Work Note (Experiment 54J)
|
||||
|
||||
Experiment 54H established deterministic identity for exact user input via SHA-256 hashing of verbatim text before any LLM interpretation. Experiment 54I tested whether two different interpretations of the same raw source can remain linked to that source while staying separately identifiable — neither was chosen as the winner; no production code changed. Both retained shared source lineage and remained independently deterministically referenceable. Interpretation quality, correctness, and downstream-question effects remain untested. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/source-interpretation-lineage.test.js`.
|
||||
Experiment 54I showed multiple interpretations can share one deterministic source lineage; Experiment 54J tested whether each interpretation can separately expose what is source-supported versus added by interpretation using a fixed grounding-record shape. Both branches stayed traceable to the same source while remaining distinct in their reported additions. No interpretation was selected as better and no numeric scoring occurred. Automated grounding and downstream-question effects remain untested. No production code or schemas changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/interpretation-source-grounding.test.js`.
|
||||
|
||||
@@ -6077,4 +6077,197 @@ No. The experiment explicitly excludes next-question derivation.
|
||||
|
||||
### Status
|
||||
|
||||
**Pending Rob's review.** No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/source-interpretation-lineage.test.js`.
|
||||
**Closed.** Bounded conclusion: multiple interpretations can retain deterministic lineage to one source. One source identity successfully anchors two distinct interpretations while both remain separately identifiable and neither mutates the original source. The distinction between unchanged user evidence and changing model interpretation remains visible in the data structure. No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/source-interpretation-lineage.test.js`.
|
||||
|
||||
## Experiment 54J — Can Source-Grounded Meaning Be Separated From Interpretation-Added Meaning? (2026-08-07)
|
||||
|
||||
### Objective
|
||||
|
||||
Take the lineage proved in Experiment 54I and test the next smallest reasoning capability:
|
||||
|
||||
> **Given one exact user source and two different interpretations linked to it, can deterministic test logic identify which parts of each interpretation are directly grounded in the source and which parts go beyond what the source actually states?**
|
||||
|
||||
This experiment must **not** decide which interpretation is ultimately correct. It tests source-grounding comparison only.
|
||||
|
||||
### Hypothesis
|
||||
|
||||
A small test-only grounding representation may be enough to distinguish:
|
||||
|
||||
- meaning directly supported by the source;
|
||||
- meaning introduced by the interpretation.
|
||||
|
||||
If that distinction can be represented cleanly, later reasoning could compare interpretations without immediately selecting one. If deterministic comparison requires semantic judgement that cannot be justified from text alone, record that boundary honestly.
|
||||
|
||||
### Context Used
|
||||
|
||||
- `docs/current-handoff.md` (Experiment 54I findings and Return-to-Work Note)
|
||||
- Experiment 54I only in `docs/design-evolution-log.md`
|
||||
- `tests/reconstruction/source-interpretation-lineage.test.js` (to confirm 54I remains intact — 15 tests pass)
|
||||
- `tests/reconstruction/deterministic-source-record.test.js` (to confirm source identity helper remains intact — 9 tests pass)
|
||||
- Node.js built-in `crypto.createHash('sha256')` — deterministic, no LLM
|
||||
|
||||
### Fixed Source
|
||||
|
||||
Raw user input:
|
||||
> Revenue is down. I think pricing may be part of the problem, but I am not sure.
|
||||
|
||||
### Two Fixed Interpretations
|
||||
|
||||
**Interpretation A:**
|
||||
> Pricing may be contributing materially to the revenue decline.
|
||||
|
||||
**Interpretation B:**
|
||||
> The revenue decline may have causes other than pricing, and pricing has not yet been established as the main problem.
|
||||
|
||||
Both are plausible readings of the source. The experiment does not claim either is correct.
|
||||
|
||||
### Grounding Record Shape (Test-Only)
|
||||
|
||||
```json
|
||||
{
|
||||
"sourceId": "...",
|
||||
"interpretationId": "...",
|
||||
"supportedBySource": ["..."],
|
||||
"addedByInterpretation": ["..."]
|
||||
}
|
||||
```
|
||||
|
||||
This is **not** a proposed production schema. It does not include: confidence, evidence type, status, scores, timestamps, model names, node kinds, or next-question information.
|
||||
|
||||
### Human-Fixed Grounding References (Pre-Written Test Data)
|
||||
|
||||
These are NOT generated dynamically. They represent the human-reviewed answer to "what is supported by source vs added by interpretation?"
|
||||
|
||||
**Interpretation A reference:**
|
||||
|
||||
`Supported by source` captures only:
|
||||
- revenue is down;
|
||||
- pricing may be part of the problem.
|
||||
|
||||
`Added by interpretation` captures that:
|
||||
- pricing may be contributing **materially** to the decline.
|
||||
|
||||
The source does not establish material impact.
|
||||
|
||||
**Interpretation B reference:**
|
||||
|
||||
`Supported by source` captures only:
|
||||
- revenue is down;
|
||||
- pricing may be part of the problem;
|
||||
- the user is unsure.
|
||||
|
||||
`Added by interpretation` captures that:
|
||||
- there may be causes other than pricing;
|
||||
- pricing is not established as the main problem.
|
||||
|
||||
These are plausible interpretations of uncertainty, but they are not directly stated as facts in the source.
|
||||
|
||||
### Focused Test Results (13 tests, all pass)
|
||||
|
||||
| # | Test | Result |
|
||||
|---|------|--------|
|
||||
| 1 | Both grounding records reference the same exact sourceId | Pass |
|
||||
| 2 | Each grounding record references its distinct interpretationId | Pass |
|
||||
| 3 | Source-supported statements remain separate from interpretation-added | Pass |
|
||||
| 4 | Interpretation A records 'materially' as added meaning | Pass |
|
||||
| 5 | Interpretation B records alternative causes as added meaning | Pass |
|
||||
| 6 | B's "not established as main problem" is interpretation-added, not user-stated fact | Pass |
|
||||
| 7 | Neither grounding record mutates the source | Pass |
|
||||
| 8 | Grounding creation does not affect interpretationId values | Pass |
|
||||
| 9 | Recreating the same grounding record produces identical results | Pass |
|
||||
| 10 | Consumer can inspect A and B: shared source / distinct interpretation / supported vs added | Pass |
|
||||
| 11 | Neither interpretation is selected as a winner | Pass |
|
||||
| 12 | No numeric scoring is present in grounding records | Pass |
|
||||
| 13 | No LLM or network call occurs | Pass |
|
||||
|
||||
### Shared Source Lineage Result
|
||||
|
||||
Confirmed: both Interpretation A and Interpretation B grounding records reference the same `sourceId`. A deterministic consumer can recover that exactly one source anchors both interpretations' grounding.
|
||||
|
||||
### Distinct Grounding Identity Result
|
||||
|
||||
Confirmed: Interpretation A and Interpretation B grounding records have different `interpretationId` values and distinct sets of `supportedBySource` and `addedByInterpretation` entries despite sharing a source.
|
||||
|
||||
### Source-Added Meaning Comparison
|
||||
|
||||
**Interpretation A adds:** stronger causal/importance language ("materially") that the source does not contain. The source says pricing "may be part of the problem" without quantifying impact. Interpretation A strengthens this to material contribution.
|
||||
|
||||
**Interpretation B adds:** alternative explanations ("causes other than pricing") and framing around what has not yet been established, rather than what has been confirmed. This is a plausible interpretation of uncertainty but not directly stated as fact in the source.
|
||||
|
||||
### Source Immutability Result
|
||||
|
||||
Confirmed: creating grounding records does not mutate the source or interpretation records in any way (fields, structure, or content remain identical).
|
||||
|
||||
### Deterministic Consumer Recovery
|
||||
|
||||
Confirmed: a consumer given the grounding records can deterministically identify (1) one shared source, (2) two distinct interpretations with distinct grounding profiles, and (3) what each interpretation adds beyond the source.
|
||||
|
||||
### Was Either Interpretation Selected as More Correct?
|
||||
|
||||
No. The experiment does not select, score, or prefer either interpretation.
|
||||
|
||||
### Was Downstream Question Selection Tested?
|
||||
|
||||
No. The experiment explicitly excludes next-question derivation.
|
||||
|
||||
### What This Experiment Establishes
|
||||
|
||||
- Source-supported meaning and interpretation-added meaning can be represented separately within a deterministic data structure.
|
||||
- Two competing interpretations of the same source can each expose their own grounding profile while sharing the same source lineage.
|
||||
- Interpretation A introduces stronger causal/importance language than the source.
|
||||
- Interpretation B introduces alternative explanations not directly stated by the source.
|
||||
- The representation preserves disagreement without treating either branch as fact.
|
||||
- A later consumer can distinguish shared source content from distinct interpretation additions.
|
||||
|
||||
### What This Experiment Does Not Establish
|
||||
|
||||
- Which interpretation is more justified or better grounded.
|
||||
- Automated semantic extraction of grounding (grounding references are human-fixed test data, not dynamically extracted).
|
||||
- Whether different interpretations would lead to different downstream questions.
|
||||
- Any semantic correctness claim about either interpretation.
|
||||
- Graph integration, persistence, multi-turn history, or production architecture.
|
||||
|
||||
### Explicitly Untested
|
||||
|
||||
- Automated semantic extraction of grounding;
|
||||
- choosing the better interpretation;
|
||||
- evaluating interpretation correctness;
|
||||
- comparing against other evidence;
|
||||
- contradiction handling;
|
||||
- confidence reduction;
|
||||
- clarification behaviour;
|
||||
- downstream question selection;
|
||||
- graph integration;
|
||||
- persistence;
|
||||
- multi-turn reasoning.
|
||||
|
||||
### Limitations
|
||||
|
||||
- Test-only implementation with human-fixed grounding references (not automated semantic extraction);
|
||||
- Only two fixed interpretations tested (not a general multi-interpretation protocol);
|
||||
- Grounding distinctions were written by human review, not derived algorithmically;
|
||||
- The `|` separator in interpretationId derivation assumes the separator does not appear in user text; this is adequate for identity stability but would need review if adopted as production code;
|
||||
- Does not test whether a consumer can use these grounding records to make downstream decisions about confidence or question selection.
|
||||
|
||||
### Evaluation Questions
|
||||
|
||||
1. Can both interpretations retain the same source lineage while exposing different added meaning? **Yes.**
|
||||
2. Can a later deterministic consumer see which claims were source-supported and which came from interpretation? **Yes.**
|
||||
3. Does Interpretation A introduce stronger causal/importance language than the source? **Yes — "materially" is not in the source.**
|
||||
4. Does Interpretation B introduce alternative explanations not explicitly stated by the source? **Yes — "causes other than pricing" is plausible but not stated.**
|
||||
5. Does this representation preserve disagreement without treating either branch as fact? **Yes — no correctness or status fields are assigned.**
|
||||
6. Does this experiment establish which interpretation is more justified? **No.**
|
||||
7. Does this experiment establish which downstream question should be asked? **No.**
|
||||
|
||||
### Evaluation Conclusion
|
||||
|
||||
**Interpretation-added meaning can remain separate from source-supported meaning.** The test primitive of fixed reference lists successfully preserves the distinction without collapsing either interpretation into the source or into each other.
|
||||
|
||||
### Regression Tests
|
||||
|
||||
- `source-interpretation-lineage.test.js`: 15 tests, all pass
|
||||
- `deterministic-source-record.test.js`: 9 tests, all pass
|
||||
|
||||
### Status
|
||||
|
||||
**Pending Rob's review.** No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/interpretation-source-grounding.test.js`.
|
||||
|
||||
Reference in New Issue
Block a user