experiment: test interpretation lineage to user source
This commit is contained in:
@@ -5919,4 +5919,162 @@ This experiment establishes **source identity** — answering "what exact materi
|
||||
|
||||
### Status
|
||||
|
||||
**Pending Rob's review.** Deterministic source identity established. No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/deterministic-source-record.test.js`.
|
||||
**Closed.** Bounded conclusion: deterministic source identity is feasible before LLM interpretation; identical tested text produces stable identity; source identity does not establish claim or graph provenance. No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/deterministic-source-record.test.js`.
|
||||
|
||||
## Experiment 54I — Can Two Different Interpretations Stay Anchored to the Same User Source? (2026-08-07)
|
||||
|
||||
### Objective
|
||||
|
||||
Take the deterministic source identity proved in Experiment 54H and test the next smallest step:
|
||||
|
||||
> **Can two different interpretations of the exact same user input remain separate while both retaining an explicit link back to that same source?**
|
||||
|
||||
This experiment deliberately does not decide which interpretation is better. It tests whether the reasoning system could preserve one source, multiple interpretations, and clear lineage back to that source without silently replacing the original input or collapsing the interpretations together. This is a passive, test-only experiment.
|
||||
|
||||
### Hypothesis
|
||||
|
||||
A very small lineage structure may be enough to demonstrate that:
|
||||
- one source identity can anchor multiple interpretations;
|
||||
- interpretations remain separately identifiable;
|
||||
- neither interpretation changes or replaces the source;
|
||||
- a later consumer can deterministically see that both interpretations came from the same exact input.
|
||||
|
||||
If even this cannot be represented cleanly without semantic ambiguity, record that honestly.
|
||||
|
||||
### Context Used
|
||||
|
||||
- `docs/current-handoff.md` (Experiment 54H findings and Return-to-Work Note)
|
||||
- Experiment 54H only in `docs/design-evolution-log.md`
|
||||
- `tests/reconstruction/deterministic-source-record.test.js` (to verify the source-identity helper remains intact)
|
||||
- Node.js built-in `crypto.createHash('sha256')` — deterministic, no LLM
|
||||
|
||||
### Fixed Source
|
||||
|
||||
Raw user input:
|
||||
> Revenue is down. I think pricing may be part of the problem, but I am not sure.
|
||||
|
||||
One deterministic source record created from that exact text using the Experiment 54H method (SHA-256 of verbatim text).
|
||||
|
||||
### Two Fixed Interpretations
|
||||
|
||||
**Interpretation A:**
|
||||
> Pricing may be contributing materially to the revenue decline.
|
||||
|
||||
**Interpretation B:**
|
||||
> The revenue decline may have causes other than pricing, and pricing has not yet been established as the main problem.
|
||||
|
||||
Both are plausible readings of the source. The experiment does not claim either is correct.
|
||||
|
||||
### Interpretation Identity Method
|
||||
|
||||
Each `interpretationId` is generated deterministically from:
|
||||
- the shared `sourceId`;
|
||||
- plus the exact interpretation text (combined as concatenation of `sourceId + "|" + interpretationText`).
|
||||
|
||||
Same source + same interpretation → same interpretation ID.
|
||||
Same source + different interpretation → different interpretation ID.
|
||||
Interpretation ID changes if interpretation text changes.
|
||||
No random UUID, no clock, no LLM, no mutable global state.
|
||||
|
||||
### Interpretation Record Shape (Test-Only)
|
||||
|
||||
```json
|
||||
{
|
||||
"interpretationId": "...",
|
||||
"sourceId": "...",
|
||||
"interpretationText": "..."
|
||||
}
|
||||
```
|
||||
|
||||
This is **not** a proposed production schema. It does not include: confidence, evidence type, status, scores, timestamps, model names, node kinds, or next-question information. The purpose is only identity and lineage.
|
||||
|
||||
### Focused Test Results (15 tests, all pass)
|
||||
|
||||
| # | Test | Result |
|
||||
|---|------|--------|
|
||||
| 1 | Source retains exact verbatim user input | Pass |
|
||||
| 2 | Interpretation A references the source's exact sourceId | Pass |
|
||||
| 3 | Interpretation B references the same exact sourceId | Pass |
|
||||
| 4 | A and B have different interpretationId values | Pass |
|
||||
| 5 | Recreating A produces exactly the same interpretationId | Pass |
|
||||
| 6 | Recreating B produces exactly the same interpretationId | Pass |
|
||||
| 7 | Changing interpretation text changes the interpretation ID | Pass |
|
||||
| 8 | Neither interpretation mutates the source record | Pass |
|
||||
| 9 | Creating one interpretation does not mutate the other | Pass |
|
||||
| 10 | Deterministic consumer identifies one shared source and two distinct interpretations | Pass |
|
||||
| 11 | No semantic judgement chooses A over B | Pass |
|
||||
| 12 | No LLM or network call occurs | Pass |
|
||||
| Cross-check | Different source + same text → different interpretationId | Pass |
|
||||
| Immutability | Source unchanged after multiple interpretation creations | Pass |
|
||||
| Lineage traceability | Consumer traces both interpretations to exact verbatim source | Pass |
|
||||
|
||||
### Shared Source Lineage Result
|
||||
|
||||
Confirmed: both Interpretation A and Interpretation B reference the same `sourceId`. A deterministic consumer can recover that exactly one source anchors both interpretations.
|
||||
|
||||
### Distinct Interpretation Identity Result
|
||||
|
||||
Confirmed: Interpretation A and Interpretation B have different `interpretationId` values despite sharing a source. The IDs remain stable across recreation and diverge when text changes.
|
||||
|
||||
### Source Immutability Result
|
||||
|
||||
Confirmed: creating one or multiple interpretations from a source does not mutate the source record in any way (fields, structure, or content remain identical).
|
||||
|
||||
### Interpretation Independence Result
|
||||
|
||||
Confirmed: creating an interpretation for A does not affect B's identity, and vice versa. Each interpretation is independently derived from the shared sourceId + its own text.
|
||||
|
||||
### Deterministic Consumer Recovery
|
||||
|
||||
Confirmed: a consumer given the source record and both interpretation records can deterministically identify (1) one shared source, (2) two distinct interpretations, and (3) that both interpretations trace back to the same exact verbatim input.
|
||||
|
||||
### Was Either Interpretation Selected as More Correct?
|
||||
|
||||
No. The experiment does not select, score, or prefer either interpretation.
|
||||
|
||||
### Was Downstream Question Selection Tested?
|
||||
|
||||
No. The experiment explicitly excludes next-question derivation.
|
||||
|
||||
### What This Experiment Establishes
|
||||
|
||||
- One source can deterministically anchor multiple interpretations.
|
||||
- Those interpretations remain independently identifiable via stable interpretationId values.
|
||||
- A later consumer can determine that two interpretations came from the same exact source.
|
||||
- Preserving multiple interpretations does not require changing the original source record.
|
||||
- The distinction between "the user supplied this" and "interpretation A/B says this may mean X/Y" survives representation cleanly in code.
|
||||
|
||||
### What This Experiment Does Not Establish
|
||||
|
||||
- Which interpretation is more justified or better grounded.
|
||||
- Whether different interpretations would lead to different next questions.
|
||||
- Any semantic correctness claim about either interpretation.
|
||||
- Graph integration, persistence, multi-turn history, or production architecture.
|
||||
|
||||
### Explicitly Untested
|
||||
|
||||
- Deciding which interpretation is better grounded.
|
||||
- Comparing interpretations against verbatim source wording.
|
||||
- Comparing interpretations against other evidence.
|
||||
- Contradictions between interpretations.
|
||||
- Whether disagreement should reduce confidence.
|
||||
- Whether disagreement should trigger clarification.
|
||||
- Downstream question selection.
|
||||
- Graph integration.
|
||||
- Persistence.
|
||||
- Multi-turn interpretation history.
|
||||
|
||||
### Limitations
|
||||
|
||||
- Test-only implementation; not integrated into any production path.
|
||||
- Only two fixed interpretations tested (not a general multi-interpretation protocol).
|
||||
- No semantic analysis of whether either interpretation faithfully represents the source.
|
||||
- The `|` separator in interpretationId derivation assumes the separator does not appear in user text; this is adequate for identity stability but would need review if adopted as production code.
|
||||
|
||||
### Experiment Conclusion
|
||||
|
||||
**Multiple interpretations can retain deterministic lineage to one source.** One source identity successfully anchors two distinct interpretations while both remain separately identifiable and neither mutates the original source. The distinction between unchanged user evidence and changing model interpretation remains visible in the data structure.
|
||||
|
||||
### Status
|
||||
|
||||
**Pending Rob's review.** No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/source-interpretation-lineage.test.js`.
|
||||
|
||||
Reference in New Issue
Block a user