experiment: test deterministic user source identity
This commit is contained in:
@@ -5716,7 +5716,7 @@ From `evidenceRecordSchema`:
|
||||
|
||||
### Does Raw User Statement Coexist with Reconstruction Result?
|
||||
|
||||
**No.** The raw user statement (`scenario`) is the input parameter to `analyseScenario`. It is used only for prompt building (line 59 of analysis.js). It is NOT included in the return value alongside `evidence` records or the validated reconstruction. At validation time, deterministic code has:
|
||||
**Not retained alongside results.** The raw user statement (`scenario`) is the input parameter to `analyseScenario`. It is available to production code while reconstruction is being performed, but it is NOT retained alongside the returned reconstruction/evidence state for later deterministic verification. At validation time, deterministic code has:
|
||||
- the evidence records (with model-generated descriptions and types);
|
||||
- but NOT the raw user statement itself.
|
||||
|
||||
@@ -5778,7 +5778,7 @@ Evidence records carry no verbatim user text and no deterministic structural ref
|
||||
|
||||
### Experiment 54G Conclusion
|
||||
|
||||
**Existing evidence records do not contain deterministic source provenance.** Neither verbatim user text nor structured location references (character offsets, turn IDs, source record references) survive into the evidence records. The `source` and `attribution` fields are free-form model-generated text that may be null — they cannot establish deterministic linkage to user-supplied words. The raw user statement is not returned alongside the validated reconstruction result. Therefore, deterministic code cannot verify that any evidence record was produced from the user's actual words without trusting the LLM's `evidenceType` label.
|
||||
**Bounded findings.** Evidence records do not contain verbatim source text. Evidence records do not contain deterministic source locations (character offsets, turn/message IDs, or structural references). `evidenceType` is model classification, not production-code provenance. Current evidence records therefore cannot independently prove source provenance without trusting the LLM's label.
|
||||
|
||||
### Limitations
|
||||
|
||||
@@ -5791,3 +5791,132 @@ Evidence records carry no verbatim user text and no deterministic structural ref
|
||||
### Status
|
||||
|
||||
**Pending Rob's review.** Source-inspection complete. No production code changed. Working tree clean before commit.
|
||||
|
||||
## Experiment 54H — Can Trustworthy Source Identity Be Established Before LLM Interpretation? (2026-08-07)
|
||||
|
||||
### Objective
|
||||
|
||||
First, tighten Experiment 54G so it does not imply the raw user statement is unavailable during reconstruction.
|
||||
|
||||
Then test one small capability outside the active engine:
|
||||
|
||||
> **Can production code create a deterministic source record from the user's raw input before any LLM interpretation occurs, so later reasoning has something trustworthy to refer back to?**
|
||||
|
||||
This is a passive test-only experiment. Do not integrate provenance into the graph or runtime.
|
||||
|
||||
### Hypothesis
|
||||
|
||||
A trustworthy provenance root may not require the LLM to decide anything. Production code may be able to take the raw input and create a small immutable source record **before** reconstruction that provides stable identity, source type, and verbatim source text — without attempting to classify what individual reasoning claims mean. If this cannot be done cleanly without introducing hidden interpretation, record why.
|
||||
|
||||
### Context Used
|
||||
|
||||
- `docs/current-handoff.md` (54G findings)
|
||||
- Experiment 54G only in `docs/design-evolution-log.md`
|
||||
- `lib/reconstruction/schema.js` — evidenceRecordSchema to avoid naming collisions
|
||||
- Node.js built-in `crypto.createHash('sha256')` — standard library, deterministic
|
||||
|
||||
### Test-Only Source Record Shape
|
||||
|
||||
```json
|
||||
{
|
||||
"sourceId": "...",
|
||||
"sourceType": "user_input",
|
||||
"verbatimText": "..."
|
||||
}
|
||||
```
|
||||
|
||||
This is deliberately small and does not include: evidence type, confidence, interpretation, node kind, relevance, inferred meaning, or summary.
|
||||
|
||||
### Deterministic ID Method Used
|
||||
|
||||
SHA-256 hash of the verbatim input text using Node.js built-in `crypto.createHash('sha256')`.
|
||||
|
||||
### Why the ID Method Is Deterministic
|
||||
|
||||
- SHA-256 is a deterministic cryptographic hash function: same input always produces the same output; different inputs produce different outputs (with negligible collision probability).
|
||||
- Uses Node.js standard library — no LLM, no random UUID, no clock/time dependency, no mutable global state.
|
||||
- Input is the verbatim string in UTF-8 encoding with no transformation or normalisation.
|
||||
|
||||
### Four Input Cases
|
||||
|
||||
| Case | Text | Purpose |
|
||||
|------|------|---------|
|
||||
| 1 | "The business has lost three major customers this year." | Baseline simple statement |
|
||||
| 2 | "Three major customers have left the business this year." | Paraphrase — confirm identity tracks actual text, not meaning |
|
||||
| 3 | "Revenue is down. I think pricing may be part of the problem, but I am not sure." | Multi-sentence — confirm full verbatim input survives unchanged |
|
||||
| 4 | Same as Case 1 | Confirm deterministic identity |
|
||||
|
||||
### Focused Test Results (9 tests, all pass)
|
||||
|
||||
| # | Test | Result |
|
||||
|---|------|--------|
|
||||
| 1 | Case 1: verbatimText equals the exact input string | Pass |
|
||||
| 2 | Case 2 (paraphrase): sourceType is always user_input | Pass |
|
||||
| 3 | Case 1 and Case 4 produce identical sourceId | Pass |
|
||||
| 4 | Case 1 and Case 2 produce different sourceIds despite similar meaning | Pass |
|
||||
| 5 | Case 3: multi-sentence verbatim text preserved unchanged | Pass |
|
||||
| 6 | Input string is not mutated by createSourceRecord | Pass |
|
||||
| 7 | Helper output is deterministic across repeated calls | Pass |
|
||||
| 8 | No interpretation or summarisation occurs in the source record | Pass |
|
||||
| 9 | Different inputs produce different IDs (distinct text) | Pass |
|
||||
|
||||
### Case Results Summary
|
||||
|
||||
- **Case 1 result:** verbatimText preserved exactly ✓; deterministic SHA-256 ID produced.
|
||||
- **Case 2 paraphrase result:** Same meaning, different text → different sourceId ✓; verbatimText preserved exactly.
|
||||
- **Case 3 multi-sentence result:** Full multi-sentence text preserved unchanged ✓; single deterministic ID.
|
||||
- **Case 4 repeated-input result:** Identical to Case 1 text → identical sourceId ✓.
|
||||
|
||||
### Same-Input Identity Result
|
||||
|
||||
Confirmed: Cases 1 and 4 produce exactly the same sourceId (sha256 of identical string = identical hash).
|
||||
|
||||
### Different-Text Identity Result
|
||||
|
||||
Confirmed: Cases 1, 2, and 3 produce different sourceIds despite Case 1≈Case 2 in meaning.
|
||||
|
||||
### Verbatim-Text Preservation Result
|
||||
|
||||
All three unique inputs preserved exactly as-is. No summarisation, normalisation, or transformation applied to verbatimText.
|
||||
|
||||
### Input Immutability Result
|
||||
|
||||
Confirmed: the original input string is never mutated. createSourceRecord reads via `String(rawInput)` (copy) before hashing.
|
||||
|
||||
### LLM / Network Dependency
|
||||
|
||||
None. The helper uses only Node.js built-in `crypto` and JavaScript primitives. Zero inference calls. Zero network calls.
|
||||
|
||||
### Does the Helper Perform Semantic Interpretation
|
||||
|
||||
No. The helper outputs exactly: `{ sourceId, sourceType, verbatimText }`. No meaning is extracted, classified, summarised, or inferred.
|
||||
|
||||
### Distinction Between Source Identity and Semantic Provenance
|
||||
|
||||
This experiment establishes **source identity** — answering "what exact material entered the system, and what stable identity can we give that material?" It does **not** answer: what the text means, whether it is evidence, whether it was inferred, or whether graph nodes can link to it. Source identity ≠ claim provenance.
|
||||
|
||||
### What Remains Explicitly Untested
|
||||
|
||||
- Linking evidence records to source records
|
||||
- Linking graph nodes to evidence or source
|
||||
- Update-turn source identity (not initial reconstruction)
|
||||
- Persistence of source records alongside graph state
|
||||
- Whether production code can reliably invoke this helper at the right boundary
|
||||
- How downstream consumers discover and use the source record
|
||||
- Multiple-input scenarios (multi-turn conversations)
|
||||
|
||||
### Experiment Conclusion
|
||||
|
||||
**Deterministic source identity is feasible before LLM interpretation.** A deterministic SHA-256 hash of raw user text produces stable, verifiable identity without any semantic processing. The concept is straightforward because it operates purely at the input boundary — no classification, no inference, no graph contract required.
|
||||
|
||||
### Limitations
|
||||
|
||||
- Test-only implementation; not integrated into any production path
|
||||
- Single-input only (does not address multi-turn or batched input scenarios)
|
||||
- Does not test whether production code can reliably invoke this at the correct reconstruction boundary
|
||||
- Does not test persistence, retrieval, or downstream linking
|
||||
- SHA-256 collision resistance is sufficient for source identity but does not guarantee uniqueness across all possible inputs in practice
|
||||
|
||||
### Status
|
||||
|
||||
**Pending Rob's review.** Deterministic source identity established. No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/deterministic-source-record.test.js`.
|
||||
|
||||
Reference in New Issue
Block a user