experiment: test semantic grounding of interpretations
This commit is contained in:
@@ -6271,3 +6271,151 @@ No. The experiment explicitly excludes next-question derivation.
|
||||
### Status
|
||||
|
||||
**Pending Rob's review.** No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/reconstruction/interpretation-source-grounding.test.js`.
|
||||
|
||||
|
||||
## Experiment 54K — Can the Model Automatically Separate Source-Supported Meaning From Interpretation-Added Meaning? (2026-08-07)
|
||||
|
||||
### Objective
|
||||
|
||||
Take the representation proved in Experiment 54J and test the smallest missing capability:
|
||||
|
||||
> **Given an exact user source and one interpretation of it, can the configured semantic model identify which meaning is supported by the source and which meaning was added by the interpretation?**
|
||||
|
||||
Experiment 54J used human-fixed grounding references.
|
||||
Experiment 54K tests whether that grounding distinction can be produced semantically without changing production behaviour.
|
||||
|
||||
This is a passive test-only experiment.
|
||||
|
||||
### Hypothesis
|
||||
|
||||
Given an exact verbatim source and one interpretation, the model may be able to separate:
|
||||
1. what the source actually supports;
|
||||
2. what the interpretation adds beyond the source.
|
||||
|
||||
If it can do this without strengthening, weakening, or rewriting the source meaning, then automated grounding is plausible enough for further investigation.
|
||||
|
||||
### Context Used
|
||||
|
||||
- `docs/current-handoff.md` (Experiment 54J findings and Return-to-Work Note);
|
||||
- Experiment 54J only in `docs/design-evolution-log.md`;
|
||||
- Existing `tests/reconstruction/interpretation-source-grounding.test.js` to confirm the grounding record shape remains valid;
|
||||
- Existing `.env.local` configuration (same Ollama host and model).
|
||||
|
||||
### Configured Host and Model
|
||||
|
||||
- **Ollama host:** `http://192.168.1.111:11434` (unchanged from production);
|
||||
- **Model:** `qwen-claude:latest` (unchanged from production).
|
||||
|
||||
### Semantic Output Contract
|
||||
|
||||
Each call receives `{ source, interpretation }` and returns exactly:
|
||||
```json
|
||||
{
|
||||
"supportedBySource": ["short factual statements"],
|
||||
"addedByInterpretation": ["short factual statements"]
|
||||
}
|
||||
```
|
||||
No confidence. No scores. No explanation field. No chain-of-thought.
|
||||
|
||||
### Semantic Instruction (identical for all cases)
|
||||
|
||||
> Compare the interpretation with the exact source text. Put only meaning directly supported by the source into `supportedBySource`. Put meaning introduced, strengthened, narrowed, or otherwise added by the interpretation into `addedByInterpretation`. Do not treat a plausible inference as source-supported merely because it is reasonable.
|
||||
|
||||
### Three Fixed Cases — Inputs and Semantic Outputs
|
||||
|
||||
#### Case 1 — Interpretation A
|
||||
|
||||
**Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
|
||||
**Interpretation:** "Pricing may be contributing materially to the revenue decline."
|
||||
|
||||
**Semantic output (live model):**
|
||||
- `supportedBySource`: ["Revenue has declined.", "Pricing might be a factor in the problem.", "The speaker expresses uncertainty about pricing's role."]
|
||||
- `addedByInterpretation`: ["Pricing has a significant or substantial impact on the revenue decline.", "The explicit expression of uncertainty is removed, strengthening the claim."]
|
||||
|
||||
**Human-fixed reference:** supported = [revenue is down, pricing may be part of the problem]; added = [pricing may be contributing materially to the decline]
|
||||
|
||||
**Classification:** grounding_correct
|
||||
|
||||
#### Case 2 — Interpretation B
|
||||
|
||||
**Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
|
||||
**Interpretation:** "The revenue decline may have causes other than pricing, and pricing has not yet been established as the main problem."
|
||||
|
||||
**Semantic output (live model):**
|
||||
- `supportedBySource`: ["Revenue has declined.", "Pricing might be one contributing factor to the decline.", "The speaker is unsure whether pricing is a factor."]
|
||||
- `addedByInterpretation`: ["Pricing has not yet been established as the main problem of the revenue decline."]
|
||||
|
||||
**Human-fixed reference:** supported = [revenue is down, pricing may be part of the problem, the user is unsure]; added = [there may be causes other than pricing, pricing has not yet been established as the main problem]
|
||||
|
||||
**Classification:** partial_grounding
|
||||
The model correctly captured "not established as main problem" on the added side. However, it did not capture "alternative causes besides pricing" on the added side — this is a genuine missed addition for Interpretation B, since alternative causality is a core element of that interpretation. The supportedBySource content was all correct including uncertainty.
|
||||
|
||||
#### Case 3 — Fully Grounded Control
|
||||
|
||||
**Source:** "Revenue is down. I think pricing may be part of the problem, but I am not sure."
|
||||
**Interpretation:** "Revenue is down, and the user thinks pricing may be part of the problem but is unsure."
|
||||
|
||||
**Semantic output (live model):**
|
||||
- `supportedBySource`: ["Revenue is down.", "Pricing may be part of the problem.", "The speaker expresses uncertainty about whether pricing is contributing to the issue."]
|
||||
- `addedByInterpretation`: []
|
||||
|
||||
**Human-fixed reference:** supported = [revenue is down, pricing may be part of the problem, the user is unsure]; added = []
|
||||
|
||||
**Classification:** grounding_correct
|
||||
The model produced no material additions for a faithful restatement. The output is an empty array on the added side, confirming it did not invent distinctions where none exist.
|
||||
|
||||
### Required Questions — Answers
|
||||
|
||||
1. **Did Case 1 correctly keep `materially` on the interpretation-added side?** Yes. The model placed "significant or substantial" strengthening in `addedByInterpretation` and kept it out of `supportedBySource`.
|
||||
2. **Did Case 2 distinguish user uncertainty from the more specific interpretation layered onto it?** Partially. It correctly kept uncertainty in supportedBySource but failed to capture one of two additions (alternative causes) on the added side.
|
||||
3. **Did Case 3 correctly produce no material added meaning?** Yes. Empty `addedByInterpretation` array for a faithful restatement.
|
||||
4. **Did any plausible inference get incorrectly promoted into `supportedBySource`?** No. The model did not promote any interpretation-specific content into the supported side across any case.
|
||||
5. **Did any genuinely source-supported meaning get incorrectly treated as interpretation-added?** No. All three cases retained their core source-supported content in the supported side. The most notable was uncertainty — the model correctly identified it as source-supported (even in Case 1 where my reference didn't include it).
|
||||
6. **How many of three cases were:** grounding_correct = 2, partial_grounding = 1, grounding_failed = 0.
|
||||
7. **Does the result suggest automated semantic grounding is plausible enough for further testing?** Yes, with caution. The model kept strengthening (materially) on the correct side in every case and did not promote interpretation content into source-supported territory. One missed addition (alternative causes) suggests occasional under-detection of additions but no false positives on the critical dimension.
|
||||
8. **Does this experiment establish which interpretation is better?** No. It does not select or score interpretations.
|
||||
9. **Does it establish what question should be asked next?** No. That remains untested in this experiment.
|
||||
|
||||
### Grounding Summary
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Grounding-correct count | 2 |
|
||||
| Partial-grounding count | 1 |
|
||||
| Grounding-failed count | 0 |
|
||||
| Interpretation-added meaning leaked into supportedBySource | No |
|
||||
| Source-supported meaning pushed into addedByInterpretation | No |
|
||||
| Fully grounded control avoided invented additions | Yes |
|
||||
|
||||
### Evidence for Automated Semantic Grounding
|
||||
|
||||
The model correctly separated strengthening ("materially" → "significant or substantial") from source meaning in every case. No interpretation-specific content leaked into supportedBySource. The fully grounded control produced an empty added array. One partial result (Case 2) missed one of two expected additions but preserved all three pieces of source-supported content.
|
||||
|
||||
### Inference Timing
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Number of live inference calls | 3 |
|
||||
| Total inference time | 96,372.37ms |
|
||||
| Average | 32,124.12ms per call |
|
||||
| Fastest | 25,575.87ms (Case 3) |
|
||||
| Slowest | 41,532.40ms (Case 2) |
|
||||
|
||||
Timing is observational only. All three cases required ~25–42 seconds of model inference time on this host/model.
|
||||
|
||||
### Limitations
|
||||
|
||||
- Single source text tested across all three cases — no cross-domain validation;
|
||||
- Only one interpretation per source tested in each case — no multi-interpretation comparison in a single call;
|
||||
- Model was `qwen-claude:latest` on host `192.168.1.111` — results may differ with other models or hosts;
|
||||
- The partially correct Case 2 still captured the core supported content — the gap was in added-content completeness, not source-meaning accuracy;
|
||||
- Evaluation used structured meaning checks (not keyword matching) but remains a heuristic approximation of semantic comparison;
|
||||
- No repeated sampling — results reflect a single run per case.
|
||||
|
||||
### Evaluation Conclusion
|
||||
|
||||
**Semantic grounding is promising but imperfect.** The model reliably kept interpretation-specific strengthening out of supportedBySource across all cases, which is the critical signal. One missed addition in Case 2 and occasional paraphrase mismatches suggest the approach needs refinement before production use, but the direction is viable.
|
||||
|
||||
### Status
|
||||
|
||||
**Pending Rob's review.** No production code changed. No schemas modified. No active engine behaviour changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/reconstruction/semantic-interpretation-grounding.test.js`.
|
||||
|
||||
Reference in New Issue
Block a user