experiment: test qualified evidence uncertainty status
This commit is contained in:
@@ -0,0 +1,171 @@
|
||||
# Experiment 58B.1 — Qualified Evidence Without Weakening Uncertainty
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Date:** 2026-08-12
|
||||
**Status:** Complete
|
||||
**Previous context:** Follows 58A.2 which exposed the core problem — status shift from `unknown` to `provisional` when evidence arrives but resolution remains open.
|
||||
|
||||
---
|
||||
|
||||
## Objective
|
||||
|
||||
When the user provides a concrete £2m figure and explicitly states it is unverified, does the engine preserve the figure **and** keep the existing savings-realism uncertainty unresolved?
|
||||
|
||||
This isolates the status decision from 58A.2's broader failure modes.
|
||||
|
||||
---
|
||||
|
||||
## Fixed Starting Graph
|
||||
|
||||
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
|
||||
```
|
||||
id: n_savings_realism
|
||||
label: Are the projected office savings from relocation realistic?
|
||||
kind: unknown
|
||||
status: unknown
|
||||
value: null
|
||||
confidence: low
|
||||
dependsOn: [n_relocation_state]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Fixed Answer
|
||||
|
||||
> The projected saving is about £2 million per year, but that figure is still unverified and I am not yet confident it is realistic.
|
||||
|
||||
Three components:
|
||||
1. **Supported information:** approximately £2 million per year
|
||||
2. **Explicit qualification:** figure is unverified
|
||||
3. **Continued uncertainty:** user not yet confident the estimate is realistic
|
||||
|
||||
No lease assumption, no disruption-cost assumption, no new competing unknown.
|
||||
|
||||
---
|
||||
|
||||
## Execution
|
||||
|
||||
```bash
|
||||
FIXTURE_MODE=updateOnly \
|
||||
ANSWER_2="The projected saving is about £2 million per year, but that figure is still unverified and I am not yet confident it is realistic." \
|
||||
CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \
|
||||
node scripts/reproduce-multi-turn-investigation.mjs
|
||||
```
|
||||
|
||||
**Host/model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
**Calls:** startCalls=0, updateCalls=1, totalCalls=1
|
||||
**Retries:** 0
|
||||
|
||||
---
|
||||
|
||||
## Results (direct API inspection)
|
||||
|
||||
### Answer meaning (inside updatedProposal.answerMeaning)
|
||||
|
||||
```json
|
||||
{
|
||||
"userSupportedMeaning": "The projected savings figure is £2 million per year, but it remains unverified and the user lacks confidence in its realism.",
|
||||
"possibleInference": null,
|
||||
"supportCategory": "uncertain",
|
||||
"resolutionGuidance": "may_resolve"
|
||||
}
|
||||
```
|
||||
|
||||
### Structural proposal
|
||||
|
||||
- `updatedNodes`: `[{nodeId: n_savings_realism, previousStatus: unknown, newStatus: unknown, previousValue: null, newValue: "£2M/year (unverified)"}]`
|
||||
- `resolvedUnknownNodeIds`: `[]`
|
||||
- `addedNodes`: `[]`
|
||||
- `addedEdges`: `[]`
|
||||
- `structuralActionRequired`: true
|
||||
|
||||
### Persistent graph after update
|
||||
|
||||
```
|
||||
node: id=n_relocation_state, kind=state, label=Engineering team relocation consideration, status=provisional
|
||||
node: id=n_savings_realism, kind=unknown, label=Are the projected office savings from relocation realistic?, status=unknown, confidence=low, value="£2M/year (unverified)"
|
||||
edge: n_savings_realism → n_relocation_state (depends_on)
|
||||
```
|
||||
|
||||
### Selected question
|
||||
|
||||
- Question: `"are the projected office savings from relocation realistic?"`
|
||||
- nodeId: `n_savings_realism`
|
||||
|
||||
---
|
||||
|
||||
## Assessment
|
||||
|
||||
| Criterion | Result | Classification |
|
||||
|-----------|--------|----------------|
|
||||
| £2m figure | `"£2M/year (unverified)"` on n_savings_realism | PRESERVED AS QUALIFIED |
|
||||
| Existing uncertainty status | `unknown` → `unknown` (unchanged) | UNCERTAINTY PRESERVED |
|
||||
| Qualification | "(unverified)" suffix in value | PRESERVED |
|
||||
| Identity of savings-realism node | Single node, unchanged id, no duplicates | ONE IDENTITY PRESERVED |
|
||||
| Next investigation | Question targets the unresolved realism question | GOOD |
|
||||
|
||||
### Answer meaning analysis
|
||||
|
||||
- **userSupportedMeaning** correctly captured all three answer components: figure amount + unverified status + user doubt
|
||||
- **possibleInference** = null — did not invent assumptions about lease, disruption, or cost structure
|
||||
- **supportCategory** = `"uncertain"` — semantically correct for qualified evidence
|
||||
- **resolutionGuidance** = `"may_resolve"` — correctly reflects that the uncertainty remains open
|
||||
|
||||
---
|
||||
|
||||
## Classification: A — QUALIFIED EVIDENCE AND UNCERTAINTY BOTH PRESERVED
|
||||
|
||||
- £2m survives as `"£2M/year (unverified)"` with explicit qualification
|
||||
- n_savings_realism stays `kind=unknown / status=unknown` — identity and unresolved nature both preserved
|
||||
- Confidence set to `low` — appropriate for unverified evidence on an uncertainty node
|
||||
- Answer meaning supportCategory = `"uncertain"` — correct semantic interpretation
|
||||
- No duplicate nodes, no resolved unknown nodes
|
||||
- Selected question continues investigating the realism concern
|
||||
|
||||
### Does the graph still clearly represent realism as unresolved?
|
||||
|
||||
**YES.** The node kind remains `unknown`, status remains `unknown`, and value contains the explicit qualification "(unverified)". Confidence is `low`. There are zero `resolvedUnknownNodeIds`. An interrogative selectedQuestion pointing to this same node confirms ongoing investigation targeting.
|
||||
|
||||
---
|
||||
|
||||
## What the engine understood correctly
|
||||
|
||||
1. **Evidence preservation:** Extracted the £2M/year figure from prose and stored it on the existing uncertainty node rather than discarding or inventing a new node.
|
||||
2. **Qualification embedding:** The value includes "(unverified)" — the model did not strip the qualification when storing evidence.
|
||||
3. **Semantic category:** Labeled supportCategory as `"uncertain"` rather than `"strong"` or `"established"`.
|
||||
4. **No fabrication:** possibleInference was null — no invented lease, disruption, or cost assumptions.
|
||||
5. **Status stability:** Status remained `unknown` (not shifted to `provisional`) — unlike 58A.2 where this was the core failure.
|
||||
6. **Open resolution:** Did not resolve n_savings_realism; resolutionGuidance = `"may_resolve"` correctly reflects the ongoing need for verification.
|
||||
7. **Question continuity:** Selected question re-targets the existing node's label rather than inventing a new uncertainty.
|
||||
|
||||
## What it overstated, weakened, or lost
|
||||
|
||||
**Nothing significant.** The update was fully correct for the constraints of this case. One minor note: `structuralActionRequired` is `true` despite no structural change (no new/removed nodes or edges). This flag means "a follow-up structural action may be needed" but does not indicate a failure — it is a forward-looking directive, not a description of what was done wrong.
|
||||
|
||||
---
|
||||
|
||||
## What this establishes
|
||||
|
||||
1. The engine **can** preserve a concrete figure with its qualification when the existing uncertainty node remains the appropriate target.
|
||||
2. Status `unknown` is stable across updates even when value transitions from `null` to a qualified string — unlike the 58A.2 failure path.
|
||||
3. Qualification embedded in `newValue` (e.g., `"£2M/year (unverified)"`) survives as persistent evidence that realism remains unconfirmed.
|
||||
4. Answer meaning extraction (`userSupportedMeaning`, `supportCategory: uncertain`, `resolutionGuidance: may_resolve`) aligns correctly with the user's actual semantics.
|
||||
|
||||
## What this does NOT prove
|
||||
|
||||
1. This is a **single controlled case** — one answer, one model invocation. It does not prove stability across different answers or models.
|
||||
2. It does not test whether `structuralActionRequired=true` with no structural change causes issues in subsequent turns.
|
||||
3. It does not test what happens when the user's qualification changes (e.g., from "unverified" to "verified").
|
||||
4. It does not test interaction with other uncertainty nodes (58A.1's scenario where multiple unknowns compete).
|
||||
5. Value format `"£2M/year (unverified)"` uses prose — whether numeric `2000000` would work equally well is untested here.
|
||||
|
||||
---
|
||||
|
||||
## Production code changed: NO
|
||||
## Prompt changed: NO
|
||||
## Validator changed: NO
|
||||
## Harness changed: NO
|
||||
## Vitest run: NO
|
||||
## Ollama calls beyond harness count: 0
|
||||
## Dev server disturbed: NO
|
||||
Reference in New Issue
Block a user