confidence-engine/docs/experiment-58b1.md

# Experiment 58B.1 — Qualified Evidence Without Weakening Uncertainty

**Branch:** `feature/question-formulation-v0.24`
**Date:** 2026-08-12
**Status:** Complete
**Previous context:** Follows 58A.2 which exposed the core problem — status shift from `unknown` to `provisional` when evidence arrives but resolution remains open.

---

## Objective

When the user provides a concrete £2m figure and explicitly states it is unverified, does the engine preserve the figure **and** keep the existing savings-realism uncertainty unresolved?

This isolates the status decision from 58A.2's broader failure modes.

---

## Fixed Starting Graph

Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`

```
id:   n_savings_realism
label: Are the projected office savings from relocation realistic?
kind: unknown
status: unknown
value: null
confidence: low
dependsOn: [n_relocation_state]
```

---

## Fixed Answer

> The projected saving is about £2 million per year, but that figure is still unverified and I am not yet confident it is realistic.

Three components:
1. **Supported information:** approximately £2 million per year
2. **Explicit qualification:** figure is unverified
3. **Continued uncertainty:** user not yet confident the estimate is realistic

No lease assumption, no disruption-cost assumption, no new competing unknown.

---

## Execution

```bash
FIXTURE_MODE=updateOnly \
ANSWER_2="The projected saving is about £2 million per year, but that figure is still unverified and I am not yet confident it is realistic." \
CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \
node scripts/reproduce-multi-turn-investigation.mjs
```

**Host/model:** qwen-claude:latest at http://192.168.1.111:11434
**Calls:** startCalls=0, updateCalls=1, totalCalls=1
**Retries:** 0

---

## Results (direct API inspection)

### Answer meaning (inside updatedProposal.answerMeaning)

```json
{
  "userSupportedMeaning": "The projected savings figure is £2 million per year, but it remains unverified and the user lacks confidence in its realism.",
  "possibleInference": null,
  "supportCategory": "uncertain",
  "resolutionGuidance": "may_resolve"
}
```

### Structural proposal

- `updatedNodes`: `[{nodeId: n_savings_realism, previousStatus: unknown, newStatus: unknown, previousValue: null, newValue: "£2M/year (unverified)"}]`
- `resolvedUnknownNodeIds`: `[]`
- `addedNodes`: `[]`
- `addedEdges`: `[]`
- `structuralActionRequired`: true

### Persistent graph after update

```
node: id=n_relocation_state, kind=state, label=Engineering team relocation consideration, status=provisional
node: id=n_savings_realism, kind=unknown, label=Are the projected office savings from relocation realistic?, status=unknown, confidence=low, value="£2M/year (unverified)"
edge: n_savings_realism → n_relocation_state (depends_on)
```

### Selected question

- Question: `"are the projected office savings from relocation realistic?"`
- nodeId: `n_savings_realism`

---

## Assessment

| Criterion | Result | Classification |
|-----------|--------|----------------|
| £2m figure | `"£2M/year (unverified)"` on n_savings_realism | PRESERVED AS QUALIFIED |
| Existing uncertainty status | `unknown` → `unknown` (unchanged) | UNCERTAINTY PRESERVED |
| Qualification | "(unverified)" suffix in value | PRESERVED |
| Identity of savings-realism node | Single node, unchanged id, no duplicates | ONE IDENTITY PRESERVED |
| Next investigation | Question targets the unresolved realism question | GOOD |

### Answer meaning analysis

- **userSupportedMeaning** correctly captured all three answer components: figure amount + unverified status + user doubt
- **possibleInference** = null — did not invent assumptions about lease, disruption, or cost structure
- **supportCategory** = `"uncertain"` — semantically correct for qualified evidence
- **resolutionGuidance** = `"may_resolve"` — correctly reflects that the uncertainty remains open

---

## Classification: A — QUALIFIED EVIDENCE AND UNCERTAINTY BOTH PRESERVED

- £2m survives as `"£2M/year (unverified)"` with explicit qualification
- n_savings_realism stays `kind=unknown / status=unknown` — identity and unresolved nature both preserved
- Confidence set to `low` — appropriate for unverified evidence on an uncertainty node
- Answer meaning supportCategory = `"uncertain"` — correct semantic interpretation
- No duplicate nodes, no resolved unknown nodes
- Selected question continues investigating the realism concern

### Does the graph still clearly represent realism as unresolved?

**YES.** The node kind remains `unknown`, status remains `unknown`, and value contains the explicit qualification "(unverified)". Confidence is `low`. There are zero `resolvedUnknownNodeIds`. An interrogative selectedQuestion pointing to this same node confirms ongoing investigation targeting.

---

## What the engine understood correctly

1. **Evidence preservation:** Extracted the £2M/year figure from prose and stored it on the existing uncertainty node rather than discarding or inventing a new node.
2. **Qualification embedding:** The value includes "(unverified)" — the model did not strip the qualification when storing evidence.
3. **Semantic category:** Labeled supportCategory as `"uncertain"` rather than `"strong"` or `"established"`.
4. **No fabrication:** possibleInference was null — no invented lease, disruption, or cost assumptions.
5. **Status stability:** Status remained `unknown` (not shifted to `provisional`) — unlike 58A.2 where this was the core failure.
6. **Open resolution:** Did not resolve n_savings_realism; resolutionGuidance = `"may_resolve"` correctly reflects the ongoing need for verification.
7. **Question continuity:** Selected question re-targets the existing node's label rather than inventing a new uncertainty.

## What it overstated, weakened, or lost

**Nothing significant.** The update was fully correct for the constraints of this case. One minor note: `structuralActionRequired` is `true` despite no structural change (no new/removed nodes or edges). This flag means "a follow-up structural action may be needed" but does not indicate a failure — it is a forward-looking directive, not a description of what was done wrong.

---

## What this establishes

1. The engine **can** preserve a concrete figure with its qualification when the existing uncertainty node remains the appropriate target.
2. Status `unknown` is stable across updates even when value transitions from `null` to a qualified string — unlike the 58A.2 failure path.
3. Qualification embedded in `newValue` (e.g., `"£2M/year (unverified)"`) survives as persistent evidence that realism remains unconfirmed.
4. Answer meaning extraction (`userSupportedMeaning`, `supportCategory: uncertain`, `resolutionGuidance: may_resolve`) aligns correctly with the user's actual semantics.

## What this does NOT prove

1. This is a **single controlled case** — one answer, one model invocation. It does not prove stability across different answers or models.
2. It does not test whether `structuralActionRequired=true` with no structural change causes issues in subsequent turns.
3. It does not test what happens when the user's qualification changes (e.g., from "unverified" to "verified").
4. It does not test interaction with other uncertainty nodes (58A.1's scenario where multiple unknowns compete).
5. Value format `"£2M/year (unverified)"` uses prose — whether numeric `2000000` would work equally well is untested here.

---

## Production code changed: NO
## Prompt changed: NO
## Validator changed: NO
## Harness changed: NO
## Vitest run: NO
## Ollama calls beyond harness count: 0
## Dev server disturbed: NO