confidence-engine/docs/experiment-58b1.md

Experiment 58B.1 — Qualified Evidence Without Weakening Uncertainty

Branch: feature/question-formulation-v0.24 Date: 2026-08-12 Status: Complete Previous context: Follows 58A.2 which exposed the core problem — status shift from unknown to provisional when evidence arrives but resolution remains open.

---

Objective

When the user provides a concrete £2m figure and explicitly states it is unverified, does the engine preserve the figure and keep the existing savings-realism uncertainty unresolved?

This isolates the status decision from 58A.2's broader failure modes.

---

Fixed Starting Graph

Fixture: tests/fixtures/pre-anchored-update-savings-realism.json

id:   n_savings_realism
label: Are the projected office savings from relocation realistic?
kind: unknown
status: unknown
value: null
confidence: low
dependsOn: [n_relocation_state]

---

Fixed Answer

The projected saving is about £2 million per year, but that figure is still unverified and I am not yet confident it is realistic.

Three components:

1. Supported information: approximately £2 million per year
2. Explicit qualification: figure is unverified
3. Continued uncertainty: user not yet confident the estimate is realistic

No lease assumption, no disruption-cost assumption, no new competing unknown.

---

Execution

FIXTURE_MODE=updateOnly \
ANSWER_2="The projected saving is about £2 million per year, but that figure is still unverified and I am not yet confident it is realistic." \
CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \
node scripts/reproduce-multi-turn-investigation.mjs

Host/model: qwen-claude:latest at http://192.168.1.111:11434 Calls: startCalls=0, updateCalls=1, totalCalls=1 Retries: 0

---

Results (direct API inspection)

Answer meaning (inside updatedProposal.answerMeaning)

{
  "userSupportedMeaning": "The projected savings figure is £2 million per year, but it remains unverified and the user lacks confidence in its realism.",
  "possibleInference": null,
  "supportCategory": "uncertain",
  "resolutionGuidance": "may_resolve"
}

Structural proposal

• updatedNodes: [{nodeId: n_savings_realism, previousStatus: unknown, newStatus: unknown, previousValue: null, newValue: "£2M/year (unverified)"}]
• resolvedUnknownNodeIds: []
• addedNodes: []
• addedEdges: []
• structuralActionRequired: true

Persistent graph after update

node: id=n_relocation_state, kind=state, label=Engineering team relocation consideration, status=provisional
node: id=n_savings_realism, kind=unknown, label=Are the projected office savings from relocation realistic?, status=unknown, confidence=low, value="£2M/year (unverified)"
edge: n_savings_realism → n_relocation_state (depends_on)

Selected question

• Question: "are the projected office savings from relocation realistic?"
• nodeId: n_savings_realism

---

Assessment

Criterion	Result	Classification
£2m figure	"£2M/year (unverified)" on n_savings_realism	PRESERVED AS QUALIFIED
Existing uncertainty status	unknown → unknown (unchanged)	UNCERTAINTY PRESERVED
Qualification	"(unverified)" suffix in value	PRESERVED
Identity of savings-realism node	Single node, unchanged id, no duplicates	ONE IDENTITY PRESERVED
Next investigation	Question targets the unresolved realism question	GOOD

Answer meaning analysis

• userSupportedMeaning correctly captured all three answer components: figure amount + unverified status + user doubt
• possibleInference = null — did not invent assumptions about lease, disruption, or cost structure
• supportCategory = "uncertain" — semantically correct for qualified evidence
• resolutionGuidance = "may_resolve" — correctly reflects that the uncertainty remains open

---

Classification: A — QUALIFIED EVIDENCE AND UNCERTAINTY BOTH PRESERVED

• £2m survives as "£2M/year (unverified)" with explicit qualification
• n_savings_realism stays kind=unknown / status=unknown — identity and unresolved nature both preserved
• Confidence set to low — appropriate for unverified evidence on an uncertainty node
• Answer meaning supportCategory = "uncertain" — correct semantic interpretation
• No duplicate nodes, no resolved unknown nodes
• Selected question continues investigating the realism concern

Does the graph still clearly represent realism as unresolved?

YES. The node kind remains unknown, status remains unknown, and value contains the explicit qualification "(unverified)". Confidence is low. There are zero resolvedUnknownNodeIds. An interrogative selectedQuestion pointing to this same node confirms ongoing investigation targeting.

---

What the engine understood correctly

1. Evidence preservation: Extracted the £2M/year figure from prose and stored it on the existing uncertainty node rather than discarding or inventing a new node.
2. Qualification embedding: The value includes "(unverified)" — the model did not strip the qualification when storing evidence.
3. Semantic category: Labeled supportCategory as "uncertain" rather than "strong" or "established".
4. No fabrication: possibleInference was null — no invented lease, disruption, or cost assumptions.
5. Status stability: Status remained unknown (not shifted to provisional) — unlike 58A.2 where this was the core failure.
6. Open resolution: Did not resolve n_savings_realism; resolutionGuidance = "may_resolve" correctly reflects the ongoing need for verification.
7. Question continuity: Selected question re-targets the existing node's label rather than inventing a new uncertainty.

What it overstated, weakened, or lost

Nothing significant. The update was fully correct for the constraints of this case. One minor note: structuralActionRequired is true despite no structural change (no new/removed nodes or edges). This flag means "a follow-up structural action may be needed" but does not indicate a failure — it is a forward-looking directive, not a description of what was done wrong.

---

What this establishes

1. The engine can preserve a concrete figure with its qualification when the existing uncertainty node remains the appropriate target.
2. Status unknown is stable across updates even when value transitions from null to a qualified string — unlike the 58A.2 failure path.
3. Qualification embedded in newValue (e.g., "£2M/year (unverified)") survives as persistent evidence that realism remains unconfirmed.
4. Answer meaning extraction (userSupportedMeaning, supportCategory: uncertain, resolutionGuidance: may_resolve) aligns correctly with the user's actual semantics.

What this does NOT prove

1. This is a single controlled case — one answer, one model invocation. It does not prove stability across different answers or models.
2. It does not test whether structuralActionRequired=true with no structural change causes issues in subsequent turns.
3. It does not test what happens when the user's qualification changes (e.g., from "unverified" to "verified").
4. It does not test interaction with other uncertainty nodes (58A.1's scenario where multiple unknowns compete).
5. Value format "£2M/year (unverified)" uses prose — whether numeric 2000000 would work equally well is untested here.

---

Production code changed: NO

Prompt changed: NO

Validator changed: NO

Harness changed: NO

Vitest run: NO

Ollama calls beyond harness count: 0

Dev server disturbed: NO