robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5ce5e7349b03173d42348bf624b68005078db637
confidence-engine/docs/experiment-58b2.md
T

7.9 KiB
Raw Blame History

Experiment 58B.2 — Verified Uncertainty Resolution

Branch: feature/question-formulation-v0.24 Date: 2026-08-12 Status: Complete Previous context: Follows 58B.1 which showed the engine preserves qualified evidence while keeping uncertainty open. This tests the opposite boundary: when the user explicitly verifies and confirms realism, does the engine resolve?

Objective

When the user explicitly says the £2m figure has now been verified and is realistic, does the engine resolve the existing n_savings_realism uncertainty rather than merely changing its value or weakening its status?

Fixed Starting Graph

Fixture: tests/fixtures/pre-anchored-update-savings-realism.json

id:   n_savings_realism
label: Are the projected office savings from relocation realistic?
kind: unknown
status: unknown
value: null
confidence: low
dependsOn: [n_relocation_state]

Fixed Answer

We have now verified the projected saving at about £2 million per year, including the relevant lease exit costs, and I am confident that estimate is realistic.

Components:

  1. Concrete value: approximately £2 million per year
  2. Verification: the estimate has now been checked
  3. Relevant dependency addressed: lease exit costs included
  4. Explicit confidence: user now believes the estimate is realistic

No new uncertainty introduced.

Execution

FIXTURE_MODE=updateOnly \
ANSWER_2="We have now verified the projected saving at about £2 million per year, including the relevant lease exit costs, and I am confident that estimate is realistic." \
CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \
node scripts/reproduce-multi-turn-investigation.mjs

Host/model: qwen-claude:latest at http://192.168.1.111:11434 Calls: startCalls=0, updateCalls=1, totalCalls=1 Retries: 0

Results

Answer meaning

Not returned in the update response (updateOnly mode accepted without including answerMeaning in the output). The structural fields below are authoritative.

Structural proposal

  • updatedNodes: [{"nodeId":"n_savings_realism","previousStatus":"unknown","newStatus":"resolved","previousValue":null,"newValue":"£2,000,000","reason":"User verified the projected annual savings including lease exit costs are realistic."}]
  • resolvedUnknownNodeIds: ["n_savings_realism"]
  • addedNodes: []
  • addedEdges: []
  • structuralActionRequired: null

Persistent graph after update

node: id=n_relocation_state, kind=state, label=Engineering team relocation consideration, status=provisional
node: id=n_savings_realism, kind=unknown, label=Are the projected office savings from relocation realistic?, status=resolved
edge: n_savings_realism → n_relocation_state (depends_on)

Selected question

None produced.

Assessment

Criterion Result Classification
£2m figure "£2,000,000" on n_savings_realism PRESERVED AS VERIFIED
Existing uncertainty status unknown → resolved CORRECTLY RESOLVED
Identity of savings-realism node Single node, unchanged id (n_savings_realism), no duplicates ORIGINAL IDENTITY RESOLVED
Verification meaning reason: "User verified the projected annual savings including lease exit costs are realistic." PRESERVED
Next investigation NONE — no consequential unresolved issue remains GOOD

£2m figure analysis

The value "£2,000,000" preserves the core monetary figure. The "per year" unit is not explicit in newValue (unlike 58B.1 which had "£2M/year (unverified)") but is preserved in the reason field ("projected annual savings"). This qualifies as PRESERVED AS VERIFIED — the amount is captured and the verification context survives.

Uncertainty resolution analysis

Status clearly changed from unknown to resolved. The node id n_savings_realism appears in resolvedUnknownNodeIds. This is unambiguous correct resolution.

Identity analysis

Exactly one savings-realism unknown node exists before and after the update. Same node id, same label, status transitions correctly. No duplicate created. ORIGINAL IDENTITY RESOLVED.

Verification meaning analysis

The reason field on the updated node explicitly states: "User verified the projected annual savings including lease exit costs are realistic." This captures all four components of the user's answer (value, verification, lease costs, confidence). PRESERVED.

Next investigation analysis

No selected question was produced. This is correct behavior — the existing uncertainty is resolved and no new consequential unresolved issue was introduced by the answer. GOOD.

Classification: A — UNCERTAINTY CORRECTLY RESOLVED

  • n_savings_realism correctly resolved (status → resolved)
  • Included in resolvedUnknownNodeIds
  • Verified £2m evidence survives as "£2,000,000" with full verification context in reason field
  • No duplicate uncertainty created
  • Same node id preserved (original identity resolved)
  • No redundant question asked about realism
  • No consequential unresolved issue remains to investigate

What the engine understood correctly

  1. Resolution trigger: The explicit "verified" and "confident...realistic" language triggered correct uncertainty resolution — status moved from unknown to resolved. This is the semantic boundary 58B.1 left open.
  2. Value extraction: The figure was captured as "£2,000,000" — a clean monetary representation.
  3. Verification context: The reason field captured all four answer components: value (£2m), verification status ("verified"), lease exit costs, and confidence ("realistic").
  4. No fabrication: No new uncertainty nodes or edges were created from this answer that contained no new uncertainty.
  5. Identity preservation: The original n_savings_realism was updated (not replaced or duplicated).
  6. Correct termination signal: No selected question was produced, correctly reflecting that the existing investigation thread is complete.

What it overstated, weakened, or lost

Minor weakening of temporal unit: The "per year" time unit is not explicit in newValue (which is "£2,000,000" rather than "£2,000,000/year"). However, the word "annual" in the reason field partially compensates. This does not affect the core resolution question — it is a secondary representation detail.

What this establishes

  1. When the user provides explicit verification AND confidence about realism, the engine correctly resolves the existing savings-realism uncertainty (status → resolved + inclusion in resolvedUnknownNodeIds).
  2. This is the semantic opposite of 58B.1 and works correctly — the engine distinguishes between "unverified but plausible" (keep open) and "verified and confident" (resolve).
  3. The verified £2m figure survives in persistent graph state with verification context captured.
  4. No duplicate uncertainty is created during resolution.
  5. The engine does not ask the resolved question again — it correctly terminates when no consequential unresolved issue remains.

What this does NOT prove

  1. Single controlled case — one answer, one model invocation. Stability across different answers or repeated runs is untested.
  2. Temporal unit preservation — whether "per year" survives in newValue depends on answer phrasing and model behavior.
  3. answerMeaning fields — the update response did not include answerMeaning, so whether userSupportedMeaning captured all four components independently of the reason field is unavailable.
  4. Multi-turn stability — what happens when subsequent turns arrive after a resolved uncertainty is untested here.
  5. Cross-domain generalisation — this is a single cost-savings domain case.

Production code changed: NO

Prompt changed: NO

Validator changed: NO

Harness changed: NO

Vitest run: NO

Ollama calls beyond harness count: 0

Dev server disturbed: NO

Reference in New Issue View Git Blame Copy Permalink