From 3f1bf7bcb078196f7feceaab721c55511753b8a6 Mon Sep 17 00:00:00 2001 From: robbond Date: Wed, 12 Aug 2026 16:26:43 +0100 Subject: [PATCH] experiment: test verified uncertainty resolution --- docs/current-handoff.md | 8 +- docs/experiment-58b2.md | 176 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 183 insertions(+), 1 deletion(-) create mode 100644 docs/experiment-58b2.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index ee6979d..6e57701 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -171,7 +171,13 @@ Answer before continuing: --- -_Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F, 56D–56H, 56L–56M, v0.8 closeout. Branch: `feature/question-formulation-v0.24`. First-pass reasoning-fidelity v0.8 complete to A–F scope._ +_Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F, 56D–56H, 56L–56M, 58B.2, v0.8 closeout. Branch: `feature/question-formulation-v0.24`. First-pass reasoning-fidelity v0.8 complete to A–F scope._ + +### Experiment 58B.2 — Verified Uncertainty Resolution + +**Objective:** When the user explicitly verifies a figure and states confidence in its realism, does the engine resolve the existing savings-realism uncertainty rather than merely changing its value? + +**Classification: A — UNCERTAINTY CORRECTLY RESOLVED.** One update-only call. The engine correctly resolved `n_savings_realism` (status unknown→resolved), included it in `resolvedUnknownNodeIds`, preserved the £2m figure as "£2,000,000" with verification captured in the reason field ("User verified the projected annual savings including lease exit costs are realistic."). No duplicate uncertainty created. No next question produced — correctly reflecting that no consequential unresolved issue remains. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. --- diff --git a/docs/experiment-58b2.md b/docs/experiment-58b2.md new file mode 100644 index 0000000..5ed366e --- /dev/null +++ b/docs/experiment-58b2.md @@ -0,0 +1,176 @@ +# Experiment 58B.2 — Verified Uncertainty Resolution + +**Branch:** `feature/question-formulation-v0.24` +**Date:** 2026-08-12 +**Status:** Complete +**Previous context:** Follows 58B.1 which showed the engine preserves qualified evidence while keeping uncertainty open. This tests the opposite boundary: when the user explicitly verifies and confirms realism, does the engine resolve? + +--- + +## Objective + +When the user explicitly says the £2m figure has now been verified and is realistic, does the engine resolve the existing `n_savings_realism` uncertainty rather than merely changing its value or weakening its status? + +--- + +## Fixed Starting Graph + +Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json` + +``` +id: n_savings_realism +label: Are the projected office savings from relocation realistic? +kind: unknown +status: unknown +value: null +confidence: low +dependsOn: [n_relocation_state] +``` + +--- + +## Fixed Answer + +> We have now verified the projected saving at about £2 million per year, including the relevant lease exit costs, and I am confident that estimate is realistic. + +Components: +1. **Concrete value:** approximately £2 million per year +2. **Verification:** the estimate has now been checked +3. **Relevant dependency addressed:** lease exit costs included +4. **Explicit confidence:** user now believes the estimate is realistic + +No new uncertainty introduced. + +--- + +## Execution + +```bash +FIXTURE_MODE=updateOnly \ +ANSWER_2="We have now verified the projected saving at about £2 million per year, including the relevant lease exit costs, and I am confident that estimate is realistic." \ +CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \ +node scripts/reproduce-multi-turn-investigation.mjs +``` + +**Host/model:** qwen-claude:latest at http://192.168.1.111:11434 +**Calls:** startCalls=0, updateCalls=1, totalCalls=1 +**Retries:** 0 + +--- + +## Results + +### Answer meaning + +Not returned in the update response (updateOnly mode accepted without including answerMeaning in the output). The structural fields below are authoritative. + +### Structural proposal + +- `updatedNodes`: `[{"nodeId":"n_savings_realism","previousStatus":"unknown","newStatus":"resolved","previousValue":null,"newValue":"£2,000,000","reason":"User verified the projected annual savings including lease exit costs are realistic."}]` +- `resolvedUnknownNodeIds`: `["n_savings_realism"]` +- `addedNodes`: `[]` +- `addedEdges`: `[]` +- `structuralActionRequired`: null + +### Persistent graph after update + +``` +node: id=n_relocation_state, kind=state, label=Engineering team relocation consideration, status=provisional +node: id=n_savings_realism, kind=unknown, label=Are the projected office savings from relocation realistic?, status=resolved +edge: n_savings_realism → n_relocation_state (depends_on) +``` + +### Selected question + +None produced. + +--- + +## Assessment + +| Criterion | Result | Classification | +|-----------|--------|----------------| +| £2m figure | `"£2,000,000"` on n_savings_realism | PRESERVED AS VERIFIED | +| Existing uncertainty status | `unknown` → `resolved` | CORRECTLY RESOLVED | +| Identity of savings-realism node | Single node, unchanged id (`n_savings_realism`), no duplicates | ORIGINAL IDENTITY RESOLVED | +| Verification meaning | reason: "User verified the projected annual savings including lease exit costs are realistic." | PRESERVED | +| Next investigation | NONE — no consequential unresolved issue remains | GOOD | + +### £2m figure analysis + +The value `"£2,000,000"` preserves the core monetary figure. The "per year" unit is not explicit in `newValue` (unlike 58B.1 which had `"£2M/year (unverified)"`) but is preserved in the reason field ("projected **annual** savings"). This qualifies as PRESERVED AS VERIFIED — the amount is captured and the verification context survives. + +### Uncertainty resolution analysis + +Status clearly changed from `unknown` to `resolved`. The node id `n_savings_realism` appears in `resolvedUnknownNodeIds`. This is unambiguous correct resolution. + +### Identity analysis + +Exactly one savings-realism unknown node exists before and after the update. Same node id, same label, status transitions correctly. No duplicate created. ORIGINAL IDENTITY RESOLVED. + +### Verification meaning analysis + +The reason field on the updated node explicitly states: "User verified the projected annual savings including lease exit costs are realistic." This captures all four components of the user's answer (value, verification, lease costs, confidence). PRESERVED. + +### Next investigation analysis + +No selected question was produced. This is correct behavior — the existing uncertainty is resolved and no new consequential unresolved issue was introduced by the answer. GOOD. + +--- + +## Classification: A — UNCERTAINTY CORRECTLY RESOLVED + +- n_savings_realism correctly resolved (status → `resolved`) +- Included in `resolvedUnknownNodeIds` +- Verified £2m evidence survives as `"£2,000,000"` with full verification context in reason field +- No duplicate uncertainty created +- Same node id preserved (original identity resolved) +- No redundant question asked about realism +- No consequential unresolved issue remains to investigate + +--- + +## What the engine understood correctly + +1. **Resolution trigger:** The explicit "verified" and "confident...realistic" language triggered correct uncertainty resolution — status moved from `unknown` to `resolved`. This is the semantic boundary 58B.1 left open. +2. **Value extraction:** The figure was captured as `"£2,000,000"` — a clean monetary representation. +3. **Verification context:** The reason field captured all four answer components: value (£2m), verification status ("verified"), lease exit costs, and confidence ("realistic"). +4. **No fabrication:** No new uncertainty nodes or edges were created from this answer that contained no new uncertainty. +5. **Identity preservation:** The original `n_savings_realism` was updated (not replaced or duplicated). +6. **Correct termination signal:** No selected question was produced, correctly reflecting that the existing investigation thread is complete. + +--- + +## What it overstated, weakened, or lost + +**Minor weakening of temporal unit:** The "per year" time unit is not explicit in `newValue` (which is `"£2,000,000"` rather than `"£2,000,000/year"`). However, the word "annual" in the reason field partially compensates. This does not affect the core resolution question — it is a secondary representation detail. + +--- + +## What this establishes + +1. When the user provides **explicit verification** AND **confidence about realism**, the engine correctly resolves the existing savings-realism uncertainty (status → `resolved` + inclusion in `resolvedUnknownNodeIds`). +2. This is the semantic opposite of 58B.1 and works correctly — the engine distinguishes between "unverified but plausible" (keep open) and "verified and confident" (resolve). +3. The verified £2m figure survives in persistent graph state with verification context captured. +4. No duplicate uncertainty is created during resolution. +5. The engine does not ask the resolved question again — it correctly terminates when no consequential unresolved issue remains. + +--- + +## What this does NOT prove + +1. **Single controlled case** — one answer, one model invocation. Stability across different answers or repeated runs is untested. +2. **Temporal unit preservation** — whether "per year" survives in `newValue` depends on answer phrasing and model behavior. +3. **answerMeaning fields** — the update response did not include answerMeaning, so whether userSupportedMeaning captured all four components independently of the reason field is unavailable. +4. **Multi-turn stability** — what happens when subsequent turns arrive after a resolved uncertainty is untested here. +5. **Cross-domain generalisation** — this is a single cost-savings domain case. + +--- + +## Production code changed: NO +## Prompt changed: NO +## Validator changed: NO +## Harness changed: NO +## Vitest run: NO +## Ollama calls beyond harness count: 0 +## Dev server disturbed: NO