experiment: test verified uncertainty resolution
This commit is contained in:
@@ -171,7 +171,13 @@ Answer before continuing:
|
||||
|
||||
---
|
||||
|
||||
_Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F, 56D–56H, 56L–56M, v0.8 closeout. Branch: `feature/question-formulation-v0.24`. First-pass reasoning-fidelity v0.8 complete to A–F scope._
|
||||
_Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F, 56D–56H, 56L–56M, 58B.2, v0.8 closeout. Branch: `feature/question-formulation-v0.24`. First-pass reasoning-fidelity v0.8 complete to A–F scope._
|
||||
|
||||
### Experiment 58B.2 — Verified Uncertainty Resolution
|
||||
|
||||
**Objective:** When the user explicitly verifies a figure and states confidence in its realism, does the engine resolve the existing savings-realism uncertainty rather than merely changing its value?
|
||||
|
||||
**Classification: A — UNCERTAINTY CORRECTLY RESOLVED.** One update-only call. The engine correctly resolved `n_savings_realism` (status unknown→resolved), included it in `resolvedUnknownNodeIds`, preserved the £2m figure as "£2,000,000" with verification captured in the reason field ("User verified the projected annual savings including lease exit costs are realistic."). No duplicate uncertainty created. No next question produced — correctly reflecting that no consequential unresolved issue remains. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
# Experiment 58B.2 — Verified Uncertainty Resolution
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Date:** 2026-08-12
|
||||
**Status:** Complete
|
||||
**Previous context:** Follows 58B.1 which showed the engine preserves qualified evidence while keeping uncertainty open. This tests the opposite boundary: when the user explicitly verifies and confirms realism, does the engine resolve?
|
||||
|
||||
---
|
||||
|
||||
## Objective
|
||||
|
||||
When the user explicitly says the £2m figure has now been verified and is realistic, does the engine resolve the existing `n_savings_realism` uncertainty rather than merely changing its value or weakening its status?
|
||||
|
||||
---
|
||||
|
||||
## Fixed Starting Graph
|
||||
|
||||
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
|
||||
```
|
||||
id: n_savings_realism
|
||||
label: Are the projected office savings from relocation realistic?
|
||||
kind: unknown
|
||||
status: unknown
|
||||
value: null
|
||||
confidence: low
|
||||
dependsOn: [n_relocation_state]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Fixed Answer
|
||||
|
||||
> We have now verified the projected saving at about £2 million per year, including the relevant lease exit costs, and I am confident that estimate is realistic.
|
||||
|
||||
Components:
|
||||
1. **Concrete value:** approximately £2 million per year
|
||||
2. **Verification:** the estimate has now been checked
|
||||
3. **Relevant dependency addressed:** lease exit costs included
|
||||
4. **Explicit confidence:** user now believes the estimate is realistic
|
||||
|
||||
No new uncertainty introduced.
|
||||
|
||||
---
|
||||
|
||||
## Execution
|
||||
|
||||
```bash
|
||||
FIXTURE_MODE=updateOnly \
|
||||
ANSWER_2="We have now verified the projected saving at about £2 million per year, including the relevant lease exit costs, and I am confident that estimate is realistic." \
|
||||
CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \
|
||||
node scripts/reproduce-multi-turn-investigation.mjs
|
||||
```
|
||||
|
||||
**Host/model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
**Calls:** startCalls=0, updateCalls=1, totalCalls=1
|
||||
**Retries:** 0
|
||||
|
||||
---
|
||||
|
||||
## Results
|
||||
|
||||
### Answer meaning
|
||||
|
||||
Not returned in the update response (updateOnly mode accepted without including answerMeaning in the output). The structural fields below are authoritative.
|
||||
|
||||
### Structural proposal
|
||||
|
||||
- `updatedNodes`: `[{"nodeId":"n_savings_realism","previousStatus":"unknown","newStatus":"resolved","previousValue":null,"newValue":"£2,000,000","reason":"User verified the projected annual savings including lease exit costs are realistic."}]`
|
||||
- `resolvedUnknownNodeIds`: `["n_savings_realism"]`
|
||||
- `addedNodes`: `[]`
|
||||
- `addedEdges`: `[]`
|
||||
- `structuralActionRequired`: null
|
||||
|
||||
### Persistent graph after update
|
||||
|
||||
```
|
||||
node: id=n_relocation_state, kind=state, label=Engineering team relocation consideration, status=provisional
|
||||
node: id=n_savings_realism, kind=unknown, label=Are the projected office savings from relocation realistic?, status=resolved
|
||||
edge: n_savings_realism → n_relocation_state (depends_on)
|
||||
```
|
||||
|
||||
### Selected question
|
||||
|
||||
None produced.
|
||||
|
||||
---
|
||||
|
||||
## Assessment
|
||||
|
||||
| Criterion | Result | Classification |
|
||||
|-----------|--------|----------------|
|
||||
| £2m figure | `"£2,000,000"` on n_savings_realism | PRESERVED AS VERIFIED |
|
||||
| Existing uncertainty status | `unknown` → `resolved` | CORRECTLY RESOLVED |
|
||||
| Identity of savings-realism node | Single node, unchanged id (`n_savings_realism`), no duplicates | ORIGINAL IDENTITY RESOLVED |
|
||||
| Verification meaning | reason: "User verified the projected annual savings including lease exit costs are realistic." | PRESERVED |
|
||||
| Next investigation | NONE — no consequential unresolved issue remains | GOOD |
|
||||
|
||||
### £2m figure analysis
|
||||
|
||||
The value `"£2,000,000"` preserves the core monetary figure. The "per year" unit is not explicit in `newValue` (unlike 58B.1 which had `"£2M/year (unverified)"`) but is preserved in the reason field ("projected **annual** savings"). This qualifies as PRESERVED AS VERIFIED — the amount is captured and the verification context survives.
|
||||
|
||||
### Uncertainty resolution analysis
|
||||
|
||||
Status clearly changed from `unknown` to `resolved`. The node id `n_savings_realism` appears in `resolvedUnknownNodeIds`. This is unambiguous correct resolution.
|
||||
|
||||
### Identity analysis
|
||||
|
||||
Exactly one savings-realism unknown node exists before and after the update. Same node id, same label, status transitions correctly. No duplicate created. ORIGINAL IDENTITY RESOLVED.
|
||||
|
||||
### Verification meaning analysis
|
||||
|
||||
The reason field on the updated node explicitly states: "User verified the projected annual savings including lease exit costs are realistic." This captures all four components of the user's answer (value, verification, lease costs, confidence). PRESERVED.
|
||||
|
||||
### Next investigation analysis
|
||||
|
||||
No selected question was produced. This is correct behavior — the existing uncertainty is resolved and no new consequential unresolved issue was introduced by the answer. GOOD.
|
||||
|
||||
---
|
||||
|
||||
## Classification: A — UNCERTAINTY CORRECTLY RESOLVED
|
||||
|
||||
- n_savings_realism correctly resolved (status → `resolved`)
|
||||
- Included in `resolvedUnknownNodeIds`
|
||||
- Verified £2m evidence survives as `"£2,000,000"` with full verification context in reason field
|
||||
- No duplicate uncertainty created
|
||||
- Same node id preserved (original identity resolved)
|
||||
- No redundant question asked about realism
|
||||
- No consequential unresolved issue remains to investigate
|
||||
|
||||
---
|
||||
|
||||
## What the engine understood correctly
|
||||
|
||||
1. **Resolution trigger:** The explicit "verified" and "confident...realistic" language triggered correct uncertainty resolution — status moved from `unknown` to `resolved`. This is the semantic boundary 58B.1 left open.
|
||||
2. **Value extraction:** The figure was captured as `"£2,000,000"` — a clean monetary representation.
|
||||
3. **Verification context:** The reason field captured all four answer components: value (£2m), verification status ("verified"), lease exit costs, and confidence ("realistic").
|
||||
4. **No fabrication:** No new uncertainty nodes or edges were created from this answer that contained no new uncertainty.
|
||||
5. **Identity preservation:** The original `n_savings_realism` was updated (not replaced or duplicated).
|
||||
6. **Correct termination signal:** No selected question was produced, correctly reflecting that the existing investigation thread is complete.
|
||||
|
||||
---
|
||||
|
||||
## What it overstated, weakened, or lost
|
||||
|
||||
**Minor weakening of temporal unit:** The "per year" time unit is not explicit in `newValue` (which is `"£2,000,000"` rather than `"£2,000,000/year"`). However, the word "annual" in the reason field partially compensates. This does not affect the core resolution question — it is a secondary representation detail.
|
||||
|
||||
---
|
||||
|
||||
## What this establishes
|
||||
|
||||
1. When the user provides **explicit verification** AND **confidence about realism**, the engine correctly resolves the existing savings-realism uncertainty (status → `resolved` + inclusion in `resolvedUnknownNodeIds`).
|
||||
2. This is the semantic opposite of 58B.1 and works correctly — the engine distinguishes between "unverified but plausible" (keep open) and "verified and confident" (resolve).
|
||||
3. The verified £2m figure survives in persistent graph state with verification context captured.
|
||||
4. No duplicate uncertainty is created during resolution.
|
||||
5. The engine does not ask the resolved question again — it correctly terminates when no consequential unresolved issue remains.
|
||||
|
||||
---
|
||||
|
||||
## What this does NOT prove
|
||||
|
||||
1. **Single controlled case** — one answer, one model invocation. Stability across different answers or repeated runs is untested.
|
||||
2. **Temporal unit preservation** — whether "per year" survives in `newValue` depends on answer phrasing and model behavior.
|
||||
3. **answerMeaning fields** — the update response did not include answerMeaning, so whether userSupportedMeaning captured all four components independently of the reason field is unavailable.
|
||||
4. **Multi-turn stability** — what happens when subsequent turns arrive after a resolved uncertainty is untested here.
|
||||
5. **Cross-domain generalisation** — this is a single cost-savings domain case.
|
||||
|
||||
---
|
||||
|
||||
## Production code changed: NO
|
||||
## Prompt changed: NO
|
||||
## Validator changed: NO
|
||||
## Harness changed: NO
|
||||
## Vitest run: NO
|
||||
## Ollama calls beyond harness count: 0
|
||||
## Dev server disturbed: NO
|
||||
Reference in New Issue
Block a user