diff --git a/docs/current-handoff.md b/docs/current-handoff.md index f583790..ee6979d 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -1794,3 +1794,9 @@ The hypothesis asked whether selecting `n_savings_realism` would now produce a c **Objective:** When the answer simply preserves the existing savings-realism uncertainty and introduces no new issue, does the live engine select that existing interrogative node and render its next question coherently through the 58A.4 fix? **Classification: E — NO QUESTION.** One update-only call via the committed harness. The update was rejected at `proposal_compatibility` with HTTP 422 because `structuralActionRequired=true` but the proposal contained no graph mutation (only `[updatedNodes: [{nodeId: n_savings_realism, newValue: null}]]`). The model correctly extracted user meaning ("The user remains unsure about whether the projected office savings from relocation are realistic.") but could not produce any structural change because the answer provided only uncertainty confirmation with zero new evidence. No next question was emitted because the apparatus blocks before question-selection. **This is a semantic gap:** pure-preservation answers provide no mutation target, yet the engine treats the uncertainty as requiring structural action. The 58A.4 interrogative fix path remains unproven live. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. + +### Experiment 58B.1 — Qualified Evidence Without Weakening Uncertainty (CONTROLLED) + +**Objective:** When the user provides a concrete £2m figure but explicitly states it is unverified, does the engine preserve the figure and keep the existing savings-realism uncertainty unresolved? + +**Classification: A — QUALIFIED EVIDENCE AND UNCERTAINTY BOTH PRESERVED.** One update-only call via the committed harness (confirmed by direct API inspection). The engine preserved the £2m/year figure as `"£2M/year (unverified)"` on n_savings_realism, kept status as `unknown` (not weakened to provisional), confidence set to low, answerMeaning.supportCategory = `"uncertain"`, possibleInference = null. No duplicate nodes, no resolved unknowns. Selected question continues investigating the realism concern. This is the best result seen for savings-realism across all 58A/B experiments — evidence and uncertainty both survive intact. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. diff --git a/docs/experiment-58b1.md b/docs/experiment-58b1.md new file mode 100644 index 0000000..677be0d --- /dev/null +++ b/docs/experiment-58b1.md @@ -0,0 +1,171 @@ +# Experiment 58B.1 — Qualified Evidence Without Weakening Uncertainty + +**Branch:** `feature/question-formulation-v0.24` +**Date:** 2026-08-12 +**Status:** Complete +**Previous context:** Follows 58A.2 which exposed the core problem — status shift from `unknown` to `provisional` when evidence arrives but resolution remains open. + +--- + +## Objective + +When the user provides a concrete £2m figure and explicitly states it is unverified, does the engine preserve the figure **and** keep the existing savings-realism uncertainty unresolved? + +This isolates the status decision from 58A.2's broader failure modes. + +--- + +## Fixed Starting Graph + +Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json` + +``` +id: n_savings_realism +label: Are the projected office savings from relocation realistic? +kind: unknown +status: unknown +value: null +confidence: low +dependsOn: [n_relocation_state] +``` + +--- + +## Fixed Answer + +> The projected saving is about £2 million per year, but that figure is still unverified and I am not yet confident it is realistic. + +Three components: +1. **Supported information:** approximately £2 million per year +2. **Explicit qualification:** figure is unverified +3. **Continued uncertainty:** user not yet confident the estimate is realistic + +No lease assumption, no disruption-cost assumption, no new competing unknown. + +--- + +## Execution + +```bash +FIXTURE_MODE=updateOnly \ +ANSWER_2="The projected saving is about £2 million per year, but that figure is still unverified and I am not yet confident it is realistic." \ +CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \ +node scripts/reproduce-multi-turn-investigation.mjs +``` + +**Host/model:** qwen-claude:latest at http://192.168.1.111:11434 +**Calls:** startCalls=0, updateCalls=1, totalCalls=1 +**Retries:** 0 + +--- + +## Results (direct API inspection) + +### Answer meaning (inside updatedProposal.answerMeaning) + +```json +{ + "userSupportedMeaning": "The projected savings figure is £2 million per year, but it remains unverified and the user lacks confidence in its realism.", + "possibleInference": null, + "supportCategory": "uncertain", + "resolutionGuidance": "may_resolve" +} +``` + +### Structural proposal + +- `updatedNodes`: `[{nodeId: n_savings_realism, previousStatus: unknown, newStatus: unknown, previousValue: null, newValue: "£2M/year (unverified)"}]` +- `resolvedUnknownNodeIds`: `[]` +- `addedNodes`: `[]` +- `addedEdges`: `[]` +- `structuralActionRequired`: true + +### Persistent graph after update + +``` +node: id=n_relocation_state, kind=state, label=Engineering team relocation consideration, status=provisional +node: id=n_savings_realism, kind=unknown, label=Are the projected office savings from relocation realistic?, status=unknown, confidence=low, value="£2M/year (unverified)" +edge: n_savings_realism → n_relocation_state (depends_on) +``` + +### Selected question + +- Question: `"are the projected office savings from relocation realistic?"` +- nodeId: `n_savings_realism` + +--- + +## Assessment + +| Criterion | Result | Classification | +|-----------|--------|----------------| +| £2m figure | `"£2M/year (unverified)"` on n_savings_realism | PRESERVED AS QUALIFIED | +| Existing uncertainty status | `unknown` → `unknown` (unchanged) | UNCERTAINTY PRESERVED | +| Qualification | "(unverified)" suffix in value | PRESERVED | +| Identity of savings-realism node | Single node, unchanged id, no duplicates | ONE IDENTITY PRESERVED | +| Next investigation | Question targets the unresolved realism question | GOOD | + +### Answer meaning analysis + +- **userSupportedMeaning** correctly captured all three answer components: figure amount + unverified status + user doubt +- **possibleInference** = null — did not invent assumptions about lease, disruption, or cost structure +- **supportCategory** = `"uncertain"` — semantically correct for qualified evidence +- **resolutionGuidance** = `"may_resolve"` — correctly reflects that the uncertainty remains open + +--- + +## Classification: A — QUALIFIED EVIDENCE AND UNCERTAINTY BOTH PRESERVED + +- £2m survives as `"£2M/year (unverified)"` with explicit qualification +- n_savings_realism stays `kind=unknown / status=unknown` — identity and unresolved nature both preserved +- Confidence set to `low` — appropriate for unverified evidence on an uncertainty node +- Answer meaning supportCategory = `"uncertain"` — correct semantic interpretation +- No duplicate nodes, no resolved unknown nodes +- Selected question continues investigating the realism concern + +### Does the graph still clearly represent realism as unresolved? + +**YES.** The node kind remains `unknown`, status remains `unknown`, and value contains the explicit qualification "(unverified)". Confidence is `low`. There are zero `resolvedUnknownNodeIds`. An interrogative selectedQuestion pointing to this same node confirms ongoing investigation targeting. + +--- + +## What the engine understood correctly + +1. **Evidence preservation:** Extracted the £2M/year figure from prose and stored it on the existing uncertainty node rather than discarding or inventing a new node. +2. **Qualification embedding:** The value includes "(unverified)" — the model did not strip the qualification when storing evidence. +3. **Semantic category:** Labeled supportCategory as `"uncertain"` rather than `"strong"` or `"established"`. +4. **No fabrication:** possibleInference was null — no invented lease, disruption, or cost assumptions. +5. **Status stability:** Status remained `unknown` (not shifted to `provisional`) — unlike 58A.2 where this was the core failure. +6. **Open resolution:** Did not resolve n_savings_realism; resolutionGuidance = `"may_resolve"` correctly reflects the ongoing need for verification. +7. **Question continuity:** Selected question re-targets the existing node's label rather than inventing a new uncertainty. + +## What it overstated, weakened, or lost + +**Nothing significant.** The update was fully correct for the constraints of this case. One minor note: `structuralActionRequired` is `true` despite no structural change (no new/removed nodes or edges). This flag means "a follow-up structural action may be needed" but does not indicate a failure — it is a forward-looking directive, not a description of what was done wrong. + +--- + +## What this establishes + +1. The engine **can** preserve a concrete figure with its qualification when the existing uncertainty node remains the appropriate target. +2. Status `unknown` is stable across updates even when value transitions from `null` to a qualified string — unlike the 58A.2 failure path. +3. Qualification embedded in `newValue` (e.g., `"£2M/year (unverified)"`) survives as persistent evidence that realism remains unconfirmed. +4. Answer meaning extraction (`userSupportedMeaning`, `supportCategory: uncertain`, `resolutionGuidance: may_resolve`) aligns correctly with the user's actual semantics. + +## What this does NOT prove + +1. This is a **single controlled case** — one answer, one model invocation. It does not prove stability across different answers or models. +2. It does not test whether `structuralActionRequired=true` with no structural change causes issues in subsequent turns. +3. It does not test what happens when the user's qualification changes (e.g., from "unverified" to "verified"). +4. It does not test interaction with other uncertainty nodes (58A.1's scenario where multiple unknowns compete). +5. Value format `"£2M/year (unverified)"` uses prose — whether numeric `2000000` would work equally well is untested here. + +--- + +## Production code changed: NO +## Prompt changed: NO +## Validator changed: NO +## Harness changed: NO +## Vitest run: NO +## Ollama calls beyond harness count: 0 +## Dev server disturbed: NO