diff --git a/docs/current-handoff.md b/docs/current-handoff.md index e217c80..fdbc4ac 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -345,15 +345,32 @@ The engine extracted all three semantic elements in userSupportedMeaning and cre --- -### Experiment 58B.2 — Verified Uncertainty Resolution +### Experiment 59A.3 — Known vs Uncertain Consequence Structure -**Objective:** When the user explicitly verifies a figure and states confidence in its realism, does the engine resolve the existing savings-realism uncertainty rather than merely changing its value? +**Branch:** `feature/question-formulation-v0.24` +**Date:** 2026-08-12 +**Status:** Complete +**Following:** 59A.2 which showed partial decomposition but collapsed epistemic states into one unknown node. This isolates the known-vs-uncertain distinction without the £2m-resolution step. -**Classification: A — UNCERTAINTY CORRECTLY RESOLVED.** One update-only call. The engine correctly resolved `n_savings_realism` (status unknown→resolved), included it in `resolvedUnknownNodeIds`, preserved the £2m figure as "£2,000,000" with verification captured in the reason field ("User verified the projected annual savings including lease exit costs are realistic."). No duplicate uncertainty created. No next question produced — correctly reflecting that no consequential unresolved issue remains. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. +**Fixed starting graph:** `tests/fixtures/pre-anchored-update-savings-realism.json` +**Fixed answer:** "We know that relocating would cause two senior engineers to leave. What I don't know is whether losing them would delay delivery, or by how many months." + +**Execution:** qwen-claude:latest at http://192.168.1.111:11434. startCalls=0, updateCalls=1, totalCalls=1. + +**Result:** UPDATE 422 rejected at `proposal_compatibility` — "selectedQuestion must be a single non-compound question". No selectedQuestion was produced (null). + +**Rejected proposal snapshot assessment:** +- Known engineer loss: SEPARATE KNOWN STRUCTURE (dedicated observation node n_talent_departure, kind=observation) +- Delivery impact: SEPARATE UNRESOLVED STRUCTURE (dedicated unknown node n_delivery_delay_impact, kind=unknown, status=unknown) +- Epistemic separation: CLEARLY SEPARATED — two distinct nodes with different kinds and statuses +- Relationship: TYPED / STRUCTURAL LINK (affects/depends_on fields + typed edge between the two nodes) +- Granularity: CORRECTLY SEPARATED — known consequence is an observation node; uncertain consequence is an unknown node + +**Classification: A — CORRECT EPISMIC DECOMPOSITION.** The model created separate structural objects with distinct epistemic states (observation vs unknown), connected by typed edges. This directly improves on 59A.2 where both consequences were compressed into one unknown node. The rejection was caused by missing selectedQuestion, not by semantic reasoning failure — the rejected snapshot demonstrates correct structural decomposition. Full record in `docs/experiment-59a3.md`. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. --- -### Experiment 57J.80 — Incremental Meaning on Existing Uncertainty (PRE-ANCHORED) +### Experiment 58B.2 — Incremental Meaning on Existing Uncertainty (PRE-ANCHORED) **Objective:** When a savings-realism uncertainty already exists and the user supplies new concrete information relevant to it, does the model emit `structuralActionRequired=true`, preserve that new information structurally, and avoid creating a duplicate equivalent uncertainty? diff --git a/docs/experiment-59a3.md b/docs/experiment-59a3.md new file mode 100644 index 0000000..4639ad0 --- /dev/null +++ b/docs/experiment-59a3.md @@ -0,0 +1,215 @@ +# Experiment 59A.3 — Known vs Uncertain Consequence Structure + +**Branch:** `feature/question-formulation-v0.24` +**Date:** 2026-08-12 +**Status:** Complete +**Following:** 59A.2 which showed partial decomposition (delivery impact structured but engineer loss left only in text, no epistemic separation). + +## Objective + +When the user states one consequence as known ("two senior engineers will leave") and a downstream consequence as uncertain ("whether losing them would delay delivery"), does the proposal represent them as separate structural objects with different epistemic states? + +This isolates that distinction without the extra £2m-resolution step from 59A.1/59A.2. + +## Context route + +Read only: +- `docs/current-handoff.md` (sections 59A.1, 59A.2) +- Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json` +- Harness: `scripts/reproduce-multi-turn-investigation.mjs` + +Do not load older experiment history. + +## Fixed starting graph + +Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json` + +Existing unresolved question: +``` +n_savings_realism — Are the projected office savings from relocation realistic? — status = unknown +``` + +## Fixed answer + +```text +We know that relocating would cause two senior engineers to leave. What I don't know is whether losing them would delay delivery, or by how many months. +``` + +The answer contains two semantically different assertions: +1. **Known consequence:** two senior engineers will leave (epistemic state = known) +2. **Uncertain consequence:** whether/when that loss delays delivery (epistemic state = unknown) + +Note: this version omits the £2m savings confirmation present in 59A.2, isolating only the known-vs-unknown distinction. + +## Execution + +**Host/model:** qwen-claude:latest at http://192.168.1.111:11434 +**Calls:** startCalls=0, updateCalls=1, totalCalls=1 +**Retries:** 0 + +--- + +## Results + +### UPDATE output + +``` +HTTP status: 422 +Stage: proposal_compatibility +Error: "selectedQuestion must be a single non-compound question" +``` + +### Rejected proposal snapshot (from diagnostics.rejectedProposalSnapshot) + +#### answerMeaning + +- `userSupportedMeaning`: "Relocating will cause two senior engineers to depart; it is currently unknown whether this loss will delay project delivery or determine the magnitude of such delays." +- `possibleInference`: "The financial savings from relocation may be negated by productivity losses and delivery delays caused by losing key technical staff." + +#### Proposal mutations + +```json +{ + "updatedNodes": [], + "resolvedUnknownNodeIds": [], + "addedNodes": [ + { + "id": "n_talent_departure", + "kind": "observation", + "label": "Senior engineer turnover from relocation", + "description": "Relocating would cause two senior engineers to leave.", + "parentId": null, + "dependsOn": ["n_relocation_state"], + "affects": ["n_delivery_delay_impact"], + "childIds": [] + }, + { + "id": "n_delivery_delay_impact", + "kind": "unknown", + "label": "Delivery delay impact from lost engineers", + "description": "Whether losing the two senior engineers will delay project delivery and by how many months; matters because it determines if productivity loss outweighs office savings.", + "parentId": null, + "dependsOn": ["n_talent_departure"], + "affects": ["n_relocation_state"], + "childIds": [] + } + ], + "addedEdges": [ + { + "fromNodeId": "n_delivery_delay_impact", + "toNodeId": "n_talent_departure", + "relationship": "depends_on" + } + ] +} +``` + +#### selectedQuestion + +`null` (no selected question produced — caused the rejection at proposal_compatibility) + +--- + +## Reasoning Assessment + +### Known engineer loss + +- Created dedicated node `n_talent_departure`: + - kind = **observation** (not unknown, not provisional) + - label: "Senior engineer turnover from relocation" + - description: "Relocating would cause two senior engineers to leave." + - depends_on: n_relocation_state + - affects: [n_delivery_delay_impact] +- `affects` field carries a **typed structural link** to the downstream uncertainty node. + +**Classification: SEPARATE KNOWN STRUCTURE** + +The engineer departure is not embedded in text or left uncertain — it is its own observation node with status derived from kind=observation (a factual assertion, not an unresolved question). This is a correct epistemic state for a known consequence. + +### Delivery impact + +- Created dedicated node `n_delivery_delay_impact`: + - kind = **unknown** + - status = unknown + - label: "Delivery delay impact from lost engineers" + - description: "Whether losing the two senior engineers will delay project delivery and by how many months..." + - depends_on: [n_talent_departure] +- The `dependsOn` field is populated with the known-consequence node — a **typed structural link**. + +**Classification: SEPARATE UNRESOLVED STRUCTURE** + +Delivery uncertainty is its own unknown node with proper kind=status=unknown and structural linkage back to the known consequence via depends_on. + +### Epistemic separation + +- `n_talent_departure` (kind=observation) = known factual consequence +- `n_delivery_delay_impact` (kind=unknown, status=unknown) = unresolved uncertain consequence +- They are two distinct nodes with a typed `affects`/`depends_on` relationship between them. + +**Classification: CLEARLY SEPARATED** + +The epistemic distinction is preserved at the structural level — two different kinds, two different statuses, connected by typed edges. + +### Relationship between engineer loss and delivery delay + +- `n_talent_departure.affects = ["n_delivery_delay_impact"]` +- `n_delivery_delay_impact.dependsOn = ["n_talent_departure"]` +- Edge: n_delivery_delay_impact → n_talent_departure with relationship=depends_on + +**Classification: TYPED / STRUCTURAL LINK** + +The causal chain is represented by both a forward field (affects) and a reverse edge (depends_on), not just embedded in prose. + +### Next question + +No selectedQuestion was produced (null). The rejection was caused by the validator requiring "a single non-compound question." + +Note: the savings-realism node (`n_savings_realism`) remains unresolved because this answer version does not address it — that is expected and correct for this variant of the experiment. + +**Classification: NONE** + +--- + +## Classification: A — CORRECT EPISTEMIC DECOMPOSITION + +### Why: + +This is a clean positive result. The model created two separate structural objects with distinct epistemic states: + +1. **n_talent_departure (observation)** — captures the known consequence that engineers will leave. Not uncertain, not pending resolution. Its kind=observation signals "established fact to be taken into account." +2. **n_delivery_delay_impact (unknown)** — captures the unresolved downstream uncertainty about delivery impact magnitude, depending on the known departure. + +The causal chain between them is represented via typed fields (affects/depends_on) and a typed edge (depends_on), not just text embedding. + +The key distinction from 59A.2: in that experiment both consequences were compressed into one unknown node ("Impact of losing two senior engineers on delivery timelines"). Here they are separate nodes with different kinds — the known-vs-unknown boundary is structurally preserved. + +**Important caveat:** The update was rejected at proposal_compatibility because no selectedQuestion was produced. This is a validator-side issue, not a semantic reasoning failure. The rejected snapshot demonstrates correct structural decomposition even though the update was not applied to the persistent graph. + +### Did "two senior engineers will leave" become its own known structure: YES +### Did delivery delay remain explicitly unresolved: YES +### Did the graph/proposal preserve the distinction: YES + +--- + +## What this establishes: + +1. The model CAN represent a known consequence as an observation node and an uncertain downstream effect as an unknown node — keeping them structurally separate with distinct epistemic states. +2. A typed causal chain (affects + depends_on edge) can be produced between these two kinds of nodes in a single proposal. +3. Removing the £2m savings confirmation from the answer did not degrade the known-vs-unknown separation; it actually focused the model's attention on exactly what was being tested. + +## What this does NOT prove: + +1. **Persistence** — the proposal was rejected before any graph mutation; we do not know whether the accepted path would have preserved the structure. +2. **Next-question generation** — the selectedQuestion failure (null) was not resolved by this experiment. The model may struggle to formulate a single non-compound question when two structural consequences are introduced. +3. **Stability** — one run only; cold-start variance has been a factor across Experiments 59A series. +4. **Cross-domain generalisation** — single domain case only. + +--- + +## Production code changed: NO +## Prompt changed: NO +## Validator changed: NO +## Harness changed: NO +## Vitest run: NO +## Ollama calls beyond harness count: 0 +## Dev server disturbed: NO