From c3c535114310132f33a735a294eb1f1fff00a7a0 Mon Sep 17 00:00:00 2001 From: robbond Date: Wed, 12 Aug 2026 16:51:39 +0100 Subject: [PATCH] experiment: test trade-off decomposition --- docs/current-handoff.md | 29 ++++++ docs/experiment-59a2.md | 221 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 250 insertions(+) create mode 100644 docs/experiment-59a2.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 3a3ea5f..e217c80 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -316,6 +316,35 @@ The engine correctly closed savings-realism, preserved verified evidence, repres ## Ollama calls beyond harness count: 0 ## Dev server disturbed: NO +### Experiment 59A.2 — Trade-off Decomposition: Known Consequence vs Uncertain Consequence + +**Branch:** `feature/question-formulation-v0.24` +**Date:** 2026-08-12 +**Status:** Complete +**Following:** 59A.1 which established the boundary shift but bundled two consequences into one node. This tests whether the engine can separate a *known* consequence (engineers leaving) from an *uncertain* consequence (delivery impact). + +**Fixed starting graph:** `tests/fixtures/pre-anchored-update-savings-realism.json` +**Fixed answer:** "I am comfortable that the £2 million annual saving is real. We know we would lose two senior engineers if we relocate. What I don't know is whether losing them would materially delay delivery, or by how much." + +**Execution:** qwen-claude:latest at http://192.168.1.111:11434. startCalls=0, updateCalls=1, totalCalls=1. + +**Result:** UPDATE 422 rejected at `proposal_compatibility` — "selectedQuestion must be a single non-compound question". No selectedQuestion was produced (null). + +**Reasoning assessment from rejected snapshot:** +- Savings-realism: CLOSED CORRECTLY (resolved, in resolvedUnknownNodeIds) +- £2m annual saving: PRESERVED AS ACCEPTED EVIDENCE (in userSupportedMeaning) +- Two senior engineers leaving: REPRESENTED BUT LEFT UNCERTAIN (extracted in meaning but no separate structural node for the known fact) +- Delivery impact: REPRESENTED AS UNRESOLVED (dedicated unknown node n_delivery_delay_impact created) +- Relationship: PARTIALLY LINKED (causal link encoded in free text within one node, not as typed edge) +- Granularity: COLLAPSED INTO UMBRELLA UNKNOWN (both consequences bundled into one unknown node: "Impact of losing two senior engineers on delivery timelines") +- Next question: NONE (null — caused rejection) + +**Classification: B — PARTIAL DECOMPOSITION** + +The engine extracted all three semantic elements in userSupportedMeaning and created a dedicated unknown for delivery impact. However, the known-vs-uncertain distinction was not preserved structurally — both consequences were compressed into one unresolved-question frame. The selectedQuestion was null, triggering validator rejection. Full record in `docs/experiment-59a2.md`. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. + +--- + ### Experiment 58B.2 — Verified Uncertainty Resolution **Objective:** When the user explicitly verifies a figure and states confidence in its realism, does the engine resolve the existing savings-realism uncertainty rather than merely changing its value? diff --git a/docs/experiment-59a2.md b/docs/experiment-59a2.md new file mode 100644 index 0000000..e6f1941 --- /dev/null +++ b/docs/experiment-59a2.md @@ -0,0 +1,221 @@ +# Experiment 59A.2 — Trade-off Decomposition: Known Consequence vs Uncertain Consequence + +**Branch:** `feature/question-formulation-v0.24` +**Date:** 2026-08-12 +**Status:** Complete + +--- + +## Objective + +When one consequence is stated as known and a second consequence remains uncertain, does the engine represent them separately rather than collapsing both into one broad "is it worth it?" unknown? + +This follows from 59A.1 which successfully shifted to trade-off reasoning but bundled key-engineer loss and delivery slowdown into one umbrella node. + +--- + +## Context route + +Read only: +- `docs/current-handoff.md` (section 59A.1) +- Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json` +- Harness: `scripts/reproduce-multi-turn-investigation.mjs` + +Do not load 57J history. Do not investigate unrelated test failures. + +--- + +## Fixed starting graph + +Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json` + +Existing unresolved question: +``` +n_savings_realism — Are the projected office savings from relocation realistic? — status = unknown +``` + +--- + +## Fixed answer + +``` +I am comfortable that the £2 million annual saving is real. We know we would lose two senior engineers if we relocate. What I don't know is whether losing them would materially delay delivery, or by how much. +``` + +The answer contains three semantically different things: +1. **Resolved factual uncertainty:** £2 million annual saving is real +2. **Known consequence:** two senior engineers would leave +3. **Unresolved consequence:** whether that loss materially delays delivery, and by how much + +The engine should not treat all three as equivalent uncertainty. + +--- + +## Execution + +**Host/model:** qwen-claude:latest at http://192.168.1.111:11434 +**Calls:** startCalls=0, updateCalls=1, totalCalls=1 +**Retries:** 0 + +--- + +## Results + +### UPDATE output + +``` +HTTP status: 422 +Stage: proposal_compatibility +Error: "selectedQuestion must be a single non-compound question" +``` + +### Rejected proposal snapshot (from diagnostics.rejectedProposalSnapshot) + +#### answerMeaning + +- `userSupportedMeaning`: "The £2 million annual office savings are confirmed as real. However, relocating would result in losing two senior engineers, and there is unresolved uncertainty regarding whether this loss would materially delay delivery or by what extent." +- `possibleInference`: null + +#### Proposal mutations + +``` +updatedNodes: [ + { "nodeId": "n_savings_realism", "newValue": null } +] +resolvedUnknownNodeIds: ["n_savings_realism"] +addedNodes: [ + { + "id": "n_delivery_delay_impact", + "kind": "unknown", + "label": "Impact of losing two senior engineers on delivery timelines", + "description": "Uncertainty about whether losing two senior engineers would materially delay engineering delivery, and by how much, because resolving this is needed to decide if the £2M relocation savings justify the potential project delays.", + "parentId": null, + "dependsOn": ["n_relocation_state"], + "affects": [], + "childIds": [] + } +] +addedEdges: [ + { + "fromNodeId": "n_delivery_delay_impact", + "toNodeId": "n_relocation_state", + "relationship": "depends_on" + } +] +``` + +#### selectedQuestion + +`null` (no selected question produced — this caused the rejection) + +--- + +## Reasoning Assessment + +### Savings-realism uncertainty + +- `n_savings_realism`: status unknown → resolved (included in resolvedUnknownNodeIds) +- **Classification: CLOSED CORRECTLY** + +The explicit "comfortable...real" language triggered resolution. No duplicate or re-asking. + +### £2m annual saving + +- Preserved in userSupportedMeaning: "The £2 million annual office savings are confirmed as real." +- **Classification: PRESERVED AS ACCEPTED EVIDENCE** + +The full figure (£2 million), time unit (annual), and confirmation status ("confirmed") survived. + +### Two senior engineers leaving + +- Extracted in userSupportedMeaning: "relocating would result in losing two senior engineers" +- No separate structural node created for this known fact +- **Classification: REPRESENTED BUT LEFT UNCERTAIN** — the model extracted it as part of a single meaning sentence rather than as a standalone known-consequence assertion. The phrase does not use tentative language ("might lose"), but it is also not separately structured. + +### Delivery impact + +- A dedicated unknown node was created: `n_delivery_delay_impact` + - label: "Impact of losing two senior engineers on delivery timelines" + - kind=unknown, status=unknown +- Description captures the causal link explicitly: "Uncertainty about whether losing two senior engineers would materially delay engineering delivery" +- **Classification: REPRESENTED AS UNRESOLVED** + +### Causal/dependency relationship + +- The node label references "losing two senior engineers" and the description links it to "materially delay engineering delivery" +- The causal chain is encoded in free text within the node's label and description, not as a typed edge +- **Classification: PARTIALLY LINKED** — semantically present but structurally flattened into one node rather than represented as a typed relationship between two distinct nodes. + +### Granularity + +The model did NOT separate the known consequence (engineers leaving) from the uncertain consequence (delivery impact). Instead, it created ONE unknown node that bundles both: "Impact of losing two senior engineers on delivery timelines." + +This is structurally one node containing both consequences — not a separation between a known-fact assertion and an unresolved-uncertainty. + +**Classification: COLLAPSED INTO UMBRELLA UNKNOWN** + +### Next question + +No selectedQuestion was produced (null). The rejection was caused by the validator requiring "a single non-compound question." + +**Classification: NONE** + +--- + +## Classification: B — PARTIAL DECOMPOSITION + +### Why: + +The engine correctly closed savings-realism, preserved £2m as accepted evidence, and created a dedicated node for delivery impact. However, it did not represent the known consequence ("two senior engineers will leave") separately from the unresolved consequence (delivery delay). Instead, both were collapsed into one unknown node whose label frames the entire issue as an unresolved question ("Impact of losing two senior engineers on delivery timelines"). This means the known fact that engineers *will* leave is structurally embedded inside a node that represents only *what the delivery impact will be* — which is subtly different but still bundles the known and the unknown. + +The selectedQuestion was null, causing a rejection at proposal_compatibility — this is the "apparatus" aspect of the partial result. + +### Did the engine preserve "two senior engineers will leave" as known: +PARTIAL — extracted in userSupportedMeaning without tentative language, but not structured as an independent known-consequence node. + +### Did it preserve delivery impact as uncertain: +YES — dedicated unknown node created with status=unknown. + +### Did it keep those epistemic states distinct: +NO — both consequences are bundled into one structural node. + +### What the engine understood correctly: + +1. Savings realism is resolved (correct resolution trigger) +2. £2m/year saving is verified evidence +3. The delivery impact from engineer loss is an unresolved question worth investigating +4. The causal link between engineer loss and delivery delay was captured in text +5. No redundant investigation of the resolved savings question + +### What it flattened or misclassified: + +1. **Epistemic states collapsed.** "We know we would lose two senior engineers" (known) and "What I don't know is whether losing them would materially delay delivery" (uncertain) were bundled into one unknown node. The node does not distinguish between what is known and what remains uncertain about those engineers. +2. **selectedQuestion was null.** The model did not produce any selected question, triggering the compound-question validator rejection. This may indicate the model recognized it was generating a complex/unanswerable query and abstained from producing one. + +### What uncertainty it chose to pursue next: +NONE — no question produced (rejection). + +### Was that the best available unresolved question: +DEBATABLE — even if produced, the question would need to distinguish "will engineers leave?" (known) from "what is the delivery impact?" (uncertain). The model appeared to struggle with this distinction. + +### What this establishes: + +1. The engine CAN extract all three semantic elements (resolved savings, known engineer loss, uncertain delivery) in userSupportedMeaning +2. A dedicated unknown node for delivery impact can be created +3. However, the known-vs-unknown distinction was not preserved structurally — both consequences were compressed into one unresolved-question frame + +### What this does NOT prove: + +1. Whether a different model or prompt variant would separate the epistemic states more cleanly +2. Stability across repeated runs +3. Whether the selectedQuestion failure is deterministic or cold-start variance + +--- + +## Production code changed: NO +## Prompt changed: NO +## Validator changed: NO +## Harness changed: NO +## Vitest run: NO +## Ollama calls beyond harness count: 0 +## Dev server disturbed: NO