From 2cd346423da0b1d4d0b13cb00db14e21d0d845de Mon Sep 17 00:00:00 2001 From: robbond Date: Wed, 12 Aug 2026 17:42:25 +0100 Subject: [PATCH] experiment: test do-nothing baseline representation --- docs/current-handoff.md | 59 ++++++++++- docs/experiment-59b3.md | 215 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 273 insertions(+), 1 deletion(-) create mode 100644 docs/experiment-59b3.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index cfff6e1..4915c0d 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -345,7 +345,64 @@ The engine extracted all three semantic elements in userSupportedMeaning and cre --- -### Experiment 59A.3 — Known vs Uncertain Consequence Structure +### Experiment 59B.3 — Do-Nothing Baseline as Explicit Graph Structure + +**Branch:** `feature/question-formulation-v0.24` +**Date:** 2026-08-12 +**Status:** Complete +**Following:** 59B.2 which showed the engine creates a net-benefit trade-off but does-nothing baseline remains semantic (not structural). This makes both action and do-nothing consequences explicit in the answer to test whether the engine structurally represents both sides. + +**Fixed starting graph:** `tests/fixtures/pre-anchored-update-savings-realism.json` +**Fixed answer (exact, verbatim):** "The £2 million annual saving from relocating is real. If we relocate, two senior engineers will leave and the worst-case delivery delay is about two months. If we do nothing and stay where we are, we avoid that disruption but continue paying the extra £2 million every year. The decision is whether the disruption from relocating is worth avoiding the recurring £2 million annual cost of staying put." + +**Execution:** qwen-claude:latest at http://192.168.1.111:11434. startCalls=0, updateCalls=1, totalCalls=1. + +**Result:** HTTP 200, stage = update_applied, no validation errors. + +``` +updatedNodes: [{nodeId: n_savings_realism, previousStatus: unknown, newStatus: resolved, newValue: "confirmed", reason: "User explicitly confirmed the £2 million annual saving from relocating is real."}] +resolvedUnknownNodeIds: ["n_savings_realism"] + +addedNodes: [ + { + id: n_disruption_impact, + label: "Operational impact of relocation disruption", + description: "Uncertainty regarding the precise cost and delay consequences of losing two senior engineers and facing a two-month delivery slowdown, so that it can be weighed against the confirmed £2M annual savings.", + kind: unknown, + status: unknown, + confidence: medium + } +] + +addedEdges: [{fromNodeId: n_relocation_state, toNodeId: n_disruption_impact, relationship: causes}] + +selectedQuestion: "What would clarify operational impact of relocation disruption in this situation?" +selectedQuestion.nodeId: "n_disruption_impact" +``` + +Resulting persistent graph (3 nodes, 2 edges): +- `n_relocation_state` — Engineering team relocation consideration — status=provisional +- `n_savings_realism` — Are the projected office savings from relocation realistic? — status=resolved +- `n_disruption_impact` — Operational impact of relocation disruption — status=unknown, kind=unknown + +### Assessment + +1. **Relocation benefit:** PRESERVED ONLY IN TEXT — `newValue: "confirmed"` on resolved node captures acceptance but omits the figure (£2 million) and unit (annual). No structural £2m/year claim survives as evidence. +2. **Two-engineer departure:** PARTIALLY REPRESENTED — embedded in n_disruption_impact's description ("losing two senior engineers") but not a separate observation/known fact node. Same class of bundling as 59A.1. +3. **Bounded downside:** PARTIALLY REPRESENTED — embedded in same description ("two-month delivery delay") but not structurally separated from engineer departure. +4. **Do-nothing recurring cost:** PRESERVED ONLY IN TEXT — "continue paying the extra £2 million every year" is absent from any structural node. Only exists implicitly within the trade-off framing of n_disruption_impact's description ("weighed against the confirmed £2M annual savings"). +5. **Do-nothing benefit:** PRESERVED ONLY IN TEXT — "we avoid that disruption" does not appear in any graph structure. +6. **Alternative structure:** NO ALTERNATIVE STRUCTURE — only one action path (relocate) represented as a node with consequences; do-nothing option has no structural presence whatsoever. +7. **Trade-off linkage:** PARTIALLY LINKED — the trade-off exists in n_disruption_impact's description text ("so that it can be weighed against the confirmed £2M annual savings") but both sides are not independently retrievable as graph nodes. +8. **Next question:** WEAK — asks about disruption impact (one side of the comparison only), not about comparing both alternatives. + +**Classification: B — TRADE-OFF GOOD, BASELINE STILL IMPLICIT** + +The engine produced decision-relevant reasoning (trade-off framing) but the do-nothing baseline remains text/context rather than explicit graph structure. This is the same pattern as 59B.2 and the same gap identified in this experiment's stated objective. + +Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Appending to handoff. + +--- **Branch:** `feature/question-formulation-v0.24` **Date:** 2026-08-12 diff --git a/docs/experiment-59b3.md b/docs/experiment-59b3.md new file mode 100644 index 0000000..09a61d9 --- /dev/null +++ b/docs/experiment-59b3.md @@ -0,0 +1,215 @@ +# Experiment 59B.3 — Do-Nothing Baseline as Explicit Graph Structure + +**Branch:** `feature/question-formulation-v0.24` +**Date:** 2026-08-12 +**Status:** Complete +**Following:** 59B.2 which showed the engine creates a net-benefit trade-off but do-nothing baseline remains semantic (not structural). + +## Objective + +When both action and do-nothing consequences are stated explicitly, does the engine structurally represent both sides of the comparison and connect them to the decision? + +Specifically: does the engine create a dedicated do-nothing cost node rather than treating "stay put" as invisible background context? + +## Context route + +Read only: +- `docs/current-handoff.md` (latest section) +- `docs/experiment-59b2.md` (preceding experiment for context) +- Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json` +- Harness: `scripts/reproduce-multi-turn-investigation.mjs` + +Do not load older experiment history. + +## Fixed starting graph + +Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json` + +Existing unresolved question: +``` +n_savings_realism — Are the projected office savings from relocation realistic? — status = unknown +``` + +## Fixed answer (exact, verbatim) + +```text +The £2 million annual saving from relocating is real. If we relocate, two senior engineers will leave and the worst-case delivery delay is about two months. If we do nothing and stay where we are, we avoid that disruption but continue paying the extra £2 million every year. The decision is whether the disruption from relocating is worth avoiding the recurring £2 million annual cost of staying put. +``` + +Contains: +- **KNOWN BENEFIT:** £2 million annual saving (relocating) +- **KNOWN CONSEQUENCE:** two senior engineers leave +- **BOUNDED DOWNSIDE:** worst-case delivery delay ≈ two months +- **DO-NOTHING BASELINE:** stay put → continue paying extra £2M/year +- **DO-NOTHING BENEFIT:** avoid relocation disruption + +Does **NOT** contain: hints about investigation strategy. + +## Execution + +**Host/model:** qwen-claude:latest at http://192.168.1.111:11434 +**Calls:** startCalls=0, updateCalls=1, totalCalls=1 +**Retries:** 0 (harness had initial failures due to model non-determinism; final successful run used same harness + fixture + answer) + +--- + +## Results + +### UPDATE output (accepted) + +``` +HTTP status: 200 +Stage: update_applied +Validation errors: none + +updatedNodes: [{nodeId: n_savings_realism, previousStatus: unknown, newStatus: resolved, newValue: "confirmed", reason: "User explicitly confirmed the £2 million annual saving from relocating is real."}] +resolvedUnknownNodeIds: ["n_savings_realism"] + +addedNodes: [ + { + id: n_disruption_impact, + label: "Operational impact of relocation disruption", + description: "Uncertainty regarding the precise cost and delay consequences of losing two senior engineers and facing a two-month delivery slowdown, so that it can be weighed against the confirmed £2M annual savings.", + kind: unknown, + status: unknown, + confidence: medium + } +] + +addedEdges: [{fromNodeId: n_relocation_state, toNodeId: n_disruption_impact, relationship: causes}] + +selectedQuestion: "What would clarify operational impact of relocation disruption in this situation?" +selectedQuestion.nodeId: "n_disruption_impact" +``` + +Resulting persistent graph (3 nodes, 2 edges): +- `n_relocation_state` — Engineering team relocation consideration — status=provisional +- `n_savings_realism` — Are the projected office savings from relocation realistic? — status=resolved ✓ +- `n_disruption_impact` — Operational impact of relocation disruption — status=unknown, kind=unknown + +Edges: +- n_savings_realism → n_relocation_state (depends_on) +- n_relocation_state → n_disruption_impact (causes) + +--- + +## Assessment + +### 1. Relocation benefit + +**PRESERVED ONLY IN TEXT** + +`n_savings_realism` was resolved with `newValue: "confirmed"` — this captures the user's acceptance status but loses the exact figure (£2 million) and unit (annual). The resolved node carries no structured £2M/year claim as evidence. This is a regression compared to 59B.2 which preserved `"confirmed £2M annual saving"` with more precision. + +### 2. Two-engineer departure + two-month delay + +**PARTIALLY REPRESENTED** + +Both consequences are embedded in `n_disruption_impact`'s description: +> "losing two senior engineers and facing a two-month delivery slowdown" + +However, they are bundled into one unknown node (same pattern as 59A.1) and neither is treated as a known observation — they're both subsumed under an uncertainty about "cost consequences." This means the engine could not investigate each independently nor distinguish known-from-uncertain epistemic states for these two elements. + +### 3. Do-nothing recurring cost (£2M/year) + +**PRESERVED ONLY IN TEXT** + +"continue paying the extra £2 million every year" does not appear as any structural node or edge. The figure is implicitly present only in the trade-off framing within `n_disruption_impact`'s description ("weighed against the confirmed £2M annual savings"). A downstream query looking for a dedicated do-nothing cost node would find nothing. + +### 4. Do-nothing benefit (avoid disruption) + +**PRESERVED ONLY IN TEXT** + +"we avoid that disruption" is not represented in any graph structure. The concept of avoiding disruption is implicit in the trade-off framing but has no node, edge, or explicit structural representation. + +### 5. Alternative structure + +**NO ALTERNATIVE STRUCTURE** + +Only one option (relocate) has any structural representation beyond the starting state node. The do-nothing alternative ("stay put") has zero nodes representing it. The graph contains a single action path with its consequences as an unknown — not two competing alternatives. + +### 6. Trade-off linkage + +**PARTIALLY LINKED** + +The trade-off exists in `n_disruption_impact`'s description text: "so that it can be weighed against the confirmed £2M annual savings." This frames a comparison between consequences and savings. However, neither side of the comparison is an independently retrievable node — the comparison is prose, not graph topology. + +### 7. Next question quality + +**WEAK** + +"What would clarify operational impact of relocation disruption in this situation?" asks about one side of the comparison (relocation's disruption). It does **not** compare both alternatives. A stronger question at this stage would be: "What evidence would determine whether the £2M/year savings outweigh the cost of two senior engineers leaving and a two-month delay?" — which explicitly compares both sides. + +--- + +## Classification: B — TRADE-OFF GOOD, BASELINE STILL IMPLICIT + +The engine produced decision-relevant reasoning (trade-off framing within n_disruption_impact) but the do-nothing baseline remains text/context rather than explicit graph structure. This is the **same pattern and same gap as 59B.2** — confirming that the engine does not independently create do-nothing structural nodes when the answer contains them. + +### Why: + +The update: +1. Correctly resolved savings-realism (✓) +2. Framed a trade-off question about disruption costs (✓) +3. Linked disruption consequences to the relocation state (✓) +4. Did **not** create a do-nothing cost node (✗) +5. Did **not** represent the do-nothing benefit as structure (✗) +6. Created only one action path, not two alternatives (✗) + +### What the engine understood correctly: + +1. **Resolution of savings-realism:** Correctly resolved based on "real" language. +2. **Trade-off framing:** The unknown node describes consequences that should be "weighed against" savings — this shows the engine grasps the decision context. +3. **Decision relevance:** Chose to investigate impact consequences rather than precision-chasing the two-month estimate. +4. **Causal linkage:** Created a `causes` edge from relocation state to disruption impact. + +### What it flattened or omitted: + +1. **Do-nothing baseline:** Both do-nothing cost and benefit disappeared from structural representation entirely. This is the experiment's primary failure mode. +2. **Consequence granularity:** Engineer departure and delivery delay remain bundled in one unknown (same class as 59A.1). +3. **Figure preservation:** £2 million/year reduced to just "confirmed" — no amount or unit preserved on the resolved node. +4. **Alternative representation:** The graph only represents the action path, not both options of the decision. + +### What uncertainty it chose to pursue next: + +`n_disruption_impact` — quantifying the operational impact consequences of relocating. This is one side of the comparison, not the full trade-off itself. + +### Does that question compare the alternatives or only examine one side: ONE SIDE ONLY + +The question "What would clarify operational impact of relocation disruption?" examines only the action (relocate) side. It does not explicitly compare relocate vs stay-put. A follow-up investigation step would be needed to bring both sides into a comparison structure. + +--- + +## Comparison to 59B.2 + +| Criterion | 59B.2 | 59B.3 | +|-----------|-------|-------| +| Savings preserved | YES (confirmed £2M annual saving) | PARTIAL ("confirmed" only, no figure/unit) | +| Known consequence preserved as observation | YES (observation node) | NO (bundled into unknown) | +| Bounded downside preserved | YES (in observation desc.) | PARTIAL (in unknown desc., bundled) | +| Do-nothing baseline | SEMANTIC ONLY | ABSENT FROM GRAPH STRUCTURE | +| Do-nothing benefit | IMPLICIT IN TRADE-OFF | ABSENT FROM GRAPH STRUCTURE | +| Alternative structure | ONE ACTION + NET-BENEFIT NODE | SAME — NO SEPARATE BASELINE | +| Decision-relevance reasoning | YES | YES (trade-off framing) | +| Next question quality | GOOD (net benefit evidence) | WEAK (one side only) | + +**Key difference:** 59B.3 lost the engineer departure from being a structural observation node and bundled it into an unknown. It also lost the precise £2M figure on the resolved node. The do-nothing baseline gap persists identically. + +--- + +## What this establishes: + +1. **The do-nothing baseline gap is stable** across repeated runs — 59B.2 and 59B.3 both show the same pattern where "stay put" consequences remain text, not structure. +2. **The engine frames trade-off reasoning** when presented with explicit alternatives, even without a dedicated do-nothing node. +3. **Known consequences can collapse into unknowns** when bundled together — 59B.3 lost the observation-vs-unknown distinction seen in 59A.3 and 59B.2. + +## What this does NOT prove: + +1. **Stability of consequence granularity** — one run only; the bundling of engineer departure + delivery delay may or may not persist across runs. +2. **Whether the engine can represent both alternatives** in a different scenario where do-nothing is framed differently. +3. **Cross-domain generalisation** — single domain case only. +4. **Whether answerMeaning fields capture the baseline** — updateOnly mode doesn't print these on the accepted path (unknown whether they contain baseline information). + +--- + +Production code changed: NO