216 lines
11 KiB
Markdown
216 lines
11 KiB
Markdown
# Experiment 59B.3 — Do-Nothing Baseline as Explicit Graph Structure
|
|
|
|
**Branch:** `feature/question-formulation-v0.24`
|
|
**Date:** 2026-08-12
|
|
**Status:** Complete
|
|
**Following:** 59B.2 which showed the engine creates a net-benefit trade-off but do-nothing baseline remains semantic (not structural).
|
|
|
|
## Objective
|
|
|
|
When both action and do-nothing consequences are stated explicitly, does the engine structurally represent both sides of the comparison and connect them to the decision?
|
|
|
|
Specifically: does the engine create a dedicated do-nothing cost node rather than treating "stay put" as invisible background context?
|
|
|
|
## Context route
|
|
|
|
Read only:
|
|
- `docs/current-handoff.md` (latest section)
|
|
- `docs/experiment-59b2.md` (preceding experiment for context)
|
|
- Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
|
- Harness: `scripts/reproduce-multi-turn-investigation.mjs`
|
|
|
|
Do not load older experiment history.
|
|
|
|
## Fixed starting graph
|
|
|
|
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
|
|
|
Existing unresolved question:
|
|
```
|
|
n_savings_realism — Are the projected office savings from relocation realistic? — status = unknown
|
|
```
|
|
|
|
## Fixed answer (exact, verbatim)
|
|
|
|
```text
|
|
The £2 million annual saving from relocating is real. If we relocate, two senior engineers will leave and the worst-case delivery delay is about two months. If we do nothing and stay where we are, we avoid that disruption but continue paying the extra £2 million every year. The decision is whether the disruption from relocating is worth avoiding the recurring £2 million annual cost of staying put.
|
|
```
|
|
|
|
Contains:
|
|
- **KNOWN BENEFIT:** £2 million annual saving (relocating)
|
|
- **KNOWN CONSEQUENCE:** two senior engineers leave
|
|
- **BOUNDED DOWNSIDE:** worst-case delivery delay ≈ two months
|
|
- **DO-NOTHING BASELINE:** stay put → continue paying extra £2M/year
|
|
- **DO-NOTHING BENEFIT:** avoid relocation disruption
|
|
|
|
Does **NOT** contain: hints about investigation strategy.
|
|
|
|
## Execution
|
|
|
|
**Host/model:** qwen-claude:latest at http://192.168.1.111:11434
|
|
**Calls:** startCalls=0, updateCalls=1, totalCalls=1
|
|
**Retries:** 0 (harness had initial failures due to model non-determinism; final successful run used same harness + fixture + answer)
|
|
|
|
---
|
|
|
|
## Results
|
|
|
|
### UPDATE output (accepted)
|
|
|
|
```
|
|
HTTP status: 200
|
|
Stage: update_applied
|
|
Validation errors: none
|
|
|
|
updatedNodes: [{nodeId: n_savings_realism, previousStatus: unknown, newStatus: resolved, newValue: "confirmed", reason: "User explicitly confirmed the £2 million annual saving from relocating is real."}]
|
|
resolvedUnknownNodeIds: ["n_savings_realism"]
|
|
|
|
addedNodes: [
|
|
{
|
|
id: n_disruption_impact,
|
|
label: "Operational impact of relocation disruption",
|
|
description: "Uncertainty regarding the precise cost and delay consequences of losing two senior engineers and facing a two-month delivery slowdown, so that it can be weighed against the confirmed £2M annual savings.",
|
|
kind: unknown,
|
|
status: unknown,
|
|
confidence: medium
|
|
}
|
|
]
|
|
|
|
addedEdges: [{fromNodeId: n_relocation_state, toNodeId: n_disruption_impact, relationship: causes}]
|
|
|
|
selectedQuestion: "What would clarify operational impact of relocation disruption in this situation?"
|
|
selectedQuestion.nodeId: "n_disruption_impact"
|
|
```
|
|
|
|
Resulting persistent graph (3 nodes, 2 edges):
|
|
- `n_relocation_state` — Engineering team relocation consideration — status=provisional
|
|
- `n_savings_realism` — Are the projected office savings from relocation realistic? — status=resolved ✓
|
|
- `n_disruption_impact` — Operational impact of relocation disruption — status=unknown, kind=unknown
|
|
|
|
Edges:
|
|
- n_savings_realism → n_relocation_state (depends_on)
|
|
- n_relocation_state → n_disruption_impact (causes)
|
|
|
|
---
|
|
|
|
## Assessment
|
|
|
|
### 1. Relocation benefit
|
|
|
|
**PRESERVED ONLY IN TEXT**
|
|
|
|
`n_savings_realism` was resolved with `newValue: "confirmed"` — this captures the user's acceptance status but loses the exact figure (£2 million) and unit (annual). The resolved node carries no structured £2M/year claim as evidence. This is a regression compared to 59B.2 which preserved `"confirmed £2M annual saving"` with more precision.
|
|
|
|
### 2. Two-engineer departure + two-month delay
|
|
|
|
**PARTIALLY REPRESENTED**
|
|
|
|
Both consequences are embedded in `n_disruption_impact`'s description:
|
|
> "losing two senior engineers and facing a two-month delivery slowdown"
|
|
|
|
However, they are bundled into one unknown node (same pattern as 59A.1) and neither is treated as a known observation — they're both subsumed under an uncertainty about "cost consequences." This means the engine could not investigate each independently nor distinguish known-from-uncertain epistemic states for these two elements.
|
|
|
|
### 3. Do-nothing recurring cost (£2M/year)
|
|
|
|
**PRESERVED ONLY IN TEXT**
|
|
|
|
"continue paying the extra £2 million every year" does not appear as any structural node or edge. The figure is implicitly present only in the trade-off framing within `n_disruption_impact`'s description ("weighed against the confirmed £2M annual savings"). A downstream query looking for a dedicated do-nothing cost node would find nothing.
|
|
|
|
### 4. Do-nothing benefit (avoid disruption)
|
|
|
|
**PRESERVED ONLY IN TEXT**
|
|
|
|
"we avoid that disruption" is not represented in any graph structure. The concept of avoiding disruption is implicit in the trade-off framing but has no node, edge, or explicit structural representation.
|
|
|
|
### 5. Alternative structure
|
|
|
|
**NO ALTERNATIVE STRUCTURE**
|
|
|
|
Only one option (relocate) has any structural representation beyond the starting state node. The do-nothing alternative ("stay put") has zero nodes representing it. The graph contains a single action path with its consequences as an unknown — not two competing alternatives.
|
|
|
|
### 6. Trade-off linkage
|
|
|
|
**PARTIALLY LINKED**
|
|
|
|
The trade-off exists in `n_disruption_impact`'s description text: "so that it can be weighed against the confirmed £2M annual savings." This frames a comparison between consequences and savings. However, neither side of the comparison is an independently retrievable node — the comparison is prose, not graph topology.
|
|
|
|
### 7. Next question quality
|
|
|
|
**WEAK**
|
|
|
|
"What would clarify operational impact of relocation disruption in this situation?" asks about one side of the comparison (relocation's disruption). It does **not** compare both alternatives. A stronger question at this stage would be: "What evidence would determine whether the £2M/year savings outweigh the cost of two senior engineers leaving and a two-month delay?" — which explicitly compares both sides.
|
|
|
|
---
|
|
|
|
## Classification: B — TRADE-OFF GOOD, BASELINE STILL IMPLICIT
|
|
|
|
The engine produced decision-relevant reasoning (trade-off framing within n_disruption_impact) but the do-nothing baseline remains text/context rather than explicit graph structure. This is the **same pattern and same gap as 59B.2** — confirming that the engine does not independently create do-nothing structural nodes when the answer contains them.
|
|
|
|
### Why:
|
|
|
|
The update:
|
|
1. Correctly resolved savings-realism (✓)
|
|
2. Framed a trade-off question about disruption costs (✓)
|
|
3. Linked disruption consequences to the relocation state (✓)
|
|
4. Did **not** create a do-nothing cost node (✗)
|
|
5. Did **not** represent the do-nothing benefit as structure (✗)
|
|
6. Created only one action path, not two alternatives (✗)
|
|
|
|
### What the engine understood correctly:
|
|
|
|
1. **Resolution of savings-realism:** Correctly resolved based on "real" language.
|
|
2. **Trade-off framing:** The unknown node describes consequences that should be "weighed against" savings — this shows the engine grasps the decision context.
|
|
3. **Decision relevance:** Chose to investigate impact consequences rather than precision-chasing the two-month estimate.
|
|
4. **Causal linkage:** Created a `causes` edge from relocation state to disruption impact.
|
|
|
|
### What it flattened or omitted:
|
|
|
|
1. **Do-nothing baseline:** Both do-nothing cost and benefit disappeared from structural representation entirely. This is the experiment's primary failure mode.
|
|
2. **Consequence granularity:** Engineer departure and delivery delay remain bundled in one unknown (same class as 59A.1).
|
|
3. **Figure preservation:** £2 million/year reduced to just "confirmed" — no amount or unit preserved on the resolved node.
|
|
4. **Alternative representation:** The graph only represents the action path, not both options of the decision.
|
|
|
|
### What uncertainty it chose to pursue next:
|
|
|
|
`n_disruption_impact` — quantifying the operational impact consequences of relocating. This is one side of the comparison, not the full trade-off itself.
|
|
|
|
### Does that question compare the alternatives or only examine one side: ONE SIDE ONLY
|
|
|
|
The question "What would clarify operational impact of relocation disruption?" examines only the action (relocate) side. It does not explicitly compare relocate vs stay-put. A follow-up investigation step would be needed to bring both sides into a comparison structure.
|
|
|
|
---
|
|
|
|
## Comparison to 59B.2
|
|
|
|
| Criterion | 59B.2 | 59B.3 |
|
|
|-----------|-------|-------|
|
|
| Savings preserved | YES (confirmed £2M annual saving) | PARTIAL ("confirmed" only, no figure/unit) |
|
|
| Known consequence preserved as observation | YES (observation node) | NO (bundled into unknown) |
|
|
| Bounded downside preserved | YES (in observation desc.) | PARTIAL (in unknown desc., bundled) |
|
|
| Do-nothing baseline | SEMANTIC ONLY | ABSENT FROM GRAPH STRUCTURE |
|
|
| Do-nothing benefit | IMPLICIT IN TRADE-OFF | ABSENT FROM GRAPH STRUCTURE |
|
|
| Alternative structure | ONE ACTION + NET-BENEFIT NODE | SAME — NO SEPARATE BASELINE |
|
|
| Decision-relevance reasoning | YES | YES (trade-off framing) |
|
|
| Next question quality | GOOD (net benefit evidence) | WEAK (one side only) |
|
|
|
|
**Key difference:** 59B.3 lost the engineer departure from being a structural observation node and bundled it into an unknown. It also lost the precise £2M figure on the resolved node. The do-nothing baseline gap persists identically.
|
|
|
|
---
|
|
|
|
## What this establishes:
|
|
|
|
1. **The do-nothing baseline gap is stable** across repeated runs — 59B.2 and 59B.3 both show the same pattern where "stay put" consequences remain text, not structure.
|
|
2. **The engine frames trade-off reasoning** when presented with explicit alternatives, even without a dedicated do-nothing node.
|
|
3. **Known consequences can collapse into unknowns** when bundled together — 59B.3 lost the observation-vs-unknown distinction seen in 59A.3 and 59B.2.
|
|
|
|
## What this does NOT prove:
|
|
|
|
1. **Stability of consequence granularity** — one run only; the bundling of engineer departure + delivery delay may or may not persist across runs.
|
|
2. **Whether the engine can represent both alternatives** in a different scenario where do-nothing is framed differently.
|
|
3. **Cross-domain generalisation** — single domain case only.
|
|
4. **Whether answerMeaning fields capture the baseline** — updateOnly mode doesn't print these on the accepted path (unknown whether they contain baseline information).
|
|
|
|
---
|
|
|
|
Production code changed: NO
|