experiment: test do-nothing baseline representation
This commit is contained in:
+58
-1
@@ -345,7 +345,64 @@ The engine extracted all three semantic elements in userSupportedMeaning and cre
|
||||
|
||||
---
|
||||
|
||||
### Experiment 59A.3 — Known vs Uncertain Consequence Structure
|
||||
### Experiment 59B.3 — Do-Nothing Baseline as Explicit Graph Structure
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Date:** 2026-08-12
|
||||
**Status:** Complete
|
||||
**Following:** 59B.2 which showed the engine creates a net-benefit trade-off but does-nothing baseline remains semantic (not structural). This makes both action and do-nothing consequences explicit in the answer to test whether the engine structurally represents both sides.
|
||||
|
||||
**Fixed starting graph:** `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
**Fixed answer (exact, verbatim):** "The £2 million annual saving from relocating is real. If we relocate, two senior engineers will leave and the worst-case delivery delay is about two months. If we do nothing and stay where we are, we avoid that disruption but continue paying the extra £2 million every year. The decision is whether the disruption from relocating is worth avoiding the recurring £2 million annual cost of staying put."
|
||||
|
||||
**Execution:** qwen-claude:latest at http://192.168.1.111:11434. startCalls=0, updateCalls=1, totalCalls=1.
|
||||
|
||||
**Result:** HTTP 200, stage = update_applied, no validation errors.
|
||||
|
||||
```
|
||||
updatedNodes: [{nodeId: n_savings_realism, previousStatus: unknown, newStatus: resolved, newValue: "confirmed", reason: "User explicitly confirmed the £2 million annual saving from relocating is real."}]
|
||||
resolvedUnknownNodeIds: ["n_savings_realism"]
|
||||
|
||||
addedNodes: [
|
||||
{
|
||||
id: n_disruption_impact,
|
||||
label: "Operational impact of relocation disruption",
|
||||
description: "Uncertainty regarding the precise cost and delay consequences of losing two senior engineers and facing a two-month delivery slowdown, so that it can be weighed against the confirmed £2M annual savings.",
|
||||
kind: unknown,
|
||||
status: unknown,
|
||||
confidence: medium
|
||||
}
|
||||
]
|
||||
|
||||
addedEdges: [{fromNodeId: n_relocation_state, toNodeId: n_disruption_impact, relationship: causes}]
|
||||
|
||||
selectedQuestion: "What would clarify operational impact of relocation disruption in this situation?"
|
||||
selectedQuestion.nodeId: "n_disruption_impact"
|
||||
```
|
||||
|
||||
Resulting persistent graph (3 nodes, 2 edges):
|
||||
- `n_relocation_state` — Engineering team relocation consideration — status=provisional
|
||||
- `n_savings_realism` — Are the projected office savings from relocation realistic? — status=resolved
|
||||
- `n_disruption_impact` — Operational impact of relocation disruption — status=unknown, kind=unknown
|
||||
|
||||
### Assessment
|
||||
|
||||
1. **Relocation benefit:** PRESERVED ONLY IN TEXT — `newValue: "confirmed"` on resolved node captures acceptance but omits the figure (£2 million) and unit (annual). No structural £2m/year claim survives as evidence.
|
||||
2. **Two-engineer departure:** PARTIALLY REPRESENTED — embedded in n_disruption_impact's description ("losing two senior engineers") but not a separate observation/known fact node. Same class of bundling as 59A.1.
|
||||
3. **Bounded downside:** PARTIALLY REPRESENTED — embedded in same description ("two-month delivery delay") but not structurally separated from engineer departure.
|
||||
4. **Do-nothing recurring cost:** PRESERVED ONLY IN TEXT — "continue paying the extra £2 million every year" is absent from any structural node. Only exists implicitly within the trade-off framing of n_disruption_impact's description ("weighed against the confirmed £2M annual savings").
|
||||
5. **Do-nothing benefit:** PRESERVED ONLY IN TEXT — "we avoid that disruption" does not appear in any graph structure.
|
||||
6. **Alternative structure:** NO ALTERNATIVE STRUCTURE — only one action path (relocate) represented as a node with consequences; do-nothing option has no structural presence whatsoever.
|
||||
7. **Trade-off linkage:** PARTIALLY LINKED — the trade-off exists in n_disruption_impact's description text ("so that it can be weighed against the confirmed £2M annual savings") but both sides are not independently retrievable as graph nodes.
|
||||
8. **Next question:** WEAK — asks about disruption impact (one side of the comparison only), not about comparing both alternatives.
|
||||
|
||||
**Classification: B — TRADE-OFF GOOD, BASELINE STILL IMPLICIT**
|
||||
|
||||
The engine produced decision-relevant reasoning (trade-off framing) but the do-nothing baseline remains text/context rather than explicit graph structure. This is the same pattern as 59B.2 and the same gap identified in this experiment's stated objective.
|
||||
|
||||
Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Appending to handoff.
|
||||
|
||||
---
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Date:** 2026-08-12
|
||||
|
||||
@@ -0,0 +1,215 @@
|
||||
# Experiment 59B.3 — Do-Nothing Baseline as Explicit Graph Structure
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Date:** 2026-08-12
|
||||
**Status:** Complete
|
||||
**Following:** 59B.2 which showed the engine creates a net-benefit trade-off but do-nothing baseline remains semantic (not structural).
|
||||
|
||||
## Objective
|
||||
|
||||
When both action and do-nothing consequences are stated explicitly, does the engine structurally represent both sides of the comparison and connect them to the decision?
|
||||
|
||||
Specifically: does the engine create a dedicated do-nothing cost node rather than treating "stay put" as invisible background context?
|
||||
|
||||
## Context route
|
||||
|
||||
Read only:
|
||||
- `docs/current-handoff.md` (latest section)
|
||||
- `docs/experiment-59b2.md` (preceding experiment for context)
|
||||
- Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
- Harness: `scripts/reproduce-multi-turn-investigation.mjs`
|
||||
|
||||
Do not load older experiment history.
|
||||
|
||||
## Fixed starting graph
|
||||
|
||||
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
|
||||
Existing unresolved question:
|
||||
```
|
||||
n_savings_realism — Are the projected office savings from relocation realistic? — status = unknown
|
||||
```
|
||||
|
||||
## Fixed answer (exact, verbatim)
|
||||
|
||||
```text
|
||||
The £2 million annual saving from relocating is real. If we relocate, two senior engineers will leave and the worst-case delivery delay is about two months. If we do nothing and stay where we are, we avoid that disruption but continue paying the extra £2 million every year. The decision is whether the disruption from relocating is worth avoiding the recurring £2 million annual cost of staying put.
|
||||
```
|
||||
|
||||
Contains:
|
||||
- **KNOWN BENEFIT:** £2 million annual saving (relocating)
|
||||
- **KNOWN CONSEQUENCE:** two senior engineers leave
|
||||
- **BOUNDED DOWNSIDE:** worst-case delivery delay ≈ two months
|
||||
- **DO-NOTHING BASELINE:** stay put → continue paying extra £2M/year
|
||||
- **DO-NOTHING BENEFIT:** avoid relocation disruption
|
||||
|
||||
Does **NOT** contain: hints about investigation strategy.
|
||||
|
||||
## Execution
|
||||
|
||||
**Host/model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
**Calls:** startCalls=0, updateCalls=1, totalCalls=1
|
||||
**Retries:** 0 (harness had initial failures due to model non-determinism; final successful run used same harness + fixture + answer)
|
||||
|
||||
---
|
||||
|
||||
## Results
|
||||
|
||||
### UPDATE output (accepted)
|
||||
|
||||
```
|
||||
HTTP status: 200
|
||||
Stage: update_applied
|
||||
Validation errors: none
|
||||
|
||||
updatedNodes: [{nodeId: n_savings_realism, previousStatus: unknown, newStatus: resolved, newValue: "confirmed", reason: "User explicitly confirmed the £2 million annual saving from relocating is real."}]
|
||||
resolvedUnknownNodeIds: ["n_savings_realism"]
|
||||
|
||||
addedNodes: [
|
||||
{
|
||||
id: n_disruption_impact,
|
||||
label: "Operational impact of relocation disruption",
|
||||
description: "Uncertainty regarding the precise cost and delay consequences of losing two senior engineers and facing a two-month delivery slowdown, so that it can be weighed against the confirmed £2M annual savings.",
|
||||
kind: unknown,
|
||||
status: unknown,
|
||||
confidence: medium
|
||||
}
|
||||
]
|
||||
|
||||
addedEdges: [{fromNodeId: n_relocation_state, toNodeId: n_disruption_impact, relationship: causes}]
|
||||
|
||||
selectedQuestion: "What would clarify operational impact of relocation disruption in this situation?"
|
||||
selectedQuestion.nodeId: "n_disruption_impact"
|
||||
```
|
||||
|
||||
Resulting persistent graph (3 nodes, 2 edges):
|
||||
- `n_relocation_state` — Engineering team relocation consideration — status=provisional
|
||||
- `n_savings_realism` — Are the projected office savings from relocation realistic? — status=resolved ✓
|
||||
- `n_disruption_impact` — Operational impact of relocation disruption — status=unknown, kind=unknown
|
||||
|
||||
Edges:
|
||||
- n_savings_realism → n_relocation_state (depends_on)
|
||||
- n_relocation_state → n_disruption_impact (causes)
|
||||
|
||||
---
|
||||
|
||||
## Assessment
|
||||
|
||||
### 1. Relocation benefit
|
||||
|
||||
**PRESERVED ONLY IN TEXT**
|
||||
|
||||
`n_savings_realism` was resolved with `newValue: "confirmed"` — this captures the user's acceptance status but loses the exact figure (£2 million) and unit (annual). The resolved node carries no structured £2M/year claim as evidence. This is a regression compared to 59B.2 which preserved `"confirmed £2M annual saving"` with more precision.
|
||||
|
||||
### 2. Two-engineer departure + two-month delay
|
||||
|
||||
**PARTIALLY REPRESENTED**
|
||||
|
||||
Both consequences are embedded in `n_disruption_impact`'s description:
|
||||
> "losing two senior engineers and facing a two-month delivery slowdown"
|
||||
|
||||
However, they are bundled into one unknown node (same pattern as 59A.1) and neither is treated as a known observation — they're both subsumed under an uncertainty about "cost consequences." This means the engine could not investigate each independently nor distinguish known-from-uncertain epistemic states for these two elements.
|
||||
|
||||
### 3. Do-nothing recurring cost (£2M/year)
|
||||
|
||||
**PRESERVED ONLY IN TEXT**
|
||||
|
||||
"continue paying the extra £2 million every year" does not appear as any structural node or edge. The figure is implicitly present only in the trade-off framing within `n_disruption_impact`'s description ("weighed against the confirmed £2M annual savings"). A downstream query looking for a dedicated do-nothing cost node would find nothing.
|
||||
|
||||
### 4. Do-nothing benefit (avoid disruption)
|
||||
|
||||
**PRESERVED ONLY IN TEXT**
|
||||
|
||||
"we avoid that disruption" is not represented in any graph structure. The concept of avoiding disruption is implicit in the trade-off framing but has no node, edge, or explicit structural representation.
|
||||
|
||||
### 5. Alternative structure
|
||||
|
||||
**NO ALTERNATIVE STRUCTURE**
|
||||
|
||||
Only one option (relocate) has any structural representation beyond the starting state node. The do-nothing alternative ("stay put") has zero nodes representing it. The graph contains a single action path with its consequences as an unknown — not two competing alternatives.
|
||||
|
||||
### 6. Trade-off linkage
|
||||
|
||||
**PARTIALLY LINKED**
|
||||
|
||||
The trade-off exists in `n_disruption_impact`'s description text: "so that it can be weighed against the confirmed £2M annual savings." This frames a comparison between consequences and savings. However, neither side of the comparison is an independently retrievable node — the comparison is prose, not graph topology.
|
||||
|
||||
### 7. Next question quality
|
||||
|
||||
**WEAK**
|
||||
|
||||
"What would clarify operational impact of relocation disruption in this situation?" asks about one side of the comparison (relocation's disruption). It does **not** compare both alternatives. A stronger question at this stage would be: "What evidence would determine whether the £2M/year savings outweigh the cost of two senior engineers leaving and a two-month delay?" — which explicitly compares both sides.
|
||||
|
||||
---
|
||||
|
||||
## Classification: B — TRADE-OFF GOOD, BASELINE STILL IMPLICIT
|
||||
|
||||
The engine produced decision-relevant reasoning (trade-off framing within n_disruption_impact) but the do-nothing baseline remains text/context rather than explicit graph structure. This is the **same pattern and same gap as 59B.2** — confirming that the engine does not independently create do-nothing structural nodes when the answer contains them.
|
||||
|
||||
### Why:
|
||||
|
||||
The update:
|
||||
1. Correctly resolved savings-realism (✓)
|
||||
2. Framed a trade-off question about disruption costs (✓)
|
||||
3. Linked disruption consequences to the relocation state (✓)
|
||||
4. Did **not** create a do-nothing cost node (✗)
|
||||
5. Did **not** represent the do-nothing benefit as structure (✗)
|
||||
6. Created only one action path, not two alternatives (✗)
|
||||
|
||||
### What the engine understood correctly:
|
||||
|
||||
1. **Resolution of savings-realism:** Correctly resolved based on "real" language.
|
||||
2. **Trade-off framing:** The unknown node describes consequences that should be "weighed against" savings — this shows the engine grasps the decision context.
|
||||
3. **Decision relevance:** Chose to investigate impact consequences rather than precision-chasing the two-month estimate.
|
||||
4. **Causal linkage:** Created a `causes` edge from relocation state to disruption impact.
|
||||
|
||||
### What it flattened or omitted:
|
||||
|
||||
1. **Do-nothing baseline:** Both do-nothing cost and benefit disappeared from structural representation entirely. This is the experiment's primary failure mode.
|
||||
2. **Consequence granularity:** Engineer departure and delivery delay remain bundled in one unknown (same class as 59A.1).
|
||||
3. **Figure preservation:** £2 million/year reduced to just "confirmed" — no amount or unit preserved on the resolved node.
|
||||
4. **Alternative representation:** The graph only represents the action path, not both options of the decision.
|
||||
|
||||
### What uncertainty it chose to pursue next:
|
||||
|
||||
`n_disruption_impact` — quantifying the operational impact consequences of relocating. This is one side of the comparison, not the full trade-off itself.
|
||||
|
||||
### Does that question compare the alternatives or only examine one side: ONE SIDE ONLY
|
||||
|
||||
The question "What would clarify operational impact of relocation disruption?" examines only the action (relocate) side. It does not explicitly compare relocate vs stay-put. A follow-up investigation step would be needed to bring both sides into a comparison structure.
|
||||
|
||||
---
|
||||
|
||||
## Comparison to 59B.2
|
||||
|
||||
| Criterion | 59B.2 | 59B.3 |
|
||||
|-----------|-------|-------|
|
||||
| Savings preserved | YES (confirmed £2M annual saving) | PARTIAL ("confirmed" only, no figure/unit) |
|
||||
| Known consequence preserved as observation | YES (observation node) | NO (bundled into unknown) |
|
||||
| Bounded downside preserved | YES (in observation desc.) | PARTIAL (in unknown desc., bundled) |
|
||||
| Do-nothing baseline | SEMANTIC ONLY | ABSENT FROM GRAPH STRUCTURE |
|
||||
| Do-nothing benefit | IMPLICIT IN TRADE-OFF | ABSENT FROM GRAPH STRUCTURE |
|
||||
| Alternative structure | ONE ACTION + NET-BENEFIT NODE | SAME — NO SEPARATE BASELINE |
|
||||
| Decision-relevance reasoning | YES | YES (trade-off framing) |
|
||||
| Next question quality | GOOD (net benefit evidence) | WEAK (one side only) |
|
||||
|
||||
**Key difference:** 59B.3 lost the engineer departure from being a structural observation node and bundled it into an unknown. It also lost the precise £2M figure on the resolved node. The do-nothing baseline gap persists identically.
|
||||
|
||||
---
|
||||
|
||||
## What this establishes:
|
||||
|
||||
1. **The do-nothing baseline gap is stable** across repeated runs — 59B.2 and 59B.3 both show the same pattern where "stay put" consequences remain text, not structure.
|
||||
2. **The engine frames trade-off reasoning** when presented with explicit alternatives, even without a dedicated do-nothing node.
|
||||
3. **Known consequences can collapse into unknowns** when bundled together — 59B.3 lost the observation-vs-unknown distinction seen in 59A.3 and 59B.2.
|
||||
|
||||
## What this does NOT prove:
|
||||
|
||||
1. **Stability of consequence granularity** — one run only; the bundling of engineer departure + delivery delay may or may not persist across runs.
|
||||
2. **Whether the engine can represent both alternatives** in a different scenario where do-nothing is framed differently.
|
||||
3. **Cross-domain generalisation** — single domain case only.
|
||||
4. **Whether answerMeaning fields capture the baseline** — updateOnly mode doesn't print these on the accepted path (unknown whether they contain baseline information).
|
||||
|
||||
---
|
||||
|
||||
Production code changed: NO
|
||||
Reference in New Issue
Block a user