experiment: test decision relevance and do-nothing baseline

2026-08-12 17:10:01 +01:00
parent 8856e66147
commit 7f27fccadc
2 changed files with 146 additions and 0 deletions
@@ -1995,3 +1995,11 @@ The hypothesis asked whether selecting `n_savings_realism` would now produce a c
 **Objective:** When the user provides a concrete £2m figure but explicitly states it is unverified, does the engine preserve the figure and keep the existing savings-realism uncertainty unresolved?

 **Classification: A — QUALIFIED EVIDENCE AND UNCERTAINTY BOTH PRESERVED.** One update-only call via the committed harness (confirmed by direct API inspection). The engine preserved the £2m/year figure as `"£2M/year (unverified)"` on n_savings_realism, kept status as `unknown` (not weakened to provisional), confidence set to low, answerMeaning.supportCategory = `"uncertain"`, possibleInference = null. No duplicate nodes, no resolved unknowns. Selected question continues investigating the realism concern. This is the best result seen for savings-realism across all 58A/B experiments — evidence and uncertainty both survive intact. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed.
+
+---
+
+### Experiment 59B.1 — Decision Relevance of Next Question vs Precision Chasing
+
+**Objective:** When the financial benefit is known, the downside is bounded, and the cost of doing nothing is explicit, does the engine compare decision consequences — or does it simply ask for more precision about the remaining uncertainty?
+
+**Classification: D — DO-NOTHING BASELINE LOST.** One update-only call via the committed harness. The engine resolved n_savings_realism (preserving £2m as known benefit) and created a new meta-level question "would a precise delivery-delay estimate change the decision?" which is HIGH decision relevance. However, the engine entirely lost: (a) the known consequence of two senior engineers departing, (b) the bounded downside of "no more than two months" delay, and (c) the do-nothing baseline ("staying put costs extra £2m every year"). A positive finding: the engine did NOT ask for exact delivery delay (no precision chasing). It asked whether precision matters at all — a valid decision-relevant step. However, this question is contextually hollow because the critical comparison elements are absent from the graph. This suggests a context-preservation deficit in the update path: when resolving one uncertainty and creating a new node, the engine drops other critical information from the answer rather than carrying it forward. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed.