experiment: test independent decision relevance

2026-08-12 17:18:18 +01:00
parent 7f27fccadc
commit 70688f91c9
2 changed files with 263 additions and 0 deletions
@@ -2003,3 +2003,30 @@ The hypothesis asked whether selecting `n_savings_realism` would now produce a c
 **Objective:** When the financial benefit is known, the downside is bounded, and the cost of doing nothing is explicit, does the engine compare decision consequences — or does it simply ask for more precision about the remaining uncertainty?

 **Classification: D — DO-NOTHING BASELINE LOST.** One update-only call via the committed harness. The engine resolved n_savings_realism (preserving £2m as known benefit) and created a new meta-level question "would a precise delivery-delay estimate change the decision?" which is HIGH decision relevance. However, the engine entirely lost: (a) the known consequence of two senior engineers departing, (b) the bounded downside of "no more than two months" delay, and (c) the do-nothing baseline ("staying put costs extra £2m every year"). A positive finding: the engine did NOT ask for exact delivery delay (no precision chasing). It asked whether precision matters at all — a valid decision-relevant step. However, this question is contextually hollow because the critical comparison elements are absent from the graph. This suggests a context-preservation deficit in the update path: when resolving one uncertainty and creating a new node, the engine drops other critical information from the answer rather than carrying it forward. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed.
+
+### Experiment 59B.2 — Independent Decision Relevance Reasoning (No User Hint)
+
+**Objective:** When the user does NOT hint about decision relevance of precision, does the engine independently recognise when more precision may not change the decision?
+
+**Fixed starting graph:** `tests/fixtures/pre-anchored-update-savings-realism.json`
+**Fixed answer:** "The £2 million annual saving is real. We know two senior engineers would leave if we relocate. The worst-case delivery impact is about two months of delay. If we stay where we are, we keep paying the extra £2 million every year."
+
+**Execution:** qwen-claude:latest at http://192.168.1.111:11434. startCalls=0, updateCalls=1, totalCalls=1.
+
+**Result:** UPDATE accepted (HTTP 200 at update_applied). No validation errors.
+
+**Assessment of four factual elements:**
+- £2m annual saving: PRESERVED AS KNOWN BENEFIT (resolved on `n_savings_realism` with "confirmed £2M annual saving")
+- Two-engineer departure: PRESERVED AS KNOWN CONSEQUENCE (dedicated observation node `n_talent_impact_clarity`, kind=observation, status=supported)
+- Two-month worst-case bound: PRESERVED AS BOUNDED DOWNSIDE (embedded in observation description: "worst-case 2-month project delay")
+- Do-nothing baseline: PRESERVED ONLY IN SEMANTIC/TEXT CONTEXT (implicit in trade-off framing within `n_net_relocation_benefit`'s description, not a dedicated structural node)
+
+**Next question chosen:** "What evidence would clarify relocation net benefit is positive?" → `n_net_relocation_benefit` (DECISION-CHANGING UNKNOWN)
+
+**Precision chasing:** NO. The engine did NOT ask for more precise delay information despite the approximate "about two months" figure. It independently chose a decision-changing unknown (net benefit).
+
+**Independent decision relevance:** YES. The engine distinguished between "uncertainty exists" and "this uncertainty is worth resolving." It resolved known benefit, preserved known consequence as an observation (not uncertainty), preserved bounded downside, and created a net-benefit trade-off unknown — without any user hint about precision relevance.
+
+**Classification: A — INDEPENDENT DECISION-RELEVANCE REASONING.** The engine preserves all four factual elements (three structural, one semantic) and independently focuses on information that could plausibly change the decision. Significant improvement over 59B.1 which lost known consequence and bounded downside from the graph entirely.
+
+Full record in `docs/experiment-59b2.md`. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed.