experiment: validate decision materiality rule live

2026-08-13 07:33:52 +01:00
parent b671681ddc
commit e8e6986d15
2 changed files with 190 additions and 0 deletions
@@ -2675,3 +2675,27 @@ Tested whether the Confidence Engine can independently recognise when quantified
 ---

 Experiment 60B.4 implemented the minimal correction: added a domain-general decision-sufficiency rule to the prompt (not schema, not validator). The rule instructs: uncertainty alone is not sufficient reason to continue; continuation requires a specific material factor that could change the preferred option; when no such factor remains, resolve the existing decision context rather than asking a generic question. Prompt-level addition only — three sentences in a new "Decision Sufficiency Rule" section in prompt-builder.js. Deterministic tests confirm all five positive and six negative criteria pass (86/86). No live regression performed. Status pending Rob's review.
+
+---
+
+## Experiment 60B.5 — Live Validation of Decision Materiality Rule (2026-08-13)
+
+Single-call live regression of experiment 60B.4's prompt-only decision materiality rule. Re-runs the exact 60B.2 case (quantified financial comparison without explicit stopping cue) with the new materiality rule active in the prompt.
+
+**Fixture:** `tests/fixtures/pre-anchored-decision-options.json`
+**Answer:** "We have now quantified the full financial impact of the relocation disruption, including replacing the two senior engineers and the delivery delay, at about £600,000 as a one-off cost. Staying put costs us an extra £2 million every year."
+
+**Host/model:** qwen-claude:latest at http://127.0.0.1:3000 (dev server). startCalls=0, updateCalls=1, totalCalls=1. No retries.
+
+### Result: Classification A — MATERIALITY RULE FIX CONFIRMED
+
+The engine resolved the existing decision context independently. All three entity identities preserved. The generic continuation failure from 60B.2 is eliminated.
+
+**Key changes vs 60B.2:**
+- `n_relocation_decision` status: unknown → **resolved** (was supported in 60B.2)
+- `resolvedUnknownNodeIds`: ["n_relocation_decision"] (was empty in 60B.2)
+- No selected question generated — decision is complete (was generic continuation in 60B.2)
+- One first-class observation node created: `n_fin_quantification` capturing both £600k one-off and £2M/year figures as known facts with typed edges to each option
+- Decision direction expressed: newValue = "Relocate to Manchester"
+
+**One live call at qwen-claude:latest on http://127.0.0.1:3000. No production code changed.**