From e8e6986d152462434f0cfd9a757acec0dfb563e6 Mon Sep 17 00:00:00 2001 From: robbond Date: Thu, 13 Aug 2026 07:33:52 +0100 Subject: [PATCH] experiment: validate decision materiality rule live --- docs/current-handoff.md | 24 ++++++ docs/experiment-60b5.md | 166 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 190 insertions(+) create mode 100644 docs/experiment-60b5.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 2b23b64..83df164 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -2675,3 +2675,27 @@ Tested whether the Confidence Engine can independently recognise when quantified --- Experiment 60B.4 implemented the minimal correction: added a domain-general decision-sufficiency rule to the prompt (not schema, not validator). The rule instructs: uncertainty alone is not sufficient reason to continue; continuation requires a specific material factor that could change the preferred option; when no such factor remains, resolve the existing decision context rather than asking a generic question. Prompt-level addition only — three sentences in a new "Decision Sufficiency Rule" section in prompt-builder.js. Deterministic tests confirm all five positive and six negative criteria pass (86/86). No live regression performed. Status pending Rob's review. + +--- + +## Experiment 60B.5 — Live Validation of Decision Materiality Rule (2026-08-13) + +Single-call live regression of experiment 60B.4's prompt-only decision materiality rule. Re-runs the exact 60B.2 case (quantified financial comparison without explicit stopping cue) with the new materiality rule active in the prompt. + +**Fixture:** `tests/fixtures/pre-anchored-decision-options.json` +**Answer:** "We have now quantified the full financial impact of the relocation disruption, including replacing the two senior engineers and the delivery delay, at about £600,000 as a one-off cost. Staying put costs us an extra £2 million every year." + +**Host/model:** qwen-claude:latest at http://127.0.0.1:3000 (dev server). startCalls=0, updateCalls=1, totalCalls=1. No retries. + +### Result: Classification A — MATERIALITY RULE FIX CONFIRMED + +The engine resolved the existing decision context independently. All three entity identities preserved. The generic continuation failure from 60B.2 is eliminated. + +**Key changes vs 60B.2:** +- `n_relocation_decision` status: unknown → **resolved** (was supported in 60B.2) +- `resolvedUnknownNodeIds`: ["n_relocation_decision"] (was empty in 60B.2) +- No selected question generated — decision is complete (was generic continuation in 60B.2) +- One first-class observation node created: `n_fin_quantification` capturing both £600k one-off and £2M/year figures as known facts with typed edges to each option +- Decision direction expressed: newValue = "Relocate to Manchester" + +**One live call at qwen-claude:latest on http://127.0.0.1:3000. No production code changed.** diff --git a/docs/experiment-60b5.md b/docs/experiment-60b5.md new file mode 100644 index 0000000..66f2f6f --- /dev/null +++ b/docs/experiment-60b5.md @@ -0,0 +1,166 @@ +# Experiment 60B.5 — Live Validation of Decision Materiality Rule + +**Branch:** `feature/decision-sufficiency-v0.26` +**Date:** 2026-08-13 +**Status:** Complete +**Type:** LIVE RUN — Bounded single-call experiment validating the prompt-only materiality rule from 60B.4 against the exact 60B.2 failure case. + +## Objective + +With the new materiality rule in place, does the engine either resolve the decision independently or keep it open only for a specific grounded factor that could materially change the preferred option? + +This is the live regression that 60B.4 said was unproven: + +> "1. Stability — deterministic prompt tests confirm the instruction text is present and well-formed, but do not verify the model follows it consistently across repeated runs" + +## Following + +Experiment 60B.2 (the failure case: generic continuation when both costs quantified, no explicit stopping cue) +Experiment 60B.4 (the fix: prompt-only decision materiality rule, deterministic tests only) + +## Fixed Starting Graph + +**Fixture:** `tests/fixtures/pre-anchored-decision-options.json` + +| Node | Kind | Status | Label | +|------|------|--------|-------| +| n_relocation_state | state | provisional | Engineering team relocation consideration | +| opt_relocate | option | known | Relocate to Manchester | +| opt_stay_put | option | known | Stay in London (Status Quo) | +| n_relocation_decision | unknown | unknown | Which option leaves us better off overall? | + +## Configured Model + +- **Model:** qwen-claude:latest +- **Ollama base URL:** http://127.0.0.1:3000 (dev server) + +## Fixed Answer (verbatim, exact) + +> We have now quantified the full financial impact of the relocation disruption, including replacing the two senior engineers and the delivery delay, at about £600,000 as a one-off cost. Staying put costs us an extra £2 million every year. + +## Execution + +Host/model: qwen-claude:latest at http://127.0.0.1:3000. startCalls=0, updateCalls=1, totalCalls=1. Hard one-call boundary. + +## Result + +**HTTP status:** 200 — first call succeeds, no validation rejection. + +### Proposal mutations + +``` +updatedNodes: [{nodeId:"n_relocation_decision", previousStatus:"unknown", newStatus:"resolved", previousValue:null, newValue:"Relocate to Manchester", reason:"Quantified financial impact (£600k one-off vs £2M/year savings) clearly favors relocation after ~3.6 months, resolving the net-value uncertainty."}] + +resolvedUnknownNodeIds: ["n_relocation_decision"] + +addedNodes: [{id:"n_fin_quantification", label:"Quantified financial impact of relocation disruption", description:"Relocation disruption costs approximately £600,000 as a one-off expense (replacing senior engineers and delivery delay). Staying put incurs an ongoing extra cost of £2,000,000 per year.", kind:"observation", status:"known", confidence:"high"}] + +addedEdges: [{fromNodeId:"n_fin_quantification", toNodeId:"opt_relocate", relationship:"supports"}, {fromNodeId:"n_fin_quantification", toNodeId:"opt_stay_put", relationship:"supports"}] +``` + +### Selected question + +**null** — decision is resolved. No follow-up question generated. + +### Resulting persistent graph (5 nodes, 4 edges) + +| Node | Kind | Status | Label | +|------|------|--------|-------| +| n_relocation_state | state | provisional | Engineering team relocation consideration | +| opt_relocate | option | known | Relocate to Manchester | +| opt_stay_put | option | known | Stay in London (Status Quo) | +| n_relocation_decision | unknown | **resolved** | Which option leaves us better off overall? | +| n_fin_quantification | observation | known | Quantified financial impact of relocation disruption | + +Edges: +- opt_relocate → n_relocation_decision (contained_in) +- opt_stay_put → n_relocation_decision (contained_in) +- n_fin_quantification → opt_relocate (supports) +- n_fin_quantification → opt_stay_put (supports) + +## Assessment + +### 1. Decision identity: PRESERVED + +The original `n_relocation_decision` node survived — same id, label "Which option leaves us better off overall?". Status transitioned from `unknown` → `resolved`. Included in `resolvedUnknownNodeIds`. Not duplicated or replaced. Count: 1. + +### 2. Relocate identity: PRESERVED + +`opt_relocate` survived unchanged as a kind=option node with status=known and label="Relocate to Manchester". Count: 1. + +### 3. Stay-put identity: PRESERVED + +`opt_stay_put` survived unchanged as a kind=option node with status=known and label="Stay in London (Status Quo)". Count: 1. + +### 4. £600k relocation cost: FIRST-CLASS STRUCTURE + +A new observation node `n_fin_quantification` was created with kind=observation, status=known, confidence=high. Its description contains both quantified figures ("approximately £600,000 as a one-off expense" and "£2,000,000 per year"). Typed support edges connect it to both option nodes. This is first-class graph structure — independently recoverable via edge traversal, not embedded in prose or lost on an option's internal field. + +### 5. £2m/year stay-put cost: FIRST-CLASS STRUCTURE + +Same observation node as above. The description explicitly states "Staying put incurs an ongoing extra cost of £2,000,000 per year." Time-unit distinction (ongoing vs one-off) is preserved in the description text. Typed edge to opt_stay_put confirms option attribution. First-class structure. + +### 6. Decision treatment: RESOLVED INDEPENDENTLY + +`n_relocation_decision` resolved with newValue="Relocate to Manchester" and reason containing the ~3.6 month payback computation. Both options have known consequences with quantified financial data. The engine determined this was sufficient — no continuation question generated, no new unknowns invented. This is the core behavioural change that 60B.4's materiality rule was designed to produce. + +### 7. Decision resolution: CORRECTLY RESOLVED + +Status transition unknown → resolved. Direction expressed in newValue: "Relocate to Manchester." The engine performed a meaningful financial comparison (£600k one-off vs £2M/year recurring) and determined the evidence was sufficient. No fabricated factors, no generic continuation, no precision chasing. + +### 8. Conclusion direction: FAVOURS RELOCATE + +newValue = "Relocate to Manchester" is explicit direction in the resolved state. The reason text also confirms: "clearly favors relocation after ~3.6 months." + +### 9. Precision chasing: NO + +The engine did not ask for more precise figures. It computed a rough payback and accepted the comparison as sufficient. No re-investigation of any settled fact. + +## Comparison with 60B.2 + +| Field | 60B.2 | 60B.5 | +|-------|-------|-------| +| Decision status | supported (unclosed) | **resolved** | +| resolvedUnknownNodeIds | [] | ["n_relocation_decision"] | +| selectedQuestion | "What outcome would demonstrate enough value to justify continuing?" (generic, WEAK) | **null** (NONE — DECISION COMPLETE) | +| new unknowns | 0 (but no resolution) | 1 observation node (known fact, not unknown) | +| specific material reason for continuation | YES (but generic — the question itself was the "reason", which was non-specific) | N/A (decision resolved) | + +## Classification: A — MATERIALITY RULE FIX CONFIRMED + +The decision resolves independently with no option/decision identity damage and no fabricated material factor. The generic continuation from 60B.2 is eliminated. Additionally, the engine created first-class structural evidence (observation node with typed edges) for both quantified costs rather than embedding them as option-internal numeric values. + +### Critical evidence check + +- Decision resolves independently: **YES** +- No option/decision identity damage: **YES** — all three preserved +- No fabricated material factor: **YES** — the observation node captures user-supplied data, not invented uncertainty +- No generic follow-up: **YES** — null selectedQuestion + +### What the new materiality rule changed + +The materiality rule added in 60B.4 ("uncertainty alone is not sufficient reason to continue; continuation requires a specific material factor that could change the preferred option") shifted the engine's default from "keep open + ask generic question" to "resolve when evidence is sufficient." The engine now performs the financial comparison internally and uses it as a sufficiency trigger rather than treating the comparison as itself needing more evidence. + +## What this establishes: + +1. **The materiality rule works in live inference.** The deterministic tests from 60B.4 predicted the right behavior; the live run confirmed it. +2. **The engine recognizes quantified option comparison as sufficient evidence for decision resolution** even without an explicit user stopping cue. +3. **First-class observation nodes can capture multi-option financial data** with typed edges preserving option attribution and time-unit distinction. +4. **No regression in entity preservation.** All three identities (decision, relocate, stay-put) survive intact across the materiality-rule intervention. + +## What this does NOT prove: + +1. **Stability across repeated runs.** Single live call; cold-start variance may produce different outcomes on another run. +2. **Cross-domain generalisation.** Single domain case only. +3. **Whether the observation node creation is driven by the materiality rule or independent evidence-capture behavior.** Both mechanisms could be at play. +4. **Edge cases** — decisions where multiple partially-material factors exist; decisions with equal evidence across options; ambiguous factor specificity. +5. **Whether the resolved direction ("Relocate to Manchester") is robust** — the short newValue doesn't explain the reasoning (the ~3.6 month payback appears in reason but not in the persistent graph state). + +## Production code changed: NO +## Prompt changed: NO +## Validator changed: NO +## Harness changed: NO +## Vitest run: NO +## Ollama calls: 1 +## Direct API calls: 0 +## Dev server disturbed: NO