confidence-engine/docs/experiment-60b5.md

Experiment 60B.5 — Live Validation of Decision Materiality Rule

Branch: feature/decision-sufficiency-v0.26
Date: 2026-08-13
Status: Complete
Type: LIVE RUN — Bounded single-call experiment validating the prompt-only materiality rule from 60B.4 against the exact 60B.2 failure case.

Objective

With the new materiality rule in place, does the engine either resolve the decision independently or keep it open only for a specific grounded factor that could materially change the preferred option?

This is the live regression that 60B.4 said was unproven:

"1. Stability — deterministic prompt tests confirm the instruction text is present and well-formed, but do not verify the model follows it consistently across repeated runs"

Following

Experiment 60B.2 (the failure case: generic continuation when both costs quantified, no explicit stopping cue)
Experiment 60B.4 (the fix: prompt-only decision materiality rule, deterministic tests only)

Fixed Starting Graph

Fixture: tests/fixtures/pre-anchored-decision-options.json

Node	Kind	Status	Label
n_relocation_state	state	provisional	Engineering team relocation consideration
opt_relocate	option	known	Relocate to Manchester
opt_stay_put	option	known	Stay in London (Status Quo)
n_relocation_decision	unknown	unknown	Which option leaves us better off overall?

Configured Model

• Model: qwen-claude:latest
• Ollama base URL: http://127.0.0.1:3000 (dev server)

Fixed Answer (verbatim, exact)

We have now quantified the full financial impact of the relocation disruption, including replacing the two senior engineers and the delivery delay, at about £600,000 as a one-off cost. Staying put costs us an extra £2 million every year.

Execution

Host/model: qwen-claude:latest at http://127.0.0.1:3000. startCalls=0, updateCalls=1, totalCalls=1. Hard one-call boundary.

Result

HTTP status: 200 — first call succeeds, no validation rejection.

Proposal mutations

updatedNodes: [{nodeId:"n_relocation_decision", previousStatus:"unknown", newStatus:"resolved", previousValue:null, newValue:"Relocate to Manchester", reason:"Quantified financial impact (£600k one-off vs £2M/year savings) clearly favors relocation after ~3.6 months, resolving the net-value uncertainty."}]

resolvedUnknownNodeIds: ["n_relocation_decision"]

addedNodes: [{id:"n_fin_quantification", label:"Quantified financial impact of relocation disruption", description:"Relocation disruption costs approximately £600,000 as a one-off expense (replacing senior engineers and delivery delay). Staying put incurs an ongoing extra cost of £2,000,000 per year.", kind:"observation", status:"known", confidence:"high"}]

addedEdges: [{fromNodeId:"n_fin_quantification", toNodeId:"opt_relocate", relationship:"supports"}, {fromNodeId:"n_fin_quantification", toNodeId:"opt_stay_put", relationship:"supports"}]

Selected question

null — decision is resolved. No follow-up question generated.

Resulting persistent graph (5 nodes, 4 edges)

Node	Kind	Status	Label
n_relocation_state	state	provisional	Engineering team relocation consideration
opt_relocate	option	known	Relocate to Manchester
opt_stay_put	option	known	Stay in London (Status Quo)
n_relocation_decision	unknown	resolved	Which option leaves us better off overall?
n_fin_quantification	observation	known	Quantified financial impact of relocation disruption

Edges:

• opt_relocate → n_relocation_decision (contained_in)
• opt_stay_put → n_relocation_decision (contained_in)
• n_fin_quantification → opt_relocate (supports)
• n_fin_quantification → opt_stay_put (supports)

Assessment

1. Decision identity: PRESERVED

The original n_relocation_decision node survived — same id, label "Which option leaves us better off overall?". Status transitioned from unknown → resolved. Included in resolvedUnknownNodeIds. Not duplicated or replaced. Count: 1.

2. Relocate identity: PRESERVED

opt_relocate survived unchanged as a kind=option node with status=known and label="Relocate to Manchester". Count: 1.

3. Stay-put identity: PRESERVED

opt_stay_put survived unchanged as a kind=option node with status=known and label="Stay in London (Status Quo)". Count: 1.

4. £600k relocation cost: FIRST-CLASS STRUCTURE

A new observation node n_fin_quantification was created with kind=observation, status=known, confidence=high. Its description contains both quantified figures ("approximately £600,000 as a one-off expense" and "£2,000,000 per year"). Typed support edges connect it to both option nodes. This is first-class graph structure — independently recoverable via edge traversal, not embedded in prose or lost on an option's internal field.

5. £2m/year stay-put cost: FIRST-CLASS STRUCTURE

Same observation node as above. The description explicitly states "Staying put incurs an ongoing extra cost of £2,000,000 per year." Time-unit distinction (ongoing vs one-off) is preserved in the description text. Typed edge to opt_stay_put confirms option attribution. First-class structure.

6. Decision treatment: RESOLVED INDEPENDENTLY

n_relocation_decision resolved with newValue="Relocate to Manchester" and reason containing the ~3.6 month payback computation. Both options have known consequences with quantified financial data. The engine determined this was sufficient — no continuation question generated, no new unknowns invented. This is the core behavioural change that 60B.4's materiality rule was designed to produce.

7. Decision resolution: CORRECTLY RESOLVED

Status transition unknown → resolved. Direction expressed in newValue: "Relocate to Manchester." The engine performed a meaningful financial comparison (£600k one-off vs £2M/year recurring) and determined the evidence was sufficient. No fabricated factors, no generic continuation, no precision chasing.

8. Conclusion direction: FAVOURS RELOCATE

newValue = "Relocate to Manchester" is explicit direction in the resolved state. The reason text also confirms: "clearly favors relocation after ~3.6 months."

9. Precision chasing: NO

The engine did not ask for more precise figures. It computed a rough payback and accepted the comparison as sufficient. No re-investigation of any settled fact.

Comparison with 60B.2

Field	60B.2	60B.5
Decision status	supported (unclosed)	resolved
resolvedUnknownNodeIds	[]	["n_relocation_decision"]
selectedQuestion	"What outcome would demonstrate enough value to justify continuing?" (generic, WEAK)	null (NONE — DECISION COMPLETE)
new unknowns	0 (but no resolution)	1 observation node (known fact, not unknown)
specific material reason for continuation	YES (but generic — the question itself was the "reason", which was non-specific)	N/A (decision resolved)

Classification: A — MATERIALITY RULE FIX CONFIRMED

The decision resolves independently with no option/decision identity damage and no fabricated material factor. The generic continuation from 60B.2 is eliminated. Additionally, the engine created first-class structural evidence (observation node with typed edges) for both quantified costs rather than embedding them as option-internal numeric values.

Critical evidence check

• Decision resolves independently: YES
• No option/decision identity damage: YES — all three preserved
• No fabricated material factor: YES — the observation node captures user-supplied data, not invented uncertainty
• No generic follow-up: YES — null selectedQuestion

What the new materiality rule changed

The materiality rule added in 60B.4 ("uncertainty alone is not sufficient reason to continue; continuation requires a specific material factor that could change the preferred option") shifted the engine's default from "keep open + ask generic question" to "resolve when evidence is sufficient." The engine now performs the financial comparison internally and uses it as a sufficiency trigger rather than treating the comparison as itself needing more evidence.

What this establishes:

1. The materiality rule works in live inference. The deterministic tests from 60B.4 predicted the right behavior; the live run confirmed it.
2. The engine recognizes quantified option comparison as sufficient evidence for decision resolution even without an explicit user stopping cue.
3. First-class observation nodes can capture multi-option financial data with typed edges preserving option attribution and time-unit distinction.
4. No regression in entity preservation. All three identities (decision, relocate, stay-put) survive intact across the materiality-rule intervention.

What this does NOT prove:

1. Stability across repeated runs. Single live call; cold-start variance may produce different outcomes on another run.
2. Cross-domain generalisation. Single domain case only.
3. Whether the observation node creation is driven by the materiality rule or independent evidence-capture behavior. Both mechanisms could be at play.
4. Edge cases — decisions where multiple partially-material factors exist; decisions with equal evidence across options; ambiguous factor specificity.
5. Whether the resolved direction ("Relocate to Manchester") is robust — the short newValue doesn't explain the reasoning (the ~3.6 month payback appears in reason but not in the persistent graph state).

Production code changed: NO

Prompt changed: NO

Validator changed: NO

Harness changed: NO

Vitest run: NO

Ollama calls: 1

Direct API calls: 0

Dev server disturbed: NO