9.3 KiB
Experiment 60B.6 — Test Materiality Rule Against Real Unresolved Factor
Branch: feature/decision-sufficiency-v0.26
Date: 2026-08-13
Status: Complete
Type: LIVE RUN — Single-call experiment testing the materiality rule from 60B.4 against a genuinely decision-changing uncertainty.
Objective
When the quantified comparison favours one option but one specific unresolved factor could realistically reverse that preference, does the engine keep the decision open for that factor rather than over-closing?
Following
Experiment 60B.2 (over-closure: resolves decision when £5M client risk remains)
Experiment 60B.4 (the fix: prompt-only materiality rule)
Experiment 60B.5 (confirmation: materiality rule can close without explicit cue)
This is the opposite boundary of 60B.5's success case. 60B.5 tested "should it close when nothing materially unresolved?" — and it did. 60B.6 tests "should it stay open when something materially unresolved exists?" — and should keep open for that specific factor.
Fixed Starting Graph
Fixture: tests/fixtures/pre-anchored-decision-options.json
| Node | Kind | Status | Label |
|---|---|---|---|
| n_relocation_state | state | provisional | Engineering team relocation consideration |
| opt_relocate | option | known | Relocate to Manchester |
| opt_stay_put | option | known | Stay in London (Status Quo) |
| n_relocation_decision | unknown | unknown | Which option leaves us better off overall? |
Configured Model
- Model: qwen-claude:latest
- Ollama base URL: http://127.0.0.1:3000 (dev server)
Fixed Answer (verbatim, exact)
We have now quantified the full financial impact of replacing the two senior engineers and the delivery delay at about £600,000 as a one-off relocation cost. Staying put costs us an extra £2 million every year. The remaining issue is our largest client: we do not yet know whether they would leave if we relocated, and losing them would cost us about £5 million per year.
Why This Case Matters
Without the client risk, Relocate (£600k one-off) vs Stay (£2M/year recurring) clearly favours Relocate. The £5M/year client-retention risk is large enough to plausibly reverse that preference — making it a genuine decision-changing uncertainty.
Execution
Single update call. No retries. Hard boundary enforced by harness.
Call Accounting
startCalls: 0
updateCalls: 1
totalCalls: 1
Retries: 0
HTTP Response
- Status: 200 — success, no rejection
- Proposal applied: YES (accepted)
Proposal Mutations
updatedNodes: []
resolvedUnknownNodeIds: []
addedNodes: [{
id: "n_client_retention",
label: "Largest client retention uncertainty",
description: "Whether the organisation's largest client will leave if the team relocates to Manchester, matters because their departure would cost ~£5M/year and likely negate the £2M annual savings needed to justify relocation.",
kind: "unknown",
status: "unknown",
confidence: "medium"
}]
addedEdges: [{
id: "e-client-to-opt",
fromNodeId: "n_client_retention",
toNodeId: "opt_relocate",
relationship: "may_cause",
confidence: "medium",
description: "Relocating to Manchester may cause the largest client to leave"
}]
selectedQuestion: "What outcome would demonstrate enough value to justify continuing?"
Resulting Persistent Graph (5 nodes, 3 edges)
| Node | Kind | Status | Label |
|---|---|---|---|
| n_relocation_state | state | provisional | Engineering team relocation consideration |
| opt_relocate | option | known | Relocate to Manchester |
| opt_stay_put | option | known | Stay in London (Status Quo) |
| n_relocation_decision | unknown | unknown | Which option leaves us better off overall? |
| n_client_retention | unknown | unknown | Largest client retention uncertainty |
Edges:
- opt_relocate → n_relocation_decision (contained_in)
- opt_stay_put → n_relocation_decision (contained_in)
- n_client_retention → opt_relocate (may_cause)
Assessment
1. Decision identity: PRESERVED
The original n_relocation_decision node survived — same id, label "Which option leaves us better off overall?", status=unknown. Not resolved. Count: 1.
2. Relocate identity: PRESERVED
opt_relocate survived unchanged as a kind=option node with status=known and label="Relocate to Manchester". Count: 1.
3. Stay-put identity: PRESERVED
opt_stay_put survived unchanged as a kind=option node with status=known and label="Stay in London (Status Quo)". Count: 1.
4. Known financial comparison (£600k + £2M/year)
No new observation node created. Unlike 60B.5 (which created n_fin_quantification), the quantified figures exist only in the user answer text, not as structured evidence nodes in the graph. They are implicitly available in the option descriptions but not explicitly captured as comparison evidence.
Classification: PARTIAL (figures survive in option descriptions but no new first-class observation structure was created)
5. Client-retention uncertainty: FIRST-CLASS UNKNOWN
Created as n_client_retention with kind=unknown, status=unknown. Has a description explaining the factor and its relevance. This is a proper first-class unknown node — not text-only, not flattened into an option's internal state.
Classification: FIRST-CLASS UNKNOWN
6. Client-risk ownership to Relate: CLEARLY OWNED BY RELOCATE
The n_client_retention node has childIds: ["opt_relocate"] and a typed edge n_client_retention → opt_relocate with relationship="may_cause". Graph-only reasoning can determine the unresolved client risk belongs to the Relocate option.
Classification: CLEARLY OWNED BY RELOCATE
7. £5M/year downside: PRESERVED WITH UNKNOWN
The "~£5M/year" figure is embedded in the description text of n_client_retention. It is not isolated as a separate structured numeric value but survives within the unknown node's epistemic container.
Classification: PRESERVED WITH UNKNOWN
8. Materiality judgment: RECOGNISED BUT WEAKLY
The engine correctly kept the decision open and created the client-retention unknown, demonstrating it recognised this factor as material. However, the recognition is structural (it created the node) but not interrogative (the selected question does not target it). The materiality rule prevented over-closure but did not fully activate the follow-up targeting the specific material factor.
Classification: RECOGNISED BUT WEAKLY
9. Decision treatment: KEPT OPEN FOR SPECIFIC MATERIAL REASON
The decision was kept open — n_relocation_decision remains unknown, nothing resolved. The newly added n_client_retention is clearly the specific material reason (a client retention uncertainty with £5M/year downside that could reverse the preference). No unrelated uncertainty invented.
Classification: KEPT OPEN FOR SPECIFIC MATERIAL REASON
10. Selected question: WEAK
The question "What outcome would demonstrate enough value to justify continuing?" is generic — the same phrasing from 60B.2. The engine has just created a specific client-retention unknown and should have targeted it with something like "Will the organisation's largest client leave if we relocate to Manchester?"
Classification: WEAK
Classification: B — MATERIAL FACTOR RECOGNISED BUT STRUCTURE PARTIAL
The engine correctly continues for the client risk (classification B requires this) but its representation or question specificity is incomplete.
Critical evidence check
- Decision remains unresolved: YES
- Client-retention uncertainty survives: YES
- Client risk attributable to Relocate: YES
- £5M/year impact survives: YES (embedded in description text, not isolated)
- Generic substitute question replaces specific targeting: NO — it uses generic phrasing instead of targeting the specific unknown
- No unnecessary new uncertainty invented: YES
What this establishes:
- The materiality rule prevents over-closure with a real decision-changing factor. When an unresolved £5M/year client-retention risk exists, the engine does NOT resolve the decision — correctly keeping it open.
- The engine can create first-class unknown nodes for option-specific risks. The
n_client_retentionnode is properly typed, attributed to the correct option via amay_causeedge, and has meaningful description text. - Option ownership survives graph construction. The
may_causerelationship from the client-retention unknown toopt_relocatemakes clear this uncertainty belongs to the Relate option — enabling future reasoning to correctly associate upside/downside with the right candidate.
What this does NOT prove:
- Whether the materiality rule can target follow-up questions at the specific material factor. The generic question suggests structural recognition but interrogative gap.
- Whether the observation-node behaviour is consistent (60B.5 created one, 60B.6 did not). This may be context-dependent or model-stochastic rather than rule-driven.
- Cross-domain robustness. Single scenario with a single model.
- What happens with multiple concurrent material uncertainties. Tested exactly one unresolved factor.