194 lines
9.3 KiB
Markdown
194 lines
9.3 KiB
Markdown
# Experiment 60B.6 — Test Materiality Rule Against Real Unresolved Factor
|
|
|
|
**Branch:** `feature/decision-sufficiency-v0.26`
|
|
**Date:** 2026-08-13
|
|
**Status:** Complete
|
|
**Type:** LIVE RUN — Single-call experiment testing the materiality rule from 60B.4 against a genuinely decision-changing uncertainty.
|
|
|
|
## Objective
|
|
|
|
When the quantified comparison favours one option but one specific unresolved factor could realistically reverse that preference, does the engine keep the decision open for that factor rather than over-closing?
|
|
|
|
## Following
|
|
|
|
Experiment 60B.2 (over-closure: resolves decision when £5M client risk remains)
|
|
Experiment 60B.4 (the fix: prompt-only materiality rule)
|
|
Experiment 60B.5 (confirmation: materiality rule can close without explicit cue)
|
|
|
|
This is the **opposite boundary** of 60B.5's success case. 60B.5 tested "should it close when nothing materially unresolved?" — and it did. 60B.6 tests "should it stay open when something materially unresolved exists?" — and should keep open for that specific factor.
|
|
|
|
## Fixed Starting Graph
|
|
|
|
**Fixture:** `tests/fixtures/pre-anchored-decision-options.json`
|
|
|
|
| Node | Kind | Status | Label |
|
|
|------|------|--------|-------|
|
|
| n_relocation_state | state | provisional | Engineering team relocation consideration |
|
|
| opt_relocate | option | known | Relocate to Manchester |
|
|
| opt_stay_put | option | known | Stay in London (Status Quo) |
|
|
| n_relocation_decision | unknown | unknown | Which option leaves us better off overall? |
|
|
|
|
## Configured Model
|
|
|
|
- **Model:** qwen-claude:latest
|
|
- **Ollama base URL:** http://127.0.0.1:3000 (dev server)
|
|
|
|
## Fixed Answer (verbatim, exact)
|
|
|
|
> We have now quantified the full financial impact of replacing the two senior engineers and the delivery delay at about £600,000 as a one-off relocation cost. Staying put costs us an extra £2 million every year. The remaining issue is our largest client: we do not yet know whether they would leave if we relocated, and losing them would cost us about £5 million per year.
|
|
|
|
## Why This Case Matters
|
|
|
|
Without the client risk, Relocate (£600k one-off) vs Stay (£2M/year recurring) clearly favours Relocate. The £5M/year client-retention risk is large enough to plausibly reverse that preference — making it a genuine decision-changing uncertainty.
|
|
|
|
## Execution
|
|
|
|
Single update call. No retries. Hard boundary enforced by harness.
|
|
|
|
## Call Accounting
|
|
|
|
```
|
|
startCalls: 0
|
|
updateCalls: 1
|
|
totalCalls: 1
|
|
Retries: 0
|
|
```
|
|
|
|
## HTTP Response
|
|
|
|
- **Status:** 200 — success, no rejection
|
|
- **Proposal applied:** YES (accepted)
|
|
|
|
## Proposal Mutations
|
|
|
|
```
|
|
updatedNodes: []
|
|
resolvedUnknownNodeIds: []
|
|
|
|
addedNodes: [{
|
|
id: "n_client_retention",
|
|
label: "Largest client retention uncertainty",
|
|
description: "Whether the organisation's largest client will leave if the team relocates to Manchester, matters because their departure would cost ~£5M/year and likely negate the £2M annual savings needed to justify relocation.",
|
|
kind: "unknown",
|
|
status: "unknown",
|
|
confidence: "medium"
|
|
}]
|
|
|
|
addedEdges: [{
|
|
id: "e-client-to-opt",
|
|
fromNodeId: "n_client_retention",
|
|
toNodeId: "opt_relocate",
|
|
relationship: "may_cause",
|
|
confidence: "medium",
|
|
description: "Relocating to Manchester may cause the largest client to leave"
|
|
}]
|
|
|
|
selectedQuestion: "What outcome would demonstrate enough value to justify continuing?"
|
|
```
|
|
|
|
## Resulting Persistent Graph (5 nodes, 3 edges)
|
|
|
|
| Node | Kind | Status | Label |
|
|
|------|------|--------|-------|
|
|
| n_relocation_state | state | provisional | Engineering team relocation consideration |
|
|
| opt_relocate | option | known | Relocate to Manchester |
|
|
| opt_stay_put | option | known | Stay in London (Status Quo) |
|
|
| n_relocation_decision | unknown | unknown | Which option leaves us better off overall? |
|
|
| n_client_retention | unknown | unknown | Largest client retention uncertainty |
|
|
|
|
Edges:
|
|
- opt_relocate → n_relocation_decision (contained_in)
|
|
- opt_stay_put → n_relocation_decision (contained_in)
|
|
- n_client_retention → opt_relocate (may_cause)
|
|
|
|
## Assessment
|
|
|
|
### 1. Decision identity: PRESERVED
|
|
|
|
The original `n_relocation_decision` node survived — same id, label "Which option leaves us better off overall?", status=unknown. Not resolved. Count: 1.
|
|
|
|
### 2. Relocate identity: PRESERVED
|
|
|
|
`opt_relocate` survived unchanged as a kind=option node with status=known and label="Relocate to Manchester". Count: 1.
|
|
|
|
### 3. Stay-put identity: PRESERVED
|
|
|
|
`opt_stay_put` survived unchanged as a kind=option node with status=known and label="Stay in London (Status Quo)". Count: 1.
|
|
|
|
### 4. Known financial comparison (£600k + £2M/year)
|
|
|
|
**No new observation node created.** Unlike 60B.5 (which created `n_fin_quantification`), the quantified figures exist only in the user answer text, not as structured evidence nodes in the graph. They are implicitly available in the option descriptions but not explicitly captured as comparison evidence.
|
|
|
|
Classification: **PARTIAL** (figures survive in option descriptions but no new first-class observation structure was created)
|
|
|
|
### 5. Client-retention uncertainty: FIRST-CLASS UNKNOWN
|
|
|
|
Created as `n_client_retention` with kind=unknown, status=unknown. Has a description explaining the factor and its relevance. This is a proper first-class unknown node — not text-only, not flattened into an option's internal state.
|
|
|
|
Classification: **FIRST-CLASS UNKNOWN**
|
|
|
|
### 6. Client-risk ownership to Relate: CLEARLY OWNED BY RELOCATE
|
|
|
|
The `n_client_retention` node has `childIds: ["opt_relocate"]` and a typed edge `n_client_retention → opt_relocate` with relationship="may_cause". Graph-only reasoning can determine the unresolved client risk belongs to the Relocate option.
|
|
|
|
Classification: **CLEARLY OWNED BY RELOCATE**
|
|
|
|
### 7. £5M/year downside: PRESERVED WITH UNKNOWN
|
|
|
|
The "~£5M/year" figure is embedded in the description text of `n_client_retention`. It is not isolated as a separate structured numeric value but survives within the unknown node's epistemic container.
|
|
|
|
Classification: **PRESERVED WITH UNKNOWN**
|
|
|
|
### 8. Materiality judgment: RECOGNISED BUT WEAKLY
|
|
|
|
The engine correctly kept the decision open and created the client-retention unknown, demonstrating it recognised this factor as material. However, the recognition is structural (it created the node) but not interrogative (the selected question does not target it). The materiality rule prevented over-closure but did not fully activate the follow-up targeting the specific material factor.
|
|
|
|
Classification: **RECOGNISED BUT WEAKLY**
|
|
|
|
### 9. Decision treatment: KEPT OPEN FOR SPECIFIC MATERIAL REASON
|
|
|
|
The decision was kept open — `n_relocation_decision` remains unknown, nothing resolved. The newly added `n_client_retention` is clearly the specific material reason (a client retention uncertainty with £5M/year downside that could reverse the preference). No unrelated uncertainty invented.
|
|
|
|
Classification: **KEPT OPEN FOR SPECIFIC MATERIAL REASON**
|
|
|
|
### 10. Selected question: WEAK
|
|
|
|
The question "What outcome would demonstrate enough value to justify continuing?" is generic — the same phrasing from 60B.2. The engine has just created a specific client-retention unknown and should have targeted it with something like "Will the organisation's largest client leave if we relocate to Manchester?"
|
|
|
|
Classification: **WEAK**
|
|
|
|
## Classification: B — MATERIAL FACTOR RECOGNISED BUT STRUCTURE PARTIAL
|
|
|
|
The engine correctly continues for the client risk (classification B requires this) but its representation or question specificity is incomplete.
|
|
|
|
### Critical evidence check
|
|
|
|
- Decision remains unresolved: **YES**
|
|
- Client-retention uncertainty survives: **YES**
|
|
- Client risk attributable to Relocate: **YES**
|
|
- £5M/year impact survives: **YES** (embedded in description text, not isolated)
|
|
- Generic substitute question replaces specific targeting: **NO — it uses generic phrasing instead of targeting the specific unknown**
|
|
- No unnecessary new uncertainty invented: **YES**
|
|
|
|
## What this establishes:
|
|
|
|
1. **The materiality rule prevents over-closure with a real decision-changing factor.** When an unresolved £5M/year client-retention risk exists, the engine does NOT resolve the decision — correctly keeping it open.
|
|
2. **The engine can create first-class unknown nodes for option-specific risks.** The `n_client_retention` node is properly typed, attributed to the correct option via a `may_cause` edge, and has meaningful description text.
|
|
3. **Option ownership survives graph construction.** The `may_cause` relationship from the client-retention unknown to `opt_relocate` makes clear this uncertainty belongs to the Relate option — enabling future reasoning to correctly associate upside/downside with the right candidate.
|
|
|
|
## What this does NOT prove:
|
|
|
|
1. **Whether the materiality rule can target follow-up questions at the specific material factor.** The generic question suggests structural recognition but interrogative gap.
|
|
2. **Whether the observation-node behaviour is consistent** (60B.5 created one, 60B.6 did not). This may be context-dependent or model-stochastic rather than rule-driven.
|
|
3. **Cross-domain robustness.** Single scenario with a single model.
|
|
4. **What happens with multiple concurrent material uncertainties.** Tested exactly one unresolved factor.
|
|
|
|
## Production code changed: NO
|
|
## Prompt changed: NO
|
|
## Validator changed: NO
|
|
## Harness changed: NO
|
|
## Vitest run: NO
|
|
## Ollama calls: 1
|
|
## Direct API calls: 0
|
|
## Dev server disturbed: NO
|