experiment: test materiality rule against real unresolved factor
This commit is contained in:
@@ -2699,3 +2699,42 @@ The engine resolved the existing decision context independently. All three entit
|
||||
- Decision direction expressed: newValue = "Relocate to Manchester"
|
||||
|
||||
**One live call at qwen-claude:latest on http://127.0.0.1:3000. No production code changed.**
|
||||
|
||||
## Experiment 60B.6 — Materiality Rule Against Real Unresolved Factor (2026-08-13)
|
||||
|
||||
Test whether the materiality rule keeps a decision open when one genuinely decision-changing uncertainty remains — the opposite boundary of 60B.5's over-closure case.
|
||||
|
||||
**Fixture:** `tests/fixtures/pre-anchored-decision-options.json`
|
||||
**Answer (verbatim):** "We have now quantified the full financial impact of replacing the two senior engineers and the delivery delay at about £600,000 as a one-off relocation cost. Staying put costs us an extra £2 million every year. The remaining issue is our largest client: we do not yet know whether they would leave if we relocated, and losing them would cost us about £5 million per year."
|
||||
|
||||
**One live call at qwen-claude:latest on http://127.0.0.1:3000.**
|
||||
|
||||
### Result: Classification B — MATERIAL FACTOR RECOGNISED BUT STRUCTURE PARTIAL
|
||||
|
||||
The engine correctly keeps the decision open because of the client-retention uncertainty, creates a first-class unknown node for it, and clearly attributes it to the Relocate option via a `may_cause` edge. However, three structural weaknesses prevent an A:
|
||||
|
||||
1. **No observation node for known financial comparison.** Unlike 60B.5 (which created `n_fin_quantification`), 60B.6 did not extract the £600k and £2M/year figures into a new observation node. The quantified comparison survives only implicitly through the original option descriptions, not as structured evidence.
|
||||
2. **The £5M/year downside is text-only.** The description mentions "~£5M/year" but it is embedded in prose rather than isolated as a structured numeric consequence.
|
||||
3. **Selected question is generic despite known material factor.** The engine asks "What outcome would demonstrate enough value to justify continuing?" — the same generic phrasing from 60B.2 — when it has just created a specific client-retention unknown. It should have targeted that uncertainty directly (e.g., "Will the organisation's largest client leave if we relocate to Manchester?").
|
||||
|
||||
### Structural details
|
||||
|
||||
```
|
||||
updatedNodes: []
|
||||
resolvedUnknownNodeIds: []
|
||||
addedNodes: [n_client_retention — kind=unknown, status=unknown, label="Largest client retention uncertainty"]
|
||||
addedEdges: [n_client_retention → opt_relocate, relationship="may_cause"]
|
||||
selectedQuestion: "What outcome would demonstrate enough value to justify continuing?" (generic)
|
||||
```
|
||||
|
||||
**What worked:**
|
||||
- Decision identity preserved (status remained unknown)
|
||||
- Both option identities preserved
|
||||
- Client-retention uncertainty created as a first-class unknown node
|
||||
- Client risk clearly attributed to Relocate via `may_cause` edge and `childIds` on the unknown node
|
||||
- No fabricated factors or over-closure
|
||||
|
||||
**What was weak:**
|
||||
- Generic question despite having just identified a specific material factor
|
||||
- No observation node for the known financial comparison (unlike 60B.5)
|
||||
- £5M/year not isolated from description text
|
||||
|
||||
@@ -0,0 +1,193 @@
|
||||
# Experiment 60B.6 — Test Materiality Rule Against Real Unresolved Factor
|
||||
|
||||
**Branch:** `feature/decision-sufficiency-v0.26`
|
||||
**Date:** 2026-08-13
|
||||
**Status:** Complete
|
||||
**Type:** LIVE RUN — Single-call experiment testing the materiality rule from 60B.4 against a genuinely decision-changing uncertainty.
|
||||
|
||||
## Objective
|
||||
|
||||
When the quantified comparison favours one option but one specific unresolved factor could realistically reverse that preference, does the engine keep the decision open for that factor rather than over-closing?
|
||||
|
||||
## Following
|
||||
|
||||
Experiment 60B.2 (over-closure: resolves decision when £5M client risk remains)
|
||||
Experiment 60B.4 (the fix: prompt-only materiality rule)
|
||||
Experiment 60B.5 (confirmation: materiality rule can close without explicit cue)
|
||||
|
||||
This is the **opposite boundary** of 60B.5's success case. 60B.5 tested "should it close when nothing materially unresolved?" — and it did. 60B.6 tests "should it stay open when something materially unresolved exists?" — and should keep open for that specific factor.
|
||||
|
||||
## Fixed Starting Graph
|
||||
|
||||
**Fixture:** `tests/fixtures/pre-anchored-decision-options.json`
|
||||
|
||||
| Node | Kind | Status | Label |
|
||||
|------|------|--------|-------|
|
||||
| n_relocation_state | state | provisional | Engineering team relocation consideration |
|
||||
| opt_relocate | option | known | Relocate to Manchester |
|
||||
| opt_stay_put | option | known | Stay in London (Status Quo) |
|
||||
| n_relocation_decision | unknown | unknown | Which option leaves us better off overall? |
|
||||
|
||||
## Configured Model
|
||||
|
||||
- **Model:** qwen-claude:latest
|
||||
- **Ollama base URL:** http://127.0.0.1:3000 (dev server)
|
||||
|
||||
## Fixed Answer (verbatim, exact)
|
||||
|
||||
> We have now quantified the full financial impact of replacing the two senior engineers and the delivery delay at about £600,000 as a one-off relocation cost. Staying put costs us an extra £2 million every year. The remaining issue is our largest client: we do not yet know whether they would leave if we relocated, and losing them would cost us about £5 million per year.
|
||||
|
||||
## Why This Case Matters
|
||||
|
||||
Without the client risk, Relocate (£600k one-off) vs Stay (£2M/year recurring) clearly favours Relocate. The £5M/year client-retention risk is large enough to plausibly reverse that preference — making it a genuine decision-changing uncertainty.
|
||||
|
||||
## Execution
|
||||
|
||||
Single update call. No retries. Hard boundary enforced by harness.
|
||||
|
||||
## Call Accounting
|
||||
|
||||
```
|
||||
startCalls: 0
|
||||
updateCalls: 1
|
||||
totalCalls: 1
|
||||
Retries: 0
|
||||
```
|
||||
|
||||
## HTTP Response
|
||||
|
||||
- **Status:** 200 — success, no rejection
|
||||
- **Proposal applied:** YES (accepted)
|
||||
|
||||
## Proposal Mutations
|
||||
|
||||
```
|
||||
updatedNodes: []
|
||||
resolvedUnknownNodeIds: []
|
||||
|
||||
addedNodes: [{
|
||||
id: "n_client_retention",
|
||||
label: "Largest client retention uncertainty",
|
||||
description: "Whether the organisation's largest client will leave if the team relocates to Manchester, matters because their departure would cost ~£5M/year and likely negate the £2M annual savings needed to justify relocation.",
|
||||
kind: "unknown",
|
||||
status: "unknown",
|
||||
confidence: "medium"
|
||||
}]
|
||||
|
||||
addedEdges: [{
|
||||
id: "e-client-to-opt",
|
||||
fromNodeId: "n_client_retention",
|
||||
toNodeId: "opt_relocate",
|
||||
relationship: "may_cause",
|
||||
confidence: "medium",
|
||||
description: "Relocating to Manchester may cause the largest client to leave"
|
||||
}]
|
||||
|
||||
selectedQuestion: "What outcome would demonstrate enough value to justify continuing?"
|
||||
```
|
||||
|
||||
## Resulting Persistent Graph (5 nodes, 3 edges)
|
||||
|
||||
| Node | Kind | Status | Label |
|
||||
|------|------|--------|-------|
|
||||
| n_relocation_state | state | provisional | Engineering team relocation consideration |
|
||||
| opt_relocate | option | known | Relocate to Manchester |
|
||||
| opt_stay_put | option | known | Stay in London (Status Quo) |
|
||||
| n_relocation_decision | unknown | unknown | Which option leaves us better off overall? |
|
||||
| n_client_retention | unknown | unknown | Largest client retention uncertainty |
|
||||
|
||||
Edges:
|
||||
- opt_relocate → n_relocation_decision (contained_in)
|
||||
- opt_stay_put → n_relocation_decision (contained_in)
|
||||
- n_client_retention → opt_relocate (may_cause)
|
||||
|
||||
## Assessment
|
||||
|
||||
### 1. Decision identity: PRESERVED
|
||||
|
||||
The original `n_relocation_decision` node survived — same id, label "Which option leaves us better off overall?", status=unknown. Not resolved. Count: 1.
|
||||
|
||||
### 2. Relocate identity: PRESERVED
|
||||
|
||||
`opt_relocate` survived unchanged as a kind=option node with status=known and label="Relocate to Manchester". Count: 1.
|
||||
|
||||
### 3. Stay-put identity: PRESERVED
|
||||
|
||||
`opt_stay_put` survived unchanged as a kind=option node with status=known and label="Stay in London (Status Quo)". Count: 1.
|
||||
|
||||
### 4. Known financial comparison (£600k + £2M/year)
|
||||
|
||||
**No new observation node created.** Unlike 60B.5 (which created `n_fin_quantification`), the quantified figures exist only in the user answer text, not as structured evidence nodes in the graph. They are implicitly available in the option descriptions but not explicitly captured as comparison evidence.
|
||||
|
||||
Classification: **PARTIAL** (figures survive in option descriptions but no new first-class observation structure was created)
|
||||
|
||||
### 5. Client-retention uncertainty: FIRST-CLASS UNKNOWN
|
||||
|
||||
Created as `n_client_retention` with kind=unknown, status=unknown. Has a description explaining the factor and its relevance. This is a proper first-class unknown node — not text-only, not flattened into an option's internal state.
|
||||
|
||||
Classification: **FIRST-CLASS UNKNOWN**
|
||||
|
||||
### 6. Client-risk ownership to Relate: CLEARLY OWNED BY RELOCATE
|
||||
|
||||
The `n_client_retention` node has `childIds: ["opt_relocate"]` and a typed edge `n_client_retention → opt_relocate` with relationship="may_cause". Graph-only reasoning can determine the unresolved client risk belongs to the Relocate option.
|
||||
|
||||
Classification: **CLEARLY OWNED BY RELOCATE**
|
||||
|
||||
### 7. £5M/year downside: PRESERVED WITH UNKNOWN
|
||||
|
||||
The "~£5M/year" figure is embedded in the description text of `n_client_retention`. It is not isolated as a separate structured numeric value but survives within the unknown node's epistemic container.
|
||||
|
||||
Classification: **PRESERVED WITH UNKNOWN**
|
||||
|
||||
### 8. Materiality judgment: RECOGNISED BUT WEAKLY
|
||||
|
||||
The engine correctly kept the decision open and created the client-retention unknown, demonstrating it recognised this factor as material. However, the recognition is structural (it created the node) but not interrogative (the selected question does not target it). The materiality rule prevented over-closure but did not fully activate the follow-up targeting the specific material factor.
|
||||
|
||||
Classification: **RECOGNISED BUT WEAKLY**
|
||||
|
||||
### 9. Decision treatment: KEPT OPEN FOR SPECIFIC MATERIAL REASON
|
||||
|
||||
The decision was kept open — `n_relocation_decision` remains unknown, nothing resolved. The newly added `n_client_retention` is clearly the specific material reason (a client retention uncertainty with £5M/year downside that could reverse the preference). No unrelated uncertainty invented.
|
||||
|
||||
Classification: **KEPT OPEN FOR SPECIFIC MATERIAL REASON**
|
||||
|
||||
### 10. Selected question: WEAK
|
||||
|
||||
The question "What outcome would demonstrate enough value to justify continuing?" is generic — the same phrasing from 60B.2. The engine has just created a specific client-retention unknown and should have targeted it with something like "Will the organisation's largest client leave if we relocate to Manchester?"
|
||||
|
||||
Classification: **WEAK**
|
||||
|
||||
## Classification: B — MATERIAL FACTOR RECOGNISED BUT STRUCTURE PARTIAL
|
||||
|
||||
The engine correctly continues for the client risk (classification B requires this) but its representation or question specificity is incomplete.
|
||||
|
||||
### Critical evidence check
|
||||
|
||||
- Decision remains unresolved: **YES**
|
||||
- Client-retention uncertainty survives: **YES**
|
||||
- Client risk attributable to Relocate: **YES**
|
||||
- £5M/year impact survives: **YES** (embedded in description text, not isolated)
|
||||
- Generic substitute question replaces specific targeting: **NO — it uses generic phrasing instead of targeting the specific unknown**
|
||||
- No unnecessary new uncertainty invented: **YES**
|
||||
|
||||
## What this establishes:
|
||||
|
||||
1. **The materiality rule prevents over-closure with a real decision-changing factor.** When an unresolved £5M/year client-retention risk exists, the engine does NOT resolve the decision — correctly keeping it open.
|
||||
2. **The engine can create first-class unknown nodes for option-specific risks.** The `n_client_retention` node is properly typed, attributed to the correct option via a `may_cause` edge, and has meaningful description text.
|
||||
3. **Option ownership survives graph construction.** The `may_cause` relationship from the client-retention unknown to `opt_relocate` makes clear this uncertainty belongs to the Relate option — enabling future reasoning to correctly associate upside/downside with the right candidate.
|
||||
|
||||
## What this does NOT prove:
|
||||
|
||||
1. **Whether the materiality rule can target follow-up questions at the specific material factor.** The generic question suggests structural recognition but interrogative gap.
|
||||
2. **Whether the observation-node behaviour is consistent** (60B.5 created one, 60B.6 did not). This may be context-dependent or model-stochastic rather than rule-driven.
|
||||
3. **Cross-domain robustness.** Single scenario with a single model.
|
||||
4. **What happens with multiple concurrent material uncertainties.** Tested exactly one unresolved factor.
|
||||
|
||||
## Production code changed: NO
|
||||
## Prompt changed: NO
|
||||
## Validator changed: NO
|
||||
## Harness changed: NO
|
||||
## Vitest run: NO
|
||||
## Ollama calls: 1
|
||||
## Direct API calls: 0
|
||||
## Dev server disturbed: NO
|
||||
Reference in New Issue
Block a user