203 lines
10 KiB
Markdown
203 lines
10 KiB
Markdown
# Experiment 60B.1 — Decision Sufficiency on Option Graph
|
|
|
|
**Branch:** `feature/decision-options-v0.25`
|
|
**Date:** 2026-08-13
|
|
**Status:** Complete
|
|
**Type:** LIVE RUN — Single bounded update to test whether the engine recognises when remaining material consequences have been quantified and resolves the existing decision rather than inventing another uncertainty.
|
|
|
|
## Objective
|
|
|
|
When the supplied answer provides fully quantified financial impacts for both options and states there are no other material differences, does the engine resolve the existing "Which option leaves us better off overall?" decision context rather than creating another generic unknown?
|
|
|
|
## Hypothesis
|
|
|
|
A strong result should:
|
|
- Preserve both existing option identities (opt_relocate, opt_stay_put)
|
|
- Preserve the shared decision context (n_relocation_decision)
|
|
- Represent the £600k one-off relocation cost as first-class structure
|
|
- Preserve the £2m/year stay-put cost structurally
|
|
- Recognise that no material comparison uncertainty remains
|
|
- Resolve the existing decision context rather than creating another generic unknown
|
|
|
|
## Fixed Starting Graph
|
|
|
|
**Fixture:** `tests/fixtures/pre-anchored-decision-options.json`
|
|
|
|
Pre-existing structure (4 nodes, 2 edges):
|
|
|
|
| Node | Kind | Status | Label |
|
|
|------|------|--------|-------|
|
|
| n_relocation_state | state | provisional | Engineering team relocation consideration |
|
|
| opt_relocate | option | known | Relocate to Manchester |
|
|
| opt_stay_put | option | known | Stay in London (Status Quo) |
|
|
| n_relocation_decision | unknown | unknown | Which option leaves us better off overall? |
|
|
|
|
Edges: opt_relocate → n_relocation_decision (contained_in); opt_stay_put → n_relocation_decision (contained_in).
|
|
|
|
## Configured Model
|
|
|
|
- **Model:** qwen-claude:latest
|
|
- **Ollama base URL:** http://192.168.1.111:11434
|
|
|
|
## Fixed Answer (verbatim)
|
|
|
|
> We have now quantified the full financial impact of the relocation disruption, including replacing the two senior engineers and the delivery delay, at about £600,000 as a one-off cost. Staying put still costs us an extra £2 million every year. There are no other material differences between the two options that we need to account for.
|
|
|
|
## Execution
|
|
|
|
Host/model: qwen-claude:latest at http://192.168.1.111:11434. startCalls=0, updateCalls=1, totalCalls=1. Hard one-call boundary.
|
|
|
|
## Result
|
|
|
|
**HTTP status:** 200 — first call succeeds, no validation rejection.
|
|
|
|
### Proposal mutations
|
|
|
|
```
|
|
updatedNodes: [
|
|
{nodeId:"n_relocation_decision", previousStatus:"unknown", newStatus:"resolved",
|
|
newValue:"Relocate to Manchester is financially superior due to lower net cost (£600k one-off vs £2M/year).",
|
|
reason:"User provided quantified financial impacts showing relocation is cheaper overall with no other material differences."},
|
|
{nodeId:"opt_relocate", previousStatus:"known", newStatus:"known", newValue:600000,
|
|
reason:"Quantified financial impact of relocation disruption established by user."},
|
|
{nodeId:"opt_stay_put", previousStatus:"known", newStatus:"known", newValue:2000000,
|
|
reason:"Quantified annual extra cost of staying in London established by user."}
|
|
]
|
|
|
|
resolvedUnknownNodeIds: ["n_relocation_decision"]
|
|
|
|
addedNodes: [
|
|
{id:"metric_relocation_cost", label:"Relocation one-off cost", kind:"metric",
|
|
status:"known", confidence:"high", value:600000, unit:"GBP", childIds:["opt_relocate"]},
|
|
{id:"metric_stay_annual_cost", label:"Staying put annual cost", kind:"metric",
|
|
status:"known", confidence:"high", value:2000000, unit:"GBP/year", childIds:["opt_stay_put"]}
|
|
]
|
|
|
|
addedEdges: [
|
|
{id:"e-metric-rel-to-opt", from:"metric_relocation_cost", to:"opt_relocate", relationship:"measures"},
|
|
{id:"e-metric-stay-to-opt", from:"metric_stay_annual_cost", to:"opt_stay_put", relationship:"measures"}
|
|
]
|
|
|
|
selectedQuestion: null (none — decision resolved)
|
|
```
|
|
|
|
### Resulting persistent graph (6 nodes, 4 edges)
|
|
|
|
| Node | Kind | Status | Label |
|
|
|------|------|--------|-------|
|
|
| n_relocation_state | state | provisional | Engineering team relocation consideration |
|
|
| opt_relocate | option | known | Relocate to Manchester |
|
|
| opt_stay_put | option | known | Stay in London (Status Quo) |
|
|
| n_relocation_decision | **unknown** → **resolved** | **resolved** | Which option leaves us better off overall? |
|
|
| metric_relocation_cost | **metric** | **known** | Relocation one-off cost |
|
|
| metric_stay_annual_cost | **metric** | **known** | Staying put annual cost |
|
|
|
|
Edges:
|
|
- opt_relocate → n_relocation_decision (contained_in)
|
|
- opt_stay_put → n_relocation_decision (contained_in)
|
|
- metric_relocation_cost → opt_relocate (measures)
|
|
- metric_stay_annual_cost → opt_stay_put (measures)
|
|
|
|
## Assessment
|
|
|
|
### 1. Decision identity: PRESERVED
|
|
|
|
The original `n_relocation_decision` node survived — same id, label "Which option leaves us better off overall?". Status transitioned from `unknown` → `resolved`. Not duplicated or replaced. Exactly one decision-context unknown.
|
|
|
|
### 2. Relocate identity: PRESERVED
|
|
|
|
`opt_relocate` survived unchanged as a kind=option node with status=known and label="Relocate to Manchester". Updated via newValue=600000 on the update list, but the original node was not replaced or duplicated. Count: 1.
|
|
|
|
### 3. Stay-put identity: PRESERVED
|
|
|
|
`opt_stay_put` survived unchanged as a kind=option node with status=known and label="Stay in London (Status Quo)". Updated via newValue=2000000 on the update list, but not replaced or duplicated. Count: 1.
|
|
|
|
### 4. £600k relocation cost: FIRST-CLASS STRUCTURE
|
|
|
|
The engine created `metric_relocation_cost` as a dedicated metric node with:
|
|
- **value:** 600000 (numeric, not prose)
|
|
- **unit:** "GBP" (structured unit field)
|
|
- **kind:** "metric"
|
|
- **status:** "known"
|
|
- **label:** "Relocation one-off cost"
|
|
- **childIds:** ["opt_relocate"]
|
|
- **edge:** measures → opt_relocate
|
|
|
|
This is first-class graph structure with typed edges and numeric value.
|
|
|
|
### 5. £2m/year stay-put cost: PRESERVED STRUCTURALLY
|
|
|
|
The engine created `metric_stay_annual_cost` as a dedicated metric node with:
|
|
- **value:** 2000000 (numeric)
|
|
- **unit:** "GBP/year" (structured unit with recurrence)
|
|
- **kind:** "metric"
|
|
- **status:** "known"
|
|
- **childIds:** ["opt_stay_put"]
|
|
- **edge:** measures → opt_stay_put
|
|
|
|
Also preserved as newValue=2000000 on the opt_stay_put updatedNode entry. Both structural and option-description preservation.
|
|
|
|
### 6. Comparison completeness: CLEAR
|
|
|
|
The resulting graph preserves the stated comparison:
|
|
- Relocate: £600k one-off cost (metric_relocation_cost, value=600000, unit="GBP")
|
|
- Stay put: £2m/year recurring cost (metric_stay_annual_cost, value=2000000, unit="GBP/year")
|
|
|
|
Both are first-class metric nodes with typed edges to their respective options. The comparison is fully represented and graph-reasonable.
|
|
|
|
### 7. Decision sufficiency: RECOGNISES SUFFICIENT EVIDENCE
|
|
|
|
The engine set `n_relocation_decision` status from "unknown" → "resolved" with newValue describing the financial superiority of Relocate. It recognised that the user-provided quantified impacts (no remaining material differences) were sufficient to close the decision context. No generic or fabricated uncertainty was created.
|
|
|
|
### 8. Decision resolution: CORRECTLY RESOLVED
|
|
|
|
The existing `n_relocation_decision` node was resolved in place — same id, correct direction. Not duplicated, not replaced with a new decision node.
|
|
|
|
### 9. Conclusion direction: FAVOURS RELOCATE
|
|
|
|
The resolved newValue states: "Relocate to Manchester is financially superior due to lower net cost (£600k one-off vs £2M/year)." This is consistent with the supplied economics — a £600k one-off versus £2m/year recurring cost clearly favours relocation on the stated criteria.
|
|
|
|
### 10. New uncertainty discipline: NONE
|
|
|
|
Zero new unknown nodes were created. Only two metric evidence nodes were added (one for each cost), and the existing decision was resolved. No generic follow-up question generated (selectedQuestion=null).
|
|
|
|
## Classification: A — DECISION SUFFICIENCY RECOGNISED
|
|
|
|
The engine preserved both existing option identities, represented both financial impacts as first-class metric structure with correct option ownership, recognised that the user had stated no remaining material differences, and resolved the existing decision context without inventing any new uncertainty. The resolution direction (favouring Relocate) is consistent with the supplied economics.
|
|
|
|
## What the engine understood correctly:
|
|
|
|
1. **Both costs are material consequences to be compared** — created distinct metric nodes for each with correct units (GBP vs GBP/year).
|
|
2. **Evidence ownership is correct** — metric_relocation_cost → opt_relocate, metric_stay_annual_cost → opt_stay_put.
|
|
3. **Sufficiency recognition** — treated the user's "no other material differences" statement as a boundary condition that closes the decision context.
|
|
4. **In-place resolution** — resolved n_relocation_decision rather than creating a new decision node or generic unknown.
|
|
5. **Correct direction** — concluded Relocate is financially superior, consistent with £600k one-off vs £2M/year recurring.
|
|
|
|
## What it did NOT do:
|
|
|
|
1. **Did not create any new uncertainty** — zero unknown nodes added.
|
|
2. **Did not generate a follow-up question** — selectedQuestion=null (decision complete).
|
|
3. **Did not duplicate option or decision nodes** — all three entities (opt_relocate, opt_stay_put, n_relocation_decision) appear exactly once.
|
|
|
|
## What this establishes:
|
|
|
|
1. The engine can recognise when the user has provided sufficient evidence to resolve a decision context, treating "no other material differences" as a valid closing condition.
|
|
2. Financial comparison facts for both options are represented as first-class structure with correct unit types (one-off vs recurring) and option ownership.
|
|
3. Decision resolution can occur in-place on an existing unknown node without creating duplication.
|
|
|
|
## What this does NOT prove:
|
|
|
|
1. **Stability across repeated runs** — one run only; cold-start variance may produce different outcomes on repeated runs.
|
|
2. **Resolution under ambiguity** — tested with clearly quantified costs and explicit "no other differences" statement; not tested with partial or ambiguous evidence.
|
|
3. **Cross-domain generalisation** — single domain case only.
|
|
4. **Quality of resolution rationale** — the engine did produce a correct direction, but we did not test whether it can distinguish between quantitatively close options (e.g., £1.8M vs £2M/year).
|
|
|
|
## Production code changed: NO
|
|
## Prompt changed: NO
|
|
## Validator changed: NO
|
|
## Harness changed: NO
|
|
## Vitest run: NO
|
|
## Ollama calls: 1
|
|
## Direct API calls: 0
|
|
## Dev server disturbed: NO
|