7.5 KiB
Experiment 59B.1 — Decision Relevance of Next Question vs Precision Chasing
Branch: feature/question-formulation-v0.24
Date: 2026-08-12
Status: Complete
Following: 59A series which showed the engine can distinguish known consequences from uncertain downstream effects at the proposal level.
Objective
When the financial benefit is known, the downside is bounded, and the cost of doing nothing is explicit, does the engine compare decision consequences — or does it simply ask for more precision about the remaining uncertainty?
Specifically: does the engine recognise when greater precision about an uncertainty may not actually matter to the decision?
Context route
Read only:
docs/current-handoff.mddocs/experiment-59a3.md(preceding reasoning context)- Fixture:
tests/fixtures/pre-anchored-update-savings-realism.json - Harness:
scripts/reproduce-multi-turn-investigation.mjs
Do not load older experiment history.
Fixed starting graph
Fixture: tests/fixtures/pre-anchored-update-savings-realism.json
Existing unresolved question:
n_savings_realism — Are the projected office savings from relocation realistic? — status = unknown
Fixed answer
The £2 million annual saving is real. We know two senior engineers would leave if we relocate. Even in the worst case, I think that would delay delivery by no more than two months. If we stay where we are, we keep paying the extra £2 million every year. I am trying to decide whether getting a more precise delivery-delay estimate would actually change the decision.
Contains:
- KNOWN BENEFIT: £2 million annual saving
- KNOWN CONSEQUENCE: two senior engineers would leave
- BOUNDED DOWNSIDE: delivery delay no worse than approximately two months
- DO-NOTHING BASELINE: staying put costs an extra £2 million every year
- DECISION-RELEVANCE QUESTION: would greater precision about delivery delay actually change the decision?
Execution
Host/model: qwen-claude:latest at http://192.168.1.111:11434 Calls: startCalls=0, updateCalls=1, totalCalls=1 Retries: 0
Results
UPDATE output
HTTP status: 200
Stage: update_applied
Validation errors: none
updatedNodes: [{nodeId:n_savings_realism, previousStatus:unknown, newStatus:resolved, newValue:'£2 million annual savings confirmed as real', reason:'User explicitly confirmed the projected office savings are real and provided exact figures.'}]
resolvedUnknownNodeIds: ['n_savings_realism']
addedNodes: [{id:n_delay_estimate_relevance, label:'Would a precise delivery-delay estimate change the decision?', kind:unknown, status:unknown}]
addedEdges: [{fromNodeId:n_delay_estimate_relevance, toNodeId:n_savings_realism, relationship:depends_on}]
selectedQuestion: 'would a precise delivery-delay estimate change the decision?'
Resulting graph (3 nodes, 2 edges):
n_relocation_state— Engineering team relocation consideration — status=provisionaln_savings_realism— Are the projected office savings from relocation realistic? — status=resolved ✓n_delay_estimate_relevance— Would a precise delivery-delay estimate change the decision? — status=unknown
Assessment
1. £2m annual saving
PRESERVED AS KNOWN BENEFIT
Node n_savings_realism resolved with value "£2 million annual savings confirmed as real".
2. Two-engineer departure
LOST No node in the graph represents "two senior engineers would leave." Not preserved structurally, not visible in any node label/description/value. The answer clearly stated this as a known consequence but it was dropped.
3. Two-month downside bound
UNAVAILABLE No node captures "no more than two months" or any upper bound on delivery delay. The engine did not weaken it to open-ended uncertainty explicitly, but the information is simply absent from the graph.
4. Do-nothing baseline
LOST
"Staying put costs an extra £2m every year" is not structurally represented as a cost node, a comparison edge, or any structural element of the graph. n_relocation_state has no do-nothing semantics.
5. Decision framing
TRADE-OFF PRESENT BUT BASELINE LOST The engine selected a question about whether precision matters to the decision — this is trade-off thinking at a meta-level. However, the baseline (cost of staying put) is lost structurally, so the trade-off has no anchoring.
6. Next-question decision relevance
HIGH DECISION RELEVANCE "Would a precise delivery-delay estimate change the decision?" — If answered yes, it would justify further investigation; if answered no, it would stop precision-seeking. This directly addresses the user's stated concern about whether more precision is worth obtaining.
7. Precision chasing
NO PRECISION CHASING The engine did NOT ask "what exactly is the delivery delay?" It asked a meta-level question about decision relevance of precision itself. However, this positive result is partially undermined by the fact that critical contextual facts (engineer departure, two-month bound) were lost before the question was formulated.
Classification: D — DO-NOTHING BASELINE LOST
The engine evaluated relocation consequences without preserving the recurring cost of staying put as a structural element. Two additional losses compound this:
- The known consequence ("two senior engineers would leave") was entirely lost from the graph.
- The bounded downside ("no more than two months") was absent from the graph.
A positive finding: the engine did not ask "what exactly is the delay?" — it asked whether precision matters at all, which is a valid decision-relevant next step. However, this question lacks structural grounding because the critical comparison elements (engineer loss, bounded impact, do-nothing cost) are not present in the graph to give the question context.
What this establishes:
- The engine can formulate a genuinely meta-level decision-relevance question when prompted by an answer that explicitly raises it ("I am trying to decide whether getting a more precise delivery-delay estimate would actually change the decision").
- The engine does not default to precision-chasing (asking for exact values) when the user signals that decision relevance matters.
- Known benefit preservation works: £2m savings survived as resolved on
n_savings_realism.
What this does NOT prove:
- That the engine would independently recognise decision irrelevance without an explicit user prompt about it — the answer text contained "I am trying to decide whether getting a more precise delivery-delay estimate would actually change the decision," which is a very strong signal that guided question selection.
- That the engine preserves known consequences alongside benefits — engineer departure was entirely lost.
- That the engine preserves bounded downside information — the two-month upper bound disappeared.
- Whether these losses are due to answerMeaning extraction limits, proposal generation limits, or node-kinds being misclassified.
Key observation
The engine's meta-level question framing is structurally intelligent but contextually hollow. It asked the right kind of question (is precision worth it?) but lost the facts that make that question meaningful (what happens if we relocate? what are the bounds? what does doing nothing cost?). This suggests a context-preservation deficit in the update path: when the engine resolves one uncertainty and creates a new decision-relevance node, it drops other critical information from the answer rather than carrying it forward.
Production code changed: NO