confidence-engine/docs/experiment-60b12.md

Experiment 60B.12 — Live Verification of Prerequisite-Aware Question Targeting

Branch: feature/question-target-alignment-v0.27
Starting HEAD: 3c6e436
Date: 2026-08-13
Status: BLOCKED (apparatus failure)
Type: LIVE RUN — Single-call verification of 60B.11 prerequisite-aware targeting

Objective

Run one bounded Update to answer:

Does the engine now keep the decision open for the client-retention uncertainty AND make that same unknown the final selected question target?

Following

Experiment 60B.6 (materiality rule with real unresolved factor)
Experiment 60B.11 (prerequisite-aware preferred targeting implemented in production code)

This is the live regression 60B.11 explicitly left unproven:

the exact 60B.6 live continuation case,
where the model selects the newly exposed client-retention factor
and the final selected target preserves that same ready material unknown

Fixed Starting Graph

Fixture: tests/fixtures/pre-anchored-decision-options.json

Node	Kind	Status	Label
n_relocation_state	state	provisional	Engineering team relocation consideration
opt_relocate	option	known	Relocate to Manchester
opt_stay_put	option	known	Stay in London (Status Quo)
n_relocation_decision	unknown	unknown	Which option leaves us better off overall?

Configured Model

• Model: qwen-claude:latest
• Ollama base URL: http://192.168.1.111:11434 (from .env.local)

Fixed Answer (verbatim, exact)

We have now quantified the full financial impact of replacing the two senior engineers and the delivery delay at about £600,000 as a one-off relocation cost. Staying put costs us an extra £2 million every year. The remaining issue is our largest client: we do not yet know whether they would leave if we relocated, and losing them would cost us about £5 million per year.

Execution

Exactly one update call through the production route via reproduce-multi-turn-investigation.mjs in updateOnly mode.

Call Accounting

startCalls: 0
updateCalls: 1
totalCalls: 1
Retries: 0

HTTP Response

• Status: 500 — rejected during validation
• Stage: result_validation
• Proposal applied: NO (rejected)

Rejection Error

Active unknown violates reasoning pattern consistency: "n_client_retention_uncertainty" is diagnosis but active pattern is decision

The model attempted to create a node with id n_client_retention_uncertainty and kind "diagnosis". The active reasoning pattern is "decision", which does not allow the "diagnosis" kind for unknown nodes. This is a structural/pattern consistency validation failure — not a materiality or targeting question.

Note: 60B.6 used node id n_client_retention with kind "unknown". The model in this run produced n_client_retention_uncertainty with kind "diagnosis" — different ID and different kind, which triggered the validator rejection before any proposal could be applied.

Structural Action Required

UNAVAILABLE (rejection occurred before structural data was exposed)

Assessment

Materiality behaviour: UNAVAILABLE

Cannot assess — no proposal applied.

Client-retention uncertainty: UNAVAILABLE

Cannot assess — model produced n_client_retention_uncertainty (kind=diagnosis) rather than a compatible unknown node.

Client-risk ownership: UNAVAILABLE

Cannot assess.

Preferred-target behaviour: UNAVAILABLE

Cannot assess — rejected before proposal application.

Question text: NONE

No question returned.

Prerequisite guard: UNAVAILABLE

Cannot assess — prerequisite checking occurs after proposal validation.

60B.6 vs 60B.12 Comparison

Field	60B.6	60B.12
Final nodeId	(created n_client_retention, but generic question)	UNAVAILABLE — rejected
Decision status	unresolved (PRESERVED)	UNAVAILABLE
Client-retention unknown	YES (n_client_retention)	UNAVAILABLE

In 60B.6 the model produced kind=unknown with id n_client_retention. In 60B.12 the model produced kind=diagnosis with id n_client_retention_uncertainty — a different node name and an incompatible kind for the active decision pattern.

Classification: H — BLOCKED

Apparatus (reasoning-pattern consistency validator) rejected the model's proposal before inference could be assessed. The blocker is not the 60B.11 targeting fix but a schema-level incompatibility between what the model produced (kind=diagnosis) and what the active pattern permits.

Critical evidence

• No production code changed during this experiment
• No prompt changes to question-targeting logic — this failure is at the pattern-consistency layer
• The node id mismatch (60B.6 used n_client_retention; 60B.12 model produced n_client_retention_uncertainty) suggests stochastic variation in model output naming
• The kind mismatch (unknown vs diagnosis) is the actual validation blocker

What this establishes

1. The live server was reachable and the update-only harness executed correctly.
2. The reasoning-pattern consistency validator catches kind mismatches between model output and active pattern before any proposal mutation.
3. Further live testing requires either (a) matching what 60B.6 did — producing kind=unknown with a compatible id — or (b) relaxing the active pattern to accept diagnosis nodes.

Production code changed: NO

Prompt changed during experiment: NO

Validator changed: NO

Schema changed: NO

Harness changed: NO

Vitest run: NO

Ollama calls: 1

Direct API calls: 0

Dev server disturbed: NO