137 lines
5.6 KiB
Markdown
137 lines
5.6 KiB
Markdown
# Experiment 60B.12 — Live Verification of Prerequisite-Aware Question Targeting
|
|
|
|
**Branch:** `feature/question-target-alignment-v0.27`
|
|
**Starting HEAD:** `3c6e436`
|
|
**Date:** 2026-08-13
|
|
**Status:** BLOCKED (apparatus failure)
|
|
**Type:** LIVE RUN — Single-call verification of 60B.11 prerequisite-aware targeting
|
|
|
|
## Objective
|
|
|
|
Run one bounded Update to answer:
|
|
|
|
> Does the engine now keep the decision open for the client-retention uncertainty AND make that same unknown the final selected question target?
|
|
|
|
## Following
|
|
|
|
Experiment 60B.6 (materiality rule with real unresolved factor)
|
|
Experiment 60B.11 (prerequisite-aware preferred targeting implemented in production code)
|
|
|
|
This is the **live regression** 60B.11 explicitly left unproven:
|
|
|
|
```text
|
|
the exact 60B.6 live continuation case,
|
|
where the model selects the newly exposed client-retention factor
|
|
and the final selected target preserves that same ready material unknown
|
|
```
|
|
|
|
## Fixed Starting Graph
|
|
|
|
**Fixture:** `tests/fixtures/pre-anchored-decision-options.json`
|
|
|
|
| Node | Kind | Status | Label |
|
|
|------|------|--------|-------|
|
|
| n_relocation_state | state | provisional | Engineering team relocation consideration |
|
|
| opt_relocate | option | known | Relocate to Manchester |
|
|
| opt_stay_put | option | known | Stay in London (Status Quo) |
|
|
| n_relocation_decision | unknown | unknown | Which option leaves us better off overall? |
|
|
|
|
## Configured Model
|
|
|
|
- **Model:** qwen-claude:latest
|
|
- **Ollama base URL:** http://192.168.1.111:11434 (from .env.local)
|
|
|
|
## Fixed Answer (verbatim, exact)
|
|
|
|
> We have now quantified the full financial impact of replacing the two senior engineers and the delivery delay at about £600,000 as a one-off relocation cost. Staying put costs us an extra £2 million every year. The remaining issue is our largest client: we do not yet know whether they would leave if we relocated, and losing them would cost us about £5 million per year.
|
|
|
|
## Execution
|
|
|
|
Exactly one update call through the production route via `reproduce-multi-turn-investigation.mjs` in `updateOnly` mode.
|
|
|
|
## Call Accounting
|
|
|
|
```
|
|
startCalls: 0
|
|
updateCalls: 1
|
|
totalCalls: 1
|
|
Retries: 0
|
|
```
|
|
|
|
## HTTP Response
|
|
|
|
- **Status:** 500 — rejected during validation
|
|
- **Stage:** result_validation
|
|
- **Proposal applied:** NO (rejected)
|
|
|
|
## Rejection Error
|
|
|
|
```
|
|
Active unknown violates reasoning pattern consistency: "n_client_retention_uncertainty" is diagnosis but active pattern is decision
|
|
```
|
|
|
|
The model attempted to create a node with id `n_client_retention_uncertainty` and kind `"diagnosis"`. The active reasoning pattern is `"decision"`, which does not allow the `"diagnosis"` kind for unknown nodes. This is a structural/pattern consistency validation failure — not a materiality or targeting question.
|
|
|
|
Note: 60B.6 used node id `n_client_retention` with kind `"unknown"`. The model in this run produced `n_client_retention_uncertainty` with kind `"diagnosis"` — different ID and different kind, which triggered the validator rejection before any proposal could be applied.
|
|
|
|
## Structural Action Required
|
|
|
|
UNAVAILABLE (rejection occurred before structural data was exposed)
|
|
|
|
## Assessment
|
|
|
|
### Materiality behaviour: UNAVAILABLE
|
|
Cannot assess — no proposal applied.
|
|
|
|
### Client-retention uncertainty: UNAVAILABLE
|
|
Cannot assess — model produced `n_client_retention_uncertainty` (kind=diagnosis) rather than a compatible unknown node.
|
|
|
|
### Client-risk ownership: UNAVAILABLE
|
|
Cannot assess.
|
|
|
|
### Preferred-target behaviour: UNAVAILABLE
|
|
Cannot assess — rejected before proposal application.
|
|
|
|
### Question text: NONE
|
|
No question returned.
|
|
|
|
### Prerequisite guard: UNAVAILABLE
|
|
Cannot assess — prerequisite checking occurs after proposal validation.
|
|
|
|
## 60B.6 vs 60B.12 Comparison
|
|
|
|
| Field | 60B.6 | 60B.12 |
|
|
|-------|-------|--------|
|
|
| Final nodeId | (created n_client_retention, but generic question) | UNAVAILABLE — rejected |
|
|
| Decision status | unresolved (PRESERVED) | UNAVAILABLE |
|
|
| Client-retention unknown | YES (n_client_retention) | UNAVAILABLE |
|
|
|
|
In 60B.6 the model produced `kind=unknown` with id `n_client_retention`. In 60B.12 the model produced `kind=diagnosis` with id `n_client_retention_uncertainty` — a different node name and an incompatible kind for the active decision pattern.
|
|
|
|
## Classification: H — BLOCKED
|
|
|
|
Apparatus (reasoning-pattern consistency validator) rejected the model's proposal before inference could be assessed. The blocker is not the 60B.11 targeting fix but a schema-level incompatibility between what the model produced (`kind=diagnosis`) and what the active pattern permits.
|
|
|
|
## Critical evidence
|
|
|
|
- No production code changed during this experiment
|
|
- No prompt changes to question-targeting logic — this failure is at the pattern-consistency layer
|
|
- The node id mismatch (60B.6 used `n_client_retention`; 60B.12 model produced `n_client_retention_uncertainty`) suggests stochastic variation in model output naming
|
|
- The kind mismatch (`unknown` vs `diagnosis`) is the actual validation blocker
|
|
|
|
## What this establishes
|
|
|
|
1. The live server was reachable and the update-only harness executed correctly.
|
|
2. The reasoning-pattern consistency validator catches kind mismatches between model output and active pattern before any proposal mutation.
|
|
3. Further live testing requires either (a) matching what 60B.6 did — producing `kind=unknown` with a compatible id — or (b) relaxing the active pattern to accept `diagnosis` nodes.
|
|
|
|
## Production code changed: NO
|
|
## Prompt changed during experiment: NO
|
|
## Validator changed: NO
|
|
## Schema changed: NO
|
|
## Harness changed: NO
|
|
## Vitest run: NO
|
|
## Ollama calls: 1
|
|
## Direct API calls: 0
|
|
## Dev server disturbed: NO
|