diff --git a/docs/current-handoff.md b/docs/current-handoff.md index c92aef7..7d0c70b 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -2813,3 +2813,7 @@ Experiment 60B.7 diagnosed why selectedQuestion targets a generic parent decisio --- Experiment 60B.8 compared three minimal mechanisms for preserving a material factor into final question selection (A: honour valid model-selected nodeId as preferred target, B: deterministic material-factor priority via scoring adjustment, C: explicit continuationTarget field). Analysis of actual code confirmed: (1) selectedQuestion.nodeId already means "candidate for next question" with full structural validation in place, (2) ignoring a valid model-selected node is semantically contradictory given the field's purpose, (3) existing validation (validateSelectedQuestion + isSelectableUnresolvedUnknown) covers all validity checks, and (4) existing fallback chain at lines 3418-3437 of apply-proposal.js handles the invalid-target case. Candidate A was chosen: honour valid model-selected nodeId as preferred target before deterministic scoring, with fallback to selectActiveImplementationCandidate when invalid. No new schema fields, no new validators, no new keyword logic. Smallest boundary: one Rule 172 clarification + one modification in buildSelectedQuestionResult. Implementation readiness: A — ready for bounded implementation. Status pending Rob's review. + +--- + +Experiment 60B.12 reran the exact 60B.6 live case to verify prerequisite-aware targeting from 60B.11. **Classification: H — BLOCKED.** The update was rejected at stage `result_validation` with error: "Active unknown violates reasoning pattern consistency: 'n_client_retention_uncertainty' is diagnosis but active pattern is decision." Three differences from 60B.6: (1) node id changed from `n_client_retention` to `n_client_retention_uncertainty`, (2) node kind changed from `unknown` to `diagnosis`, (3) rejection occurred before any proposal applied — neither materiality nor targeting could be assessed. The blocker is a schema-level pattern-consistency mismatch between the model's output and the active decision pattern, not a question-targeting problem. Fix path: either constrain the model prompt to produce kind=unknown nodes or relax the active pattern validator to accept diagnosis nodes for decision-mode continuations. diff --git a/docs/experiment-60b12.md b/docs/experiment-60b12.md new file mode 100644 index 0000000..18d4c96 --- /dev/null +++ b/docs/experiment-60b12.md @@ -0,0 +1,136 @@ +# Experiment 60B.12 — Live Verification of Prerequisite-Aware Question Targeting + +**Branch:** `feature/question-target-alignment-v0.27` +**Starting HEAD:** `3c6e436` +**Date:** 2026-08-13 +**Status:** BLOCKED (apparatus failure) +**Type:** LIVE RUN — Single-call verification of 60B.11 prerequisite-aware targeting + +## Objective + +Run one bounded Update to answer: + +> Does the engine now keep the decision open for the client-retention uncertainty AND make that same unknown the final selected question target? + +## Following + +Experiment 60B.6 (materiality rule with real unresolved factor) +Experiment 60B.11 (prerequisite-aware preferred targeting implemented in production code) + +This is the **live regression** 60B.11 explicitly left unproven: + +```text +the exact 60B.6 live continuation case, +where the model selects the newly exposed client-retention factor +and the final selected target preserves that same ready material unknown +``` + +## Fixed Starting Graph + +**Fixture:** `tests/fixtures/pre-anchored-decision-options.json` + +| Node | Kind | Status | Label | +|------|------|--------|-------| +| n_relocation_state | state | provisional | Engineering team relocation consideration | +| opt_relocate | option | known | Relocate to Manchester | +| opt_stay_put | option | known | Stay in London (Status Quo) | +| n_relocation_decision | unknown | unknown | Which option leaves us better off overall? | + +## Configured Model + +- **Model:** qwen-claude:latest +- **Ollama base URL:** http://192.168.1.111:11434 (from .env.local) + +## Fixed Answer (verbatim, exact) + +> We have now quantified the full financial impact of replacing the two senior engineers and the delivery delay at about £600,000 as a one-off relocation cost. Staying put costs us an extra £2 million every year. The remaining issue is our largest client: we do not yet know whether they would leave if we relocated, and losing them would cost us about £5 million per year. + +## Execution + +Exactly one update call through the production route via `reproduce-multi-turn-investigation.mjs` in `updateOnly` mode. + +## Call Accounting + +``` +startCalls: 0 +updateCalls: 1 +totalCalls: 1 +Retries: 0 +``` + +## HTTP Response + +- **Status:** 500 — rejected during validation +- **Stage:** result_validation +- **Proposal applied:** NO (rejected) + +## Rejection Error + +``` +Active unknown violates reasoning pattern consistency: "n_client_retention_uncertainty" is diagnosis but active pattern is decision +``` + +The model attempted to create a node with id `n_client_retention_uncertainty` and kind `"diagnosis"`. The active reasoning pattern is `"decision"`, which does not allow the `"diagnosis"` kind for unknown nodes. This is a structural/pattern consistency validation failure — not a materiality or targeting question. + +Note: 60B.6 used node id `n_client_retention` with kind `"unknown"`. The model in this run produced `n_client_retention_uncertainty` with kind `"diagnosis"` — different ID and different kind, which triggered the validator rejection before any proposal could be applied. + +## Structural Action Required + +UNAVAILABLE (rejection occurred before structural data was exposed) + +## Assessment + +### Materiality behaviour: UNAVAILABLE +Cannot assess — no proposal applied. + +### Client-retention uncertainty: UNAVAILABLE +Cannot assess — model produced `n_client_retention_uncertainty` (kind=diagnosis) rather than a compatible unknown node. + +### Client-risk ownership: UNAVAILABLE +Cannot assess. + +### Preferred-target behaviour: UNAVAILABLE +Cannot assess — rejected before proposal application. + +### Question text: NONE +No question returned. + +### Prerequisite guard: UNAVAILABLE +Cannot assess — prerequisite checking occurs after proposal validation. + +## 60B.6 vs 60B.12 Comparison + +| Field | 60B.6 | 60B.12 | +|-------|-------|--------| +| Final nodeId | (created n_client_retention, but generic question) | UNAVAILABLE — rejected | +| Decision status | unresolved (PRESERVED) | UNAVAILABLE | +| Client-retention unknown | YES (n_client_retention) | UNAVAILABLE | + +In 60B.6 the model produced `kind=unknown` with id `n_client_retention`. In 60B.12 the model produced `kind=diagnosis` with id `n_client_retention_uncertainty` — a different node name and an incompatible kind for the active decision pattern. + +## Classification: H — BLOCKED + +Apparatus (reasoning-pattern consistency validator) rejected the model's proposal before inference could be assessed. The blocker is not the 60B.11 targeting fix but a schema-level incompatibility between what the model produced (`kind=diagnosis`) and what the active pattern permits. + +## Critical evidence + +- No production code changed during this experiment +- No prompt changes to question-targeting logic — this failure is at the pattern-consistency layer +- The node id mismatch (60B.6 used `n_client_retention`; 60B.12 model produced `n_client_retention_uncertainty`) suggests stochastic variation in model output naming +- The kind mismatch (`unknown` vs `diagnosis`) is the actual validation blocker + +## What this establishes + +1. The live server was reachable and the update-only harness executed correctly. +2. The reasoning-pattern consistency validator catches kind mismatches between model output and active pattern before any proposal mutation. +3. Further live testing requires either (a) matching what 60B.6 did — producing `kind=unknown` with a compatible id — or (b) relaxing the active pattern to accept `diagnosis` nodes. + +## Production code changed: NO +## Prompt changed during experiment: NO +## Validator changed: NO +## Schema changed: NO +## Harness changed: NO +## Vitest run: NO +## Ollama calls: 1 +## Direct API calls: 0 +## Dev server disturbed: NO