From 32e668969ebf47f2c8f1f9f8c34c445805219998 Mon Sep 17 00:00:00 2001 From: robbond Date: Thu, 13 Aug 2026 09:54:42 +0100 Subject: [PATCH] experiment: diagnose decision-pattern kind mismatch --- docs/current-handoff.md | 4 + docs/experiment-60b13.md | 171 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 175 insertions(+) create mode 100644 docs/experiment-60b13.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 7d0c70b..184da3b 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -2817,3 +2817,7 @@ Experiment 60B.8 compared three minimal mechanisms for preserving a material fac --- Experiment 60B.12 reran the exact 60B.6 live case to verify prerequisite-aware targeting from 60B.11. **Classification: H — BLOCKED.** The update was rejected at stage `result_validation` with error: "Active unknown violates reasoning pattern consistency: 'n_client_retention_uncertainty' is diagnosis but active pattern is decision." Three differences from 60B.6: (1) node id changed from `n_client_retention` to `n_client_retention_uncertainty`, (2) node kind changed from `unknown` to `diagnosis`, (3) rejection occurred before any proposal applied — neither materiality nor targeting could be assessed. The blocker is a schema-level pattern-consistency mismatch between the model's output and the active decision pattern, not a question-targeting problem. Fix path: either constrain the model prompt to produce kind=unknown nodes or relax the active pattern validator to accept diagnosis nodes for decision-mode continuations. + +--- + +Experiment 60B.13 performed read-only architectural diagnosis of the 60B.12 kind mismatch. **Classification: A + D — Prompt kind ambiguity + missing normalisation path.** Analysis confirmed: (1) `kind=diagnosis` is NOT a valid SituationKind — it exists only as a reasoning pattern and as the default fallback in `selectReasoningPattern`; (2) the validator at line 3927/3998 of apply-proposal.js correctly rejects diagnosis under active decision pattern — semantically, diagnosis and decision are distinct reasoning types; (3) the model likely produced kind=unknown with diagnostic-inferred text analysis, not kind=diagnosis directly (which would fail zod immediately); (4) `hasDecisionContext`'s keyword list (`whether to|build|launch|continue|proceed|invest|commercially justified|viability`) does not include "relocate"/"relocation", so material factors about relocation decisions get inferred as diagnosis; (5) the prompt's kind rules cover decision questions and candidate options but have no rule for material unresolved factors within a decision. Minimum corrective boundary: B — one clarifying rule in Prompt Proposal Rules section stating that new material factors affecting a decision outcome use kind=unknown, with reasoning pattern determined by graph context. Implementation readiness: A. diff --git a/docs/experiment-60b13.md b/docs/experiment-60b13.md new file mode 100644 index 0000000..f6a395d --- /dev/null +++ b/docs/experiment-60b13.md @@ -0,0 +1,171 @@ +# Experiment 60B.13 — Why the Model Classified a Decision-Relevant Factor as `diagnosis` + +**Branch:** `feature/question-target-alignment-v0.27` +**Starting HEAD:** `3a4dda9` +**Date:** 2026-08-13 +**Status:** COMPLETE (read-only diagnosis) +**Type:** ARCHITECTURAL DIAGNOSIS — Read-only investigation of kind mismatch blocker from 60B.12 + +## Objective + +Answer one question: + +> What current prompt/schema/pattern-classification rule caused or allowed a decision-relevant client-retention uncertainty to be emitted as `diagnosis`, and what is the smallest correct architectural boundary for preventing that mismatch? + +## Findings by Checkpoint + +### PATTERN OWNERSHIP + +**Active pattern source:** DETERMINISTIC CODE + +The active reasoning pattern "decision" comes from `selectReasoningPattern` in `lib/graph/question-formulator.js` (line 1030), which is computed deterministically from graph state via `hasDecisionContext`, `isDefinitionPatternCandidate`, etc. It is **not** model-chosen, not persisted on the graph, and not hybrid — it is recomputed fresh each update cycle from current graph topology and text analysis. + +**Persisted on graph:** NO + +No field in the SituationGraph schema stores an active reasoning pattern as a persistent value. The pattern is derived on-demand from `selectReasoningPattern` or inherited via `determineActiveReasoningPattern` (lines 1804–1832 of apply-proposal.js). + +**Model may change pattern mid-update:** CONDITIONAL + +The model cannot directly set the active pattern. However, if the model's proposal materially changes graph state (e.g., adds new nodes that alter `hasDecisionContext` for subsequent unknowns), `determineActiveReasoningPattern` will recompute during decomposition. This is indirect: the pattern follows from graph state, not from model intent. + +### DIAGNOSIS SEMANTICS + +**Architectural meaning of kind=diagnosis:** + +`kind=diagnosis` does **not exist** in the SituationKind schema enum (`lib/graph/schema.js` line 11–22). Valid kinds are: `observation`, `reported_claim`, `metric`, `state`, `transition`, `relationship`, `assumption`, `unknown`, `conclusion`, `option`. + +The term "diagnosis" exists **only as a reasoning pattern** in `ALL_REASONING_PATTERNS` (question-formulator.js line 849) and as the **default/fallback** pattern in `selectReasoningPattern` (line 1095–1099): + +> "Selected diagnosis as the default because the active unknown needs clarifying evidence or mechanism-level investigation." + +When the model emits `kind="diagnosis"`, it produces an invalid kind that would fail zod schema validation — **but** if the proposal's selected question references a newly added unknown with a compatible kind (e.g., kind=unknown), the pattern compatibility check at line 3927 of apply-proposal.js runs before zod and may reject the proposal first. + +**Valid only under diagnosis pattern:** CONDITIONAL + +Since kind="diagnosis" is not a valid kind, this question is partially unanswerable as stated. However, nodes whose *text* triggers `inferIntrinsicNodePattern` to return "diagnosis" would need an active pattern of "diagnosis" or its allowed set ["diagnosis", "comparison", "definition"] to be compatible. + +**Can coexist inside decision pattern:** NO + +Under active pattern "decision", only node patterns "decision" and "definition" are allowed (ALLOWED_NODE_PATTERNS_BY_ACTIVE_PATTERN, apply-proposal.js line 1795). Any node whose inferred pattern is "diagnosis" will be rejected. + +**Causal uncertainty alone implies diagnosis:** NO + +The architecture clearly separates the node's kind from its reasoning pattern. A causal uncertainty within a decision should use kind=unknown with reasoning pattern=decision — this is exactly what the schema and validation expect. + +### DECISION UNKNOWN SEMANTICS + +**Correct kind for material unresolved decision factor:** unknown + +By definition, an unresolved factor has unknown truth value or unknown impact. Encoding it as anything other than kind=unknown creates a semantic contradiction (e.g., an "assumption" implies a stated belief, not genuine uncertainty). The reasoning pattern determines the investigation type; the kind captures the nature of the node's content. + +**60B.12 client-retention factor:** UNKNOWN + +The semantically correct encoding is: +- **kind**: unknown (material unresolved fact) +- **reasoning pattern**: decision (it affects option comparison) +- **label/description**: should contain decision keywords or be connected to a decision-context node for `hasDecisionContext` to detect + +**Why the model failed:** + +The model correctly identified the concept (client retention matters £5M). It created a node whose text did not trigger any of `hasDecisionContext`'s keyword list (`whether to|build|launch|continue|proceed|invest|commercially justified|commercial justification|commercial value|business case|viability`) because "relocate"/"relocation"/"leaving"/"better off" are not in that set. With no keyword match, `selectReasoningPattern` returned its default: "diagnosis". + +### PROMPT ANALYSIS + +**Decision uncertainty vs diagnostic explanation clearly distinguished:** PARTIAL + +The prompt lists valid kinds (line 55-56 of prompt-builder.js) and explicitly excludes "diagnosis" as a kind. However, the kind guidance section is narrow: +- Line 148: "Create exactly one node of kind 'unknown' to carry the **decision question**" +- Line 150: "For each candidate path, create exactly one node of kind 'option'" + +Neither rule covers the case of a material *causal factor* within an existing decision. The distinction between "uncertain factor in a decision" and "diagnostic explanation of an observed problem" is not explicitly stated. + +**Decision-pattern material risks explicitly stay unknown:** NO + +There is no rule stating: "When adding a new unresolved factor that may affect the outcome of an ongoing decision, use kind=unknown." The closest guidance (Rule 7/Rule 9) says to add unknowns for "genuinely new" concepts relevant to the case — but it doesn't specify what kind they should be. + +**Prompt may pull causal uncertainty toward diagnosis:** PARTIAL + +The prompt does not explicitly mention "diagnosis" as a prohibited kind, only listing allowed kinds. A model interpreting a material factor like client-retention (a causal downside) might infer that since the concept describes a diagnostic inquiry ("will this happen to us?"), it should use a diagnostic-semantic kind — even though no valid kind supports that intent. + +### VALIDATOR ANALYSIS + +**Rejects diagnosis under active decision:** YES + +The validator correctly rejects inferred pattern "diagnosis" when active pattern is "decision". This is `ALLOWED_NODE_PATTERNS_BY_ACTIVE_PATTERN["decision"] = ["decision", "definition"]` — line 1795. + +**Semantically correct to reject:** YES + +Diagnosis is architecturally distinct from decision reasoning. The architecture's design separates kind from pattern precisely because the same structural type (unknown) can serve different investigation modes. Allowing diagnosis inside decision would conflate two distinct reasoning types. + +**Repair/coercion path exists:** NO + +The validator performs only rejection — no normalization, no repair, no second-chance. The whole proposal is discarded. There is no mechanism to check if a "diagnosis" node is actually semantically compatible (e.g., kind=unknown with diagnostic-inferred pattern that would be decision-compatible) and normalize it. + +**Harmless drift distinguished from real pattern transition:** NO + +The validator has no capability to determine whether the model's inferred pattern represents genuine semantic mismatch or merely a harmless kind drift. It treats all mismatches equally. + +**Whole proposal discarded:** YES + +Rejection at result_validation (line 3977-4002) discards the entire proposal — no partial application, no node-level rejection, no selective repair. + +### 60B.12 RECONSTRUCTION + +**How the model could emit diagnosis for client-retention uncertainty:** + +1. Model receives user answer about £5M client-retention risk +2. Model correctly identifies this as a material unresolved factor for the relocation decision +3. Model creates node `n_client_retention_uncertainty` with kind=unknown (valid) +4. Node label/description describes causal uncertainty about client retention +5. Text analysis runs: no keywords from `hasDecisionContext`'s list match ("whether to", "build", etc.) +6. `selectReasoningPattern` falls through all pattern-specific checks and returns default "diagnosis" +7. Compatibility check: diagnosis not in ["decision", "definition"] → incompatible +8. Validation rejects the entire proposal with "violates reasoning pattern consistency" + +**Classification:** A + D + +### A — PROMPT KIND AMBIGUITY + +The kind guidance rules cover decision questions and candidate options explicitly but do not address material unresolved factors within a decision. The model correctly identifies the uncertainty as needing kind=unknown structurally, but the semantic description of that unknown ("will our largest client leave") triggers diagnostic pattern inference because it doesn't match decision context keywords. The prompt does not prevent this misalignment. + +### D — MISSING COMPATIBILITY / NORMALISATION PATH + +The validator rejects without checking if the mismatch is genuinely semantic (diagnosis really should investigate something) or a harmless drift (model correctly identified an unknown but described it in diagnostic language). A deterministic normalizer could safely map kind=unknown + diagnosed-as-uncertain → kind=unknown with pattern re-inference, rather than rejecting outright. + +### Why not B (Model enum drift despite clear contract)? + +The model didn't produce "diagnosis" as a kind value directly — if it had, zod would have rejected immediately. The model likely produced kind=unknown but the *inferred pattern* was "diagnosis". The issue is not that the model ignored the contract; it's that the contract doesn't address this gap (what kind do I use for a new material factor in an existing decision?). + +### Why not C (Validator too strict)? + +The validator is correct. A diagnosis node inside a decision pattern would conflate two architecturally distinct reasoning types. The separation of kind=unknown from reasoning-pattern=decision vs =diagnosis is a deliberate design choice that the validator faithfully enforces. + +### MINIMUM CORRECTIVE BOUNDARY + +**Choice:** B — PROMPT KIND CLARIFICATION + +**Why:** This addresses the root cause (missing guidance for material unresolved factors) without adding unnecessary complexity. Normalization (option C) would mask the underlying ambiguity rather than prevent it. Prompt clarification is a single addition to the Proposal Rules section of the update prompt, approximately 1-2 sentences. + +The clarifying rule should state: +> "When the answer introduces a new material factor that may affect the outcome of an ongoing decision or investigation, create it as kind='unknown' — not as any other kind. Its reasoning pattern is determined automatically from the graph context; your role is to encode it structurally as unknown and connect it to the relevant parent node." + +### IMPLEMENTATION READINESS + +**A — READY FOR BOUNDED IMPLEMENTATION** + +One unresolved question (if any): +- Does the prompt's existing "Decision Option Structure Rules" section need similar clarification for option-level causal factors? (Answer: No — options are covered by Rule 150.) + +**Smallest implementation boundary:** One new rule (Rule #33 or a numbered insertion) in the Proposal Rules section of `buildGraphUpdatePrompt` in `prompt-builder.js`. + +## Critical Analysis Summary + +The root cause is **not** a validator defect or model stochastic failure. It is a prompt guidance gap: + +1. The SituationKind enum does not include "diagnosis" — it's a reasoning pattern, not a node kind. +2. The prompt lists valid kinds but the kind-specific rules (lines 148-150) only cover decision questions and candidate options. +3. Material unresolved factors that are *causal* to a decision (client retention, regulatory impact, market size) have no explicit kind guidance. +4. When these factors lack decision-context keywords in their label/description, `hasDecisionContext` returns false, causing pattern inference to default to "diagnosis" — which is incompatible with the active decision pattern. +5. The validator correctly rejects this mismatch but without a repair path, causing complete proposal loss. + +The architecture correctly separates kind (what the node is) from reasoning pattern (how to investigate it). The prompt should make this distinction explicit for the model's benefit.