confidence-engine/docs/experiment-60b13.md

Experiment 60B.13 — Why the Model Classified a Decision-Relevant Factor as diagnosis

Branch: feature/question-target-alignment-v0.27
Starting HEAD: 3a4dda9
Date: 2026-08-13
Status: COMPLETE (read-only diagnosis)
Type: ARCHITECTURAL DIAGNOSIS — Read-only investigation of kind mismatch blocker from 60B.12

Objective

Answer one question:

What current prompt/schema/pattern-classification rule caused or allowed a decision-relevant client-retention uncertainty to be emitted as diagnosis, and what is the smallest correct architectural boundary for preventing that mismatch?

Findings by Checkpoint

PATTERN OWNERSHIP

Active pattern source: DETERMINISTIC CODE

The active reasoning pattern "decision" comes from selectReasoningPattern in lib/graph/question-formulator.js (line 1030), which is computed deterministically from graph state via hasDecisionContext, isDefinitionPatternCandidate, etc. It is not model-chosen, not persisted on the graph, and not hybrid — it is recomputed fresh each update cycle from current graph topology and text analysis.

Persisted on graph: NO

No field in the SituationGraph schema stores an active reasoning pattern as a persistent value. The pattern is derived on-demand from selectReasoningPattern or inherited via determineActiveReasoningPattern (lines 1804–1832 of apply-proposal.js).

Model may change pattern mid-update: CONDITIONAL

The model cannot directly set the active pattern. However, if the model's proposal materially changes graph state (e.g., adds new nodes that alter hasDecisionContext for subsequent unknowns), determineActiveReasoningPattern will recompute during decomposition. This is indirect: the pattern follows from graph state, not from model intent.

DIAGNOSIS SEMANTICS

Architectural meaning of kind=diagnosis:

kind=diagnosis does not exist in the SituationKind schema enum (lib/graph/schema.js line 11–22). Valid kinds are: observation, reported_claim, metric, state, transition, relationship, assumption, unknown, conclusion, option.

The term "diagnosis" exists only as a reasoning pattern in ALL_REASONING_PATTERNS (question-formulator.js line 849) and as the default/fallback pattern in selectReasoningPattern (line 1095–1099):

"Selected diagnosis as the default because the active unknown needs clarifying evidence or mechanism-level investigation."

When the model emits kind="diagnosis", it produces an invalid kind that would fail zod schema validation — but if the proposal's selected question references a newly added unknown with a compatible kind (e.g., kind=unknown), the pattern compatibility check at line 3927 of apply-proposal.js runs before zod and may reject the proposal first.

Valid only under diagnosis pattern: CONDITIONAL

Since kind="diagnosis" is not a valid kind, this question is partially unanswerable as stated. However, nodes whose text triggers inferIntrinsicNodePattern to return "diagnosis" would need an active pattern of "diagnosis" or its allowed set ["diagnosis", "comparison", "definition"] to be compatible.

Can coexist inside decision pattern: NO

Under active pattern "decision", only node patterns "decision" and "definition" are allowed (ALLOWED_NODE_PATTERNS_BY_ACTIVE_PATTERN, apply-proposal.js line 1795). Any node whose inferred pattern is "diagnosis" will be rejected.

Causal uncertainty alone implies diagnosis: NO

The architecture clearly separates the node's kind from its reasoning pattern. A causal uncertainty within a decision should use kind=unknown with reasoning pattern=decision — this is exactly what the schema and validation expect.

DECISION UNKNOWN SEMANTICS

Correct kind for material unresolved decision factor: unknown

By definition, an unresolved factor has unknown truth value or unknown impact. Encoding it as anything other than kind=unknown creates a semantic contradiction (e.g., an "assumption" implies a stated belief, not genuine uncertainty). The reasoning pattern determines the investigation type; the kind captures the nature of the node's content.

60B.12 client-retention factor: UNKNOWN

The semantically correct encoding is:

• kind: unknown (material unresolved fact)
• reasoning pattern: decision (it affects option comparison)
• label/description: should contain decision keywords or be connected to a decision-context node for hasDecisionContext to detect

Why the model failed:

The model correctly identified the concept (client retention matters £5M). It created a node whose text did not trigger any of hasDecisionContext's keyword list (whether to|build|launch|continue|proceed|invest|commercially justified|commercial justification|commercial value|business case|viability) because "relocate"/"relocation"/"leaving"/"better off" are not in that set. With no keyword match, selectReasoningPattern returned its default: "diagnosis".

PROMPT ANALYSIS

Decision uncertainty vs diagnostic explanation clearly distinguished: PARTIAL

The prompt lists valid kinds (line 55-56 of prompt-builder.js) and explicitly excludes "diagnosis" as a kind. However, the kind guidance section is narrow:

• Line 148: "Create exactly one node of kind 'unknown' to carry the decision question"
• Line 150: "For each candidate path, create exactly one node of kind 'option'"

Neither rule covers the case of a material causal factor within an existing decision. The distinction between "uncertain factor in a decision" and "diagnostic explanation of an observed problem" is not explicitly stated.

Decision-pattern material risks explicitly stay unknown: NO

There is no rule stating: "When adding a new unresolved factor that may affect the outcome of an ongoing decision, use kind=unknown." The closest guidance (Rule 7/Rule 9) says to add unknowns for "genuinely new" concepts relevant to the case — but it doesn't specify what kind they should be.

Prompt may pull causal uncertainty toward diagnosis: PARTIAL

The prompt does not explicitly mention "diagnosis" as a prohibited kind, only listing allowed kinds. A model interpreting a material factor like client-retention (a causal downside) might infer that since the concept describes a diagnostic inquiry ("will this happen to us?"), it should use a diagnostic-semantic kind — even though no valid kind supports that intent.

VALIDATOR ANALYSIS

Rejects diagnosis under active decision: YES

The validator correctly rejects inferred pattern "diagnosis" when active pattern is "decision". This is ALLOWED_NODE_PATTERNS_BY_ACTIVE_PATTERN["decision"] = ["decision", "definition"] — line 1795.

Semantically correct to reject: YES

Diagnosis is architecturally distinct from decision reasoning. The architecture's design separates kind from pattern precisely because the same structural type (unknown) can serve different investigation modes. Allowing diagnosis inside decision would conflate two distinct reasoning types.

Repair/coercion path exists: NO

The validator performs only rejection — no normalization, no repair, no second-chance. The whole proposal is discarded. There is no mechanism to check if a "diagnosis" node is actually semantically compatible (e.g., kind=unknown with diagnostic-inferred pattern that would be decision-compatible) and normalize it.

Harmless drift distinguished from real pattern transition: NO

The validator has no capability to determine whether the model's inferred pattern represents genuine semantic mismatch or merely a harmless kind drift. It treats all mismatches equally.

Whole proposal discarded: YES

Rejection at result_validation (line 3977-4002) discards the entire proposal — no partial application, no node-level rejection, no selective repair.

60B.12 RECONSTRUCTION

How the model could emit diagnosis for client-retention uncertainty:

1. Model receives user answer about £5M client-retention risk
2. Model correctly identifies this as a material unresolved factor for the relocation decision
3. Model creates node n_client_retention_uncertainty with kind=unknown (valid)
4. Node label/description describes causal uncertainty about client retention
5. Text analysis runs: no keywords from hasDecisionContext's list match ("whether to", "build", etc.)
6. selectReasoningPattern falls through all pattern-specific checks and returns default "diagnosis"
7. Compatibility check: diagnosis not in ["decision", "definition"] → incompatible
8. Validation rejects the entire proposal with "violates reasoning pattern consistency"

Classification: A + D

A — PROMPT KIND AMBIGUITY

The kind guidance rules cover decision questions and candidate options explicitly but do not address material unresolved factors within a decision. The model correctly identifies the uncertainty as needing kind=unknown structurally, but the semantic description of that unknown ("will our largest client leave") triggers diagnostic pattern inference because it doesn't match decision context keywords. The prompt does not prevent this misalignment.

D — MISSING COMPATIBILITY / NORMALISATION PATH

The validator rejects without checking if the mismatch is genuinely semantic (diagnosis really should investigate something) or a harmless drift (model correctly identified an unknown but described it in diagnostic language). A deterministic normalizer could safely map kind=unknown + diagnosed-as-uncertain → kind=unknown with pattern re-inference, rather than rejecting outright.

Why not B (Model enum drift despite clear contract)?

The model didn't produce "diagnosis" as a kind value directly — if it had, zod would have rejected immediately. The model likely produced kind=unknown but the inferred pattern was "diagnosis". The issue is not that the model ignored the contract; it's that the contract doesn't address this gap (what kind do I use for a new material factor in an existing decision?).

Why not C (Validator too strict)?

The validator is correct. A diagnosis node inside a decision pattern would conflate two architecturally distinct reasoning types. The separation of kind=unknown from reasoning-pattern=decision vs =diagnosis is a deliberate design choice that the validator faithfully enforces.

MINIMUM CORRECTIVE BOUNDARY

Choice: B — PROMPT KIND CLARIFICATION

Why: This addresses the root cause (missing guidance for material unresolved factors) without adding unnecessary complexity. Normalization (option C) would mask the underlying ambiguity rather than prevent it. Prompt clarification is a single addition to the Proposal Rules section of the update prompt, approximately 1-2 sentences.

The clarifying rule should state:

"When the answer introduces a new material factor that may affect the outcome of an ongoing decision or investigation, create it as kind='unknown' — not as any other kind. Its reasoning pattern is determined automatically from the graph context; your role is to encode it structurally as unknown and connect it to the relevant parent node."

IMPLEMENTATION READINESS

A — READY FOR BOUNDED IMPLEMENTATION

One unresolved question (if any):

• Does the prompt's existing "Decision Option Structure Rules" section need similar clarification for option-level causal factors? (Answer: No — options are covered by Rule 150.)

Smallest implementation boundary: One new rule (Rule #33 or a numbered insertion) in the Proposal Rules section of buildGraphUpdatePrompt in prompt-builder.js.

Critical Analysis Summary

The root cause is not a validator defect or model stochastic failure. It is a prompt guidance gap:

1. The SituationKind enum does not include "diagnosis" — it's a reasoning pattern, not a node kind.
2. The prompt lists valid kinds but the kind-specific rules (lines 148-150) only cover decision questions and candidate options.
3. Material unresolved factors that are causal to a decision (client retention, regulatory impact, market size) have no explicit kind guidance.
4. When these factors lack decision-context keywords in their label/description, hasDecisionContext returns false, causing pattern inference to default to "diagnosis" — which is incompatible with the active decision pattern.
5. The validator correctly rejects this mismatch but without a repair path, causing complete proposal loss.

The architecture correctly separates kind (what the node is) from reasoning pattern (how to investigate it). The prompt should make this distinction explicit for the model's benefit.