confidence-engine/docs/experiment-60b7.md

# Experiment 60B.7 — Why does selectedQuestion fail to target the specific material unknown just created?

**Branch:** `feature/decision-sufficiency-v0.26`  
**Date:** 2026-08-13  
**Status:** Complete (diagnosis only)  
**Type:** READ-ONLY DIAGNOSIS — Architecture-level tracing of selectedQuestion lifecycle.

## Objective

Explain why experiment 60B.6 produced:

```
addedNodes: n_client_retention (correct material unknown)
selectedQuestion.nodeId: n_relocation_decision (generic parent decision)
selectedQuestion text: "What outcome would demonstrate enough value to justify continuing?" (generic template question)
```

The engine identifies the correct material unknown structurally but fails to target it interrogatively.

## Following

Experiment 60B.4 (materiality rule added to prompt)  
Experiment 60B.5 (decision closes without material uncertainty)  
Experiment 60B.6 (decision stays open for £5M client risk, but question targets parent decision generically)

## Method

Code tracing only. No live calls. No code changes. Read-only inspection of:
- `lib/graph/prompt-builder.js` — prompt rules for selectedQuestion
- `lib/graph/apply-proposal.js` — validation and deterministic selection
- `lib/graph/orchestrator.js` — updateCase flow ordering
- `lib/graph/utils.js` — scoreUnknownCandidate and selectActiveUnknownCandidate
- `lib/graph/question-formulator.js` — template-based question generation

## Checkpoint 1 — selectedQuestion ownership

### Who creates the final selectedQuestion?

**HYBRID in proposal, ENTIRELY DETERMINISTIC in output.**

The model produces `selectedQuestion: {nodeId, question, reason}` inside its proposal JSON. However:

```
In apply-proposal.js line ~3734-3861 (applyValidatedProposal):

const effectiveSelectedQuestion =
    deterministicSelection?.status === "selected"
      ? {
          nodeId: deterministicSelection.nodeId,              // <-- deterministic
          question: effectiveFormulatedQuestion?.question ||   // <-- deterministic
                    deterministicSelection.question,
          ...all other fields from formulatedQuestion           // <-- deterministic
        }
      : null;
```

The final `effectiveSelectedQuestion` that gets returned to the orchestrator is **100% deterministic**. Both nodeId and question text come from the deterministic pipeline:

1. `selectActiveUnknownCandidate(graph, resolvedNodeIds)` — scores ALL unresolved unknowns by text-pattern matching and structural metrics (downstream count, unresolved parent dependencies)
2. The highest-scoring node becomes `deterministicSelection.nodeId`
3. `formulateQuestion({ node, graph, context })` — generates question text from deterministic templates (`buildQuestionFromFamily`, `buildFoundationalDirectQuestion`, etc.)

### Is selectedQuestion.nodeId model-generated?

**NO.** The model's `selectedQuestion.nodeId` is only structurally validated (line 3318):
- Does the node exist? (in graph OR addedNodes)
- Is it an unknown kind?
- Is it unresolved?
- Is the question non-compound?

It is NOT used as a priority signal. It does not boost score. It does not bias selection. It does not appear in `deterministicSelection`.

### Is selectedQuestion text model-generated?

**NO.** The model's question text is completely discarded at line 3746 / 3836:
```js
question: effectiveFormulatedQuestion?.question || deterministicSelection.question
```

The text comes from `buildQuestionFromFamily` or `buildDeterministicQuestionForUnknown` — deterministic template functions that match keywords in the selected node's label/description and produce one of ~20 predefined question templates.

## Checkpoint 2 — timing

### The exact ordering in apply-proposal.js:

```
Line ~3318     validateSelectedQuestion(situationGraph, validatedProposal)
                [validates model's nodeId against graph + addedNodes]

Line ~3358     applyGraphUpdate(graphSnapshot, proposalSnapshot)
                [mutation applied — new nodes NOW in graph]

Line ~3439     deterministicSelection = selectActiveUnknownCandidate(
                 updatedSituationGraph, resolvedNodeIds)
                [scores ALL unresolved unknowns including newly added ones]

Line ~3715     formulatedQuestion = formulateQuestion({ node, graph, context })
                [deterministic question text from templates]

Line ~3734     effectiveSelectedQuestion built from deterministicSelection + formulatedQuestion
                [final output — 100% deterministic]
```

### Can selectedQuestion target a node created in the same proposal?

**YES.** `validateSelectedQuestion` at line 218 uses:
```js
const nodeById = buildNodeById(graph, proposal.addedNodes);
```
This includes newly-added nodes. And `selectActiveUnknownCandidate` scores against `updatedSituationGraph` which already contains the mutations (line 3439).

### Is the newly-added unknown available before selectedQuestion is finalized?

**YES.** By line 3439, the mutation has been applied and the new unknown is in the graph. It is scored alongside all existing unresolved unknowns.

## Checkpoint 3 — prompt contract

### Rules relevant to selectedQuestion (from prompt-builder.js):

**Rule 16:** *"When your proposal adds one or more new unresolved unknowns, you MUST include a selectedQuestion identifying one of those as a candidate unknown node."*

→ "one of those" = ANY one. Not the most material one. Not the most consequential one. Just any valid unresolved unknown from addedNodes.

**Rule 17:** *"selectedQuestion.nodeId must reference an unresolved unknown node that exists either already in the graph or in addedNodes."*

→ Pure structural constraint. No semantic prioritization required.

**Rule 18:** *"selectedQuestion.question must be one narrow non-compound question about that one unknown."*

→ The model's text is validated structurally but discarded at output (see Checkpoint 1).

**Rule 20:** *"Return selectedQuestion as null only when no consequential unresolved unknown remains."*

→ Does not require selecting the MOST material factor. Only requires not returning null if any consequential unresolved unknown exists.

**Rule 22 + Additional Guidance line 172:** *"When selectedQuestion is provided, your role ends at supplying one valid unresolved unknown node from the graph or addedNodes — the engine retains deterministic final-priority selection and may choose a different question if multiple candidates exist."*

→ **Explicitly acknowledges** that the model's choice does not determine the final selection. The engine has full override authority.

### Does materiality rule connect to selectedQuestion targeting?

**NO.** The materiality rule (lines 137-143) says:

> *"Keep a decision context unresolved only when you can identify a specific unresolved factor that could materially change which option is preferred. If the currently supported evidence is sufficient to distinguish the options and no such material unresolved factor remains, resolve the existing decision context and do not ask a generic continuation question."*

This governs **whether** to keep open. It does NOT say: *"When you keep open for a specific material factor, your selectedQuestion must target that factor."* There is no rule that bridges materiality recognition → question targeting.

## Checkpoint 4 — deterministic selection / validation

### Does scoreUnknownCandidate prioritize newly-created unknowns?

**NO.** The scoring function (utils.js line 332) uses:
- `downstreamCount × 4` — how many other nodes depend on this one
- Text pattern matches from `classifyUnknownPriority`:
  - objective (+12), criteria (+11), actor (+10), constraint (+9), measure (+8), terminology (+7)
  - pricing penalty (-8), implementation penalty (-10), optimisation penalty, speculative penalty
- **Zero** recency or "newly-created" bonus

### Does scoring prioritize the material unknown that justified continuation?

**NOT BY DESIGN.** Scoring only looks at text keywords and structural position. A newly created unknown like `n_client_retention` scores based on keyword density in its label+description. The pre-existing parent decision node (`n_relocation_decision`) may have accumulated more matching text through its label ("Which option leaves us better off overall?") and description context.

There is no "materiality" concept computed or passed to the scorer.

### Does validation reject a generic question when a specific unresolved node exists?

**NO.** `validateSelectedQuestion` (line 215) checks:
- nodeId exists ✓
- nodeId is unknown kind ✓
- nodeId is unresolved ✓
- Question is non-compound ✓

It does NOT check:
- Whether the selected node is the most consequential
- Whether a more specific factor exists
- Whether the question is generic vs targeted

### Does `validateQuestionSelectionRequirement` enforce specificity?

**NO.** (line 283) Only checks: if consequential unresolved unknowns exist, selectedQuestion must not be null. It does NOT check that the selected node matches the most material factor.

## Checkpoint 5 — reconstruct 60B.6

### How could n_client_retention coexist with a generic parent-decision question?

The flow in 60B.6:

1. **Model proposes:**
   - `addedNodes: [n_client_retention]` — correct material unknown
   - `selectedQuestion.nodeId: n_relocation_decision` — parent decision (valid but not optimal)
   - `selectedQuestion.question: "What outcome would demonstrate enough value to justify continuing?"`

2. **Validation** passes because n_relocation_decision is an existing unresolved unknown.

3. **Mutation applied** — n_client_retention now exists in the graph.

4. **Deterministic selection** scores ALL unresolved unknowns (n_relocation_decision + n_client_retention):
   - Both are candidates
   - Scoring based on text pattern matches and downstream count
   - Whichever scored higher was selected by the deterministic pipeline
   - The model's nodeId had no influence on score

5. **Question text** generated from template for the deterministically-selected node:
   - For n_relocation_decision, the label "Which option leaves us better off overall?" triggers one of the decision-foundation templates
   - Result: "What outcome would demonstrate enough value to justify continuing?" — a generic template match

### Contract-valid?

**YES.** Every contract rule was satisfied:
- addedNodes: valid new unknown with description ✓
- selectedQuestion.nodeId referenced an existing unresolved unknown ✓
- Question text is narrow and non-compound ✓
- Decision correctly kept open (materiality rule) ✓
- No duplicate unknowns ✓
- Added edge connecting n_client_retention to opt_relocate ✓

### Semantically aligned with materiality rule?

**PARTIAL.** The engine recognized the material factor structurally (created the node, connected it, kept the decision open). But the follow-up question did not target the material factor — it targeted the parent decision generically. The spirit of "do not ask a generic continuation question" (rule 143) was violated in practice even though no explicit contract rule forbids this combination.

### What exact rule was missing?

**No rule connects materiality recognition to question targeting.** The prompt says:
- Rule 16: include selectedQuestion identifying "one of those" unknowns as a candidate ✓
- Rule 172: engine retains deterministic final-priority selection ✓

But neither rule says: when the decision is kept open for a specific material factor, the follow-up question must target that factor. The system treats all unresolved unknowns as equally valid question targets, and deterministic scoring has no awareness of which factor justified continuation.

## Classification: E — MULTIPLE FACTORS

### Combination identified:

**A + B + D**

- **A (Prompt Alignment Gap):** Rule 16 says "identify one of those" — not the material one. Rule 172 explicitly acknowledges model's choice is advisory, not binding. No rule requires the follow-up to target the material factor that justified continuation.

- **B (Selection Priority Gap):** `scoreUnknownCandidate` has no recency bonus and no materiality awareness. It scores all unresolved unknowns purely by text keywords and structural position. Newly-created consequential unknowns get zero priority boost.

- **D (Validation Gap):** `validateSelectedQuestion` accepts any structurally valid unresolved unknown. There is no check that the selected node matches the most consequential unresolved factor. Generic questions are not rejected when specific unresolved nodes exist.

## Minimum Missing Distinction: B — CONTINUATION-REASON → QUESTION-TARGET RULE

**Why:** Adding a rule equivalent to:
> "When a decision remains unresolved because of a specific material factor, selectedQuestion must target that factor rather than the parent decision generically."

This is the narrowest change that closes the gap. Options C (new-unknown priority) and D (validation rejection of generic questions) are related but either too broad or too late in the pipeline. Option B addresses the root cause: no semantic bridge exists between "why we're staying open" and "what we should ask next."

## Implementation Readiness: A — READY FOR BOUNDED IMPLEMENTATION

**Smallest implementation boundary:**
1. Add one rule to prompt-builder.js saying that when a decision remains open for a specific material factor, the selectedQuestion node must be constrained to that factor or its direct children.
2. Optionally add `validateQuestionSelectionAlignment` in apply-proposal.js that checks whether the deterministically-selected node matches the materiality reason — as an advisory diagnostic (not rejection).

This requires prompt-only changes plus optional lightweight validation. No schema changes, no new scoring dimensions, no architecture overhaul.

## Documentation

- Created: docs/experiment-60b7.md
- Appended to: docs/current-handoff.md
- Commit message: experiment: diagnose material-factor question targeting

## Git status