confidence-engine/docs/experiment-57j58.md

Experiment 57J.58 — Selected-Question Ownership Diagnosis (Read-Only Deterministic)

Branch: feature/uncertainty-identity-v0.21 Starting HEAD: f25b1f5 (experiment: validate equivalent uncertainty reuse live) Experiment commit: pending

Objective

Answer exactly:

Why did 57J.57 reject a proposal that correctly introduced a dedicated savings-realism unknown because selectedQuestion was missing, and which component currently owns responsibility for supplying that next question?

---

Part 1 — Exact Rejection Trace

Function: validateQuestionSelectionRequirement(graph, proposal) at lib/graph/apply-proposal.js:283

Exact condition:

const addedConsequentialUnknowns = proposal.addedNodes.filter(
  (node) => node.kind === "unknown" && node.status !== "resolved",
);

if (
  proposal.selectedQuestion == null &&
  addedConsequentialUnknowns.length > 0
) {
  return [
    "selectedQuestion is required when consequential unresolved unknowns remain after resolving the answered unknown",
  ];
}

Inputs used by the condition (from 57J.57's parsed proposal):

• proposal.selectedQuestion → null (absent; Zod default from .nullable().default(null) on schema.js:191)
• proposal.addedNodes → [ { id: "nsavings_reality", kind: "unknown", status: <valid non-resolved enum>, confidence: <valid enum>, parentId, dependsOn, affects, childIds } ]. The status field was required by Zod (situationNodeSchema line 60: status: z.enum(Object.values(SituationStatus))). The diagnostic snapshot omits it for brevity but it must exist because Zod parsing succeeded at the proposal_compatibility stage.
• Filter result → [nsavings_reality] because kind === "unknown" and status !== "resolved"

Why the condition evaluates true:

1. proposal.selectedQuestion == null is true — the model did not include a selectedQuestion in its JSON output. Zod defaults absent to null.
2. addedConsequentialUnknowns.length > 0 is true — one new node with kind: "unknown" and a non-resolved status exists in addedNodes.

Dependencies:

• Does requirement depend on updatedNodes: NO — the function never inspects updatedNodes.
• Does requirement depend on resolvedUnknownNodeIds: NO — the function never inspects this field.
• Does requirement depend on addedNodes: YES — this is the sole input to the condition.
• Does requirement depend on remaining unresolved unknowns (existing graph): NO — the function does not consult graph.nodes. It only looks at what the model added in addedNodes.
• Does requirement depend on activeUnknown: NO.
• Does requirement depend on answerMeaning: NO.

Critical finding: The error message says "after resolving the answered unknown" but the actual condition does NOT check resolvedUnknownNodeIds, does NOT check whether any node was resolved, and does NOT count existing unresolved unknowns. It fires whenever ANY new unresolved unknown appears in addedNodes, regardless of whether an existing node was resolved or even whether the model resolved anything at all. The message is operationally misleading.

---

Part 2 — Selected-Question Owner

Current owner: MODEL-PROVIDED (with engine validation/override)

Evidence trace:

1. The model must include selectedQuestion in its JSON proposal per prompt rules #16 and #20.
2. Zod defaulting (selectedQuestionSchema.nullable().default(null) at schema.js:191) means absent → null.
3. validateQuestionSelectionRequirement catches absence when addedNodes contains unresolved unknowns (57J.57's trigger).
4. After validation, in applyValidatedProposal (apply-proposal.js:3418–3420): the engine uses validatedProposal.selectedQuestion.nodeId as the active unknown if present.
5. If no valid selectedQuestion survives (lines 3432–3436), the engine falls back to deterministic selectActiveUnknownCandidate().

This is constrained MODEL-PROVIDED: the model must produce a candidate; the engine validates it and may override via deterministic scoring when the model's candidate is invalid or absent.

---

Part 3 — Ordering Problem

Current order of operations:

1. Zod schema parse (proposal_validation stage)
2. reconcileResolutionSemantics (synthetic updates for resolved nodes)
3. validateAddedUnknowns (duplicate detection, count ≤ 3)
4. validateSelectedQuestionBelongsToChild (structural check)
5. validateSelectedQuestion (if present: node existence, unknown kind, unresolved status, compound check, scoring)
6. validateAnswerMeaningCompatibilityWithRawAnswer
7. validateAnswerMeaningAlignment
8. validateQuestionSelectionRequirement ← 57J.57 triggered here
9. If all pass → applyGraphUpdate (mutation)
10. selectActiveUnknownCandidate (deterministic engine selection)

Can the system deterministically know which unknown should be asked next before mutation: YES

Why: At step 8, the validator already sees addedNodes from the proposal and all existing graph nodes from situationGraph. The scoring function (scoreUnknownCandidate, called in line 270 of validateSelectedQuestion) can evaluate information value for all candidate unknowns without mutation. However, there is a timing tension: the validator requires the model to provide selectedQuestion before mutation occurs, but at that point some nodes may not yet be integrated into the graph (addedNodes exists as a separate array). The engine handles this by checking both graph.nodes and proposal.addedNodes in buildNodeById (line 218).

---

Part 4 — Prompt Contract

Operational completeness: PARTIAL

What it tells the model:

• Rule #16: "If consequential unresolved unknowns exist, selectedQuestion may identify one valid candidate unknown, but the engine will deterministically choose final priority after validation."
• Rule #20: "Return selectedQuestion as null only when no consequential unresolved unknown remains."
• Rule #17: "selectedQuestion.nodeId must reference an unresolved unknown node that exists either already in the graph or in addedNodes."
• Rule #18: "selectedQuestion.question must be one narrow non-compound question about that one unknown."
• Required shape (line 94–95): "selectedQuestion: either null or an object using these exact keys: nodeId, question, reason"

What it does NOT tell the model:

• The word "may" in rule #16 semantically means optionality. This directly conflicts with rule #20's mandatory framing (null is only acceptable when nothing remains unresolved). When the model adds a new unknown (not resolving an existing one), there is no positive instruction stating "you MUST include selectedQuestion."
• Rule #6 requires structural mutation for consequential uncertainty but does not explicitly connect this to selectedQuestion obligation.
• No explicit mapping from condition "I added an unresolved unknown" → "therefore selectedQuestion is mandatory."

---

Part 5 — Controlled Cases

Case A — update resolves current unknown, other unresolved unknowns remain

SelectedQuestion required: DEPENDS Why: Only if the update ALSO adds new unknown nodes. If only existing nodes are updated/resolved without adding new unknowns, validateQuestionSelectionRequirement never fires (it only checks addedNodes). Other validators may still require it depending on downstream flow. Matches current behaviour: YES — this validator only checks addedNodes, not existing graph state.

Case B — update introduces a new unresolved unknown and resolves nothing

SelectedQuestion required: YES Why: Any new unresolved unknown triggers the requirement unconditionally. Correct behavior: without a selected question, there's no way to determine what to ask next. Matches current behaviour: YES — this is exactly what happened in 57J.57.

Case C — evidence/state change, unresolved set unchanged

SelectedQuestion required: DEPENDS Why: This validator does NOT fire (no new unknown nodes). The question requirement here comes from other parts of the pipeline (e.g., validateAnswerMeaningAlignment or downstream engine logic) if the active unknown changed. Matches current behaviour: YES — this validator stays silent; other mechanisms handle it.

Case D — proposal leaves no consequential unresolved unknowns

SelectedQuestion required: NO Why: Either no unknowns exist (investigation complete) or selectedQuestion was null by rule #20 and no new unknowns were added. Matches current behaviour: YES.

---

Part 6 — Architecture Ownership Classification

Evaluation of four explanations:

A — MODEL OMISSION The prompt has rules addressing selectedQuestion but uses contradictory language ("may" vs "only when null"). The model correctly understood the semantics (created the savings-realism node) but omitted the field because the prompt made it appear optional via rule #16.

B — PROMPT CONTRACT GAP ✅ BEST FIT Rule #16's "may identify" is semantically permissive, while rule #20 only defines when null is acceptable (via negation). No positive statement says "you MUST include selectedQuestion whenever you add an unresolved unknown." The contradiction between these two rules creates genuine ambiguity about obligation.

C — VALIDATION ORDER GAP The validator fires before mutation but correctly sees addedNodes. This is NOT the primary problem — the validator has sufficient information. The deeper timing tension (requiring pre-mutation question when engine can only determine post-mutation) exists but is secondary to the prompt ambiguity.

D — RESPONSIBILITY SPLIT GAP The model provides a candidate; the engine validates and may override. Rule #16's "engine will deterministically choose final priority" could make the model defer selection entirely. This split contributes to confusion but originates from the prompt's ambiguous language.

Best classification: B — PROMPT CONTRACT GAP

---

Part 7 — Smallest Next Boundary

Selected: B — prompt-only clarification

The smallest change is to clarify rule #16:

• Change "may identify" to mandatory language ("MUST include a candidate selectedQuestion identifying one unresolved unknown").
• Clarify the trigger condition: "When you add any new unresolved unknown (status !== 'resolved'), you must provide selectedQuestion even if you did not resolve any existing node."

This does NOT require validator changes, scoring changes, or question-selection ownership transfer. It only removes the semantic ambiguity that made selectedQuestion appear optional in rule #16.