experiment: diagnose selected-question ownership
This commit is contained in:
@@ -0,0 +1,163 @@
|
||||
# Experiment 57J.58 — Selected-Question Ownership Diagnosis (Read-Only Deterministic)
|
||||
|
||||
**Branch:** `feature/uncertainty-identity-v0.21`
|
||||
**Starting HEAD:** `f25b1f5` (experiment: validate equivalent uncertainty reuse live)
|
||||
**Experiment commit:** pending
|
||||
|
||||
## Objective
|
||||
|
||||
Answer exactly:
|
||||
|
||||
> Why did 57J.57 reject a proposal that correctly introduced a dedicated savings-realism unknown because `selectedQuestion` was missing, and which component currently owns responsibility for supplying that next question?
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — Exact Rejection Trace
|
||||
|
||||
**Function:** `validateQuestionSelectionRequirement(graph, proposal)` at `lib/graph/apply-proposal.js:283`
|
||||
|
||||
**Exact condition:**
|
||||
```javascript
|
||||
const addedConsequentialUnknowns = proposal.addedNodes.filter(
|
||||
(node) => node.kind === "unknown" && node.status !== "resolved",
|
||||
);
|
||||
|
||||
if (
|
||||
proposal.selectedQuestion == null &&
|
||||
addedConsequentialUnknowns.length > 0
|
||||
) {
|
||||
return [
|
||||
"selectedQuestion is required when consequential unresolved unknowns remain after resolving the answered unknown",
|
||||
];
|
||||
}
|
||||
```
|
||||
|
||||
**Inputs used by the condition (from 57J.57's parsed proposal):**
|
||||
- `proposal.selectedQuestion` → `null` (absent; Zod default from `.nullable().default(null)` on schema.js:191)
|
||||
- `proposal.addedNodes` → `[ { id: "nsavings_reality", kind: "unknown", status: <valid non-resolved enum>, confidence: <valid enum>, parentId, dependsOn, affects, childIds } ]`. The `status` field was required by Zod (situationNodeSchema line 60: `status: z.enum(Object.values(SituationStatus))`). The diagnostic snapshot omits it for brevity but it must exist because Zod parsing succeeded at the `proposal_compatibility` stage.
|
||||
- Filter result → `[nsavings_reality]` because `kind === "unknown"` and `status !== "resolved"`
|
||||
|
||||
**Why the condition evaluates true:**
|
||||
1. `proposal.selectedQuestion == null` is **true** — the model did not include a `selectedQuestion` in its JSON output. Zod defaults absent to null.
|
||||
2. `addedConsequentialUnknowns.length > 0` is **true** — one new node with `kind: "unknown"` and a non-resolved status exists in `addedNodes`.
|
||||
|
||||
**Dependencies:**
|
||||
- Does requirement depend on `updatedNodes`: **NO** — the function never inspects `updatedNodes`.
|
||||
- Does requirement depend on `resolvedUnknownNodeIds`: **NO** — the function never inspects this field.
|
||||
- Does requirement depend on `addedNodes`: **YES** — this is the sole input to the condition.
|
||||
- Does requirement depend on remaining unresolved unknowns (existing graph): **NO** — the function does not consult `graph.nodes`. It only looks at what the model added in `addedNodes`.
|
||||
- Does requirement depend on `activeUnknown`: **NO**.
|
||||
- Does requirement depend on `answerMeaning`: **NO**.
|
||||
|
||||
**Critical finding:** The error message says "after resolving the answered unknown" but the actual condition does NOT check `resolvedUnknownNodeIds`, does NOT check whether any node was resolved, and does NOT count existing unresolved unknowns. It fires whenever ANY new unresolved unknown appears in `addedNodes`, regardless of whether an existing node was resolved or even whether the model resolved anything at all. The message is operationally misleading.
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — Selected-Question Owner
|
||||
|
||||
**Current owner: MODEL-PROVIDED (with engine validation/override)**
|
||||
|
||||
Evidence trace:
|
||||
1. The model must include `selectedQuestion` in its JSON proposal per prompt rules #16 and #20.
|
||||
2. Zod defaulting (`selectedQuestionSchema.nullable().default(null)` at schema.js:191) means absent → null.
|
||||
3. `validateQuestionSelectionRequirement` catches absence when `addedNodes` contains unresolved unknowns (57J.57's trigger).
|
||||
4. After validation, in `applyValidatedProposal` (apply-proposal.js:3418–3420): the engine uses `validatedProposal.selectedQuestion.nodeId` as the active unknown if present.
|
||||
5. If no valid selectedQuestion survives (lines 3432–3436), the engine falls back to deterministic `selectActiveUnknownCandidate()`.
|
||||
|
||||
This is **constrained MODEL-PROVIDED**: the model must produce a candidate; the engine validates it and may override via deterministic scoring when the model's candidate is invalid or absent.
|
||||
|
||||
---
|
||||
|
||||
## Part 3 — Ordering Problem
|
||||
|
||||
**Current order of operations:**
|
||||
```
|
||||
1. Zod schema parse (proposal_validation stage)
|
||||
2. reconcileResolutionSemantics (synthetic updates for resolved nodes)
|
||||
3. validateAddedUnknowns (duplicate detection, count ≤ 3)
|
||||
4. validateSelectedQuestionBelongsToChild (structural check)
|
||||
5. validateSelectedQuestion (if present: node existence, unknown kind, unresolved status, compound check, scoring)
|
||||
6. validateAnswerMeaningCompatibilityWithRawAnswer
|
||||
7. validateAnswerMeaningAlignment
|
||||
8. validateQuestionSelectionRequirement ← 57J.57 triggered here
|
||||
9. If all pass → applyGraphUpdate (mutation)
|
||||
10. selectActiveUnknownCandidate (deterministic engine selection)
|
||||
```
|
||||
|
||||
**Can the system deterministically know which unknown should be asked next before mutation:** YES
|
||||
|
||||
**Why:** At step 8, the validator already sees `addedNodes` from the proposal and all existing graph nodes from `situationGraph`. The scoring function (`scoreUnknownCandidate`, called in line 270 of `validateSelectedQuestion`) can evaluate information value for all candidate unknowns without mutation. However, there is a timing tension: the validator requires the model to provide selectedQuestion *before* mutation occurs, but at that point some nodes may not yet be integrated into the graph (addedNodes exists as a separate array). The engine handles this by checking both `graph.nodes` and `proposal.addedNodes` in `buildNodeById` (line 218).
|
||||
|
||||
---
|
||||
|
||||
## Part 4 — Prompt Contract
|
||||
|
||||
**Operational completeness: PARTIAL**
|
||||
|
||||
**What it tells the model:**
|
||||
- Rule #16: "If consequential unresolved unknowns exist, selectedQuestion **may** identify one valid candidate unknown, but the engine will deterministically choose final priority after validation."
|
||||
- Rule #20: "Return selectedQuestion as null only when no consequential unresolved unknown remains."
|
||||
- Rule #17: "selectedQuestion.nodeId must reference an unresolved unknown node that exists either already in the graph or in addedNodes."
|
||||
- Rule #18: "selectedQuestion.question must be one narrow non-compound question about that one unknown."
|
||||
- Required shape (line 94–95): "selectedQuestion: either null or an object using these exact keys: nodeId, question, reason"
|
||||
|
||||
**What it does NOT tell the model:**
|
||||
- The word "**may**" in rule #16 semantically means optionality. This directly conflicts with rule #20's mandatory framing (null is only acceptable when nothing remains unresolved). When the model adds a new unknown (not resolving an existing one), there is no positive instruction stating "you MUST include selectedQuestion."
|
||||
- Rule #6 requires structural mutation for consequential uncertainty but does not explicitly connect this to selectedQuestion obligation.
|
||||
- No explicit mapping from condition "I added an unresolved unknown" → "therefore selectedQuestion is mandatory."
|
||||
|
||||
---
|
||||
|
||||
## Part 5 — Controlled Cases
|
||||
|
||||
### Case A — update resolves current unknown, other unresolved unknowns remain
|
||||
**SelectedQuestion required:** DEPENDS
|
||||
**Why:** Only if the update ALSO adds new unknown nodes. If only existing nodes are updated/resolved without adding new unknowns, `validateQuestionSelectionRequirement` never fires (it only checks `addedNodes`). Other validators may still require it depending on downstream flow.
|
||||
**Matches current behaviour:** YES — this validator only checks addedNodes, not existing graph state.
|
||||
|
||||
### Case B — update introduces a new unresolved unknown and resolves nothing
|
||||
**SelectedQuestion required:** YES
|
||||
**Why:** Any new unresolved unknown triggers the requirement unconditionally. Correct behavior: without a selected question, there's no way to determine what to ask next.
|
||||
**Matches current behaviour:** YES — this is exactly what happened in 57J.57.
|
||||
|
||||
### Case C — evidence/state change, unresolved set unchanged
|
||||
**SelectedQuestion required:** DEPENDS
|
||||
**Why:** This validator does NOT fire (no new unknown nodes). The question requirement here comes from other parts of the pipeline (e.g., `validateAnswerMeaningAlignment` or downstream engine logic) if the active unknown changed.
|
||||
**Matches current behaviour:** YES — this validator stays silent; other mechanisms handle it.
|
||||
|
||||
### Case D — proposal leaves no consequential unresolved unknowns
|
||||
**SelectedQuestion required:** NO
|
||||
**Why:** Either no unknowns exist (investigation complete) or selectedQuestion was null by rule #20 and no new unknowns were added.
|
||||
**Matches current behaviour:** YES.
|
||||
|
||||
---
|
||||
|
||||
## Part 6 — Architecture Ownership Classification
|
||||
|
||||
### Evaluation of four explanations:
|
||||
|
||||
**A — MODEL OMISSION**
|
||||
The prompt has rules addressing selectedQuestion but uses contradictory language ("may" vs "only when null"). The model correctly understood the semantics (created the savings-realism node) but omitted the field because the prompt made it appear optional via rule #16.
|
||||
|
||||
**B — PROMPT CONTRACT GAP** ✅ BEST FIT
|
||||
Rule #16's "may identify" is semantically permissive, while rule #20 only defines when null is acceptable (via negation). No positive statement says "you MUST include selectedQuestion whenever you add an unresolved unknown." The contradiction between these two rules creates genuine ambiguity about obligation.
|
||||
|
||||
**C — VALIDATION ORDER GAP**
|
||||
The validator fires before mutation but correctly sees `addedNodes`. This is NOT the primary problem — the validator has sufficient information. The deeper timing tension (requiring pre-mutation question when engine can only determine post-mutation) exists but is secondary to the prompt ambiguity.
|
||||
|
||||
**D — RESPONSIBILITY SPLIT GAP**
|
||||
The model provides a candidate; the engine validates and may override. Rule #16's "engine will deterministically choose final priority" could make the model defer selection entirely. This split contributes to confusion but originates from the prompt's ambiguous language.
|
||||
|
||||
### Best classification: **B — PROMPT CONTRACT GAP**
|
||||
|
||||
---
|
||||
|
||||
## Part 7 — Smallest Next Boundary
|
||||
|
||||
**Selected: B — prompt-only clarification**
|
||||
|
||||
The smallest change is to clarify rule #16:
|
||||
- Change "may identify" to mandatory language ("MUST include a candidate selectedQuestion identifying one unresolved unknown").
|
||||
- Clarify the trigger condition: "When you add any new unresolved unknown (status !== 'resolved'), you must provide selectedQuestion even if you did not resolve any existing node."
|
||||
|
||||
This does NOT require validator changes, scoring changes, or question-selection ownership transfer. It only removes the semantic ambiguity that made `selectedQuestion` appear optional in rule #16.
|
||||
Reference in New Issue
Block a user