218 lines
11 KiB
Markdown
218 lines
11 KiB
Markdown
# Experiment 60B.38 — Stale question target after resolution
|
||
|
||
**Date:** 2026-08-14
|
||
**Branch:** `feature/customer-signing-followup-fixture-v0.35`
|
||
|
||
## Purpose
|
||
|
||
Diagnose why a node transitioned to `status=known` in the same update still survives as the final `selectedQuestion` target — despite 60B.37 confirming that both status-level closure (`status=known`) and reasoning-level correctness were achieved.
|
||
|
||
**DO NOT MODIFY PRODUCTION CODE. DO NOT RUN TESTS. DO NOT CALL OLLAMA.**
|
||
|
||
This is a pure code-path diagnosis experiment.
|
||
|
||
## Fixed observations from 60B.37
|
||
|
||
```
|
||
updatedNodes (2):
|
||
n_enterprise_customer_signing: unknown → resolved
|
||
n_product_launch_decision: unknown → known
|
||
|
||
resolvedUnknownNodeIds:
|
||
["n_enterprise_customer_signing"]
|
||
|
||
addedNodes / addedEdges:
|
||
[] / []
|
||
|
||
proposal.selectedQuestion.nodeId:
|
||
n_product_launch_decision
|
||
|
||
final.selectedQuestion.nodeId:
|
||
n_product_launch_decision (status=known)
|
||
```
|
||
|
||
## Analysis approach
|
||
|
||
Trace `applyValidatedProposal` line-by-line through the deterministic lifecycle:
|
||
- validateSelectedQuestion → applyGraphUpdate → decomposition → propagation → selection reformulation
|
||
- Identify exact predicates that filter candidates
|
||
- Check whether each excludes `status=known` nodes
|
||
|
||
## Key production functions inspected
|
||
|
||
### lib/graph/apply-proposal.js
|
||
|
||
1. **validateSelectedQuestion** (line 215–281)
|
||
- Line 246: checks `effectiveStatus === "resolved"` only
|
||
- Does NOT check `effectiveStatus === "known"`
|
||
|
||
2. **isSelectableUnresolvedUnknown** (line 1658–1665)
|
||
- Line 1663: excludes `["resolved", "contradicted"]`
|
||
- Does NOT exclude `"known"`
|
||
|
||
3. **selectActiveUnknownCandidate** (lib/graph/utils.js line 593–642)
|
||
- Line 596: filters only `n.kind === "unknown" && !resolvedNodeIds.includes(n.id)`
|
||
- No status check at all
|
||
|
||
4. **listUnresolvedUnknownCandidates** (line 1668–1679)
|
||
- Line 1677: excludes `["resolved", "contradicted"]`
|
||
- Does NOT exclude `"known"`
|
||
|
||
5. **remainingUnknownExists** check (line 3705–3712)
|
||
- Checks only `node.kind === "unknown" && !resolvedNodeIds.includes(node.id)`
|
||
- No status check
|
||
|
||
6. **carriedActiveUnknownStillUnresolved** (line 3847–3854)
|
||
- Excludes `["resolved", "contradicted"]`
|
||
- Does NOT exclude `"known"`
|
||
|
||
### lib/graph/utils.js
|
||
|
||
- **scoreUnknownCandidate** (line 332): no status filtering — only priority, dependencies, text matching
|
||
- **selectActiveUnknownCandidate** (line 593): same gap — kind=unknown only, resolvedNodeIds only
|
||
|
||
## Lifecycle ordering in applyValidatedProposal
|
||
|
||
1. Graph validation (situationGraphSchema)
|
||
2. Proposal compatibility validation (graphUpdateSchema)
|
||
3. reconcileResolutionSemantics
|
||
4. validateGraphUpdate
|
||
5. **validateSelectedQuestion** ← pre-mutation check at line 3594
|
||
6. validateAnswerMeaningCompatibilityWithRawAnswer
|
||
7. validateAnswerMeaningAlignment
|
||
8. validateQuestionSelectionRequirement
|
||
9. Detect structural errors (proposalsCompatibilityErrors)
|
||
10. Collect structurally admitted node IDs
|
||
11. **applyGraphUpdate** ← mutation happens here at line 3641
|
||
12. Build reasoning state
|
||
13. Run deterministic decomposition
|
||
14. Propagate resolved child evidence
|
||
15. Post-propagation candidate assessment
|
||
16. Model-selection honour path (line 3972)
|
||
17. Deterministic fallback selection (line 4008)
|
||
18. Set final selectedQuestion from deterministicSelection
|
||
|
||
## Root cause
|
||
|
||
### Two independent gaps in the selectable-node predicate chain:
|
||
|
||
**Gap 1 — validateSelectedQuestion (pre-mutation)** at line 246:
|
||
```javascript
|
||
if (resolvesNode || effectiveStatus === "resolved") {
|
||
```
|
||
This rejects `selectedQuestion.nodeId` when the node is explicitly in `resolvedUnknownNodeIds` OR when its new status via `updatedNodes` is `"resolved"`. But it does NOT check for `effectiveStatus === "known"`.
|
||
|
||
A node transitioned to `status=known` via `updatedNodes` passes this validation silently.
|
||
|
||
**Gap 2 — isSelectableUnresolvedUnknown (post-mutation)** at line 1663:
|
||
```javascript
|
||
!["resolved", "contradicted"].includes(node.status)
|
||
```
|
||
This predicate is used throughout the pipeline to determine whether a node can be selected as the next question target. It correctly excludes `"resolved"` and `"contradicted"` but does NOT exclude `"known"`.
|
||
|
||
Since `kind` stays `"unknown"` while `status` changes to `"known"`, the predicate returns true for known-status nodes that should not be selectable.
|
||
|
||
This gap propagates through:
|
||
- `isSelectableUnresolvedUnknown` (used at lines 2406, 2434, 2466, 3764, 3984, 3764)
|
||
- `selectActiveUnknownCandidate` in utils.js (used at line 3722, used as fallback selector)
|
||
- `listUnresolvedUnknownCandidates` / `listEligibleUnknownCandidates`
|
||
- `remainingUnknownExists` check at line 3705
|
||
|
||
## The exact path in 60B.37
|
||
|
||
1. **Model proposes**: `updatedNodes[n_product_launch_decision] = { newStatus: "known" }`, `selectedQuestion.nodeId = "n_product_launch_decision"`
|
||
2. **validateSelectedQuestion** (pre-mutation): effectiveStatus = "known" → line 246 check fails (only catches "resolved") → NO ERROR
|
||
3. **applyGraphUpdate** (line 3641): n_product_launch_decision gets status=known in the updated graph
|
||
4. **Lines 3705-3712 remainingUnknownExists**: kind=unknown ✓, not in resolvedNodeIds ✓ → returns true → no reselection triggered
|
||
5. **Line 3722 selectActiveUnknownCandidate**: filters by kind=unknown + not in resolvedNodeIds. n_product_launch_decision passes (kind=unknown, NOT in resolvedUnknownNodeIds). Returns { nodeId: "n_product_launch_decision", status: "selected" }
|
||
6. **Lines 3783-3792 preservation check**: isSelectableUnresolvedUnknown returns true for known-status node → preservedSelectedChildNode set to decision node
|
||
7. **Line 4055 finalSelectedQuestion**: built from deterministicSelection.nodeId = "n_product_launch_decision" (status=known)
|
||
8. **Result**: A node with status=known receives a follow-up question despite decision-level closure being complete
|
||
|
||
## Asymmetry between resolution paths
|
||
|
||
**resolvedUnknownNodeIds exclusion:** YES — nodes in this array are checked at line 238 and excluded by resolvedNodeIds throughout the pipeline.
|
||
|
||
**updated-to-known exclusion:** NO — no function in the entire chain checks `status !== "known"` as a filter condition. `"known"` is not in any exclusion list.
|
||
|
||
**Asymmetry exists:** YES
|
||
|
||
The path via `resolvedUnknownNodeIds` (explicit resolution) is fully guarded. The path via `updatedNodes[n].newStatus = "known"` (implicit resolution) is NOT guarded because:
|
||
- validateSelectedQuestion only catches "resolved" status, not "known"
|
||
- isSelectableUnresolvedUnknown only excludes ["resolved", "contradicted"], not "known"
|
||
- selectActiveUnknownCandidate has no status check at all
|
||
|
||
## Active unknown lifecycle for this case
|
||
|
||
```
|
||
Pre-update active node: n_enterprise_customer_signing
|
||
Post-mutation active node before reselection: null (cleared at line 3698 because previous was resolved)
|
||
Final active node: n_product_launch_decision (set at line 3702 from proposal.selectedQuestion, then NOT re-evaluated for status validity)
|
||
```
|
||
|
||
A known-status unknown can remain `activeUnknownNodeId` because `remainingUnknownExists` only checks kind and resolvedNodeIds.
|
||
|
||
## Cause assessment
|
||
|
||
### Candidate A — EARLY VALIDATION / LATE MUTATION
|
||
|
||
**Evidence:** MEDIUM-HIGH
|
||
- validateSelectedQuestion is called at line 3594 (before mutation at line 3641)
|
||
- But the gap is not about timing — even a post-mutation check would miss "known" because the predicate doesn't filter it
|
||
- The validation exists but has an incomplete status filter
|
||
|
||
**Explains 60B.37:** PARTIAL — captures the pre-mutation aspect but not the status filtering gap
|
||
|
||
### Candidate B — `resolvedUnknownNodeIds`-ONLY FILTER
|
||
|
||
**Evidence:** HIGH
|
||
- Every predicate in the pipeline (`isSelectableUnresolvedUnknown`, `selectActiveUnknownCandidate`, `listUnresolvedUnknownCandidates`) that should exclude resolved nodes only checks:
|
||
- kind === "unknown" (always true for unknown-type nodes)
|
||
- not in resolvedNodeIds/resolvedUnknownNodeIds
|
||
- None check status against the full set of terminal statuses ["resolved", "known", "contradicted"]
|
||
|
||
**Explains 60B.37:** YES — this is the precise mechanism. The decision node transitions via updatedNodes.newStatus="known" rather than resolvedUnknownNodeIds, and no predicate catches the gap.
|
||
|
||
### Candidate C — PREFERRED-TARGET PATH BYPASSES NORMAL SELECTABILITY
|
||
|
||
**Evidence:** MEDIUM
|
||
- Model-selected target at line 3976 has an explicit isSelectableUnresolvedUnknown check (line 3984)
|
||
- This check would pass for known-status nodes due to the predicate gap
|
||
- However, in 60B.37 the decision node was NOT newly added, so this path doesn't apply
|
||
- The model-selection honour path correctly skips it
|
||
|
||
**Explains 60B.37:** PARTIAL — the gap exists but the specific path is blocked by the "newly added" check
|
||
|
||
### Candidate D — ACTIVE NODE LIFECYCLE STALE
|
||
|
||
**Evidence:** MEDIUM
|
||
- remainingUnknownExists at line 3705 doesn't check status
|
||
- But in 60B.37, n_product_launch_decision becomes active via line 3702 (from proposal.selectedQuestion), not from remainingUnknownExists
|
||
- The real issue is the target selection path, not active node management per se
|
||
|
||
**Explains 60B.37:** PARTIAL — contributes to the stale state but isn't the root cause
|
||
|
||
## Critical distinction
|
||
|
||
**Choice: B — QUESTION TARGET VALIDATION IS WRONG**
|
||
|
||
Why: The graph resolution itself was observed as correct in 60B.37 (n_product_launch_decision correctly became status=known with correct rationale). The failure is specifically at the question-target validation layer: multiple predicates filter terminal statuses but collectively miss "known". This is a validation predicate gap, not a resolution state error or active node lifecycle issue.
|
||
|
||
## Minimum corrective boundary
|
||
|
||
**Choice: C — UNIFY ALL FINAL TARGETS THROUGH ONE SELECTABILITY PREDICATE**
|
||
|
||
Why: The fix requires making `isSelectableUnresolvedUnknown` correctly exclude `status === "known"` nodes AND ensuring `validateSelectedQuestion` checks effective status against all terminal states including "known". This ensures whether the target comes from model preference, active node persistence, or deterministic selector, it passes one canonical post-mutation unresolved/selectable check.
|
||
|
||
Would preserve valid unresolved preferred targets: YES — only known/resolved/contradicted nodes are excluded
|
||
Would preserve prerequisite-first fallback: YES — unaffected by status filtering changes
|
||
Would prevent known decision nodes from receiving final questions: YES — all selection paths would use the corrected predicate
|
||
|
||
## Implementation readiness
|
||
|
||
**A — READY FOR BOUNDED IMPLEMENTATION**
|
||
|
||
The diagnosis is complete. The exact code paths and predicates are identified. The fix is a single-predicate correction to `isSelectableUnresolvedUnknown` and one status check addition in `validateSelectedQuestion`.
|
||
|
||
Smallest implementation boundary: Two changes — (1) add "known" to the exclusion list in `isSelectableUnresolvedUnknown`, (2) add `effectiveStatus === "known"` check in `validateSelectedQuestion` at line 246.
|