diff --git a/docs/current-handoff.md b/docs/current-handoff.md index b25eafd..7adf109 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -2953,3 +2953,11 @@ Experiment 60B.34 was a live regression on `feature/bare-whether-proposition-v0. --- Experiment 60B.35 validated that a bare `Whether...` description survives the **full** applyValidatedProposal runtime path end-to-end. **Classification: A — FULL LIFECYCLE CONFIRMED.** The model-produced node `n_enterprise_customer_signing` (kind=unknown, status=unknown) with a depends_on edge reached the full production apply-proposal pipeline without proposition degradation. Final selected question was `"What evidence would clarify whether the prospective enterprise customer will commit this year?"` — proposition-specific evidence framing confirmed at the output boundary. No rationale contamination, decision identity preserved, no unrelated uncertainty invented. This closes the verification gap from 60B.34: bare Whether... preservation now proven end-to-end in both partial and full runtime paths. One live call. No production code changed. Deterministic test added (removed in cleanup commit). + +--- + +Experiment 60B.37 ran the customer-signing followup fixture through a single bounded update to test whether resolving the last material uncertainty of an existing unresolved decision updates that same factor in place and closes the decision cleanly. **Classification: C — FACTOR RESOLVES BUT GENERIC CONTINUATION REMAINS.** The enterprise-customer unknown (`n_enterprise_customer_signing`) resolved correctly in place (unknown → resolved) with no duplication. The product-launch decision (`n_product_launch_decision`) was correctly transitioned to status=known with rationale "Prerequisite uncertainty resolved and user confirms no other material uncertainties remain between the options." Both option identities preserved. No new nodes or edges created. **However**, the engine still produced a `selectedQuestion` ("What outcome would demonstrate enough value to justify launching?") targeting `n_product_launch_decision`, which has status=known. This is a structural inconsistency: decision-level closure at the reasoning level but continuation at the orchestration level. One live call. No production code changed. Status pending review. + +--- + +Experiment 60B.38 was a read-only diagnosis of why `n_product_launch_decision` becomes known in the same update yet still survives as the final selectedQuestion target. **Classification: B — QUESTION TARGET VALIDATION IS WRONG.** Two independent gaps discovered in the selectable-node predicate chain: (1) `validateSelectedQuestion` at lib/graph/apply-proposal.js:246 checks `effectiveStatus === "resolved"` only, NOT `"known"`; (2) `isSelectableUnresolvedUnknown` at lib/graph/apply-proposal.js:1663 excludes ["resolved", "contradicted"] but NOT "known". These gaps propagate through the entire pipeline: `selectActiveUnknownCandidate` in utils.js has no status filter at all. The decision node transitioned via `updatedNodes.newStatus="known"` rather than `resolvedUnknownNodeIds`, and no predicate catches this gap. **Minimum corrective boundary:** C — unify all final targets through one canonical post-mutation unresolved/selectable check. Add "known" to the exclusion list in `isSelectableUnresolvedUnknown` and add `effectiveStatus === "known"` check in `validateSelectedQuestion`. No production code changed. 0 Ollama calls. Pure code inspection. Full trace in docs/experiment-60b38.md. diff --git a/docs/experiment-60b38.md b/docs/experiment-60b38.md new file mode 100644 index 0000000..680209e --- /dev/null +++ b/docs/experiment-60b38.md @@ -0,0 +1,217 @@ +# Experiment 60B.38 — Stale question target after resolution + +**Date:** 2026-08-14 +**Branch:** `feature/customer-signing-followup-fixture-v0.35` + +## Purpose + +Diagnose why a node transitioned to `status=known` in the same update still survives as the final `selectedQuestion` target — despite 60B.37 confirming that both status-level closure (`status=known`) and reasoning-level correctness were achieved. + +**DO NOT MODIFY PRODUCTION CODE. DO NOT RUN TESTS. DO NOT CALL OLLAMA.** + +This is a pure code-path diagnosis experiment. + +## Fixed observations from 60B.37 + +``` +updatedNodes (2): + n_enterprise_customer_signing: unknown → resolved + n_product_launch_decision: unknown → known + +resolvedUnknownNodeIds: + ["n_enterprise_customer_signing"] + +addedNodes / addedEdges: + [] / [] + +proposal.selectedQuestion.nodeId: + n_product_launch_decision + +final.selectedQuestion.nodeId: + n_product_launch_decision (status=known) +``` + +## Analysis approach + +Trace `applyValidatedProposal` line-by-line through the deterministic lifecycle: +- validateSelectedQuestion → applyGraphUpdate → decomposition → propagation → selection reformulation +- Identify exact predicates that filter candidates +- Check whether each excludes `status=known` nodes + +## Key production functions inspected + +### lib/graph/apply-proposal.js + +1. **validateSelectedQuestion** (line 215–281) + - Line 246: checks `effectiveStatus === "resolved"` only + - Does NOT check `effectiveStatus === "known"` + +2. **isSelectableUnresolvedUnknown** (line 1658–1665) + - Line 1663: excludes `["resolved", "contradicted"]` + - Does NOT exclude `"known"` + +3. **selectActiveUnknownCandidate** (lib/graph/utils.js line 593–642) + - Line 596: filters only `n.kind === "unknown" && !resolvedNodeIds.includes(n.id)` + - No status check at all + +4. **listUnresolvedUnknownCandidates** (line 1668–1679) + - Line 1677: excludes `["resolved", "contradicted"]` + - Does NOT exclude `"known"` + +5. **remainingUnknownExists** check (line 3705–3712) + - Checks only `node.kind === "unknown" && !resolvedNodeIds.includes(node.id)` + - No status check + +6. **carriedActiveUnknownStillUnresolved** (line 3847–3854) + - Excludes `["resolved", "contradicted"]` + - Does NOT exclude `"known"` + +### lib/graph/utils.js + +- **scoreUnknownCandidate** (line 332): no status filtering — only priority, dependencies, text matching +- **selectActiveUnknownCandidate** (line 593): same gap — kind=unknown only, resolvedNodeIds only + +## Lifecycle ordering in applyValidatedProposal + +1. Graph validation (situationGraphSchema) +2. Proposal compatibility validation (graphUpdateSchema) +3. reconcileResolutionSemantics +4. validateGraphUpdate +5. **validateSelectedQuestion** ← pre-mutation check at line 3594 +6. validateAnswerMeaningCompatibilityWithRawAnswer +7. validateAnswerMeaningAlignment +8. validateQuestionSelectionRequirement +9. Detect structural errors (proposalsCompatibilityErrors) +10. Collect structurally admitted node IDs +11. **applyGraphUpdate** ← mutation happens here at line 3641 +12. Build reasoning state +13. Run deterministic decomposition +14. Propagate resolved child evidence +15. Post-propagation candidate assessment +16. Model-selection honour path (line 3972) +17. Deterministic fallback selection (line 4008) +18. Set final selectedQuestion from deterministicSelection + +## Root cause + +### Two independent gaps in the selectable-node predicate chain: + +**Gap 1 — validateSelectedQuestion (pre-mutation)** at line 246: +```javascript +if (resolvesNode || effectiveStatus === "resolved") { +``` +This rejects `selectedQuestion.nodeId` when the node is explicitly in `resolvedUnknownNodeIds` OR when its new status via `updatedNodes` is `"resolved"`. But it does NOT check for `effectiveStatus === "known"`. + +A node transitioned to `status=known` via `updatedNodes` passes this validation silently. + +**Gap 2 — isSelectableUnresolvedUnknown (post-mutation)** at line 1663: +```javascript +!["resolved", "contradicted"].includes(node.status) +``` +This predicate is used throughout the pipeline to determine whether a node can be selected as the next question target. It correctly excludes `"resolved"` and `"contradicted"` but does NOT exclude `"known"`. + +Since `kind` stays `"unknown"` while `status` changes to `"known"`, the predicate returns true for known-status nodes that should not be selectable. + +This gap propagates through: +- `isSelectableUnresolvedUnknown` (used at lines 2406, 2434, 2466, 3764, 3984, 3764) +- `selectActiveUnknownCandidate` in utils.js (used at line 3722, used as fallback selector) +- `listUnresolvedUnknownCandidates` / `listEligibleUnknownCandidates` +- `remainingUnknownExists` check at line 3705 + +## The exact path in 60B.37 + +1. **Model proposes**: `updatedNodes[n_product_launch_decision] = { newStatus: "known" }`, `selectedQuestion.nodeId = "n_product_launch_decision"` +2. **validateSelectedQuestion** (pre-mutation): effectiveStatus = "known" → line 246 check fails (only catches "resolved") → NO ERROR +3. **applyGraphUpdate** (line 3641): n_product_launch_decision gets status=known in the updated graph +4. **Lines 3705-3712 remainingUnknownExists**: kind=unknown ✓, not in resolvedNodeIds ✓ → returns true → no reselection triggered +5. **Line 3722 selectActiveUnknownCandidate**: filters by kind=unknown + not in resolvedNodeIds. n_product_launch_decision passes (kind=unknown, NOT in resolvedUnknownNodeIds). Returns { nodeId: "n_product_launch_decision", status: "selected" } +6. **Lines 3783-3792 preservation check**: isSelectableUnresolvedUnknown returns true for known-status node → preservedSelectedChildNode set to decision node +7. **Line 4055 finalSelectedQuestion**: built from deterministicSelection.nodeId = "n_product_launch_decision" (status=known) +8. **Result**: A node with status=known receives a follow-up question despite decision-level closure being complete + +## Asymmetry between resolution paths + +**resolvedUnknownNodeIds exclusion:** YES — nodes in this array are checked at line 238 and excluded by resolvedNodeIds throughout the pipeline. + +**updated-to-known exclusion:** NO — no function in the entire chain checks `status !== "known"` as a filter condition. `"known"` is not in any exclusion list. + +**Asymmetry exists:** YES + +The path via `resolvedUnknownNodeIds` (explicit resolution) is fully guarded. The path via `updatedNodes[n].newStatus = "known"` (implicit resolution) is NOT guarded because: +- validateSelectedQuestion only catches "resolved" status, not "known" +- isSelectableUnresolvedUnknown only excludes ["resolved", "contradicted"], not "known" +- selectActiveUnknownCandidate has no status check at all + +## Active unknown lifecycle for this case + +``` +Pre-update active node: n_enterprise_customer_signing +Post-mutation active node before reselection: null (cleared at line 3698 because previous was resolved) +Final active node: n_product_launch_decision (set at line 3702 from proposal.selectedQuestion, then NOT re-evaluated for status validity) +``` + +A known-status unknown can remain `activeUnknownNodeId` because `remainingUnknownExists` only checks kind and resolvedNodeIds. + +## Cause assessment + +### Candidate A — EARLY VALIDATION / LATE MUTATION + +**Evidence:** MEDIUM-HIGH +- validateSelectedQuestion is called at line 3594 (before mutation at line 3641) +- But the gap is not about timing — even a post-mutation check would miss "known" because the predicate doesn't filter it +- The validation exists but has an incomplete status filter + +**Explains 60B.37:** PARTIAL — captures the pre-mutation aspect but not the status filtering gap + +### Candidate B — `resolvedUnknownNodeIds`-ONLY FILTER + +**Evidence:** HIGH +- Every predicate in the pipeline (`isSelectableUnresolvedUnknown`, `selectActiveUnknownCandidate`, `listUnresolvedUnknownCandidates`) that should exclude resolved nodes only checks: + - kind === "unknown" (always true for unknown-type nodes) + - not in resolvedNodeIds/resolvedUnknownNodeIds +- None check status against the full set of terminal statuses ["resolved", "known", "contradicted"] + +**Explains 60B.37:** YES — this is the precise mechanism. The decision node transitions via updatedNodes.newStatus="known" rather than resolvedUnknownNodeIds, and no predicate catches the gap. + +### Candidate C — PREFERRED-TARGET PATH BYPASSES NORMAL SELECTABILITY + +**Evidence:** MEDIUM +- Model-selected target at line 3976 has an explicit isSelectableUnresolvedUnknown check (line 3984) +- This check would pass for known-status nodes due to the predicate gap +- However, in 60B.37 the decision node was NOT newly added, so this path doesn't apply +- The model-selection honour path correctly skips it + +**Explains 60B.37:** PARTIAL — the gap exists but the specific path is blocked by the "newly added" check + +### Candidate D — ACTIVE NODE LIFECYCLE STALE + +**Evidence:** MEDIUM +- remainingUnknownExists at line 3705 doesn't check status +- But in 60B.37, n_product_launch_decision becomes active via line 3702 (from proposal.selectedQuestion), not from remainingUnknownExists +- The real issue is the target selection path, not active node management per se + +**Explains 60B.37:** PARTIAL — contributes to the stale state but isn't the root cause + +## Critical distinction + +**Choice: B — QUESTION TARGET VALIDATION IS WRONG** + +Why: The graph resolution itself was observed as correct in 60B.37 (n_product_launch_decision correctly became status=known with correct rationale). The failure is specifically at the question-target validation layer: multiple predicates filter terminal statuses but collectively miss "known". This is a validation predicate gap, not a resolution state error or active node lifecycle issue. + +## Minimum corrective boundary + +**Choice: C — UNIFY ALL FINAL TARGETS THROUGH ONE SELECTABILITY PREDICATE** + +Why: The fix requires making `isSelectableUnresolvedUnknown` correctly exclude `status === "known"` nodes AND ensuring `validateSelectedQuestion` checks effective status against all terminal states including "known". This ensures whether the target comes from model preference, active node persistence, or deterministic selector, it passes one canonical post-mutation unresolved/selectable check. + +Would preserve valid unresolved preferred targets: YES — only known/resolved/contradicted nodes are excluded +Would preserve prerequisite-first fallback: YES — unaffected by status filtering changes +Would prevent known decision nodes from receiving final questions: YES — all selection paths would use the corrected predicate + +## Implementation readiness + +**A — READY FOR BOUNDED IMPLEMENTATION** + +The diagnosis is complete. The exact code paths and predicates are identified. The fix is a single-predicate correction to `isSelectableUnresolvedUnknown` and one status check addition in `validateSelectedQuestion`. + +Smallest implementation boundary: Two changes — (1) add "known" to the exclusion list in `isSelectableUnresolvedUnknown`, (2) add `effectiveStatus === "known"` check in `validateSelectedQuestion` at line 246.