experiment: diagnose stale question after resolution
This commit is contained in:
@@ -2953,3 +2953,11 @@ Experiment 60B.34 was a live regression on `feature/bare-whether-proposition-v0.
|
||||
---
|
||||
|
||||
Experiment 60B.35 validated that a bare `Whether...` description survives the **full** applyValidatedProposal runtime path end-to-end. **Classification: A — FULL LIFECYCLE CONFIRMED.** The model-produced node `n_enterprise_customer_signing` (kind=unknown, status=unknown) with a depends_on edge reached the full production apply-proposal pipeline without proposition degradation. Final selected question was `"What evidence would clarify whether the prospective enterprise customer will commit this year?"` — proposition-specific evidence framing confirmed at the output boundary. No rationale contamination, decision identity preserved, no unrelated uncertainty invented. This closes the verification gap from 60B.34: bare Whether... preservation now proven end-to-end in both partial and full runtime paths. One live call. No production code changed. Deterministic test added (removed in cleanup commit).
|
||||
|
||||
---
|
||||
|
||||
Experiment 60B.37 ran the customer-signing followup fixture through a single bounded update to test whether resolving the last material uncertainty of an existing unresolved decision updates that same factor in place and closes the decision cleanly. **Classification: C — FACTOR RESOLVES BUT GENERIC CONTINUATION REMAINS.** The enterprise-customer unknown (`n_enterprise_customer_signing`) resolved correctly in place (unknown → resolved) with no duplication. The product-launch decision (`n_product_launch_decision`) was correctly transitioned to status=known with rationale "Prerequisite uncertainty resolved and user confirms no other material uncertainties remain between the options." Both option identities preserved. No new nodes or edges created. **However**, the engine still produced a `selectedQuestion` ("What outcome would demonstrate enough value to justify launching?") targeting `n_product_launch_decision`, which has status=known. This is a structural inconsistency: decision-level closure at the reasoning level but continuation at the orchestration level. One live call. No production code changed. Status pending review.
|
||||
|
||||
---
|
||||
|
||||
Experiment 60B.38 was a read-only diagnosis of why `n_product_launch_decision` becomes known in the same update yet still survives as the final selectedQuestion target. **Classification: B — QUESTION TARGET VALIDATION IS WRONG.** Two independent gaps discovered in the selectable-node predicate chain: (1) `validateSelectedQuestion` at lib/graph/apply-proposal.js:246 checks `effectiveStatus === "resolved"` only, NOT `"known"`; (2) `isSelectableUnresolvedUnknown` at lib/graph/apply-proposal.js:1663 excludes ["resolved", "contradicted"] but NOT "known". These gaps propagate through the entire pipeline: `selectActiveUnknownCandidate` in utils.js has no status filter at all. The decision node transitioned via `updatedNodes.newStatus="known"` rather than `resolvedUnknownNodeIds`, and no predicate catches this gap. **Minimum corrective boundary:** C — unify all final targets through one canonical post-mutation unresolved/selectable check. Add "known" to the exclusion list in `isSelectableUnresolvedUnknown` and add `effectiveStatus === "known"` check in `validateSelectedQuestion`. No production code changed. 0 Ollama calls. Pure code inspection. Full trace in docs/experiment-60b38.md.
|
||||
|
||||
@@ -0,0 +1,217 @@
|
||||
# Experiment 60B.38 — Stale question target after resolution
|
||||
|
||||
**Date:** 2026-08-14
|
||||
**Branch:** `feature/customer-signing-followup-fixture-v0.35`
|
||||
|
||||
## Purpose
|
||||
|
||||
Diagnose why a node transitioned to `status=known` in the same update still survives as the final `selectedQuestion` target — despite 60B.37 confirming that both status-level closure (`status=known`) and reasoning-level correctness were achieved.
|
||||
|
||||
**DO NOT MODIFY PRODUCTION CODE. DO NOT RUN TESTS. DO NOT CALL OLLAMA.**
|
||||
|
||||
This is a pure code-path diagnosis experiment.
|
||||
|
||||
## Fixed observations from 60B.37
|
||||
|
||||
```
|
||||
updatedNodes (2):
|
||||
n_enterprise_customer_signing: unknown → resolved
|
||||
n_product_launch_decision: unknown → known
|
||||
|
||||
resolvedUnknownNodeIds:
|
||||
["n_enterprise_customer_signing"]
|
||||
|
||||
addedNodes / addedEdges:
|
||||
[] / []
|
||||
|
||||
proposal.selectedQuestion.nodeId:
|
||||
n_product_launch_decision
|
||||
|
||||
final.selectedQuestion.nodeId:
|
||||
n_product_launch_decision (status=known)
|
||||
```
|
||||
|
||||
## Analysis approach
|
||||
|
||||
Trace `applyValidatedProposal` line-by-line through the deterministic lifecycle:
|
||||
- validateSelectedQuestion → applyGraphUpdate → decomposition → propagation → selection reformulation
|
||||
- Identify exact predicates that filter candidates
|
||||
- Check whether each excludes `status=known` nodes
|
||||
|
||||
## Key production functions inspected
|
||||
|
||||
### lib/graph/apply-proposal.js
|
||||
|
||||
1. **validateSelectedQuestion** (line 215–281)
|
||||
- Line 246: checks `effectiveStatus === "resolved"` only
|
||||
- Does NOT check `effectiveStatus === "known"`
|
||||
|
||||
2. **isSelectableUnresolvedUnknown** (line 1658–1665)
|
||||
- Line 1663: excludes `["resolved", "contradicted"]`
|
||||
- Does NOT exclude `"known"`
|
||||
|
||||
3. **selectActiveUnknownCandidate** (lib/graph/utils.js line 593–642)
|
||||
- Line 596: filters only `n.kind === "unknown" && !resolvedNodeIds.includes(n.id)`
|
||||
- No status check at all
|
||||
|
||||
4. **listUnresolvedUnknownCandidates** (line 1668–1679)
|
||||
- Line 1677: excludes `["resolved", "contradicted"]`
|
||||
- Does NOT exclude `"known"`
|
||||
|
||||
5. **remainingUnknownExists** check (line 3705–3712)
|
||||
- Checks only `node.kind === "unknown" && !resolvedNodeIds.includes(node.id)`
|
||||
- No status check
|
||||
|
||||
6. **carriedActiveUnknownStillUnresolved** (line 3847–3854)
|
||||
- Excludes `["resolved", "contradicted"]`
|
||||
- Does NOT exclude `"known"`
|
||||
|
||||
### lib/graph/utils.js
|
||||
|
||||
- **scoreUnknownCandidate** (line 332): no status filtering — only priority, dependencies, text matching
|
||||
- **selectActiveUnknownCandidate** (line 593): same gap — kind=unknown only, resolvedNodeIds only
|
||||
|
||||
## Lifecycle ordering in applyValidatedProposal
|
||||
|
||||
1. Graph validation (situationGraphSchema)
|
||||
2. Proposal compatibility validation (graphUpdateSchema)
|
||||
3. reconcileResolutionSemantics
|
||||
4. validateGraphUpdate
|
||||
5. **validateSelectedQuestion** ← pre-mutation check at line 3594
|
||||
6. validateAnswerMeaningCompatibilityWithRawAnswer
|
||||
7. validateAnswerMeaningAlignment
|
||||
8. validateQuestionSelectionRequirement
|
||||
9. Detect structural errors (proposalsCompatibilityErrors)
|
||||
10. Collect structurally admitted node IDs
|
||||
11. **applyGraphUpdate** ← mutation happens here at line 3641
|
||||
12. Build reasoning state
|
||||
13. Run deterministic decomposition
|
||||
14. Propagate resolved child evidence
|
||||
15. Post-propagation candidate assessment
|
||||
16. Model-selection honour path (line 3972)
|
||||
17. Deterministic fallback selection (line 4008)
|
||||
18. Set final selectedQuestion from deterministicSelection
|
||||
|
||||
## Root cause
|
||||
|
||||
### Two independent gaps in the selectable-node predicate chain:
|
||||
|
||||
**Gap 1 — validateSelectedQuestion (pre-mutation)** at line 246:
|
||||
```javascript
|
||||
if (resolvesNode || effectiveStatus === "resolved") {
|
||||
```
|
||||
This rejects `selectedQuestion.nodeId` when the node is explicitly in `resolvedUnknownNodeIds` OR when its new status via `updatedNodes` is `"resolved"`. But it does NOT check for `effectiveStatus === "known"`.
|
||||
|
||||
A node transitioned to `status=known` via `updatedNodes` passes this validation silently.
|
||||
|
||||
**Gap 2 — isSelectableUnresolvedUnknown (post-mutation)** at line 1663:
|
||||
```javascript
|
||||
!["resolved", "contradicted"].includes(node.status)
|
||||
```
|
||||
This predicate is used throughout the pipeline to determine whether a node can be selected as the next question target. It correctly excludes `"resolved"` and `"contradicted"` but does NOT exclude `"known"`.
|
||||
|
||||
Since `kind` stays `"unknown"` while `status` changes to `"known"`, the predicate returns true for known-status nodes that should not be selectable.
|
||||
|
||||
This gap propagates through:
|
||||
- `isSelectableUnresolvedUnknown` (used at lines 2406, 2434, 2466, 3764, 3984, 3764)
|
||||
- `selectActiveUnknownCandidate` in utils.js (used at line 3722, used as fallback selector)
|
||||
- `listUnresolvedUnknownCandidates` / `listEligibleUnknownCandidates`
|
||||
- `remainingUnknownExists` check at line 3705
|
||||
|
||||
## The exact path in 60B.37
|
||||
|
||||
1. **Model proposes**: `updatedNodes[n_product_launch_decision] = { newStatus: "known" }`, `selectedQuestion.nodeId = "n_product_launch_decision"`
|
||||
2. **validateSelectedQuestion** (pre-mutation): effectiveStatus = "known" → line 246 check fails (only catches "resolved") → NO ERROR
|
||||
3. **applyGraphUpdate** (line 3641): n_product_launch_decision gets status=known in the updated graph
|
||||
4. **Lines 3705-3712 remainingUnknownExists**: kind=unknown ✓, not in resolvedNodeIds ✓ → returns true → no reselection triggered
|
||||
5. **Line 3722 selectActiveUnknownCandidate**: filters by kind=unknown + not in resolvedNodeIds. n_product_launch_decision passes (kind=unknown, NOT in resolvedUnknownNodeIds). Returns { nodeId: "n_product_launch_decision", status: "selected" }
|
||||
6. **Lines 3783-3792 preservation check**: isSelectableUnresolvedUnknown returns true for known-status node → preservedSelectedChildNode set to decision node
|
||||
7. **Line 4055 finalSelectedQuestion**: built from deterministicSelection.nodeId = "n_product_launch_decision" (status=known)
|
||||
8. **Result**: A node with status=known receives a follow-up question despite decision-level closure being complete
|
||||
|
||||
## Asymmetry between resolution paths
|
||||
|
||||
**resolvedUnknownNodeIds exclusion:** YES — nodes in this array are checked at line 238 and excluded by resolvedNodeIds throughout the pipeline.
|
||||
|
||||
**updated-to-known exclusion:** NO — no function in the entire chain checks `status !== "known"` as a filter condition. `"known"` is not in any exclusion list.
|
||||
|
||||
**Asymmetry exists:** YES
|
||||
|
||||
The path via `resolvedUnknownNodeIds` (explicit resolution) is fully guarded. The path via `updatedNodes[n].newStatus = "known"` (implicit resolution) is NOT guarded because:
|
||||
- validateSelectedQuestion only catches "resolved" status, not "known"
|
||||
- isSelectableUnresolvedUnknown only excludes ["resolved", "contradicted"], not "known"
|
||||
- selectActiveUnknownCandidate has no status check at all
|
||||
|
||||
## Active unknown lifecycle for this case
|
||||
|
||||
```
|
||||
Pre-update active node: n_enterprise_customer_signing
|
||||
Post-mutation active node before reselection: null (cleared at line 3698 because previous was resolved)
|
||||
Final active node: n_product_launch_decision (set at line 3702 from proposal.selectedQuestion, then NOT re-evaluated for status validity)
|
||||
```
|
||||
|
||||
A known-status unknown can remain `activeUnknownNodeId` because `remainingUnknownExists` only checks kind and resolvedNodeIds.
|
||||
|
||||
## Cause assessment
|
||||
|
||||
### Candidate A — EARLY VALIDATION / LATE MUTATION
|
||||
|
||||
**Evidence:** MEDIUM-HIGH
|
||||
- validateSelectedQuestion is called at line 3594 (before mutation at line 3641)
|
||||
- But the gap is not about timing — even a post-mutation check would miss "known" because the predicate doesn't filter it
|
||||
- The validation exists but has an incomplete status filter
|
||||
|
||||
**Explains 60B.37:** PARTIAL — captures the pre-mutation aspect but not the status filtering gap
|
||||
|
||||
### Candidate B — `resolvedUnknownNodeIds`-ONLY FILTER
|
||||
|
||||
**Evidence:** HIGH
|
||||
- Every predicate in the pipeline (`isSelectableUnresolvedUnknown`, `selectActiveUnknownCandidate`, `listUnresolvedUnknownCandidates`) that should exclude resolved nodes only checks:
|
||||
- kind === "unknown" (always true for unknown-type nodes)
|
||||
- not in resolvedNodeIds/resolvedUnknownNodeIds
|
||||
- None check status against the full set of terminal statuses ["resolved", "known", "contradicted"]
|
||||
|
||||
**Explains 60B.37:** YES — this is the precise mechanism. The decision node transitions via updatedNodes.newStatus="known" rather than resolvedUnknownNodeIds, and no predicate catches the gap.
|
||||
|
||||
### Candidate C — PREFERRED-TARGET PATH BYPASSES NORMAL SELECTABILITY
|
||||
|
||||
**Evidence:** MEDIUM
|
||||
- Model-selected target at line 3976 has an explicit isSelectableUnresolvedUnknown check (line 3984)
|
||||
- This check would pass for known-status nodes due to the predicate gap
|
||||
- However, in 60B.37 the decision node was NOT newly added, so this path doesn't apply
|
||||
- The model-selection honour path correctly skips it
|
||||
|
||||
**Explains 60B.37:** PARTIAL — the gap exists but the specific path is blocked by the "newly added" check
|
||||
|
||||
### Candidate D — ACTIVE NODE LIFECYCLE STALE
|
||||
|
||||
**Evidence:** MEDIUM
|
||||
- remainingUnknownExists at line 3705 doesn't check status
|
||||
- But in 60B.37, n_product_launch_decision becomes active via line 3702 (from proposal.selectedQuestion), not from remainingUnknownExists
|
||||
- The real issue is the target selection path, not active node management per se
|
||||
|
||||
**Explains 60B.37:** PARTIAL — contributes to the stale state but isn't the root cause
|
||||
|
||||
## Critical distinction
|
||||
|
||||
**Choice: B — QUESTION TARGET VALIDATION IS WRONG**
|
||||
|
||||
Why: The graph resolution itself was observed as correct in 60B.37 (n_product_launch_decision correctly became status=known with correct rationale). The failure is specifically at the question-target validation layer: multiple predicates filter terminal statuses but collectively miss "known". This is a validation predicate gap, not a resolution state error or active node lifecycle issue.
|
||||
|
||||
## Minimum corrective boundary
|
||||
|
||||
**Choice: C — UNIFY ALL FINAL TARGETS THROUGH ONE SELECTABILITY PREDICATE**
|
||||
|
||||
Why: The fix requires making `isSelectableUnresolvedUnknown` correctly exclude `status === "known"` nodes AND ensuring `validateSelectedQuestion` checks effective status against all terminal states including "known". This ensures whether the target comes from model preference, active node persistence, or deterministic selector, it passes one canonical post-mutation unresolved/selectable check.
|
||||
|
||||
Would preserve valid unresolved preferred targets: YES — only known/resolved/contradicted nodes are excluded
|
||||
Would preserve prerequisite-first fallback: YES — unaffected by status filtering changes
|
||||
Would prevent known decision nodes from receiving final questions: YES — all selection paths would use the corrected predicate
|
||||
|
||||
## Implementation readiness
|
||||
|
||||
**A — READY FOR BOUNDED IMPLEMENTATION**
|
||||
|
||||
The diagnosis is complete. The exact code paths and predicates are identified. The fix is a single-predicate correction to `isSelectableUnresolvedUnknown` and one status check addition in `validateSelectedQuestion`.
|
||||
|
||||
Smallest implementation boundary: Two changes — (1) add "known" to the exclusion list in `isSelectableUnresolvedUnknown`, (2) add `effectiveStatus === "known"` check in `validateSelectedQuestion` at line 246.
|
||||
Reference in New Issue
Block a user