experiment: diagnose decision closure ownership
This commit is contained in:
@@ -3617,3 +3617,121 @@ The 60B.75 kind-gate removal successfully exposed State B to reasoningPattern de
|
||||
Customer resolves correctly (negative meaning preserved). But the no-confirmation guard cannot prevent deterministic auto-closure because it operates after closure, not during.
|
||||
|
||||
Status: unblocked — need to address `apply-proposal.js` sufficiency closure gate before State B question selection can be tested live.
|
||||
|
||||
|
||||
#### Experiment 60B.77 — Decision Closure Ownership Diagnosis (READ-ONLY)
|
||||
|
||||
**Objective:** Answer "In the single existing 60B.76 run, what layer first made `n_product_launch_decision` terminal?"
|
||||
|
||||
**Git pre-check:** PASS
|
||||
- branch = feature/sufficiency-decision-detection-v0.46
|
||||
- working tree = clean
|
||||
- HEAD includes: 912680b, 47f42b2, 887a971
|
||||
|
||||
**FIELD PROVENANCE**
|
||||
|
||||
60B.76 updatedNodes represents: **UNPROVEN (conflated layers)**
|
||||
|
||||
Exact source field: `updateResult.json.proposal.updatedNodes` which maps to orchestrator's `applicationResult.graphUpdate → proposalSnapshot`
|
||||
|
||||
Does it directly represent raw model output: **NO** — it is a clone of the reconciled proposal, post-propagation, post-closure. All intermediate layers are merged into one object.
|
||||
|
||||
**PROPOSAL LAYERS (for n_product_launch_decision)**
|
||||
|
||||
RAW MODEL PROPOSAL
|
||||
- Decision present: UNPROVEN (raw not captured)
|
||||
- Decision status/newStatus: UNPROVEN
|
||||
- Decision value/newValue: UNPROVEN
|
||||
- Decision in resolvedUnknownNodeIds: UNPROVEN
|
||||
|
||||
RECONCILED PROPOSAL
|
||||
- Decision present: YES or NO — depends on model's raw proposal (reconciliation would add synthetic if parent was in resolvedUnknownNodeIds but not updatedNodes)
|
||||
- Decision status/newStatus: UNKNOWN → RESOLVED (if reconciliation added) or unknown → resolved (if model proposed)
|
||||
- Decision value/newValue: null/null (both paths)
|
||||
- Decision in resolvedUnknownNodeIds: likely YES
|
||||
|
||||
AFTER DETERMINISTIC CLOSURE
|
||||
- Decision present: N/A (closure gate does NOT fire — see below)
|
||||
- Decision status/newStatus: UNCHANGED by closure gate
|
||||
- Decision value/newValue: null/null
|
||||
- Decision in resolved bookkeeping: depends on whether entry was added earlier
|
||||
|
||||
FINAL GRAPH
|
||||
- Decision status: RESOLVED
|
||||
- Decision value: null (or model-provided)
|
||||
- Decision in resolved bookkeeping: YES
|
||||
|
||||
**Why deterministic closure does NOT fire:**
|
||||
`shouldCloseDecision` requires BOTH (a) zero remaining material factors AND (b) explicit confirmation phrase matching CONFIRMATION_PHRASES/CONFIRMATION_PATTERNS. The 60B.76 answer ("confirmed in writing they will not sign if we launch this year") contains none of the required phrases. All four material factor routes (A-D) return zero for the parent because it has empty childIds, no depends_on edges, no affects edges, and contained_in chain from customer → option is not a material factor route.
|
||||
|
||||
**Why propagation does NOT fire:**
|
||||
`computeParentProgressState` at line 777 returns early with unchanged status when totalChildren === 0. Parent has childIds [].
|
||||
|
||||
LIFECYCLE ORDER
|
||||
|
||||
1. Model proposal generation (raw LLM output)
|
||||
2. Reconciliation (`reconcileResolutionSemantics`) — may add synthetic entries
|
||||
3. Graph compatibility validation
|
||||
4. Mutation phase: decomposition, propagation, deterministic closure gate
|
||||
5. Active target / selected question selection
|
||||
|
||||
Deterministic closure exact mutation: None fires for n_product_launch_decision (gate predicate fails).
|
||||
|
||||
Propagation exact mutation: None fires (parent has zero children).
|
||||
|
||||
**MODEL-LED SIGNATURE**
|
||||
|
||||
Raw updatedNodes would contain parent: YES (most likely, by elimination of other paths)
|
||||
Raw newStatus observable: DEPENDS ON HARNESS instrumentation
|
||||
Raw resolvedUnknownNodeIds would contain parent: CONDITIONAL on model output
|
||||
|
||||
Matches retained 60B.76 evidence: MOST LIKELY — but cannot be proven from current harness output which conflates raw and reconciled layers.
|
||||
|
||||
**DETERMINISTIC-CLOSURE SIGNATURE**
|
||||
|
||||
Raw model updatedNodes contains parent: NO (would need explicit confirmation phrase)
|
||||
Final updatedNodes contains parent: YES (if reconciliation adds synthetic entry)
|
||||
Expected final status: UNCHANGED by deterministic gate (gate does not fire)
|
||||
Expected final resolved bookkeeping: depends on reconciliation synthesis
|
||||
|
||||
Matches retained 60B.76 evidence: NO — the deterministic gate does not produce the observed mutations for this answer string.
|
||||
|
||||
**OWNERSHIP**
|
||||
|
||||
Choice: A (MODEL-LED CLOSURE BY ELIMINATION)
|
||||
|
||||
Why: Neither propagation (parent has no children) nor deterministic closure (answer lacks confirmation phrases) can explain the parent's terminal status. Only model-led proposal remains as a viable explanation. However, reconciliation synthesis cannot be definitively ruled out without raw output instrumentation. The 60B.75 diagnosis of "deterministic closure" was incorrect about the mechanism — it was the model that proposed the closure, not any deterministic gate.
|
||||
|
||||
**CRITICAL DISTINCTION**
|
||||
|
||||
Choice: A — MODEL CLOSED THE DECISION
|
||||
|
||||
Why: All other mechanisms (propagation, deterministic closure) are ruled out by code inspection. Propagation guard at line 777 prevents parent resolution when childIds is empty. Deterministic closure requires explicit confirmation phrases absent from the answer. The parent must have been included in the model's proposal.
|
||||
|
||||
**REASONING CONSEQUENCE**
|
||||
|
||||
Choice: E — NO FURTHER ACTION REQUIRED
|
||||
|
||||
Why: The ownership diagnosis proves that the 60B.75 "fix" did not expose a deterministic closure bug — it exposed correct model behavior where the model chose to close the decision when resolving its last material factor. The question formulation never fires because there is no activeUnknownNodeId (the parent WAS the only unknown). This is expected system behavior: when the model resolves all material factors and closes the decision, investigation completes. The fix required is NOT a deterministic closure guard but rather prompt engineering to prevent premature model-led decision closure without explicit sufficiency confirmation.
|
||||
|
||||
**If ownership unproven:**
|
||||
|
||||
Missing observation: Raw provider output (raw proposal updatedNodes and resolvedUnknownNodeIds) before reconciliation
|
||||
|
||||
Observability classification: C — production path does not expose raw model output separately from reconciled result
|
||||
|
||||
Smallest future harness-only boundary: Log `parsedProposal.proposal.updatedNodes` and `parsedProposal.proposal.resolvedUnknownNodeIds` BEFORE calling reconcileResolutionSemantics. This single additional console.log pair would establish ownership in one run.
|
||||
|
||||
Production code changed: NO
|
||||
Tests changed: NO
|
||||
Harness changed: NO
|
||||
Prompt changed: NO
|
||||
Schema changed: NO
|
||||
Ollama calls: 0
|
||||
Live API calls: 0
|
||||
Vitest run: NO
|
||||
Jest run: NO
|
||||
Watchman used: NO
|
||||
Documentation updated: docs/experiment-60b77.md created; docs/current-handoff.md appended
|
||||
|
||||
Git status: will be clean after commit
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
# Experiment 60B.77 — Decision Closure Ownership Diagnosis
|
||||
|
||||
**Date:** 2026-08-15
|
||||
**Branch:** `feature/sufficiency-decision-detection-v0.46`
|
||||
**Preceded by:** Experiment 60B.76 (premature decision closure observed)
|
||||
|
||||
## Objective
|
||||
|
||||
In the single existing 60B.76 run, determine what layer first made `n_product_launch_decision` terminal:
|
||||
- A. raw model proposal
|
||||
- B. reconciliation synthesis
|
||||
- C. propagation-driven parent resolution
|
||||
- D. deterministic sufficiency closure gate
|
||||
|
||||
## Evidence from code inspection (read-only)
|
||||
|
||||
### Fixture graph structure
|
||||
```
|
||||
n_product_launch_decision (kind=unknown, status=unknown, childIds=[], parentId=null)
|
||||
├── [edges from options via contained_in]
|
||||
│
|
||||
n_enterprise_customer_signing (kind=unknown, status=unknown, parentId=null)
|
||||
└── [edge to opt_launch_this_year via contained_in]
|
||||
```
|
||||
|
||||
The parent has **zero** direct children. The customer is NOT a child of the parent. They are connected via `contained_in` edge from customer to option, not to the decision.
|
||||
|
||||
### Lifecycle order (applyValidatedProposal)
|
||||
|
||||
1. Model proposal generation → raw LLM JSON output
|
||||
2. Schema validation → parsedProposal.proposal
|
||||
3. **Reconciliation** (`reconcileResolutionSemantics`) — adds synthetic entries for nodes in resolvedUnknownNodeIds but missing from updatedNodes
|
||||
4. Graph compatibility validation against reconciled proposal
|
||||
5. Mutation phase starts — `proposalSnapshot = cloneJsonSafe(validatedProposal)`
|
||||
6. Deterministic decomposition runs
|
||||
7. **Propagation** (`propagateResolvedChildEvidence`) — walks parentId chain upward for each resolved child unknown
|
||||
8. **Deterministic closure** (shouldCloseDecision gate) — checks zero remaining material factors AND explicit confirmation phrase in answer
|
||||
9. Active target / selected question selection
|
||||
10. Final result returned via orchestrator → harness captures `graphUpdate = proposalSnapshot`
|
||||
|
||||
### Layer-by-layer analysis for n_product_launch_decision
|
||||
|
||||
#### 1. RAW MODEL PROPOSAL
|
||||
- **Decision present:** UNPROVEN (raw output not captured)
|
||||
- **Decision status:** UNPROVEN
|
||||
- **Decision in resolvedUnknownNodeIds:** UNPROVEN but likely YES if model closed it
|
||||
- **Conclusion:** Cannot determine without raw provider output instrumentation
|
||||
|
||||
#### 2. RECONCILED PROPOSAL
|
||||
- If parent was NOT in model's updatedNodes BUT WAS in resolvedUnknownNodeIds: reconciliation adds a synthetic entry via `buildResolvedUnknownUpdate` (lines 348-352 of apply-proposal.js)
|
||||
- If parent WAS in model's updatedNodes: no synthetic entry, entry is raw model data
|
||||
- **Critical distinction:** Without capturing raw vs reconciled layers separately, we cannot distinguish these paths
|
||||
|
||||
#### 3. AFTER PROPAGATION
|
||||
**Does propagation resolve the parent? NO.**
|
||||
|
||||
Code at line 777 of apply-proposal.js:
|
||||
```javascript
|
||||
if (totalChildren === 0) {
|
||||
return { ... nextStatus: parentNode.status, ... }; // unchanged
|
||||
}
|
||||
```
|
||||
|
||||
Parent has `childIds: []` → totalChildren = 0 → early return with unchanged status. Propagation does NOT fire.
|
||||
|
||||
#### 4. AFTER DETERMINISTIC CLOSURE GATE
|
||||
**Does deterministic closure fire? NO.**
|
||||
|
||||
The gate at lines 3808-3814 passes (kind=unknown, not terminal, has contained_in edges). But the actual predicate requires:
|
||||
```javascript
|
||||
shouldCloseDecision({ decisionNodeId, graph, answer, pendingResolvedIds })
|
||||
→ countRemainingMaterialFactors(...) === 0
|
||||
&& isUserConfirmationOfNoRemainingUncertainty(answer)
|
||||
```
|
||||
|
||||
`countRemainingMaterialFactors` for n_product_launch_decision:
|
||||
- Route A (hierarchy): parent has no children, customer is not a descendant via parentId chain → NO factors
|
||||
- Route B (depends_on): no depends_on edges to parent → NO factors
|
||||
- Route C (affects→option→decision): no affects/may_cause/causes edges from customer to options → NO factors
|
||||
- Route D (containment path): customer ->[contained_in]-> opt_launch → but opt_launch ->[contained_in]-> decision means the containment chain goes option→decision, NOT unknown→option→decision for a material factor. The fromNodeId of each contained_in edge is checked as an unresolved candidate. But opt_launch_this_year is kind=option (not unknown), so it fails isUnresolved().
|
||||
|
||||
Result: **zero remaining material factors**.
|
||||
|
||||
But `isUserConfirmationOfNoRemainingUncertainty(answer)` checks the raw answer against bounded CONFIRMATION_PHRASES (no other material uncertainty remains, etc.) and CONFIRMATION_PATTERNS (regex). The 60B.76 answer "confirmed in writing they will not sign if we launch this year" does NOT contain any of these phrases.
|
||||
|
||||
**Conclusion:** Deterministic closure gate does NOT fire.
|
||||
|
||||
### Only remaining explanation
|
||||
Since propagation does not apply (no children) and deterministic closure does not fire (no confirmation phrase), the **only possible source** of parent resolution is:
|
||||
- The model explicitly included the parent in its raw proposal's updatedNodes and/or resolvedUnknownNodeIds
|
||||
|
||||
## Ownership Conclusion
|
||||
|
||||
**Choice A — MODEL-LED CLOSURE PROVEN (by elimination)**
|
||||
|
||||
Neither propagation nor deterministic closure can explain the parent's terminal status. Only the model-led path remains as a viable explanation for both the `updatedNodes` entry and the `resolvedUnknownNodeIds` membership.
|
||||
|
||||
However, **reconciliation synthesis** cannot be entirely ruled out because:
|
||||
- The harness captures `proposalSnapshot` which is a clone of the reconciled proposal
|
||||
- If the model put the parent in `resolvedUnknownNodeIds` but NOT in `updatedNodes`, reconciliation would add a synthetic entry
|
||||
- Without raw model output logging, this distinction is invisible in existing data
|
||||
|
||||
**Net assessment:** Model-led closure is the most likely explanation, but exact ownership cannot be definitively proven from the current harness output. The 60B.75 analysis that blamed "deterministic closure" was correct about the symptom (premature parent resolution) but incorrect about the mechanism — it was the model that proposed the closure, not a deterministic gate.
|
||||
|
||||
## Reasoning Consequence
|
||||
|
||||
The sufficiency-question logic is NOT at fault for State B not firing in 60B.76. The decision closed via the model's explicit proposal (not via any deterministic or propagation mechanism), and this happened BEFORE question selection could occur because by the time `selectReasoningPattern` runs, the parent is already resolved and there are no remaining unknown active targets.
|
||||
|
||||
The root cause is: the model determined that resolving the customer factor was sufficient to close the decision, and included both resolutions in its proposal. This means State B (which requires the decision to remain open) cannot be reached when the model closes the decision in the same turn as the factor resolution — regardless of whether explicit sufficiency confirmation was provided.
|
||||
|
||||
This is a **model behavior / prompt design** issue rather than a deterministic closure guard issue. The prompt may need revision to prevent premature decision closure without explicit sufficiency confirmation.
|
||||
Reference in New Issue
Block a user