robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: diagnose stale question after resolution

Browse Source
This commit is contained in:
robbond
2026-08-14 08:21:22 +01:00
parent 1331fe94f1
commit 88a80180b7
2 changed files with 225 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+8
View File
@@ -2953,3 +2953,11 @@ Experiment 60B.34 was a live regression on `feature/bare-whether-proposition-v0.
---
Experiment 60B.35 validated that a bare `Whether...` description survives the **full** applyValidatedProposal runtime path end-to-end. **Classification: A — FULL LIFECYCLE CONFIRMED.** The model-produced node `n_enterprise_customer_signing` (kind=unknown, status=unknown) with a depends_on edge reached the full production apply-proposal pipeline without proposition degradation. Final selected question was `"What evidence would clarify whether the prospective enterprise customer will commit this year?"` — proposition-specific evidence framing confirmed at the output boundary. No rationale contamination, decision identity preserved, no unrelated uncertainty invented. This closes the verification gap from 60B.34: bare Whether... preservation now proven end-to-end in both partial and full runtime paths. One live call. No production code changed. Deterministic test added (removed in cleanup commit).
---
Experiment 60B.37 ran the customer-signing followup fixture through a single bounded update to test whether resolving the last material uncertainty of an existing unresolved decision updates that same factor in place and closes the decision cleanly. **Classification: C — FACTOR RESOLVES BUT GENERIC CONTINUATION REMAINS.** The enterprise-customer unknown (`n_enterprise_customer_signing`) resolved correctly in place (unknown → resolved) with no duplication. The product-launch decision (`n_product_launch_decision`) was correctly transitioned to status=known with rationale "Prerequisite uncertainty resolved and user confirms no other material uncertainties remain between the options." Both option identities preserved. No new nodes or edges created. **However**, the engine still produced a `selectedQuestion` ("What outcome would demonstrate enough value to justify launching?") targeting `n_product_launch_decision`, which has status=known. This is a structural inconsistency: decision-level closure at the reasoning level but continuation at the orchestration level. One live call. No production code changed. Status pending review.
---
Experiment 60B.38 was a read-only diagnosis of why `n_product_launch_decision` becomes known in the same update yet still survives as the final selectedQuestion target. **Classification: B — QUESTION TARGET VALIDATION IS WRONG.** Two independent gaps discovered in the selectable-node predicate chain: (1) `validateSelectedQuestion` at lib/graph/apply-proposal.js:246 checks `effectiveStatus === "resolved"` only, NOT `"known"`; (2) `isSelectableUnresolvedUnknown` at lib/graph/apply-proposal.js:1663 excludes ["resolved", "contradicted"] but NOT "known". These gaps propagate through the entire pipeline: `selectActiveUnknownCandidate` in utils.js has no status filter at all. The decision node transitioned via `updatedNodes.newStatus="known"` rather than `resolvedUnknownNodeIds`, and no predicate catches this gap. **Minimum corrective boundary:** C — unify all final targets through one canonical post-mutation unresolved/selectable check. Add "known" to the exclusion list in `isSelectableUnresolvedUnknown` and add `effectiveStatus === "known"` check in `validateSelectedQuestion`. No production code changed. 0 Ollama calls. Pure code inspection. Full trace in docs/experiment-60b38.md.