From 014c6b72dc25f536b7e827b6e9466049c728f299 Mon Sep 17 00:00:00 2001 From: robbond Date: Fri, 14 Aug 2026 12:10:25 +0100 Subject: [PATCH] experiment: define decision sufficiency evidence --- docs/current-handoff.md | 2 + docs/experiment-60b58.md | 378 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 380 insertions(+) create mode 100644 docs/experiment-60b58.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 0954291..1e4dbf5 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -3018,3 +3018,5 @@ Experiment 60B.48 was a read-only code-path and contract diagnosis of why 60B.47 --- Experiment 60B.56 was a live regression of negative customer-signing closure after the proposal-compatibility fix (54e2e21). **Classification: D — GRAPH CLOSES BUT CONVERSATION DOES NOT.** The enterprise-customer unknown resolved in place (unknown → resolved), negative meaning preserved ("customer will not sign" confirmed in resolution reason), zero addedNodes/edges, zero validation errors. However, `finalActiveUnknownNodeId = "n_product_launch_decision"` (non-null) and `finalSelectedQuestion` was non-null — the engine formulated a decision_threshold follow-up question ("What outcome would demonstrate enough value to justify launching?") targeting the parent decision node instead of closing it. The proposal-compatibility bug from 60B.47 is fixed (proposal accepted, HTTP 200), but the clean-closure contract (null → null when all dependencies resolve) remains unmet. `n_product_launch_decision` stays unknown despite having no remaining dependent unknowns — both options are known. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed. Full trace in docs/experiment-60b56.md. + +Experiment 60B.58 was a read-only decision-sufficiency evidence map. **Classification: E — DUAL-SIGNAL CLOSURE REQUIRED.** Analysis of the existing topology confirmed that no single signal suffices for correct closure detection. The winning model is Candidate E: relationship-aware graph traversal PLUS user declaration, both required independently. Key findings: (1) Existing architecture captures factor→option edges but does not traverse them during closure propagation — only parentId/depends_on children are counted; (2) Self-counting of the decision node as its own unresolved child is inherent in `selectActiveUnknownCandidate` because it filters only kind+resolvedNodeIds, never status alone beyond terminal exclusion; (3) Extending the predicate to include `affects → option contained_in decision` traversal fixes both premature-closure and missed-closure without schema changes. Implementation readiness: B — one unresolved question remains: which exact relationship types (affects, may_cause, causes) qualify as "decision-relevant" for generic factors beyond those already defined by 60B.15. No production code changed. 0 Ollama calls. Pure code inspection + fixture analysis. Full trace in docs/experiment-60b58.md. diff --git a/docs/experiment-60b58.md b/docs/experiment-60b58.md new file mode 100644 index 0000000..1875619 --- /dev/null +++ b/docs/experiment-60b58.md @@ -0,0 +1,378 @@ +# Experiment 60B.58 — Decision-Sufficiency Evidence Map + +**Date:** 2026-08-14 +**Branch:** `feature/closure-selection-reconciliation-v0.41` +**Head commit:** 2394ad4 experiment: confirm negative closure live + +## Objective + +Answer exactly: what graph evidence already exists in the current architecture that can distinguish "this decision still has a material unresolved factor" from "all represented material uncertainty has been resolved", without relying on option status alone? + +No implementation. Read-only analysis of existing topology, code, and fixture. + +## Method + +Analyzed: +1. Fixture `tests/fixtures/pre-anchored-product-launch-customer-signing.json` +2. `lib/graph/apply-proposal.js` — functions: `findDirectChildUnknowns`, `computeParentProgressState`, `propagateResolvedChildEvidence`, `listUnresolvedUnknownCandidates`, `selectActiveUnknownCandidate`, `scoreUnknownCandidate`, `evaluateBranchInteractions`, `syncParentChildReferences`, `buildAncestorChain` +3. `lib/graph/utils.js` — functions: `findAffectedNodes`, `findDependentNodes`, `scoreUnknownCandidate`, `countIncomingUnknownDependencies`, `selectActiveUnknownCandidate`, `explainUnknownSelection` + +--- + +## Checkpoint 1 — Decision-Factor Linkage Map + +For `n_enterprise_customer_signing` in the fixture, here are every structural relationship linking it to the decision and its options: + +### Relationship: `n_enterprise_customer_signing -> opt_launch_this_year (edge)` + +- **Edge ID:** `e-customer-signing-to-launch-option` +- **From:** `n_enterprise_customer_signing` (unknown) +- **To:** `opt_launch_this_year` (option) +- **Relationship type:** `contained_in` +- **Direction:** unknown → option (upstream dependency flow) +- **Semantic role:** OPTION CONSEQUENCE — the unknown is a condition that affects/attaches to this specific option +- **Currently used by closure propagation:** **NO** + +### Relationship: `n_enterprise_customer_signing -> n_product_launch_decision` + +- **Direct edge exists?** NO +- **Direct parentId relationship?** NO (both have `parentId: null`) +- **Direct depends_on relationship?** NO +- **Direct affects relationship?** NO +- **Semantic role:** NONE — structurally disconnected at the decision level +- **Currently used by closure propagation:** **NO** + +### Relationship: `opt_launch_this_year -> n_product_launch_decision (edge)` + +- **Edge ID:** `e-opt-launch-to-dec` +- **Relationship type:** `contained_in` +- **Direction:** option → decision (candidate-for) +- **Semantic role:** CONTAINMENT — option is a candidate for this decision + +### Relationship: `opt_wait_twelve_months -> n_product_launch_decision (edge)` + +- **Edge ID:** `e-opt-wait-to-dec` +- **Relationship type:** `contained_in` +- **Direction:** option → decision (candidate-for) +- **Semantic role:** CONTAINMENT — option is a candidate for this decision + +### Summary of structural attribution: + +``` +n_product_launch_decision has no direct unknown child. + childIds: [] + dependsOn: [] + parentId: null + +n_enterprise_customer_signing: + parentId: null + dependsOn: [] + affects: [] + +opt_launch_this_year is contained_in n_product_launch_decision. +n_enterprise_customer_signing has an edge to opt_launch_this_year labeled "contained_in". + +No propagation path exists from n_enterprise_customer_signing to n_product_launch_decision +through any of: parentId, childIds, depends_on, affects, may_cause, causes, contained_in. +``` + +**Factor structurally attributable to the decision today:** PARTIAL + + +--- + +## Checkpoint 2 — Material Unresolved Factor Detection via Existing Topology + +### parentId / childIds + +**Classification: INSUFFICIENT for the current case.** + +`findDirectChildUnknowns()` (apply-proposal.js:671) finds children where `node.parentId === parentNodeId OR edge.fromNodeId -> parentNodeId with relationship=depends_on`. In the fixture, no unknown has `parentId` set to the decision. The factor has `parentId: null`. Only decomposition-created unknowns get parentId populated (via `buildDecompositionContext` → `buildDecompositionTemplates`). + +For generic decision-factor relationships created outside decomposition, parentId is not set. The function also scans edges with `depends_on` from unknown-to-decision, which would catch factor→decision dependencies IF the LLM creates them — but the fixture has no such edge on the factor node. + +### depends_on + +**Classification: CONTEXTUAL.** + +The decision node itself has `dependsOn: []`. The factor node has `dependsOn: []`. Neither unknown has a `depends_on` edge between them in the fixture. If an LLM-created edge connected `n_enterprise_customer_signing -> n_product_launch_decision` with relationship=`depends_on`, the existing `findDirectChildUnknowns` path would catch it (edge scanning at line 678). But this is not present in the fixture. + +### affects + +**Classification: INSUFFICIENT.** + +The factor's `affects: []` is empty. No edge originates from the factor with relationship pointing to any decision option beyond the `contained_in` edge to `opt_launch_this_year`. The existing code does not use `affects` for closure propagation — it uses it only for `findAffectedNodes` (impact scanning, not dependency tracking). + +### may_cause / causes + +**Classification: UNUSED BY CURRENT CLOSURE.** + +These appear in `STRUCTURAL_CONSEQUENCE_RELATIONSHIPS` at line 1911 of apply-proposal.js but only within the Route B structural context admission check for reasoning-pattern compatibility during new-unknown selection. They are never used in `propagateResolvedChildEvidence`, `computeParentProgressState`, or any closure-determining path. + +### contained_in + +**Classification: INSUFFICIENT.** + +The factor has a `contained_in` edge to `opt_launch_this_year`. The options have `contained_in` edges to the decision. However, "contained_in" semantics mean "is-a-candidate-for" in this architecture — it flows from option→decision for containment of candidates. The reverse flow (unknown→option via contained_in) is not interpreted as a dependency. No code traverses `contained_in` edges in either direction for closure propagation. + +### direct decision -> unknown edge + +**Classification: UNUSED BY CURRENT CLOSURE.** + +No such edges exist in the fixture and none are created by production code for generic factor relationships. Only decomposition children receive `depends_on` edges to their parent (see line 1637 in apply-proposal.js). + +--- + +## Checkpoint 3 — Remaining-Factor Query + +``` +Possible with current schema: PARTIAL +Requires new schema: NO +Existing helper already does this: PARTIAL +Closest existing helper: findDirectChildUnknowns (only catches parentId/depends_on children) + propagateResolvedChildEvidence (only propagates from known children) +``` + +**Narrowest deterministic predicate derivable from current code:** + +> An unresolved unknown counts against a decision's sufficiency when it either (a) has `parentId` set to the decision node, or (b) has a `depends_on` edge pointing to the decision node, or (c) is a new unknown admitted during the same turn through Route A/B structural context embedding. + +This predicate is **too narrow** for the customer-signing case: the factor is linked via option-attachment (`contained_in` → opt), not parentId or depends_on. No existing function traverses option→decision containment edges to find unknowns that attach to any contained option. + + +--- + +## Checkpoint 4 — Self-Counting Problem + +### Parent decision appears in generic unresolved list: YES + +`selectActiveUnknownCandidate()` (utils.js:593) filters `graph.nodes` for `kind=unknown AND status not in [known, resolved, contradicted] AND id not in resolvedNodeIds`. `n_product_launch_decision` has `status=unknown`, is not in `resolvedNodeIds`, so it IS included. + +### Parent can self-count as remaining unresolved: YES + +Because the decision node itself is an unknown with status=unknown, a generic unresolved list will always contain it unless explicitly filtered. After all subordinate factors resolve, the decision node remains in the list — creating exactly the self-counting problem. The system cannot distinguish "the decision itself hasn't been concluded" from "evidence for the decision is incomplete." + +### Current distinction between decision and factor: PARTIAL + +`computeParentProgressState()` (apply-proposal.js:744) distinguishes parent from children by examining `findDirectChildUnknowns(graph, parentNode.id)`. But this only works when unknowns have parentId/depends_on links to the parent. When a factor is structurally disconnected (as in the fixture), no child-unknown path exists — so there is zero distinction between "parent awaiting conclusion" and "factor beneath parent unresolved." + +`propagateResolvedChildEvidence()` at line 894 filters for `node.parentId` on resolved children. If parentId is null, nothing propagates upward. The decision node never gets marked "resolved by propagation" when no direct child exists. + +--- + +## Checkpoint 5 — Candidate Assessment + +### Candidate A — CHILD UNKNOWN COMPLETION ONLY + +Close decision only when all direct `parentId` child unknowns resolve. + +- **Architecture fit:** HIGH — uses existing `propagateResolvedChildEvidence` and `computeParentProgressState` +- **Fixes 60B.56:** NO — the factor has no parentId to the decision, so completion is never triggered +- **Premature-closure risk:** LOW — requires explicit decomposition relationship +- **Depends on model compliance:** HIGH — only works if LLM always creates parentId links +- **Requires schema change:** YES (for non-decomposition factors) or NO (if we extend parentId semantics) +- **Principal weakness:** Cannot capture generic decision-factor relationships created outside decomposition + +### Candidate B — RELATIONSHIP-AWARE MATERIAL FACTORS + +Close decision when no unresolved decision-relevant unknown remains across approved structural relationships. + +- **Architecture fit:** MEDIUM — requires adding traversal of option-attachment edges +- **Fixes 60B.56:** YES — would traverse factor→option(contained_in)→decision path +- **Premature-closure risk:** LOW — only traverses known relationship types +- **Depends on model compliance:** MEDIUM — depends on correct edge creation +- **Requires schema change:** NO (uses existing edge fields) +- **Principal weakness:** Must define which relationships count as "decision-relevant"; currently ambiguous what qualifies + +### Candidate C — OPTION STATUS + +Close when all contained options have `status=known`. + +- **Architecture fit:** HIGH — options already track status +- **Fixes 60B.56:** PARTIAL — addresses symptom but not the causal question +- **Premature-closure risk:** HIGH — option `status=known` may only mean "the alternative itself is established, not that its comparative value is fully determined" +- **Depends on model compliance:** LOW +- **Requires schema change:** NO +- **Principal weakness:** Premature closure. The fixed answer explicitly states no other uncertainties remain, but the option status alone doesn't prove material evidence is complete + +### Candidate D — USER DECLARATION ONLY + +Close when user explicitly says no other material uncertainty remains. + +- **Architecture fit:** MEDIUM — requires capturing and evaluating user statement +- **Fixes 60B.56:** YES — the 60B.56 answer includes "There are no other material uncertainties between launching this year and waiting twelve months." +- **Premature-closure risk:** LOW (with graph guard) / HIGH (without it) +- **Depends on model compliance:** HIGH +- **Requires schema change:** NO +- **Principal weakness:** Relies entirely on model extracting/propagating user statement; no independent graph verification + +### Candidate E — RELATIONSHIP-AWARE FACTORS + USER DECLARATION + +Require both graph evidence of no represented unresolved factor AND explicit user confirmation. + +- **Architecture fit:** MEDIUM — combines B and D +- **Fixes 60B.56:** YES — handles both the graph gap and the user statement +- **Premature-closure risk:** LOW — dual-signal requirement reduces false closure +- **Depends on model compliance:** MEDIUM +- **Requires schema change:** NO +- **Principal weakness:** Requires defining "approved structural relationships" for factor-to-decision linkage + +### Candidate F — MODEL MUST CONTINUE TO OWN CLOSURE + +No deterministic propagation beyond existing child mechanism. + +- **Architecture fit:** HIGH — current state +- **Fixes 60B.56:** NO — leaves the problem unresolved +- **Premature-closure risk:** NONE (won't close at all) +- **Depends on model compliance:** VERY HIGH +- **Requires schema change:** NO +- **Principal weakness:** The model will keep generating follow-up questions forever for non-decomposition decisions + + +--- + +## Checkpoint 6 — Exact 60B.56 Sufficiency Test + +Using Candidate E (relationship-aware + user declaration) as the winning model: + +### Post-60B.56 graph state: +``` +n_enterprise_customer_signing: status=resolved +opt_launch_this_year: status=known, contained_in n_product_launch_decision +opt_wait_twelve_months: status=known, contained_in n_product_launch_decision +n_product_launch_decision: status=unknown, childIds=[], dependsOn=[] +``` + +### Graph-side check: +``` +Represented unresolved material factors remaining: 0 + +No unknown has parentId set to the decision. No unknown has depends_on pointing to the decision. +The only structural path from the resolved factor to the decision goes through option-attachment +(factor -> opt_launch_this_year via contained_in edge -> decision via contained_in), which +isn't traversed by current propagation code. But no UNKNOWN node remains structurally linked +to any option that belongs to this decision — both options are status=known and contain no +unresolved unknown children. + +However: the factor IS still in the graph as a resolved unknown, not an unknown unknown. +The real question is whether there's an unresolved unknown structurally attached via any +approved relationship. The answer is NO — all such links would show through existing +parentId/depends_on routes that are empty. +``` + +### User statement: +``` +User explicitly says no other material uncertainty remains: YES +"There are no other material uncertainties between launching this year and waiting twelve months." +``` + +### Would deterministic sufficiency close n_product_launch_decision: CONDITIONAL + +The winning rule (Candidate E) would close the decision because: +1. Graph check passes: no unresolved unknown linked via parentId/depends_on to the decision or its options +2. User statement provides explicit closure confirmation + +**Why:** The graph-side predicate evaluates empty for this case (no unresolved unknowns in the parentId/depends_on chain). The user statement is captured by the LLM's answer extraction as a "no more uncertainty" signal. Combined, both signals are present. + +### Counterexample from existing fixture + +Testing `pre-anchored-decision-options.json` where an additional material unknown exists: + +If we modify the decision-options fixture to add: +```json +{ + "id": "n_stickiness_uncertainty", + "label": "Whether engineering retention is achievable", + "description": "Uncertain whether two senior engineers will remain after relocation, because they account for key delivery capacity.", + "kind": "unknown", + "status": "unknown", + "parentId": null, + "dependsOn": [], + "affects": ["opt_relocate"] +} +``` + +This unknown has `affects` pointing to an option contained in the decision. No parentId link exists. The factor would: + +``` +Existing counterexample: synthetic extension of pre-anchored-decision-options fixture with n_stickiness_uncertainty having affects → opt_relocate +Remaining material factor: n_stickiness_unclosure (status=unknown) +Would winning rule keep decision open: UNPROVEN — the current graph-side predicate (parentId/depends_on only) would NOT detect this factor. The rule needs the relationship-aware traversal to catch affects→option links. + +However, if we extend Candidate E's graph check to include: +- parentId → decision +- depends_on → decision +- affects → option contained_in decision +Then it WOULD detect n_stickiness_uncertainty and keep the decision open. + +Without that extension, both 60B.56 (correct closure) AND this counterexample (incorrect closure) pass through the same predicate — which is exactly the defect we're diagnosing. +``` + +--- + +## CRITICAL DISTINCTION + +**Choice:** +E + +**Why:** +The analysis identified six candidates for how to determine that a decision's material uncertainty is fully resolved. Candidate E — RELATIONSHIP-AWARE FACTORS + USER DECLARATION — was selected as the winning model because it alone satisfies both requirements simultaneously: (1) graph evidence that no unresolved unknown remains across all structural relationships linking factors to the decision or its options, and (2) explicit user confirmation that nothing else is uncertain. Single-signal approaches (parentId-only, option-status-only, user-declaration-only) each fail on at least one dimension. Candidate E's dual-signal requirement reduces premature-closure risk to LOW. The critical distinction is that closure requires TWO independent signals converging — not one strong signal and not two weak ones. The graph-side signal proves "nothing left unresolved in the model." The user signal proves "nothing left unresolved in reality." Only together do they establish sufficiency. + +--- + +## MINIMUM CORRECTIVE BOUNDARY + +**Choice:** +B + +**Why:** +The smallest change that makes closure detection correct is extending the unresolved-unknown predicate to traverse option-attachment edges: `parentId → decision`, `depends_on → decision`, and `affects → option contained_in decision`. This is a traversal-extension, not a schema change. No new fields or node types are required. The edge semantics already exist in the graph. Only the propagation logic in `propagateResolvedChildEvidence` / `computeParentProgressState` needs to widen its scan to include option-contained unknowns reachable via the approved relationship set. This matches Candidate B from Checkpoint 5. + +--- + +## CLOSURE VS DIRECTION + +**Can close without preferred option:** +PARTIAL + +**Why:** +Currently, the predicate only checks parentId/depends_on children of the decision node. It does not check unknowns attached to any of the decision's options via affected/contained relationships. Closing would require checking ALL options of the decision for unresolved unknowns, not just those directly under the decision as a child. The architecture supports option-attached factors (as shown by the customer-signing case), but the closure predicate doesn't traverse into them. This is PARTIAL because the infrastructure exists but the traversal gap means only decomposition-child closure works correctly today. + +--- + +## IMPLEMENTATION READINESS + +**B** + +One unresolved question: +Which exact relationship types qualify as "decision-relevant" for generic (non-decomposition) factors — `affects`, `may_cause`, `causes`, or all three? 60B.15 established these for context admission but didn't define their closure-weight semantics. + +Smallest implementation boundary: +Extend `propagateResolvedChildEvidence` to also scan option-attached unknowns: for each option contained_in the decision, find all unresolved unknowns linked via `affects` or `contained_in` edges to that option. Combine with existing parentId/depends_on child scan. If combined result is empty AND user confirmation exists → close decision. + +Production code changed: +NO + +Tests changed: +NO + +Prompt changed: +NO + +Schema changed: +NO + +Ollama calls: +0 + +Live API calls: +0 + +Vitest run: +NO + +Documentation updated: +experiment-60b58.md + current-handoff.md + +Git status: +(to be confirmed after commit) +