From c335bf0a9a36dd00c67d16c3b76c975959bf7e9f Mon Sep 17 00:00:00 2001 From: robbond Date: Fri, 14 Aug 2026 17:23:12 +0100 Subject: [PATCH] experiment: define missing sufficiency confirmation question --- docs/current-handoff.md | 15 +++++ docs/experiment-60b72.md | 131 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+) create mode 100644 docs/experiment-60b72.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 29a73f2..13a34a5 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -3492,3 +3492,18 @@ The no-confirmation guard works live post-refactor. Parent decision stays open w --- + +--- + +--- + +### Experiment 60B.72 — Missing Sufficiency Confirmation Question Diagnosis (READ-ONLY) + +**Date:** 2026-08-14 +**Branch:** `feature/decision-sufficiency-module-v0.44` +**Parent:** 60B.71 (no-confirmation guard confirmed working) +**Type:** Read-only reasoning diagnosis + +Inverses the question-target problem of 60B.71: when the guard correctly keeps the decision open, the next question should be a sufficiency-confirmation question — not a generic decision_threshold continuation. Currently it cannot distinguish "need to confirm no remaining factors" from "need to investigate an unresolved factor." + +**Status:** pending Rob's review. Full analysis in docs/experiment-60b72.md. diff --git a/docs/experiment-60b72.md b/docs/experiment-60b72.md new file mode 100644 index 0000000..4ac6f65 --- /dev/null +++ b/docs/experiment-60b72.md @@ -0,0 +1,131 @@ +# Experiment 60B.72 — Missing Sufficiency Confirmation Question Diagnosis + +**Date:** 2026-08-14 +**Branch:** `feature/decision-sufficiency-module-v0.44` +**Parent:** 60B.71 (no-confirmation guard confirmed working) +**Type:** Read-only reasoning diagnosis + +## Problem Statement + +When no material factors remain but the user has not explicitly confirmed sufficiency, +the engine asks a generic decision_threshold question ("What outcome would demonstrate +enough value to justify X?") instead of asking whether what's already been presented +is sufficient. + +The core distinction: State A (genuine unresolved factor exists) and State B (no +factor remains, no confirmation given) both collapse to `decision_threshold` because +`selectInvestigationStrategy` does not consult `hasRemainingMaterialFactors()`. + +## Fixed Diagnosis + +- `hasRemainingMaterialFactors(decisionNodeId, graph) === false` for State B ✓ +- `isUserConfirmationOfNoRemainingUncertainty(answer) === false` for State B ✓ +- Decision status remains unknown ✓ +- Selector sees unresolved decision → selector does not see remaining-factor state +- `decision_threshold` wins by normal unresolved-decision logic + +## Candidate Assessment + +### Candidate A — KEEP CURRENT DECISION_THRESHOLD +Architecture fit: HIGH | Premature-closure risk: MEDIUM | Generic-loop risk: HIGH +Reopening resolved evidence risk: LOW | User burden: MEDIUM +New state field: NO | New question family: NO | Existing target reusable: YES +Principal weakness: "What outcome would demonstrate enough value to justify X?" is a +continuation prompt (asks for MORE justification) rather than the missing sufficiency +confirmation. Creates high generic-loop risk when no factors remain. + +### Candidate B — DIRECT SUFFICIENCY CONFIRMATION +Architecture fit: MEDIUM | Premature-closure risk: LOW | Generic-loop risk: MEDIUM +Reopening resolved evidence risk: LOW | User burden: MEDIUM +New state field: NO | New question family: PARTIAL (one new template) | Existing target reusable: YES +Principal weakness: Binary yes/no framing may elicit "yes" without specifics. + +### Candidate C — DISCOVER A MISSING FACTOR +Architecture fit: MEDIUM | Premature-closure risk: LOW | Generic-loop risk: LOW +Reopening resolved evidence risk: MEDIUM | User burden: HIGH +New state field: NO | New question family: PARTIAL (one new template) | Existing target reusable: YES +Principal weakness: Puts all discovery burden on the user. Silent if user forgets something. + +### Candidate D — CLOSE ANYWAY +Architecture fit: LOW | Premature-closure risk: HIGH | Generic-loop risk: NONE +Reopening resolved evidence risk: NONE | User burden: NONE +New state field: NO | New question family: NO | Existing target reusable: NO (target should transition) +Principal weakness: Directly contradicts 60B.71's conservative guard. Closes without explicit confirmation. + +### Candidate E — MODEL CHOOSES BETWEEN B/C +Architecture fit: LOW | Premature-closure risk: UNPROVEN | Generic-loop risk: UNPROVEN +Reopening resolved evidence risk: UNPROVEN | User burden: MEDIUM +New state field: NO | New question family: YES | Existing target reusable: MAYBE +Principal weakness: Adds non-determinism where determinism is possible. The distinction +between B vs C IS deterministically knowable from `hasRemainingMaterialFactors()`. + +## Winning Intent: D — BOTH CONFIRMATION + DISCOVERY IN ONE QUESTION + +Structure: "Is there anything else material you haven't mentioned that could change +which option is better?" + +This asks about sufficiency (confirmation) while allowing identification of a remaining +factor (discovery). Deterministic branching on the answer: +- "No" → closure proceeds +- Names factor → that factor becomes next unknown + +## Existing Question Machinery + +Family reusable: decision_threshold (or decision_evidence) — PARTIAL reuse needed. +One new deterministic template suffices. No new family required. + +The `decision_threshold` family maps `{family: "decision_threshold", template: "decision_threshold_outcome"}` +and produces questions via `buildQuestionFromStrategy({key: "decision_threshold"})`. +Adding a new State B template here changes the question text without affecting which +strategy is selected or which target is active. + +## State Representation + +Choice: B — TRANSIENT DETERMINISTIC BRANCH IS SUFFICIENT + +All four signals available at selection time: +1. `target.kind === "unknown"` and target is decision +2. `hasRemainingMaterialFactors(target.id, graph) === false` +3. Raw confirmation absent from answer context +4. Active target still unknown (not closed/resolved) + +No persisted field required. The state exists entirely in the current turn's context. + +## Branch Location: C — QUESTION FORMULATION + +Location A (active-target selection): Too high-level. Target identity logic should not +depend on remaining-factor state. MEDIUM coupling. + +Location B (investigation strategy selection): Addresses root cause but mixes text-pattern +matching with graph-quantitative logic. HIGH coupling. + +Location C (question formulation): Cleanest boundary. Changes only the question OUTPUT +without affecting inputs or control flow. LOW coupling. + +Preferred: C — `buildQuestionFromFamily` receives all needed signals (node, graph, +investigationStrategy) and is where "how to ask" decisions belong. + +## Conservative Behaviour + +- One confirmation/discovery turn supported: YES +- False-open-over-false-closed preserved: YES +- Resolved factors stay closed: UNPROVEN (theoretical risk if user mentions resolved item, but it's user-initiated) +- New genuine factor can be surfaced: YES + +## Critical Distinction: B — MISSING CONFIRMATION NEEDS DISTINCT QUESTION INTENT + +Current `decision_threshold` asks "what MORE justification is needed?" when the correct +question for State B is "is what we have sufficient?" These are different information goals. + +## Minimum Corrective Boundary: C — ONE NEW TEMPLATE IN EXISTING FAMILY + +Transitive deterministic branch + one new template in `decision_threshold` family. + +Prevents premature closure (one more turn), prevents generic looping (distinct intent), +asks only for missing information, leaves decision identity stable. + +## Implementation Readiness: A — READY FOR BOUNDED IMPLEMENTATION + +No unresolved design question. Smallest boundary: add State B detection at formulation +time + one new sufficiency confirmation/discovery template in `decision_threshold` family. +