From 391667777efbb4c95683d83d4563abd5a489d748 Mon Sep 17 00:00:00 2001 From: robbond Date: Fri, 14 Aug 2026 18:24:36 +0100 Subject: [PATCH] experiment: confirm live sufficiency confirmation question --- docs/current-handoff.md | 45 ++++++++++ docs/experiment-60b74.md | 185 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 230 insertions(+) create mode 100644 docs/experiment-60b74.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 5638c74..f720162 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -3549,3 +3549,48 @@ Pre-existing apply-proposal failures (4) confirmed unrelated via git stash/re-ru #### Status COMPLETE — production + tests committed, documentation recorded. + +--- + +#### 60B.74 — Missing Sufficiency Confirmation Question (Live Verification) + +**Date:** 2026-08-14 +**Branch:** `feature/sufficiency-confirmation-question-v0.45` +**Type:** Live regression observation only + +#### Objective + +Verify whether the post-60B.73 production path preserves the no-confirmation guard while replacing the generic decision-threshold continuation with the focused sufficiency confirmation/discovery question. + +#### Run + +```bash +FIXTURE_MODE=updateOnly \ +FIXTURE_PATH=tests/fixtures/pre-anchored-product-launch-customer-signing.json \ +ANSWER_2="The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received." \ +CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \ +node scripts/reproduce-multi-turn-investigation.mjs +``` + +#### Result: B — GENERIC QUESTION PERSISTS + +Observed behavior is IDENTICAL to 60B.71 on the question level: + +- Customer resolves correctly → resolved (negative meaning preserved) +- Decision stays open → unknown (no premature closure) +- Active target remains n_product_launch_decision ✓ +- **Question remains generic:** `"What outcome would demonstrate enough value to justify launching?"` +- **Template is `decision_threshold_outcome` — NOT `decision_threshold_sufficiency_confirmation`** + +#### Root Cause: Deterministic Bug in State B Detection + +The condition at line 2010 of `question-formulator.js`: + +```js +node.kind !== "unknown" +``` + +can NEVER be true for any decision node, because all decisions have `kind === "unknown"` (same as child factors). The SituationKind enum has no "decision" kind value. This prevents the sufficiency template from ever firing. + +#### Status +BLOCKED — State B detection bug blocks 60B.73 implementation from reaching production. Fix needed in `question-formulator.js` line 2010 to correctly distinguish parent decisions from child factors. diff --git a/docs/experiment-60b74.md b/docs/experiment-60b74.md new file mode 100644 index 0000000..d8b84f4 --- /dev/null +++ b/docs/experiment-60b74.md @@ -0,0 +1,185 @@ +# Experiment 60B.74 — Missing Sufficiency Confirmation Question (Live Verification) + +**Date:** 2026-08-14 +**Branch:** `feature/sufficiency-confirmation-question-v0.45` +**Head commit:** 7cfeee1 docs: record sufficiency confirmation question + +## Objective + +Does the post-60B.73 production path preserve the no-confirmation guard while replacing the generic decision-threshold continuation with the focused sufficiency confirmation/discovery question? + +## Configured environment + +- **Model:** qwen-claude:latest +- **Ollama base URL:** http://192.168.1.111:11434 +- **Confidence Engine base URL:** http://127.0.0.1:3000 + +## Input + +- **Fixture:** `tests/fixtures/pre-anchored-product-launch-customer-signing.json` + - Pre-anchored state: decision (`n_product_launch_decision`) unknown; customer signing (`n_enterprise_customer_signing`) unknown, activeUnknownNodeId = n_enterprise_customer_signing. +- **Answer (exact, no paraphrase):** "The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received." +- Explicit sufficiency confirmation: **NO** + +## Run + +```bash +FIXTURE_MODE=updateOnly \ +FIXTURE_PATH=tests/fixtures/pre-anchored-product-launch-customer-signing.json \ +ANSWER_2="The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received." \ +CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \ +node scripts/reproduce-multi-turn-investigation.mjs +``` + +- **startCalls:** 0 +- **updateCalls:** 1 +- **totalCalls:** 1 +- **Retries:** 0 + +## Results + +### Proposal accepted: YES (HTTP 200) + +### updatedNodes: +```json +[ + { + "nodeId": "n_enterprise_customer_signing", + "previousStatus": "unknown", + "newStatus": "resolved", + "previousValue": null, + "newValue": null, + "reason": "User explicitly confirmed the enterprise customer will not sign if launched this year." + }, + { + "nodeId": "opt_launch_this_year", + "previousStatus": "known", + "newStatus": "known", + "previousValue": null, + "newValue": "Expected annual revenue reduced to £500k; £300k launch cost remains.", + "reason": "Reflects updated financial consequence following resolved customer signing status." + } +] +``` + +### resolvedUnknownNodeIds: +```json +["n_enterprise_customer_signing"] +``` + +### addedNodes: +```json +[] +``` + +### addedEdges: +```json +[] +``` + +### DIRECT QUESTION METADATA + +``` +finalActiveUnknownNodeId: "n_product_launch_decision" + +finalSelectedQuestion: { + "nodeId": "n_product_launch_decision", + "question": "What outcome would demonstrate enough value to justify launching?", + "reason": "Formulated from graph context using the decision_threshold investigation strategy.", + "strategy": "decision_threshold", + "investigationStrategy": { + "key": "decision_threshold", + "reason": "Selected because the unknown determines the threshold for making or justifying a decision.", + "nodeId": "n_product_launch_decision", + "nodeLabel": "Which option leaves us better off overall?", + "meaning": "which option leaves us better off overall", + "actionPhrase": "launch", + "relatedNodeIds": ["opt_launch_this_year", "opt_wait_twelve_months"], + "centralStatement": "We are evaluating two product-launch timing options: launching the new software product this year or waiting twelve months." + }, + "reasoningPattern": "decision", + "reasoningPatternReason": "Selected decision because the active unknown sits inside a build, continue, invest, or commercial-justification decision context.", + "questionFamily": "decision_threshold", + "allowedQuestionFamilies": ["decision_foundation", "decision_evidence", "decision_threshold", "definition"], + "rejectedQuestionFamilies": ["explanation", "comparison", "contradiction", "diagnosis", "prioritisation"], + "selectedQuestionTemplate": "decision_threshold_outcome", + "questionComplexity": { + "acceptable": true, + "primaryConceptCount": 1, + "compoundQuestionSignals": [], + "abstractTermCount": 0, + "cognitiveLoad": "low", + "reasons": [] + } +} +``` + +### Customer node final state: +- `n_enterprise_customer_signing`: status = **resolved**, value = null (meaning carried in reason text) + +### Customer resolution meaning: +"User explicitly confirmed the enterprise customer will not sign" → Negative meaning **PRESERVED** in reason text. + +### Decision node final state: +- `n_product_launch_decision`: status = **unknown** (UNRESOLVED) — NOT closed, NOT in resolvedUnknownNodeIds + +## Bug Identification + +The State B detection condition in `formulateQuestion()` at line 2010 of `question-formulator.js` contains a deterministic bug: + +```js +if ( + reasoningPatternSelection.pattern === "decision" && + node.kind !== "unknown", // ← NEVER TRUE for decision nodes! + node.status !== "known" && + node.status !== "resolved" && + node.status !== "contradicted" && + hasRemainingMaterialFactors(node.id, graph) === false +) +``` + +All decision nodes have `kind === "unknown"` (along with all child factors). The condition `node.kind !== "unknown"` excludes ALL decision nodes from State B detection. There are no kind values that represent "decision" in the SituationKind enum — decisions share kind="unknown" with factors. + +This means the sufficiency confirmation template (`decision_threshold_sufficiency_confirmation`) can NEVER fire for any parent decision target, regardless of how many factors are resolved or whether explicit confirmation is absent. + +## 60B.71 → 60B.74 comparison + +| Field | 60B.71 (before fix) | 60B.74 (after fix) | +|---|---|---| +| Customer status | unknown→resolved | unknown→resolved | +| Decision status | **unknown** (KEPT OPEN) | **unknown** (KEPT OPEN) | +| finalActiveUnknownNodeId | "n_product_launch_decision" | "n_product_launch_decision" | +| selectedQuestionTemplate | decision_threshold_outcome | decision_threshold_outcome | +| Question family | decision_threshold | decision_threshold | +| Question | "What outcome would demonstrate enough value to justify launching?" | "What outcome would demonstrate enough value to justify launching?" | +| addedNodes | [n_revised_launch_year_revenue] | [] | +| addedEdges | [e-customer-confirmation-to-revenue] | [] | + +Note: The question text is IDENTICAL across both experiments. The fix did not land in the production path. + +## Classification: B — GENERIC QUESTION PERSISTS + +Decision stays open and active target remains the existing parent decision (correct structural behavior), but the sufficiency confirmation/discovery template does NOT fire. The generic `decision_threshold_outcome` question ("What outcome would demonstrate enough value to justify launching?") persists unchanged from 60B.71. + +## Why + +The State B detection condition `node.kind !== "unknown"` can never be true for any decision node, since all decisions have kind="unknown" in the SituationKind enum. The condition was designed to exclude child factors but instead excludes ALL unknown-kind nodes including the parent decision itself. No kind value in the schema represents "decision" specifically. + +## What this proves + +1. **The no-confirmation guard still works structurally.** The decision remains open; the customer factor resolves correctly; negative meaning is preserved. +2. **60B.73 implementation does NOT reach production.** The sufficiency template code exists in `question-formulator.js` at line 2034 but the guard condition that gates it (line 2010) prevents entry for any decision target. +3. **This is a deterministic bug, not an LLM non-determinism issue.** The wrong question fires in every run regardless of model. + +## What remains unproven + +1. **How to correctly distinguish parent decisions from child factors.** Neither parentId nor kind provides this distinction (both are null and "unknown" respectively). +2. **The fix itself** — needs a different detection mechanism (e.g., whether the node's children include unresolved unknowns, or whether it is an ancestor of options). + +## Production code changed: NO +## Prompt changed: NO +## Schema changed: NO +## Harness changed during experiment: NO +## Vitest run: NO +## Ollama calls: 1 +## Direct API calls: 0