From 887a9710c82367b6cf326457aebde998794fd83b Mon Sep 17 00:00:00 2001 From: robbond Date: Sat, 15 Aug 2026 05:59:35 +0100 Subject: [PATCH] experiment: confirm live sufficiency decision detection --- docs/current-handoff.md | 16 +++++ docs/experiment-60b76.md | 125 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 141 insertions(+) create mode 100644 docs/experiment-60b76.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 7d69cb0..213399d 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -3601,3 +3601,19 @@ Fixed the deterministic bug: removed `node.kind !== "unknown"` from State B dete Tests updated to production shape (`kind: "unknown"` for decisions). Added 6 focused tests in 60B.75 block. All 22 tests pass. No schema/prompt/provider/harness changes. Status: resolved. + +#### Experiment 60B.76 — Live Verification of Sufficiency Question Fix (FAIL) + +**Date:** 2026-08-15 +**Branch:** `feature/sufficiency-decision-detection-v0.46` +**Type:** Live regression of 60B.75 fix + +**Result: C — DECISION CLOSES + F — STRUCTURAL REGRESSION** + +The deterministic sufficiency-closure gate in `apply-proposal.js` fires *before* question formulation when all material factors resolve. The decision (`n_product_launch_decision`) was auto-resolved with status=resolved before `formulateQuestion()` could fire the sufficiency confirmation template. + +The 60B.75 kind-gate removal successfully exposed State B to reasoningPattern detection in the tests, but the live path hits an earlier gate: deterministic closure via `hasRemainingMaterialFactors()` returning false. The decision closes prematurely, no question fires, and activeUnknownNodeId becomes null. + +Customer resolves correctly (negative meaning preserved). But the no-confirmation guard cannot prevent deterministic auto-closure because it operates after closure, not during. + +Status: unblocked — need to address `apply-proposal.js` sufficiency closure gate before State B question selection can be tested live. diff --git a/docs/experiment-60b76.md b/docs/experiment-60b76.md new file mode 100644 index 0000000..cef313e --- /dev/null +++ b/docs/experiment-60b76.md @@ -0,0 +1,125 @@ +# Experiment 60B.76 — Live Verification of Sufficiency Question Fix + +**Date:** 2026-08-15 +**Branch:** `feature/sufficiency-decision-detection-v0.46` +**Preceded by:** Experiment 60B.75 (fix applied + tests pass) + +## Objective + +Does the 60B.75 fix make the exact no-confirmation production case select the focused sufficiency confirmation/discovery question while keeping the decision open? + +## Configured environment + +- **Model:** qwen-claude:latest +- **Ollama base URL:** http://192.168.1.111:11434 +- **Confidence Engine base URL:** http://127.0.0.1:3000 + +## Input + +- **Fixture:** `tests/fixtures/pre-anchored-product-launch-customer-signing.json` + - Pre-anchored state: decision (`n_product_launch_decision`) unknown; customer signing (`n_enterprise_customer_signing`) unknown, activeUnknownNodeId = n_enterprise_customer_signing. +- **Answer (exact, no paraphrase):** "The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received." +- Explicit sufficiency confirmation: **NO** + +## Run + +```bash +FIXTURE_MODE=updateOnly \ +FIXTURE_PATH=tests/fixtures/pre-anchored-product-launch-customer-signing.json \ +ANSWER_2="The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received." \ +CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \ +node scripts/reproduce-multi-turn-investigation.mjs +``` + +- **startCalls:** 0 +- **updateCalls:** 1 +- **totalCalls:** 1 +- **Retries:** 0 + +## Results + +### Proposal accepted: YES (HTTP 200) + +### updatedNodes: +```json +[ + { + "nodeId": "n_enterprise_customer_signing", + "previousStatus": "unknown", + "newStatus": "resolved", + "reason": "Enterprise customer confirmed in writing they will not sign if we launch this year." + }, + { + "nodeId": "n_product_launch_decision", + "previousStatus": "unknown", + "newStatus": "resolved", + "reason": "Material uncertainty resolved; evidence now sufficiently distinguishes net value of launch this year versus wait twelve months." + } +] +``` + +### resolvedUnknownNodeIds: +```json +["n_enterprise_customer_signing", "n_product_launch_decision"] +``` + +### addedNodes: [] +### addedEdges: [] + +### structuralActionRequired: null +### finalActiveUnknownNodeId: null +### finalSelectedQuestion: null + +## Analysis + +The sufficiency confirmation question did **not** fire. Instead, the decision was closed deterministically during `apply-proposal.js` processing — before `formulateQuestion()` was ever called. + +**Root cause:** The deterministic sufficiency-closure gate (line 1673 of `apply-proposal.js`) fires when: +1. All material uncertainties are resolved → `hasRemainingMaterialFactors()` returns false +2. This triggers automatic decision closure with reason "sufficient evidence distinguishes net value" + +The 60B.75 fix exposed State B in `selectReasoningPattern()` (so `reasoningPattern === "decision"` now reaches the sufficiency template selection), but did not address the earlier deterministic closure gate in `apply-proposal.js`. The decision resolves before question formulation can occur. + +### Structural impact: + +- **Customer:** Resolves correctly → status = resolved, negative meaning preserved +- **Decision:** Prematurely closes → status = resolved (should be unknown) +- **Active target:** Gone → activeUnknownNodeId = null (decision was the only unknown) +- **Question:** Never formulated → finalSelectedQuestion = null +- **No new nodes/edges** + +### Classification: C — DECISION CLOSES + +The deterministic sufficiency closure in `apply-proposal.js` resolves the decision before the question-formulation path is reached. This is also **F — STRUCTURAL REGRESSION** because meaning loss does not occur, but the structural behavior (premature closure) prevents testing of State B's question selection. + +## Comparison with 60B.74 + +| Field | 60B.74 | 60B.76 | +|---|---|---| +| Customer status | resolved | resolved ✓ | +| Customer meaning preserved | yes | yes ✓ | +| Decision status | **unknown** (kept open) | **resolved** (closed prematurely) ✗ | +| activeUnknownNodeId | n_product_launch_decision | null | +| finalSelectedQuestion | decision_threshold_outcome | null (never reached) | +| Classification | B | C + F | + +The decision closed in 60B.76 because `hasRemainingMaterialFactors()` correctly returns false after the customer factor resolves — and the deterministic closure gate fires *before* question formulation can evaluate sufficiency confirmation presence. + +## What this proves + +1. **The 60B.75 fix successfully exposes State B to reasoningPattern detection.** The decision-context path is now reachable. +2. **But an earlier gate prevents reaching that path:** the `apply-proposal.js` deterministic closure fires before question formulation, closing the decision prematurely when no material factors remain. +3. **The sufficiency confirmation check must either be lifted from deterministic closure or moved into the closure gate itself** — checking for explicit confirmation *before* auto-closure. + +## What remains unproven + +1. Whether State B's sufficiency template selection works correctly (question formulation is not reached). +2. Whether removing the kind gate in `selectReasoningPattern` causes any unintended side effects when question formulation IS reached. + +## Production code changed: NO +## Prompt changed: NO +## Schema changed: NO +## Harness changed during experiment: NO +## Vitest run: NO +## Ollama calls: 1 +## Direct API calls: 0