experiment: confirm live sufficiency decision detection

This commit is contained in:
2026-08-15 05:59:35 +01:00
parent 47f42b24cc
commit 887a9710c8
2 changed files with 141 additions and 0 deletions
+16
View File
@@ -3601,3 +3601,19 @@ Fixed the deterministic bug: removed `node.kind !== "unknown"` from State B dete
Tests updated to production shape (`kind: "unknown"` for decisions). Added 6 focused tests in 60B.75 block. All 22 tests pass. No schema/prompt/provider/harness changes. Tests updated to production shape (`kind: "unknown"` for decisions). Added 6 focused tests in 60B.75 block. All 22 tests pass. No schema/prompt/provider/harness changes.
Status: resolved. Status: resolved.
#### Experiment 60B.76 — Live Verification of Sufficiency Question Fix (FAIL)
**Date:** 2026-08-15
**Branch:** `feature/sufficiency-decision-detection-v0.46`
**Type:** Live regression of 60B.75 fix
**Result: C — DECISION CLOSES + F — STRUCTURAL REGRESSION**
The deterministic sufficiency-closure gate in `apply-proposal.js` fires *before* question formulation when all material factors resolve. The decision (`n_product_launch_decision`) was auto-resolved with status=resolved before `formulateQuestion()` could fire the sufficiency confirmation template.
The 60B.75 kind-gate removal successfully exposed State B to reasoningPattern detection in the tests, but the live path hits an earlier gate: deterministic closure via `hasRemainingMaterialFactors()` returning false. The decision closes prematurely, no question fires, and activeUnknownNodeId becomes null.
Customer resolves correctly (negative meaning preserved). But the no-confirmation guard cannot prevent deterministic auto-closure because it operates after closure, not during.
Status: unblocked — need to address `apply-proposal.js` sufficiency closure gate before State B question selection can be tested live.
+125
View File
@@ -0,0 +1,125 @@
# Experiment 60B.76 — Live Verification of Sufficiency Question Fix
**Date:** 2026-08-15
**Branch:** `feature/sufficiency-decision-detection-v0.46`
**Preceded by:** Experiment 60B.75 (fix applied + tests pass)
## Objective
Does the 60B.75 fix make the exact no-confirmation production case select the focused sufficiency confirmation/discovery question while keeping the decision open?
## Configured environment
- **Model:** qwen-claude:latest
- **Ollama base URL:** http://192.168.1.111:11434
- **Confidence Engine base URL:** http://127.0.0.1:3000
## Input
- **Fixture:** `tests/fixtures/pre-anchored-product-launch-customer-signing.json`
- Pre-anchored state: decision (`n_product_launch_decision`) unknown; customer signing (`n_enterprise_customer_signing`) unknown, activeUnknownNodeId = n_enterprise_customer_signing.
- **Answer (exact, no paraphrase):** "The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received."
- Explicit sufficiency confirmation: **NO**
## Run
```bash
FIXTURE_MODE=updateOnly \
FIXTURE_PATH=tests/fixtures/pre-anchored-product-launch-customer-signing.json \
ANSWER_2="The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received." \
CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \
node scripts/reproduce-multi-turn-investigation.mjs
```
- **startCalls:** 0
- **updateCalls:** 1
- **totalCalls:** 1
- **Retries:** 0
## Results
### Proposal accepted: YES (HTTP 200)
### updatedNodes:
```json
[
{
"nodeId": "n_enterprise_customer_signing",
"previousStatus": "unknown",
"newStatus": "resolved",
"reason": "Enterprise customer confirmed in writing they will not sign if we launch this year."
},
{
"nodeId": "n_product_launch_decision",
"previousStatus": "unknown",
"newStatus": "resolved",
"reason": "Material uncertainty resolved; evidence now sufficiently distinguishes net value of launch this year versus wait twelve months."
}
]
```
### resolvedUnknownNodeIds:
```json
["n_enterprise_customer_signing", "n_product_launch_decision"]
```
### addedNodes: []
### addedEdges: []
### structuralActionRequired: null
### finalActiveUnknownNodeId: null
### finalSelectedQuestion: null
## Analysis
The sufficiency confirmation question did **not** fire. Instead, the decision was closed deterministically during `apply-proposal.js` processing — before `formulateQuestion()` was ever called.
**Root cause:** The deterministic sufficiency-closure gate (line 1673 of `apply-proposal.js`) fires when:
1. All material uncertainties are resolved → `hasRemainingMaterialFactors()` returns false
2. This triggers automatic decision closure with reason "sufficient evidence distinguishes net value"
The 60B.75 fix exposed State B in `selectReasoningPattern()` (so `reasoningPattern === "decision"` now reaches the sufficiency template selection), but did not address the earlier deterministic closure gate in `apply-proposal.js`. The decision resolves before question formulation can occur.
### Structural impact:
- **Customer:** Resolves correctly → status = resolved, negative meaning preserved
- **Decision:** Prematurely closes → status = resolved (should be unknown)
- **Active target:** Gone → activeUnknownNodeId = null (decision was the only unknown)
- **Question:** Never formulated → finalSelectedQuestion = null
- **No new nodes/edges**
### Classification: C — DECISION CLOSES
The deterministic sufficiency closure in `apply-proposal.js` resolves the decision before the question-formulation path is reached. This is also **F — STRUCTURAL REGRESSION** because meaning loss does not occur, but the structural behavior (premature closure) prevents testing of State B's question selection.
## Comparison with 60B.74
| Field | 60B.74 | 60B.76 |
|---|---|---|
| Customer status | resolved | resolved ✓ |
| Customer meaning preserved | yes | yes ✓ |
| Decision status | **unknown** (kept open) | **resolved** (closed prematurely) ✗ |
| activeUnknownNodeId | n_product_launch_decision | null |
| finalSelectedQuestion | decision_threshold_outcome | null (never reached) |
| Classification | B | C + F |
The decision closed in 60B.76 because `hasRemainingMaterialFactors()` correctly returns false after the customer factor resolves — and the deterministic closure gate fires *before* question formulation can evaluate sufficiency confirmation presence.
## What this proves
1. **The 60B.75 fix successfully exposes State B to reasoningPattern detection.** The decision-context path is now reachable.
2. **But an earlier gate prevents reaching that path:** the `apply-proposal.js` deterministic closure fires before question formulation, closing the decision prematurely when no material factors remain.
3. **The sufficiency confirmation check must either be lifted from deterministic closure or moved into the closure gate itself** — checking for explicit confirmation *before* auto-closure.
## What remains unproven
1. Whether State B's sufficiency template selection works correctly (question formulation is not reached).
2. Whether removing the kind gate in `selectReasoningPattern` causes any unintended side effects when question formulation IS reached.
## Production code changed: NO
## Prompt changed: NO
## Schema changed: NO
## Harness changed during experiment: NO
## Vitest run: NO
## Ollama calls: 1
## Direct API calls: 0