experiment: confirm live sufficiency decision detection
This commit is contained in:
@@ -3601,3 +3601,19 @@ Fixed the deterministic bug: removed `node.kind !== "unknown"` from State B dete
|
||||
Tests updated to production shape (`kind: "unknown"` for decisions). Added 6 focused tests in 60B.75 block. All 22 tests pass. No schema/prompt/provider/harness changes.
|
||||
|
||||
Status: resolved.
|
||||
|
||||
#### Experiment 60B.76 — Live Verification of Sufficiency Question Fix (FAIL)
|
||||
|
||||
**Date:** 2026-08-15
|
||||
**Branch:** `feature/sufficiency-decision-detection-v0.46`
|
||||
**Type:** Live regression of 60B.75 fix
|
||||
|
||||
**Result: C — DECISION CLOSES + F — STRUCTURAL REGRESSION**
|
||||
|
||||
The deterministic sufficiency-closure gate in `apply-proposal.js` fires *before* question formulation when all material factors resolve. The decision (`n_product_launch_decision`) was auto-resolved with status=resolved before `formulateQuestion()` could fire the sufficiency confirmation template.
|
||||
|
||||
The 60B.75 kind-gate removal successfully exposed State B to reasoningPattern detection in the tests, but the live path hits an earlier gate: deterministic closure via `hasRemainingMaterialFactors()` returning false. The decision closes prematurely, no question fires, and activeUnknownNodeId becomes null.
|
||||
|
||||
Customer resolves correctly (negative meaning preserved). But the no-confirmation guard cannot prevent deterministic auto-closure because it operates after closure, not during.
|
||||
|
||||
Status: unblocked — need to address `apply-proposal.js` sufficiency closure gate before State B question selection can be tested live.
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
# Experiment 60B.76 — Live Verification of Sufficiency Question Fix
|
||||
|
||||
**Date:** 2026-08-15
|
||||
**Branch:** `feature/sufficiency-decision-detection-v0.46`
|
||||
**Preceded by:** Experiment 60B.75 (fix applied + tests pass)
|
||||
|
||||
## Objective
|
||||
|
||||
Does the 60B.75 fix make the exact no-confirmation production case select the focused sufficiency confirmation/discovery question while keeping the decision open?
|
||||
|
||||
## Configured environment
|
||||
|
||||
- **Model:** qwen-claude:latest
|
||||
- **Ollama base URL:** http://192.168.1.111:11434
|
||||
- **Confidence Engine base URL:** http://127.0.0.1:3000
|
||||
|
||||
## Input
|
||||
|
||||
- **Fixture:** `tests/fixtures/pre-anchored-product-launch-customer-signing.json`
|
||||
- Pre-anchored state: decision (`n_product_launch_decision`) unknown; customer signing (`n_enterprise_customer_signing`) unknown, activeUnknownNodeId = n_enterprise_customer_signing.
|
||||
- **Answer (exact, no paraphrase):** "The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received."
|
||||
- Explicit sufficiency confirmation: **NO**
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
FIXTURE_MODE=updateOnly \
|
||||
FIXTURE_PATH=tests/fixtures/pre-anchored-product-launch-customer-signing.json \
|
||||
ANSWER_2="The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received." \
|
||||
CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \
|
||||
node scripts/reproduce-multi-turn-investigation.mjs
|
||||
```
|
||||
|
||||
- **startCalls:** 0
|
||||
- **updateCalls:** 1
|
||||
- **totalCalls:** 1
|
||||
- **Retries:** 0
|
||||
|
||||
## Results
|
||||
|
||||
### Proposal accepted: YES (HTTP 200)
|
||||
|
||||
### updatedNodes:
|
||||
```json
|
||||
[
|
||||
{
|
||||
"nodeId": "n_enterprise_customer_signing",
|
||||
"previousStatus": "unknown",
|
||||
"newStatus": "resolved",
|
||||
"reason": "Enterprise customer confirmed in writing they will not sign if we launch this year."
|
||||
},
|
||||
{
|
||||
"nodeId": "n_product_launch_decision",
|
||||
"previousStatus": "unknown",
|
||||
"newStatus": "resolved",
|
||||
"reason": "Material uncertainty resolved; evidence now sufficiently distinguishes net value of launch this year versus wait twelve months."
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
### resolvedUnknownNodeIds:
|
||||
```json
|
||||
["n_enterprise_customer_signing", "n_product_launch_decision"]
|
||||
```
|
||||
|
||||
### addedNodes: []
|
||||
### addedEdges: []
|
||||
|
||||
### structuralActionRequired: null
|
||||
### finalActiveUnknownNodeId: null
|
||||
### finalSelectedQuestion: null
|
||||
|
||||
## Analysis
|
||||
|
||||
The sufficiency confirmation question did **not** fire. Instead, the decision was closed deterministically during `apply-proposal.js` processing — before `formulateQuestion()` was ever called.
|
||||
|
||||
**Root cause:** The deterministic sufficiency-closure gate (line 1673 of `apply-proposal.js`) fires when:
|
||||
1. All material uncertainties are resolved → `hasRemainingMaterialFactors()` returns false
|
||||
2. This triggers automatic decision closure with reason "sufficient evidence distinguishes net value"
|
||||
|
||||
The 60B.75 fix exposed State B in `selectReasoningPattern()` (so `reasoningPattern === "decision"` now reaches the sufficiency template selection), but did not address the earlier deterministic closure gate in `apply-proposal.js`. The decision resolves before question formulation can occur.
|
||||
|
||||
### Structural impact:
|
||||
|
||||
- **Customer:** Resolves correctly → status = resolved, negative meaning preserved
|
||||
- **Decision:** Prematurely closes → status = resolved (should be unknown)
|
||||
- **Active target:** Gone → activeUnknownNodeId = null (decision was the only unknown)
|
||||
- **Question:** Never formulated → finalSelectedQuestion = null
|
||||
- **No new nodes/edges**
|
||||
|
||||
### Classification: C — DECISION CLOSES
|
||||
|
||||
The deterministic sufficiency closure in `apply-proposal.js` resolves the decision before the question-formulation path is reached. This is also **F — STRUCTURAL REGRESSION** because meaning loss does not occur, but the structural behavior (premature closure) prevents testing of State B's question selection.
|
||||
|
||||
## Comparison with 60B.74
|
||||
|
||||
| Field | 60B.74 | 60B.76 |
|
||||
|---|---|---|
|
||||
| Customer status | resolved | resolved ✓ |
|
||||
| Customer meaning preserved | yes | yes ✓ |
|
||||
| Decision status | **unknown** (kept open) | **resolved** (closed prematurely) ✗ |
|
||||
| activeUnknownNodeId | n_product_launch_decision | null |
|
||||
| finalSelectedQuestion | decision_threshold_outcome | null (never reached) |
|
||||
| Classification | B | C + F |
|
||||
|
||||
The decision closed in 60B.76 because `hasRemainingMaterialFactors()` correctly returns false after the customer factor resolves — and the deterministic closure gate fires *before* question formulation can evaluate sufficiency confirmation presence.
|
||||
|
||||
## What this proves
|
||||
|
||||
1. **The 60B.75 fix successfully exposes State B to reasoningPattern detection.** The decision-context path is now reachable.
|
||||
2. **But an earlier gate prevents reaching that path:** the `apply-proposal.js` deterministic closure fires before question formulation, closing the decision prematurely when no material factors remain.
|
||||
3. **The sufficiency confirmation check must either be lifted from deterministic closure or moved into the closure gate itself** — checking for explicit confirmation *before* auto-closure.
|
||||
|
||||
## What remains unproven
|
||||
|
||||
1. Whether State B's sufficiency template selection works correctly (question formulation is not reached).
|
||||
2. Whether removing the kind gate in `selectReasoningPattern` causes any unintended side effects when question formulation IS reached.
|
||||
|
||||
## Production code changed: NO
|
||||
## Prompt changed: NO
|
||||
## Schema changed: NO
|
||||
## Harness changed during experiment: NO
|
||||
## Vitest run: NO
|
||||
## Ollama calls: 1
|
||||
## Direct API calls: 0
|
||||
Reference in New Issue
Block a user