126 lines
5.5 KiB
Markdown
126 lines
5.5 KiB
Markdown
# Experiment 60B.76 — Live Verification of Sufficiency Question Fix
|
|
|
|
**Date:** 2026-08-15
|
|
**Branch:** `feature/sufficiency-decision-detection-v0.46`
|
|
**Preceded by:** Experiment 60B.75 (fix applied + tests pass)
|
|
|
|
## Objective
|
|
|
|
Does the 60B.75 fix make the exact no-confirmation production case select the focused sufficiency confirmation/discovery question while keeping the decision open?
|
|
|
|
## Configured environment
|
|
|
|
- **Model:** qwen-claude:latest
|
|
- **Ollama base URL:** http://192.168.1.111:11434
|
|
- **Confidence Engine base URL:** http://127.0.0.1:3000
|
|
|
|
## Input
|
|
|
|
- **Fixture:** `tests/fixtures/pre-anchored-product-launch-customer-signing.json`
|
|
- Pre-anchored state: decision (`n_product_launch_decision`) unknown; customer signing (`n_enterprise_customer_signing`) unknown, activeUnknownNodeId = n_enterprise_customer_signing.
|
|
- **Answer (exact, no paraphrase):** "The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received."
|
|
- Explicit sufficiency confirmation: **NO**
|
|
|
|
## Run
|
|
|
|
```bash
|
|
FIXTURE_MODE=updateOnly \
|
|
FIXTURE_PATH=tests/fixtures/pre-anchored-product-launch-customer-signing.json \
|
|
ANSWER_2="The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received." \
|
|
CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \
|
|
node scripts/reproduce-multi-turn-investigation.mjs
|
|
```
|
|
|
|
- **startCalls:** 0
|
|
- **updateCalls:** 1
|
|
- **totalCalls:** 1
|
|
- **Retries:** 0
|
|
|
|
## Results
|
|
|
|
### Proposal accepted: YES (HTTP 200)
|
|
|
|
### updatedNodes:
|
|
```json
|
|
[
|
|
{
|
|
"nodeId": "n_enterprise_customer_signing",
|
|
"previousStatus": "unknown",
|
|
"newStatus": "resolved",
|
|
"reason": "Enterprise customer confirmed in writing they will not sign if we launch this year."
|
|
},
|
|
{
|
|
"nodeId": "n_product_launch_decision",
|
|
"previousStatus": "unknown",
|
|
"newStatus": "resolved",
|
|
"reason": "Material uncertainty resolved; evidence now sufficiently distinguishes net value of launch this year versus wait twelve months."
|
|
}
|
|
]
|
|
```
|
|
|
|
### resolvedUnknownNodeIds:
|
|
```json
|
|
["n_enterprise_customer_signing", "n_product_launch_decision"]
|
|
```
|
|
|
|
### addedNodes: []
|
|
### addedEdges: []
|
|
|
|
### structuralActionRequired: null
|
|
### finalActiveUnknownNodeId: null
|
|
### finalSelectedQuestion: null
|
|
|
|
## Analysis
|
|
|
|
The sufficiency confirmation question did **not** fire. Instead, the decision was closed deterministically during `apply-proposal.js` processing — before `formulateQuestion()` was ever called.
|
|
|
|
**Root cause:** The deterministic sufficiency-closure gate (line 1673 of `apply-proposal.js`) fires when:
|
|
1. All material uncertainties are resolved → `hasRemainingMaterialFactors()` returns false
|
|
2. This triggers automatic decision closure with reason "sufficient evidence distinguishes net value"
|
|
|
|
The 60B.75 fix exposed State B in `selectReasoningPattern()` (so `reasoningPattern === "decision"` now reaches the sufficiency template selection), but did not address the earlier deterministic closure gate in `apply-proposal.js`. The decision resolves before question formulation can occur.
|
|
|
|
### Structural impact:
|
|
|
|
- **Customer:** Resolves correctly → status = resolved, negative meaning preserved
|
|
- **Decision:** Prematurely closes → status = resolved (should be unknown)
|
|
- **Active target:** Gone → activeUnknownNodeId = null (decision was the only unknown)
|
|
- **Question:** Never formulated → finalSelectedQuestion = null
|
|
- **No new nodes/edges**
|
|
|
|
### Classification: C — DECISION CLOSES
|
|
|
|
The deterministic sufficiency closure in `apply-proposal.js` resolves the decision before the question-formulation path is reached. This is also **F — STRUCTURAL REGRESSION** because meaning loss does not occur, but the structural behavior (premature closure) prevents testing of State B's question selection.
|
|
|
|
## Comparison with 60B.74
|
|
|
|
| Field | 60B.74 | 60B.76 |
|
|
|---|---|---|
|
|
| Customer status | resolved | resolved ✓ |
|
|
| Customer meaning preserved | yes | yes ✓ |
|
|
| Decision status | **unknown** (kept open) | **resolved** (closed prematurely) ✗ |
|
|
| activeUnknownNodeId | n_product_launch_decision | null |
|
|
| finalSelectedQuestion | decision_threshold_outcome | null (never reached) |
|
|
| Classification | B | C + F |
|
|
|
|
The decision closed in 60B.76 because `hasRemainingMaterialFactors()` correctly returns false after the customer factor resolves — and the deterministic closure gate fires *before* question formulation can evaluate sufficiency confirmation presence.
|
|
|
|
## What this proves
|
|
|
|
1. **The 60B.75 fix successfully exposes State B to reasoningPattern detection.** The decision-context path is now reachable.
|
|
2. **But an earlier gate prevents reaching that path:** the `apply-proposal.js` deterministic closure fires before question formulation, closing the decision prematurely when no material factors remain.
|
|
3. **The sufficiency confirmation check must either be lifted from deterministic closure or moved into the closure gate itself** — checking for explicit confirmation *before* auto-closure.
|
|
|
|
## What remains unproven
|
|
|
|
1. Whether State B's sufficiency template selection works correctly (question formulation is not reached).
|
|
2. Whether removing the kind gate in `selectReasoningPattern` causes any unintended side effects when question formulation IS reached.
|
|
|
|
## Production code changed: NO
|
|
## Prompt changed: NO
|
|
## Schema changed: NO
|
|
## Harness changed during experiment: NO
|
|
## Vitest run: NO
|
|
## Ollama calls: 1
|
|
## Direct API calls: 0
|