233 lines
10 KiB
Markdown
233 lines
10 KiB
Markdown
# Experiment 60B.71 — No-Confirmation Premature-Closure Guard (Live)
|
|
|
|
**Date:** 2026-08-14
|
|
**Branch:** `feature/decision-sufficiency-module-v0.44`
|
|
**Head commit:** a5b71ad experiment: map remaining apply proposal boundaries
|
|
|
|
## Objective
|
|
|
|
Does the engine avoid premature decision closure when the final factor resolves but the user does NOT explicitly confirm that no other material uncertainty remains?
|
|
|
|
## Hypothesis
|
|
|
|
When `hasRemainingMaterialFactors(decisionId, graph) === false` BUT raw user answer lacks explicit no-further-uncertainty confirmation:
|
|
|
|
```
|
|
parent decision: should remain unresolved (unknown)
|
|
activeUnknownNodeId: should become non-null (the decision itself)
|
|
selectedQuestion: should be non-null and materially specific
|
|
no deterministic closure should fire
|
|
```
|
|
|
|
## Context
|
|
|
|
This is the inverse boundary of experiment 60B.66, which verified that explicit confirmation + resolved factors → deterministic closure.
|
|
|
|
**60B.66 input:** "No. The enterprise customer has now confirmed... There are no other material uncertainties between launching this year and waiting twelve months."
|
|
**60B.71 input:** "The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received."
|
|
|
|
The omission phrase is: "There are no other material uncertainties..." — deliberately absent.
|
|
|
|
## Configured environment
|
|
|
|
- **Model:** qwen-claude:latest
|
|
- **Ollama base URL:** http://192.168.1.111:11434
|
|
- **Confidence Engine base URL:** http://127.0.0.1:3000
|
|
|
|
## Input
|
|
|
|
- **Fixture:** `tests/fixtures/pre-anchored-product-launch-customer-signing.json`
|
|
- Pre-anchored state: decision (`n_product_launch_decision`) unknown; customer signing (`n_enterprise_customer_signing`) unknown, activeUnknownNodeId = n_enterprise_customer_signing.
|
|
- **Answer (exact, no paraphrase):** "The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received."
|
|
|
|
## Run
|
|
|
|
```bash
|
|
FIXTURE_MODE=updateOnly \
|
|
FIXTURE_PATH=tests/fixtures/pre-anchored-product-launch-customer-signing.json \
|
|
ANSWER_2="The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received." \
|
|
CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \
|
|
node scripts/reproduce-multi-turn-investigation.mjs
|
|
```
|
|
|
|
- **startCalls:** 0
|
|
- **updateCalls:** 1
|
|
- **totalCalls:** 1
|
|
- **Retries:** 0
|
|
|
|
## Results
|
|
|
|
### Proposal accepted: YES (HTTP 200)
|
|
|
|
### updatedNodes:
|
|
```json
|
|
[
|
|
{
|
|
"nodeId": "n_enterprise_customer_signing",
|
|
"previousStatus": "unknown",
|
|
"newStatus": "resolved",
|
|
"previousValue": null,
|
|
"newValue": null,
|
|
"reason": "Enterprise customer confirmed in writing they will not sign; the uncertainty is fully resolved."
|
|
}
|
|
]
|
|
```
|
|
|
|
### resolvedUnknownNodeIds:
|
|
```json
|
|
["n_enterprise_customer_signing"]
|
|
```
|
|
|
|
**Note:** `n_product_launch_decision` is NOT in resolvedUnknownNodeIds. Only the customer factor was resolved.
|
|
|
|
### addedNodes:
|
|
```json
|
|
[
|
|
{
|
|
"id": "n_revised_launch_year_revenue",
|
|
"label": "Revised first-year expected revenue for launch this year",
|
|
"description": "Expected additional recurring revenue drops to approximately £500k per year after the enterprise customer confirmed they will not sign, because it directly updates the financial baseline needed to compare against waiting.",
|
|
"kind": "observation",
|
|
"status": "known",
|
|
"confidence": "high",
|
|
"value": 500000,
|
|
"unit": "GBP",
|
|
"dependsOn": ["n_enterprise_customer_signing"],
|
|
"affects": ["opt_launch_this_year", "n_product_launch_decision"]
|
|
}
|
|
]
|
|
```
|
|
|
|
### addedEdges:
|
|
```json
|
|
[
|
|
{
|
|
"id": "e-customer-confirmation-to-revenue",
|
|
"fromNodeId": "n_revised_launch_year_revenue",
|
|
"toNodeId": "opt_launch_this_year",
|
|
"relationship": "supports"
|
|
}
|
|
]
|
|
```
|
|
|
|
### Customer node final state:
|
|
- `n_enterprise_customer_signing`: status = **resolved**, value = null (no explicit newValue set; meaning carried in reason text)
|
|
|
|
### Customer resolution meaning:
|
|
"Enterprise customer confirmed in writing they will not sign" → Negative meaning **PRESERVED** in reason text. Note: unlike 60B.66 where `newValue = "confirmed_no_signing"`, here the model chose to leave newValue as null while preserving the negative meaning in the reason string.
|
|
|
|
### Decision node final state:
|
|
- `n_product_launch_decision`: status = **unknown** (UNRESOLVED) — NOT in updatedNodes, NOT in resolvedUnknownNodeIds
|
|
|
|
### Launch option final state:
|
|
- `opt_launch_this_year`: status = known (unchanged; its description was not mutated in this pass)
|
|
|
|
### Wait option final state:
|
|
- `opt_wait_twelve_months`: status = known (unchanged)
|
|
|
|
### DIRECT CLOSURE METADATA
|
|
|
|
```
|
|
finalActiveUnknownNodeId: "n_product_launch_decision"
|
|
finalSelectedQuestion: {
|
|
"nodeId": "n_product_launch_decision",
|
|
"question": "What outcome would demonstrate enough value to justify launching?",
|
|
"reason": "Formulated from graph context using the decision_threshold investigation strategy.",
|
|
"strategy": "decision_threshold"
|
|
}
|
|
```
|
|
|
|
## Proposal Ownership
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Parent decision in model proposal updatedNodes | NO |
|
|
| Parent decision terminal in model proposal | NO |
|
|
| Parent decision in proposal resolvedUnknownNodeIds | NO |
|
|
|
|
The model did NOT close the decision in its proposal. The decision remained unknown throughout the entire production pipeline — neither the model nor deterministic closure closed it.
|
|
|
|
## Assessment
|
|
|
|
### Customer factor: RESOLVED IN PLACE
|
|
|
|
`n_enterprise_customer_signing` updated from `unknown` → `resolved`. Correct.
|
|
|
|
### Negative meaning: PRESERVED
|
|
|
|
The reason text explicitly states "Enterprise customer confirmed in writing they will not sign." The negative outcome is preserved. Note the newValue is null (not "confirmed_no_signing" as in 60B.66) — this is a minor semantic drift in value encoding but does not weaken the meaning.
|
|
|
|
### Decision outcome: KEPT OPEN
|
|
|
|
Decision `n_product_launch_decision` remains status `unknown`. It was neither closed by model proposal mutation nor by deterministic sufficiency closure. This is the correct conservative outcome when raw confirmation is absent.
|
|
|
|
### Active lifecycle: GENUINE UNRESOLVED TARGET
|
|
|
|
`finalActiveUnknownNodeId = "n_product_launch_decision"` — the decision itself becomes the active target because its single material factor has been resolved but no explicit confirmation was given. This is genuine unresolved state, not stale.
|
|
|
|
### Final question: SPECIFIC MATERIAL FOLLOW-UP
|
|
|
|
`"What outcome would demonstrate enough value to justify launching?"` — A decision_threshold strategy question specifically targeted at `n_product_launch_decision`. It asks what the customer must see to justify the launch, which directly engages with the remaining unresolved comparison that the resolution of the customer factor has revealed. This is materially specific (not generic continuation).
|
|
|
|
### Additional observation: new structural element introduced by model
|
|
|
|
The model created a new observation node `n_revised_launch_year_revenue` (£500k/year revised revenue) derived from the customer's statement about losing £700k enterprise revenue against the original £1.2M expected. This is not new material uncertainty — it's quantified financial context for the remaining decision. Its status is known, and it feeds into both the launch option and the decision node.
|
|
|
|
## 60B.66 → 60B.71 comparison
|
|
|
|
| Field | 60B.66 (confirmation PRESENT) | 60B.71 (confirmation ABSENT) |
|
|
|---|---|---|
|
|
| Customer status | unknown→resolved | unknown→resolved |
|
|
| Customer value | confirmed_no_signing | null (meaning in reason only) |
|
|
| Decision status | **resolved** (CLOSED) | **unknown** (KEPT OPEN) |
|
|
| Decision in updatedNodes | YES | NO |
|
|
| Decision in resolvedUnknownNodeIds | YES (both nodes listed) | NO (only customer node) |
|
|
| addedNodes | [] | [n_revised_launch_year_revenue] |
|
|
| addedEdges | [] | [e-customer-confirmation-to-revenue] |
|
|
| finalActiveUnknownNodeId | null | "n_product_launch_decision" |
|
|
| finalSelectedQuestion | null | non-null (decision_threshold) |
|
|
|
|
## Classification: E — NEW MATERIAL FACTOR IDENTIFIED
|
|
|
|
The decision remains open and the model identifies a specific consequential factor (revised revenue observation) and produces a materially specific follow-up question targeting the unresolved decision.
|
|
|
|
Additionally, **Classification A criteria are also met**:
|
|
- Customer resolves correctly ✓
|
|
- Negative meaning preserved ✓
|
|
- Decision remains unresolved ✓
|
|
- No deterministic closure without confirmation ✓
|
|
- Active target is genuine unresolved state (decision itself) ✓
|
|
|
|
The distinguishing feature that makes E primary is the introduction of a new observation node as material context for the decision evaluation, plus the specific material follow-up.
|
|
|
|
## What this proves
|
|
|
|
1. **The no-confirmation guard works live.** The parent decision remains open when raw user answer lacks explicit no-further-uncertainty confirmation. This confirms experiment 60B.71's core hypothesis.
|
|
|
|
2. **No deterministic closure without confirmation — even post-refactor.** The extracted `decision-sufficiency.js` module correctly does not close the decision without explicit sufficiency confirmation, matching the pre-refactor baseline (60B.66) inverse case.
|
|
|
|
3. **The model generates materially specific continuation when guard triggers.** Rather than generic "keep thinking" question, it formulates a decision_threshold question about what value demonstration would justify launching.
|
|
|
|
4. **Model introduces quantified financial observation node** rather than duplicating or inventing uncertainty. This is constructive reasoning support, not spurious structural change.
|
|
|
|
## What remains unproven
|
|
|
|
1. **Model behavior under other factor resolution patterns** — this test covers only the enterprise-customer-signing case.
|
|
2. **Whether explicit confirmation + no remaining factors still closes correctly post-refactor** — that's 60B.66 (previously verified).
|
|
3. **Multi-factor scenarios where some confirm but others don't** — single-factor resolution tested here.
|
|
|
|
## Behavioural baseline
|
|
|
|
```
|
|
NO-CONFIRMATION / DECISION-SUFFICIENCY THREAD:
|
|
GUARD CONFIRMED — PARENT REMAINS UNKNOWN, ACTIVE TARGET BECOMES DECISION ITSELF, SPECIFIC MATERIAL FOLLOW-UP GENERATED
|
|
```
|
|
|
|
## Production code changed: NO
|
|
## Prompt changed: NO
|
|
## Schema changed: NO
|
|
## Harness changed during experiment: NO
|
|
## Vitest run: NO
|
|
## Ollama calls: 1
|
|
## Direct API calls: 0
|