experiment: establish live decision closure baseline
This commit is contained in:
@@ -3389,3 +3389,25 @@ Smallest zero-semantic-change refactor: extract all decision-sufficiency evaluat
|
|||||||
### VITEST RUN: NO
|
### VITEST RUN: NO
|
||||||
### JEST RUN: NO
|
### JEST RUN: NO
|
||||||
### WATCHMAN USED: NO
|
### WATCHMAN USED: NO
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### EXPERIMENT 60B.66 — LIVE DECISION-CLOSURE REGRESSION (PASS)
|
||||||
|
|
||||||
|
**Date:** 2026-08-14
|
||||||
|
**Branch:** `feature/decision-closure-integration-v0.43`
|
||||||
|
**Experiment commit:** bce05f7 feat(reasoning): integrate explicit decision-sufficiency closure (60B.64)
|
||||||
|
**Result:** A — LIVE REASONING THREAD CLOSED
|
||||||
|
|
||||||
|
The committed 60B.64 deterministic closure rule was verified live against the exact negative customer-signing case from 60B.56.
|
||||||
|
|
||||||
|
Key outcomes:
|
||||||
|
- Decision `n_product_launch_decision` resolved unknown→resolved (was KEPT OPEN in 60B.56)
|
||||||
|
- `finalActiveUnknownNodeId = null`, `finalSelectedQuestion = null` (both were non-null in 60B.56)
|
||||||
|
- Customer factor resolves in place with negative meaning preserved
|
||||||
|
- Zero addedNodes, zero addedEdges
|
||||||
|
- Decision identity preserved; both options preserved
|
||||||
|
|
||||||
|
Pre-refactor behavioural baseline established.
|
||||||
|
Next structural task may be zero-semantic-change decision-sufficiency extraction.
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,209 @@
|
|||||||
|
# Experiment 60B.66 — Live Decision-Closure Regression (60B.64 Fix)
|
||||||
|
|
||||||
|
**Date:** 2026-08-14
|
||||||
|
**Branch:** `feature/decision-closure-integration-v0.43`
|
||||||
|
**Head commit:** bce05f7 feat(reasoning): integrate explicit decision-sufficiency closure (60B.64)
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Does the committed 60B.64 production path now close the exact 60B.56 negative customer-signing case with no stale active target, no follow-up question, no new uncertainty, and no invented recommendation direction?
|
||||||
|
|
||||||
|
## Hypothesis
|
||||||
|
|
||||||
|
```
|
||||||
|
hasRemainingMaterialFactors(decisionId, graph) === false
|
||||||
|
AND
|
||||||
|
raw user answer explicitly confirms no other material uncertainty remains
|
||||||
|
=> resolve the existing parent decision before another question is selected
|
||||||
|
|
||||||
|
Result:
|
||||||
|
customer factor = resolved
|
||||||
|
decision = resolved
|
||||||
|
activeUnknownNodeId = null
|
||||||
|
selectedQuestion = null
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configured environment
|
||||||
|
|
||||||
|
- **Model:** qwen-claude:latest
|
||||||
|
- **Ollama base URL:** http://192.168.1.111:11434
|
||||||
|
- **Confidence Engine base URL:** http://127.0.0.1:3000
|
||||||
|
|
||||||
|
## Input
|
||||||
|
|
||||||
|
- **Fixture:** `tests/fixtures/pre-anchored-product-launch-customer-signing.json`
|
||||||
|
- Pre-anchored state: decision (`n_product_launch_decision`) in unknown status; enterprise customer signing (`n_enterprise_customer_signing`) in unknown status, activeUnknownNodeId = n_enterprise_customer_signing.
|
||||||
|
- **Answer:** "No. The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received. There are no other material uncertainties between launching this year and waiting twelve months."
|
||||||
|
|
||||||
|
## Run
|
||||||
|
|
||||||
|
```bash
|
||||||
|
FIXTURE_MODE=updateOnly \
|
||||||
|
FIXTURE_PATH=tests/fixtures/pre-anchored-product-launch-customer-signing.json \
|
||||||
|
ANSWER_2="No. The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received. There are no other material uncertainties between launching this year and waiting twelve months." \
|
||||||
|
CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \
|
||||||
|
node scripts/reproduce-multi-turn-investigation.mjs
|
||||||
|
```
|
||||||
|
|
||||||
|
- **startCalls:** 0
|
||||||
|
- **updateCalls:** 1
|
||||||
|
- **totalCalls:** 1
|
||||||
|
- **Retries:** 0
|
||||||
|
|
||||||
|
## Results
|
||||||
|
|
||||||
|
### Proposal accepted: YES (HTTP 200)
|
||||||
|
|
||||||
|
### updatedNodes:
|
||||||
|
```json
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"nodeId": "n_enterprise_customer_signing",
|
||||||
|
"previousStatus": "unknown",
|
||||||
|
"newStatus": "resolved",
|
||||||
|
"newValue": "confirmed_no_signing",
|
||||||
|
"reason": "User explicitly confirmed in writing the enterprise customer will not sign, resolving this material uncertainty."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"nodeId": "opt_launch_this_year",
|
||||||
|
"previousStatus": "known",
|
||||||
|
"newStatus": "known",
|
||||||
|
"newValue": "Revised financial impact: ~£500k/year expected additional recurring revenue (excluding the confirmed lost £700k enterprise customer), £300k one-off launch/support cost.",
|
||||||
|
"reason": "Update option description to reflect the resolved financial consequence of the now-resolved unknown."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"nodeId": "n_product_launch_decision",
|
||||||
|
"previousStatus": "unknown",
|
||||||
|
"newStatus": "resolved",
|
||||||
|
"newValue": null,
|
||||||
|
"reason": "All represented material factors resolved and raw user answer explicitly confirmed no further material uncertainty remains."
|
||||||
|
}
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
### resolvedUnknownNodeIds:
|
||||||
|
```json
|
||||||
|
["n_enterprise_customer_signing", "n_product_launch_decision"]
|
||||||
|
```
|
||||||
|
|
||||||
|
### addedNodes:
|
||||||
|
```json
|
||||||
|
[]
|
||||||
|
```
|
||||||
|
|
||||||
|
### addedEdges:
|
||||||
|
```json
|
||||||
|
[]
|
||||||
|
```
|
||||||
|
|
||||||
|
### structuralActionRequired: null
|
||||||
|
|
||||||
|
### Customer node final state:
|
||||||
|
- `n_enterprise_customer_signing`: status = **resolved**, value = confirmed_no_signing
|
||||||
|
|
||||||
|
### Customer resolution meaning:
|
||||||
|
"User explicitly confirmed in writing the enterprise customer will not sign, resolving this material uncertainty." → Negative meaning **preserved**.
|
||||||
|
|
||||||
|
### Decision node final state:
|
||||||
|
- `n_product_launch_decision`: status = **resolved** (CLOSED)
|
||||||
|
|
||||||
|
### Launch option final state:
|
||||||
|
- `opt_launch_this_year`: status = known
|
||||||
|
|
||||||
|
### Wait option final state:
|
||||||
|
- `opt_wait_twelve_months`: status = known
|
||||||
|
|
||||||
|
### DIRECT CLOSURE METADATA
|
||||||
|
|
||||||
|
```
|
||||||
|
finalActiveUnknownNodeId: null
|
||||||
|
finalSelectedQuestion: null
|
||||||
|
```
|
||||||
|
|
||||||
|
## Assessment
|
||||||
|
|
||||||
|
### Customer factor: RESOLVED IN PLACE
|
||||||
|
The enterprise-customer-signing node was updated in place from `unknown` → `resolved` with value `confirmed_no_signing`.
|
||||||
|
|
||||||
|
### Negative meaning: PRESERVED
|
||||||
|
The resolution reason and newValue ("confirmed_no_signing") both explicitly preserve the negative meaning — the customer will not sign.
|
||||||
|
|
||||||
|
### Parent decision: RESOLVED IN PLACE (KNOWN/CLOSED)
|
||||||
|
`n_product_launch_decision` transitioned from `unknown` → `resolved`. The 60B.64 deterministic closure rule fired correctly: all material factors resolved + raw user answer explicitly confirmed no further uncertainty => decision closed in place.
|
||||||
|
|
||||||
|
### Identity preservation:
|
||||||
|
- Decision node: PRESERVED
|
||||||
|
- Launch option: PRESERVED
|
||||||
|
- Wait option: PRESERVED
|
||||||
|
|
||||||
|
### Active lifecycle: NULL — CLEARED
|
||||||
|
`finalActiveUnknownNodeId` is directly `null`. No stale or genuine unresolved target remains.
|
||||||
|
|
||||||
|
### Final question: NULL — DECISION COMPLETE
|
||||||
|
`finalSelectedQuestion` is directly `null`. No continuation question was generated.
|
||||||
|
|
||||||
|
### New uncertainty discipline: NONE (no new nodes, no new edges)
|
||||||
|
|
||||||
|
### Recommendation direction: NONE (not inventoried by this run)
|
||||||
|
|
||||||
|
## 60B.56 → 60B.66 comparison
|
||||||
|
|
||||||
|
| Field | 60B.56 (FAILURE) | 60B.66 (PASS) |
|
||||||
|
|---|---|---|
|
||||||
|
| Proposal accepted | YES | YES |
|
||||||
|
| Customer status | unknown→resolved | unknown→resolved |
|
||||||
|
| Customer meaning | PRESERVED | PRESERVED |
|
||||||
|
| Decision status | **unknown** (KEPT OPEN) | **resolved** (CLOSED) |
|
||||||
|
| finalActiveUnknownNodeId | "n_product_launch_decision" | **null** |
|
||||||
|
| finalSelectedQuestion | non-null decision_threshold | **null** |
|
||||||
|
| addedNodes | [] | [] |
|
||||||
|
| addedEdges | [] | [] |
|
||||||
|
| resolvedUnknownNodeIds | ["n_enterprise_customer_signing"] | ["n_enterprise_customer_signing", "n_product_launch_decision"] |
|
||||||
|
|
||||||
|
**Progress from 60B.56 → 60B.66:** The clean-closure contract is now met. The decision node auto-resolves when all its dependency unknowns resolve and the user explicitly confirms no further material uncertainty remains. Both `activeUnknownNodeId` and `selectedQuestion` are null.
|
||||||
|
|
||||||
|
## Classification: A — LIVE REASONING THREAD CLOSED
|
||||||
|
|
||||||
|
All success criteria directly observed:
|
||||||
|
- Proposal accepted ✓
|
||||||
|
- Customer resolves in place ✓
|
||||||
|
- Negative meaning preserved ✓
|
||||||
|
- Decision resolves/closes in place ✓
|
||||||
|
- Decision identity preserved ✓
|
||||||
|
- Both options preserved ✓
|
||||||
|
- addedNodes = [] ✓
|
||||||
|
- addedEdges = [] ✓
|
||||||
|
- finalActiveUnknownNodeId = null ✓
|
||||||
|
- finalSelectedQuestion = null ✓
|
||||||
|
|
||||||
|
## What this proves
|
||||||
|
|
||||||
|
1. **The 60B.64 decision-sufficiency closure rule works live.** When `hasRemainingMaterialFactors(decisionId, graph) === false` AND the raw user answer explicitly confirms no other material uncertainty remains, the parent decision node is correctly resolved before any follow-up question is selected.
|
||||||
|
2. **No regression in customer-signing factor resolution.** The negative meaning (customer will not sign) is preserved exactly.
|
||||||
|
3. **Zero spurious mutations.** No nodes or edges added during this resolution pass.
|
||||||
|
4. **The exact live failure that drove experiments 60B.47→60B.64 is now fixed.**
|
||||||
|
|
||||||
|
## Behavioural baseline
|
||||||
|
|
||||||
|
```
|
||||||
|
CUSTOMER-SIGNING / DECISION-SUFFICIENCY THREAD:
|
||||||
|
BEHAVIOURALLY CLOSED FOR CURRENT REGRESSION BASELINE
|
||||||
|
```
|
||||||
|
|
||||||
|
This establishes a pre-refactor behavioural baseline. The behaviour may now be frozen as the baseline before any zero-semantic-change decision-sufficiency extraction refactor.
|
||||||
|
|
||||||
|
## What remains unproven
|
||||||
|
|
||||||
|
1. **All reasoning behaviour is complete** — NOT claimed. This experiment only covers the single customer-signing negative case on the product-launch decision graph.
|
||||||
|
2. **All decision domains are proven** — NOT claimed. Other domains (savings, relocation, etc.) are not covered.
|
||||||
|
3. **Production is universally correct** — NOT claimed.
|
||||||
|
|
||||||
|
## Production code changed: NO
|
||||||
|
## Prompt changed: NO
|
||||||
|
## Validator changed: NO
|
||||||
|
## Schema changed: NO
|
||||||
|
## Harness changed during experiment: NO
|
||||||
|
## Vitest run: NO
|
||||||
|
## Ollama calls: 1
|
||||||
|
## Direct API calls: 0
|
||||||
|
## Dev server disturbed: NO
|
||||||
Reference in New Issue
Block a user