experiment: establish live decision closure baseline
This commit is contained in:
@@ -3389,3 +3389,25 @@ Smallest zero-semantic-change refactor: extract all decision-sufficiency evaluat
|
||||
### VITEST RUN: NO
|
||||
### JEST RUN: NO
|
||||
### WATCHMAN USED: NO
|
||||
|
||||
---
|
||||
|
||||
### EXPERIMENT 60B.66 — LIVE DECISION-CLOSURE REGRESSION (PASS)
|
||||
|
||||
**Date:** 2026-08-14
|
||||
**Branch:** `feature/decision-closure-integration-v0.43`
|
||||
**Experiment commit:** bce05f7 feat(reasoning): integrate explicit decision-sufficiency closure (60B.64)
|
||||
**Result:** A — LIVE REASONING THREAD CLOSED
|
||||
|
||||
The committed 60B.64 deterministic closure rule was verified live against the exact negative customer-signing case from 60B.56.
|
||||
|
||||
Key outcomes:
|
||||
- Decision `n_product_launch_decision` resolved unknown→resolved (was KEPT OPEN in 60B.56)
|
||||
- `finalActiveUnknownNodeId = null`, `finalSelectedQuestion = null` (both were non-null in 60B.56)
|
||||
- Customer factor resolves in place with negative meaning preserved
|
||||
- Zero addedNodes, zero addedEdges
|
||||
- Decision identity preserved; both options preserved
|
||||
|
||||
Pre-refactor behavioural baseline established.
|
||||
Next structural task may be zero-semantic-change decision-sufficiency extraction.
|
||||
|
||||
|
||||
@@ -0,0 +1,209 @@
|
||||
# Experiment 60B.66 — Live Decision-Closure Regression (60B.64 Fix)
|
||||
|
||||
**Date:** 2026-08-14
|
||||
**Branch:** `feature/decision-closure-integration-v0.43`
|
||||
**Head commit:** bce05f7 feat(reasoning): integrate explicit decision-sufficiency closure (60B.64)
|
||||
|
||||
## Objective
|
||||
|
||||
Does the committed 60B.64 production path now close the exact 60B.56 negative customer-signing case with no stale active target, no follow-up question, no new uncertainty, and no invented recommendation direction?
|
||||
|
||||
## Hypothesis
|
||||
|
||||
```
|
||||
hasRemainingMaterialFactors(decisionId, graph) === false
|
||||
AND
|
||||
raw user answer explicitly confirms no other material uncertainty remains
|
||||
=> resolve the existing parent decision before another question is selected
|
||||
|
||||
Result:
|
||||
customer factor = resolved
|
||||
decision = resolved
|
||||
activeUnknownNodeId = null
|
||||
selectedQuestion = null
|
||||
```
|
||||
|
||||
## Configured environment
|
||||
|
||||
- **Model:** qwen-claude:latest
|
||||
- **Ollama base URL:** http://192.168.1.111:11434
|
||||
- **Confidence Engine base URL:** http://127.0.0.1:3000
|
||||
|
||||
## Input
|
||||
|
||||
- **Fixture:** `tests/fixtures/pre-anchored-product-launch-customer-signing.json`
|
||||
- Pre-anchored state: decision (`n_product_launch_decision`) in unknown status; enterprise customer signing (`n_enterprise_customer_signing`) in unknown status, activeUnknownNodeId = n_enterprise_customer_signing.
|
||||
- **Answer:** "No. The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received. There are no other material uncertainties between launching this year and waiting twelve months."
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
FIXTURE_MODE=updateOnly \
|
||||
FIXTURE_PATH=tests/fixtures/pre-anchored-product-launch-customer-signing.json \
|
||||
ANSWER_2="No. The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received. There are no other material uncertainties between launching this year and waiting twelve months." \
|
||||
CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \
|
||||
node scripts/reproduce-multi-turn-investigation.mjs
|
||||
```
|
||||
|
||||
- **startCalls:** 0
|
||||
- **updateCalls:** 1
|
||||
- **totalCalls:** 1
|
||||
- **Retries:** 0
|
||||
|
||||
## Results
|
||||
|
||||
### Proposal accepted: YES (HTTP 200)
|
||||
|
||||
### updatedNodes:
|
||||
```json
|
||||
[
|
||||
{
|
||||
"nodeId": "n_enterprise_customer_signing",
|
||||
"previousStatus": "unknown",
|
||||
"newStatus": "resolved",
|
||||
"newValue": "confirmed_no_signing",
|
||||
"reason": "User explicitly confirmed in writing the enterprise customer will not sign, resolving this material uncertainty."
|
||||
},
|
||||
{
|
||||
"nodeId": "opt_launch_this_year",
|
||||
"previousStatus": "known",
|
||||
"newStatus": "known",
|
||||
"newValue": "Revised financial impact: ~£500k/year expected additional recurring revenue (excluding the confirmed lost £700k enterprise customer), £300k one-off launch/support cost.",
|
||||
"reason": "Update option description to reflect the resolved financial consequence of the now-resolved unknown."
|
||||
},
|
||||
{
|
||||
"nodeId": "n_product_launch_decision",
|
||||
"previousStatus": "unknown",
|
||||
"newStatus": "resolved",
|
||||
"newValue": null,
|
||||
"reason": "All represented material factors resolved and raw user answer explicitly confirmed no further material uncertainty remains."
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
### resolvedUnknownNodeIds:
|
||||
```json
|
||||
["n_enterprise_customer_signing", "n_product_launch_decision"]
|
||||
```
|
||||
|
||||
### addedNodes:
|
||||
```json
|
||||
[]
|
||||
```
|
||||
|
||||
### addedEdges:
|
||||
```json
|
||||
[]
|
||||
```
|
||||
|
||||
### structuralActionRequired: null
|
||||
|
||||
### Customer node final state:
|
||||
- `n_enterprise_customer_signing`: status = **resolved**, value = confirmed_no_signing
|
||||
|
||||
### Customer resolution meaning:
|
||||
"User explicitly confirmed in writing the enterprise customer will not sign, resolving this material uncertainty." → Negative meaning **preserved**.
|
||||
|
||||
### Decision node final state:
|
||||
- `n_product_launch_decision`: status = **resolved** (CLOSED)
|
||||
|
||||
### Launch option final state:
|
||||
- `opt_launch_this_year`: status = known
|
||||
|
||||
### Wait option final state:
|
||||
- `opt_wait_twelve_months`: status = known
|
||||
|
||||
### DIRECT CLOSURE METADATA
|
||||
|
||||
```
|
||||
finalActiveUnknownNodeId: null
|
||||
finalSelectedQuestion: null
|
||||
```
|
||||
|
||||
## Assessment
|
||||
|
||||
### Customer factor: RESOLVED IN PLACE
|
||||
The enterprise-customer-signing node was updated in place from `unknown` → `resolved` with value `confirmed_no_signing`.
|
||||
|
||||
### Negative meaning: PRESERVED
|
||||
The resolution reason and newValue ("confirmed_no_signing") both explicitly preserve the negative meaning — the customer will not sign.
|
||||
|
||||
### Parent decision: RESOLVED IN PLACE (KNOWN/CLOSED)
|
||||
`n_product_launch_decision` transitioned from `unknown` → `resolved`. The 60B.64 deterministic closure rule fired correctly: all material factors resolved + raw user answer explicitly confirmed no further uncertainty => decision closed in place.
|
||||
|
||||
### Identity preservation:
|
||||
- Decision node: PRESERVED
|
||||
- Launch option: PRESERVED
|
||||
- Wait option: PRESERVED
|
||||
|
||||
### Active lifecycle: NULL — CLEARED
|
||||
`finalActiveUnknownNodeId` is directly `null`. No stale or genuine unresolved target remains.
|
||||
|
||||
### Final question: NULL — DECISION COMPLETE
|
||||
`finalSelectedQuestion` is directly `null`. No continuation question was generated.
|
||||
|
||||
### New uncertainty discipline: NONE (no new nodes, no new edges)
|
||||
|
||||
### Recommendation direction: NONE (not inventoried by this run)
|
||||
|
||||
## 60B.56 → 60B.66 comparison
|
||||
|
||||
| Field | 60B.56 (FAILURE) | 60B.66 (PASS) |
|
||||
|---|---|---|
|
||||
| Proposal accepted | YES | YES |
|
||||
| Customer status | unknown→resolved | unknown→resolved |
|
||||
| Customer meaning | PRESERVED | PRESERVED |
|
||||
| Decision status | **unknown** (KEPT OPEN) | **resolved** (CLOSED) |
|
||||
| finalActiveUnknownNodeId | "n_product_launch_decision" | **null** |
|
||||
| finalSelectedQuestion | non-null decision_threshold | **null** |
|
||||
| addedNodes | [] | [] |
|
||||
| addedEdges | [] | [] |
|
||||
| resolvedUnknownNodeIds | ["n_enterprise_customer_signing"] | ["n_enterprise_customer_signing", "n_product_launch_decision"] |
|
||||
|
||||
**Progress from 60B.56 → 60B.66:** The clean-closure contract is now met. The decision node auto-resolves when all its dependency unknowns resolve and the user explicitly confirms no further material uncertainty remains. Both `activeUnknownNodeId` and `selectedQuestion` are null.
|
||||
|
||||
## Classification: A — LIVE REASONING THREAD CLOSED
|
||||
|
||||
All success criteria directly observed:
|
||||
- Proposal accepted ✓
|
||||
- Customer resolves in place ✓
|
||||
- Negative meaning preserved ✓
|
||||
- Decision resolves/closes in place ✓
|
||||
- Decision identity preserved ✓
|
||||
- Both options preserved ✓
|
||||
- addedNodes = [] ✓
|
||||
- addedEdges = [] ✓
|
||||
- finalActiveUnknownNodeId = null ✓
|
||||
- finalSelectedQuestion = null ✓
|
||||
|
||||
## What this proves
|
||||
|
||||
1. **The 60B.64 decision-sufficiency closure rule works live.** When `hasRemainingMaterialFactors(decisionId, graph) === false` AND the raw user answer explicitly confirms no other material uncertainty remains, the parent decision node is correctly resolved before any follow-up question is selected.
|
||||
2. **No regression in customer-signing factor resolution.** The negative meaning (customer will not sign) is preserved exactly.
|
||||
3. **Zero spurious mutations.** No nodes or edges added during this resolution pass.
|
||||
4. **The exact live failure that drove experiments 60B.47→60B.64 is now fixed.**
|
||||
|
||||
## Behavioural baseline
|
||||
|
||||
```
|
||||
CUSTOMER-SIGNING / DECISION-SUFFICIENCY THREAD:
|
||||
BEHAVIOURALLY CLOSED FOR CURRENT REGRESSION BASELINE
|
||||
```
|
||||
|
||||
This establishes a pre-refactor behavioural baseline. The behaviour may now be frozen as the baseline before any zero-semantic-change decision-sufficiency extraction refactor.
|
||||
|
||||
## What remains unproven
|
||||
|
||||
1. **All reasoning behaviour is complete** — NOT claimed. This experiment only covers the single customer-signing negative case on the product-launch decision graph.
|
||||
2. **All decision domains are proven** — NOT claimed. Other domains (savings, relocation, etc.) are not covered.
|
||||
3. **Production is universally correct** — NOT claimed.
|
||||
|
||||
## Production code changed: NO
|
||||
## Prompt changed: NO
|
||||
## Validator changed: NO
|
||||
## Schema changed: NO
|
||||
## Harness changed during experiment: NO
|
||||
## Vitest run: NO
|
||||
## Ollama calls: 1
|
||||
## Direct API calls: 0
|
||||
## Dev server disturbed: NO
|
||||
Reference in New Issue
Block a user