experiment: confirm negative closure live

This commit is contained in:
2026-08-14 11:16:58 +01:00
parent 54e2e2186b
commit 2394ad4c0c
2 changed files with 163 additions and 0 deletions
+4
View File
@@ -3014,3 +3014,7 @@ Experiment 60B.47 tested whether resolving the customer-signing factor with a **
--- ---
Experiment 60B.48 was a read-only code-path and contract diagnosis of why 60B.47's negative closure produced `updatedNodes` with both nodes at status=resolved but omitted the decision node from `resolvedUnknownNodeIds`. **Classification: E — MULTIPLE FACTORS (deterministic normalisation gap primary, prompt gap secondary, model output variance as symptom).** Root cause identified in `reconcileResolutionSemantics()` at lib/graph/apply-proposal.js:312-370. The function reconciles only one direction (`resolvedUnknownNodeIds → updatedNodes`) but never adds a node from `updatedNodes` into `resolvedUnknownNodeIds`. The prompt contains no explicit rule mandating the bidirectional structural tie between `newStatus="resolved"` and `resolvedUnknownNodeIds` membership. Validator at line 364-367 catches the mismatch but does not auto-fix it (only reports error). **Minimum corrective boundary: B — deterministic normalisation.** Add every unknown node updated to resolved into `resolvedUnknownNodeIds` inside `reconcileResolutionSemantics()` before validation (~4 lines of code). This preserves positive closure, makes negative closure structurally valid, and does not weaken semantic validation. No production code changed. 0 Ollama calls. Pure code inspection. Full analysis in docs/experiment-60b48.md. Experiment 60B.48 was a read-only code-path and contract diagnosis of why 60B.47's negative closure produced `updatedNodes` with both nodes at status=resolved but omitted the decision node from `resolvedUnknownNodeIds`. **Classification: E — MULTIPLE FACTORS (deterministic normalisation gap primary, prompt gap secondary, model output variance as symptom).** Root cause identified in `reconcileResolutionSemantics()` at lib/graph/apply-proposal.js:312-370. The function reconciles only one direction (`resolvedUnknownNodeIds → updatedNodes`) but never adds a node from `updatedNodes` into `resolvedUnknownNodeIds`. The prompt contains no explicit rule mandating the bidirectional structural tie between `newStatus="resolved"` and `resolvedUnknownNodeIds` membership. Validator at line 364-367 catches the mismatch but does not auto-fix it (only reports error). **Minimum corrective boundary: B — deterministic normalisation.** Add every unknown node updated to resolved into `resolvedUnknownNodeIds` inside `reconcileResolutionSemantics()` before validation (~4 lines of code). This preserves positive closure, makes negative closure structurally valid, and does not weaken semantic validation. No production code changed. 0 Ollama calls. Pure code inspection. Full analysis in docs/experiment-60b48.md.
---
Experiment 60B.56 was a live regression of negative customer-signing closure after the proposal-compatibility fix (54e2e21). **Classification: D — GRAPH CLOSES BUT CONVERSATION DOES NOT.** The enterprise-customer unknown resolved in place (unknown → resolved), negative meaning preserved ("customer will not sign" confirmed in resolution reason), zero addedNodes/edges, zero validation errors. However, `finalActiveUnknownNodeId = "n_product_launch_decision"` (non-null) and `finalSelectedQuestion` was non-null — the engine formulated a decision_threshold follow-up question ("What outcome would demonstrate enough value to justify launching?") targeting the parent decision node instead of closing it. The proposal-compatibility bug from 60B.47 is fixed (proposal accepted, HTTP 200), but the clean-closure contract (null → null when all dependencies resolve) remains unmet. `n_product_launch_decision` stays unknown despite having no remaining dependent unknowns — both options are known. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed. Full trace in docs/experiment-60b56.md.
+159
View File
@@ -0,0 +1,159 @@
# Experiment 60B.56 — Negative Customer-Signing Clean Closure (Live)
**Date:** 2026-08-14
**Branch:** `feature/closure-selection-reconciliation-v0.41`
**Head commit:** 54e2e21 fix(reasoning): reconcile closure selection state
## Objective
Does the negative customer-signing outcome now close cleanly live — i.e., does the full production runtime resolve the same customer factor and decision with no stale active target or follow-up question?
## Hypothesis (from committed deterministic fix)
```
updated unknown -> resolved
=> mirrored into resolvedUnknownNodeIds
selectedQuestion targeting same-turn resolved node
=> cleared before strict validation
if no genuine unresolved unknown remains
=> activeUnknownNodeId = null
=> selectedQuestion = null
```
## Input
- **Fixture:** `tests/fixtures/pre-anchored-product-launch-customer-signing.json`
- Pre-anchored state: decision (`n_product_launch_decision`) in unknown status; enterprise customer signing (`n_enterprise_customer_signing`) in unknown status, activeUnknownNodeId = n_enterprise_customer_signing.
- **Answer:** "No. The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received. There are no other material uncertainties between launching this year and waiting twelve months."
## Configured environment
- **Model:** qwen-claude:latest
- **Ollama base URL:** http://192.168.1.111:11434
- **Confidence Engine base URL:** http://127.0.0.1:3000
## Run
```bash
FIXTURE_MODE=updateOnly \
FIXTURE_PATH=tests/fixtures/pre-anchored-product-launch-customer-signing.json \
ANSWER_2="No. The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received. There are no other material uncertainties between launching this year and waiting twelve months." \
CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \
node scripts/reproduce-multi-turn-investigation.mjs
```
- **startCalls:** 0
- **updateCalls:** 1
- **totalCalls:** 1
- **Retries:** 0
## Results
### Proposal accepted: YES (HTTP 200)
### updatedNodes:
```json
[{"nodeId":"n_enterprise_customer_signing","previousStatus":"unknown","newStatus":"resolved","previousValue":null,"newValue":null,"reason":"User confirmed in writing the customer will not sign if launched this year, resolving the active material uncertainty."}]
```
### resolvedUnknownNodeIds:
```json
["n_enterprise_customer_signing"]
```
### addedNodes:
```json
[]
```
### addedEdges:
```json
[]
```
### customer node final state:
- `n_enterprise_customer_signing`: status = **resolved**
### customer resolution meaning:
"User confirmed in writing the customer will not sign if launched this year, resolving the active material uncertainty." → Negative meaning **preserved**.
### decision node final state:
- `n_product_launch_decision`: status = **unknown** (still open)
### launch option final state:
- `opt_launch_this_year`: status = known
### wait option final state:
- `opt_wait_twelve_months`: status = known
### DIRECT CLOSURE METADATA
```
finalActiveUnknownNodeId: "n_product_launch_decision"
finalSelectedQuestion: {"nodeId":"n_product_launch_decision","question":"What outcome would demonstrate enough value to justify launching?","reason":"Formulated from graph context using the decision_threshold investigation strategy.",...}
```
## Assessment
### Customer factor: RESOLVED IN PLACE
The enterprise-customer-signing node was updated in place from `unknown``resolved`.
### Negative meaning: PRESERVED
The resolution reason explicitly states "customer will not sign" — the negative meaning is intact.
### Decision state: KEPT OPEN FOR SPECIFIC MATERIAL REASON
`n_product_launch_decision` remains `status=unknown` with `activeUnknownNodeId = n_product_launch_decision` and a non-null `selectedQuestion` targeting it. The fix's goal of closing the decision when all its dependency unknowns resolve was **not achieved**.
### Identity preservation:
- Decision node: PRESERVED
- Launch option: PRESERVED
- Wait option: PRESERVED
### Active lifecycle: GENUINE UNRESOLVED TARGET (but arguably stale)
`n_product_launch_decision` is still the active target. It has no remaining dependent unknowns — both `opt_launch_this_year` and `opt_wait_twelve_months` are known. Its resolution depends on evaluating the remaining evidence, which was the point of having the customer-signing unknown as a dependency.
### Final question: SPECIFIC MATERIAL FOLLOW-UP
The engine formulated a decision_threshold question ("What outcome would demonstrate enough value to justify launching?") targeting `n_product_launch_decision`.
### New uncertainty discipline: NONE (no new nodes created)
## 60B.47 comparison
| Field | 60B.47 | 60B.56 |
|---|---|---|
| Proposal accepted | NO (422 proposal_compatibility) | YES |
| resolvedUnknownNodeIds | UNAVAILABLE | ["n_enterprise_customer_signing"] |
| Customer final state | UNAVAILABLE | RESOLVED |
| Decision final state | UNAVAILABLE | UNKNOWN (kept open) |
| finalActiveUnknownNodeId | UNAVAILABLE | "n_product_launch_decision" |
| finalSelectedQuestion | UNAVAILABLE | non-null (decision_threshold) |
**Progress from 60B.47 → 60B.56:** The proposal-compatibility validation bug is fixed — the update is accepted. However, the clean-closure contract was not met.
## Classification: D — GRAPH CLOSES BUT CONVERSATION DOES NOT
The customer-signing factor resolves correctly in place, and negative meaning is preserved. No nodes or edges are added. But `finalActiveUnknownNodeId` is non-null (`"n_product_launch_decision"`) and `finalSelectedQuestion` is non-null (a decision_threshold question). The graph-level closure of the dependency succeeded, but the parent decision node was not resolved — it remains open with a new follow-up question rather than closing.
## What this proves
1. **The proposal-compatibility validation bug is fixed.** Experiment 60B.47's 422 rejection no longer occurs.
2. **Customer-signing resolves in place** with the correct status transition and meaning preserved.
3. **No spurious graph mutations** — zero addedNodes, zero addedEdges.
## What remains weak or unproven
1. **Decision-node auto-resolution when all dependencies resolve.** The deterministic fix's primary goal was to close `n_product_launch_decision` when its only dependency (`n_enterprise_customer_signing`) resolves. This did not happen.
2. **selectedQuestion handling after full resolution.** When the sole unresolved unknown in a decision context is resolved, the system should produce null for both `activeUnknownNodeId` and `selectedQuestion`. Instead, it generated a new investigation question targeting the decision node itself.
3. **The clean-closure contract** (null → null on all known options with no remaining unknowns) remains unverified in live runs.
## Production code changed: NO
## Prompt changed: NO
## Validator changed: NO
## Schema changed: NO
## Harness changed during experiment: NO
## Vitest run: NO
## Ollama calls: 1 MAXIMUM
## Direct API calls: 0
## Dev server disturbed: NO