experiment: confirm negative closure live
This commit is contained in:
@@ -3014,3 +3014,7 @@ Experiment 60B.47 tested whether resolving the customer-signing factor with a **
|
|||||||
---
|
---
|
||||||
|
|
||||||
Experiment 60B.48 was a read-only code-path and contract diagnosis of why 60B.47's negative closure produced `updatedNodes` with both nodes at status=resolved but omitted the decision node from `resolvedUnknownNodeIds`. **Classification: E — MULTIPLE FACTORS (deterministic normalisation gap primary, prompt gap secondary, model output variance as symptom).** Root cause identified in `reconcileResolutionSemantics()` at lib/graph/apply-proposal.js:312-370. The function reconciles only one direction (`resolvedUnknownNodeIds → updatedNodes`) but never adds a node from `updatedNodes` into `resolvedUnknownNodeIds`. The prompt contains no explicit rule mandating the bidirectional structural tie between `newStatus="resolved"` and `resolvedUnknownNodeIds` membership. Validator at line 364-367 catches the mismatch but does not auto-fix it (only reports error). **Minimum corrective boundary: B — deterministic normalisation.** Add every unknown node updated to resolved into `resolvedUnknownNodeIds` inside `reconcileResolutionSemantics()` before validation (~4 lines of code). This preserves positive closure, makes negative closure structurally valid, and does not weaken semantic validation. No production code changed. 0 Ollama calls. Pure code inspection. Full analysis in docs/experiment-60b48.md.
|
Experiment 60B.48 was a read-only code-path and contract diagnosis of why 60B.47's negative closure produced `updatedNodes` with both nodes at status=resolved but omitted the decision node from `resolvedUnknownNodeIds`. **Classification: E — MULTIPLE FACTORS (deterministic normalisation gap primary, prompt gap secondary, model output variance as symptom).** Root cause identified in `reconcileResolutionSemantics()` at lib/graph/apply-proposal.js:312-370. The function reconciles only one direction (`resolvedUnknownNodeIds → updatedNodes`) but never adds a node from `updatedNodes` into `resolvedUnknownNodeIds`. The prompt contains no explicit rule mandating the bidirectional structural tie between `newStatus="resolved"` and `resolvedUnknownNodeIds` membership. Validator at line 364-367 catches the mismatch but does not auto-fix it (only reports error). **Minimum corrective boundary: B — deterministic normalisation.** Add every unknown node updated to resolved into `resolvedUnknownNodeIds` inside `reconcileResolutionSemantics()` before validation (~4 lines of code). This preserves positive closure, makes negative closure structurally valid, and does not weaken semantic validation. No production code changed. 0 Ollama calls. Pure code inspection. Full analysis in docs/experiment-60b48.md.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
Experiment 60B.56 was a live regression of negative customer-signing closure after the proposal-compatibility fix (54e2e21). **Classification: D — GRAPH CLOSES BUT CONVERSATION DOES NOT.** The enterprise-customer unknown resolved in place (unknown → resolved), negative meaning preserved ("customer will not sign" confirmed in resolution reason), zero addedNodes/edges, zero validation errors. However, `finalActiveUnknownNodeId = "n_product_launch_decision"` (non-null) and `finalSelectedQuestion` was non-null — the engine formulated a decision_threshold follow-up question ("What outcome would demonstrate enough value to justify launching?") targeting the parent decision node instead of closing it. The proposal-compatibility bug from 60B.47 is fixed (proposal accepted, HTTP 200), but the clean-closure contract (null → null when all dependencies resolve) remains unmet. `n_product_launch_decision` stays unknown despite having no remaining dependent unknowns — both options are known. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed. Full trace in docs/experiment-60b56.md.
|
||||||
|
|||||||
@@ -0,0 +1,159 @@
|
|||||||
|
# Experiment 60B.56 — Negative Customer-Signing Clean Closure (Live)
|
||||||
|
|
||||||
|
**Date:** 2026-08-14
|
||||||
|
**Branch:** `feature/closure-selection-reconciliation-v0.41`
|
||||||
|
**Head commit:** 54e2e21 fix(reasoning): reconcile closure selection state
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Does the negative customer-signing outcome now close cleanly live — i.e., does the full production runtime resolve the same customer factor and decision with no stale active target or follow-up question?
|
||||||
|
|
||||||
|
## Hypothesis (from committed deterministic fix)
|
||||||
|
|
||||||
|
```
|
||||||
|
updated unknown -> resolved
|
||||||
|
=> mirrored into resolvedUnknownNodeIds
|
||||||
|
|
||||||
|
selectedQuestion targeting same-turn resolved node
|
||||||
|
=> cleared before strict validation
|
||||||
|
|
||||||
|
if no genuine unresolved unknown remains
|
||||||
|
=> activeUnknownNodeId = null
|
||||||
|
=> selectedQuestion = null
|
||||||
|
```
|
||||||
|
|
||||||
|
## Input
|
||||||
|
|
||||||
|
- **Fixture:** `tests/fixtures/pre-anchored-product-launch-customer-signing.json`
|
||||||
|
- Pre-anchored state: decision (`n_product_launch_decision`) in unknown status; enterprise customer signing (`n_enterprise_customer_signing`) in unknown status, activeUnknownNodeId = n_enterprise_customer_signing.
|
||||||
|
- **Answer:** "No. The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received. There are no other material uncertainties between launching this year and waiting twelve months."
|
||||||
|
|
||||||
|
## Configured environment
|
||||||
|
|
||||||
|
- **Model:** qwen-claude:latest
|
||||||
|
- **Ollama base URL:** http://192.168.1.111:11434
|
||||||
|
- **Confidence Engine base URL:** http://127.0.0.1:3000
|
||||||
|
|
||||||
|
## Run
|
||||||
|
|
||||||
|
```bash
|
||||||
|
FIXTURE_MODE=updateOnly \
|
||||||
|
FIXTURE_PATH=tests/fixtures/pre-anchored-product-launch-customer-signing.json \
|
||||||
|
ANSWER_2="No. The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received. There are no other material uncertainties between launching this year and waiting twelve months." \
|
||||||
|
CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \
|
||||||
|
node scripts/reproduce-multi-turn-investigation.mjs
|
||||||
|
```
|
||||||
|
|
||||||
|
- **startCalls:** 0
|
||||||
|
- **updateCalls:** 1
|
||||||
|
- **totalCalls:** 1
|
||||||
|
- **Retries:** 0
|
||||||
|
|
||||||
|
## Results
|
||||||
|
|
||||||
|
### Proposal accepted: YES (HTTP 200)
|
||||||
|
|
||||||
|
### updatedNodes:
|
||||||
|
```json
|
||||||
|
[{"nodeId":"n_enterprise_customer_signing","previousStatus":"unknown","newStatus":"resolved","previousValue":null,"newValue":null,"reason":"User confirmed in writing the customer will not sign if launched this year, resolving the active material uncertainty."}]
|
||||||
|
```
|
||||||
|
|
||||||
|
### resolvedUnknownNodeIds:
|
||||||
|
```json
|
||||||
|
["n_enterprise_customer_signing"]
|
||||||
|
```
|
||||||
|
|
||||||
|
### addedNodes:
|
||||||
|
```json
|
||||||
|
[]
|
||||||
|
```
|
||||||
|
|
||||||
|
### addedEdges:
|
||||||
|
```json
|
||||||
|
[]
|
||||||
|
```
|
||||||
|
|
||||||
|
### customer node final state:
|
||||||
|
- `n_enterprise_customer_signing`: status = **resolved**
|
||||||
|
|
||||||
|
### customer resolution meaning:
|
||||||
|
"User confirmed in writing the customer will not sign if launched this year, resolving the active material uncertainty." → Negative meaning **preserved**.
|
||||||
|
|
||||||
|
### decision node final state:
|
||||||
|
- `n_product_launch_decision`: status = **unknown** (still open)
|
||||||
|
|
||||||
|
### launch option final state:
|
||||||
|
- `opt_launch_this_year`: status = known
|
||||||
|
|
||||||
|
### wait option final state:
|
||||||
|
- `opt_wait_twelve_months`: status = known
|
||||||
|
|
||||||
|
### DIRECT CLOSURE METADATA
|
||||||
|
|
||||||
|
```
|
||||||
|
finalActiveUnknownNodeId: "n_product_launch_decision"
|
||||||
|
finalSelectedQuestion: {"nodeId":"n_product_launch_decision","question":"What outcome would demonstrate enough value to justify launching?","reason":"Formulated from graph context using the decision_threshold investigation strategy.",...}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Assessment
|
||||||
|
|
||||||
|
### Customer factor: RESOLVED IN PLACE
|
||||||
|
The enterprise-customer-signing node was updated in place from `unknown` → `resolved`.
|
||||||
|
|
||||||
|
### Negative meaning: PRESERVED
|
||||||
|
The resolution reason explicitly states "customer will not sign" — the negative meaning is intact.
|
||||||
|
|
||||||
|
### Decision state: KEPT OPEN FOR SPECIFIC MATERIAL REASON
|
||||||
|
`n_product_launch_decision` remains `status=unknown` with `activeUnknownNodeId = n_product_launch_decision` and a non-null `selectedQuestion` targeting it. The fix's goal of closing the decision when all its dependency unknowns resolve was **not achieved**.
|
||||||
|
|
||||||
|
### Identity preservation:
|
||||||
|
- Decision node: PRESERVED
|
||||||
|
- Launch option: PRESERVED
|
||||||
|
- Wait option: PRESERVED
|
||||||
|
|
||||||
|
### Active lifecycle: GENUINE UNRESOLVED TARGET (but arguably stale)
|
||||||
|
`n_product_launch_decision` is still the active target. It has no remaining dependent unknowns — both `opt_launch_this_year` and `opt_wait_twelve_months` are known. Its resolution depends on evaluating the remaining evidence, which was the point of having the customer-signing unknown as a dependency.
|
||||||
|
|
||||||
|
### Final question: SPECIFIC MATERIAL FOLLOW-UP
|
||||||
|
The engine formulated a decision_threshold question ("What outcome would demonstrate enough value to justify launching?") targeting `n_product_launch_decision`.
|
||||||
|
|
||||||
|
### New uncertainty discipline: NONE (no new nodes created)
|
||||||
|
|
||||||
|
## 60B.47 comparison
|
||||||
|
|
||||||
|
| Field | 60B.47 | 60B.56 |
|
||||||
|
|---|---|---|
|
||||||
|
| Proposal accepted | NO (422 proposal_compatibility) | YES |
|
||||||
|
| resolvedUnknownNodeIds | UNAVAILABLE | ["n_enterprise_customer_signing"] |
|
||||||
|
| Customer final state | UNAVAILABLE | RESOLVED |
|
||||||
|
| Decision final state | UNAVAILABLE | UNKNOWN (kept open) |
|
||||||
|
| finalActiveUnknownNodeId | UNAVAILABLE | "n_product_launch_decision" |
|
||||||
|
| finalSelectedQuestion | UNAVAILABLE | non-null (decision_threshold) |
|
||||||
|
|
||||||
|
**Progress from 60B.47 → 60B.56:** The proposal-compatibility validation bug is fixed — the update is accepted. However, the clean-closure contract was not met.
|
||||||
|
|
||||||
|
## Classification: D — GRAPH CLOSES BUT CONVERSATION DOES NOT
|
||||||
|
|
||||||
|
The customer-signing factor resolves correctly in place, and negative meaning is preserved. No nodes or edges are added. But `finalActiveUnknownNodeId` is non-null (`"n_product_launch_decision"`) and `finalSelectedQuestion` is non-null (a decision_threshold question). The graph-level closure of the dependency succeeded, but the parent decision node was not resolved — it remains open with a new follow-up question rather than closing.
|
||||||
|
|
||||||
|
## What this proves
|
||||||
|
|
||||||
|
1. **The proposal-compatibility validation bug is fixed.** Experiment 60B.47's 422 rejection no longer occurs.
|
||||||
|
2. **Customer-signing resolves in place** with the correct status transition and meaning preserved.
|
||||||
|
3. **No spurious graph mutations** — zero addedNodes, zero addedEdges.
|
||||||
|
|
||||||
|
## What remains weak or unproven
|
||||||
|
|
||||||
|
1. **Decision-node auto-resolution when all dependencies resolve.** The deterministic fix's primary goal was to close `n_product_launch_decision` when its only dependency (`n_enterprise_customer_signing`) resolves. This did not happen.
|
||||||
|
2. **selectedQuestion handling after full resolution.** When the sole unresolved unknown in a decision context is resolved, the system should produce null for both `activeUnknownNodeId` and `selectedQuestion`. Instead, it generated a new investigation question targeting the decision node itself.
|
||||||
|
3. **The clean-closure contract** (null → null on all known options with no remaining unknowns) remains unverified in live runs.
|
||||||
|
|
||||||
|
## Production code changed: NO
|
||||||
|
## Prompt changed: NO
|
||||||
|
## Validator changed: NO
|
||||||
|
## Schema changed: NO
|
||||||
|
## Harness changed during experiment: NO
|
||||||
|
## Vitest run: NO
|
||||||
|
## Ollama calls: 1 MAXIMUM
|
||||||
|
## Direct API calls: 0
|
||||||
|
## Dev server disturbed: NO
|
||||||
Reference in New Issue
Block a user