From 2394ad4c0ccf4f21a2ffb39a1322b8de5d16a25b Mon Sep 17 00:00:00 2001 From: robbond Date: Fri, 14 Aug 2026 11:16:58 +0100 Subject: [PATCH] experiment: confirm negative closure live --- docs/current-handoff.md | 4 + docs/experiment-60b56.md | 159 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 163 insertions(+) create mode 100644 docs/experiment-60b56.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 3cd33fa..0954291 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -3014,3 +3014,7 @@ Experiment 60B.47 tested whether resolving the customer-signing factor with a ** --- Experiment 60B.48 was a read-only code-path and contract diagnosis of why 60B.47's negative closure produced `updatedNodes` with both nodes at status=resolved but omitted the decision node from `resolvedUnknownNodeIds`. **Classification: E — MULTIPLE FACTORS (deterministic normalisation gap primary, prompt gap secondary, model output variance as symptom).** Root cause identified in `reconcileResolutionSemantics()` at lib/graph/apply-proposal.js:312-370. The function reconciles only one direction (`resolvedUnknownNodeIds → updatedNodes`) but never adds a node from `updatedNodes` into `resolvedUnknownNodeIds`. The prompt contains no explicit rule mandating the bidirectional structural tie between `newStatus="resolved"` and `resolvedUnknownNodeIds` membership. Validator at line 364-367 catches the mismatch but does not auto-fix it (only reports error). **Minimum corrective boundary: B — deterministic normalisation.** Add every unknown node updated to resolved into `resolvedUnknownNodeIds` inside `reconcileResolutionSemantics()` before validation (~4 lines of code). This preserves positive closure, makes negative closure structurally valid, and does not weaken semantic validation. No production code changed. 0 Ollama calls. Pure code inspection. Full analysis in docs/experiment-60b48.md. + +--- + +Experiment 60B.56 was a live regression of negative customer-signing closure after the proposal-compatibility fix (54e2e21). **Classification: D — GRAPH CLOSES BUT CONVERSATION DOES NOT.** The enterprise-customer unknown resolved in place (unknown → resolved), negative meaning preserved ("customer will not sign" confirmed in resolution reason), zero addedNodes/edges, zero validation errors. However, `finalActiveUnknownNodeId = "n_product_launch_decision"` (non-null) and `finalSelectedQuestion` was non-null — the engine formulated a decision_threshold follow-up question ("What outcome would demonstrate enough value to justify launching?") targeting the parent decision node instead of closing it. The proposal-compatibility bug from 60B.47 is fixed (proposal accepted, HTTP 200), but the clean-closure contract (null → null when all dependencies resolve) remains unmet. `n_product_launch_decision` stays unknown despite having no remaining dependent unknowns — both options are known. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed. Full trace in docs/experiment-60b56.md. diff --git a/docs/experiment-60b56.md b/docs/experiment-60b56.md new file mode 100644 index 0000000..5e6cbf8 --- /dev/null +++ b/docs/experiment-60b56.md @@ -0,0 +1,159 @@ +# Experiment 60B.56 — Negative Customer-Signing Clean Closure (Live) + +**Date:** 2026-08-14 +**Branch:** `feature/closure-selection-reconciliation-v0.41` +**Head commit:** 54e2e21 fix(reasoning): reconcile closure selection state + +## Objective + +Does the negative customer-signing outcome now close cleanly live — i.e., does the full production runtime resolve the same customer factor and decision with no stale active target or follow-up question? + +## Hypothesis (from committed deterministic fix) + +``` +updated unknown -> resolved +=> mirrored into resolvedUnknownNodeIds + +selectedQuestion targeting same-turn resolved node +=> cleared before strict validation + +if no genuine unresolved unknown remains +=> activeUnknownNodeId = null +=> selectedQuestion = null +``` + +## Input + +- **Fixture:** `tests/fixtures/pre-anchored-product-launch-customer-signing.json` + - Pre-anchored state: decision (`n_product_launch_decision`) in unknown status; enterprise customer signing (`n_enterprise_customer_signing`) in unknown status, activeUnknownNodeId = n_enterprise_customer_signing. +- **Answer:** "No. The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received. There are no other material uncertainties between launching this year and waiting twelve months." + +## Configured environment + +- **Model:** qwen-claude:latest +- **Ollama base URL:** http://192.168.1.111:11434 +- **Confidence Engine base URL:** http://127.0.0.1:3000 + +## Run + +```bash +FIXTURE_MODE=updateOnly \ +FIXTURE_PATH=tests/fixtures/pre-anchored-product-launch-customer-signing.json \ +ANSWER_2="No. The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received. There are no other material uncertainties between launching this year and waiting twelve months." \ +CONFIDENCE_ENGINE_BASE_URL=http://127.0.0.1:3000 \ +node scripts/reproduce-multi-turn-investigation.mjs +``` + +- **startCalls:** 0 +- **updateCalls:** 1 +- **totalCalls:** 1 +- **Retries:** 0 + +## Results + +### Proposal accepted: YES (HTTP 200) + +### updatedNodes: +```json +[{"nodeId":"n_enterprise_customer_signing","previousStatus":"unknown","newStatus":"resolved","previousValue":null,"newValue":null,"reason":"User confirmed in writing the customer will not sign if launched this year, resolving the active material uncertainty."}] +``` + +### resolvedUnknownNodeIds: +```json +["n_enterprise_customer_signing"] +``` + +### addedNodes: +```json +[] +``` + +### addedEdges: +```json +[] +``` + +### customer node final state: +- `n_enterprise_customer_signing`: status = **resolved** + +### customer resolution meaning: +"User confirmed in writing the customer will not sign if launched this year, resolving the active material uncertainty." → Negative meaning **preserved**. + +### decision node final state: +- `n_product_launch_decision`: status = **unknown** (still open) + +### launch option final state: +- `opt_launch_this_year`: status = known + +### wait option final state: +- `opt_wait_twelve_months`: status = known + +### DIRECT CLOSURE METADATA + +``` +finalActiveUnknownNodeId: "n_product_launch_decision" +finalSelectedQuestion: {"nodeId":"n_product_launch_decision","question":"What outcome would demonstrate enough value to justify launching?","reason":"Formulated from graph context using the decision_threshold investigation strategy.",...} +``` + +## Assessment + +### Customer factor: RESOLVED IN PLACE +The enterprise-customer-signing node was updated in place from `unknown` → `resolved`. + +### Negative meaning: PRESERVED +The resolution reason explicitly states "customer will not sign" — the negative meaning is intact. + +### Decision state: KEPT OPEN FOR SPECIFIC MATERIAL REASON +`n_product_launch_decision` remains `status=unknown` with `activeUnknownNodeId = n_product_launch_decision` and a non-null `selectedQuestion` targeting it. The fix's goal of closing the decision when all its dependency unknowns resolve was **not achieved**. + +### Identity preservation: +- Decision node: PRESERVED +- Launch option: PRESERVED +- Wait option: PRESERVED + +### Active lifecycle: GENUINE UNRESOLVED TARGET (but arguably stale) +`n_product_launch_decision` is still the active target. It has no remaining dependent unknowns — both `opt_launch_this_year` and `opt_wait_twelve_months` are known. Its resolution depends on evaluating the remaining evidence, which was the point of having the customer-signing unknown as a dependency. + +### Final question: SPECIFIC MATERIAL FOLLOW-UP +The engine formulated a decision_threshold question ("What outcome would demonstrate enough value to justify launching?") targeting `n_product_launch_decision`. + +### New uncertainty discipline: NONE (no new nodes created) + +## 60B.47 comparison + +| Field | 60B.47 | 60B.56 | +|---|---|---| +| Proposal accepted | NO (422 proposal_compatibility) | YES | +| resolvedUnknownNodeIds | UNAVAILABLE | ["n_enterprise_customer_signing"] | +| Customer final state | UNAVAILABLE | RESOLVED | +| Decision final state | UNAVAILABLE | UNKNOWN (kept open) | +| finalActiveUnknownNodeId | UNAVAILABLE | "n_product_launch_decision" | +| finalSelectedQuestion | UNAVAILABLE | non-null (decision_threshold) | + +**Progress from 60B.47 → 60B.56:** The proposal-compatibility validation bug is fixed — the update is accepted. However, the clean-closure contract was not met. + +## Classification: D — GRAPH CLOSES BUT CONVERSATION DOES NOT + +The customer-signing factor resolves correctly in place, and negative meaning is preserved. No nodes or edges are added. But `finalActiveUnknownNodeId` is non-null (`"n_product_launch_decision"`) and `finalSelectedQuestion` is non-null (a decision_threshold question). The graph-level closure of the dependency succeeded, but the parent decision node was not resolved — it remains open with a new follow-up question rather than closing. + +## What this proves + +1. **The proposal-compatibility validation bug is fixed.** Experiment 60B.47's 422 rejection no longer occurs. +2. **Customer-signing resolves in place** with the correct status transition and meaning preserved. +3. **No spurious graph mutations** — zero addedNodes, zero addedEdges. + +## What remains weak or unproven + +1. **Decision-node auto-resolution when all dependencies resolve.** The deterministic fix's primary goal was to close `n_product_launch_decision` when its only dependency (`n_enterprise_customer_signing`) resolves. This did not happen. +2. **selectedQuestion handling after full resolution.** When the sole unresolved unknown in a decision context is resolved, the system should produce null for both `activeUnknownNodeId` and `selectedQuestion`. Instead, it generated a new investigation question targeting the decision node itself. +3. **The clean-closure contract** (null → null on all known options with no remaining unknowns) remains unverified in live runs. + +## Production code changed: NO +## Prompt changed: NO +## Validator changed: NO +## Schema changed: NO +## Harness changed during experiment: NO +## Vitest run: NO +## Ollama calls: 1 MAXIMUM +## Direct API calls: 0 +## Dev server disturbed: NO