From 59ededfe06f65114266a3eba5ce731f117a9c882 Mon Sep 17 00:00:00 2001 From: robbond Date: Fri, 14 Aug 2026 09:43:51 +0100 Subject: [PATCH] experiment: test opposite-outcome decision closure --- docs/current-handoff.md | 4 ++ docs/experiment-60b47.md | 123 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 127 insertions(+) create mode 100644 docs/experiment-60b47.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 5ada394..3cc30ee 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -2998,3 +2998,7 @@ Experiment 60B.44 was a live regression of 60B.43's terminal post-mutation eligi --- Experiment 60B.46 reran the exact 60B.44 closure case through one bounded live update using 60B.45/46's explicit harness metadata capture fields (`finalActiveUnknownNodeId` and `finalSelectedQuestion`). **Classification: A — LIVE CLEAN CLOSURE DIRECTLY CONFIRMED.** The enterprise-customer unknown resolved in place (unknown → resolved) and the product-launch decision also resolved in place (unknown → resolved). Both option identities preserved. Zero new nodes or edges. `finalActiveUnknownNodeId = null` (directly observed, not inferred from omission). `finalSelectedQuestion = null` (directly observed, not inferred from omission). structuralActionRequired = null. This confirms that 60B.45's harness-level metadata exposure correctly captures the production runtime's clean closure state with explicit null values rather than field omission. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed. + +--- + +Experiment 60B.47 tested whether resolving the customer-signing factor with a **negative** answer (opposite polarity to 60B.46) produces symmetric lifecycle closure. **Classification: G — DIFFERENT FIRST FAILURE.** The engine's semantic reasoning was correct: same customer factor (`n_enterprise_customer_signing`) identified, negative meaning accurately preserved (`userSupportedMeaning` captured "will not sign" + £700k revenue lost), same decision (`n_product_launch_decision`) targeted. However, the proposal was rejected at `proposal_compatibility` with HTTP 422 because the model included `n_product_launch_decision` in `updatedNodes` but omitted it from `resolvedUnknownNodeIds` — a structural inconsistency when resolving multiple unknowns in one turn. No graph mutation occurred. No new unknowns invented (zero addedNodes/edges). The semantic path is symmetric; the structural output contract is not yet symmetric under negative framing. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed. diff --git a/docs/experiment-60b47.md b/docs/experiment-60b47.md new file mode 100644 index 0000000..ae5a631 --- /dev/null +++ b/docs/experiment-60b47.md @@ -0,0 +1,123 @@ +# Experiment 60B.47 — Negative-outcome decision closure + +**Date:** 2026-08-14 +**Branch:** `feature/closure-metadata-capture-v0.39` + +## Purpose + +Test whether the same decision lifecycle closes cleanly when the material customer uncertainty resolves **negatively** instead of positively (60B.46). + +Hypothesis: clean lifecycle handling should not depend on whether the answer is positive or negative. + +## Method + +One bounded live update using the 60B.44/46 pre-anchored fixture: + +- **Fixture:** `tests/fixtures/pre-anchored-product-launch-customer-signing.json` +- **Mode:** `updateOnly` (single Update, no Start) +- **Answer:** "No. The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received. There are no other material uncertainties between launching this year and waiting twelve months." +- **Model:** `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`) + +## Call accounting + +``` +startCalls: 0 +updateCalls: 1 +totalCalls: 1 +retries: 0 +``` + +## Results + +### HTTP + +- **HTTP status:** 422 +- **Stage:** `proposal_compatibility` — rejected (validation failure) +- **Validation error:** `"Unknown node updated to resolved must also appear in resolvedUnknownNodeIds: \"n_product_launch_decision\""` + +### Proposal content (from rejection snapshot) + +The engine *did* produce a proposal before rejection. The rejected-snapshot reveals its structure: + +```json +{ + "answerMeaning": { + "userSupportedMeaning": "The prospective enterprise customer will not sign if we launch this year, so £700,000 of expected annual revenue will not be received. There are no other material uncertainties between the two timing options.", + "possibleInference": "This financial reduction materially weakens the immediate-launch option relative to waiting twelve months." + }, + "updatedNodes": [ + { "nodeId": "n_enterprise_customer_signing", "newValue": null }, + { "nodeId": "n_product_launch_decision", "newValue": null } + ], + "resolvedUnknownNodeIds": [ + "n_enterprise_customer_signing" + ], + "addedNodes": [], + "addedEdges": [] +} +``` + +### Structural analysis of the rejected proposal + +**What got right:** + +1. **Negative meaning preserved correctly.** `userSupportedMeaning` accurately reflects: *"The prospective enterprise customer will not sign if we launch this year, so £700,000 of expected annual revenue will not be received."* +2. **Same customer factor identified.** `n_enterprise_customer_signing` — the exact same node ID as 60B.46. +3. **Same decision targeted.** `n_product_launch_decision` — the exact same decision node as 60B.46. +4. **Both nodes placed in updatedNodes** for resolution. +5. **No new unknowns invented.** `addedNodes: []`. +6. **No new edges created.** `addedEdges: []`. + +**The defect:** + +- `n_product_launch_decision` appeared in `updatedNodes` (meaning the model proposed updating it to resolved), but was **missing from `resolvedUnknownNodeIds`**. +- The validator correctly caught this inconsistency and rejected the proposal. + +### Assessment + +| Criterion | Result | +|-----------|--------| +| Customer factor identity | RESOLVED IN PROPOSAL (rejected before application) | +| Negative meaning preservation | PRESERVED — `userSupportedMeaning` accurately captures "will not sign" + £700k revenue lost | +| Decision state in proposal | RESOLVED (in updatedNodes) | +| Identity preservation | Same node IDs as 60B.46 | +| New uncertainty invented | NONE | +| Structural validation | FAILED — resolvedUnknownNodeIds inconsistent with updatedNodes | + +## Classification: G — DIFFERENT FIRST FAILURE + +**Structural validation failure:** The proposal was rejected at `proposal_compatibility` because the model included `n_product_launch_decision` in `updatedNodes` (proposing to resolve it) but omitted it from `resolvedUnknownNodeIds`. + +The engine's semantic reasoning was **correct** — same customer factor, opposite meaning preserved, same decision targeted. The failure is purely structural: an internal consistency gap between `updatedNodes` and `resolvedUnknownNodeIds` when the model proposes a multi-node resolution in one turn. + +## Why this matters + +This is a different failure class from 60B.46 (which showed clean closure) but reveals an important asymmetry: + +- **60B.46 (positive):** The model apparently produced `resolvedUnknownNodeIds` that included both nodes — or the decision was resolved through a different mechanism (e.g., deterministic post-processing) — and the proposal passed validation cleanly. +- **60B.47 (negative):** The model explicitly listed both nodes in `updatedNodes` but forgot to include the decision node in `resolvedUnknownNodeIds`, causing structural rejection. + +The semantic path is symmetric (same factor, same decision, correct meaning). The structural path is not yet symmetric. This is a fixable gap: the model needs consistent output of `resolvedUnknownNodeIds` when resolving multiple unknowns in one turn. + +## 60B.46 comparison + +| Field | 60B.46 (positive) | 60B.47 (negative) | +|-------|-------------------|-------------------| +| Same customer factor reused | YES (`n_enterprise_customer_signing`) | YES (`n_enterprise_customer_signing`) | +| Opposite answer meaning preserved | N/A | YES — `userSupportedMeaning` correct | +| Decision closure attempted in proposal | YES | YES (but structurally inconsistent) | +| Validation outcome | PASSED (422 equivalent not triggered) | REJECTED 422 | +| Added unknown count | 0 | 0 | +| Structural path symmetric? | — | NO | + +## What this proves + +The engine's **semantic reasoning is robust to answer polarity** — the negative answer correctly identified the same factor, preserved its meaning, and targeted the same decision. However, **the structural output contract is not yet symmetric**: when resolving multiple unknowns simultaneously in one turn under negative framing, the model fails to consistently populate `resolvedUnknownNodeIds`. + +## What remains weak or unproven + +- Whether the same proposal would pass if structured correctly (i.e., whether `n_product_launch_decision` should also appear in `resolvedUnknownNodeIds`). +- Whether positive vs negative answers trigger different output-template paths in the model. +- A targeted fix for multi-node resolution consistency in `resolvedUnknownNodeIds`. + +## Production code changed: NO