124 lines
6.2 KiB
Markdown
124 lines
6.2 KiB
Markdown
# Experiment 60B.47 — Negative-outcome decision closure
|
|
|
|
**Date:** 2026-08-14
|
|
**Branch:** `feature/closure-metadata-capture-v0.39`
|
|
|
|
## Purpose
|
|
|
|
Test whether the same decision lifecycle closes cleanly when the material customer uncertainty resolves **negatively** instead of positively (60B.46).
|
|
|
|
Hypothesis: clean lifecycle handling should not depend on whether the answer is positive or negative.
|
|
|
|
## Method
|
|
|
|
One bounded live update using the 60B.44/46 pre-anchored fixture:
|
|
|
|
- **Fixture:** `tests/fixtures/pre-anchored-product-launch-customer-signing.json`
|
|
- **Mode:** `updateOnly` (single Update, no Start)
|
|
- **Answer:** "No. The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received. There are no other material uncertainties between launching this year and waiting twelve months."
|
|
- **Model:** `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`)
|
|
|
|
## Call accounting
|
|
|
|
```
|
|
startCalls: 0
|
|
updateCalls: 1
|
|
totalCalls: 1
|
|
retries: 0
|
|
```
|
|
|
|
## Results
|
|
|
|
### HTTP
|
|
|
|
- **HTTP status:** 422
|
|
- **Stage:** `proposal_compatibility` — rejected (validation failure)
|
|
- **Validation error:** `"Unknown node updated to resolved must also appear in resolvedUnknownNodeIds: \"n_product_launch_decision\""`
|
|
|
|
### Proposal content (from rejection snapshot)
|
|
|
|
The engine *did* produce a proposal before rejection. The rejected-snapshot reveals its structure:
|
|
|
|
```json
|
|
{
|
|
"answerMeaning": {
|
|
"userSupportedMeaning": "The prospective enterprise customer will not sign if we launch this year, so £700,000 of expected annual revenue will not be received. There are no other material uncertainties between the two timing options.",
|
|
"possibleInference": "This financial reduction materially weakens the immediate-launch option relative to waiting twelve months."
|
|
},
|
|
"updatedNodes": [
|
|
{ "nodeId": "n_enterprise_customer_signing", "newValue": null },
|
|
{ "nodeId": "n_product_launch_decision", "newValue": null }
|
|
],
|
|
"resolvedUnknownNodeIds": [
|
|
"n_enterprise_customer_signing"
|
|
],
|
|
"addedNodes": [],
|
|
"addedEdges": []
|
|
}
|
|
```
|
|
|
|
### Structural analysis of the rejected proposal
|
|
|
|
**What got right:**
|
|
|
|
1. **Negative meaning preserved correctly.** `userSupportedMeaning` accurately reflects: *"The prospective enterprise customer will not sign if we launch this year, so £700,000 of expected annual revenue will not be received."*
|
|
2. **Same customer factor identified.** `n_enterprise_customer_signing` — the exact same node ID as 60B.46.
|
|
3. **Same decision targeted.** `n_product_launch_decision` — the exact same decision node as 60B.46.
|
|
4. **Both nodes placed in updatedNodes** for resolution.
|
|
5. **No new unknowns invented.** `addedNodes: []`.
|
|
6. **No new edges created.** `addedEdges: []`.
|
|
|
|
**The defect:**
|
|
|
|
- `n_product_launch_decision` appeared in `updatedNodes` (meaning the model proposed updating it to resolved), but was **missing from `resolvedUnknownNodeIds`**.
|
|
- The validator correctly caught this inconsistency and rejected the proposal.
|
|
|
|
### Assessment
|
|
|
|
| Criterion | Result |
|
|
|-----------|--------|
|
|
| Customer factor identity | RESOLVED IN PROPOSAL (rejected before application) |
|
|
| Negative meaning preservation | PRESERVED — `userSupportedMeaning` accurately captures "will not sign" + £700k revenue lost |
|
|
| Decision state in proposal | RESOLVED (in updatedNodes) |
|
|
| Identity preservation | Same node IDs as 60B.46 |
|
|
| New uncertainty invented | NONE |
|
|
| Structural validation | FAILED — resolvedUnknownNodeIds inconsistent with updatedNodes |
|
|
|
|
## Classification: G — DIFFERENT FIRST FAILURE
|
|
|
|
**Structural validation failure:** The proposal was rejected at `proposal_compatibility` because the model included `n_product_launch_decision` in `updatedNodes` (proposing to resolve it) but omitted it from `resolvedUnknownNodeIds`.
|
|
|
|
The engine's semantic reasoning was **correct** — same customer factor, opposite meaning preserved, same decision targeted. The failure is purely structural: an internal consistency gap between `updatedNodes` and `resolvedUnknownNodeIds` when the model proposes a multi-node resolution in one turn.
|
|
|
|
## Why this matters
|
|
|
|
This is a different failure class from 60B.46 (which showed clean closure) but reveals an important asymmetry:
|
|
|
|
- **60B.46 (positive):** The model apparently produced `resolvedUnknownNodeIds` that included both nodes — or the decision was resolved through a different mechanism (e.g., deterministic post-processing) — and the proposal passed validation cleanly.
|
|
- **60B.47 (negative):** The model explicitly listed both nodes in `updatedNodes` but forgot to include the decision node in `resolvedUnknownNodeIds`, causing structural rejection.
|
|
|
|
The semantic path is symmetric (same factor, same decision, correct meaning). The structural path is not yet symmetric. This is a fixable gap: the model needs consistent output of `resolvedUnknownNodeIds` when resolving multiple unknowns in one turn.
|
|
|
|
## 60B.46 comparison
|
|
|
|
| Field | 60B.46 (positive) | 60B.47 (negative) |
|
|
|-------|-------------------|-------------------|
|
|
| Same customer factor reused | YES (`n_enterprise_customer_signing`) | YES (`n_enterprise_customer_signing`) |
|
|
| Opposite answer meaning preserved | N/A | YES — `userSupportedMeaning` correct |
|
|
| Decision closure attempted in proposal | YES | YES (but structurally inconsistent) |
|
|
| Validation outcome | PASSED (422 equivalent not triggered) | REJECTED 422 |
|
|
| Added unknown count | 0 | 0 |
|
|
| Structural path symmetric? | — | NO |
|
|
|
|
## What this proves
|
|
|
|
The engine's **semantic reasoning is robust to answer polarity** — the negative answer correctly identified the same factor, preserved its meaning, and targeted the same decision. However, **the structural output contract is not yet symmetric**: when resolving multiple unknowns simultaneously in one turn under negative framing, the model fails to consistently populate `resolvedUnknownNodeIds`.
|
|
|
|
## What remains weak or unproven
|
|
|
|
- Whether the same proposal would pass if structured correctly (i.e., whether `n_product_launch_decision` should also appear in `resolvedUnknownNodeIds`).
|
|
- Whether positive vs negative answers trigger different output-template paths in the model.
|
|
- A targeted fix for multi-node resolution consistency in `resolvedUnknownNodeIds`.
|
|
|
|
## Production code changed: NO
|