experiment: test opposite-outcome decision closure
This commit is contained in:
@@ -2998,3 +2998,7 @@ Experiment 60B.44 was a live regression of 60B.43's terminal post-mutation eligi
|
||||
---
|
||||
|
||||
Experiment 60B.46 reran the exact 60B.44 closure case through one bounded live update using 60B.45/46's explicit harness metadata capture fields (`finalActiveUnknownNodeId` and `finalSelectedQuestion`). **Classification: A — LIVE CLEAN CLOSURE DIRECTLY CONFIRMED.** The enterprise-customer unknown resolved in place (unknown → resolved) and the product-launch decision also resolved in place (unknown → resolved). Both option identities preserved. Zero new nodes or edges. `finalActiveUnknownNodeId = null` (directly observed, not inferred from omission). `finalSelectedQuestion = null` (directly observed, not inferred from omission). structuralActionRequired = null. This confirms that 60B.45's harness-level metadata exposure correctly captures the production runtime's clean closure state with explicit null values rather than field omission. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed.
|
||||
|
||||
---
|
||||
|
||||
Experiment 60B.47 tested whether resolving the customer-signing factor with a **negative** answer (opposite polarity to 60B.46) produces symmetric lifecycle closure. **Classification: G — DIFFERENT FIRST FAILURE.** The engine's semantic reasoning was correct: same customer factor (`n_enterprise_customer_signing`) identified, negative meaning accurately preserved (`userSupportedMeaning` captured "will not sign" + £700k revenue lost), same decision (`n_product_launch_decision`) targeted. However, the proposal was rejected at `proposal_compatibility` with HTTP 422 because the model included `n_product_launch_decision` in `updatedNodes` but omitted it from `resolvedUnknownNodeIds` — a structural inconsistency when resolving multiple unknowns in one turn. No graph mutation occurred. No new unknowns invented (zero addedNodes/edges). The semantic path is symmetric; the structural output contract is not yet symmetric under negative framing. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed.
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
# Experiment 60B.47 — Negative-outcome decision closure
|
||||
|
||||
**Date:** 2026-08-14
|
||||
**Branch:** `feature/closure-metadata-capture-v0.39`
|
||||
|
||||
## Purpose
|
||||
|
||||
Test whether the same decision lifecycle closes cleanly when the material customer uncertainty resolves **negatively** instead of positively (60B.46).
|
||||
|
||||
Hypothesis: clean lifecycle handling should not depend on whether the answer is positive or negative.
|
||||
|
||||
## Method
|
||||
|
||||
One bounded live update using the 60B.44/46 pre-anchored fixture:
|
||||
|
||||
- **Fixture:** `tests/fixtures/pre-anchored-product-launch-customer-signing.json`
|
||||
- **Mode:** `updateOnly` (single Update, no Start)
|
||||
- **Answer:** "No. The enterprise customer has now confirmed in writing that they will not sign if we launch this year, so the £700,000 of expected annual revenue from them will not be received. There are no other material uncertainties between launching this year and waiting twelve months."
|
||||
- **Model:** `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`)
|
||||
|
||||
## Call accounting
|
||||
|
||||
```
|
||||
startCalls: 0
|
||||
updateCalls: 1
|
||||
totalCalls: 1
|
||||
retries: 0
|
||||
```
|
||||
|
||||
## Results
|
||||
|
||||
### HTTP
|
||||
|
||||
- **HTTP status:** 422
|
||||
- **Stage:** `proposal_compatibility` — rejected (validation failure)
|
||||
- **Validation error:** `"Unknown node updated to resolved must also appear in resolvedUnknownNodeIds: \"n_product_launch_decision\""`
|
||||
|
||||
### Proposal content (from rejection snapshot)
|
||||
|
||||
The engine *did* produce a proposal before rejection. The rejected-snapshot reveals its structure:
|
||||
|
||||
```json
|
||||
{
|
||||
"answerMeaning": {
|
||||
"userSupportedMeaning": "The prospective enterprise customer will not sign if we launch this year, so £700,000 of expected annual revenue will not be received. There are no other material uncertainties between the two timing options.",
|
||||
"possibleInference": "This financial reduction materially weakens the immediate-launch option relative to waiting twelve months."
|
||||
},
|
||||
"updatedNodes": [
|
||||
{ "nodeId": "n_enterprise_customer_signing", "newValue": null },
|
||||
{ "nodeId": "n_product_launch_decision", "newValue": null }
|
||||
],
|
||||
"resolvedUnknownNodeIds": [
|
||||
"n_enterprise_customer_signing"
|
||||
],
|
||||
"addedNodes": [],
|
||||
"addedEdges": []
|
||||
}
|
||||
```
|
||||
|
||||
### Structural analysis of the rejected proposal
|
||||
|
||||
**What got right:**
|
||||
|
||||
1. **Negative meaning preserved correctly.** `userSupportedMeaning` accurately reflects: *"The prospective enterprise customer will not sign if we launch this year, so £700,000 of expected annual revenue will not be received."*
|
||||
2. **Same customer factor identified.** `n_enterprise_customer_signing` — the exact same node ID as 60B.46.
|
||||
3. **Same decision targeted.** `n_product_launch_decision` — the exact same decision node as 60B.46.
|
||||
4. **Both nodes placed in updatedNodes** for resolution.
|
||||
5. **No new unknowns invented.** `addedNodes: []`.
|
||||
6. **No new edges created.** `addedEdges: []`.
|
||||
|
||||
**The defect:**
|
||||
|
||||
- `n_product_launch_decision` appeared in `updatedNodes` (meaning the model proposed updating it to resolved), but was **missing from `resolvedUnknownNodeIds`**.
|
||||
- The validator correctly caught this inconsistency and rejected the proposal.
|
||||
|
||||
### Assessment
|
||||
|
||||
| Criterion | Result |
|
||||
|-----------|--------|
|
||||
| Customer factor identity | RESOLVED IN PROPOSAL (rejected before application) |
|
||||
| Negative meaning preservation | PRESERVED — `userSupportedMeaning` accurately captures "will not sign" + £700k revenue lost |
|
||||
| Decision state in proposal | RESOLVED (in updatedNodes) |
|
||||
| Identity preservation | Same node IDs as 60B.46 |
|
||||
| New uncertainty invented | NONE |
|
||||
| Structural validation | FAILED — resolvedUnknownNodeIds inconsistent with updatedNodes |
|
||||
|
||||
## Classification: G — DIFFERENT FIRST FAILURE
|
||||
|
||||
**Structural validation failure:** The proposal was rejected at `proposal_compatibility` because the model included `n_product_launch_decision` in `updatedNodes` (proposing to resolve it) but omitted it from `resolvedUnknownNodeIds`.
|
||||
|
||||
The engine's semantic reasoning was **correct** — same customer factor, opposite meaning preserved, same decision targeted. The failure is purely structural: an internal consistency gap between `updatedNodes` and `resolvedUnknownNodeIds` when the model proposes a multi-node resolution in one turn.
|
||||
|
||||
## Why this matters
|
||||
|
||||
This is a different failure class from 60B.46 (which showed clean closure) but reveals an important asymmetry:
|
||||
|
||||
- **60B.46 (positive):** The model apparently produced `resolvedUnknownNodeIds` that included both nodes — or the decision was resolved through a different mechanism (e.g., deterministic post-processing) — and the proposal passed validation cleanly.
|
||||
- **60B.47 (negative):** The model explicitly listed both nodes in `updatedNodes` but forgot to include the decision node in `resolvedUnknownNodeIds`, causing structural rejection.
|
||||
|
||||
The semantic path is symmetric (same factor, same decision, correct meaning). The structural path is not yet symmetric. This is a fixable gap: the model needs consistent output of `resolvedUnknownNodeIds` when resolving multiple unknowns in one turn.
|
||||
|
||||
## 60B.46 comparison
|
||||
|
||||
| Field | 60B.46 (positive) | 60B.47 (negative) |
|
||||
|-------|-------------------|-------------------|
|
||||
| Same customer factor reused | YES (`n_enterprise_customer_signing`) | YES (`n_enterprise_customer_signing`) |
|
||||
| Opposite answer meaning preserved | N/A | YES — `userSupportedMeaning` correct |
|
||||
| Decision closure attempted in proposal | YES | YES (but structurally inconsistent) |
|
||||
| Validation outcome | PASSED (422 equivalent not triggered) | REJECTED 422 |
|
||||
| Added unknown count | 0 | 0 |
|
||||
| Structural path symmetric? | — | NO |
|
||||
|
||||
## What this proves
|
||||
|
||||
The engine's **semantic reasoning is robust to answer polarity** — the negative answer correctly identified the same factor, preserved its meaning, and targeted the same decision. However, **the structural output contract is not yet symmetric**: when resolving multiple unknowns simultaneously in one turn under negative framing, the model fails to consistently populate `resolvedUnknownNodeIds`.
|
||||
|
||||
## What remains weak or unproven
|
||||
|
||||
- Whether the same proposal would pass if structured correctly (i.e., whether `n_product_launch_decision` should also appear in `resolvedUnknownNodeIds`).
|
||||
- Whether positive vs negative answers trigger different output-template paths in the model.
|
||||
- A targeted fix for multi-node resolution consistency in `resolvedUnknownNodeIds`.
|
||||
|
||||
## Production code changed: NO
|
||||
Reference in New Issue
Block a user