experiment: test shift into trade-off reasoning
This commit is contained in:
+144
-1
@@ -171,7 +171,150 @@ Answer before continuing:
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
_Created by Experiment 34. Updated by Experiments 38–53, 54A–54Z, 55A–55F, 56D–56H, 56L–56M, 58B.2, v0.8 closeout. Branch: `feature/question-formulation-v0.24`. First-pass reasoning-fidelity v0.8 complete to A–F scope._
|
### Experiment 59A.1 — Shift from Savings Realism to Worth-It Reasoning
|
||||||
|
|
||||||
|
**Branch:** `feature/question-formulation-v0.24`
|
||||||
|
**Date:** 2026-08-12
|
||||||
|
**Status:** Complete
|
||||||
|
**Following:** 58B.1 and 58B.2 which established verified uncertainty resolution. This tests whether the engine can shift its investigation boundary when the user accepts the saving but introduces new consequence-based uncertainty.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
When the financial saving is accepted as real but the user introduces retention and delivery consequences, does the engine stop re-investigating the saving and shift the investigation toward whether the relocation is actually worth doing?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Fixed Starting Graph
|
||||||
|
|
||||||
|
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||||
|
|
||||||
|
Existing uncertainty:
|
||||||
|
```
|
||||||
|
n_savings_realism — Are the projected office savings from relocation realistic? — status = unknown
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Fixed Answer
|
||||||
|
|
||||||
|
> I am comfortable that the £2 million annual saving is real. What I am not sure about now is whether relocating is actually worth it if we lose key engineers or slow delivery for a year.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Execution
|
||||||
|
|
||||||
|
**Host/model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||||
|
**Calls:** startCalls=0, updateCalls=1, totalCalls=1
|
||||||
|
**Retries:** 0
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Results
|
||||||
|
|
||||||
|
### Savings-realism uncertainty
|
||||||
|
|
||||||
|
- `n_savings_realism`: status `unknown` → `resolved`
|
||||||
|
- Included in `resolvedUnknownNodeIds`: `["n_savings_realism"]`
|
||||||
|
- newValue: `"Confirmed realistic: £2M annual saving is real"`
|
||||||
|
- reason: `"User explicitly confirmed comfort with the projected savings."`
|
||||||
|
|
||||||
|
**Classification: CLOSED CORRECTLY** — The explicit "comfortable...real" language triggered resolution. No duplicate or re-asking.
|
||||||
|
|
||||||
|
### £2m/year saving
|
||||||
|
|
||||||
|
Preserved as `"Confirmed realistic: £2M annual saving is real"` on the resolved node. Captures amount, time unit (annual), and acceptance status.
|
||||||
|
|
||||||
|
**Classification: PRESERVED AS ACCEPTED EVIDENCE**
|
||||||
|
|
||||||
|
### Added consequence nodes
|
||||||
|
|
||||||
|
```
|
||||||
|
n_relocation_net_value — "Does relocating provide net value despite potential staff loss or delivery slowdown?"
|
||||||
|
kind=unknown, status=unknown, confidence=low
|
||||||
|
depends on n_relocation_state (state node)
|
||||||
|
parentId chain includes n_relocation_state
|
||||||
|
description: "Uncertainty about whether the £2M annual savings outweigh the costs of losing key engineers or slowing engineering delivery for a year..."
|
||||||
|
```
|
||||||
|
|
||||||
|
The node simultaneously represents both consequences mentioned in the answer:
|
||||||
|
1. **Key-engineer retention risk** — "potential staff loss" / "losing key engineers"
|
||||||
|
2. **Delivery slowdown** — "slowing engineering delivery for a year"
|
||||||
|
|
||||||
|
### Selected question
|
||||||
|
|
||||||
|
`"Does relocating provide net value despite potential staff loss or delivery slowdown?"` (nodeId: `n_relocation_net_value`)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Assessment
|
||||||
|
|
||||||
|
| Criterion | Result | Classification |
|
||||||
|
|-----------|--------|----------------|
|
||||||
|
| Savings-realism uncertainty | `unknown` → `resolved`, in resolvedUnknownNodeIds | CLOSED CORRECTLY |
|
||||||
|
| £2m/year saving | `"Confirmed realistic: £2M annual saving is real"` on resolved node | PRESERVED AS ACCEPTED EVIDENCE |
|
||||||
|
| Key-engineer retention risk | Represented in node label + description ("losing key engineers") | STRUCTURALLY REPRESENTED |
|
||||||
|
| Delivery slowdown | Represented in node label + description ("slowing engineering delivery for a year") | STRUCTURALLY REPRESENTED |
|
||||||
|
| Decision shift | New unknown about net value, not savings realism | SHIFTED TO WORTH-IT / CONSEQUENCE REASONING |
|
||||||
|
| Next question quality | Asks the right meta-question: does benefit outweigh consequences? | GOOD |
|
||||||
|
|
||||||
|
### Why:
|
||||||
|
|
||||||
|
The engine correctly closed the savings-realism uncertainty and created a new trade-off unknown that captures both consequences (retention risk and delivery slowdown) in a single structural node. The selected question asks whether net value survives — this is exactly the boundary shift the experiment was designed to test. No re-asking of the savings-realism question occurred.
|
||||||
|
|
||||||
|
### What the engine understood correctly:
|
||||||
|
|
||||||
|
1. **Resolution trigger:** "Comfortable...real" triggered correct resolution of `n_savings_realism`.
|
||||||
|
2. **Evidence preservation:** The £2m figure survived as accepted evidence, not qualified or weakened.
|
||||||
|
3. **Boundary shift recognition:** Instead of asking another savings-related question, the engine formulated a consequence-based trade-off question.
|
||||||
|
4. **Dual-consequence capture:** Both key-engineer retention risk and delivery slowdown were captured in one node's label and description.
|
||||||
|
5. **No redundant investigation:** Did not ask about savings realism again.
|
||||||
|
|
||||||
|
### What it lost or flattened:
|
||||||
|
|
||||||
|
**Single-node compression of two distinct risks.** The engine bundled "losing key engineers" and "slowing delivery for a year" into one unknown node. This is structurally represented but loses the ability to investigate each consequence independently — resolving staff retention wouldn't automatically resolve delivery impact. A more granular representation might have created separate nodes or at least preserved the distinction in the description with clearer structural separation.
|
||||||
|
|
||||||
|
**The `newValue` field lost precision.** "£2 million annual" became `"£2M"` — the exact figure is preserved but the grammatical form is less precise than what 58B.2 produced (`"£2,000,000"`). The annual unit survives in text but not as a structured unit field.
|
||||||
|
|
||||||
|
### What uncertainty it chose to pursue next:
|
||||||
|
|
||||||
|
The engine chose `n_relocation_net_value` — whether the £2M/year benefit outweighs the operational consequences (staff loss + delivery slowdown) for one year. This is the core decision question at this stage of the investigation.
|
||||||
|
|
||||||
|
### Does that uncertainty materially affect whether relocation is worth doing: YES
|
||||||
|
|
||||||
|
Without knowing whether the net value survives the consequences, no relocation decision can be made. This is the right next investigation target.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Classification: A — SUCCESSFUL DECISION SHIFT
|
||||||
|
|
||||||
|
The engine correctly closed savings-realism, preserved verified evidence, represented both key consequences structurally, shifted to a trade-off question, and chose the right next investigation — whether relocation provides net value given the consequences.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What this establishes:
|
||||||
|
|
||||||
|
1. The engine can shift its investigation boundary when explicitly told an existing uncertainty is resolved and new concerns are introduced.
|
||||||
|
2. A consequence-based trade-off unknown (net value) can be created in a single update call.
|
||||||
|
3. Multiple consequences can be captured in one structural node.
|
||||||
|
|
||||||
|
## What this does NOT prove:
|
||||||
|
|
||||||
|
1. **Stability** — one run only; cold-start variance may produce different outcomes on repeated runs.
|
||||||
|
2. **Granularity** — bundling two distinct risks into one node works but loses independent resolution paths.
|
||||||
|
3. **Whether the engine distinguishes** between consequences that matter differently (key engineers leaving permanently vs. temporary delivery slowdown).
|
||||||
|
4. **Cross-domain generalisation** — single domain case only.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Production code changed: NO
|
||||||
|
## Prompt changed: NO
|
||||||
|
## Validator changed: NO
|
||||||
|
## Harness changed: NO
|
||||||
|
## Vitest run: NO
|
||||||
|
## Ollama calls beyond harness count: 0
|
||||||
|
## Dev server disturbed: NO
|
||||||
|
|
||||||
### Experiment 58B.2 — Verified Uncertainty Resolution
|
### Experiment 58B.2 — Verified Uncertainty Resolution
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,105 @@
|
|||||||
|
# Experiment 59A.1 — Shift from Savings Realism to Worth-It Reasoning
|
||||||
|
|
||||||
|
**Branch:** `feature/question-formulation-v0.24`
|
||||||
|
**Date:** 2026-08-12
|
||||||
|
**Status:** Complete
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
When the financial saving is accepted as real but the user introduces retention and delivery consequences, does the engine stop re-investigating the saving and shift the investigation toward whether the relocation is actually worth doing?
|
||||||
|
|
||||||
|
## Context route
|
||||||
|
|
||||||
|
Follows 58B.1 (unverified savings kept open) and 58B.2 (verified savings correctly resolved). This tests the semantic boundary *after* resolution: what does the engine do when the user says "the saving is real, but is it worth the cost?"
|
||||||
|
|
||||||
|
## Fixed starting graph
|
||||||
|
|
||||||
|
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||||
|
|
||||||
|
```
|
||||||
|
n_savings_realism — Are the projected office savings from relocation realistic? — status = unknown
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fixed answer
|
||||||
|
|
||||||
|
> I am comfortable that the £2 million annual saving is real. What I am not sure about now is whether relocating is actually worth it if we lose key engineers or slow delivery for a year.
|
||||||
|
|
||||||
|
## Execution
|
||||||
|
|
||||||
|
**Host/model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||||
|
**Calls:** startCalls=0, updateCalls=1, totalCalls=1
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Results
|
||||||
|
|
||||||
|
### UPDATE output
|
||||||
|
|
||||||
|
```
|
||||||
|
updatedNodes: [{"nodeId":"n_savings_realism","previousStatus":"unknown","newStatus":"resolved","previousValue":null,"newValue":"Confirmed realistic: £2M annual saving is real","reason":"User explicitly confirmed comfort with the projected savings."}]
|
||||||
|
resolvedUnknownNodeIds: ["n_savings_realism"]
|
||||||
|
addedNodes: [{"id":"n_relocation_net_value","label":"Does relocating provide net value despite potential staff loss or delivery slowdown?","description":"Uncertainty about whether the £2M annual savings outweigh the costs of losing key engineers or slowing engineering delivery for a year, so that we can determine if the relocation is actually worth pursuing.","kind":"unknown","status":"unknown","confidence":"low","value":null,"unit":null,"evidenceIds":[],"dependsOn":[],"affects":[],"parentId":null,"childIds":["n_relocation_state"]}]
|
||||||
|
addedEdges: [{"id":"e-net-value-to-state","fromNodeId":"n_relocation_net_value","toNodeId":"n_relocation_state","relationship":"depends_on","confidence":"medium","description":"Net value assessment depends on the relocation consideration state."}]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Resulting graph (3 nodes, 2 edges)
|
||||||
|
|
||||||
|
| Node | Kind | Status | Label |
|
||||||
|
|------|------|--------|-------|
|
||||||
|
| n_relocation_state | state | provisional | Engineering team relocation consideration |
|
||||||
|
| n_savings_realism | unknown | **resolved** | Are the projected office savings from relocation realistic? |
|
||||||
|
| **n_relocation_net_value** | unknown | unknown | Does relocating provide net value despite potential staff loss or delivery slowdown? |
|
||||||
|
|
||||||
|
### Selected question
|
||||||
|
|
||||||
|
`"Does relocating provide net value despite potential staff loss or delivery slowdown?"` → nodeId: `n_relocation_net_value`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Reasoning Assessment
|
||||||
|
|
||||||
|
| Criterion | Result |
|
||||||
|
|-----------|--------|
|
||||||
|
| Savings-realism question | CLOSED CORRECTLY |
|
||||||
|
| £2m/year saving | PRESERVED AS ACCEPTED EVIDENCE |
|
||||||
|
| Key-engineer retention risk | STRUCTURALLY REPRESENTED |
|
||||||
|
| Delivery slowdown | STRUCTURALLY REPRESENTED |
|
||||||
|
| Decision shift | SHIFTED TO WORTH-IT / CONSEQUENCE REASONING |
|
||||||
|
| Next question quality | GOOD |
|
||||||
|
|
||||||
|
### What the engine understood correctly
|
||||||
|
|
||||||
|
1. "Comfortable...real" triggered correct resolution of `n_savings_realism`
|
||||||
|
2. The £2m figure survived as accepted evidence
|
||||||
|
3. Boundary shift: formulated a consequence-based trade-off question, not another savings question
|
||||||
|
4. Both key consequences captured in one structural node
|
||||||
|
5. No redundant investigation of the resolved question
|
||||||
|
|
||||||
|
### What it lost or flattened
|
||||||
|
|
||||||
|
- Two distinct risks (staff loss, delivery slowdown) bundled into one unknown — structurally represented but loses independent resolution paths
|
||||||
|
- "£2 million annual" → `"£2M"` in newValue; precise form less granular than 58B.2's `"£2,000,000"`
|
||||||
|
|
||||||
|
### Classification: A — SUCCESSFUL DECISION SHIFT
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What this establishes
|
||||||
|
|
||||||
|
1. Engine can shift investigation boundary when explicitly told an existing uncertainty is resolved
|
||||||
|
2. Consequence-based trade-off unknown can be created in a single update call
|
||||||
|
3. Multiple consequences can be captured in one structural node
|
||||||
|
|
||||||
|
## What this does NOT prove
|
||||||
|
|
||||||
|
1. Stability across repeated runs
|
||||||
|
2. Whether the engine distinguishes between consequences that matter differently
|
||||||
|
3. Cross-domain generalisation
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Production code changed:** NO
|
||||||
|
**Prompt changed:** NO
|
||||||
|
**Ollama calls beyond harness count:** 0
|
||||||
Reference in New Issue
Block a user