docs: archive historical Confidence Engine evidence
This commit is contained in:
@@ -0,0 +1,105 @@
|
||||
# Experiment 59A.1 — Shift from Savings Realism to Worth-It Reasoning
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Date:** 2026-08-12
|
||||
**Status:** Complete
|
||||
|
||||
---
|
||||
|
||||
## Objective
|
||||
|
||||
When the financial saving is accepted as real but the user introduces retention and delivery consequences, does the engine stop re-investigating the saving and shift the investigation toward whether the relocation is actually worth doing?
|
||||
|
||||
## Context route
|
||||
|
||||
Follows 58B.1 (unverified savings kept open) and 58B.2 (verified savings correctly resolved). This tests the semantic boundary *after* resolution: what does the engine do when the user says "the saving is real, but is it worth the cost?"
|
||||
|
||||
## Fixed starting graph
|
||||
|
||||
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
|
||||
```
|
||||
n_savings_realism — Are the projected office savings from relocation realistic? — status = unknown
|
||||
```
|
||||
|
||||
## Fixed answer
|
||||
|
||||
> I am comfortable that the £2 million annual saving is real. What I am not sure about now is whether relocating is actually worth it if we lose key engineers or slow delivery for a year.
|
||||
|
||||
## Execution
|
||||
|
||||
**Host/model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
**Calls:** startCalls=0, updateCalls=1, totalCalls=1
|
||||
|
||||
---
|
||||
|
||||
## Results
|
||||
|
||||
### UPDATE output
|
||||
|
||||
```
|
||||
updatedNodes: [{"nodeId":"n_savings_realism","previousStatus":"unknown","newStatus":"resolved","previousValue":null,"newValue":"Confirmed realistic: £2M annual saving is real","reason":"User explicitly confirmed comfort with the projected savings."}]
|
||||
resolvedUnknownNodeIds: ["n_savings_realism"]
|
||||
addedNodes: [{"id":"n_relocation_net_value","label":"Does relocating provide net value despite potential staff loss or delivery slowdown?","description":"Uncertainty about whether the £2M annual savings outweigh the costs of losing key engineers or slowing engineering delivery for a year, so that we can determine if the relocation is actually worth pursuing.","kind":"unknown","status":"unknown","confidence":"low","value":null,"unit":null,"evidenceIds":[],"dependsOn":[],"affects":[],"parentId":null,"childIds":["n_relocation_state"]}]
|
||||
addedEdges: [{"id":"e-net-value-to-state","fromNodeId":"n_relocation_net_value","toNodeId":"n_relocation_state","relationship":"depends_on","confidence":"medium","description":"Net value assessment depends on the relocation consideration state."}]
|
||||
```
|
||||
|
||||
### Resulting graph (3 nodes, 2 edges)
|
||||
|
||||
| Node | Kind | Status | Label |
|
||||
|------|------|--------|-------|
|
||||
| n_relocation_state | state | provisional | Engineering team relocation consideration |
|
||||
| n_savings_realism | unknown | **resolved** | Are the projected office savings from relocation realistic? |
|
||||
| **n_relocation_net_value** | unknown | unknown | Does relocating provide net value despite potential staff loss or delivery slowdown? |
|
||||
|
||||
### Selected question
|
||||
|
||||
`"Does relocating provide net value despite potential staff loss or delivery slowdown?"` → nodeId: `n_relocation_net_value`
|
||||
|
||||
---
|
||||
|
||||
## Reasoning Assessment
|
||||
|
||||
| Criterion | Result |
|
||||
|-----------|--------|
|
||||
| Savings-realism question | CLOSED CORRECTLY |
|
||||
| £2m/year saving | PRESERVED AS ACCEPTED EVIDENCE |
|
||||
| Key-engineer retention risk | STRUCTURALLY REPRESENTED |
|
||||
| Delivery slowdown | STRUCTURALLY REPRESENTED |
|
||||
| Decision shift | SHIFTED TO WORTH-IT / CONSEQUENCE REASONING |
|
||||
| Next question quality | GOOD |
|
||||
|
||||
### What the engine understood correctly
|
||||
|
||||
1. "Comfortable...real" triggered correct resolution of `n_savings_realism`
|
||||
2. The £2m figure survived as accepted evidence
|
||||
3. Boundary shift: formulated a consequence-based trade-off question, not another savings question
|
||||
4. Both key consequences captured in one structural node
|
||||
5. No redundant investigation of the resolved question
|
||||
|
||||
### What it lost or flattened
|
||||
|
||||
- Two distinct risks (staff loss, delivery slowdown) bundled into one unknown — structurally represented but loses independent resolution paths
|
||||
- "£2 million annual" → `"£2M"` in newValue; precise form less granular than 58B.2's `"£2,000,000"`
|
||||
|
||||
### Classification: A — SUCCESSFUL DECISION SHIFT
|
||||
|
||||
---
|
||||
|
||||
## What this establishes
|
||||
|
||||
1. Engine can shift investigation boundary when explicitly told an existing uncertainty is resolved
|
||||
2. Consequence-based trade-off unknown can be created in a single update call
|
||||
3. Multiple consequences can be captured in one structural node
|
||||
|
||||
## What this does NOT prove
|
||||
|
||||
1. Stability across repeated runs
|
||||
2. Whether the engine distinguishes between consequences that matter differently
|
||||
3. Cross-domain generalisation
|
||||
|
||||
---
|
||||
|
||||
**Production code changed:** NO
|
||||
**Prompt changed:** NO
|
||||
**Ollama calls beyond harness count:** 0
|
||||
@@ -0,0 +1,221 @@
|
||||
# Experiment 59A.2 — Trade-off Decomposition: Known Consequence vs Uncertain Consequence
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Date:** 2026-08-12
|
||||
**Status:** Complete
|
||||
|
||||
---
|
||||
|
||||
## Objective
|
||||
|
||||
When one consequence is stated as known and a second consequence remains uncertain, does the engine represent them separately rather than collapsing both into one broad "is it worth it?" unknown?
|
||||
|
||||
This follows from 59A.1 which successfully shifted to trade-off reasoning but bundled key-engineer loss and delivery slowdown into one umbrella node.
|
||||
|
||||
---
|
||||
|
||||
## Context route
|
||||
|
||||
Read only:
|
||||
- `docs/current-handoff.md` (section 59A.1)
|
||||
- Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
- Harness: `scripts/reproduce-multi-turn-investigation.mjs`
|
||||
|
||||
Do not load 57J history. Do not investigate unrelated test failures.
|
||||
|
||||
---
|
||||
|
||||
## Fixed starting graph
|
||||
|
||||
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
|
||||
Existing unresolved question:
|
||||
```
|
||||
n_savings_realism — Are the projected office savings from relocation realistic? — status = unknown
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Fixed answer
|
||||
|
||||
```
|
||||
I am comfortable that the £2 million annual saving is real. We know we would lose two senior engineers if we relocate. What I don't know is whether losing them would materially delay delivery, or by how much.
|
||||
```
|
||||
|
||||
The answer contains three semantically different things:
|
||||
1. **Resolved factual uncertainty:** £2 million annual saving is real
|
||||
2. **Known consequence:** two senior engineers would leave
|
||||
3. **Unresolved consequence:** whether that loss materially delays delivery, and by how much
|
||||
|
||||
The engine should not treat all three as equivalent uncertainty.
|
||||
|
||||
---
|
||||
|
||||
## Execution
|
||||
|
||||
**Host/model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
**Calls:** startCalls=0, updateCalls=1, totalCalls=1
|
||||
**Retries:** 0
|
||||
|
||||
---
|
||||
|
||||
## Results
|
||||
|
||||
### UPDATE output
|
||||
|
||||
```
|
||||
HTTP status: 422
|
||||
Stage: proposal_compatibility
|
||||
Error: "selectedQuestion must be a single non-compound question"
|
||||
```
|
||||
|
||||
### Rejected proposal snapshot (from diagnostics.rejectedProposalSnapshot)
|
||||
|
||||
#### answerMeaning
|
||||
|
||||
- `userSupportedMeaning`: "The £2 million annual office savings are confirmed as real. However, relocating would result in losing two senior engineers, and there is unresolved uncertainty regarding whether this loss would materially delay delivery or by what extent."
|
||||
- `possibleInference`: null
|
||||
|
||||
#### Proposal mutations
|
||||
|
||||
```
|
||||
updatedNodes: [
|
||||
{ "nodeId": "n_savings_realism", "newValue": null }
|
||||
]
|
||||
resolvedUnknownNodeIds: ["n_savings_realism"]
|
||||
addedNodes: [
|
||||
{
|
||||
"id": "n_delivery_delay_impact",
|
||||
"kind": "unknown",
|
||||
"label": "Impact of losing two senior engineers on delivery timelines",
|
||||
"description": "Uncertainty about whether losing two senior engineers would materially delay engineering delivery, and by how much, because resolving this is needed to decide if the £2M relocation savings justify the potential project delays.",
|
||||
"parentId": null,
|
||||
"dependsOn": ["n_relocation_state"],
|
||||
"affects": [],
|
||||
"childIds": []
|
||||
}
|
||||
]
|
||||
addedEdges: [
|
||||
{
|
||||
"fromNodeId": "n_delivery_delay_impact",
|
||||
"toNodeId": "n_relocation_state",
|
||||
"relationship": "depends_on"
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
#### selectedQuestion
|
||||
|
||||
`null` (no selected question produced — this caused the rejection)
|
||||
|
||||
---
|
||||
|
||||
## Reasoning Assessment
|
||||
|
||||
### Savings-realism uncertainty
|
||||
|
||||
- `n_savings_realism`: status unknown → resolved (included in resolvedUnknownNodeIds)
|
||||
- **Classification: CLOSED CORRECTLY**
|
||||
|
||||
The explicit "comfortable...real" language triggered resolution. No duplicate or re-asking.
|
||||
|
||||
### £2m annual saving
|
||||
|
||||
- Preserved in userSupportedMeaning: "The £2 million annual office savings are confirmed as real."
|
||||
- **Classification: PRESERVED AS ACCEPTED EVIDENCE**
|
||||
|
||||
The full figure (£2 million), time unit (annual), and confirmation status ("confirmed") survived.
|
||||
|
||||
### Two senior engineers leaving
|
||||
|
||||
- Extracted in userSupportedMeaning: "relocating would result in losing two senior engineers"
|
||||
- No separate structural node created for this known fact
|
||||
- **Classification: REPRESENTED BUT LEFT UNCERTAIN** — the model extracted it as part of a single meaning sentence rather than as a standalone known-consequence assertion. The phrase does not use tentative language ("might lose"), but it is also not separately structured.
|
||||
|
||||
### Delivery impact
|
||||
|
||||
- A dedicated unknown node was created: `n_delivery_delay_impact`
|
||||
- label: "Impact of losing two senior engineers on delivery timelines"
|
||||
- kind=unknown, status=unknown
|
||||
- Description captures the causal link explicitly: "Uncertainty about whether losing two senior engineers would materially delay engineering delivery"
|
||||
- **Classification: REPRESENTED AS UNRESOLVED**
|
||||
|
||||
### Causal/dependency relationship
|
||||
|
||||
- The node label references "losing two senior engineers" and the description links it to "materially delay engineering delivery"
|
||||
- The causal chain is encoded in free text within the node's label and description, not as a typed edge
|
||||
- **Classification: PARTIALLY LINKED** — semantically present but structurally flattened into one node rather than represented as a typed relationship between two distinct nodes.
|
||||
|
||||
### Granularity
|
||||
|
||||
The model did NOT separate the known consequence (engineers leaving) from the uncertain consequence (delivery impact). Instead, it created ONE unknown node that bundles both: "Impact of losing two senior engineers on delivery timelines."
|
||||
|
||||
This is structurally one node containing both consequences — not a separation between a known-fact assertion and an unresolved-uncertainty.
|
||||
|
||||
**Classification: COLLAPSED INTO UMBRELLA UNKNOWN**
|
||||
|
||||
### Next question
|
||||
|
||||
No selectedQuestion was produced (null). The rejection was caused by the validator requiring "a single non-compound question."
|
||||
|
||||
**Classification: NONE**
|
||||
|
||||
---
|
||||
|
||||
## Classification: B — PARTIAL DECOMPOSITION
|
||||
|
||||
### Why:
|
||||
|
||||
The engine correctly closed savings-realism, preserved £2m as accepted evidence, and created a dedicated node for delivery impact. However, it did not represent the known consequence ("two senior engineers will leave") separately from the unresolved consequence (delivery delay). Instead, both were collapsed into one unknown node whose label frames the entire issue as an unresolved question ("Impact of losing two senior engineers on delivery timelines"). This means the known fact that engineers *will* leave is structurally embedded inside a node that represents only *what the delivery impact will be* — which is subtly different but still bundles the known and the unknown.
|
||||
|
||||
The selectedQuestion was null, causing a rejection at proposal_compatibility — this is the "apparatus" aspect of the partial result.
|
||||
|
||||
### Did the engine preserve "two senior engineers will leave" as known:
|
||||
PARTIAL — extracted in userSupportedMeaning without tentative language, but not structured as an independent known-consequence node.
|
||||
|
||||
### Did it preserve delivery impact as uncertain:
|
||||
YES — dedicated unknown node created with status=unknown.
|
||||
|
||||
### Did it keep those epistemic states distinct:
|
||||
NO — both consequences are bundled into one structural node.
|
||||
|
||||
### What the engine understood correctly:
|
||||
|
||||
1. Savings realism is resolved (correct resolution trigger)
|
||||
2. £2m/year saving is verified evidence
|
||||
3. The delivery impact from engineer loss is an unresolved question worth investigating
|
||||
4. The causal link between engineer loss and delivery delay was captured in text
|
||||
5. No redundant investigation of the resolved savings question
|
||||
|
||||
### What it flattened or misclassified:
|
||||
|
||||
1. **Epistemic states collapsed.** "We know we would lose two senior engineers" (known) and "What I don't know is whether losing them would materially delay delivery" (uncertain) were bundled into one unknown node. The node does not distinguish between what is known and what remains uncertain about those engineers.
|
||||
2. **selectedQuestion was null.** The model did not produce any selected question, triggering the compound-question validator rejection. This may indicate the model recognized it was generating a complex/unanswerable query and abstained from producing one.
|
||||
|
||||
### What uncertainty it chose to pursue next:
|
||||
NONE — no question produced (rejection).
|
||||
|
||||
### Was that the best available unresolved question:
|
||||
DEBATABLE — even if produced, the question would need to distinguish "will engineers leave?" (known) from "what is the delivery impact?" (uncertain). The model appeared to struggle with this distinction.
|
||||
|
||||
### What this establishes:
|
||||
|
||||
1. The engine CAN extract all three semantic elements (resolved savings, known engineer loss, uncertain delivery) in userSupportedMeaning
|
||||
2. A dedicated unknown node for delivery impact can be created
|
||||
3. However, the known-vs-unknown distinction was not preserved structurally — both consequences were compressed into one unresolved-question frame
|
||||
|
||||
### What this does NOT prove:
|
||||
|
||||
1. Whether a different model or prompt variant would separate the epistemic states more cleanly
|
||||
2. Stability across repeated runs
|
||||
3. Whether the selectedQuestion failure is deterministic or cold-start variance
|
||||
|
||||
---
|
||||
|
||||
## Production code changed: NO
|
||||
## Prompt changed: NO
|
||||
## Validator changed: NO
|
||||
## Harness changed: NO
|
||||
## Vitest run: NO
|
||||
## Ollama calls beyond harness count: 0
|
||||
## Dev server disturbed: NO
|
||||
@@ -0,0 +1,215 @@
|
||||
# Experiment 59A.3 — Known vs Uncertain Consequence Structure
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Date:** 2026-08-12
|
||||
**Status:** Complete
|
||||
**Following:** 59A.2 which showed partial decomposition (delivery impact structured but engineer loss left only in text, no epistemic separation).
|
||||
|
||||
## Objective
|
||||
|
||||
When the user states one consequence as known ("two senior engineers will leave") and a downstream consequence as uncertain ("whether losing them would delay delivery"), does the proposal represent them as separate structural objects with different epistemic states?
|
||||
|
||||
This isolates that distinction without the extra £2m-resolution step from 59A.1/59A.2.
|
||||
|
||||
## Context route
|
||||
|
||||
Read only:
|
||||
- `docs/current-handoff.md` (sections 59A.1, 59A.2)
|
||||
- Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
- Harness: `scripts/reproduce-multi-turn-investigation.mjs`
|
||||
|
||||
Do not load older experiment history.
|
||||
|
||||
## Fixed starting graph
|
||||
|
||||
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
|
||||
Existing unresolved question:
|
||||
```
|
||||
n_savings_realism — Are the projected office savings from relocation realistic? — status = unknown
|
||||
```
|
||||
|
||||
## Fixed answer
|
||||
|
||||
```text
|
||||
We know that relocating would cause two senior engineers to leave. What I don't know is whether losing them would delay delivery, or by how many months.
|
||||
```
|
||||
|
||||
The answer contains two semantically different assertions:
|
||||
1. **Known consequence:** two senior engineers will leave (epistemic state = known)
|
||||
2. **Uncertain consequence:** whether/when that loss delays delivery (epistemic state = unknown)
|
||||
|
||||
Note: this version omits the £2m savings confirmation present in 59A.2, isolating only the known-vs-unknown distinction.
|
||||
|
||||
## Execution
|
||||
|
||||
**Host/model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
**Calls:** startCalls=0, updateCalls=1, totalCalls=1
|
||||
**Retries:** 0
|
||||
|
||||
---
|
||||
|
||||
## Results
|
||||
|
||||
### UPDATE output
|
||||
|
||||
```
|
||||
HTTP status: 422
|
||||
Stage: proposal_compatibility
|
||||
Error: "selectedQuestion must be a single non-compound question"
|
||||
```
|
||||
|
||||
### Rejected proposal snapshot (from diagnostics.rejectedProposalSnapshot)
|
||||
|
||||
#### answerMeaning
|
||||
|
||||
- `userSupportedMeaning`: "Relocating will cause two senior engineers to depart; it is currently unknown whether this loss will delay project delivery or determine the magnitude of such delays."
|
||||
- `possibleInference`: "The financial savings from relocation may be negated by productivity losses and delivery delays caused by losing key technical staff."
|
||||
|
||||
#### Proposal mutations
|
||||
|
||||
```json
|
||||
{
|
||||
"updatedNodes": [],
|
||||
"resolvedUnknownNodeIds": [],
|
||||
"addedNodes": [
|
||||
{
|
||||
"id": "n_talent_departure",
|
||||
"kind": "observation",
|
||||
"label": "Senior engineer turnover from relocation",
|
||||
"description": "Relocating would cause two senior engineers to leave.",
|
||||
"parentId": null,
|
||||
"dependsOn": ["n_relocation_state"],
|
||||
"affects": ["n_delivery_delay_impact"],
|
||||
"childIds": []
|
||||
},
|
||||
{
|
||||
"id": "n_delivery_delay_impact",
|
||||
"kind": "unknown",
|
||||
"label": "Delivery delay impact from lost engineers",
|
||||
"description": "Whether losing the two senior engineers will delay project delivery and by how many months; matters because it determines if productivity loss outweighs office savings.",
|
||||
"parentId": null,
|
||||
"dependsOn": ["n_talent_departure"],
|
||||
"affects": ["n_relocation_state"],
|
||||
"childIds": []
|
||||
}
|
||||
],
|
||||
"addedEdges": [
|
||||
{
|
||||
"fromNodeId": "n_delivery_delay_impact",
|
||||
"toNodeId": "n_talent_departure",
|
||||
"relationship": "depends_on"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### selectedQuestion
|
||||
|
||||
`null` (no selected question produced — caused the rejection at proposal_compatibility)
|
||||
|
||||
---
|
||||
|
||||
## Reasoning Assessment
|
||||
|
||||
### Known engineer loss
|
||||
|
||||
- Created dedicated node `n_talent_departure`:
|
||||
- kind = **observation** (not unknown, not provisional)
|
||||
- label: "Senior engineer turnover from relocation"
|
||||
- description: "Relocating would cause two senior engineers to leave."
|
||||
- depends_on: n_relocation_state
|
||||
- affects: [n_delivery_delay_impact]
|
||||
- `affects` field carries a **typed structural link** to the downstream uncertainty node.
|
||||
|
||||
**Classification: SEPARATE KNOWN STRUCTURE**
|
||||
|
||||
The engineer departure is not embedded in text or left uncertain — it is its own observation node with status derived from kind=observation (a factual assertion, not an unresolved question). This is a correct epistemic state for a known consequence.
|
||||
|
||||
### Delivery impact
|
||||
|
||||
- Created dedicated node `n_delivery_delay_impact`:
|
||||
- kind = **unknown**
|
||||
- status = unknown
|
||||
- label: "Delivery delay impact from lost engineers"
|
||||
- description: "Whether losing the two senior engineers will delay project delivery and by how many months..."
|
||||
- depends_on: [n_talent_departure]
|
||||
- The `dependsOn` field is populated with the known-consequence node — a **typed structural link**.
|
||||
|
||||
**Classification: SEPARATE UNRESOLVED STRUCTURE**
|
||||
|
||||
Delivery uncertainty is its own unknown node with proper kind=status=unknown and structural linkage back to the known consequence via depends_on.
|
||||
|
||||
### Epistemic separation
|
||||
|
||||
- `n_talent_departure` (kind=observation) = known factual consequence
|
||||
- `n_delivery_delay_impact` (kind=unknown, status=unknown) = unresolved uncertain consequence
|
||||
- They are two distinct nodes with a typed `affects`/`depends_on` relationship between them.
|
||||
|
||||
**Classification: CLEARLY SEPARATED**
|
||||
|
||||
The epistemic distinction is preserved at the structural level — two different kinds, two different statuses, connected by typed edges.
|
||||
|
||||
### Relationship between engineer loss and delivery delay
|
||||
|
||||
- `n_talent_departure.affects = ["n_delivery_delay_impact"]`
|
||||
- `n_delivery_delay_impact.dependsOn = ["n_talent_departure"]`
|
||||
- Edge: n_delivery_delay_impact → n_talent_departure with relationship=depends_on
|
||||
|
||||
**Classification: TYPED / STRUCTURAL LINK**
|
||||
|
||||
The causal chain is represented by both a forward field (affects) and a reverse edge (depends_on), not just embedded in prose.
|
||||
|
||||
### Next question
|
||||
|
||||
No selectedQuestion was produced (null). The rejection was caused by the validator requiring "a single non-compound question."
|
||||
|
||||
Note: the savings-realism node (`n_savings_realism`) remains unresolved because this answer version does not address it — that is expected and correct for this variant of the experiment.
|
||||
|
||||
**Classification: NONE**
|
||||
|
||||
---
|
||||
|
||||
## Classification: A — CORRECT EPISTEMIC DECOMPOSITION
|
||||
|
||||
### Why:
|
||||
|
||||
This is a clean positive result. The model created two separate structural objects with distinct epistemic states:
|
||||
|
||||
1. **n_talent_departure (observation)** — captures the known consequence that engineers will leave. Not uncertain, not pending resolution. Its kind=observation signals "established fact to be taken into account."
|
||||
2. **n_delivery_delay_impact (unknown)** — captures the unresolved downstream uncertainty about delivery impact magnitude, depending on the known departure.
|
||||
|
||||
The causal chain between them is represented via typed fields (affects/depends_on) and a typed edge (depends_on), not just text embedding.
|
||||
|
||||
The key distinction from 59A.2: in that experiment both consequences were compressed into one unknown node ("Impact of losing two senior engineers on delivery timelines"). Here they are separate nodes with different kinds — the known-vs-unknown boundary is structurally preserved.
|
||||
|
||||
**Important caveat:** The update was rejected at proposal_compatibility because no selectedQuestion was produced. This is a validator-side issue, not a semantic reasoning failure. The rejected snapshot demonstrates correct structural decomposition even though the update was not applied to the persistent graph.
|
||||
|
||||
### Did "two senior engineers will leave" become its own known structure: YES
|
||||
### Did delivery delay remain explicitly unresolved: YES
|
||||
### Did the graph/proposal preserve the distinction: YES
|
||||
|
||||
---
|
||||
|
||||
## What this establishes:
|
||||
|
||||
1. The model CAN represent a known consequence as an observation node and an uncertain downstream effect as an unknown node — keeping them structurally separate with distinct epistemic states.
|
||||
2. A typed causal chain (affects + depends_on edge) can be produced between these two kinds of nodes in a single proposal.
|
||||
3. Removing the £2m savings confirmation from the answer did not degrade the known-vs-unknown separation; it actually focused the model's attention on exactly what was being tested.
|
||||
|
||||
## What this does NOT prove:
|
||||
|
||||
1. **Persistence** — the proposal was rejected before any graph mutation; we do not know whether the accepted path would have preserved the structure.
|
||||
2. **Next-question generation** — the selectedQuestion failure (null) was not resolved by this experiment. The model may struggle to formulate a single non-compound question when two structural consequences are introduced.
|
||||
3. **Stability** — one run only; cold-start variance has been a factor across Experiments 59A series.
|
||||
4. **Cross-domain generalisation** — single domain case only.
|
||||
|
||||
---
|
||||
|
||||
## Production code changed: NO
|
||||
## Prompt changed: NO
|
||||
## Validator changed: NO
|
||||
## Harness changed: NO
|
||||
## Vitest run: NO
|
||||
## Ollama calls beyond harness count: 0
|
||||
## Dev server disturbed: NO
|
||||
@@ -0,0 +1,138 @@
|
||||
# Experiment 59B.1 — Decision Relevance of Next Question vs Precision Chasing
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Date:** 2026-08-12
|
||||
**Status:** Complete
|
||||
**Following:** 59A series which showed the engine can distinguish known consequences from uncertain downstream effects at the proposal level.
|
||||
|
||||
## Objective
|
||||
|
||||
When the financial benefit is known, the downside is bounded, and the cost of doing nothing is explicit, does the engine compare decision consequences — or does it simply ask for more precision about the remaining uncertainty?
|
||||
|
||||
Specifically: does the engine recognise when greater precision about an uncertainty may not actually matter to the decision?
|
||||
|
||||
## Context route
|
||||
|
||||
Read only:
|
||||
- `docs/current-handoff.md`
|
||||
- `docs/experiment-59a3.md` (preceding reasoning context)
|
||||
- Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
- Harness: `scripts/reproduce-multi-turn-investigation.mjs`
|
||||
|
||||
Do not load older experiment history.
|
||||
|
||||
## Fixed starting graph
|
||||
|
||||
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
|
||||
Existing unresolved question:
|
||||
```
|
||||
n_savings_realism — Are the projected office savings from relocation realistic? — status = unknown
|
||||
```
|
||||
|
||||
## Fixed answer
|
||||
|
||||
```text
|
||||
The £2 million annual saving is real. We know two senior engineers would leave if we relocate. Even in the worst case, I think that would delay delivery by no more than two months. If we stay where we are, we keep paying the extra £2 million every year. I am trying to decide whether getting a more precise delivery-delay estimate would actually change the decision.
|
||||
```
|
||||
|
||||
Contains:
|
||||
- **KNOWN BENEFIT:** £2 million annual saving
|
||||
- **KNOWN CONSEQUENCE:** two senior engineers would leave
|
||||
- **BOUNDED DOWNSIDE:** delivery delay no worse than approximately two months
|
||||
- **DO-NOTHING BASELINE:** staying put costs an extra £2 million every year
|
||||
- **DECISION-RELEVANCE QUESTION:** would greater precision about delivery delay actually change the decision?
|
||||
|
||||
## Execution
|
||||
|
||||
**Host/model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
**Calls:** startCalls=0, updateCalls=1, totalCalls=1
|
||||
**Retries:** 0
|
||||
|
||||
---
|
||||
|
||||
## Results
|
||||
|
||||
### UPDATE output
|
||||
|
||||
```
|
||||
HTTP status: 200
|
||||
Stage: update_applied
|
||||
Validation errors: none
|
||||
|
||||
updatedNodes: [{nodeId:n_savings_realism, previousStatus:unknown, newStatus:resolved, newValue:'£2 million annual savings confirmed as real', reason:'User explicitly confirmed the projected office savings are real and provided exact figures.'}]
|
||||
resolvedUnknownNodeIds: ['n_savings_realism']
|
||||
addedNodes: [{id:n_delay_estimate_relevance, label:'Would a precise delivery-delay estimate change the decision?', kind:unknown, status:unknown}]
|
||||
addedEdges: [{fromNodeId:n_delay_estimate_relevance, toNodeId:n_savings_realism, relationship:depends_on}]
|
||||
selectedQuestion: 'would a precise delivery-delay estimate change the decision?'
|
||||
```
|
||||
|
||||
Resulting graph (3 nodes, 2 edges):
|
||||
- `n_relocation_state` — Engineering team relocation consideration — status=provisional
|
||||
- `n_savings_realism` — Are the projected office savings from relocation realistic? — status=resolved ✓
|
||||
- `n_delay_estimate_relevance` — Would a precise delivery-delay estimate change the decision? — status=unknown
|
||||
|
||||
---
|
||||
|
||||
## Assessment
|
||||
|
||||
### 1. £2m annual saving
|
||||
**PRESERVED AS KNOWN BENEFIT**
|
||||
Node `n_savings_realism` resolved with value `"£2 million annual savings confirmed as real"`.
|
||||
|
||||
### 2. Two-engineer departure
|
||||
**LOST**
|
||||
No node in the graph represents "two senior engineers would leave." Not preserved structurally, not visible in any node label/description/value. The answer clearly stated this as a known consequence but it was dropped.
|
||||
|
||||
### 3. Two-month downside bound
|
||||
**UNAVAILABLE**
|
||||
No node captures "no more than two months" or any upper bound on delivery delay. The engine did not weaken it to open-ended uncertainty explicitly, but the information is simply absent from the graph.
|
||||
|
||||
### 4. Do-nothing baseline
|
||||
**LOST**
|
||||
"Staying put costs an extra £2m every year" is not structurally represented as a cost node, a comparison edge, or any structural element of the graph. `n_relocation_state` has no do-nothing semantics.
|
||||
|
||||
### 5. Decision framing
|
||||
**TRADE-OFF PRESENT BUT BASELINE LOST**
|
||||
The engine selected a question about whether precision matters to the decision — this is trade-off thinking at a meta-level. However, the baseline (cost of staying put) is lost structurally, so the trade-off has no anchoring.
|
||||
|
||||
### 6. Next-question decision relevance
|
||||
**HIGH DECISION RELEVANCE**
|
||||
"Would a precise delivery-delay estimate change the decision?" — If answered yes, it would justify further investigation; if answered no, it would stop precision-seeking. This directly addresses the user's stated concern about whether more precision is worth obtaining.
|
||||
|
||||
### 7. Precision chasing
|
||||
**NO PRECISION CHASING**
|
||||
The engine did NOT ask "what exactly is the delivery delay?" It asked a meta-level question about decision relevance of precision itself. However, this positive result is partially undermined by the fact that critical contextual facts (engineer departure, two-month bound) were lost before the question was formulated.
|
||||
|
||||
---
|
||||
|
||||
## Classification: D — DO-NOTHING BASELINE LOST
|
||||
|
||||
The engine evaluated relocation consequences without preserving the recurring cost of staying put as a structural element. Two additional losses compound this:
|
||||
- The known consequence ("two senior engineers would leave") was entirely lost from the graph.
|
||||
- The bounded downside ("no more than two months") was absent from the graph.
|
||||
|
||||
A positive finding: the engine did **not** ask "what exactly is the delay?" — it asked whether precision matters at all, which is a valid decision-relevant next step. However, this question lacks structural grounding because the critical comparison elements (engineer loss, bounded impact, do-nothing cost) are not present in the graph to give the question context.
|
||||
|
||||
---
|
||||
|
||||
## What this establishes:
|
||||
|
||||
1. The engine can formulate a genuinely meta-level decision-relevance question when prompted by an answer that explicitly raises it ("I am trying to decide whether getting a more precise delivery-delay estimate would actually change the decision").
|
||||
2. The engine does not default to precision-chasing (asking for exact values) when the user signals that decision relevance matters.
|
||||
3. Known benefit preservation works: £2m savings survived as resolved on `n_savings_realism`.
|
||||
|
||||
## What this does NOT prove:
|
||||
|
||||
1. That the engine would independently recognise decision irrelevance without an explicit user prompt about it — the answer text contained "I am trying to decide whether getting a more precise delivery-delay estimate would actually change the decision," which is a very strong signal that guided question selection.
|
||||
2. That the engine preserves known consequences alongside benefits — engineer departure was entirely lost.
|
||||
3. That the engine preserves bounded downside information — the two-month upper bound disappeared.
|
||||
4. Whether these losses are due to answerMeaning extraction limits, proposal generation limits, or node-kinds being misclassified.
|
||||
|
||||
---
|
||||
|
||||
## Key observation
|
||||
|
||||
The engine's meta-level question framing is structurally intelligent but contextually hollow. It asked the right *kind* of question (is precision worth it?) but lost the facts that make that question meaningful (what happens if we relocate? what are the bounds? what does doing nothing cost?). This suggests a **context-preservation deficit** in the update path: when the engine resolves one uncertainty and creates a new decision-relevance node, it drops other critical information from the answer rather than carrying it forward.
|
||||
|
||||
Production code changed: NO
|
||||
@@ -0,0 +1,236 @@
|
||||
# Experiment 59B.2 — Independent Decision Relevance Reasoning
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Date:** 2026-08-12
|
||||
**Status:** Complete
|
||||
**Following:** 59A.3 (known-vs-uncertain decomposition) and 59B.1 (user-hinted decision relevance). This removes the user's hint about precision relevance.
|
||||
|
||||
## Objective
|
||||
|
||||
When the benefit, known consequence, bounded downside, and do-nothing cost are all stated, does the engine independently reason about decision relevance rather than merely asking for greater precision?
|
||||
|
||||
Specifically: does the engine recognise when more precision may not change the decision?
|
||||
|
||||
## Context route
|
||||
|
||||
Read only:
|
||||
- `docs/current-handoff.md` (latest section)
|
||||
- `docs/experiment-59b1.md` (preceding experiment 59B.1 for context)
|
||||
- Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
- Harness: `scripts/reproduce-multi-turn-investigation.mjs`
|
||||
|
||||
Do not load older experiment history.
|
||||
|
||||
## Fixed starting graph
|
||||
|
||||
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
|
||||
Existing unresolved question:
|
||||
```
|
||||
n_savings_realism — Are the projected office savings from relocation realistic? — status = unknown
|
||||
```
|
||||
|
||||
## Fixed answer (exact, verbatim)
|
||||
|
||||
```text
|
||||
The £2 million annual saving is real. We know two senior engineers would leave if we relocate. The worst-case delivery impact is about two months of delay. If we stay where we are, we keep paying the extra £2 million every year.
|
||||
```
|
||||
|
||||
Contains:
|
||||
- **KNOWN BENEFIT:** £2 million annual saving
|
||||
- **KNOWN CONSEQUENCE:** two senior engineers would leave
|
||||
- **BOUNDED DOWNSIDE:** worst-case delivery delay ≈ two months
|
||||
- **DO-NOTHING BASELINE:** stay put → continue paying extra £2 million every year
|
||||
|
||||
Does **NOT** contain:
|
||||
- "Would more precision change the decision?"
|
||||
- "How exactly many weeks will delivery slip?"
|
||||
|
||||
The engine must decide what matters next.
|
||||
|
||||
## Execution
|
||||
|
||||
**Host/model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
**Calls:** startCalls=0, updateCalls=1, totalCalls=1
|
||||
**Retries:** 0
|
||||
|
||||
---
|
||||
|
||||
## Results
|
||||
|
||||
### UPDATE output (accepted)
|
||||
|
||||
```
|
||||
HTTP status: 200
|
||||
Stage: update_applied
|
||||
Validation errors: none
|
||||
|
||||
updatedNodes: [{nodeId: n_savings_realism, previousStatus: unknown, newStatus: resolved, newValue: "confirmed £2M annual saving", reason: "User explicitly confirms the projected office savings are real."}]
|
||||
resolvedUnknownNodeIds: ["n_savings_realism"]
|
||||
|
||||
addedNodes: [
|
||||
{
|
||||
id: n_talent_impact_clarity,
|
||||
label: "Impact of senior engineer departure on delivery",
|
||||
description: "Relocation causes two senior engineers to leave, creating a worst-case 2-month project delay.",
|
||||
kind: observation,
|
||||
status: supported,
|
||||
confidence: high
|
||||
},
|
||||
{
|
||||
id: n_net_relocation_benefit,
|
||||
label: "Whether relocation net benefit is positive",
|
||||
description: "Uncertainty about if the confirmed £2M annual savings outweigh the loss of two senior engineers and up to 2-month delay because it determines if the relocation achieves its core operational objective.",
|
||||
kind: unknown,
|
||||
status: unknown,
|
||||
confidence: medium
|
||||
}
|
||||
]
|
||||
|
||||
addedEdges: [
|
||||
{fromNodeId: n_talent_impact_clarity, toNodeId: n_relocation_state, relationship: weakens},
|
||||
{fromNodeId: n_net_relocation_benefit, toNodeId: n_savings_realism, relationship: depends_on}
|
||||
]
|
||||
|
||||
selectedQuestion: "What evidence would clarify relocation net benefit is positive?"
|
||||
selectedQuestion.nodeId: "n_net_relocation_benefit"
|
||||
```
|
||||
|
||||
Resulting persistent graph (4 nodes, 3 edges):
|
||||
- `n_relocation_state` — Engineering team relocation consideration — status=provisional
|
||||
- `n_savings_realism` — Are the projected office savings from relocation realistic? — status=resolved ✓
|
||||
- `n_talent_impact_clarity` — Impact of senior engineer departure on delivery — status=supported, kind=observation
|
||||
- `n_net_relocation_benefit` — Whether relocation net benefit is positive — status=unknown, kind=unknown
|
||||
|
||||
---
|
||||
|
||||
## Assessment
|
||||
|
||||
### 1. £2m annual saving
|
||||
**PRESERVED AS KNOWN BENEFIT**
|
||||
|
||||
Node `n_savings_realism` resolved with value `"confirmed £2M annual saving"`. The exact figure (£2 million) and the unit (annual) are preserved on a resolved node.
|
||||
|
||||
### 2. Two-engineer departure
|
||||
**PRESERVED AS KNOWN CONSEQUENCE**
|
||||
|
||||
New dedicated observation node `n_talent_impact_clarity`, kind=observation, status=supported, label="Impact of senior engineer departure on delivery", description explicitly states "Relocation causes two senior engineers to leave". This is structurally separate from uncertainty — an observation, not a question.
|
||||
|
||||
### 3. Two-month worst-case bound
|
||||
**PRESERVED AS BOUNDED DOWNSIDE**
|
||||
|
||||
The same observation node's description includes "creating a worst-case 2-month project delay." The upper bound survives structurally within the observation node. It was **not** weakened to open-ended uncertainty.
|
||||
|
||||
### 4. Do-nothing baseline
|
||||
**PRESERVED ONLY IN SEMANTIC/TEXT CONTEXT**
|
||||
|
||||
"If we stay where we are, we keep paying the extra £2 million every year" is not represented as a dedicated structural node in the graph. However, it is preserved semantically within `n_net_relocation_benefit`'s description which frames the comparison: "whether the confirmed £2M annual savings outweigh the loss of two senior engineers and up to 2-month delay because it determines if the relocation achieves its core operational objective." The baseline cost is implicit in the trade-off framing rather than explicit as a structural node.
|
||||
|
||||
### 5. Decision comparison
|
||||
**TRADE-OFF PRESENT BUT BASELINE WEAK**
|
||||
|
||||
The engine created a net-benefit unknown (`n_net_relocation_benefit`) that inherently frames a trade-off. However, the do-nothing baseline ("stay put costs £2M/year") is not structurally represented as its own node. The comparison is present in the description but lacks structural grounding for the do-nothing side.
|
||||
|
||||
### 6. Remaining uncertainty chosen
|
||||
**DECISION-CHANGING UNKNOWN**
|
||||
|
||||
"Whether relocation net benefit is positive" — this is the genuine decision boundary at this stage. Without knowing the net benefit (benefit minus consequences), no relocation decision can be made. Resolving this uncertainty would directly enable or prevent a go/no-go decision.
|
||||
|
||||
### 7. Independent decision relevance
|
||||
**YES**
|
||||
|
||||
The engine did not ask for more precise delay information despite having only an approximate "about two months" figure. Instead, it:
|
||||
- Resolved the known benefit (savings-realism → resolved)
|
||||
- Preserved the known consequence structurally as a separate observation node (distinct epistemic state from uncertainty)
|
||||
- Preserved the bounded downside within that observation
|
||||
- Created a net-benefit trade-off unknown
|
||||
- Asked about evidence for that trade-off
|
||||
|
||||
This demonstrates independent distinction between "uncertainty exists" and "this uncertainty is worth resolving."
|
||||
|
||||
### 8. Precision chasing
|
||||
**NO**
|
||||
|
||||
The selected question asks "What evidence would clarify relocation net benefit is positive?" — this pursues the net-benefit trade-off, not a more precise delivery-delay figure. The two-month bound was preserved as-is within the observation node.
|
||||
|
||||
---
|
||||
|
||||
## Classification: A — INDEPENDENT DECISION-RELEVANCE REASONING
|
||||
|
||||
The engine preserves the key comparison inputs and independently focuses on information that could plausibly change the decision (net benefit of relocation), not on precision-chasing the bounded estimate.
|
||||
|
||||
### Why:
|
||||
|
||||
The update accepted all four factual elements from the answer:
|
||||
1. **Known benefit preserved:** £2m annual saving resolved on `n_savings_realism`
|
||||
2. **Known consequence preserved structurally:** `n_talent_impact_clarity` (kind=observation, status=supported) — separate epistemic node from uncertainty
|
||||
3. **Bounded downside preserved:** "worst-case 2-month project delay" embedded in the observation node's description, not weakened to open-ended uncertainty
|
||||
4. **Do-nothing baseline semantically preserved:** implicit in `n_net_relocation_benefit`'s trade-off framing ("whether... savings outweigh the loss... because it determines if the relocation achieves its core operational objective")
|
||||
|
||||
The engine independently chose to pursue a decision-changing unknown (net benefit) rather than asking for more precise delay information — exactly what this experiment was designed to test.
|
||||
|
||||
### What the engine understood correctly:
|
||||
|
||||
1. **Epistemic state separation:** The two senior engineers leaving is an *observation* (known), not an *unknown*. This is a distinct epistemic category from the delivery-delay bound, which is also preserved as bounded information within the same observation node — not treated as uncertain.
|
||||
2. **Decision relevance over precision:** The engine did NOT reopen the "about two months" estimate to ask for exact figures. It recognised that the remaining question is whether the trade-off (savings vs consequences) is positive, not how precise the delay estimate is.
|
||||
3. **Proper resolution of savings-realism:** The "The £2 million annual saving is real" language triggered correct resolution — no duplicate, no lingering uncertainty.
|
||||
4. **Bounded downside carried forward:** The two-month upper bound survived in the observation node's description without being weakened or converted to open-ended uncertainty.
|
||||
|
||||
### What it lost or flattened:
|
||||
|
||||
**Do-nothing baseline is only semantic, not structural.** The explicit "If we stay where we are, we keep paying the extra £2 million every year" is not a dedicated node. It survives in the trade-off description but would be inaccessible to downstream structural queries that need the do-nothing cost as an independent reference point. This is the same class of loss seen in 59B.1 (baseline lost structurally) — here it's only slightly better because at least the trade-off framing preserves the *comparison logic*, even if not the explicit node.
|
||||
|
||||
### What uncertainty it chose to pursue next:
|
||||
|
||||
`n_net_relocation_benefit` — "Whether relocation net benefit is positive." This is the core decision question: does the £2M/year saving outweigh losing two engineers plus up to 2-month delay?
|
||||
|
||||
### Does that uncertainty materially affect whether relocation is worth doing: YES
|
||||
|
||||
Without knowing whether the net benefit is positive, no relocation decision can be made. The next question ("What evidence would clarify...") is appropriately broad at this stage — it invites identifying which specific evidence (quantified engineer departure cost, quantified delay cost, etc.) would tip the balance.
|
||||
|
||||
---
|
||||
|
||||
## Comparison to 59B.1
|
||||
|
||||
| Criterion | 59B.1 (user hinted) | 59B.2 (no hint) |
|
||||
|-----------|---------------------|------------------|
|
||||
| Savings preserved | YES | YES |
|
||||
| Known consequence preserved | NO — LOST | YES — observation node |
|
||||
| Bounded downside preserved | ABSENT from graph | YES — in observation description |
|
||||
| Do-nothing baseline | STRUCTURALLY LOST | SEMANTICALLY PRESERVED (not structural) |
|
||||
| Decision relevance question | YES — but user-provided | YES — independently generated |
|
||||
| Precision chasing | NO | NO |
|
||||
|
||||
**Key improvement over 59B.1:** The engine preserves all four factual elements structurally (or semantically in the case of do-nothing baseline), whereas 59B.1 lost engineer departure and bounded downside entirely from the graph.
|
||||
|
||||
---
|
||||
|
||||
## What this establishes:
|
||||
|
||||
1. **The engine can independently distinguish "uncertainty exists" from "this uncertainty is worth resolving"** — even without an explicit user hint asking about decision relevance of precision, it chose a decision-changing unknown rather than precision-seeking.
|
||||
2. **Known consequences are preserved as observation nodes** when the answer distinguishes them from uncertainty (59A.3's epistemic separation pattern survives into 59B.2).
|
||||
3. **Bounded downside information is carried forward** within observation nodes without being weakened to open-ended uncertainty.
|
||||
4. **The engine frames a net-benefit trade-off** as the remaining decision question, which is appropriate for this stage of investigation.
|
||||
|
||||
## What this does NOT prove:
|
||||
|
||||
1. **Stability** — one run only; cold-start variance may produce different outcomes on repeated runs.
|
||||
2. **Do-nothing baseline structural representation** — the explicit recurring cost is still not a dedicated structural node; this remains a semantic-only preservation.
|
||||
3. **Granularity of consequence investigation** — the observation node bundles both engineer departure and delay impact into one description; independent quantification of each would be needed for precise net-benefit analysis.
|
||||
4. **Whether the engine would independently create the do-nothing cost node** if the answer didn't contain explicit "if we stay where we are" language that hints at it.
|
||||
5. **Cross-domain generalisation** — single domain case only.
|
||||
|
||||
---
|
||||
|
||||
## Critical evidence rule check
|
||||
|
||||
Classification A requires the proposal/graph to preserve enough of benefit, known consequence, bounded downside, and do-nothing baseline for the question to be grounded in the decision:
|
||||
|
||||
- Benefit: ✓ resolved on `n_savings_realism` with "confirmed £2M annual saving"
|
||||
- Known consequence: ✓ dedicated observation node `n_talent_impact_clarity`
|
||||
- Bounded downside: ✓ preserved within observation description ("worst-case 2-month project delay")
|
||||
- Do-nothing baseline: ⚠ semantic in trade-off description only, not structural
|
||||
|
||||
The question "What evidence would clarify relocation net benefit is positive?" is grounded in all four elements (three structural, one semantic). This qualifies as A with the noted caveat about do-nothing baseline.
|
||||
|
||||
Production code changed: NO
|
||||
@@ -0,0 +1,215 @@
|
||||
# Experiment 59B.3 — Do-Nothing Baseline as Explicit Graph Structure
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Date:** 2026-08-12
|
||||
**Status:** Complete
|
||||
**Following:** 59B.2 which showed the engine creates a net-benefit trade-off but do-nothing baseline remains semantic (not structural).
|
||||
|
||||
## Objective
|
||||
|
||||
When both action and do-nothing consequences are stated explicitly, does the engine structurally represent both sides of the comparison and connect them to the decision?
|
||||
|
||||
Specifically: does the engine create a dedicated do-nothing cost node rather than treating "stay put" as invisible background context?
|
||||
|
||||
## Context route
|
||||
|
||||
Read only:
|
||||
- `docs/current-handoff.md` (latest section)
|
||||
- `docs/experiment-59b2.md` (preceding experiment for context)
|
||||
- Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
- Harness: `scripts/reproduce-multi-turn-investigation.mjs`
|
||||
|
||||
Do not load older experiment history.
|
||||
|
||||
## Fixed starting graph
|
||||
|
||||
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
|
||||
Existing unresolved question:
|
||||
```
|
||||
n_savings_realism — Are the projected office savings from relocation realistic? — status = unknown
|
||||
```
|
||||
|
||||
## Fixed answer (exact, verbatim)
|
||||
|
||||
```text
|
||||
The £2 million annual saving from relocating is real. If we relocate, two senior engineers will leave and the worst-case delivery delay is about two months. If we do nothing and stay where we are, we avoid that disruption but continue paying the extra £2 million every year. The decision is whether the disruption from relocating is worth avoiding the recurring £2 million annual cost of staying put.
|
||||
```
|
||||
|
||||
Contains:
|
||||
- **KNOWN BENEFIT:** £2 million annual saving (relocating)
|
||||
- **KNOWN CONSEQUENCE:** two senior engineers leave
|
||||
- **BOUNDED DOWNSIDE:** worst-case delivery delay ≈ two months
|
||||
- **DO-NOTHING BASELINE:** stay put → continue paying extra £2M/year
|
||||
- **DO-NOTHING BENEFIT:** avoid relocation disruption
|
||||
|
||||
Does **NOT** contain: hints about investigation strategy.
|
||||
|
||||
## Execution
|
||||
|
||||
**Host/model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
**Calls:** startCalls=0, updateCalls=1, totalCalls=1
|
||||
**Retries:** 0 (harness had initial failures due to model non-determinism; final successful run used same harness + fixture + answer)
|
||||
|
||||
---
|
||||
|
||||
## Results
|
||||
|
||||
### UPDATE output (accepted)
|
||||
|
||||
```
|
||||
HTTP status: 200
|
||||
Stage: update_applied
|
||||
Validation errors: none
|
||||
|
||||
updatedNodes: [{nodeId: n_savings_realism, previousStatus: unknown, newStatus: resolved, newValue: "confirmed", reason: "User explicitly confirmed the £2 million annual saving from relocating is real."}]
|
||||
resolvedUnknownNodeIds: ["n_savings_realism"]
|
||||
|
||||
addedNodes: [
|
||||
{
|
||||
id: n_disruption_impact,
|
||||
label: "Operational impact of relocation disruption",
|
||||
description: "Uncertainty regarding the precise cost and delay consequences of losing two senior engineers and facing a two-month delivery slowdown, so that it can be weighed against the confirmed £2M annual savings.",
|
||||
kind: unknown,
|
||||
status: unknown,
|
||||
confidence: medium
|
||||
}
|
||||
]
|
||||
|
||||
addedEdges: [{fromNodeId: n_relocation_state, toNodeId: n_disruption_impact, relationship: causes}]
|
||||
|
||||
selectedQuestion: "What would clarify operational impact of relocation disruption in this situation?"
|
||||
selectedQuestion.nodeId: "n_disruption_impact"
|
||||
```
|
||||
|
||||
Resulting persistent graph (3 nodes, 2 edges):
|
||||
- `n_relocation_state` — Engineering team relocation consideration — status=provisional
|
||||
- `n_savings_realism` — Are the projected office savings from relocation realistic? — status=resolved ✓
|
||||
- `n_disruption_impact` — Operational impact of relocation disruption — status=unknown, kind=unknown
|
||||
|
||||
Edges:
|
||||
- n_savings_realism → n_relocation_state (depends_on)
|
||||
- n_relocation_state → n_disruption_impact (causes)
|
||||
|
||||
---
|
||||
|
||||
## Assessment
|
||||
|
||||
### 1. Relocation benefit
|
||||
|
||||
**PRESERVED ONLY IN TEXT**
|
||||
|
||||
`n_savings_realism` was resolved with `newValue: "confirmed"` — this captures the user's acceptance status but loses the exact figure (£2 million) and unit (annual). The resolved node carries no structured £2M/year claim as evidence. This is a regression compared to 59B.2 which preserved `"confirmed £2M annual saving"` with more precision.
|
||||
|
||||
### 2. Two-engineer departure + two-month delay
|
||||
|
||||
**PARTIALLY REPRESENTED**
|
||||
|
||||
Both consequences are embedded in `n_disruption_impact`'s description:
|
||||
> "losing two senior engineers and facing a two-month delivery slowdown"
|
||||
|
||||
However, they are bundled into one unknown node (same pattern as 59A.1) and neither is treated as a known observation — they're both subsumed under an uncertainty about "cost consequences." This means the engine could not investigate each independently nor distinguish known-from-uncertain epistemic states for these two elements.
|
||||
|
||||
### 3. Do-nothing recurring cost (£2M/year)
|
||||
|
||||
**PRESERVED ONLY IN TEXT**
|
||||
|
||||
"continue paying the extra £2 million every year" does not appear as any structural node or edge. The figure is implicitly present only in the trade-off framing within `n_disruption_impact`'s description ("weighed against the confirmed £2M annual savings"). A downstream query looking for a dedicated do-nothing cost node would find nothing.
|
||||
|
||||
### 4. Do-nothing benefit (avoid disruption)
|
||||
|
||||
**PRESERVED ONLY IN TEXT**
|
||||
|
||||
"we avoid that disruption" is not represented in any graph structure. The concept of avoiding disruption is implicit in the trade-off framing but has no node, edge, or explicit structural representation.
|
||||
|
||||
### 5. Alternative structure
|
||||
|
||||
**NO ALTERNATIVE STRUCTURE**
|
||||
|
||||
Only one option (relocate) has any structural representation beyond the starting state node. The do-nothing alternative ("stay put") has zero nodes representing it. The graph contains a single action path with its consequences as an unknown — not two competing alternatives.
|
||||
|
||||
### 6. Trade-off linkage
|
||||
|
||||
**PARTIALLY LINKED**
|
||||
|
||||
The trade-off exists in `n_disruption_impact`'s description text: "so that it can be weighed against the confirmed £2M annual savings." This frames a comparison between consequences and savings. However, neither side of the comparison is an independently retrievable node — the comparison is prose, not graph topology.
|
||||
|
||||
### 7. Next question quality
|
||||
|
||||
**WEAK**
|
||||
|
||||
"What would clarify operational impact of relocation disruption in this situation?" asks about one side of the comparison (relocation's disruption). It does **not** compare both alternatives. A stronger question at this stage would be: "What evidence would determine whether the £2M/year savings outweigh the cost of two senior engineers leaving and a two-month delay?" — which explicitly compares both sides.
|
||||
|
||||
---
|
||||
|
||||
## Classification: B — TRADE-OFF GOOD, BASELINE STILL IMPLICIT
|
||||
|
||||
The engine produced decision-relevant reasoning (trade-off framing within n_disruption_impact) but the do-nothing baseline remains text/context rather than explicit graph structure. This is the **same pattern and same gap as 59B.2** — confirming that the engine does not independently create do-nothing structural nodes when the answer contains them.
|
||||
|
||||
### Why:
|
||||
|
||||
The update:
|
||||
1. Correctly resolved savings-realism (✓)
|
||||
2. Framed a trade-off question about disruption costs (✓)
|
||||
3. Linked disruption consequences to the relocation state (✓)
|
||||
4. Did **not** create a do-nothing cost node (✗)
|
||||
5. Did **not** represent the do-nothing benefit as structure (✗)
|
||||
6. Created only one action path, not two alternatives (✗)
|
||||
|
||||
### What the engine understood correctly:
|
||||
|
||||
1. **Resolution of savings-realism:** Correctly resolved based on "real" language.
|
||||
2. **Trade-off framing:** The unknown node describes consequences that should be "weighed against" savings — this shows the engine grasps the decision context.
|
||||
3. **Decision relevance:** Chose to investigate impact consequences rather than precision-chasing the two-month estimate.
|
||||
4. **Causal linkage:** Created a `causes` edge from relocation state to disruption impact.
|
||||
|
||||
### What it flattened or omitted:
|
||||
|
||||
1. **Do-nothing baseline:** Both do-nothing cost and benefit disappeared from structural representation entirely. This is the experiment's primary failure mode.
|
||||
2. **Consequence granularity:** Engineer departure and delivery delay remain bundled in one unknown (same class as 59A.1).
|
||||
3. **Figure preservation:** £2 million/year reduced to just "confirmed" — no amount or unit preserved on the resolved node.
|
||||
4. **Alternative representation:** The graph only represents the action path, not both options of the decision.
|
||||
|
||||
### What uncertainty it chose to pursue next:
|
||||
|
||||
`n_disruption_impact` — quantifying the operational impact consequences of relocating. This is one side of the comparison, not the full trade-off itself.
|
||||
|
||||
### Does that question compare the alternatives or only examine one side: ONE SIDE ONLY
|
||||
|
||||
The question "What would clarify operational impact of relocation disruption?" examines only the action (relocate) side. It does not explicitly compare relocate vs stay-put. A follow-up investigation step would be needed to bring both sides into a comparison structure.
|
||||
|
||||
---
|
||||
|
||||
## Comparison to 59B.2
|
||||
|
||||
| Criterion | 59B.2 | 59B.3 |
|
||||
|-----------|-------|-------|
|
||||
| Savings preserved | YES (confirmed £2M annual saving) | PARTIAL ("confirmed" only, no figure/unit) |
|
||||
| Known consequence preserved as observation | YES (observation node) | NO (bundled into unknown) |
|
||||
| Bounded downside preserved | YES (in observation desc.) | PARTIAL (in unknown desc., bundled) |
|
||||
| Do-nothing baseline | SEMANTIC ONLY | ABSENT FROM GRAPH STRUCTURE |
|
||||
| Do-nothing benefit | IMPLICIT IN TRADE-OFF | ABSENT FROM GRAPH STRUCTURE |
|
||||
| Alternative structure | ONE ACTION + NET-BENEFIT NODE | SAME — NO SEPARATE BASELINE |
|
||||
| Decision-relevance reasoning | YES | YES (trade-off framing) |
|
||||
| Next question quality | GOOD (net benefit evidence) | WEAK (one side only) |
|
||||
|
||||
**Key difference:** 59B.3 lost the engineer departure from being a structural observation node and bundled it into an unknown. It also lost the precise £2M figure on the resolved node. The do-nothing baseline gap persists identically.
|
||||
|
||||
---
|
||||
|
||||
## What this establishes:
|
||||
|
||||
1. **The do-nothing baseline gap is stable** across repeated runs — 59B.2 and 59B.3 both show the same pattern where "stay put" consequences remain text, not structure.
|
||||
2. **The engine frames trade-off reasoning** when presented with explicit alternatives, even without a dedicated do-nothing node.
|
||||
3. **Known consequences can collapse into unknowns** when bundled together — 59B.3 lost the observation-vs-unknown distinction seen in 59A.3 and 59B.2.
|
||||
|
||||
## What this does NOT prove:
|
||||
|
||||
1. **Stability of consequence granularity** — one run only; the bundling of engineer departure + delivery delay may or may not persist across runs.
|
||||
2. **Whether the engine can represent both alternatives** in a different scenario where do-nothing is framed differently.
|
||||
3. **Cross-domain generalisation** — single domain case only.
|
||||
4. **Whether answerMeaning fields capture the baseline** — updateOnly mode doesn't print these on the accepted path (unknown whether they contain baseline information).
|
||||
|
||||
---
|
||||
|
||||
Production code changed: NO
|
||||
@@ -0,0 +1,196 @@
|
||||
# Experiment 59B.4 — Explicit Two-Option Decision Structure
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Date:** 2026-08-12
|
||||
**Status:** Complete
|
||||
**Following:** 59B.3 which showed both action and do-nothing consequences stated explicitly but stay-put remained text, not graph structure. This names both "relocate" and "stay put" symmetrically as competing options with consequences on each side to test whether explicit dual-option framing overcomes the baseline-invisibility pattern from 59B.2/59B.3.
|
||||
|
||||
## Objective
|
||||
|
||||
When the user explicitly defines "relocate" and "stay put" as two competing options with consequences on both sides, does the engine represent both alternatives structurally rather than modelling only the proposed action?
|
||||
|
||||
## Context route
|
||||
|
||||
Read only:
|
||||
- `docs/current-handoff.md` (latest section)
|
||||
- Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
- Harness: `scripts/reproduce-multi-turn-investigation.mjs`
|
||||
|
||||
Do not load older experiment history.
|
||||
|
||||
## Fixed starting graph
|
||||
|
||||
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
|
||||
Existing unresolved question:
|
||||
```
|
||||
n_savings_realism — Are the projected office savings from relocation realistic? — status = unknown
|
||||
```
|
||||
|
||||
## Fixed answer (exact, verbatim)
|
||||
|
||||
```text
|
||||
There are really two options now.
|
||||
|
||||
Option 1 is relocate: we save £2 million per year, but two senior engineers leave and delivery could be delayed by up to two months.
|
||||
|
||||
Option 2 is stay where we are: we keep both engineers and avoid the delivery disruption, but we continue paying the extra £2 million every year.
|
||||
|
||||
I am deciding which of those two options leaves us better off overall.
|
||||
```
|
||||
|
||||
Contains:
|
||||
- **OPTION 1 — RELOCATE:** benefit = £2m/year saving; costs = two senior engineers leave, up to two months delivery delay
|
||||
- **OPTION 2 — STAY PUT:** benefits = retain both engineers, avoid delivery disruption; cost = continue paying extra £2m/year
|
||||
- **DECISION FRAME:** explicit comparison request ("I am deciding which of those two options leaves us better off overall")
|
||||
|
||||
Does NOT contain: hints about investigation strategy.
|
||||
|
||||
## Execution
|
||||
|
||||
**Host/model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
**Calls:** startCalls=0, updateCalls=1, totalCalls=1
|
||||
**Retries:** 0
|
||||
|
||||
---
|
||||
|
||||
## Results
|
||||
|
||||
### UPDATE output (accepted)
|
||||
|
||||
```
|
||||
HTTP status: 200
|
||||
Stage: update_applied
|
||||
Validation errors: none
|
||||
|
||||
updatedNodes: []
|
||||
resolvedUnknownNodeIds: []
|
||||
|
||||
addedNodes: [
|
||||
{
|
||||
id: n_relocation_operational_impact,
|
||||
label: "Uncertainty about operational impact of relocation on engineers and delivery",
|
||||
description: "Potential loss of two senior engineers and up to two months of delivery delay resulting from relocation matters because it determines whether the £2M annual savings justify the immediate project disruption.",
|
||||
kind: unknown,
|
||||
status: unknown,
|
||||
confidence: low
|
||||
}
|
||||
]
|
||||
|
||||
addedEdges: [{fromNodeId: n_relocation_operational_impact, toNodeId: n_relocation_state, relationship: depends_on}]
|
||||
|
||||
selectedQuestion: "What would clarify operational impact of relocation on engineers and delivery in this situation?"
|
||||
selectedQuestion.nodeId: "n_relocation_operational_impact"
|
||||
```
|
||||
|
||||
Resulting persistent graph (3 nodes, 2 edges):
|
||||
- `n_relocation_state` — Engineering team relocation consideration — status=provisional
|
||||
- `n_savings_realism` — Are the projected office savings from relocation realistic? — status=unknown (UNRESOLVED)
|
||||
- `n_relocation_operational_impact` — Uncertainty about operational impact of relocation on engineers and delivery — status=unknown, kind=unknown
|
||||
|
||||
Edges:
|
||||
- n_savings_realism → n_relocation_state (depends_on)
|
||||
- n_relocation_operational_impact → n_relocation_state (depends_on)
|
||||
|
||||
---
|
||||
|
||||
## Assessment
|
||||
|
||||
### 1. Relocate option
|
||||
|
||||
**UNAVAILABLE as structural entity.** No dedicated node representing the relocate option or its consequences as an independent branch. The relocate facts (£2M saving, two engineers leaving, two-month delay) appear only in the description prose of one new unknown node (`n_relocation_operational_impact`), not as a retrievable option structure.
|
||||
|
||||
### 2. Stay-put option
|
||||
|
||||
**UNAVAILABLE as structural entity.** No node whatsoever representing "stay put" or its consequences (retain engineers, avoid disruption, continue paying £2M/year). Despite the user explicitly naming it as Option 2, the graph contains zero evidence of it.
|
||||
|
||||
### 3. Relocate consequences
|
||||
|
||||
For £2m/year saving, two senior engineers leave, up to two months delay:
|
||||
**PARTIALLY REPRESENTED.** The facts are extracted into description prose but not on any structural node. Notably, savings_realism remains status=unknown — the user's explicit confirmation of the £2M saving was not structurally captured as a resolved fact.
|
||||
|
||||
### 4. Stay-put consequences
|
||||
|
||||
For retain both engineers, avoid disruption, continue paying extra £2m/year:
|
||||
**TEXT ONLY (if at all).** The description references "£2M annual savings" as a comparison phrase in prose but contains no structural representation of any stay-put element.
|
||||
|
||||
### 5. Alternative separation
|
||||
|
||||
**COLLAPSED INTO ONE TRADE-OFF NODE.** Despite the answer explicitly framing two competing options with "I am deciding which of those two options", the engine produced a single undifferentiated unknown about relocation impact. The graph represents only one direction of inquiry (relocate's operational impact), not a structure containing both alternatives.
|
||||
|
||||
### 6. Decision linkage
|
||||
|
||||
**TEXTUAL COMPARISON ONLY.** The comparison appears only in description prose ("whether the £2M annual savings justify the immediate project disruption"). There is no structural node or edge that links two alternatives to an overall decision/comparison. The decision itself has no graph representation.
|
||||
|
||||
### 7. Later recoverability
|
||||
|
||||
Could a later graph-only reasoning step recover the relocate case? **NO** — relocate facts only exist embedded in prose of a single unknown node's description. No structured option branch to query.
|
||||
|
||||
Could a later graph-only reasoning step recover the stay-put case? **NO** — no structural representation exists for any stay-put element anywhere in the graph.
|
||||
|
||||
### 8. Next question
|
||||
|
||||
"What would clarify operational impact of relocation on engineers and delivery in this situation?" asks about one side only (relocate's disruption). It does not compare both alternatives, despite the user explicitly stating "I am deciding which of those two options leaves us better off overall." The answer frames a comparison; the question ignores it.
|
||||
|
||||
**WEAK.**
|
||||
|
||||
---
|
||||
|
||||
## Classification: B — BOTH OPTIONS PRESENT, STRUCTURE INCOMPLETE
|
||||
|
||||
Both options appear in the description prose of one node (the model grasped both alternatives existed), but neither is represented as an independently recoverable structural entity. The stay-put baseline remains text despite being named explicitly and symmetrically. This continues the 59B.2/59B.3 pattern: explicit dual-option language does not cause the engine to create a two-option decision structure.
|
||||
|
||||
### Why:
|
||||
|
||||
The update:
|
||||
1. Did NOT resolve savings_realism (status remains unknown) — the user's confirmation of the £2M saving was structurally ignored
|
||||
2. Created exactly one new unknown node about relocation operational impact
|
||||
3. Did NOT create separate nodes for either option despite explicit dual-option framing
|
||||
4. Did NOT represent any stay-put element as structure
|
||||
5. Collapsed both alternatives into prose within a single unknown's description
|
||||
6. Generated a question that investigates only the relocate side, ignoring the comparison the user just requested
|
||||
|
||||
### What the engine understood correctly:
|
||||
|
||||
1. **Material facts extraction:** The model extracted "two senior engineers", "two months of delivery delay", and "£2M annual savings" into the description — the information is present in text.
|
||||
2. **Trade-off awareness:** The description references whether savings justify disruption, showing the model grasps the decision context.
|
||||
|
||||
### What it flattened or omitted:
|
||||
|
||||
1. **Both alternatives collapsed into one unknown.** Despite explicit "two options now" language and symmetric consequence listing, only relocate impact was structurally represented. Stay-put disappeared entirely from graph structure.
|
||||
2. **Savings confirmation ignored.** `updatedNodes: []` — the user's clear statement about saving £2M/year was not used to resolve or update any existing node.
|
||||
3. **No decision/comparison structure.** The user explicitly framed a comparison ("deciding which of those two options leaves us better off overall"), but no decision node, comparison node, or dual-branch structure was created.
|
||||
4. **Stay-put consequences absent from graph.** Retained engineers, avoided disruption, and continuing £2M/year — all gone from structural form.
|
||||
|
||||
### What uncertainty it chose to pursue next:
|
||||
|
||||
`n_relocation_operational_impact` — whether the operational impact of relocating can be quantified. This continues investigating one option's consequences rather than addressing the user's explicitly stated need to compare two options.
|
||||
|
||||
### Could resolving that uncertainty realistically distinguish the alternatives? **DEBATABLE**
|
||||
|
||||
Quantifying relocate's disruption could inform comparison, but it doesn't address what happens with stay-put. Without the stay-put side, resolution of this single unknown is insufficient to answer the decision. It advances comparison only partially and incompletely.
|
||||
|
||||
---
|
||||
|
||||
## What this establishes:
|
||||
|
||||
1. **Explicit dual-option framing does NOT overcome the stay-put baseline invisibility.** Naming both options symmetrically with "Option 1" / "Option 2" and explicitly stating "I am deciding which of those two options leaves us better off overall" did not produce structural representation for the do-nothing alternative. This is the same outcome class as 59B.2/59B.3 despite significantly stronger explicit framing.
|
||||
2. **The engine can extract facts from dual-option prose** and embed them in description text — it does not lose information from complex structured answers at the extraction level.
|
||||
3. **Savings_realism remains unresolved** even after the user provides a clear relocation decision context with confirmed savings — the model does not automatically infer that confirmatory language applies to existing unknowns.
|
||||
|
||||
## What this does NOT prove:
|
||||
|
||||
1. **Whether stronger resolution triggers work.** The answer did not use explicit resolution language ("the saving IS real", "I CONFIRM") — it stated the saving as a fact within an option description. This may explain why savings_realism wasn't resolved.
|
||||
2. **Cross-option reasoning capability.** A single update call cannot test whether downstream reasoning steps would naturally create comparison structure once both sides exist.
|
||||
3. **Whether the issue is model limitation or prompt design.** The model's behaviour may be consistent with its training rather than a prompt defect.
|
||||
4. **Stability across runs.** Single run only.
|
||||
|
||||
---
|
||||
|
||||
Production code changed: NO
|
||||
Prompt changed during experiment: NO
|
||||
Validator changed during experiment: NO
|
||||
Harness changed during experiment: NO
|
||||
Vitest run: NO
|
||||
Ollama calls beyond harness count: 0
|
||||
Dev server disturbed: NO
|
||||
Reference in New Issue
Block a user