216 lines
8.9 KiB
Markdown
216 lines
8.9 KiB
Markdown
# Experiment 59A.3 — Known vs Uncertain Consequence Structure
|
|
|
|
**Branch:** `feature/question-formulation-v0.24`
|
|
**Date:** 2026-08-12
|
|
**Status:** Complete
|
|
**Following:** 59A.2 which showed partial decomposition (delivery impact structured but engineer loss left only in text, no epistemic separation).
|
|
|
|
## Objective
|
|
|
|
When the user states one consequence as known ("two senior engineers will leave") and a downstream consequence as uncertain ("whether losing them would delay delivery"), does the proposal represent them as separate structural objects with different epistemic states?
|
|
|
|
This isolates that distinction without the extra £2m-resolution step from 59A.1/59A.2.
|
|
|
|
## Context route
|
|
|
|
Read only:
|
|
- `docs/current-handoff.md` (sections 59A.1, 59A.2)
|
|
- Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
|
- Harness: `scripts/reproduce-multi-turn-investigation.mjs`
|
|
|
|
Do not load older experiment history.
|
|
|
|
## Fixed starting graph
|
|
|
|
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
|
|
|
Existing unresolved question:
|
|
```
|
|
n_savings_realism — Are the projected office savings from relocation realistic? — status = unknown
|
|
```
|
|
|
|
## Fixed answer
|
|
|
|
```text
|
|
We know that relocating would cause two senior engineers to leave. What I don't know is whether losing them would delay delivery, or by how many months.
|
|
```
|
|
|
|
The answer contains two semantically different assertions:
|
|
1. **Known consequence:** two senior engineers will leave (epistemic state = known)
|
|
2. **Uncertain consequence:** whether/when that loss delays delivery (epistemic state = unknown)
|
|
|
|
Note: this version omits the £2m savings confirmation present in 59A.2, isolating only the known-vs-unknown distinction.
|
|
|
|
## Execution
|
|
|
|
**Host/model:** qwen-claude:latest at http://192.168.1.111:11434
|
|
**Calls:** startCalls=0, updateCalls=1, totalCalls=1
|
|
**Retries:** 0
|
|
|
|
---
|
|
|
|
## Results
|
|
|
|
### UPDATE output
|
|
|
|
```
|
|
HTTP status: 422
|
|
Stage: proposal_compatibility
|
|
Error: "selectedQuestion must be a single non-compound question"
|
|
```
|
|
|
|
### Rejected proposal snapshot (from diagnostics.rejectedProposalSnapshot)
|
|
|
|
#### answerMeaning
|
|
|
|
- `userSupportedMeaning`: "Relocating will cause two senior engineers to depart; it is currently unknown whether this loss will delay project delivery or determine the magnitude of such delays."
|
|
- `possibleInference`: "The financial savings from relocation may be negated by productivity losses and delivery delays caused by losing key technical staff."
|
|
|
|
#### Proposal mutations
|
|
|
|
```json
|
|
{
|
|
"updatedNodes": [],
|
|
"resolvedUnknownNodeIds": [],
|
|
"addedNodes": [
|
|
{
|
|
"id": "n_talent_departure",
|
|
"kind": "observation",
|
|
"label": "Senior engineer turnover from relocation",
|
|
"description": "Relocating would cause two senior engineers to leave.",
|
|
"parentId": null,
|
|
"dependsOn": ["n_relocation_state"],
|
|
"affects": ["n_delivery_delay_impact"],
|
|
"childIds": []
|
|
},
|
|
{
|
|
"id": "n_delivery_delay_impact",
|
|
"kind": "unknown",
|
|
"label": "Delivery delay impact from lost engineers",
|
|
"description": "Whether losing the two senior engineers will delay project delivery and by how many months; matters because it determines if productivity loss outweighs office savings.",
|
|
"parentId": null,
|
|
"dependsOn": ["n_talent_departure"],
|
|
"affects": ["n_relocation_state"],
|
|
"childIds": []
|
|
}
|
|
],
|
|
"addedEdges": [
|
|
{
|
|
"fromNodeId": "n_delivery_delay_impact",
|
|
"toNodeId": "n_talent_departure",
|
|
"relationship": "depends_on"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
#### selectedQuestion
|
|
|
|
`null` (no selected question produced — caused the rejection at proposal_compatibility)
|
|
|
|
---
|
|
|
|
## Reasoning Assessment
|
|
|
|
### Known engineer loss
|
|
|
|
- Created dedicated node `n_talent_departure`:
|
|
- kind = **observation** (not unknown, not provisional)
|
|
- label: "Senior engineer turnover from relocation"
|
|
- description: "Relocating would cause two senior engineers to leave."
|
|
- depends_on: n_relocation_state
|
|
- affects: [n_delivery_delay_impact]
|
|
- `affects` field carries a **typed structural link** to the downstream uncertainty node.
|
|
|
|
**Classification: SEPARATE KNOWN STRUCTURE**
|
|
|
|
The engineer departure is not embedded in text or left uncertain — it is its own observation node with status derived from kind=observation (a factual assertion, not an unresolved question). This is a correct epistemic state for a known consequence.
|
|
|
|
### Delivery impact
|
|
|
|
- Created dedicated node `n_delivery_delay_impact`:
|
|
- kind = **unknown**
|
|
- status = unknown
|
|
- label: "Delivery delay impact from lost engineers"
|
|
- description: "Whether losing the two senior engineers will delay project delivery and by how many months..."
|
|
- depends_on: [n_talent_departure]
|
|
- The `dependsOn` field is populated with the known-consequence node — a **typed structural link**.
|
|
|
|
**Classification: SEPARATE UNRESOLVED STRUCTURE**
|
|
|
|
Delivery uncertainty is its own unknown node with proper kind=status=unknown and structural linkage back to the known consequence via depends_on.
|
|
|
|
### Epistemic separation
|
|
|
|
- `n_talent_departure` (kind=observation) = known factual consequence
|
|
- `n_delivery_delay_impact` (kind=unknown, status=unknown) = unresolved uncertain consequence
|
|
- They are two distinct nodes with a typed `affects`/`depends_on` relationship between them.
|
|
|
|
**Classification: CLEARLY SEPARATED**
|
|
|
|
The epistemic distinction is preserved at the structural level — two different kinds, two different statuses, connected by typed edges.
|
|
|
|
### Relationship between engineer loss and delivery delay
|
|
|
|
- `n_talent_departure.affects = ["n_delivery_delay_impact"]`
|
|
- `n_delivery_delay_impact.dependsOn = ["n_talent_departure"]`
|
|
- Edge: n_delivery_delay_impact → n_talent_departure with relationship=depends_on
|
|
|
|
**Classification: TYPED / STRUCTURAL LINK**
|
|
|
|
The causal chain is represented by both a forward field (affects) and a reverse edge (depends_on), not just embedded in prose.
|
|
|
|
### Next question
|
|
|
|
No selectedQuestion was produced (null). The rejection was caused by the validator requiring "a single non-compound question."
|
|
|
|
Note: the savings-realism node (`n_savings_realism`) remains unresolved because this answer version does not address it — that is expected and correct for this variant of the experiment.
|
|
|
|
**Classification: NONE**
|
|
|
|
---
|
|
|
|
## Classification: A — CORRECT EPISTEMIC DECOMPOSITION
|
|
|
|
### Why:
|
|
|
|
This is a clean positive result. The model created two separate structural objects with distinct epistemic states:
|
|
|
|
1. **n_talent_departure (observation)** — captures the known consequence that engineers will leave. Not uncertain, not pending resolution. Its kind=observation signals "established fact to be taken into account."
|
|
2. **n_delivery_delay_impact (unknown)** — captures the unresolved downstream uncertainty about delivery impact magnitude, depending on the known departure.
|
|
|
|
The causal chain between them is represented via typed fields (affects/depends_on) and a typed edge (depends_on), not just text embedding.
|
|
|
|
The key distinction from 59A.2: in that experiment both consequences were compressed into one unknown node ("Impact of losing two senior engineers on delivery timelines"). Here they are separate nodes with different kinds — the known-vs-unknown boundary is structurally preserved.
|
|
|
|
**Important caveat:** The update was rejected at proposal_compatibility because no selectedQuestion was produced. This is a validator-side issue, not a semantic reasoning failure. The rejected snapshot demonstrates correct structural decomposition even though the update was not applied to the persistent graph.
|
|
|
|
### Did "two senior engineers will leave" become its own known structure: YES
|
|
### Did delivery delay remain explicitly unresolved: YES
|
|
### Did the graph/proposal preserve the distinction: YES
|
|
|
|
---
|
|
|
|
## What this establishes:
|
|
|
|
1. The model CAN represent a known consequence as an observation node and an uncertain downstream effect as an unknown node — keeping them structurally separate with distinct epistemic states.
|
|
2. A typed causal chain (affects + depends_on edge) can be produced between these two kinds of nodes in a single proposal.
|
|
3. Removing the £2m savings confirmation from the answer did not degrade the known-vs-unknown separation; it actually focused the model's attention on exactly what was being tested.
|
|
|
|
## What this does NOT prove:
|
|
|
|
1. **Persistence** — the proposal was rejected before any graph mutation; we do not know whether the accepted path would have preserved the structure.
|
|
2. **Next-question generation** — the selectedQuestion failure (null) was not resolved by this experiment. The model may struggle to formulate a single non-compound question when two structural consequences are introduced.
|
|
3. **Stability** — one run only; cold-start variance has been a factor across Experiments 59A series.
|
|
4. **Cross-domain generalisation** — single domain case only.
|
|
|
|
---
|
|
|
|
## Production code changed: NO
|
|
## Prompt changed: NO
|
|
## Validator changed: NO
|
|
## Harness changed: NO
|
|
## Vitest run: NO
|
|
## Ollama calls beyond harness count: 0
|
|
## Dev server disturbed: NO
|