experiment: test known-vs-uncertain consequence structure
This commit is contained in:
@@ -0,0 +1,215 @@
|
||||
# Experiment 59A.3 — Known vs Uncertain Consequence Structure
|
||||
|
||||
**Branch:** `feature/question-formulation-v0.24`
|
||||
**Date:** 2026-08-12
|
||||
**Status:** Complete
|
||||
**Following:** 59A.2 which showed partial decomposition (delivery impact structured but engineer loss left only in text, no epistemic separation).
|
||||
|
||||
## Objective
|
||||
|
||||
When the user states one consequence as known ("two senior engineers will leave") and a downstream consequence as uncertain ("whether losing them would delay delivery"), does the proposal represent them as separate structural objects with different epistemic states?
|
||||
|
||||
This isolates that distinction without the extra £2m-resolution step from 59A.1/59A.2.
|
||||
|
||||
## Context route
|
||||
|
||||
Read only:
|
||||
- `docs/current-handoff.md` (sections 59A.1, 59A.2)
|
||||
- Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
- Harness: `scripts/reproduce-multi-turn-investigation.mjs`
|
||||
|
||||
Do not load older experiment history.
|
||||
|
||||
## Fixed starting graph
|
||||
|
||||
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
|
||||
|
||||
Existing unresolved question:
|
||||
```
|
||||
n_savings_realism — Are the projected office savings from relocation realistic? — status = unknown
|
||||
```
|
||||
|
||||
## Fixed answer
|
||||
|
||||
```text
|
||||
We know that relocating would cause two senior engineers to leave. What I don't know is whether losing them would delay delivery, or by how many months.
|
||||
```
|
||||
|
||||
The answer contains two semantically different assertions:
|
||||
1. **Known consequence:** two senior engineers will leave (epistemic state = known)
|
||||
2. **Uncertain consequence:** whether/when that loss delays delivery (epistemic state = unknown)
|
||||
|
||||
Note: this version omits the £2m savings confirmation present in 59A.2, isolating only the known-vs-unknown distinction.
|
||||
|
||||
## Execution
|
||||
|
||||
**Host/model:** qwen-claude:latest at http://192.168.1.111:11434
|
||||
**Calls:** startCalls=0, updateCalls=1, totalCalls=1
|
||||
**Retries:** 0
|
||||
|
||||
---
|
||||
|
||||
## Results
|
||||
|
||||
### UPDATE output
|
||||
|
||||
```
|
||||
HTTP status: 422
|
||||
Stage: proposal_compatibility
|
||||
Error: "selectedQuestion must be a single non-compound question"
|
||||
```
|
||||
|
||||
### Rejected proposal snapshot (from diagnostics.rejectedProposalSnapshot)
|
||||
|
||||
#### answerMeaning
|
||||
|
||||
- `userSupportedMeaning`: "Relocating will cause two senior engineers to depart; it is currently unknown whether this loss will delay project delivery or determine the magnitude of such delays."
|
||||
- `possibleInference`: "The financial savings from relocation may be negated by productivity losses and delivery delays caused by losing key technical staff."
|
||||
|
||||
#### Proposal mutations
|
||||
|
||||
```json
|
||||
{
|
||||
"updatedNodes": [],
|
||||
"resolvedUnknownNodeIds": [],
|
||||
"addedNodes": [
|
||||
{
|
||||
"id": "n_talent_departure",
|
||||
"kind": "observation",
|
||||
"label": "Senior engineer turnover from relocation",
|
||||
"description": "Relocating would cause two senior engineers to leave.",
|
||||
"parentId": null,
|
||||
"dependsOn": ["n_relocation_state"],
|
||||
"affects": ["n_delivery_delay_impact"],
|
||||
"childIds": []
|
||||
},
|
||||
{
|
||||
"id": "n_delivery_delay_impact",
|
||||
"kind": "unknown",
|
||||
"label": "Delivery delay impact from lost engineers",
|
||||
"description": "Whether losing the two senior engineers will delay project delivery and by how many months; matters because it determines if productivity loss outweighs office savings.",
|
||||
"parentId": null,
|
||||
"dependsOn": ["n_talent_departure"],
|
||||
"affects": ["n_relocation_state"],
|
||||
"childIds": []
|
||||
}
|
||||
],
|
||||
"addedEdges": [
|
||||
{
|
||||
"fromNodeId": "n_delivery_delay_impact",
|
||||
"toNodeId": "n_talent_departure",
|
||||
"relationship": "depends_on"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### selectedQuestion
|
||||
|
||||
`null` (no selected question produced — caused the rejection at proposal_compatibility)
|
||||
|
||||
---
|
||||
|
||||
## Reasoning Assessment
|
||||
|
||||
### Known engineer loss
|
||||
|
||||
- Created dedicated node `n_talent_departure`:
|
||||
- kind = **observation** (not unknown, not provisional)
|
||||
- label: "Senior engineer turnover from relocation"
|
||||
- description: "Relocating would cause two senior engineers to leave."
|
||||
- depends_on: n_relocation_state
|
||||
- affects: [n_delivery_delay_impact]
|
||||
- `affects` field carries a **typed structural link** to the downstream uncertainty node.
|
||||
|
||||
**Classification: SEPARATE KNOWN STRUCTURE**
|
||||
|
||||
The engineer departure is not embedded in text or left uncertain — it is its own observation node with status derived from kind=observation (a factual assertion, not an unresolved question). This is a correct epistemic state for a known consequence.
|
||||
|
||||
### Delivery impact
|
||||
|
||||
- Created dedicated node `n_delivery_delay_impact`:
|
||||
- kind = **unknown**
|
||||
- status = unknown
|
||||
- label: "Delivery delay impact from lost engineers"
|
||||
- description: "Whether losing the two senior engineers will delay project delivery and by how many months..."
|
||||
- depends_on: [n_talent_departure]
|
||||
- The `dependsOn` field is populated with the known-consequence node — a **typed structural link**.
|
||||
|
||||
**Classification: SEPARATE UNRESOLVED STRUCTURE**
|
||||
|
||||
Delivery uncertainty is its own unknown node with proper kind=status=unknown and structural linkage back to the known consequence via depends_on.
|
||||
|
||||
### Epistemic separation
|
||||
|
||||
- `n_talent_departure` (kind=observation) = known factual consequence
|
||||
- `n_delivery_delay_impact` (kind=unknown, status=unknown) = unresolved uncertain consequence
|
||||
- They are two distinct nodes with a typed `affects`/`depends_on` relationship between them.
|
||||
|
||||
**Classification: CLEARLY SEPARATED**
|
||||
|
||||
The epistemic distinction is preserved at the structural level — two different kinds, two different statuses, connected by typed edges.
|
||||
|
||||
### Relationship between engineer loss and delivery delay
|
||||
|
||||
- `n_talent_departure.affects = ["n_delivery_delay_impact"]`
|
||||
- `n_delivery_delay_impact.dependsOn = ["n_talent_departure"]`
|
||||
- Edge: n_delivery_delay_impact → n_talent_departure with relationship=depends_on
|
||||
|
||||
**Classification: TYPED / STRUCTURAL LINK**
|
||||
|
||||
The causal chain is represented by both a forward field (affects) and a reverse edge (depends_on), not just embedded in prose.
|
||||
|
||||
### Next question
|
||||
|
||||
No selectedQuestion was produced (null). The rejection was caused by the validator requiring "a single non-compound question."
|
||||
|
||||
Note: the savings-realism node (`n_savings_realism`) remains unresolved because this answer version does not address it — that is expected and correct for this variant of the experiment.
|
||||
|
||||
**Classification: NONE**
|
||||
|
||||
---
|
||||
|
||||
## Classification: A — CORRECT EPISTEMIC DECOMPOSITION
|
||||
|
||||
### Why:
|
||||
|
||||
This is a clean positive result. The model created two separate structural objects with distinct epistemic states:
|
||||
|
||||
1. **n_talent_departure (observation)** — captures the known consequence that engineers will leave. Not uncertain, not pending resolution. Its kind=observation signals "established fact to be taken into account."
|
||||
2. **n_delivery_delay_impact (unknown)** — captures the unresolved downstream uncertainty about delivery impact magnitude, depending on the known departure.
|
||||
|
||||
The causal chain between them is represented via typed fields (affects/depends_on) and a typed edge (depends_on), not just text embedding.
|
||||
|
||||
The key distinction from 59A.2: in that experiment both consequences were compressed into one unknown node ("Impact of losing two senior engineers on delivery timelines"). Here they are separate nodes with different kinds — the known-vs-unknown boundary is structurally preserved.
|
||||
|
||||
**Important caveat:** The update was rejected at proposal_compatibility because no selectedQuestion was produced. This is a validator-side issue, not a semantic reasoning failure. The rejected snapshot demonstrates correct structural decomposition even though the update was not applied to the persistent graph.
|
||||
|
||||
### Did "two senior engineers will leave" become its own known structure: YES
|
||||
### Did delivery delay remain explicitly unresolved: YES
|
||||
### Did the graph/proposal preserve the distinction: YES
|
||||
|
||||
---
|
||||
|
||||
## What this establishes:
|
||||
|
||||
1. The model CAN represent a known consequence as an observation node and an uncertain downstream effect as an unknown node — keeping them structurally separate with distinct epistemic states.
|
||||
2. A typed causal chain (affects + depends_on edge) can be produced between these two kinds of nodes in a single proposal.
|
||||
3. Removing the £2m savings confirmation from the answer did not degrade the known-vs-unknown separation; it actually focused the model's attention on exactly what was being tested.
|
||||
|
||||
## What this does NOT prove:
|
||||
|
||||
1. **Persistence** — the proposal was rejected before any graph mutation; we do not know whether the accepted path would have preserved the structure.
|
||||
2. **Next-question generation** — the selectedQuestion failure (null) was not resolved by this experiment. The model may struggle to formulate a single non-compound question when two structural consequences are introduced.
|
||||
3. **Stability** — one run only; cold-start variance has been a factor across Experiments 59A series.
|
||||
4. **Cross-domain generalisation** — single domain case only.
|
||||
|
||||
---
|
||||
|
||||
## Production code changed: NO
|
||||
## Prompt changed: NO
|
||||
## Validator changed: NO
|
||||
## Harness changed: NO
|
||||
## Vitest run: NO
|
||||
## Ollama calls beyond harness count: 0
|
||||
## Dev server disturbed: NO
|
||||
Reference in New Issue
Block a user