robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: rerun option consequence structure once

Browse Source
This commit is contained in:
robbond
2026-08-13 06:08:23 +01:00
parent 56a04ddd0e
commit 2016a024c5
2 changed files with 284 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+72
View File
@@ -556,6 +556,78 @@ This is an improvement over 59B.4 (where do-nothing had no structural presence)
---
### Experiment 60A.6 — First-Call Option Consequence Structure Confirmation
**Branch:** `feature/decision-options-v0.25`
**Date:** 2026-08-13
**Status:** Complete
**Type:** LIVE RUN — Hard one-call boundary repeat of 60A.5's reasoning case, testing whether the first proposal succeeds and preserves option-attributed consequences without retry.
## Objective
Does the model preserve known consequences under the correct option branch in the first proposal, without requiring a retry?
## Fixed Starting Graph
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
Pre-existing uncertainty: `n_savings_realism` (status=unknown)
## Fixed Answer (verbatim)
> There are two options.
>
> If we relocate, we save £2 million per year, two senior engineers will definitely leave, and delivery will be delayed by no more than two months.
>
> If we stay put, we retain both senior engineers, avoid the relocation delay, and continue paying the extra £2 million every year.
>
> Those consequences are known. What I still do not know is which option leaves us better off overall.
## Execution
Host/model: qwen-claude:latest at http://192.168.1.111:11434. startCalls=0, updateCalls=1, totalCalls=1. Hard one-call boundary.
## Result
**HTTP status:** 200 — first call succeeds, no validation rejection.
### Structure (5 nodes, 3 edges)
| Node | Kind | Status |
|------|------|--------|
| n_savings_realism | unknown | resolved |
| opt_relocate | option | **known** |
| opt_stay_put | option | **known** |
| n_relocation_decision | unknown | unknown |
Edges: both options → n_relocation_decision via `contained_in`; n_savings_realism → n_relocation_state via `depends_on`.
### Consequences — OPTION-OWNED DESCRIPTION
Both relocate and stay-put consequences present in their respective option descriptions with status=known. All six facts preserved correctly with no cross-contamination. Option epistemic status now correctly reflects the user's stated position ("Those consequences are known").
### Epistemic correctness: CORRECT
n_savings_realism resolved; known consequences remain known (not reopened as unknowns).
### Classification: A — FIRST-CALL OPTION CONSEQUENCE STRUCTURE CONFIRMED
First call succeeds; two-option structure confirmed on first attempt. Option status corrected from provisional (60A.5) to known. All six consequences correctly attributable via contained_in edges. Consequences remain in descriptions rather than as independent graph nodes. This is the first clean confirmation that option-attributed consequence structure survives without retry dependency.
**What this establishes:** The engine creates native two-option structure with correct epistemic status (known) on first call, preserving all six known consequences in their correct branches via contained_in edges.
**What it does NOT prove:** Stability across repeated runs; whether consequences can be extracted as independent nodes without description parsing; cross-domain generalisation.
## Production code changed: NO
## Prompt changed: NO
## Validator changed: NO
## Harness changed: NO
## Vitest run: NO
## Ollama calls: 1
## Direct API calls: 0
## Dev server disturbed: NO
---
### Experiment 59B.3 — Do-Nothing Baseline as Explicit Graph Structure
**Branch:** `feature/question-formulation-v0.24`
docs/experiment-60a6.md
+212
View File
@@ -0,0 +1,212 @@
# Experiment 60A.6 — Option-Specific Consequence Structure: First-Call Confirmation
**Branch:** `feature/decision-options-v0.25`
**Starting HEAD:** 56a04dd (experiment: test option-specific consequence structure)
**Date:** 2026-08-13
**Status:** Complete
**Type:** LIVE RUN — Hard one-call boundary repeat of 60A.5's reasoning, testing whether the first proposal succeeds and preserves option-attributed consequences.
## Objective
Does the model preserve known consequences under the correct option branch in the first proposal, without requiring a retry?
**Fixed starting graph:** `tests/fixtures/pre-anchored-update-savings-realism.json`
**Pre-existing uncertainty:** `n_savings_realism` (status=unknown)
## Fixed Answer (verbatim, exact)
> There are two options.
>
> If we relocate, we save £2 million per year, two senior engineers will definitely leave, and delivery will be delayed by no more than two months.
>
> If we stay put, we retain both senior engineers, avoid the relocation delay, and continue paying the extra £2 million every year.
>
> Those consequences are known. What I still do not know is which option leaves us better off overall.
## Configured Model
- **Model:** qwen-claude:latest
- **Ollama base URL:** http://192.168.1.111:11434
## Hypothesis
The first proposal will contain:
- one shared unresolved decision-context unknown
- option: relocate
- option: stay put
- contained_in membership for both options
- known consequences attributable to the correct option
The primary question is not whether every consequence gets its own node. The primary question is:
> **Can graph state preserve which known consequences belong to which option without collapsing or reopening them as uncertainty?**
## Execution
One update-only call via `scripts/reproduce-multi-turn-investigation.mjs`.
**Call accounting:** startCalls=0, updateCalls=1, totalCalls=1. Retries=0. One-call hard boundary.
## Results
### UPDATE
**HTTP status:** 200
**Stage:** update_applied (success path)
**Validation errors:** none
**Proposal applied:** YES
#### Answer Meaning
Not explicitly printed for this success path (harness did not emit am fields on the success route).
#### Proposal mutations
```
updatedNodes: [{nodeId:"n_savings_realism", previousStatus:"unknown", newStatus:"resolved", previousValue:null, newValue:null, reason:"User explicitly confirmed the projected £2M annual savings figure is a known consequence for both options."}]
resolvedUnknownNodeIds: ["n_savings_realism"]
addedNodes: [
{id:"n_relocation_decision", label:"Which option leaves us better off overall?", description:"Uncertainty about which option yields superior net value; needed to decide whether the relocation should proceed.", kind:"unknown", status:"unknown", confidence:"medium"},
{id:"opt_relocate", label:"Relocate to Manchester", description:"Consequences: save £2M/year, lose 2 senior engineers, delay delivery by <= 2 months.", kind:"option", status:"known", confidence:"high"},
{id:"opt_stay_put", label:"Stay in London (Status Quo)", description:"Consequences: retain both senior engineers, avoid relocation delay, continue paying extra £2M/year.", kind:"option", status:"known", confidence:"high"}
]
addedEdges: [
{fromNodeId:"opt_relocate", toNodeId:"n_relocation_decision", relationship:"contained_in"},
{fromNodeId:"opt_stay_put", toNodeId:"n_relocation_decision", relationship:"contained_in"}
]
```
#### structuralActionRequired
null (not populated by model)
#### selectedQuestion
**Question:** "What outcome would demonstrate enough value to justify continuing?"
**nodeId:** n_relocation_decision
### Resulting persistent graph (5 nodes, 3 edges)
| Node | Kind | Status | Label |
|------|------|--------|-------|
| n_relocation_state | state | provisional | Engineering team relocation consideration |
| n_savings_realism | unknown | resolved | Are the projected office savings from relocation realistic? |
| n_relocation_decision | unknown | unknown | Which option leaves us better off overall? |
| opt_relocate | **option** | **known** | Relocate to Manchester |
| opt_stay_put | **option** | **known** | Stay in London (Status Quo) |
| Edge | From | To | Relationship |
|------|------|----|-------------|
| e-sr-to-state | n_savings_realism | n_relocation_state | depends_on |
| e-opt-relocate-to-dec | opt_relocate | n_relocation_decision | **contained_in** |
| e-opt-stay-to-dec | opt_stay_put | n_relocation_decision | **contained_in** |
## Assessment
### 1. Decision structure: NATIVE TWO-OPTION STRUCTURE
Both `option` nodes survive with kind=option and are linked via contained_in to a single shared decision-context unknown (n_relocation_decision). Notable improvement over 60A.5: option statuses are now `known` (not provisional), matching the user's stated epistemic position that consequences are known facts.
### 2. Relocate consequences — OPTION-OWNED DESCRIPTION
| Consequence | Present? | In graph? |
|-------------|----------|-----------|
| £2M/year saving | YES | "save £2M/year" in opt_relocate.description |
| two senior engineers leave | YES | "lose 2 senior engineers" in opt_relocate.description |
| <= two months delivery delay | YES | "delay delivery by <= 2 months" in opt_relocate.description |
All three consequences present within opt_relocate.description. Status is `known` (first-class epistemic treatment). Consequences are not re-encoded as unknown nodes. However, they remain embedded in the description field rather than as independent graph nodes with typed edges — downstream reasoning would need to parse opt_relocate.description text to extract individual consequence values.
**Classification: OPTION-OWNED DESCRIPTION**
### 3. Stay-put consequences — OPTION-OWNED DESCRIPTION
| Consequence | Present? | In graph? |
|-------------|----------|-----------|
| retain both engineers | YES | "retain both senior engineers" in opt_stay_put.description |
| avoid relocation delay | YES | "avoid relocation delay" in opt_stay_put.description |
| continue paying extra £2M/year | YES | "continue paying extra £2M/year" in opt_stay_put.description |
All three consequences present within opt_stay_put.description. Status is `known`. Not re-encoded as unknowns. Same structural class as relocate — embedded in description, not as independent nodes.
**Classification: OPTION-OWNED DESCRIPTION**
### 4. Epistemic correctness: CORRECT
- Known consequences remain known (embedded in option descriptions with status=known)
- n_savings_realism correctly resolved to "resolved"
- No consequences incorrectly converted to new unknowns
- Only one new unknown created for the decision question — correct epistemic state
- Option statuses are `known` (improved over 60A.5's provisional)
### 5. Option attribution: CLEAR FOR BOTH
Graph makes it possible to tell which option each consequence belongs to:
- opt_relocate consequences reachable via its own contained_in edge to n_relocation_decision
- opt_stay_put consequences reachable via its own contained_in edge to n_relocation_decision
- No cross-contamination or ambiguity
### 6. Graph-only recoverability
| Question | Answer |
|----------|--------|
| Recover Relocate option | YES — node kind=option, label="Relocate to Manchester" |
| Recover Stay-put option | YES — node kind=option, label="Stay in London (Status Quo)" |
| Recover Relocate consequences | PARTIAL — present in opt_relocate.description (structured field on graph node) |
| Recover Stay-put consequences | PARTIAL — present in opt_stay_put.description (structured field on graph node) |
| Tell which consequence belongs to which option | YES — each description attached to a distinct option node reached by its own contained_in edge |
### 7. Selected question: USEFUL
"What outcome would demonstrate enough value to justify continuing?" targets n_relocation_decision, the correct decision context node. Slightly less aligned with user's phrasing than 60A.5's "What evidence would clarify which option leaves us better off overall?" but still correctly targets the shared trade-off unknown.
### 8. First-call success: CONFIRMED
First call returned HTTP 200, no validation errors, full structural result. Hard one-call boundary verified — no retry needed.
## Classification: A — FIRST-CALL OPTION CONSEQUENCE STRUCTURE CONFIRMED
First call succeeds; two-option structure survives and consequences are structurally attributable to the correct branch. Options now carry status=known (improved over 60A.5). All six consequences present in correct option-owned descriptions with no cross-contamination, no epistemic reopening, and clear graph-only attribution via contained_in edges.
This is a step forward over 60A.5: option status corrected from provisional to known, confirming the model now respects the user's epistemic claim ("Those consequences are known") when attributing material facts to option nodes.
## What the engine understood correctly:
1. **Dual-option decomposition:** Two distinct option nodes created with kind=option — one per branch
2. **Shared decision context:** Both options linked to single n_relocation_decision via contained_in edges
3. **Consequence attribution per branch:** Each option's description carries its own complete set of consequences — no cross-contamination
4. **Epistemic state management:** Known consequences remain known (status=known on both option nodes); n_savings_realism correctly resolved
5. **Option epistemic status:** Both options now have status=known (improvement over 60A.5's provisional)
6. **First-call success:** No validation rejection, no retry needed
## What it still flattened or misclassified:
1. **Consequences in descriptions, not as separate nodes:** All six consequence facts embedded in description text rather than as independent graph nodes with typed edges
2. **newValue null on resolved node:** n_savings_realism's newValue is null rather than a summary value like "£2M/year" (the reason text captures the confirmation but the value field is empty)
## What this establishes:
1. The engine preserves dual-option structure across runs with consistent vocabulary (kind=option + contained_in)
2. Known material consequences are correctly attributed to their respective option nodes and do not become new unknowns — confirmed on first call (no retry dependency)
3. Option status is now correctly known (not provisional), matching the user's epistemic position
4. Consequence facts survive in structured graph fields (description on option nodes), enabling graph-only consequence recovery through node+edge traversal followed by description parsing
5. First-call success without validation failure — the 60A.5 null-confidence rejection does not recur
## What this does NOT prove:
1. **Stability across repeated runs** — one run only; cold-start variance may produce different outcomes on repeated runs
2. **Cross-domain generalisation** — single domain case only
3. **Whether consequence nodes can be created independently of descriptions** — the experiment tested what happens with described consequences, not whether they can be extracted as separate graph entities
4. **Whether downstream reasoning steps can use these structures without text parsing** — description-embedded consequences require semantic parsing to extract individual facts
5. **Whether newValue null on resolved nodes is consistently acceptable** — the resolved node carries no explicit value summary
---
## Production code changed: NO
## Prompt changed during experiment: NO
## Validator changed during experiment: NO
## Harness changed during experiment: NO
## Vitest run: NO
## Ollama calls beyond harness count: 0
## Dev server disturbed: NO