robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: test option-specific consequence structure

Browse Source
This commit is contained in:
robbond
2026-08-13 05:53:20 +01:00
parent 3db6f40fdc
commit 56a04ddd0e
2 changed files with 412 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+211
View File
@@ -345,6 +345,217 @@ The engine extracted all three semantic elements in userSupportedMeaning and cre
---
### Experiment 60A.5 — Option-Specific Consequence Structure Confirmation
**Branch:** `feature/decision-options-v0.25`
**Date:** 2026-08-13
**Status:** Complete
**Type:** LIVE RUN — Bounded single-call experiment to verify whether known consequences for two alternatives become independently recoverable graph structure attached to the correct option.
**Following:** 60A.4 which confirmed native two-option structure with contained_in edges.
## Objective
When the user explicitly separates known consequences for two alternatives ("If we relocate... If we stay put..."), does the live engine create consequence structure that remains attributable to the correct option — without converting known material into new unresolved unknowns?
## Fixed Starting Graph
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
Pre-existing uncertainty: `n_savings_realism` (status=unknown)
## Fixed Answer (verbatim, exact)
> There are two options.
>
> If we relocate, we save £2 million per year, two senior engineers will definitely leave, and delivery will be delayed by no more than two months.
>
> If we stay put, we retain both senior engineers, avoid the relocation delay, and continue paying the extra £2 million every year.
>
> Those consequences are known. What I still do not know is which option leaves us better off overall.
## Configured Model
- **Model:** qwen-claude:latest
- **Ollama base URL:** http://192.168.1.111:11434
## Hypothesis
A strong result should preserve: one shared decision-context unknown; option: relocate; option: stay put; and create independently recoverable consequence/evidence structure associated with the correct option.
Known consequences ≠ unresolved decision. The engine should not turn them into new unknowns merely because they matter to the decision.
## Execution
One update-only call via `scripts/reproduce-multi-turn-investigation.mjs`. (First attempt returned HTTP 422 at proposal_validation due to null confidence fields on added nodes; second attempt — one additional call — returned HTTP 200 with full structural result.)
**Call accounting:** startCalls=0, updateCalls=2 (1 rejection + 1 success), totalCalls=2. Only the successful run carries assessable inference results.
### CALL ACCOUNTING (successful run)
| Field | Value |
|-------|-------|
| startCalls | 0 |
| updateCalls | 1 |
| totalCalls | 1 |
| Retries | 1 (initial rejection at proposal_validation) |
## Results — Successful Run
### UPDATE
**HTTP status:** 200
**Stage:** update_applied (implied by success path; no validation errors)
#### updatedNodes
```
[{nodeId: "n_savings_realism", previousStatus: "unknown", newStatus: "resolved", newValue: "£2m/year", reason: "User explicitly states the £2m annual saving consequence is known."}]
```
**resolvedUnknownNodeIds:** `["n_savings_realism"]`
### addedNodes (3 nodes)
1. **opt_relocate** — kind=option, status=provisional, confidence=high
- Label: "Relocate to Manchester"
- Description: "Move the engineering team to Manchester. Consequences: saves £2m/year, loses two senior engineers, delays delivery by up to two months."
- childIds: ["n_overall_tradeoff"]
2. **opt_stay** — kind=option, status=provisional, confidence=high
- Label: "Stay in London"
- Description: "Remain in the current office. Consequences: retain both senior engineers, avoid relocation delay, continue paying the extra £2m/year."
- childIds: ["n_overall_tradeoff"]
3. **n_overall_tradeoff** — kind=unknown, status=unknown, confidence=high
- Label: "Which option leaves us better off overall?"
- description: "Uncertainty regarding the net trade-off between relocation savings and retention/delay costs compared to staying put, needed to decide on the best path forward."
- dependsOn: ["opt_relocate", "opt_stay"]
### addedEdges (3 edges)
1. **opt_relocate → n_overall_tradeoff** — contained_in (confidence=high)
2. **opt_stay → n_overall_tradeoff** — contained_in (confidence=high)
3. **n_overall_tradeoff → n_savings_realism** — depends_on (confidence=medium)
### selectedQuestion
**Question:** "What evidence would clarify which option leaves us better off overall?"
**nodeId:** `n_overall_tradeoff`
### Resulting persistent graph (5 nodes, 4 edges)
| Node | Kind | Status | Label |
|------|------|--------|-------|
| n_relocation_state | state | provisional | Engineering team relocation consideration |
| n_savings_realism | unknown | resolved | Are the projected office savings from relocation realistic? |
| opt_relocate | **option** | provisional | Relocate to Manchester |
| opt_stay | **option** | provisional | Stay in London |
| n_overall_tradeoff | unknown | unknown | Which option leaves us better off overall? |
| Edge | From | To | Relationship |
|------|------|----|-------------|
| e-sr-to-state | n_savings_realism | n_relocation_state | depends_on |
| opt-rel-to-trad | opt_relocate | n_overall_tradeoff | **contained_in** |
| opt-stay-to-trad | opt_stay | n_overall_tradeoff | **contained_in** |
| trad-to-savings | n_overall_tradeoff | n_savings_realism | depends_on |
## Assessment
### 1. Decision structure: NATIVE TWO-OPTION STRUCTURE PRESERVED
Both `option` nodes survive with kind=option and are linked via contained_in to the same unresolved decision context (n_overall_tradeoff). This matches the 60A.4 pattern. Two minor differences from 60A.4:
- Option statuses are provisional instead of known (both have confidence=high, so ambiguity is low)
- Node IDs use lowercase abbreviations (opt_relocate/opt_stay vs n_option_relocate/n_option_stay)
### 2. Relocate consequences — ALL INDEPENDENTLY STRUCTURED (in descriptions)
| Consequence | Present? | Classification |
|-------------|----------|----------------|
| £2m/year saving | YES | "saves £2m/year" — exact figure and unit preserved |
| two senior engineers leave | YES | "loses two senior engineers" — exact headcount preserved |
| <= two months delivery delay | YES | "delays delivery by up to two months" — bounded phrasing preserved |
All three consequences present within opt_relocate's description. The description also serves as a structured field on an option node reachable via contained_in edge.
### 3. Stay-put consequences — ALL INDEPENDENTLY STRUCTURED (in descriptions)
| Consequence | Present? | Classification |
|-------------|----------|----------------|
| retain both engineers | YES | "retain both senior engineers" |
| avoid relocation delay | YES | "avoid relocation delay" |
| continue paying extra £2m/year | YES | "continue paying the extra £2m/year" |
All three consequences present within opt_stay's description.
### 4. Epistemic correctness — CORRECT
- Known consequences remain known (embedded in option descriptions, not unresolved)
- n_savings_realism correctly resolved to "resolved" with newValue="£2m/year"
- No consequences incorrectly converted to new unknowns
- Only one new unknown created (n_overall_tradeoff for the decision question) — this is the correct epistemic state
### 5. Option attribution — CLEAR FOR BOTH OPTIONS
Graph makes it possible to tell which option each consequence belongs to:
- opt_relocate consequences are embedded in opt_relocate.description, reachable via contained_in edge from opt_relocate to n_overall_tradeoff
- opt_stay consequences are embedded in opt_stay.description, reachable via contained_in edge from opt_stay to n_overall_tradeoff
- No cross-contamination or ambiguity
### 6. Relationship direction — SEMANTICALLY CLEAR
| From | To | Relationship | Assessment |
|------|----|-------------|------------|
| opt_relocate | n_overall_tradeoff | contained_in | Clear — relocation is a candidate for the decision |
| opt_stay | n_overall_tradeoff | contained_in | Clear — staying put is a candidate for the decision |
| n_overall_tradeoff | n_savings_realism | depends_on | Workable but slightly odd — the tradeoff assessment depending on the savings figure makes semantic sense (you need the savings value to assess whether relocation is better), though the direction could be read as the unknown "needs" the resolved node, which is epistemically inverted. |
### 7. Graph-only recoverability
| Question | Answer |
|----------|--------|
| Recover Relocate option | YES — node kind=option, label="Relocate to Manchester" |
| Recover Stay-put option | YES — node kind=option, label="Stay in London" |
| Recover Relocate consequences | PARTIAL — present in opt_relocate.description (structured field on graph node) |
| Recover Stay-put consequences | PARTIAL — present in opt_stay.description (structured field on graph node) |
| Tell which consequence belongs to which option | YES — each description is attached to a distinct option node reached by its own contained_in edge |
### 8. Selected question — GOOD
"What evidence would clarify which option leaves us better off overall?" targets n_overall_tradeoff, the correct decision context node. The question aligns with the user's stated unresolved issue. No penalty for asking about a genuinely decision-relevant comparison criterion.
## Classification: B — CONSEQUENCE STRUCTURE PARTIAL
Both option branches survive as structurally distinct nodes (kind=option) with correct containment relationships to a shared decision context. All known consequences for both options are present and correctly attributable. However, consequences remain embedded in option descriptions rather than as independent graph nodes with typed edges — a downstream reasoning step would need to parse opt_relocate.description vs opt_stay.description text to extract specific consequence values.
This is an improvement over 59B.4 (where do-nothing had no structural presence) but does not reach A-level because consequences are not first-class independently recoverable nodes.
## What the engine understood correctly:
1. **Dual-option decomposition:** Two distinct option nodes created with kind=option — one per branch
2. **Shared decision context:** Both options linked to single n_overall_tradeoff via contained_in edges
3. **Consequence attribution per branch:** Each option's description carries its own complete set of consequences — no cross-contamination
4. **Epistemic state management:** Known consequences remain known; n_savings_realism correctly resolved
5. **Decision-question alignment:** Selected question mirrors the user's stated unresolved issue
## What it still flattened or misclassified:
1. **Consequences in descriptions, not as separate nodes:** All six consequence facts are embedded in opt_relocate.description and opt_stay.description rather than as independent graph nodes. A downstream step cannot query "what consequences does opt_relocate have?" via graph traversal alone — it must parse the description text.
2. **Option status is provisional, not known:** Both option nodes have status=provisional rather than status=known (the user stated consequences are KNOWN). This is a minor epistemic mismatch — options themselves are known possibilities, not uncertain states.
## What this establishes:
1. The engine preserves dual-option structure across runs with consistent vocabulary (kind=option + contained_in)
2. Known material consequences are correctly attributed to their respective option nodes and do not become new unknowns
3. Consequence facts survive in structured graph fields (description on option nodes), enabling graph-only consequence recovery through node+edge traversal followed by description parsing
## What this does NOT prove:
1. **Stability across repeated runs** — the first run failed at proposal_validation; the successful inference was on a second attempt
2. **Cross-domain generalisation** — single domain case only
3. **Whether consequence nodes can be created independently of descriptions** — the experiment tested what happens with described consequences, not whether they can be extracted as separate graph entities
4. **Whether downstream reasoning steps can use these structures without text parsing** — description-embedded consequences require semantic parsing to extract individual facts
---
### Experiment 59B.3 — Do-Nothing Baseline as Explicit Graph Structure
**Branch:** `feature/question-formulation-v0.24`