From 2927509ca5903833738d1d109b3ce9e06969fe76 Mon Sep 17 00:00:00 2001 From: robbond Date: Tue, 11 Aug 2026 19:15:51 +0100 Subject: [PATCH] experiment: validate selected-question contract live --- docs/current-handoff.md | 6 ++ docs/experiment-57j60.md | 125 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 131 insertions(+) create mode 100644 docs/experiment-57j60.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 0bfd424..dea4bfb 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -197,6 +197,12 @@ Two-turn run (fixed scenario: "We are considering relocating the engineering tea --- +### Experiment 57J.60 — Selected-Question Contract Live Validation + +**Objective:** Validate that v0.22's mandatory selectedQuestion rule (rule #16) works live when the model adds a new dedicated savings-realism unknown. **Classification: A — V0.22 FIX WORKS LIVE.** One start + one update through the production `updateCase()` path. Update 1 added one dedicated unknown node (`n_proj_savings_realism`, kind=unknown, status=unknown) and included a valid selectedQuestion candidate (nodeId=`n_proj_savings_realism`) referencing that same unresolved node. The old 57J.57 contract rejection — new unresolved unknown + selectedQuestion=null — did not recur. HTTP 200 at `update_applied`, no `proposal_compatibility` error, no rejectedProposalSnapshot. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Full record in `docs/experiment-57j60.md`. + +--- + ### Experiment 57E — Irrelevant Decomposition Question Boundary **Objective:** Identify the exact graph node that triggered the decomposition producing "How the two observations were measured", and determine whether the parent was genuinely about comparison/measurement/timing before decomposition. diff --git a/docs/experiment-57j60.md b/docs/experiment-57j60.md new file mode 100644 index 0000000..e019ea0 --- /dev/null +++ b/docs/experiment-57j60.md @@ -0,0 +1,125 @@ +# Experiment 57J.60 — Selected-Question Contract Live Validation + +**Branch:** `feature/selected-question-contract-v0.22` +**Starting HEAD:** `3baa77e` (docs: record selected-question contract alignment) + +## Objective + +Answer exactly: + +> When the model adds a new savings-realism unknown, does v0.22 now also provide the mandatory `selectedQuestion` candidate so the proposal gets past the exact 57J.57 rejection boundary? + +## Fixed scenario + +``` +We are considering relocating the engineering team to reduce operating costs. +``` + +## Fixed answer + +``` +I am unsure whether the projected office savings from the relocation are realistic. +``` + +## Pre-written expectation + +> If the model creates a new unresolved savings-realism unknown, the proposal should now also contain a valid selectedQuestion candidate. The exact 57J.57 failure — new unresolved unknown plus selectedQuestion=null — should therefore not recur. + +## Run + +**Call accounting:** start: 1, update: 1, total: 2. No retries. + +### START + +- **HTTP:** 200 +- **Stage:** `unknown` (start stage) +- **Nodes:** 7 + - `ni0j5v4` — state/provisional — "A management group is evaluating whether relocating its engineering team will achieve the stated goal of lowering operating expenses." + - `njc0evc` — observation/supported — "The 'We' entity proposing or considering the relocation" + - `n7d1qbz` — observation/supported — "The workforce whose physical or operational base is proposed to be moved" + - `nchkvdm` — metric/known — "The financial expenses associated with maintaining the engineering team at its current and potential new locations" + - `nhstb08` — metric/known — "The proposed strategy of moving the team's location or operational hub" + - `n7m99es` — unknown/unknown — "Current monthly operating costs, target location expenses, and one-time relocation transition costs" + - `n992ndq` — unknown/unknown — "Effect of the move on team turnover, hiring difficulty, output quality, or delivery timelines" +- **Edges:** 4 + - `njc0evc` → `ni0j5v4` (supports) + - `n7d1qbz` → `ni0j5v4` (supports) + - `n7m99es` → `ni0j5v4` (depends_on) + - `n992ndq` → `ni0j5v4` (depends_on) +- **Selected question:** `nodeId=n7m99es`, question: "What evidence would confirm or rule out current monthly operating costs, target location expenses, and one-time relocation transition costs?" + +### UPDATE 1 + +- **HTTP:** 200 +- **Stage:** `update_applied` +- **First error:** None (no rejection) + +**Answer meaning fields:** +- **userSupportedMeaning:** "I am unsure whether the projected office savings from the relocation are realistic." +- **supportCategory:** `uncertain` +- **resolutionGuidance:** `may_resolve` + +**Graph changes:** +- **updatedNodes:** 0 (direct) — note: `nibve28`'s dependsOn was modified but not in updatedNodes list; the new node appears via addedNodes only +- **resolvedUnknownNodeIds:** [] (empty) +- **addedNodes:** 1 + - `n_proj_savings_realism` — unknown/unknown — "Uncertainty regarding the realism of projected office savings from relocation" + - Description: "Uncertainty remains about whether the projected office savings are realistic, matters because validating these projections is needed to decide if the engineering relocation will actually lower operating expenses as intended." + - affects: [`nibve28`] (summary state) +- **addedEdges:** 1 + - `e-new-n_proj_savings_realism`: `n_proj_savings_realism` → `nibve28` (depends_on, confidence=medium) +- **Node count:** 6→10 (+4 nodes: 3 observations + 2 metrics merged into summary + 1 new unknown) +- **Edge count:** 4→7 (+3 edges) + +**selectedQuestion (Update 1):** +- **nodeId:** `n_proj_savings_realism` +- **question:** "What would clarify realism of projected office savings from relocation in this situation?" +- **reason:** "Formulated as a neutral clarification question because no narrower investigation strategy clearly applied." +- **status:** the candidate node is `unknown/unknown` — unresolved + +## Structural identity classification + +**DEDICATED UNKNOWN** + +The savings-realism uncertainty is represented as an independent unknown node (`n_proj_savings_realism`) with its own label, description, and graph edge. It was not absorbed into an existing cost node. + +## selectedQuestion check + +- **selectedQuestion present:** YES +- **nodeId:** `n_proj_savings_realism` +- **question:** "What would clarify realism of projected office savings from relocation in this situation?" +- **reason:** "Formulated as a neutral clarification question because no narrower investigation strategy clearly applied." +- **Candidate references an unresolved node:** YES — `n_proj_savings_realism` has status=`unknown` (unresolved) + +## Old 57J.57 failure check + +The old 57J.57 rejection was: `proposal_compatibility` rejection because `selectedQuestion` was null when new unresolved unknowns were added. **Not observed.** No `proposal_compatibility` error, no rejected proposal snapshot, HTTP 200 at `update_applied`. + +## Classification: A — V0.22 FIX WORKS LIVE + +All criteria met: +1. New unresolved unknown added (`n_proj_savings_realism`, status=unknown) ✓ +2. selectedQuestion present with nodeId=`n_proj_savings_realism` ✓ +3. Candidate structurally valid (references a node that exists in updatedSituationGraph, is of kind=unknown, status=unknown) ✓ +4. Old selectedQuestion-null rejection absent ✓ + +## What this establishes + +The v0.22 prompt change (rule #16: "must include" instead of "may identify") now drives the model to supply a valid `selectedQuestion` candidate even when it adds a new dedicated savings-realism unknown. The proposal passes through `update_applied` without the 57J.57 contract rejection. The exact v0.22 trigger — adding a new unresolved unknown alongside selectedQuestion — is confirmed live. + +## What this does NOT prove + +- Downstream investigation viability past Update 2+ (engine now has too_broad health with 4 active unknowns) +- Stability across repeated identical runs +- Cross-domain generalisation +- Whether the engine's deterministic priority selection actually exercises its scoring against this candidate or always accepts it +- Behaviour with other answer types beyond the single "uncertain" test case +- Whether the selectedQuestion contract holds when the model absorbs uncertainty into an existing node instead of adding a new one + +## Configured Ollama + +qwen-claude:latest at http://192.168.1.111:11434 + +## Production code changed + +NO — prompt + tests only (v0.22 branch baseline). No harness code changed beyond temporary scenario/answer config.