experiment: validate selected-question contract live
This commit is contained in:
@@ -197,6 +197,12 @@ Two-turn run (fixed scenario: "We are considering relocating the engineering tea
|
||||
|
||||
---
|
||||
|
||||
### Experiment 57J.60 — Selected-Question Contract Live Validation
|
||||
|
||||
**Objective:** Validate that v0.22's mandatory selectedQuestion rule (rule #16) works live when the model adds a new dedicated savings-realism unknown. **Classification: A — V0.22 FIX WORKS LIVE.** One start + one update through the production `updateCase()` path. Update 1 added one dedicated unknown node (`n_proj_savings_realism`, kind=unknown, status=unknown) and included a valid selectedQuestion candidate (nodeId=`n_proj_savings_realism`) referencing that same unresolved node. The old 57J.57 contract rejection — new unresolved unknown + selectedQuestion=null — did not recur. HTTP 200 at `update_applied`, no `proposal_compatibility` error, no rejectedProposalSnapshot. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Full record in `docs/experiment-57j60.md`.
|
||||
|
||||
---
|
||||
|
||||
### Experiment 57E — Irrelevant Decomposition Question Boundary
|
||||
|
||||
**Objective:** Identify the exact graph node that triggered the decomposition producing "How the two observations were measured", and determine whether the parent was genuinely about comparison/measurement/timing before decomposition.
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
# Experiment 57J.60 — Selected-Question Contract Live Validation
|
||||
|
||||
**Branch:** `feature/selected-question-contract-v0.22`
|
||||
**Starting HEAD:** `3baa77e` (docs: record selected-question contract alignment)
|
||||
|
||||
## Objective
|
||||
|
||||
Answer exactly:
|
||||
|
||||
> When the model adds a new savings-realism unknown, does v0.22 now also provide the mandatory `selectedQuestion` candidate so the proposal gets past the exact 57J.57 rejection boundary?
|
||||
|
||||
## Fixed scenario
|
||||
|
||||
```
|
||||
We are considering relocating the engineering team to reduce operating costs.
|
||||
```
|
||||
|
||||
## Fixed answer
|
||||
|
||||
```
|
||||
I am unsure whether the projected office savings from the relocation are realistic.
|
||||
```
|
||||
|
||||
## Pre-written expectation
|
||||
|
||||
> If the model creates a new unresolved savings-realism unknown, the proposal should now also contain a valid selectedQuestion candidate. The exact 57J.57 failure — new unresolved unknown plus selectedQuestion=null — should therefore not recur.
|
||||
|
||||
## Run
|
||||
|
||||
**Call accounting:** start: 1, update: 1, total: 2. No retries.
|
||||
|
||||
### START
|
||||
|
||||
- **HTTP:** 200
|
||||
- **Stage:** `unknown` (start stage)
|
||||
- **Nodes:** 7
|
||||
- `ni0j5v4` — state/provisional — "A management group is evaluating whether relocating its engineering team will achieve the stated goal of lowering operating expenses."
|
||||
- `njc0evc` — observation/supported — "The 'We' entity proposing or considering the relocation"
|
||||
- `n7d1qbz` — observation/supported — "The workforce whose physical or operational base is proposed to be moved"
|
||||
- `nchkvdm` — metric/known — "The financial expenses associated with maintaining the engineering team at its current and potential new locations"
|
||||
- `nhstb08` — metric/known — "The proposed strategy of moving the team's location or operational hub"
|
||||
- `n7m99es` — unknown/unknown — "Current monthly operating costs, target location expenses, and one-time relocation transition costs"
|
||||
- `n992ndq` — unknown/unknown — "Effect of the move on team turnover, hiring difficulty, output quality, or delivery timelines"
|
||||
- **Edges:** 4
|
||||
- `njc0evc` → `ni0j5v4` (supports)
|
||||
- `n7d1qbz` → `ni0j5v4` (supports)
|
||||
- `n7m99es` → `ni0j5v4` (depends_on)
|
||||
- `n992ndq` → `ni0j5v4` (depends_on)
|
||||
- **Selected question:** `nodeId=n7m99es`, question: "What evidence would confirm or rule out current monthly operating costs, target location expenses, and one-time relocation transition costs?"
|
||||
|
||||
### UPDATE 1
|
||||
|
||||
- **HTTP:** 200
|
||||
- **Stage:** `update_applied`
|
||||
- **First error:** None (no rejection)
|
||||
|
||||
**Answer meaning fields:**
|
||||
- **userSupportedMeaning:** "I am unsure whether the projected office savings from the relocation are realistic."
|
||||
- **supportCategory:** `uncertain`
|
||||
- **resolutionGuidance:** `may_resolve`
|
||||
|
||||
**Graph changes:**
|
||||
- **updatedNodes:** 0 (direct) — note: `nibve28`'s dependsOn was modified but not in updatedNodes list; the new node appears via addedNodes only
|
||||
- **resolvedUnknownNodeIds:** [] (empty)
|
||||
- **addedNodes:** 1
|
||||
- `n_proj_savings_realism` — unknown/unknown — "Uncertainty regarding the realism of projected office savings from relocation"
|
||||
- Description: "Uncertainty remains about whether the projected office savings are realistic, matters because validating these projections is needed to decide if the engineering relocation will actually lower operating expenses as intended."
|
||||
- affects: [`nibve28`] (summary state)
|
||||
- **addedEdges:** 1
|
||||
- `e-new-n_proj_savings_realism`: `n_proj_savings_realism` → `nibve28` (depends_on, confidence=medium)
|
||||
- **Node count:** 6→10 (+4 nodes: 3 observations + 2 metrics merged into summary + 1 new unknown)
|
||||
- **Edge count:** 4→7 (+3 edges)
|
||||
|
||||
**selectedQuestion (Update 1):**
|
||||
- **nodeId:** `n_proj_savings_realism`
|
||||
- **question:** "What would clarify realism of projected office savings from relocation in this situation?"
|
||||
- **reason:** "Formulated as a neutral clarification question because no narrower investigation strategy clearly applied."
|
||||
- **status:** the candidate node is `unknown/unknown` — unresolved
|
||||
|
||||
## Structural identity classification
|
||||
|
||||
**DEDICATED UNKNOWN**
|
||||
|
||||
The savings-realism uncertainty is represented as an independent unknown node (`n_proj_savings_realism`) with its own label, description, and graph edge. It was not absorbed into an existing cost node.
|
||||
|
||||
## selectedQuestion check
|
||||
|
||||
- **selectedQuestion present:** YES
|
||||
- **nodeId:** `n_proj_savings_realism`
|
||||
- **question:** "What would clarify realism of projected office savings from relocation in this situation?"
|
||||
- **reason:** "Formulated as a neutral clarification question because no narrower investigation strategy clearly applied."
|
||||
- **Candidate references an unresolved node:** YES — `n_proj_savings_realism` has status=`unknown` (unresolved)
|
||||
|
||||
## Old 57J.57 failure check
|
||||
|
||||
The old 57J.57 rejection was: `proposal_compatibility` rejection because `selectedQuestion` was null when new unresolved unknowns were added. **Not observed.** No `proposal_compatibility` error, no rejected proposal snapshot, HTTP 200 at `update_applied`.
|
||||
|
||||
## Classification: A — V0.22 FIX WORKS LIVE
|
||||
|
||||
All criteria met:
|
||||
1. New unresolved unknown added (`n_proj_savings_realism`, status=unknown) ✓
|
||||
2. selectedQuestion present with nodeId=`n_proj_savings_realism` ✓
|
||||
3. Candidate structurally valid (references a node that exists in updatedSituationGraph, is of kind=unknown, status=unknown) ✓
|
||||
4. Old selectedQuestion-null rejection absent ✓
|
||||
|
||||
## What this establishes
|
||||
|
||||
The v0.22 prompt change (rule #16: "must include" instead of "may identify") now drives the model to supply a valid `selectedQuestion` candidate even when it adds a new dedicated savings-realism unknown. The proposal passes through `update_applied` without the 57J.57 contract rejection. The exact v0.22 trigger — adding a new unresolved unknown alongside selectedQuestion — is confirmed live.
|
||||
|
||||
## What this does NOT prove
|
||||
|
||||
- Downstream investigation viability past Update 2+ (engine now has too_broad health with 4 active unknowns)
|
||||
- Stability across repeated identical runs
|
||||
- Cross-domain generalisation
|
||||
- Whether the engine's deterministic priority selection actually exercises its scoring against this candidate or always accepts it
|
||||
- Behaviour with other answer types beyond the single "uncertain" test case
|
||||
- Whether the selectedQuestion contract holds when the model absorbs uncertainty into an existing node instead of adding a new one
|
||||
|
||||
## Configured Ollama
|
||||
|
||||
qwen-claude:latest at http://192.168.1.111:11434
|
||||
|
||||
## Production code changed
|
||||
|
||||
NO — prompt + tests only (v0.22 branch baseline). No harness code changed beyond temporary scenario/answer config.
|
||||
Reference in New Issue
Block a user