126 lines
6.4 KiB
Markdown
126 lines
6.4 KiB
Markdown
# Experiment 57J.60 — Selected-Question Contract Live Validation
|
|
|
|
**Branch:** `feature/selected-question-contract-v0.22`
|
|
**Starting HEAD:** `3baa77e` (docs: record selected-question contract alignment)
|
|
|
|
## Objective
|
|
|
|
Answer exactly:
|
|
|
|
> When the model adds a new savings-realism unknown, does v0.22 now also provide the mandatory `selectedQuestion` candidate so the proposal gets past the exact 57J.57 rejection boundary?
|
|
|
|
## Fixed scenario
|
|
|
|
```
|
|
We are considering relocating the engineering team to reduce operating costs.
|
|
```
|
|
|
|
## Fixed answer
|
|
|
|
```
|
|
I am unsure whether the projected office savings from the relocation are realistic.
|
|
```
|
|
|
|
## Pre-written expectation
|
|
|
|
> If the model creates a new unresolved savings-realism unknown, the proposal should now also contain a valid selectedQuestion candidate. The exact 57J.57 failure — new unresolved unknown plus selectedQuestion=null — should therefore not recur.
|
|
|
|
## Run
|
|
|
|
**Call accounting:** start: 1, update: 1, total: 2. No retries.
|
|
|
|
### START
|
|
|
|
- **HTTP:** 200
|
|
- **Stage:** `unknown` (start stage)
|
|
- **Nodes:** 7
|
|
- `ni0j5v4` — state/provisional — "A management group is evaluating whether relocating its engineering team will achieve the stated goal of lowering operating expenses."
|
|
- `njc0evc` — observation/supported — "The 'We' entity proposing or considering the relocation"
|
|
- `n7d1qbz` — observation/supported — "The workforce whose physical or operational base is proposed to be moved"
|
|
- `nchkvdm` — metric/known — "The financial expenses associated with maintaining the engineering team at its current and potential new locations"
|
|
- `nhstb08` — metric/known — "The proposed strategy of moving the team's location or operational hub"
|
|
- `n7m99es` — unknown/unknown — "Current monthly operating costs, target location expenses, and one-time relocation transition costs"
|
|
- `n992ndq` — unknown/unknown — "Effect of the move on team turnover, hiring difficulty, output quality, or delivery timelines"
|
|
- **Edges:** 4
|
|
- `njc0evc` → `ni0j5v4` (supports)
|
|
- `n7d1qbz` → `ni0j5v4` (supports)
|
|
- `n7m99es` → `ni0j5v4` (depends_on)
|
|
- `n992ndq` → `ni0j5v4` (depends_on)
|
|
- **Selected question:** `nodeId=n7m99es`, question: "What evidence would confirm or rule out current monthly operating costs, target location expenses, and one-time relocation transition costs?"
|
|
|
|
### UPDATE 1
|
|
|
|
- **HTTP:** 200
|
|
- **Stage:** `update_applied`
|
|
- **First error:** None (no rejection)
|
|
|
|
**Answer meaning fields:**
|
|
- **userSupportedMeaning:** "I am unsure whether the projected office savings from the relocation are realistic."
|
|
- **supportCategory:** `uncertain`
|
|
- **resolutionGuidance:** `may_resolve`
|
|
|
|
**Graph changes:**
|
|
- **updatedNodes:** 0 (direct) — note: `nibve28`'s dependsOn was modified but not in updatedNodes list; the new node appears via addedNodes only
|
|
- **resolvedUnknownNodeIds:** [] (empty)
|
|
- **addedNodes:** 1
|
|
- `n_proj_savings_realism` — unknown/unknown — "Uncertainty regarding the realism of projected office savings from relocation"
|
|
- Description: "Uncertainty remains about whether the projected office savings are realistic, matters because validating these projections is needed to decide if the engineering relocation will actually lower operating expenses as intended."
|
|
- affects: [`nibve28`] (summary state)
|
|
- **addedEdges:** 1
|
|
- `e-new-n_proj_savings_realism`: `n_proj_savings_realism` → `nibve28` (depends_on, confidence=medium)
|
|
- **Node count:** 6→10 (+4 nodes: 3 observations + 2 metrics merged into summary + 1 new unknown)
|
|
- **Edge count:** 4→7 (+3 edges)
|
|
|
|
**selectedQuestion (Update 1):**
|
|
- **nodeId:** `n_proj_savings_realism`
|
|
- **question:** "What would clarify realism of projected office savings from relocation in this situation?"
|
|
- **reason:** "Formulated as a neutral clarification question because no narrower investigation strategy clearly applied."
|
|
- **status:** the candidate node is `unknown/unknown` — unresolved
|
|
|
|
## Structural identity classification
|
|
|
|
**DEDICATED UNKNOWN**
|
|
|
|
The savings-realism uncertainty is represented as an independent unknown node (`n_proj_savings_realism`) with its own label, description, and graph edge. It was not absorbed into an existing cost node.
|
|
|
|
## selectedQuestion check
|
|
|
|
- **selectedQuestion present:** YES
|
|
- **nodeId:** `n_proj_savings_realism`
|
|
- **question:** "What would clarify realism of projected office savings from relocation in this situation?"
|
|
- **reason:** "Formulated as a neutral clarification question because no narrower investigation strategy clearly applied."
|
|
- **Candidate references an unresolved node:** YES — `n_proj_savings_realism` has status=`unknown` (unresolved)
|
|
|
|
## Old 57J.57 failure check
|
|
|
|
The old 57J.57 rejection was: `proposal_compatibility` rejection because `selectedQuestion` was null when new unresolved unknowns were added. **Not observed.** No `proposal_compatibility` error, no rejected proposal snapshot, HTTP 200 at `update_applied`.
|
|
|
|
## Classification: A — V0.22 FIX WORKS LIVE
|
|
|
|
All criteria met:
|
|
1. New unresolved unknown added (`n_proj_savings_realism`, status=unknown) ✓
|
|
2. selectedQuestion present with nodeId=`n_proj_savings_realism` ✓
|
|
3. Candidate structurally valid (references a node that exists in updatedSituationGraph, is of kind=unknown, status=unknown) ✓
|
|
4. Old selectedQuestion-null rejection absent ✓
|
|
|
|
## What this establishes
|
|
|
|
The v0.22 prompt change (rule #16: "must include" instead of "may identify") now drives the model to supply a valid `selectedQuestion` candidate even when it adds a new dedicated savings-realism unknown. The proposal passes through `update_applied` without the 57J.57 contract rejection. The exact v0.22 trigger — adding a new unresolved unknown alongside selectedQuestion — is confirmed live.
|
|
|
|
## What this does NOT prove
|
|
|
|
- Downstream investigation viability past Update 2+ (engine now has too_broad health with 4 active unknowns)
|
|
- Stability across repeated identical runs
|
|
- Cross-domain generalisation
|
|
- Whether the engine's deterministic priority selection actually exercises its scoring against this candidate or always accepts it
|
|
- Behaviour with other answer types beyond the single "uncertain" test case
|
|
- Whether the selectedQuestion contract holds when the model absorbs uncertainty into an existing node instead of adding a new one
|
|
|
|
## Configured Ollama
|
|
|
|
qwen-claude:latest at http://192.168.1.111:11434
|
|
|
|
## Production code changed
|
|
|
|
NO — prompt + tests only (v0.22 branch baseline). No harness code changed beyond temporary scenario/answer config.
|