confidence-engine/docs/experiment-57j60.md

Experiment 57J.60 — Selected-Question Contract Live Validation

Branch: feature/selected-question-contract-v0.22 Starting HEAD: 3baa77e (docs: record selected-question contract alignment)

Objective

Answer exactly:

When the model adds a new savings-realism unknown, does v0.22 now also provide the mandatory selectedQuestion candidate so the proposal gets past the exact 57J.57 rejection boundary?

Fixed scenario

We are considering relocating the engineering team to reduce operating costs.

Fixed answer

I am unsure whether the projected office savings from the relocation are realistic.

Pre-written expectation

If the model creates a new unresolved savings-realism unknown, the proposal should now also contain a valid selectedQuestion candidate. The exact 57J.57 failure — new unresolved unknown plus selectedQuestion=null — should therefore not recur.

Run

Call accounting: start: 1, update: 1, total: 2. No retries.

START

• HTTP: 200
• Stage: unknown (start stage)
• Nodes: 7
  • ni0j5v4 — state/provisional — "A management group is evaluating whether relocating its engineering team will achieve the stated goal of lowering operating expenses."
  • njc0evc — observation/supported — "The 'We' entity proposing or considering the relocation"
  • n7d1qbz — observation/supported — "The workforce whose physical or operational base is proposed to be moved"
  • nchkvdm — metric/known — "The financial expenses associated with maintaining the engineering team at its current and potential new locations"
  • nhstb08 — metric/known — "The proposed strategy of moving the team's location or operational hub"
  • n7m99es — unknown/unknown — "Current monthly operating costs, target location expenses, and one-time relocation transition costs"
  • n992ndq — unknown/unknown — "Effect of the move on team turnover, hiring difficulty, output quality, or delivery timelines"
• Edges: 4
  • njc0evc → ni0j5v4 (supports)
  • n7d1qbz → ni0j5v4 (supports)
  • n7m99es → ni0j5v4 (depends_on)
  • n992ndq → ni0j5v4 (depends_on)
• Selected question: nodeId=n7m99es, question: "What evidence would confirm or rule out current monthly operating costs, target location expenses, and one-time relocation transition costs?"

UPDATE 1

• HTTP: 200
• Stage: update_applied
• First error: None (no rejection)

Answer meaning fields:

• userSupportedMeaning: "I am unsure whether the projected office savings from the relocation are realistic."
• supportCategory: uncertain
• resolutionGuidance: may_resolve

Graph changes:

• updatedNodes: 0 (direct) — note: nibve28's dependsOn was modified but not in updatedNodes list; the new node appears via addedNodes only
• resolvedUnknownNodeIds: [] (empty)
• addedNodes: 1
  • n_proj_savings_realism — unknown/unknown — "Uncertainty regarding the realism of projected office savings from relocation"
    • Description: "Uncertainty remains about whether the projected office savings are realistic, matters because validating these projections is needed to decide if the engineering relocation will actually lower operating expenses as intended."
    • affects: [nibve28] (summary state)
• addedEdges: 1
  • e-new-n_proj_savings_realism: n_proj_savings_realism → nibve28 (depends_on, confidence=medium)
• Node count: 6→10 (+4 nodes: 3 observations + 2 metrics merged into summary + 1 new unknown)
• Edge count: 4→7 (+3 edges)

selectedQuestion (Update 1):

• nodeId: n_proj_savings_realism
• question: "What would clarify realism of projected office savings from relocation in this situation?"
• reason: "Formulated as a neutral clarification question because no narrower investigation strategy clearly applied."
• status: the candidate node is unknown/unknown — unresolved

Structural identity classification

DEDICATED UNKNOWN

The savings-realism uncertainty is represented as an independent unknown node (n_proj_savings_realism) with its own label, description, and graph edge. It was not absorbed into an existing cost node.

selectedQuestion check

• selectedQuestion present: YES
• nodeId: n_proj_savings_realism
• question: "What would clarify realism of projected office savings from relocation in this situation?"
• reason: "Formulated as a neutral clarification question because no narrower investigation strategy clearly applied."
• Candidate references an unresolved node: YES — n_proj_savings_realism has status=unknown (unresolved)

Old 57J.57 failure check

The old 57J.57 rejection was: proposal_compatibility rejection because selectedQuestion was null when new unresolved unknowns were added. Not observed. No proposal_compatibility error, no rejected proposal snapshot, HTTP 200 at update_applied.

Classification: A — V0.22 FIX WORKS LIVE

All criteria met:

1. New unresolved unknown added (n_proj_savings_realism, status=unknown) ✓
2. selectedQuestion present with nodeId=n_proj_savings_realism ✓
3. Candidate structurally valid (references a node that exists in updatedSituationGraph, is of kind=unknown, status=unknown) ✓
4. Old selectedQuestion-null rejection absent ✓

What this establishes

The v0.22 prompt change (rule #16: "must include" instead of "may identify") now drives the model to supply a valid selectedQuestion candidate even when it adds a new dedicated savings-realism unknown. The proposal passes through update_applied without the 57J.57 contract rejection. The exact v0.22 trigger — adding a new unresolved unknown alongside selectedQuestion — is confirmed live.

What this does NOT prove

• Downstream investigation viability past Update 2+ (engine now has too_broad health with 4 active unknowns)
• Stability across repeated identical runs
• Cross-domain generalisation
• Whether the engine's deterministic priority selection actually exercises its scoring against this candidate or always accepts it
• Behaviour with other answer types beyond the single "uncertain" test case
• Whether the selectedQuestion contract holds when the model absorbs uncertainty into an existing node instead of adding a new one

Configured Ollama

qwen-claude:latest at http://192.168.1.111:11434

Production code changed

NO — prompt + tests only (v0.22 branch baseline). No harness code changed beyond temporary scenario/answer config.