6.5 KiB
Experiment 57J.26 — Post-Admission Investigation Progress (Live)
Objective: Answer whether the engine makes genuine investigative progress after admitting two user-supported unknowns, by continuing past the first meaningful v0.15 question with a concrete savings-realism answer.
Pre-written expectation recorded: YES
Answer 2 provides concrete support for the savings-realism uncertainty. The investigation should therefore make progress rather than repeat the same question. The next move should concern another genuine unresolved aspect of the relocation decision. Retention impact is an obvious remaining issue, but the experiment does not require that exact question if another grounded unresolved issue is legitimately selected.
A return to unsupported comparison/measurement/timing framing, repetition of the resolved savings-realism question, or a new validation failure counts as the first meaningful failure.
Configured apparatus
- Ollama base URL:
http://192.168.1.111:11434 - Model:
qwen-claude:latest - Harness:
scripts/reproduce-multi-turn-investigation.mjs(canonical) - Branch:
feature/user-supported-unknown-admission-v0.15 - HEAD:
fbbd271— experiment: validate user-supported unknown admission live - Production API path:
/api/cases/start→/api/cases/update
Fixed scenario and answers
Scenario: "We are considering relocating the engineering team to reduce operating costs."
Answer 1: "Before deciding, I need evidence that the projected office savings are realistic and evidence that the move will not materially increase loss of key engineers."
Answer 2: "The projected savings are based on the current London lease, business rates, service charges, utilities and facilities costs that would no longer be incurred at the same level after the move. The estimate is approximately £2M per year."
Live-call count
start: 1
update 1: 1
total: 2
(Run 2 - exact 57J.25 scenario): start: 1, update 1: 1)
total: 2
Run 1 (57J.26 scenario + answer pair)
START
- HTTP status: 200
- Stage: unknown (initial state)
- Node count: 7
- Edge count: 5
- Selected question: "What was the comparable state before proportion of fixed versus variable operating costs tied to the team's physical location?"
- Active unknown:
ncouucp— "Proportion of fixed versus variable operating costs tied to the team's physical location"
The start created two unknowns: (1) geographic locations cost structures (n2sve83) and (2) proportion of fixed vs variable costs (ncouucp). Neither is about savings realism or retention — different node set from 57J.25.
UPDATE 1
- HTTP status: 200
- Stage:
proposal_compatibility - Success: false
- Error/validation summary:
"New unknown must be explicitly related to an answer-derived node: \"u-engineer-retention\"" - Node count: 7 (before rejection — one new node
u-engineer-retentionwas created but the update rolled back) - Edge count: 4
The old provenance-link gate has returned. A new unknown introduced by Answer 1 (u-engineer-retention, capturing retention impact from "move will not materially increase loss of key engineers") was rejected because it lacks an answer-derived provenance edge. This is a 57J.11 regression.
Run 2 (exact 57J.25 scenario + answer pair for comparison)
START
- HTTP status: 200
- Node count: 9
- Edge count: 6
- Unknowns created: 3 (primary goal, team performance/deadlines, budget/costs)
- Selected question: "What would clarify primary goal of the relocation..."
Different start graph from both 57J.25 and Run 1 — confirming significant run-to-run variance in initial graph construction for different scenarios.
UPDATE 1
- HTTP status: 200
- Stage:
proposal_compatibility - Success: false
- Error/validation summary:
"Proposal cannot resolve beyond an unclassified answer by introducing unsupported stronger meaning than answerMeaning.userSupportedMeaning establishes."
Different rejection — a semantic compatibility error about unclassified answer meaning, not the provenance-link gate. Still blocks Update 2.
Comparison with 57J.25
| Dimension | 57J.25 | 57J.26 Run 1 | 57J.26 Run 2 |
|---|---|---|---|
| Start nodes | 6 | 7 | 9 |
| Update stage | update_applied (HTTP 200) |
proposal_compatibility (rejected) |
proposal_compatibility (rejected) |
| Savings target admitted | YES | NO (rejected) | NOT tested |
| Retention target admitted | YES | NO (rejected) | NOT tested |
| First rejection error | None | Provenance-link gate | Semantic compatibility |
Classification: D — NEW VALIDATION / REASONING FAILURE
The first meaningful failure across both runs is a provenance-link rejection at proposal_compatibility (Run 1), which directly contradicts what 57J.25 established: that the v0.15 update path admits user-supported unknowns without requiring answer-derived provenance edges. Run 2 produced a different rejection (semantic compatibility for unclassified meaning) — indicating a second, distinct validation error also blocks the same scenario under the same commit.
First failure only:
Run 1: "New unknown must be explicitly related to an answer-derived node: \"u-engineer-retention\"" at stage proposal_compatibility. The v0.15 candidate no longer admits user-supported unknowns from Answer 1 into the graph — the old provenance-link gate has returned. Run 2 produced a different error at the same stage, confirming the update path is broken under this commit for these inputs.
What remains unproven:
- That any version of v0.15 continues investigation past Update 1 without validation failures
- That downstream investigation (Update 2+) proceeds correctly if Update 1 succeeds
- Whether the provenance-link regression or semantic compatibility error is run-dependent, scenario-dependent, or deterministic under fixed inputs
- Whether
too_broadconversation health would eventually block later turns
This experiment does NOT prove:
- That the v0.15 unknown admission fix works (the 57J.25 result cannot be reproduced)
- Any claim about investigation progress past Update 1
- That other scenarios are unaffected