experiment: test next-question formulation

2026-08-12 13:53:03 +01:00
parent 20e4b58440
commit b1914f5da7
2 changed files with 148 additions and 0 deletions
@@ -529,6 +529,14 @@ Scenario, answers, and maxUpdates restored to canonical defaults before commit.

 ---

+### Experiment 58A.3 — Next-Question Formulation Quality on Correct Target
+
+**Objective:** When the existing savings-realism uncertainty is the correct next target, does the engine formulate a coherent, useful question for it?
+
+**Classification: C — TARGET GOOD, QUESTION MALFORMED.** One update-only call via the committed harness. The engine selected `n_savings_realism` as the next-question target — this is a good target choice given the existing state. However, the question text "What would clarify are the projected office savings from relocation realistic in this situation?" is grammatically broken: it fuses "What would clarify [X] in this situation?" with the full unknown label "are the projected office savings from relocation realistic", producing an ungrammatical sentence that cannot be answered. **The exact same malformed-template failure from 58A.2 persists.** Additionally, a new regression appeared relative to 58A.2: the model changed n_savings_realism status from `unknown` → `weakened` (instead of 58A.2's `provisional`), setting newValue=null — meaning the £2m figure was not even carried forward in this run (58A.2 preserved it as "~£2,000,000/year (pending lease exit cost)"). So the question-formulation defect persists and an evidence-preservation regression appeared simultaneously. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed.
+
+---
+
 ### Experiment 57J.61 — Equivalent Uncertainty Identity Live Test

 **Objective:** Once a dedicated savings-realism uncertainty exists, does a second semantically equivalent statement reuse that same unresolved node rather than create a duplicate? **Classification: D — UPDATE 1 FAILED.** One start + two updates. Start HTTP 200 (6 nodes). Update 1 returned HTTP 200 at update_applied but the harness crash prevented detailed proposal capture. A cold-start variant confirmed that when userSupportedMeaning is populated for savings-realism uncertainty, the model extracts meaning but proposes zero graph mutations — updatedNodes=[{nodeId: X, newValue: null}], addedNodes=[], addedEdges=[]. The gateway rejects this at proposal_compatibility with "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation." Update 2 was reached (total 3 calls) and was rejected for the same reason. **Neither turn established a persistent savings-realism unknown.** The identity invariant cannot be tested when neither turn produces a valid, persistent unknown node. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Full record in `docs/experiment-57j61.md`.