experiment: test qualified-answer reasoning

2026-08-12 13:35:40 +01:00
parent a78f3edb10
commit 32184694c5
2 changed files with 127 additions and 0 deletions
@@ -513,6 +513,14 @@ Scenario, answers, and maxUpdates restored to canonical defaults before commit.

 ---

+### Experiment 58A.1 — Qualified Answer Reasoning Test
+
+**Objective:** When the user gives a mixed answer containing useful evidence, doubt, and a new assumption, does the engine preserve all three without over-resolving the existing uncertainty, and does it ask the right next question?
+
+**Classification: B — MOSTLY GOOD, INFORMATION LOSS.** One update-only call via the committed harness. The engine preserved the original savings-realism uncertainty (n_savings_realism remains unresolved, not duplicated) and structurally represented both newly exposed assumptions as dedicated unknown nodes (n_lease_penalty and n_disruption_cost). A next question targeting disruption cost was selected — relevant and material but debatable versus lease penalty. **Defect:** The £2m figure was lost — updatedNodes is empty, so no node carries the numerical claim that motivated the answer. Harness does not print answerMeaning fields for updateOnly mode's accepted path, so whether userSupportedMeaning captured doubt or the £2m reference is unavailable. Core reasoning direction is right but meaningful evidence loss prevents an A classification. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Full record in `docs/experiment-58a1.md`.
+
+---
+
 ### Experiment 57J.61 — Equivalent Uncertainty Identity Live Test

 **Objective:** Once a dedicated savings-realism uncertainty exists, does a second semantically equivalent statement reuse that same unresolved node rather than create a duplicate? **Classification: D — UPDATE 1 FAILED.** One start + two updates. Start HTTP 200 (6 nodes). Update 1 returned HTTP 200 at update_applied but the harness crash prevented detailed proposal capture. A cold-start variant confirmed that when userSupportedMeaning is populated for savings-realism uncertainty, the model extracts meaning but proposes zero graph mutations — updatedNodes=[{nodeId: X, newValue: null}], addedNodes=[], addedEdges=[]. The gateway rejects this at proposal_compatibility with "answerMeaning.userSupportedMeaning is populated, but the proposal contains no graph mutation." Update 2 was reached (total 3 calls) and was rejected for the same reason. **Neither turn established a persistent savings-realism unknown.** The identity invariant cannot be tested when neither turn produces a valid, persistent unknown node. Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No production code changed. Full record in `docs/experiment-57j61.md`.