experiment: validate raw-answer safeguard for weak priority

2026-08-09 12:06:50 +01:00
parent e6f784261b
commit b06538ee91
2 changed files with 119 additions and 0 deletions
@@ -26,6 +26,8 @@ Experiment 56D confirmed that Regression B (conditional trade-off resolution) wo

 Experiment 56E tested whether the weak-priority answer ("Risk matters more to me.") survives the full `updateCase()` production path without strengthening beyond relative importance. Result: **FAIL - semantic interpretation**. The LLM extracted userSupportedMeaning as "Avoiding additional risk is a preference/trade-off rather than a hard constraint" — asserting that risk is not a hard constraint, which goes beyond what the answer establishes (only relative importance). The deterministic guard passed because it saw the already-strengthened meaning. n-risk-constraint was incorrectly treated as resolved to "preference/trade-off". No emergent unknown created. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Status pending Rob's review.

+Experiment 56F re-tested Regression A with the canonical live harness after Codex commit `4aa1492` (refine raw-answer boundary for answer meaning). Result: **PASS - strengthening safely rejected**. The LLM still produced semantic strengthening in `userSupportedMeaning` ("Avoiding additional risk is a strongly weighted preference/trade-off rather than a hard constraint") — the same class of over-resolution as 56E. However, the pre-mutation safeguard chain correctly rejected the proposal: deterministic derivation produced `proposedMeaningCategory: hard_constraint` which mismatched `rawAnswerCategory: relative_importance`, causing `proposalValidation.success: false` and preventing compatibility guard from passing. No graph mutation occurred — `n-risk-constraint` remained unresolved (status=unknown, value=null). One live call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed. Status pending Rob's review.
+
 Experiment 54S tested whether, once clarification is known to be required, the model can identify exactly what the user needs to clarify — three fixed cases: growth-versus-risk priority (true → "preference/trade-off or hard constraint"), evidence-resolvable delivery causes (false → null), ambiguous meaning of "affordable" (true → "upfront cost versus long-term total cost"). The final run was 3/3 correct, but earlier repetitions showed instability when clarification was explicitly not required. Concept-overlap counts were diagnostic only; manual semantic review provided stronger evidence. Case 2 instability is an observed behaviour, not merely a test warning. Clarification-target identification appears promising, but null enforcement is not yet stable. Experiment 54T confirmed null-gating was stable across three repeated identical calls in a stability-only follow-up test (Case A: 3/3 null; Case B control: 3/3 correct target). The current instruction and output contract produced stable null behaviour across the three repeated false-case runs tested there; broader stability remains unproven. Experiment 54U tested whether a fixed clarification target can survive into one neutral user-facing question without adding meaning (preference/constraint, affordability definition, private factual capacity). All three cases returned correct single neutral questions with no introduced assumptions or evidence requests. The clarification-target → question step worked cleanly across the three tested targets; broader wording quality and user experience remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review.

 Experiment 54V tested whether the user's answer can resolve only that target without rewriting the rest of the source meaning. Three fixed cases: hard constraint resolved (true/null), affordability definition resolved (true/null), incomplete answer preserved (false/uncertainty). All three correct across boundary preservation, no forced interpretations, and no unsupported consequences or new questions generated. Clarification answers resolved only the intended target across all tested cases. **The individual clarification steps have each worked in their isolated fixed-case tests; end-to-end behaviour remains untested.** Graph updates, next-question choice, Behaviour Selection, and UI remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review.