From b06538ee91d1f0990c01fc61ce22ac294924d0b4 Mon Sep 17 00:00:00 2001 From: robbond Date: Sun, 9 Aug 2026 12:06:50 +0100 Subject: [PATCH] experiment: validate raw-answer safeguard for weak priority --- docs/current-handoff.md | 2 + docs/experiment-56f.md | 117 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 119 insertions(+) create mode 100644 docs/experiment-56f.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 8758d3f..c1fb654 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -26,6 +26,8 @@ Experiment 56D confirmed that Regression B (conditional trade-off resolution) wo Experiment 56E tested whether the weak-priority answer ("Risk matters more to me.") survives the full `updateCase()` production path without strengthening beyond relative importance. Result: **FAIL - semantic interpretation**. The LLM extracted userSupportedMeaning as "Avoiding additional risk is a preference/trade-off rather than a hard constraint" — asserting that risk is not a hard constraint, which goes beyond what the answer establishes (only relative importance). The deterministic guard passed because it saw the already-strengthened meaning. n-risk-constraint was incorrectly treated as resolved to "preference/trade-off". No emergent unknown created. Same host/model (qwen-claude:latest on http://192.168.1.111:11434). Status pending Rob's review. +Experiment 56F re-tested Regression A with the canonical live harness after Codex commit `4aa1492` (refine raw-answer boundary for answer meaning). Result: **PASS - strengthening safely rejected**. The LLM still produced semantic strengthening in `userSupportedMeaning` ("Avoiding additional risk is a strongly weighted preference/trade-off rather than a hard constraint") — the same class of over-resolution as 56E. However, the pre-mutation safeguard chain correctly rejected the proposal: deterministic derivation produced `proposedMeaningCategory: hard_constraint` which mismatched `rawAnswerCategory: relative_importance`, causing `proposalValidation.success: false` and preventing compatibility guard from passing. No graph mutation occurred — `n-risk-constraint` remained unresolved (status=unknown, value=null). One live call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed. Status pending Rob's review. + Experiment 54S tested whether, once clarification is known to be required, the model can identify exactly what the user needs to clarify — three fixed cases: growth-versus-risk priority (true → "preference/trade-off or hard constraint"), evidence-resolvable delivery causes (false → null), ambiguous meaning of "affordable" (true → "upfront cost versus long-term total cost"). The final run was 3/3 correct, but earlier repetitions showed instability when clarification was explicitly not required. Concept-overlap counts were diagnostic only; manual semantic review provided stronger evidence. Case 2 instability is an observed behaviour, not merely a test warning. Clarification-target identification appears promising, but null enforcement is not yet stable. Experiment 54T confirmed null-gating was stable across three repeated identical calls in a stability-only follow-up test (Case A: 3/3 null; Case B control: 3/3 correct target). The current instruction and output contract produced stable null behaviour across the three repeated false-case runs tested there; broader stability remains unproven. Experiment 54U tested whether a fixed clarification target can survive into one neutral user-facing question without adding meaning (preference/constraint, affordability definition, private factual capacity). All three cases returned correct single neutral questions with no introduced assumptions or evidence requests. The clarification-target → question step worked cleanly across the three tested targets; broader wording quality and user experience remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review. Experiment 54V tested whether the user's answer can resolve only that target without rewriting the rest of the source meaning. Three fixed cases: hard constraint resolved (true/null), affordability definition resolved (true/null), incomplete answer preserved (false/uncertainty). All three correct across boundary preservation, no forced interpretations, and no unsupported consequences or new questions generated. Clarification answers resolved only the intended target across all tested cases. **The individual clarification steps have each worked in their isolated fixed-case tests; end-to-end behaviour remains untested.** Graph updates, next-question choice, Behaviour Selection, and UI remain untested. Same host/model (qwen-claude:latest on http://192.168.1.111:11434); no production code changed. Status pending Rob's review. diff --git a/docs/experiment-56f.md b/docs/experiment-56f.md new file mode 100644 index 0000000..dc0c194 --- /dev/null +++ b/docs/experiment-56f.md @@ -0,0 +1,117 @@ +# Experiment 56F — Weak Priority Re-tested with Canonical Live Harness + +**Date**: 2026-08-09 +**Commit**: 4aa1492 (refine raw-answer boundary for answer meaning) +**Type**: Observation-only — no code changes +**Objective**: After Codex commit `4aa1492`, does Regression A now leave constraint status unresolved instead of allowing "Risk matters more to me." to become "not a hard constraint" or equivalent? + +--- + +## Input (Fixed — Regression A) + +**Source:** "I want the business to grow, but I don't want to take on more risk." +**Answer:** "Risk matters more to me." + +## Graph Setup + +Pre-update graph state matched Regression A fixture: +- `n-risk-constraint` (unknown/unknown) — active unknown, status=unknown +- No source observation node +- 0 edges + +--- + +## Results + +| # | Checkpoint | Result | +|---|-----------|--------| +| 1 | `userSupportedMeaning` extracted | **"Avoiding additional risk is a strongly weighted preference/trade-off rather than a hard constraint."** — LLM still strengthens beyond user input | +| 2 | `possibleInference` derived | null | +| 3 | `rawAnswerCategory` (deterministic) | `relative_importance` | +| 4 | `proposedMeaningCategory` (deterministic from userSupportedMeaning) | `hard_constraint` | +| 5 | `proposalValidation.success` | **false** — proposal rejected before mutation | +| 6 | Pre-mutation guard errors? | Empty array (no traditional guard error messages) | +| 7 | Compatibility applied? | **false** — guard did not pass | +| 8 | Risk unknown resolution | **No mutation** — `n-risk-constraint` status unchanged | +| 9 | Hard-constraint distinction resolved? | **NO** | + +--- + +## Analysis Against Regression A Contract + +### Expected preserved meaning +> Risk is of greater relative importance than growth; no hard-constraint or non-hard-constraint boundary established. + +### What the model extracted (userSupportedMeaning) +> "Avoiding additional risk is a strongly weighted preference/trade-off **rather than a hard constraint**." + +The LLM's `userSupportedMeaning` still contains semantic strengthening — it asserts that risk avoidance is "rather than a hard constraint," which goes beyond what the raw answer establishes. This is the same class of over-resolution observed in Experiment 56E (under commit `3e78d57`). + +### What prevented graph mutation +The pre-mutation safeguard chain rejected the proposal: +- **Deterministic derivation** produced `proposedMeaningCategory: hard_constraint` from the strengthened meaning. +- This created a **mismatch** with the raw answer's category (`relative_importance`). +- The mismatch caused `proposalValidation.success: false` and prevented the compatibility guard from passing. +- **No graph mutation occurred.** `n-risk-constraint` remained unresolved (status=unknown, value=null). + +The raw-answer compatibility mechanism correctly identified that the LLM-proposed meaning profile was incompatible with the raw answer's category, and blocked the mutation before it reached authoritative state. + +### Verdict + +**PASS - strengthening safely rejected** + +The final authoritative graph state does **not** establish either: +- risk is a hard constraint; nor +- risk is not a hard constraint; + +from "Risk matters more to me." alone. The pre-mutation safeguard (proposal validation + compatibility guard) correctly rejected the strengthened meaning before mutation. + +--- + +## Key Find + +1. **Semantic strengthening in `userSupportedMeaning` persists.** After commit `4aa1492`, the LLM still converts "Risk matters more to me." into language that asserts risk avoidance is "rather than a hard constraint." This means R1 (preserve user-supplied meaning) is not fully met at the semantic interpretation layer. + +2. **Pre-mutation safeguard works.** Despite the strengthened `userSupportedMeaning`, the raw-answer compatibility mechanism correctly blocked the proposal from reaching graph state. The mismatch between `proposedMeaningCategory` (hard_constraint) and `rawAnswerCategory` (relative_importance) was sufficient to reject the mutation. + +3. **No emergent unknown created.** Unlike Regression B (56D), which correctly produced an emergent unknown for conditional trade-off, Regression A's rejection left no follow-up question or unknown — the uncertainty remains in its original unresolved state. + +4. **Deterministic derivation is functional.** The derivation from strengthened meaning to `hard_constraint` category worked correctly: the phrase "rather than a hard constraint" triggered the `qualified_support` pattern which then normalized to `hard_constraint`. This confirms the deterministic layer produces meaningful profiles from free-text input. + +--- + +## What this established + +- After commit `4aa1492`, Regression A no longer allows unsupported constraint status to reach graph state via the production path. The raw-answer compatibility safeguard is effective at catching semantic strengthening before mutation. +- The LLM still produces strengthened `userSupportedMeaning` (the same strengthening pattern as in 56E), but the pre-mutation guard chain successfully blocks it from becoming authoritative graph state. + +## What remains untested + +- Whether the LLM's tendency to strengthen weak-priority answers can be reduced at the prompt/interpretation layer (this is a question for the semantic interpretation model, not just the guard). +- Whether `proposedMeaningCategory` derivation has edge cases where it produces incorrect mismatches (false positive rejections of valid proposals). +- Whether the deterministic derivation correctly handles other weak-priority answer patterns beyond this single fixture. +- Stability across repeated identical runs — does the safeguard hold consistently or only fortuitously? + +--- + +## Configuration + +- **Host:** http://192.168.1.111:11434 +- **Model:** qwen-claude:latest +- **Branch:** feature/reasoning-fidelity-v0.8 +- **Harness:** tests/graph/live-update-experiment-helper.cjs (canonical) +- **Runner:** experiment-56f-runner.mjs (temporary, removed after capture) +- **Production code changed:** NO +- **Live calls:** 1 + +--- + +## Regression A Result Summary + +| Aspect | Before 4aa1492 (Exp 56E) | After 4aa1492 (Exp 56F) | +|--------|--------------------------|--------------------------| +| Semantic strengthening in `userSupportedMeaning` | YES | YES (persisted) | +| Pre-mutation safeguard rejection | Not observed / unclear | YES — proposalValidation false, compatibilityGuard false | +| Graph mutation for risk-constraint | YES (status→known, value="preference/trade-off") | NO (no mutation) | +| Hard-constraint distinction resolved? | YES (to "preference/trade-off") | NO | +| Verdict | FAIL - semantic interpretation | PASS - strengthening safely rejected |