experiment: validate raw-answer safeguard for weak priority
This commit is contained in:
@@ -0,0 +1,117 @@
|
||||
# Experiment 56F — Weak Priority Re-tested with Canonical Live Harness
|
||||
|
||||
**Date**: 2026-08-09
|
||||
**Commit**: 4aa1492 (refine raw-answer boundary for answer meaning)
|
||||
**Type**: Observation-only — no code changes
|
||||
**Objective**: After Codex commit `4aa1492`, does Regression A now leave constraint status unresolved instead of allowing "Risk matters more to me." to become "not a hard constraint" or equivalent?
|
||||
|
||||
---
|
||||
|
||||
## Input (Fixed — Regression A)
|
||||
|
||||
**Source:** "I want the business to grow, but I don't want to take on more risk."
|
||||
**Answer:** "Risk matters more to me."
|
||||
|
||||
## Graph Setup
|
||||
|
||||
Pre-update graph state matched Regression A fixture:
|
||||
- `n-risk-constraint` (unknown/unknown) — active unknown, status=unknown
|
||||
- No source observation node
|
||||
- 0 edges
|
||||
|
||||
---
|
||||
|
||||
## Results
|
||||
|
||||
| # | Checkpoint | Result |
|
||||
|---|-----------|--------|
|
||||
| 1 | `userSupportedMeaning` extracted | **"Avoiding additional risk is a strongly weighted preference/trade-off rather than a hard constraint."** — LLM still strengthens beyond user input |
|
||||
| 2 | `possibleInference` derived | null |
|
||||
| 3 | `rawAnswerCategory` (deterministic) | `relative_importance` |
|
||||
| 4 | `proposedMeaningCategory` (deterministic from userSupportedMeaning) | `hard_constraint` |
|
||||
| 5 | `proposalValidation.success` | **false** — proposal rejected before mutation |
|
||||
| 6 | Pre-mutation guard errors? | Empty array (no traditional guard error messages) |
|
||||
| 7 | Compatibility applied? | **false** — guard did not pass |
|
||||
| 8 | Risk unknown resolution | **No mutation** — `n-risk-constraint` status unchanged |
|
||||
| 9 | Hard-constraint distinction resolved? | **NO** |
|
||||
|
||||
---
|
||||
|
||||
## Analysis Against Regression A Contract
|
||||
|
||||
### Expected preserved meaning
|
||||
> Risk is of greater relative importance than growth; no hard-constraint or non-hard-constraint boundary established.
|
||||
|
||||
### What the model extracted (userSupportedMeaning)
|
||||
> "Avoiding additional risk is a strongly weighted preference/trade-off **rather than a hard constraint**."
|
||||
|
||||
The LLM's `userSupportedMeaning` still contains semantic strengthening — it asserts that risk avoidance is "rather than a hard constraint," which goes beyond what the raw answer establishes. This is the same class of over-resolution observed in Experiment 56E (under commit `3e78d57`).
|
||||
|
||||
### What prevented graph mutation
|
||||
The pre-mutation safeguard chain rejected the proposal:
|
||||
- **Deterministic derivation** produced `proposedMeaningCategory: hard_constraint` from the strengthened meaning.
|
||||
- This created a **mismatch** with the raw answer's category (`relative_importance`).
|
||||
- The mismatch caused `proposalValidation.success: false` and prevented the compatibility guard from passing.
|
||||
- **No graph mutation occurred.** `n-risk-constraint` remained unresolved (status=unknown, value=null).
|
||||
|
||||
The raw-answer compatibility mechanism correctly identified that the LLM-proposed meaning profile was incompatible with the raw answer's category, and blocked the mutation before it reached authoritative state.
|
||||
|
||||
### Verdict
|
||||
|
||||
**PASS - strengthening safely rejected**
|
||||
|
||||
The final authoritative graph state does **not** establish either:
|
||||
- risk is a hard constraint; nor
|
||||
- risk is not a hard constraint;
|
||||
|
||||
from "Risk matters more to me." alone. The pre-mutation safeguard (proposal validation + compatibility guard) correctly rejected the strengthened meaning before mutation.
|
||||
|
||||
---
|
||||
|
||||
## Key Find
|
||||
|
||||
1. **Semantic strengthening in `userSupportedMeaning` persists.** After commit `4aa1492`, the LLM still converts "Risk matters more to me." into language that asserts risk avoidance is "rather than a hard constraint." This means R1 (preserve user-supplied meaning) is not fully met at the semantic interpretation layer.
|
||||
|
||||
2. **Pre-mutation safeguard works.** Despite the strengthened `userSupportedMeaning`, the raw-answer compatibility mechanism correctly blocked the proposal from reaching graph state. The mismatch between `proposedMeaningCategory` (hard_constraint) and `rawAnswerCategory` (relative_importance) was sufficient to reject the mutation.
|
||||
|
||||
3. **No emergent unknown created.** Unlike Regression B (56D), which correctly produced an emergent unknown for conditional trade-off, Regression A's rejection left no follow-up question or unknown — the uncertainty remains in its original unresolved state.
|
||||
|
||||
4. **Deterministic derivation is functional.** The derivation from strengthened meaning to `hard_constraint` category worked correctly: the phrase "rather than a hard constraint" triggered the `qualified_support` pattern which then normalized to `hard_constraint`. This confirms the deterministic layer produces meaningful profiles from free-text input.
|
||||
|
||||
---
|
||||
|
||||
## What this established
|
||||
|
||||
- After commit `4aa1492`, Regression A no longer allows unsupported constraint status to reach graph state via the production path. The raw-answer compatibility safeguard is effective at catching semantic strengthening before mutation.
|
||||
- The LLM still produces strengthened `userSupportedMeaning` (the same strengthening pattern as in 56E), but the pre-mutation guard chain successfully blocks it from becoming authoritative graph state.
|
||||
|
||||
## What remains untested
|
||||
|
||||
- Whether the LLM's tendency to strengthen weak-priority answers can be reduced at the prompt/interpretation layer (this is a question for the semantic interpretation model, not just the guard).
|
||||
- Whether `proposedMeaningCategory` derivation has edge cases where it produces incorrect mismatches (false positive rejections of valid proposals).
|
||||
- Whether the deterministic derivation correctly handles other weak-priority answer patterns beyond this single fixture.
|
||||
- Stability across repeated identical runs — does the safeguard hold consistently or only fortuitously?
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
- **Host:** http://192.168.1.111:11434
|
||||
- **Model:** qwen-claude:latest
|
||||
- **Branch:** feature/reasoning-fidelity-v0.8
|
||||
- **Harness:** tests/graph/live-update-experiment-helper.cjs (canonical)
|
||||
- **Runner:** experiment-56f-runner.mjs (temporary, removed after capture)
|
||||
- **Production code changed:** NO
|
||||
- **Live calls:** 1
|
||||
|
||||
---
|
||||
|
||||
## Regression A Result Summary
|
||||
|
||||
| Aspect | Before 4aa1492 (Exp 56E) | After 4aa1492 (Exp 56F) |
|
||||
|--------|--------------------------|--------------------------|
|
||||
| Semantic strengthening in `userSupportedMeaning` | YES | YES (persisted) |
|
||||
| Pre-mutation safeguard rejection | Not observed / unclear | YES — proposalValidation false, compatibilityGuard false |
|
||||
| Graph mutation for risk-constraint | YES (status→known, value="preference/trade-off") | NO (no mutation) |
|
||||
| Hard-constraint distinction resolved? | YES (to "preference/trade-off") | NO |
|
||||
| Verdict | FAIL - semantic interpretation | PASS - strengthening safely rejected |
|
||||
Reference in New Issue
Block a user