103 lines
5.9 KiB
Markdown
103 lines
5.9 KiB
Markdown
# Experiment 56E — Weak Priority Through Live Production Path
|
|
|
|
**Date**: 2026-08-09
|
|
**Commit**: 3e78d57 (refine answer meaning derivation for negation and qualification)
|
|
**Type**: Observation-only — no code changes
|
|
**Objective**: Validate that the production path preserves only what the weak-priority answer establishes (relative importance) without inventing whether risk is or is not a hard constraint.
|
|
|
|
---
|
|
|
|
## Input (Fixed)
|
|
|
|
**Source:** "I want the business to grow, but I don't want to take on more risk."
|
|
**Answer:** "Risk matters more to me."
|
|
|
|
## Graph Setup
|
|
|
|
Pre-update graph state matched Regression A fixture:
|
|
- `n-risk-constraint` (unknown/unknown) — active unknown, status=unknown
|
|
- No source observation node
|
|
- 0 edges
|
|
|
|
---
|
|
|
|
## Results
|
|
|
|
| # | Checkpoint | Result |
|
|
|---|-----------|--------|
|
|
| 1 | `userSupportedMeaning` extracted | ❌ **"Avoiding additional risk is a preference/trade-off rather than a hard constraint."** — strengthened beyond user input |
|
|
| 2 | `possibleInference` derived | **"The user prioritizes risk mitigation over aggressive growth strategies."** |
|
|
| 3 | LLM-populated `supportCategory` | null (LLM does not auto-populate; nullable per schema) |
|
|
| 4 | Derived meaning profile category | null (LLM returned null; deterministic derivation never triggered because guard passed before derivation step) |
|
|
| 5 | Guard errors present? | ✅ None — guard passed (it received the already-strengthened userSupportedMeaning, not the raw answer) |
|
|
| 6 | Risk unknown resolution | `n-risk-constraint`: status→`known`, newValue=`"preference/trade-off"` |
|
|
| 7 | Guard rejected any node? | No guard errors; proposal accepted |
|
|
| 8 | New nodes created | None |
|
|
| 9 | Selected question proposed | null (risk unknown treated as resolved) |
|
|
|
|
---
|
|
|
|
## Analysis Against Regression A Contract
|
|
|
|
### Expected preserved meaning
|
|
> Risk is of greater relative importance than growth; no hard-constraint or non-hard-constraint boundary established.
|
|
|
|
### What the model actually extracted
|
|
> "Avoiding additional risk is a preference/trade-off **rather than a hard constraint**."
|
|
|
|
### Violation
|
|
The user answered only "Risk matters more to me." — this establishes relative importance only. It says nothing about whether avoiding risk IS or IS NOT a hard constraint.
|
|
|
|
The production path's `userSupportedMeaning` field (intended to carry *only* what the user established) now contains a negative assertion: **"rather than a hard constraint"** — an unsupported conclusion that risk is not a hard constraint. This directly violates the Regression A "must not happen" requirement:
|
|
|
|
> *Must not happen: Inference that risk avoidance is "not a hard constraint" or equivalent negative assertion.*
|
|
|
|
### Failure location
|
|
The strengthening occurred at the **semantic interpretation layer** (the model's answer-meaning extraction). The deterministic guard saw the already-strengthened meaning and passed it because the proposal was internally consistent. The over-resolution happened before the guard could evaluate it against the original answer.
|
|
|
|
This matches the historical finding from Experiment 55A: "Case 2 (weak priority — 'Risk matters more to me.') over-resolved: the model set targetResolved=true and inferred 'not a rigid, non-negotiable constraint' — meaning stronger than the user supplied." The same failure pattern reproduced through the full production path.
|
|
|
|
---
|
|
|
|
## Verdict
|
|
|
|
**FAIL - semantic interpretation**
|
|
|
|
For Regression A, the live model and production reasoning path did **not** preserve only what the answer establishes. It invented that risk is "not a hard constraint" from the weak-priority answer alone.
|
|
|
|
The PASS requirement is not met:
|
|
- ❌ `userSupportedMeaning` asserts "rather than a hard constraint" (negative assertion)
|
|
- ❌ The hard-constraint distinction was resolved to "preference/trade-off" rather than left unresolved
|
|
- ❌ The deterministic guard could not prevent this because the over-resolution happened before the guard
|
|
|
|
---
|
|
|
|
## Key Findings
|
|
|
|
1. **The strengthening defect persists through commit 3e78d57.** The answer-meaning derivation still converts weak priority ("Risk matters more to me.") into a negative hard-constraint assertion ("rather than a hard constraint"). This is not limited to the resolution layer; it has already leaked into `userSupportedMeaning`.
|
|
|
|
2. **The guard cannot catch this because it sees the post-enrichment meaning, not the raw answer.** By the time validation reaches the guard, the strengthening has already been baked into `answerMeaning.userSupportedMeaning`.
|
|
|
|
3. **Run-to-run variation in inference field.** Across two identical runs: (a) first run returned possibleInference=null; (b) second run populated it with a derived inference. Both contained the over-resolution in userSupportedMeaning. The enrichment is unstable across runs for the weak-priority case.
|
|
|
|
4. **No emergent unknown created.** Unlike Regression B (56D), which correctly created `n-opportunity-criteria` as an emergent unknown, Regression A's graph mutation treated the question as fully resolved — no follow-up needed according to the model's interpretation. This is incorrect: the hard-constraint distinction should remain open.
|
|
|
|
---
|
|
|
|
## What remains untested
|
|
|
|
- Whether separating userSupportedMeaning from inference (as attempted in 55D) actually prevents this strengthening when the contract is enforced end-to-end
|
|
- Whether the fix from 36faf70 (conditional_qualification normalisation) or 3e78d57 (negation/qualification refinement) addresses weak-priority specifically
|
|
- Whether adding a post-guard verification layer that compares `userSupportedMeaning` against the original answer text can catch this class of over-resolution
|
|
|
|
---
|
|
|
|
## Configuration
|
|
|
|
- **Host:** http://192.168.1.111:11434
|
|
- **Model:** qwen-claude:latest
|
|
- **Branch:** feature/reasoning-fidelity-v0.8
|
|
- **Production code changed:** NO
|
|
- **Temporary instrumentation:** minimal Node script only — removed after capture
|
|
|