experiment: record 56D (Regression B) and 56E (weak-priority strengthening) results
This commit is contained in:
@@ -0,0 +1,62 @@
|
||||
# Experiment 56D — Regression B via Real Production Path
|
||||
|
||||
**Date**: 2026-08-09
|
||||
**Commit**: 3e78d57 (refine answer meaning derivation for negation and qualification)
|
||||
**Type**: Observation-only — no code changes
|
||||
**Objective**: Verify that deterministic derivation refinement works end-to-end for conditional trade-off scenarios
|
||||
|
||||
---
|
||||
|
||||
## Input (Fixed)
|
||||
|
||||
**Source**: "I want the business to grow, but I don't want to take on more risk."
|
||||
**Answer**: "I'd normally avoid more risk, but for the right opportunity I might accept some."
|
||||
|
||||
## Graph Setup
|
||||
|
||||
Pre-update graph state matched Regression B fixture:
|
||||
- `n-risk-constraint` (unknown/unknown) — active unknown
|
||||
- `obs-source-statement` (observation/supported) — source observation
|
||||
- 1 edge connecting source to risk unknown
|
||||
|
||||
---
|
||||
|
||||
## Results
|
||||
|
||||
| # | Checkpoint | Result |
|
||||
|---|-----------|--------|
|
||||
| 1 | `userSupportedMeaning` extracted | ✅ `"Risk avoidance is a strong default preference that can be overridden for specific opportunities deemed suitable, rather than an absolute hard constraint."` |
|
||||
| 2 | `possibleInference` derived | ✅ `"Growth strategy should focus on identifying and qualifying high-potential opportunities with clearly defined, bounded risk parameters instead of broad or unconditional expansion."` |
|
||||
| 3 | LLM-populated `supportCategory` | null (LLM does not auto-populate; nullable per schema) |
|
||||
| 4 | Derived meaning profile category | **conditional_tradeoff** (derived from userSupportedMeaning via deterministic logic) |
|
||||
| 5 | Guard errors present? | ✅ None — guard passed successfully |
|
||||
| 6 | Risk unknown resolved correctly | `n-risk-constraint`: status→`resolved`, newValue=null, reason=preference vs constraint distinction clarified |
|
||||
| 7 | Proposed graph mutation valid | Updated n-risk-constraint as resolved; created new unknown `n-opportunity-criteria` (unknown/unknown) with dependsOn=[n-risk-constraint] |
|
||||
| 8 | Newly proposed question | `"What specific criteria define an acceptable 'right opportunity' that justifies taking on additional risk?"` targeting the emergent unknown |
|
||||
|
||||
## Key Findings
|
||||
|
||||
1. **Meaning derivation correctly identifies conditional tradeoff**: The `userSupportedMeaning` extraction cleanly separated the default stance (avoid risk) from the qualification (override for right opportunity). This is precisely the Regression B scenario.
|
||||
|
||||
2. **Deterministic profile categorization works end-to-end**: Despite LLM returning null for `supportCategory`, our inline derivation logic (triggered by `hasDefaultPref && hasException` pattern matching on "normally" + "might/accept") correctly derives `conditional_tradeoff`.
|
||||
|
||||
3. **Guard validation passes through**: No guard errors — the resolved node and newly added unknown are both compatible with the source scenario.
|
||||
|
||||
4. **Emergent conditional unknown created successfully**: The system created `n-opportunity-criteria` (kind=unknown, status=unknown) with a description that directly operationalizes the conditional nature: *"Needs explicit criteria to define when additional risk is justified."* This confirms the pipeline correctly recognizes that a conditional tradeoff requires further exploration.
|
||||
|
||||
5. **selectedQuestion targets emergent unknown**: The proposal correctly includes `selectedQuestion` pointing to `n-opportunity-criteria`, maintaining conversation flow toward resolution of the remaining uncertainty.
|
||||
|
||||
6. **LLM does not auto-populate `supportCategory`**: Across runs, `answerMeaning.supportCategory` is consistently null. This confirms the derivation logic in `readDiagnostics` (and the inline pipeline) is the mechanism by which the meaning profile gets determined. This is expected design — the LLM produces the raw meaning; the deterministic layer categorizes it.
|
||||
|
||||
---
|
||||
|
||||
## Verdict
|
||||
|
||||
**Regression B PASSES via real production path.** The full updateCase() pipeline correctly:
|
||||
- Extracts conditional tradeoff semantics from userAnswer
|
||||
- Derives `conditional_tradeoff` category via deterministic profile matching
|
||||
- Resolves the active unknown while creating an emergent conditional/threshold unknown
|
||||
- Passes all guard constraints
|
||||
- Proposes a follow-up question targeting the remaining uncertainty
|
||||
|
||||
No regression detected. The meaning derivation refinement from commit 3e78d57 works as intended for conditional trade-off scenarios.
|
||||
@@ -0,0 +1,102 @@
|
||||
# Experiment 56E — Weak Priority Through Live Production Path
|
||||
|
||||
**Date**: 2026-08-09
|
||||
**Commit**: 3e78d57 (refine answer meaning derivation for negation and qualification)
|
||||
**Type**: Observation-only — no code changes
|
||||
**Objective**: Validate that the production path preserves only what the weak-priority answer establishes (relative importance) without inventing whether risk is or is not a hard constraint.
|
||||
|
||||
---
|
||||
|
||||
## Input (Fixed)
|
||||
|
||||
**Source:** "I want the business to grow, but I don't want to take on more risk."
|
||||
**Answer:** "Risk matters more to me."
|
||||
|
||||
## Graph Setup
|
||||
|
||||
Pre-update graph state matched Regression A fixture:
|
||||
- `n-risk-constraint` (unknown/unknown) — active unknown, status=unknown
|
||||
- No source observation node
|
||||
- 0 edges
|
||||
|
||||
---
|
||||
|
||||
## Results
|
||||
|
||||
| # | Checkpoint | Result |
|
||||
|---|-----------|--------|
|
||||
| 1 | `userSupportedMeaning` extracted | ❌ **"Avoiding additional risk is a preference/trade-off rather than a hard constraint."** — strengthened beyond user input |
|
||||
| 2 | `possibleInference` derived | **"The user prioritizes risk mitigation over aggressive growth strategies."** |
|
||||
| 3 | LLM-populated `supportCategory` | null (LLM does not auto-populate; nullable per schema) |
|
||||
| 4 | Derived meaning profile category | null (LLM returned null; deterministic derivation never triggered because guard passed before derivation step) |
|
||||
| 5 | Guard errors present? | ✅ None — guard passed (it received the already-strengthened userSupportedMeaning, not the raw answer) |
|
||||
| 6 | Risk unknown resolution | `n-risk-constraint`: status→`known`, newValue=`"preference/trade-off"` |
|
||||
| 7 | Guard rejected any node? | No guard errors; proposal accepted |
|
||||
| 8 | New nodes created | None |
|
||||
| 9 | Selected question proposed | null (risk unknown treated as resolved) |
|
||||
|
||||
---
|
||||
|
||||
## Analysis Against Regression A Contract
|
||||
|
||||
### Expected preserved meaning
|
||||
> Risk is of greater relative importance than growth; no hard-constraint or non-hard-constraint boundary established.
|
||||
|
||||
### What the model actually extracted
|
||||
> "Avoiding additional risk is a preference/trade-off **rather than a hard constraint**."
|
||||
|
||||
### Violation
|
||||
The user answered only "Risk matters more to me." — this establishes relative importance only. It says nothing about whether avoiding risk IS or IS NOT a hard constraint.
|
||||
|
||||
The production path's `userSupportedMeaning` field (intended to carry *only* what the user established) now contains a negative assertion: **"rather than a hard constraint"** — an unsupported conclusion that risk is not a hard constraint. This directly violates the Regression A "must not happen" requirement:
|
||||
|
||||
> *Must not happen: Inference that risk avoidance is "not a hard constraint" or equivalent negative assertion.*
|
||||
|
||||
### Failure location
|
||||
The strengthening occurred at the **semantic interpretation layer** (the model's answer-meaning extraction). The deterministic guard saw the already-strengthened meaning and passed it because the proposal was internally consistent. The over-resolution happened before the guard could evaluate it against the original answer.
|
||||
|
||||
This matches the historical finding from Experiment 55A: "Case 2 (weak priority — 'Risk matters more to me.') over-resolved: the model set targetResolved=true and inferred 'not a rigid, non-negotiable constraint' — meaning stronger than the user supplied." The same failure pattern reproduced through the full production path.
|
||||
|
||||
---
|
||||
|
||||
## Verdict
|
||||
|
||||
**FAIL - semantic interpretation**
|
||||
|
||||
For Regression A, the live model and production reasoning path did **not** preserve only what the answer establishes. It invented that risk is "not a hard constraint" from the weak-priority answer alone.
|
||||
|
||||
The PASS requirement is not met:
|
||||
- ❌ `userSupportedMeaning` asserts "rather than a hard constraint" (negative assertion)
|
||||
- ❌ The hard-constraint distinction was resolved to "preference/trade-off" rather than left unresolved
|
||||
- ❌ The deterministic guard could not prevent this because the over-resolution happened before the guard
|
||||
|
||||
---
|
||||
|
||||
## Key Findings
|
||||
|
||||
1. **The strengthening defect persists through commit 3e78d57.** The answer-meaning derivation still converts weak priority ("Risk matters more to me.") into a negative hard-constraint assertion ("rather than a hard constraint"). This is not limited to the resolution layer; it has already leaked into `userSupportedMeaning`.
|
||||
|
||||
2. **The guard cannot catch this because it sees the post-enrichment meaning, not the raw answer.** By the time validation reaches the guard, the strengthening has already been baked into `answerMeaning.userSupportedMeaning`.
|
||||
|
||||
3. **Run-to-run variation in inference field.** Across two identical runs: (a) first run returned possibleInference=null; (b) second run populated it with a derived inference. Both contained the over-resolution in userSupportedMeaning. The enrichment is unstable across runs for the weak-priority case.
|
||||
|
||||
4. **No emergent unknown created.** Unlike Regression B (56D), which correctly created `n-opportunity-criteria` as an emergent unknown, Regression A's graph mutation treated the question as fully resolved — no follow-up needed according to the model's interpretation. This is incorrect: the hard-constraint distinction should remain open.
|
||||
|
||||
---
|
||||
|
||||
## What remains untested
|
||||
|
||||
- Whether separating userSupportedMeaning from inference (as attempted in 55D) actually prevents this strengthening when the contract is enforced end-to-end
|
||||
- Whether the fix from 36faf70 (conditional_qualification normalisation) or 3e78d57 (negation/qualification refinement) addresses weak-priority specifically
|
||||
- Whether adding a post-guard verification layer that compares `userSupportedMeaning` against the original answer text can catch this class of over-resolution
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
- **Host:** http://192.168.1.111:11434
|
||||
- **Model:** qwen-claude:latest
|
||||
- **Branch:** feature/reasoning-fidelity-v0.8
|
||||
- **Production code changed:** NO
|
||||
- **Temporary instrumentation:** minimal Node script only — removed after capture
|
||||
|
||||
Reference in New Issue
Block a user