diff --git a/docs/experiment-56d.md b/docs/experiment-56d.md new file mode 100644 index 0000000..be77605 --- /dev/null +++ b/docs/experiment-56d.md @@ -0,0 +1,62 @@ +# Experiment 56D — Regression B via Real Production Path + +**Date**: 2026-08-09 +**Commit**: 3e78d57 (refine answer meaning derivation for negation and qualification) +**Type**: Observation-only — no code changes +**Objective**: Verify that deterministic derivation refinement works end-to-end for conditional trade-off scenarios + +--- + +## Input (Fixed) + +**Source**: "I want the business to grow, but I don't want to take on more risk." +**Answer**: "I'd normally avoid more risk, but for the right opportunity I might accept some." + +## Graph Setup + +Pre-update graph state matched Regression B fixture: +- `n-risk-constraint` (unknown/unknown) — active unknown +- `obs-source-statement` (observation/supported) — source observation +- 1 edge connecting source to risk unknown + +--- + +## Results + +| # | Checkpoint | Result | +|---|-----------|--------| +| 1 | `userSupportedMeaning` extracted | ✅ `"Risk avoidance is a strong default preference that can be overridden for specific opportunities deemed suitable, rather than an absolute hard constraint."` | +| 2 | `possibleInference` derived | ✅ `"Growth strategy should focus on identifying and qualifying high-potential opportunities with clearly defined, bounded risk parameters instead of broad or unconditional expansion."` | +| 3 | LLM-populated `supportCategory` | null (LLM does not auto-populate; nullable per schema) | +| 4 | Derived meaning profile category | **conditional_tradeoff** (derived from userSupportedMeaning via deterministic logic) | +| 5 | Guard errors present? | ✅ None — guard passed successfully | +| 6 | Risk unknown resolved correctly | `n-risk-constraint`: status→`resolved`, newValue=null, reason=preference vs constraint distinction clarified | +| 7 | Proposed graph mutation valid | Updated n-risk-constraint as resolved; created new unknown `n-opportunity-criteria` (unknown/unknown) with dependsOn=[n-risk-constraint] | +| 8 | Newly proposed question | `"What specific criteria define an acceptable 'right opportunity' that justifies taking on additional risk?"` targeting the emergent unknown | + +## Key Findings + +1. **Meaning derivation correctly identifies conditional tradeoff**: The `userSupportedMeaning` extraction cleanly separated the default stance (avoid risk) from the qualification (override for right opportunity). This is precisely the Regression B scenario. + +2. **Deterministic profile categorization works end-to-end**: Despite LLM returning null for `supportCategory`, our inline derivation logic (triggered by `hasDefaultPref && hasException` pattern matching on "normally" + "might/accept") correctly derives `conditional_tradeoff`. + +3. **Guard validation passes through**: No guard errors — the resolved node and newly added unknown are both compatible with the source scenario. + +4. **Emergent conditional unknown created successfully**: The system created `n-opportunity-criteria` (kind=unknown, status=unknown) with a description that directly operationalizes the conditional nature: *"Needs explicit criteria to define when additional risk is justified."* This confirms the pipeline correctly recognizes that a conditional tradeoff requires further exploration. + +5. **selectedQuestion targets emergent unknown**: The proposal correctly includes `selectedQuestion` pointing to `n-opportunity-criteria`, maintaining conversation flow toward resolution of the remaining uncertainty. + +6. **LLM does not auto-populate `supportCategory`**: Across runs, `answerMeaning.supportCategory` is consistently null. This confirms the derivation logic in `readDiagnostics` (and the inline pipeline) is the mechanism by which the meaning profile gets determined. This is expected design — the LLM produces the raw meaning; the deterministic layer categorizes it. + +--- + +## Verdict + +**Regression B PASSES via real production path.** The full updateCase() pipeline correctly: +- Extracts conditional tradeoff semantics from userAnswer +- Derives `conditional_tradeoff` category via deterministic profile matching +- Resolves the active unknown while creating an emergent conditional/threshold unknown +- Passes all guard constraints +- Proposes a follow-up question targeting the remaining uncertainty + +No regression detected. The meaning derivation refinement from commit 3e78d57 works as intended for conditional trade-off scenarios. diff --git a/docs/experiment-56e.md b/docs/experiment-56e.md new file mode 100644 index 0000000..040af53 --- /dev/null +++ b/docs/experiment-56e.md @@ -0,0 +1,102 @@ +# Experiment 56E — Weak Priority Through Live Production Path + +**Date**: 2026-08-09 +**Commit**: 3e78d57 (refine answer meaning derivation for negation and qualification) +**Type**: Observation-only — no code changes +**Objective**: Validate that the production path preserves only what the weak-priority answer establishes (relative importance) without inventing whether risk is or is not a hard constraint. + +--- + +## Input (Fixed) + +**Source:** "I want the business to grow, but I don't want to take on more risk." +**Answer:** "Risk matters more to me." + +## Graph Setup + +Pre-update graph state matched Regression A fixture: +- `n-risk-constraint` (unknown/unknown) — active unknown, status=unknown +- No source observation node +- 0 edges + +--- + +## Results + +| # | Checkpoint | Result | +|---|-----------|--------| +| 1 | `userSupportedMeaning` extracted | ❌ **"Avoiding additional risk is a preference/trade-off rather than a hard constraint."** — strengthened beyond user input | +| 2 | `possibleInference` derived | **"The user prioritizes risk mitigation over aggressive growth strategies."** | +| 3 | LLM-populated `supportCategory` | null (LLM does not auto-populate; nullable per schema) | +| 4 | Derived meaning profile category | null (LLM returned null; deterministic derivation never triggered because guard passed before derivation step) | +| 5 | Guard errors present? | ✅ None — guard passed (it received the already-strengthened userSupportedMeaning, not the raw answer) | +| 6 | Risk unknown resolution | `n-risk-constraint`: status→`known`, newValue=`"preference/trade-off"` | +| 7 | Guard rejected any node? | No guard errors; proposal accepted | +| 8 | New nodes created | None | +| 9 | Selected question proposed | null (risk unknown treated as resolved) | + +--- + +## Analysis Against Regression A Contract + +### Expected preserved meaning +> Risk is of greater relative importance than growth; no hard-constraint or non-hard-constraint boundary established. + +### What the model actually extracted +> "Avoiding additional risk is a preference/trade-off **rather than a hard constraint**." + +### Violation +The user answered only "Risk matters more to me." — this establishes relative importance only. It says nothing about whether avoiding risk IS or IS NOT a hard constraint. + +The production path's `userSupportedMeaning` field (intended to carry *only* what the user established) now contains a negative assertion: **"rather than a hard constraint"** — an unsupported conclusion that risk is not a hard constraint. This directly violates the Regression A "must not happen" requirement: + +> *Must not happen: Inference that risk avoidance is "not a hard constraint" or equivalent negative assertion.* + +### Failure location +The strengthening occurred at the **semantic interpretation layer** (the model's answer-meaning extraction). The deterministic guard saw the already-strengthened meaning and passed it because the proposal was internally consistent. The over-resolution happened before the guard could evaluate it against the original answer. + +This matches the historical finding from Experiment 55A: "Case 2 (weak priority — 'Risk matters more to me.') over-resolved: the model set targetResolved=true and inferred 'not a rigid, non-negotiable constraint' — meaning stronger than the user supplied." The same failure pattern reproduced through the full production path. + +--- + +## Verdict + +**FAIL - semantic interpretation** + +For Regression A, the live model and production reasoning path did **not** preserve only what the answer establishes. It invented that risk is "not a hard constraint" from the weak-priority answer alone. + +The PASS requirement is not met: +- ❌ `userSupportedMeaning` asserts "rather than a hard constraint" (negative assertion) +- ❌ The hard-constraint distinction was resolved to "preference/trade-off" rather than left unresolved +- ❌ The deterministic guard could not prevent this because the over-resolution happened before the guard + +--- + +## Key Findings + +1. **The strengthening defect persists through commit 3e78d57.** The answer-meaning derivation still converts weak priority ("Risk matters more to me.") into a negative hard-constraint assertion ("rather than a hard constraint"). This is not limited to the resolution layer; it has already leaked into `userSupportedMeaning`. + +2. **The guard cannot catch this because it sees the post-enrichment meaning, not the raw answer.** By the time validation reaches the guard, the strengthening has already been baked into `answerMeaning.userSupportedMeaning`. + +3. **Run-to-run variation in inference field.** Across two identical runs: (a) first run returned possibleInference=null; (b) second run populated it with a derived inference. Both contained the over-resolution in userSupportedMeaning. The enrichment is unstable across runs for the weak-priority case. + +4. **No emergent unknown created.** Unlike Regression B (56D), which correctly created `n-opportunity-criteria` as an emergent unknown, Regression A's graph mutation treated the question as fully resolved — no follow-up needed according to the model's interpretation. This is incorrect: the hard-constraint distinction should remain open. + +--- + +## What remains untested + +- Whether separating userSupportedMeaning from inference (as attempted in 55D) actually prevents this strengthening when the contract is enforced end-to-end +- Whether the fix from 36faf70 (conditional_qualification normalisation) or 3e78d57 (negation/qualification refinement) addresses weak-priority specifically +- Whether adding a post-guard verification layer that compares `userSupportedMeaning` against the original answer text can catch this class of over-resolution + +--- + +## Configuration + +- **Host:** http://192.168.1.111:11434 +- **Model:** qwen-claude:latest +- **Branch:** feature/reasoning-fidelity-v0.8 +- **Production code changed:** NO +- **Temporary instrumentation:** minimal Node script only — removed after capture +