diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 192ea37..4aea1c1 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -173,3 +173,7 @@ Isolated the answer-resolution step using one fixed target (preference/trade-off ### Experiment 56A Summary — Regression B Proposal Validation Enum Mismatch The first implementation pass added proposal-level `answerMeaning` with a pre-mutation compatibility guard. Deterministic regression tests A-D passed, but live Ollama runs showed Regression B failing at `proposal_validation` before the pre-mutation guard could execute. Experiment 56A traced this to a schema mismatch: Qwen returned `supportCategory: "conditional_qualification"` while the production Zod schema only accepts `conditional_tradeoff` among five values. The value survives normalization unchanged (normalize step handles node kind aliases, not supportCategory). The failure is at Zod validation — a proposal-contract issue, not a guard failure. **Hypothesis confirmed.** No fix was attempted. Branch: `feature/reasoning-fidelity-v0.8`. First file to inspect when resuming: `lib/graph/schema.js` line 165 (Zod enum for supportCategory) or the experiment record at `docs/experiment-56a.md`. Status pending Rob's review. + +### Experiment 56B Summary — Regression B Live Run After Normalisation + +Commit `36faf70` added normalization for `conditional_qualification → conditional_tradeoff`, but a live Regression B run returned a *different* variant: `supportCategory: "conditional_preference"`. The existing normalisation map does not cover this value. Two independent Zod errors occurred: (1) `conditional_preference` not in the supportCategory enum, and (2) `resolutionGuidance` was free-text instead of an enum value. **Run-to-run model variation confirmed** — the same fixed input produced `conditional_qualification` in Ex 56A and `conditional_preference` in Ex 56B. The pre-mutation guard remains unreachable because proposal_validation rejects first. Failure classification: `FAIL — normalization / proposal contract`. Branch: `feature/reasoning-fidelity-v0.8`. File to inspect when resuming: `docs/experiment-56b.md`. Status pending Rob's review. diff --git a/docs/experiment-56b.md b/docs/experiment-56b.md new file mode 100644 index 0000000..5948431 --- /dev/null +++ b/docs/experiment-56b.md @@ -0,0 +1,103 @@ +# Experiment 56B — Regression B Live Run After Normalisation + +**Date:** 2026-08-09 +**Branch:** `feature/reasoning-fidelity-v0.8` +**Status:** observation complete, no fix attempted + +## Hypothesis + +Regression B passes proposal validation after the normalisation added in commit `36faf70`, reaches the pre-mutation guard in `applyValidatedProposal()`, and preserves its conditional meaning through the graph outcome. + +## Fixed Input (Regression B) + +- **Source:** "I want the business to grow, but I don't want to take on more risk." +- **Answer:** "I'd normally avoid more risk, but for the right opportunity I might accept some." +- **Graph state:** Single unknown node `n-risk-constraint` (status: unknown) +- **Previous question:** "Is avoiding additional risk a hard constraint or a preference/trade-off?" + +## Configuration + +- **Ollama endpoint:** `http://192.168.1.111:11434` (from `.env.local`) +- **Model:** `qwen-claude:latest` + +## Observations + +### 1. Raw answerMeaning + +Inferred from Zod rejection errors (the model did not produce a validated proposal): + +- `supportCategory`: `"conditional_preference"` +- `resolutionGuidance`: `"Identify and quantify the threshold conditions that trigger risk acceptance."` (free-text string, not an enum value) + +### 2. Raw supportCategory at schema boundary + +**Observed value:** `conditional_preference` + +### 3. Normalised supportCategory + +**Result:** Unchanged — `conditional_preference` + +The normalisation map in `update-proposal.js` line 15 contains only: + +```js +const ANSWER_SUPPORT_CATEGORY_ALIASES = { + conditional_qualification: "conditional_tradeoff", +}; +``` + +It does **not** handle `conditional_preference`. The value passes through normalization untouched to Zod validation. + +### 4. Proposal validation result + +**FAILED — two errors:** + +1. **supportCategory:** `"conditional_preference"` is not in the Zod enum (`relative_priority_only | conditional_tradeoff | uncertain | explicit_hard_constraint | other`) +2. **resolutionGuidance:** Free-text string `"Identify and quantify the threshold conditions that trigger risk acceptance."` is not in the Zod enum (`must_remain_unresolved | may_resolve | must_resolve`) + +### 5. applyValidatedProposal reached? + +**NO.** The failure occurs at `proposal_validation` stage, before the pre-mutation guard in `applyValidatedProposal()` can execute. + +### 6. Guard result + +Not applicable — never reached. + +### 7. Resolution/update intent + +The model's free-text `resolutionGuidance` (`"Identify and quantify the threshold conditions that trigger risk acceptance."`) indicates it was attempting to produce conditional-resolution guidance, but failed the enum contract entirely. + +### 8. Final graph state + +**No mutation.** The proposal was rejected at validation; the SituationGraph remains unchanged (still contains `n-risk-constraint` with status `unknown`). + +## Additional Finding — Run-to-Run Model Variation + +Experiment 56A observed `supportCategory: "conditional_qualification"`. Experiment 56B observed `supportCategory: "conditional_preference"`. The same fixed input and model produce different category strings across runs. This means the normalisation map is incomplete by definition — no finite alias list can cover all possible model-generated variants. + +The two observations confirm the same root cause (model returns a non-enum supportCategory string) but with different values, reinforcing that this is an instability in the model's output contract compliance. + +## Result + +**FAIL — normalization / proposal contract** + +The hypothesis is not confirmed. Regression B fails at `proposal_validation` for the same class of defect as Experiment 56A (non-enum supportCategory), but with a *different* invalid value (`conditional_preference` instead of `conditional_qualification`). The existing normalisation map does not cover this variant. + +## What This Established + +1. Run-to-run model variation confirmed: `conditional_qualification` → `conditional_preference`. +2. The normalisation alias list (`ANSWER_SUPPORT_CATEGORY_ALIASES`) is insufficient — it only covers one of at least two observed variants. +3. The pre-mutation guard in `applyValidatedProposal()` remains unreachable because proposal_validation rejects first. +4. Even if the normalisation map were extended to cover `conditional_preference → conditional_tradeoff`, the `resolutionGuidance` field also failed (free-text instead of enum), indicating a second independent compliance gap. + +## What Remains Untested + +- Cases A, C, D, E, F +- Whether the model will consistently return one variant vs the other under repeated identical input +- The pre-mutation guard behaviour once a proposal successfully passes validation +- Downstream graph mutation consequences +- Other models' compliance with the answerMeaning output contract + +## Production reasoning code changed: NO +## Temporary instrumentation removed: YES +## Documentation updated: experiment-56b.md, current-handoff.md +## Git status: clean (pending commit)