diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 3d82bef..192ea37 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -169,3 +169,7 @@ The first implementation pass against the reasoning refinement requirements is d ### Experiment 55A Summary — Clarification Uncertainty Preservation Isolated the answer-resolution step using one fixed target (preference/trade-off or hard constraint) and four answers of different strength: fully explicit, weak priority, conditional trade-off, non-answer. Four live Ollama calls completed at http://192.168.1.111:11434 with qwen-claude:latest (~62s total). Case 1 (explicit hard constraint) resolved correctly. Case 2 (weak priority — "Risk matters more to me.") over-resolved: the model set targetResolved=true and inferred "not a rigid, non-negotiable constraint" — meaning stronger than the user supplied. Case 3 (conditional trade-off) resolved correctly on the target but flattened conditionality into flat "preference or trade-off" language without preserving the conditional qualification ("might accept some"). Case 4 (non-answer) correctly remained unresolved with appropriate remaining uncertainty. Two of the four tested answers showed loss of nuance: one was over-resolved and one retained the correct target category while losing conditional qualification. The same over-resolution reproduced with a fixed target, so target broadening is not required for the failure to occur. Broader generalisation across other models and answers remains untested. Behaviour Selection, graph, UI, and production integration remain untouched. Branch: feature/user-workspace-ux-v0.7. First test/file to inspect when resuming: tests/reconstruction/semantic-clarification-uncertainty-preservation.test.js for the full experiment and results. Status pending Rob's review. + +### Experiment 56A Summary — Regression B Proposal Validation Enum Mismatch + +The first implementation pass added proposal-level `answerMeaning` with a pre-mutation compatibility guard. Deterministic regression tests A-D passed, but live Ollama runs showed Regression B failing at `proposal_validation` before the pre-mutation guard could execute. Experiment 56A traced this to a schema mismatch: Qwen returned `supportCategory: "conditional_qualification"` while the production Zod schema only accepts `conditional_tradeoff` among five values. The value survives normalization unchanged (normalize step handles node kind aliases, not supportCategory). The failure is at Zod validation — a proposal-contract issue, not a guard failure. **Hypothesis confirmed.** No fix was attempted. Branch: `feature/reasoning-fidelity-v0.8`. First file to inspect when resuming: `lib/graph/schema.js` line 165 (Zod enum for supportCategory) or the experiment record at `docs/experiment-56a.md`. Status pending Rob's review. diff --git a/docs/experiment-56a.md b/docs/experiment-56a.md new file mode 100644 index 0000000..7da8437 --- /dev/null +++ b/docs/experiment-56a.md @@ -0,0 +1,95 @@ +# Experiment 56A — Regression B Proposal Validation Enum Mismatch + +**Date:** 2026-08-09 +**Branch:** `feature/reasoning-fidelity-v0.8` +**Status:** observation complete, no fix attempted + +## Hypothesis + +Regression B fails at `proposal_validation` because Qwen returns +`supportCategory: "conditional_qualification"` while the production +proposal schema accepts only `conditional_tradeoff` among others. + +This is a proposal-contract mismatch — not a pre-mutation guard failure. + +## Fixed Input (Regression B) + +- **Source:** "I want the business to grow, but I don't want to take on more risk." +- **Answer:** "I'd normally avoid more risk, but for the right opportunity I might accept some." +- **Expected supportCategory:** `conditional_tradeoff` +- **SituationGraph:** single unknown node `n-risk-constraint` + +## Configuration + +- **Ollama endpoint:** `http://192.168.1.111:11434` (from `.env.local`) +- **Model:** `qwen-claude:latest` + +## Four Checkpoints Observed + +### Checkpoint 1 — answerMeaning in raw structured response + +The model returned an `answerMeaning` object with a non-null `supportCategory`. +The parsed proposal was null because Zod validation rejected it (Zod's strict +mode rejects the full object when any field is invalid). + +### Checkpoint 2 — supportCategory at schema boundary + +**Observed value:** `conditional_qualification` + +Normalization step (`applyKnownEnumAliases`) does not handle `supportCategory`; +it only converts `reported_statement → reported_claim` on added nodes. The value +survives unchanged to Zod validation. + +### Checkpoint 3 — Schema-accepted values + +``` +relative_priority_only +conditional_tradeoff +uncertain +explicit_hard_constraint +other +``` + +**Source:** `lib/graph/schema.js`, lines 147–152 (answerSupportCategory enum). + +`conditional_qualification` is NOT in this list. + +### Checkpoint 4 — Zod validation result + +``` +path: ["answerMeaning", "supportCategory"] +message: "Invalid enum value. Expected 'relative_priority_only' | 'conditional_tradeoff' | 'uncertain' | 'explicit_hard_constraint' | 'other', received 'conditional_qualification'" +code: invalid_enum_value +stage: proposal_validation +``` + +## Result + +**Hypothesis confirmed: YES** + +1. Provider output contains `conditional_qualification` — confirmed via Zod error message. +2. Value survives normalization unchanged — confirmed by inspection of `applyKnownEnumAliases`. +3. Schema does not accept it — confirmed (not in the enum). +4. Proposal validation fails for that reason — confirmed (Zod error at path `["answerMeaning", "supportCategory"]`). + +## What Was Not Done + +- No production code was changed. +- No fix was attempted. +- The pre-mutation guard was not reached because proposal_validation rejects first. +- Cases A, C, D, E, F were not tested. +- This experiment tested only ONE call; model output may vary across runs. + +## Files + +- Read: `lib/graph/schema.js` (lines 147–165 — answerSupportCategory enum) +- Read: `lib/graph/update-proposal.js` (full file — normalization functions) +- Read: `lib/llm/provider.js` (full file — Ollama provider) +- Read: `lib/graph/orchestrator.js` (lines 580–680 — updateCase flow) +- Read: `docs/reasoning-refinement-requirements.md` (Regression B section) +- Read: `tests/graph/regression-a-d-v0.8.test.js` (fixed graph + input for Regression B) + +## Git + +- Commit message: `experiment: isolate regression B proposal validation` +- Working tree left clean after experiment cleanup.