experiment: isolate regression B proposal validation

2026-08-09 07:55:11 +01:00
parent 0d7ad5775c
commit b2329d8608
2 changed files with 99 additions and 0 deletions
@@ -0,0 +1,95 @@
+# Experiment 56A — Regression B Proposal Validation Enum Mismatch
+
+**Date:** 2026-08-09  
+**Branch:** `feature/reasoning-fidelity-v0.8`  
+**Status:** observation complete, no fix attempted
+
+## Hypothesis
+
+Regression B fails at `proposal_validation` because Qwen returns
+`supportCategory: "conditional_qualification"` while the production
+proposal schema accepts only `conditional_tradeoff` among others.
+
+This is a proposal-contract mismatch — not a pre-mutation guard failure.
+
+## Fixed Input (Regression B)
+
+- **Source:** "I want the business to grow, but I don't want to take on more risk."
+- **Answer:** "I'd normally avoid more risk, but for the right opportunity I might accept some."
+- **Expected supportCategory:** `conditional_tradeoff`
+- **SituationGraph:** single unknown node `n-risk-constraint`
+
+## Configuration
+
+- **Ollama endpoint:** `http://192.168.1.111:11434` (from `.env.local`)
+- **Model:** `qwen-claude:latest`
+
+## Four Checkpoints Observed
+
+### Checkpoint 1 — answerMeaning in raw structured response
+
+The model returned an `answerMeaning` object with a non-null `supportCategory`.  
+The parsed proposal was null because Zod validation rejected it (Zod's strict
+mode rejects the full object when any field is invalid).
+
+### Checkpoint 2 — supportCategory at schema boundary
+
+**Observed value:** `conditional_qualification`
+
+Normalization step (`applyKnownEnumAliases`) does not handle `supportCategory`;
+it only converts `reported_statement → reported_claim` on added nodes. The value
+survives unchanged to Zod validation.
+
+### Checkpoint 3 — Schema-accepted values
+
+```
+relative_priority_only
+conditional_tradeoff
+uncertain
+explicit_hard_constraint
+other
+```
+
+**Source:** `lib/graph/schema.js`, lines 147–152 (answerSupportCategory enum).
+
+`conditional_qualification` is NOT in this list.
+
+### Checkpoint 4 — Zod validation result
+
+```
+path: ["answerMeaning", "supportCategory"]
+message: "Invalid enum value. Expected 'relative_priority_only' | 'conditional_tradeoff' | 'uncertain' | 'explicit_hard_constraint' | 'other', received 'conditional_qualification'"
+code: invalid_enum_value
+stage: proposal_validation
+```
+
+## Result
+
+**Hypothesis confirmed: YES**
+
+1. Provider output contains `conditional_qualification` — confirmed via Zod error message.
+2. Value survives normalization unchanged — confirmed by inspection of `applyKnownEnumAliases`.
+3. Schema does not accept it — confirmed (not in the enum).
+4. Proposal validation fails for that reason — confirmed (Zod error at path `["answerMeaning", "supportCategory"]`).
+
+## What Was Not Done
+
+- No production code was changed.
+- No fix was attempted.
+- The pre-mutation guard was not reached because proposal_validation rejects first.
+- Cases A, C, D, E, F were not tested.
+- This experiment tested only ONE call; model output may vary across runs.
+
+## Files
+
+- Read: `lib/graph/schema.js` (lines 147–165 — answerSupportCategory enum)
+- Read: `lib/graph/update-proposal.js` (full file — normalization functions)
+- Read: `lib/llm/provider.js` (full file — Ollama provider)
+- Read: `lib/graph/orchestrator.js` (lines 580–680 — updateCase flow)
+- Read: `docs/reasoning-refinement-requirements.md` (Regression B section)
+- Read: `tests/graph/regression-a-d-v0.8.test.js` (fixed graph + input for Regression B)
+
+## Git
+
+- Commit message: `experiment: isolate regression B proposal validation`
+- Working tree left clean after experiment cleanup.