96 lines
3.4 KiB
Markdown
96 lines
3.4 KiB
Markdown
# Experiment 56A — Regression B Proposal Validation Enum Mismatch
|
||
|
||
**Date:** 2026-08-09
|
||
**Branch:** `feature/reasoning-fidelity-v0.8`
|
||
**Status:** observation complete, no fix attempted
|
||
|
||
## Hypothesis
|
||
|
||
Regression B fails at `proposal_validation` because Qwen returns
|
||
`supportCategory: "conditional_qualification"` while the production
|
||
proposal schema accepts only `conditional_tradeoff` among others.
|
||
|
||
This is a proposal-contract mismatch — not a pre-mutation guard failure.
|
||
|
||
## Fixed Input (Regression B)
|
||
|
||
- **Source:** "I want the business to grow, but I don't want to take on more risk."
|
||
- **Answer:** "I'd normally avoid more risk, but for the right opportunity I might accept some."
|
||
- **Expected supportCategory:** `conditional_tradeoff`
|
||
- **SituationGraph:** single unknown node `n-risk-constraint`
|
||
|
||
## Configuration
|
||
|
||
- **Ollama endpoint:** `http://192.168.1.111:11434` (from `.env.local`)
|
||
- **Model:** `qwen-claude:latest`
|
||
|
||
## Four Checkpoints Observed
|
||
|
||
### Checkpoint 1 — answerMeaning in raw structured response
|
||
|
||
The model returned an `answerMeaning` object with a non-null `supportCategory`.
|
||
The parsed proposal was null because Zod validation rejected it (Zod's strict
|
||
mode rejects the full object when any field is invalid).
|
||
|
||
### Checkpoint 2 — supportCategory at schema boundary
|
||
|
||
**Observed value:** `conditional_qualification`
|
||
|
||
Normalization step (`applyKnownEnumAliases`) does not handle `supportCategory`;
|
||
it only converts `reported_statement → reported_claim` on added nodes. The value
|
||
survives unchanged to Zod validation.
|
||
|
||
### Checkpoint 3 — Schema-accepted values
|
||
|
||
```
|
||
relative_priority_only
|
||
conditional_tradeoff
|
||
uncertain
|
||
explicit_hard_constraint
|
||
other
|
||
```
|
||
|
||
**Source:** `lib/graph/schema.js`, lines 147–152 (answerSupportCategory enum).
|
||
|
||
`conditional_qualification` is NOT in this list.
|
||
|
||
### Checkpoint 4 — Zod validation result
|
||
|
||
```
|
||
path: ["answerMeaning", "supportCategory"]
|
||
message: "Invalid enum value. Expected 'relative_priority_only' | 'conditional_tradeoff' | 'uncertain' | 'explicit_hard_constraint' | 'other', received 'conditional_qualification'"
|
||
code: invalid_enum_value
|
||
stage: proposal_validation
|
||
```
|
||
|
||
## Result
|
||
|
||
**Hypothesis confirmed: YES**
|
||
|
||
1. Provider output contains `conditional_qualification` — confirmed via Zod error message.
|
||
2. Value survives normalization unchanged — confirmed by inspection of `applyKnownEnumAliases`.
|
||
3. Schema does not accept it — confirmed (not in the enum).
|
||
4. Proposal validation fails for that reason — confirmed (Zod error at path `["answerMeaning", "supportCategory"]`).
|
||
|
||
## What Was Not Done
|
||
|
||
- No production code was changed.
|
||
- No fix was attempted.
|
||
- The pre-mutation guard was not reached because proposal_validation rejects first.
|
||
- Cases A, C, D, E, F were not tested.
|
||
- This experiment tested only ONE call; model output may vary across runs.
|
||
|
||
## Files
|
||
|
||
- Read: `lib/graph/schema.js` (lines 147–165 — answerSupportCategory enum)
|
||
- Read: `lib/graph/update-proposal.js` (full file — normalization functions)
|
||
- Read: `lib/llm/provider.js` (full file — Ollama provider)
|
||
- Read: `lib/graph/orchestrator.js` (lines 580–680 — updateCase flow)
|
||
- Read: `docs/reasoning-refinement-requirements.md` (Regression B section)
|
||
- Read: `tests/graph/regression-a-d-v0.8.test.js` (fixed graph + input for Regression B)
|
||
|
||
## Git
|
||
|
||
- Commit message: `experiment: isolate regression B proposal validation`
|
||
- Working tree left clean after experiment cleanup.
|