104 lines
4.9 KiB
Markdown
104 lines
4.9 KiB
Markdown
# Experiment 56B — Regression B Live Run After Normalisation
|
|
|
|
**Date:** 2026-08-09
|
|
**Branch:** `feature/reasoning-fidelity-v0.8`
|
|
**Status:** observation complete, no fix attempted
|
|
|
|
## Hypothesis
|
|
|
|
Regression B passes proposal validation after the normalisation added in commit `36faf70`, reaches the pre-mutation guard in `applyValidatedProposal()`, and preserves its conditional meaning through the graph outcome.
|
|
|
|
## Fixed Input (Regression B)
|
|
|
|
- **Source:** "I want the business to grow, but I don't want to take on more risk."
|
|
- **Answer:** "I'd normally avoid more risk, but for the right opportunity I might accept some."
|
|
- **Graph state:** Single unknown node `n-risk-constraint` (status: unknown)
|
|
- **Previous question:** "Is avoiding additional risk a hard constraint or a preference/trade-off?"
|
|
|
|
## Configuration
|
|
|
|
- **Ollama endpoint:** `http://192.168.1.111:11434` (from `.env.local`)
|
|
- **Model:** `qwen-claude:latest`
|
|
|
|
## Observations
|
|
|
|
### 1. Raw answerMeaning
|
|
|
|
Inferred from Zod rejection errors (the model did not produce a validated proposal):
|
|
|
|
- `supportCategory`: `"conditional_preference"`
|
|
- `resolutionGuidance`: `"Identify and quantify the threshold conditions that trigger risk acceptance."` (free-text string, not an enum value)
|
|
|
|
### 2. Raw supportCategory at schema boundary
|
|
|
|
**Observed value:** `conditional_preference`
|
|
|
|
### 3. Normalised supportCategory
|
|
|
|
**Result:** Unchanged — `conditional_preference`
|
|
|
|
The normalisation map in `update-proposal.js` line 15 contains only:
|
|
|
|
```js
|
|
const ANSWER_SUPPORT_CATEGORY_ALIASES = {
|
|
conditional_qualification: "conditional_tradeoff",
|
|
};
|
|
```
|
|
|
|
It does **not** handle `conditional_preference`. The value passes through normalization untouched to Zod validation.
|
|
|
|
### 4. Proposal validation result
|
|
|
|
**FAILED — two errors:**
|
|
|
|
1. **supportCategory:** `"conditional_preference"` is not in the Zod enum (`relative_priority_only | conditional_tradeoff | uncertain | explicit_hard_constraint | other`)
|
|
2. **resolutionGuidance:** Free-text string `"Identify and quantify the threshold conditions that trigger risk acceptance."` is not in the Zod enum (`must_remain_unresolved | may_resolve | must_resolve`)
|
|
|
|
### 5. applyValidatedProposal reached?
|
|
|
|
**NO.** The failure occurs at `proposal_validation` stage, before the pre-mutation guard in `applyValidatedProposal()` can execute.
|
|
|
|
### 6. Guard result
|
|
|
|
Not applicable — never reached.
|
|
|
|
### 7. Resolution/update intent
|
|
|
|
The model's free-text `resolutionGuidance` (`"Identify and quantify the threshold conditions that trigger risk acceptance."`) indicates it was attempting to produce conditional-resolution guidance, but failed the enum contract entirely.
|
|
|
|
### 8. Final graph state
|
|
|
|
**No mutation.** The proposal was rejected at validation; the SituationGraph remains unchanged (still contains `n-risk-constraint` with status `unknown`).
|
|
|
|
## Additional Finding — Run-to-Run Model Variation
|
|
|
|
Experiment 56A observed `supportCategory: "conditional_qualification"`. Experiment 56B observed `supportCategory: "conditional_preference"`. The same fixed input and model produce different category strings across runs. This means the normalisation map is incomplete by definition — no finite alias list can cover all possible model-generated variants.
|
|
|
|
The two observations confirm the same root cause (model returns a non-enum supportCategory string) but with different values, reinforcing that this is an instability in the model's output contract compliance.
|
|
|
|
## Result
|
|
|
|
**FAIL — normalization / proposal contract**
|
|
|
|
The hypothesis is not confirmed. Regression B fails at `proposal_validation` for the same class of defect as Experiment 56A (non-enum supportCategory), but with a *different* invalid value (`conditional_preference` instead of `conditional_qualification`). The existing normalisation map does not cover this variant.
|
|
|
|
## What This Established
|
|
|
|
1. Run-to-run model variation confirmed: `conditional_qualification` → `conditional_preference`.
|
|
2. The normalisation alias list (`ANSWER_SUPPORT_CATEGORY_ALIASES`) is insufficient — it only covers one of at least two observed variants.
|
|
3. The pre-mutation guard in `applyValidatedProposal()` remains unreachable because proposal_validation rejects first.
|
|
4. Even if the normalisation map were extended to cover `conditional_preference → conditional_tradeoff`, the `resolutionGuidance` field also failed (free-text instead of enum), indicating a second independent compliance gap.
|
|
|
|
## What Remains Untested
|
|
|
|
- Cases A, C, D, E, F
|
|
- Whether the model will consistently return one variant vs the other under repeated identical input
|
|
- The pre-mutation guard behaviour once a proposal successfully passes validation
|
|
- Downstream graph mutation consequences
|
|
- Other models' compliance with the answerMeaning output contract
|
|
|
|
## Production reasoning code changed: NO
|
|
## Temporary instrumentation removed: YES
|
|
## Documentation updated: experiment-56b.md, current-handoff.md
|
|
## Git status: clean (pending commit)
|