confidence-engine/docs/experiment-56b.md

Experiment 56B — Regression B Live Run After Normalisation

Date: 2026-08-09
Branch: feature/reasoning-fidelity-v0.8
Status: observation complete, no fix attempted

Hypothesis

Regression B passes proposal validation after the normalisation added in commit 36faf70, reaches the pre-mutation guard in applyValidatedProposal(), and preserves its conditional meaning through the graph outcome.

Fixed Input (Regression B)

• Source: "I want the business to grow, but I don't want to take on more risk."
• Answer: "I'd normally avoid more risk, but for the right opportunity I might accept some."
• Graph state: Single unknown node n-risk-constraint (status: unknown)
• Previous question: "Is avoiding additional risk a hard constraint or a preference/trade-off?"

Configuration

• Ollama endpoint: http://192.168.1.111:11434 (from .env.local)
• Model: qwen-claude:latest

Observations

1. Raw answerMeaning

Inferred from Zod rejection errors (the model did not produce a validated proposal):

• supportCategory: "conditional_preference"
• resolutionGuidance: "Identify and quantify the threshold conditions that trigger risk acceptance." (free-text string, not an enum value)

2. Raw supportCategory at schema boundary

Observed value: conditional_preference

3. Normalised supportCategory

Result: Unchanged — conditional_preference

The normalisation map in update-proposal.js line 15 contains only:

const ANSWER_SUPPORT_CATEGORY_ALIASES = {
  conditional_qualification: "conditional_tradeoff",
};

It does not handle conditional_preference. The value passes through normalization untouched to Zod validation.

4. Proposal validation result

FAILED — two errors:

1. supportCategory: "conditional_preference" is not in the Zod enum (relative_priority_only | conditional_tradeoff | uncertain | explicit_hard_constraint | other)
2. resolutionGuidance: Free-text string "Identify and quantify the threshold conditions that trigger risk acceptance." is not in the Zod enum (must_remain_unresolved | may_resolve | must_resolve)

5. applyValidatedProposal reached?

NO. The failure occurs at proposal_validation stage, before the pre-mutation guard in applyValidatedProposal() can execute.

6. Guard result

Not applicable — never reached.

7. Resolution/update intent

The model's free-text resolutionGuidance ("Identify and quantify the threshold conditions that trigger risk acceptance.") indicates it was attempting to produce conditional-resolution guidance, but failed the enum contract entirely.

8. Final graph state

No mutation. The proposal was rejected at validation; the SituationGraph remains unchanged (still contains n-risk-constraint with status unknown).

Additional Finding — Run-to-Run Model Variation

Experiment 56A observed supportCategory: "conditional_qualification". Experiment 56B observed supportCategory: "conditional_preference". The same fixed input and model produce different category strings across runs. This means the normalisation map is incomplete by definition — no finite alias list can cover all possible model-generated variants.

The two observations confirm the same root cause (model returns a non-enum supportCategory string) but with different values, reinforcing that this is an instability in the model's output contract compliance.

Result

FAIL — normalization / proposal contract

The hypothesis is not confirmed. Regression B fails at proposal_validation for the same class of defect as Experiment 56A (non-enum supportCategory), but with a different invalid value (conditional_preference instead of conditional_qualification). The existing normalisation map does not cover this variant.

What This Established

1. Run-to-run model variation confirmed: conditional_qualification → conditional_preference.
2. The normalisation alias list (ANSWER_SUPPORT_CATEGORY_ALIASES) is insufficient — it only covers one of at least two observed variants.
3. The pre-mutation guard in applyValidatedProposal() remains unreachable because proposal_validation rejects first.
4. Even if the normalisation map were extended to cover conditional_preference → conditional_tradeoff, the resolutionGuidance field also failed (free-text instead of enum), indicating a second independent compliance gap.

What Remains Untested

• Cases A, C, D, E, F
• Whether the model will consistently return one variant vs the other under repeated identical input
• The pre-mutation guard behaviour once a proposal successfully passes validation
• Downstream graph mutation consequences
• Other models' compliance with the answerMeaning output contract

Production reasoning code changed: NO

Temporary instrumentation removed: YES

Documentation updated: experiment-56b.md, current-handoff.md

Git status: clean (pending commit)