confidence-engine/docs/experiment-57a.md

Experiment 57A — Contaminated / Aborted

Status: ABORTED / CONTAMINATED AFTER FIRST VALID OBSERVATION

Baseline: 14d68f1 (merged v0.8 first pass)

Branch: main

Summary

Experiment 57A exposed one valid production defect but the observation run was contaminated after Claude modified production code (lib/graph/apply-proposal.js, lib/graph/schema.js). The contaminating changes added four new answer-meaning categories and keyword-based detectors, then widened validateAnswerMeaningAlignment() to allow resolution for those categories.

Contaminated changes were reverted to HEAD. Repository production state is restored to the merged v0.8 baseline (14d68f1).

Valid Observation (preserved)

An ordinary decision-advancing answer such as "We want cost reduction" can fall into the existing other answer-meaning category and then be rejected by validateAnswerMeaningAlignment(), preventing a legitimate unknown resolution.

Failure boundary:

• The raw answer itself is not inherently ambiguous — it conveys a clear affirmative stance advancing the decision.
• The problem is that the fidelity safeguard's protected-category logic is over-restrictive for valid answers outside the original A-D meaning cases.
• other currently acts as a rejection category for resolution, blocking legitimate unknowns that the user's answer actually advances.

Attempted Four-Category Fix — DISCARDED

The following changes were made during 57A and must not be preserved:

• New categories added to answerSupportCategory: supports_decision, contradicts_decision, conditional_support, strong_preference
• Keyword-based detectors for each new category (mentionsSupportiveStance, mentionsFactualEvidence, mentionsContradictoryStance)
• Widened validateAnswerMeaningAlignment() to allow resolution for these four categories

Reason discarded: This widened the semantic taxonomy beyond what a single observed failure case warrants and reintroduced brittle closed-vocabulary / keyword-classification risk. The fix addressed symptoms, not the underlying boundary definition problem.

Observations NOT established by 57A

These were explored during contamination but are NOT established defects and must not be treated as current findings:

• Question explosion — not established; may be investigated later if cleanly reproduced.
• Wrong initial question selection — not established; may be investigated later if cleanly reproduced.
• Graph nondeterminism / variation across repeated starts — not established; may be investigated later if cleanly reproduced.

Open Implementation Question

How to prevent the fidelity guard from blocking legitimate answers outside its protected cases without adding a growing answer-category taxonomy.

Validation

Baseline tests restored and passing: npx vitest run tests/graph/apply-proposal.test.js tests/graph/schema.test.js — 92/92 passed.