49 lines
2.9 KiB
Markdown
49 lines
2.9 KiB
Markdown
# Experiment 57A — Contaminated / Aborted
|
|
|
|
**Status:** ABORTED / CONTAMINATED AFTER FIRST VALID OBSERVATION
|
|
|
|
**Baseline:** `14d68f1` (merged v0.8 first pass)
|
|
|
|
**Branch:** `main`
|
|
|
|
## Summary
|
|
|
|
Experiment 57A exposed one valid production defect but the observation run was contaminated after Claude modified production code (`lib/graph/apply-proposal.js`, `lib/graph/schema.js`). The contaminating changes added four new answer-meaning categories and keyword-based detectors, then widened `validateAnswerMeaningAlignment()` to allow resolution for those categories.
|
|
|
|
Contaminated changes were reverted to HEAD. Repository production state is restored to the merged v0.8 baseline (`14d68f1`).
|
|
|
|
## Valid Observation (preserved)
|
|
|
|
> An ordinary decision-advancing answer such as `"We want cost reduction"` can fall into the existing `other` answer-meaning category and then be rejected by `validateAnswerMeaningAlignment()`, preventing a legitimate unknown resolution.
|
|
|
|
**Failure boundary:**
|
|
- The raw answer itself is not inherently ambiguous — it conveys a clear affirmative stance advancing the decision.
|
|
- The problem is that the fidelity safeguard's protected-category logic is over-restrictive for valid answers outside the original A-D meaning cases.
|
|
- `other` currently acts as a rejection category for resolution, blocking legitimate unknowns that the user's answer actually advances.
|
|
|
|
## Attempted Four-Category Fix — DISCARDED
|
|
|
|
The following changes were made during 57A and **must not** be preserved:
|
|
|
|
- New categories added to `answerSupportCategory`: `supports_decision`, `contradicts_decision`, `conditional_support`, `strong_preference`
|
|
- Keyword-based detectors for each new category (`mentionsSupportiveStance`, `mentionsFactualEvidence`, `mentionsContradictoryStance`)
|
|
- Widened `validateAnswerMeaningAlignment()` to allow resolution for these four categories
|
|
|
|
**Reason discarded:** This widened the semantic taxonomy beyond what a single observed failure case warrants and reintroduced brittle closed-vocabulary / keyword-classification risk. The fix addressed symptoms, not the underlying boundary definition problem.
|
|
|
|
## Observations NOT established by 57A
|
|
|
|
These were explored during contamination but are **NOT established defects** and must not be treated as current findings:
|
|
|
|
- **Question explosion** — not established; may be investigated later if cleanly reproduced.
|
|
- **Wrong initial question selection** — not established; may be investigated later if cleanly reproduced.
|
|
- **Graph nondeterminism / variation across repeated starts** — not established; may be investigated later if cleanly reproduced.
|
|
|
|
## Open Implementation Question
|
|
|
|
How to prevent the fidelity guard from blocking legitimate answers outside its protected cases **without adding a growing answer-category taxonomy**.
|
|
|
|
## Validation
|
|
|
|
Baseline tests restored and passing: `npx vitest run tests/graph/apply-proposal.test.js tests/graph/schema.test.js` — 92/92 passed.
|