diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 31613a5..1bfcfcc 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -272,3 +272,7 @@ The two important production capabilities now present are: - end-to-end interaction behaviour across graph update, question choice, Behaviour Selection and UI; - multilingual robustness; - any future defect exposed by real use. + +### Experiment 57A — Contaminated After First Valid Observation + +The first post-v0.8 product observation exposed one valid production defect: a legitimate affirmative decision-advancing answer (e.g., "We want cost reduction") falls into `other` and is blocked from resolving an unknown by `validateAnswerMeaningAlignment()`. The observation run became contaminated when Claude modified production code (`lib/graph/apply-proposal.js`, `lib/graph/schema.js`) adding four new categories and keyword detectors. Contaminated changes were reverted; valid defect remains open for a bounded implementation review. Repository production state is back at the merged v0.8 baseline (commit `14d68f1`). Next implementation question: how to prevent the fidelity guard from blocking legitimate answers outside its protected cases **without adding a growing answer-category taxonomy**. diff --git a/docs/experiment-57a.md b/docs/experiment-57a.md new file mode 100644 index 0000000..457cd6d --- /dev/null +++ b/docs/experiment-57a.md @@ -0,0 +1,48 @@ +# Experiment 57A — Contaminated / Aborted + +**Status:** ABORTED / CONTAMINATED AFTER FIRST VALID OBSERVATION + +**Baseline:** `14d68f1` (merged v0.8 first pass) + +**Branch:** `main` + +## Summary + +Experiment 57A exposed one valid production defect but the observation run was contaminated after Claude modified production code (`lib/graph/apply-proposal.js`, `lib/graph/schema.js`). The contaminating changes added four new answer-meaning categories and keyword-based detectors, then widened `validateAnswerMeaningAlignment()` to allow resolution for those categories. + +Contaminated changes were reverted to HEAD. Repository production state is restored to the merged v0.8 baseline (`14d68f1`). + +## Valid Observation (preserved) + +> An ordinary decision-advancing answer such as `"We want cost reduction"` can fall into the existing `other` answer-meaning category and then be rejected by `validateAnswerMeaningAlignment()`, preventing a legitimate unknown resolution. + +**Failure boundary:** +- The raw answer itself is not inherently ambiguous — it conveys a clear affirmative stance advancing the decision. +- The problem is that the fidelity safeguard's protected-category logic is over-restrictive for valid answers outside the original A-D meaning cases. +- `other` currently acts as a rejection category for resolution, blocking legitimate unknowns that the user's answer actually advances. + +## Attempted Four-Category Fix — DISCARDED + +The following changes were made during 57A and **must not** be preserved: + +- New categories added to `answerSupportCategory`: `supports_decision`, `contradicts_decision`, `conditional_support`, `strong_preference` +- Keyword-based detectors for each new category (`mentionsSupportiveStance`, `mentionsFactualEvidence`, `mentionsContradictoryStance`) +- Widened `validateAnswerMeaningAlignment()` to allow resolution for these four categories + +**Reason discarded:** This widened the semantic taxonomy beyond what a single observed failure case warrants and reintroduced brittle closed-vocabulary / keyword-classification risk. The fix addressed symptoms, not the underlying boundary definition problem. + +## Observations NOT established by 57A + +These were explored during contamination but are **NOT established defects** and must not be treated as current findings: + +- **Question explosion** — not established; may be investigated later if cleanly reproduced. +- **Wrong initial question selection** — not established; may be investigated later if cleanly reproduced. +- **Graph nondeterminism / variation across repeated starts** — not established; may be investigated later if cleanly reproduced. + +## Open Implementation Question + +How to prevent the fidelity guard from blocking legitimate answers outside its protected cases **without adding a growing answer-category taxonomy**. + +## Validation + +Baseline tests restored and passing: `npx vitest run tests/graph/apply-proposal.test.js tests/graph/schema.test.js` — 92/92 passed.