docs: record contaminated experiment 57a observation
This commit is contained in:
@@ -272,3 +272,7 @@ The two important production capabilities now present are:
|
||||
- end-to-end interaction behaviour across graph update, question choice, Behaviour Selection and UI;
|
||||
- multilingual robustness;
|
||||
- any future defect exposed by real use.
|
||||
|
||||
### Experiment 57A — Contaminated After First Valid Observation
|
||||
|
||||
The first post-v0.8 product observation exposed one valid production defect: a legitimate affirmative decision-advancing answer (e.g., "We want cost reduction") falls into `other` and is blocked from resolving an unknown by `validateAnswerMeaningAlignment()`. The observation run became contaminated when Claude modified production code (`lib/graph/apply-proposal.js`, `lib/graph/schema.js`) adding four new categories and keyword detectors. Contaminated changes were reverted; valid defect remains open for a bounded implementation review. Repository production state is back at the merged v0.8 baseline (commit `14d68f1`). Next implementation question: how to prevent the fidelity guard from blocking legitimate answers outside its protected cases **without adding a growing answer-category taxonomy**.
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
# Experiment 57A — Contaminated / Aborted
|
||||
|
||||
**Status:** ABORTED / CONTAMINATED AFTER FIRST VALID OBSERVATION
|
||||
|
||||
**Baseline:** `14d68f1` (merged v0.8 first pass)
|
||||
|
||||
**Branch:** `main`
|
||||
|
||||
## Summary
|
||||
|
||||
Experiment 57A exposed one valid production defect but the observation run was contaminated after Claude modified production code (`lib/graph/apply-proposal.js`, `lib/graph/schema.js`). The contaminating changes added four new answer-meaning categories and keyword-based detectors, then widened `validateAnswerMeaningAlignment()` to allow resolution for those categories.
|
||||
|
||||
Contaminated changes were reverted to HEAD. Repository production state is restored to the merged v0.8 baseline (`14d68f1`).
|
||||
|
||||
## Valid Observation (preserved)
|
||||
|
||||
> An ordinary decision-advancing answer such as `"We want cost reduction"` can fall into the existing `other` answer-meaning category and then be rejected by `validateAnswerMeaningAlignment()`, preventing a legitimate unknown resolution.
|
||||
|
||||
**Failure boundary:**
|
||||
- The raw answer itself is not inherently ambiguous — it conveys a clear affirmative stance advancing the decision.
|
||||
- The problem is that the fidelity safeguard's protected-category logic is over-restrictive for valid answers outside the original A-D meaning cases.
|
||||
- `other` currently acts as a rejection category for resolution, blocking legitimate unknowns that the user's answer actually advances.
|
||||
|
||||
## Attempted Four-Category Fix — DISCARDED
|
||||
|
||||
The following changes were made during 57A and **must not** be preserved:
|
||||
|
||||
- New categories added to `answerSupportCategory`: `supports_decision`, `contradicts_decision`, `conditional_support`, `strong_preference`
|
||||
- Keyword-based detectors for each new category (`mentionsSupportiveStance`, `mentionsFactualEvidence`, `mentionsContradictoryStance`)
|
||||
- Widened `validateAnswerMeaningAlignment()` to allow resolution for these four categories
|
||||
|
||||
**Reason discarded:** This widened the semantic taxonomy beyond what a single observed failure case warrants and reintroduced brittle closed-vocabulary / keyword-classification risk. The fix addressed symptoms, not the underlying boundary definition problem.
|
||||
|
||||
## Observations NOT established by 57A
|
||||
|
||||
These were explored during contamination but are **NOT established defects** and must not be treated as current findings:
|
||||
|
||||
- **Question explosion** — not established; may be investigated later if cleanly reproduced.
|
||||
- **Wrong initial question selection** — not established; may be investigated later if cleanly reproduced.
|
||||
- **Graph nondeterminism / variation across repeated starts** — not established; may be investigated later if cleanly reproduced.
|
||||
|
||||
## Open Implementation Question
|
||||
|
||||
How to prevent the fidelity guard from blocking legitimate answers outside its protected cases **without adding a growing answer-category taxonomy**.
|
||||
|
||||
## Validation
|
||||
|
||||
Baseline tests restored and passing: `npx vitest run tests/graph/apply-proposal.test.js tests/graph/schema.test.js` — 92/92 passed.
|
||||
Reference in New Issue
Block a user