robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: validate grounded unclassified answer live

Browse Source
This commit is contained in:
robbond
2026-08-09 20:03:30 +01:00
parent 4e4d0fa732
commit 19a42ca7f7
3 changed files with 175 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+4
View File
@@ -273,6 +273,10 @@ The two important production capabilities now present are:
- multilingual robustness;
- any future defect exposed by real use.
### Experiment 57B — Grounded Unclassified Affirmative Answer (Live)
Validated one live semantic probe: the configured Qwen model (`qwen-claude:latest` at `http://192.168.1.111:11434`) produced grounded user-supported meaning for a legitimate affirmative answer that falls into `other` (unclassified) — exactly the case blocked by v0.8's guard. **Result: PASS.** `userSupportedMeaning`: "The user directly states that cost reduction, specifically approximately £2M in annual office-overhead savings, is a primary stated reason for considering the relocation." No unsupported strengthening beyond the raw answer. One live call (20,351 ms). This confirms the live semantic premise behind Codex `4e4d0fa` (semantic token overlap grounding). **What remains unproven:** token-overlap adequacy across diverse answers, stability across runs, production guard integration. Full results in `docs/experiment-57b.md`. Status pending Rob's review.
### Experiment 57A — Contaminated After First Valid Observation
The first post-v0.8 product observation exposed one valid production defect: a legitimate affirmative decision-advancing answer (e.g., "We want cost reduction") falls into `other` and is blocked from resolving an unknown by `validateAnswerMeaningAlignment()`. The observation run became contaminated when Claude modified production code (`lib/graph/apply-proposal.js`, `lib/graph/schema.js`) adding four new categories and keyword detectors. Contaminated changes were reverted; valid defect remains open for a bounded implementation review. Repository production state is back at the merged v0.8 baseline (commit `14d68f1`). Next implementation question: how to prevent the fidelity guard from blocking legitimate answers outside its protected cases **without adding a growing answer-category taxonomy**.