robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: validate semantic compatibility live

Browse Source
This commit is contained in:
robbond
2026-08-10 08:46:31 +01:00
parent 69efc5d1b9
commit 1a31949a48
2 changed files with 140 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+4
View File
@@ -324,3 +324,7 @@ The first post-v0.8 product observation exposed one valid production defect: a l
### Experiment 57F — Decomposition Relevance Fix Live Validation (BLOCKED)
Tested whether v0.11 (`7e4c506` — prevent unsupported comparison decomposition) works on the live relocation scenario from Experiments 57E/57B. **Classification: BLOCKED.** Start returned HTTP 200 with a valid graph. Update 1 failed at `proposal_compatibility` stage with HTTP 422: "Proposal cannot resolve beyond an unclassified answer by introducing an unsupported constraint or preference/trade-off distinction." The semantic grounding gate (`validateAnswerMeaningAlignment()`) blocked cost-reduction answers classified as class "other" before any graph update could occur. No decomposition ran — prohibited children ("two observations", "measured", "different timing") cannot be confirmed absent. **New defect exposed:** the semantic grounding gate rejects legitimate cost-reduction/savings answers that fall into unclassified ("other") rather than protected categories, preventing any Update 1 evaluation of the v0.11 fix. This is a separate blocking issue from decomposition relevance. What remains unproven: whether v0.11 works when Update 1 does succeed (answers in supported classes), and whether it correctly allows appropriate comparison/measurement decomposition for genuine parents. Branch: `feature/decomposition-relevance-v0.11`. Status blocked by semantic grounding gate.
### Experiment 57G — Semantic Compatibility Live Validation
Tested whether the cost-reduction answer (unclassified "other") now passes proposal compatibility through v0.12 (`69efc5d`). **Classification: PASS.** Both invocations returned HTTP 200 at `update_applied` — no more `proposal_compatibility` rejection. The selected next question ("What changed during that period...") is grounded in the relocation scenario (reasoning pattern: explanation, strategy: evidence_gathering). No prohibited decomposition children appeared. The v0.11 decomposition fix held on this successful update. **Cold-start observation:** second invocation started with 5 nodes instead of expected 8+ — inconsistent initial graph construction worth investigating separately. Branch: `feature/semantic-compatibility-v0.12`. Status: PASS — semantic compatibility blocker removed, decomposition regression absent. Full results in `docs/experiment-57g.md`.