From 4ea664d0d8c0d392a61dc4928ac5392615120cb6 Mon Sep 17 00:00:00 2001 From: robbond Date: Mon, 10 Aug 2026 08:04:59 +0100 Subject: [PATCH] experiment: validate decomposition relevance live --- docs/current-handoff.md | 4 ++ docs/experiment-57f.md | 91 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+) create mode 100644 docs/experiment-57f.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 1932f7a..3b099d3 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -320,3 +320,7 @@ Validated one live semantic probe: the configured Qwen model (`qwen-claude:lates ### Experiment 57A — Contaminated After First Valid Observation The first post-v0.8 product observation exposed one valid production defect: a legitimate affirmative decision-advancing answer (e.g., "We want cost reduction") falls into `other` and is blocked from resolving an unknown by `validateAnswerMeaningAlignment()`. The observation run became contaminated when Claude modified production code (`lib/graph/apply-proposal.js`, `lib/graph/schema.js`) adding four new categories and keyword detectors. Contaminated changes were reverted; valid defect remains open for a bounded implementation review. Repository production state is back at the merged v0.8 baseline (commit `14d68f1`). Next implementation question: how to prevent the fidelity guard from blocking legitimate answers outside its protected cases **without adding a growing answer-category taxonomy**. + +### Experiment 57F — Decomposition Relevance Fix Live Validation (BLOCKED) + +Tested whether v0.11 (`7e4c506` — prevent unsupported comparison decomposition) works on the live relocation scenario from Experiments 57E/57B. **Classification: BLOCKED.** Start returned HTTP 200 with a valid graph. Update 1 failed at `proposal_compatibility` stage with HTTP 422: "Proposal cannot resolve beyond an unclassified answer by introducing an unsupported constraint or preference/trade-off distinction." The semantic grounding gate (`validateAnswerMeaningAlignment()`) blocked cost-reduction answers classified as class "other" before any graph update could occur. No decomposition ran — prohibited children ("two observations", "measured", "different timing") cannot be confirmed absent. **New defect exposed:** the semantic grounding gate rejects legitimate cost-reduction/savings answers that fall into unclassified ("other") rather than protected categories, preventing any Update 1 evaluation of the v0.11 fix. This is a separate blocking issue from decomposition relevance. What remains unproven: whether v0.11 works when Update 1 does succeed (answers in supported classes), and whether it correctly allows appropriate comparison/measurement decomposition for genuine parents. Branch: `feature/decomposition-relevance-v0.11`. Status blocked by semantic grounding gate. diff --git a/docs/experiment-57f.md b/docs/experiment-57f.md new file mode 100644 index 0000000..237fe47 --- /dev/null +++ b/docs/experiment-57f.md @@ -0,0 +1,91 @@ +# Experiment 57F — Decomposition Relevance Fix Live Validation + +**Date:** 2026-08-10 +**Branch:** `feature/decomposition-relevance-v0.11` +**Codex refinement validated:** `7e4c506` — reasoning: prevent unsupported comparison decomposition +**Model:** qwen-claude:latest at http://192.168.1.111:11434 + +## Objective + +After v0.11, does Update 1 avoid manufacturing the unsupported "two observations / measured" decomposition and produce a next question grounded in the actual relocation investigation? + +This is observation-only validation. + +## Fixed inputs + +- **Scenario:** Should I relocate my engineering team from London to Manchester? +- **Answer 1:** We're looking at this mainly for cost reduction — roughly £2M annual savings on office overhead. +- **Answer 2:** NOT submitted (fixed budget: Start + Update 1 = 2 live calls) + +## Pre-written human expectation (recorded before run) + +> The engine must not generate or select an unsupported measurement/comparison unknown such as "How the two observations were measured" or "Whether the two observations reflect different timing" unless the live graph actually contains a parent that establishes a genuine comparison/measurement problem. For this relocation/cost-reduction turn, the next question should remain grounded in a real unresolved aspect of the relocation decision. A broad unresolved parent is preferable to an invented measurement question. +> +> Do not define in advance what the replacement question *must* be. + +## Canonical execution route + +- Dev server: `npx next dev` → http://localhost:3000 +- Start: `POST /api/cases/start` +- Update 1: `POST /api/cases/update` +- Script: `scripts/reproduce-multi-turn-investigation.mjs` (temporarily instrumented, then restored) + +## Live call budget + +- Start: 1 +- Update 1: 1 +- Update 2: 0 +- **Total:** 2 live Ollama calls + +## Results + +### Classification: BLOCKED + +### Production result + +- **HTTP status:** 422 (Unprocessable Entity) +- **Stage:** `proposal_compatibility` +- **Node count:** 8 (unchanged from start) +- **Edge count:** 4 (unchanged from start) +- **Selected question:** null (Update failed before selection) +- **Proposal validation error:** `"Proposal cannot resolve beyond an unclassified answer by introducing an unsupported constraint or preference/trade-off distinction."` + +### What happened + +1. Start returned HTTP 200 with a valid graph (8 nodes, 4 edges) and a selected question about the viability of the engineering team relocation. +2. Update 1 submitted Answer 1 (cost reduction / £2M savings). The LLM produced grounded `userSupportedMeaning` at the prompt level. However, the answer was classified as "other" (unclassified) rather than falling into any of the protected categories. The semantic grounding check in `validateAnswerMeaningAlignment()` rejected the proposal because it could not establish that the unclassified answer supports resolving any specific unknown. +3. The graph was NOT updated. No decomposition occurred. No new nodes were added. + +## What this experiment established + +- The v0.11 fix (`7e4c506`) cannot be evaluated in this run because Update 1 fails at the semantic grounding layer before decomposition can be reached. +- The `validateAnswerMeaningAlignment()` check (from the semantic grounding mechanism validated in Experiments 57A–57B) continues to block legitimate cost-reduction answers that land in class "other". +- No prohibited decomposition children ("two observations", "measured", "different timing") can be confirmed absent because no graph update occurred. +- The blocking error is **not** a decomposition defect — it is the pre-existing semantic grounding gate preventing unclassified answers from producing any proposal. + +## What this experiment does NOT prove + +- Whether the v0.11 decomposition relevance fix works when Update 1 *does* succeed (i.e., when the answer falls into a supported class). +- Whether the semantic grounding blocker is itself correct or over-aggressive for cost-reduction scenarios. +- Whether valid comparison/measurement parents would still trigger appropriate decomposition under v0.11. + +## New meaningful product failure exposed + +The semantic grounding check (`validateAnswerMeaningAlignment()`) rejects legitimate cost-reduction answers that fall into class "other" (unclassified). This prevents any graph update for scenarios where the primary driver is expressed in non-protected language such as "cost reduction", "savings", or "economic benefit". This is a **separate** defect from decomposition relevance — it blocks the entire Update 1 path, not just question selection. + +## What remains unproven + +- Whether the v0.11 decomposition fix correctly allows *appropriate* comparison/measurement decomposition when the parent genuinely supports it. +- Whether the decomposition fix correctly prevents *inappropriate* decomposition for parents that lack comparison semantics (when Update 1 does succeed). +- The semantic grounding gate's behavior with diverse answer phrasings. + +## Cleanup + +- Production code changed: NO +- Canonical script restored: YES +- Retries/additional runs: 0 +- Ollama calls beyond budget: 0 + +--- + +*Branch: `feature/decomposition-relevance-v0.11`. Status: BLOCKED — semantic grounding gate prevents Update 1 evaluation.*