92 lines
5.1 KiB
Markdown
92 lines
5.1 KiB
Markdown
# Experiment 57F — Decomposition Relevance Fix Live Validation
|
||
|
||
**Date:** 2026-08-10
|
||
**Branch:** `feature/decomposition-relevance-v0.11`
|
||
**Codex refinement validated:** `7e4c506` — reasoning: prevent unsupported comparison decomposition
|
||
**Model:** qwen-claude:latest at http://192.168.1.111:11434
|
||
|
||
## Objective
|
||
|
||
After v0.11, does Update 1 avoid manufacturing the unsupported "two observations / measured" decomposition and produce a next question grounded in the actual relocation investigation?
|
||
|
||
This is observation-only validation.
|
||
|
||
## Fixed inputs
|
||
|
||
- **Scenario:** Should I relocate my engineering team from London to Manchester?
|
||
- **Answer 1:** We're looking at this mainly for cost reduction — roughly £2M annual savings on office overhead.
|
||
- **Answer 2:** NOT submitted (fixed budget: Start + Update 1 = 2 live calls)
|
||
|
||
## Pre-written human expectation (recorded before run)
|
||
|
||
> The engine must not generate or select an unsupported measurement/comparison unknown such as "How the two observations were measured" or "Whether the two observations reflect different timing" unless the live graph actually contains a parent that establishes a genuine comparison/measurement problem. For this relocation/cost-reduction turn, the next question should remain grounded in a real unresolved aspect of the relocation decision. A broad unresolved parent is preferable to an invented measurement question.
|
||
>
|
||
> Do not define in advance what the replacement question *must* be.
|
||
|
||
## Canonical execution route
|
||
|
||
- Dev server: `npx next dev` → http://localhost:3000
|
||
- Start: `POST /api/cases/start`
|
||
- Update 1: `POST /api/cases/update`
|
||
- Script: `scripts/reproduce-multi-turn-investigation.mjs` (temporarily instrumented, then restored)
|
||
|
||
## Live call budget
|
||
|
||
- Start: 1
|
||
- Update 1: 1
|
||
- Update 2: 0
|
||
- **Total:** 2 live Ollama calls
|
||
|
||
## Results
|
||
|
||
### Classification: BLOCKED
|
||
|
||
### Production result
|
||
|
||
- **HTTP status:** 422 (Unprocessable Entity)
|
||
- **Stage:** `proposal_compatibility`
|
||
- **Node count:** 8 (unchanged from start)
|
||
- **Edge count:** 4 (unchanged from start)
|
||
- **Selected question:** null (Update failed before selection)
|
||
- **Proposal validation error:** `"Proposal cannot resolve beyond an unclassified answer by introducing an unsupported constraint or preference/trade-off distinction."`
|
||
|
||
### What happened
|
||
|
||
1. Start returned HTTP 200 with a valid graph (8 nodes, 4 edges) and a selected question about the viability of the engineering team relocation.
|
||
2. Update 1 submitted Answer 1 (cost reduction / £2M savings). The LLM produced grounded `userSupportedMeaning` at the prompt level. However, the answer was classified as "other" (unclassified) rather than falling into any of the protected categories. The semantic grounding check in `validateAnswerMeaningAlignment()` rejected the proposal because it could not establish that the unclassified answer supports resolving any specific unknown.
|
||
3. The graph was NOT updated. No decomposition occurred. No new nodes were added.
|
||
|
||
## What this experiment established
|
||
|
||
- The v0.11 fix (`7e4c506`) cannot be evaluated in this run because Update 1 fails at the semantic grounding layer before decomposition can be reached.
|
||
- The `validateAnswerMeaningAlignment()` check (from the semantic grounding mechanism validated in Experiments 57A–57B) continues to block legitimate cost-reduction answers that land in class "other".
|
||
- No prohibited decomposition children ("two observations", "measured", "different timing") can be confirmed absent because no graph update occurred.
|
||
- The blocking error is **not** a decomposition defect — it is the pre-existing semantic grounding gate preventing unclassified answers from producing any proposal.
|
||
|
||
## What this experiment does NOT prove
|
||
|
||
- Whether the v0.11 decomposition relevance fix works when Update 1 *does* succeed (i.e., when the answer falls into a supported class).
|
||
- Whether the semantic grounding blocker is itself correct or over-aggressive for cost-reduction scenarios.
|
||
- Whether valid comparison/measurement parents would still trigger appropriate decomposition under v0.11.
|
||
|
||
## New meaningful product failure exposed
|
||
|
||
The semantic grounding check (`validateAnswerMeaningAlignment()`) rejects legitimate cost-reduction answers that fall into class "other" (unclassified). This prevents any graph update for scenarios where the primary driver is expressed in non-protected language such as "cost reduction", "savings", or "economic benefit". This is a **separate** defect from decomposition relevance — it blocks the entire Update 1 path, not just question selection.
|
||
|
||
## What remains unproven
|
||
|
||
- Whether the v0.11 decomposition fix correctly allows *appropriate* comparison/measurement decomposition when the parent genuinely supports it.
|
||
- Whether the decomposition fix correctly prevents *inappropriate* decomposition for parents that lack comparison semantics (when Update 1 does succeed).
|
||
- The semantic grounding gate's behavior with diverse answer phrasings.
|
||
|
||
## Cleanup
|
||
|
||
- Production code changed: NO
|
||
- Canonical script restored: YES
|
||
- Retries/additional runs: 0
|
||
- Ollama calls beyond budget: 0
|
||
|
||
---
|
||
|
||
*Branch: `feature/decomposition-relevance-v0.11`. Status: BLOCKED — semantic grounding gate prevents Update 1 evaluation.*
|