experiment: locate irrelevant decomposition question boundary

This commit is contained in:
2026-08-10 07:19:52 +01:00
parent 62593eea14
commit 513c483501
2 changed files with 130 additions and 0 deletions
+32
View File
@@ -172,6 +172,38 @@ Answer before continuing:
*Created by Experiment 34. Updated by Experiments 3853, 54A54Z, 55A55F, 56D56H, 56L56M, v0.8 closeout. Branch: `feature/reasoning-fidelity-v0.8`. First-pass reasoning-fidelity v0.8 complete to AF scope.*
---
### Experiment 57E — Irrelevant Decomposition Question Boundary
**Objective:** Identify the exact graph node that triggered the decomposition producing "How the two observations were measured", and determine whether the parent was genuinely about comparison/measurement/timing before decomposition.
**Run results (one start + one update, scenario: team relocation London→Manchester, answer: cost reduction £2M savings):**
- **Parent node:** `nagtmgmg` — label: "Explanation for why Should I relocate my engineering team from London to Manchester", description: "Need to understand what change or event could explain why these observations differ..."
- **Parent kind/status:** unknown/unknown
- **Selected child (Update 1):** `np6zcaw` — label: "How the two observations were measured"
- **Reasoning pattern assigned to selected child:** comparison
- **Investigation strategy:** evidence_gathering
**Key finding:** The parent node `nagtmgmg` has NO semantics of comparison, measurement validity, or timing. Its description only references "these observations differ" in a generic explanatory sense (what change/event explains the difference between initial state and current state). It does not establish that there are two measured observations to compare. Yet decomposition produced five children including hardcoded "two observations" templates:
1. Whether the two observations reflect different timing
2. How the two observations were measured
3. Possible change mainly affecting [first observation]
4. Possible change mainly affecting [second observation]
5. Possible one-off event during the period
The parent itself is a generic "explanation for difference" unknown — structurally similar to any post-hoc explanation query — and does NOT contain comparison/measurement semantics. The "two observations" language in decomposition children originates from `buildDecompositionTemplates()` default template (line 14841510 of `lib/graph/apply-proposal.js`) which unconditionally injects these children for any unknown parent that doesn't match the special-case regex patterns.
**Classification: A — decomposition trigger defect.** The parent node is not genuinely about comparing observations, measurement validity, or timing comparability, yet deterministic decomposition generates measurement/comparison children. The decomposition mechanism fires on an inappropriate parent whose semantics are purely explanatory ("what change/event explains the state difference").
**What this establishes:** The "two observations" decomposition children are template-injected regardless of parent meaning. They appear whenever `buildDecompositionTemplates()` runs for a generic unknown node that doesn't match special-case regex patterns. This is not a case of semantically appropriate decomposition with overly-specific templates — the decomposition trigger itself is firing on an irrelevant parent.
**What this does NOT prove:** That every decomposition is irrelevant (some parents genuinely concern comparison/measurement). That fixing the trigger won't break valid decompositions elsewhere. Whether other template children (change affecting X/Y, one-off event) share the same defect pattern or have independent justification issues.
---
### Experiment 57C — Post-v0.9 Investigation Flow Observation
**STOPPED AT FIRST PRODUCTION-PATH FAILURE**
+98
View File
@@ -0,0 +1,98 @@
# Experiment 57E — Irrelevant Decomposition Question Boundary
**Date:** 2026-08-10
**Branch:** `feature/relationship-contract-v0.10`
**Model:** qwen-claude:latest at http://192.168.1.111:11434
**Live calls:** 1 start + 1 update = 2 (within budget)
## Objective
Identify the exact graph node that triggered the decomposition producing "How the two observations were measured", and determine whether the parent was genuinely about comparison/measurement/timing before decomposition.
## Fixed inputs
- **Scenario:** Should I relocate my engineering team from London to Manchester?
- **Answer 1:** We're looking at this mainly for cost reduction — roughly £2M annual savings on office overhead.
## Canonical execution route
- Dev server: `npx next dev``http://localhost:3000`
- Start: `POST /api/cases/start`
- Update 1: `POST /api/cases/update`
- Script: `scripts/reproduce-multi-turn-investigation.mjs` (temporarily instrumented, then restored)
## Results
### Selected question
- **Exact text:** "What evidence would clarify how the two observations were measured?"
- **Node ID:** np6zcaw
- **Reasoning pattern:** comparison
- **Investigation strategy:** evidence_gathering
### Selected active unknown node (np6zcaw)
- **id:** np6zcaw
- **label:** How the two observations were measured
- **description:** Need evidence about the measure used for each observation, because that could help explain Should I relocate my engineering team from London to Manchester.
- **kind:** unknown
- **status:** unknown
- **parentId:** nagtgmg
- **childIds:** ["nagtgmg"]
### Parent node (nagtmgmg)
- **id:** nagtgmg
- **label:** Explanation for why Should I relocate my engineering team from London to Manchester
- **description:** Need to understand what change or event could explain why these observations differ, because that is needed to investigate their relationship.
- **kind:** unknown
- **status:** unknown
- **parentId:** null (top-level)
- **childIds:** [none populated — children added via graph edges]
### Sibling/decomposition children of parent nagtgmg
1. **id:** nlymgp2, **label:** Whether the two observations reflect different timing, **description:** Need to know whether the two observations reflect different timing, because that could help explain Should I relocate my engineering team from London to Manchester., **kind:** unknown, **status:** unknown
2. **id:** np6zcaw, **label:** How the two observations were measured, **description:** Need evidence about the measure used for each observation, because that could help explain Should I relocate my engineering team from London to Manchester., **kind:** unknown, **status:** unknown
3. **id:** ndya37c, **label:** Possible change mainly affecting engineering team is currently operational in london, **description:** Need to know whether a possible change mainly affected engineering team is currently operational in london, because that could help explain Should I relocate my engineering team from London to Manchester., **kind:** unknown, **status:** unknown
4. **id:** nmak7da, **label:** Possible change mainly affecting relocation to manchester is actively being evaluated by the decision-maker, **description:** Need to know whether a possible change mainly affected relocation to manchester is actively being evaluated by the decision-maker, because that could help explain Should I relocate my engineering team from London to Manchester., **kind:** unknown, **status:** unknown
5. **id:** nqajgbf, **label:** Possible one-off event during the period, **description:** Need to know whether a possible one-off event happened during the period, because that could help explain Should I relocate my engineering team from London to Manchester., **kind:** unknown, **status:** unknown
### Decomposition diagnostic fields retained by API
None — the production API does not expose decomposition parent/child diagnostics in its response.
## Pre-written decision rule (recorded before run)
- **Outcome A** — decomposition trigger defect: parent is NOT genuinely about comparing observations/measurement/timing, yet decomposition generates those children
- **Outcome B** — decomposition template defect: parent IS comparison-related but child template is over-specific
- **Outcome C** — both
- **Outcome D** — insufficient evidence
### Classification: A — decomposition trigger defect
## Rationale
The parent node `nagtmgmg` has NO semantics of comparison, measurement validity, or timing. Its description only references "these observations differ" in a generic explanatory sense (what change/event explains the difference between initial state and current state). It does not establish that there are two measured observations to compare. Yet decomposition produced five children including hardcoded "two observations" templates.
The parent itself is a generic "explanation for difference" unknown — structurally similar to any post-hoc explanation query — and does NOT contain comparison/measurement semantics. The "two observations" language in decomposition children originates from `buildDecompositionTemplates()` default template (line 14841510 of `lib/graph/apply-proposal.js`) which unconditionally injects these children for any unknown parent that doesn't match special-case regex patterns.
## What this experiment established
- The "two observations" decomposition children are template-injected regardless of parent meaning
- They appear whenever `buildDecompositionTemplates()` runs for a generic unknown node that doesn't match special-case regex patterns
- The selected question was assigned reasoning pattern "comparison" despite the parent having no comparison semantics
- This is a decomposition trigger defect, not merely an over-specific template
## What this experiment does NOT prove
- That every decomposition is irrelevant (some parents genuinely concern comparison/measurement)
- That fixing the trigger won't break valid decompositions elsewhere
- Whether other template children (change affecting X/Y, one-off event) share the same defect pattern or have independent justification issues
## Cleanup
- Production code changed: NO
- Canonical script restored after temporary instrumentation: YES
- Retries/additional runs: 0
- Ollama calls beyond budget: 0