experiment: test scope coherence against unknown count

docs/current-handoff.md

@@ -101,8 +101,8 @@ Answer before continuing:
 
 ---
 
-*Created by Experiment 34. Updated by Experiments 38, 39, 40, 41, 42, 43, 44, 45. Branch: `feature/user-workspace-ux-v0.7`.*
+*Created by Experiment 34. Updated by Experiments 38, 39, 40, 41, 42, 43, 44, 45, 46. Branch: `feature/user-workspace-ux-v0.7`.*
 
 ### Return-to-Work Note (Experiment 45)
 
-Experiment 45 tested the too_broad boundary from two to five competing unknowns using identical synthetic fixtures varying only in unknown count. The assessor switched at exactly three→four active unknowns; resolved-item gate works correctly (one stays too_broad, two clears it). The boundary appears mechanically clear but conceptually uncertain — real-scenario validation remains open. No production code changed. What remains: whether health should be healthy (not cannot_determine) for 2–3 unknowns with no question; whether the threshold needs widening. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/investigation-state-assessor.too-broad-boundary.test.js` for results, then `docs/design-evolution-log.md` Experiment 45 section for full data.
+Experiment 46 compared two four-unknown investigations with identical structural counts — one coherent (four unknowns contributing to one decision) and one scattered (four unrelated threads). Both returned too_broad with Clarify eligible, confirming the assessor cannot distinguish semantic coherence from scatter using active-unknown count alone. No production behaviour changed. What remains open: whether a coherence signal should be added, and what threshold would feel right in real-user data. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/investigation-state-assessor.scope-coherence.test.js` for results, then `docs/design-evolution-log.md` Experiment 46 section for full analysis.

docs/design-evolution-log.md

@@ -2928,3 +2928,141 @@ Pending Rob's review. No production behaviour changed. The next logical step wou
 ### Production Assessor Status
 
 **Unchanged.** No code was modified. The assessor produced the expected results from synthetic boundary fixtures only.
+
+---
+
+## Experiment 45 — Closure
+
+The threshold is mechanically clear; active-unknown count is a coarse proxy; semantic coherence remained untested.
+
+---
+
+## Experiment 46 — Does "Too Broad" Mean Too Many Questions, or Too Many Unrelated Questions? (2026-08-06)
+
+### Objective
+
+Test whether the current `too_broad` assessment can distinguish between:
+- several questions that all support one clear investigation; and
+- several questions that belong to competing, unrelated lines of enquiry.
+
+This is a passive diagnostic experiment. No production code changes.
+
+### Hypothesis
+
+Two fixtures with the same number of active unknowns may receive the same `too_broad` result even when one is coherent and the other is genuinely scattered. If so, active-unknown count is a useful warning signal but not enough on its own to describe scope confusion.
+
+### Context Pack Used
+
+Engine Experiment Work pack (Pack 1). Documents loaded:
+- `docs/current-project-state.md`, `docs/current-working-principles.md`, `.claude/architecture-guardrails.md`, `docs/current-implementation-verification.md`
+- `lib/assessment/investigation-state-assessor.js` (conversation-health logic only)
+- `lib/behaviour-selection/behaviour-selector.js` (Clarify rule only)
+- `tests/investigation-state-assessor.too-broad-boundary.test.js`
+- `tests/investigation-state-assessor.unclear-start.test.js`
+- Experiment 45 section in `docs/design-evolution-log.md`
+
+No additional documents loaded.
+
+### Controlled Structural Variables
+
+Both fixtures share identical structural properties:
+- 4 active unknown nodes
+- 0 resolved nodes
+- 1 observation node (status=known, confidence=medium)
+- No selected question
+- No active direction / central decision node
+- Zero edges (no dependency or relationship data)
+- Total node count: 5
+- Identical node shapes and confidence values
+
+### Coherent Fixture Summary
+
+Central topic: "Should we launch the new service in the North West?"
+
+Four unknowns all contributing to one decision:
+1. Whether customer demand exists in the North West region
+2. What price point the North West market would accept
+3. Whether delivery infrastructure can support the North West region
+4. Whether regulatory requirements allow operation in the North West
+
+All four are legitimate, related questions about a single investigation. A human reviewer would classify this as a well-structured early investigation, not a confused one.
+
+### Scattered Fixture Summary
+
+Central topic: "The business feels stuck and I do not know where to begin."
+
+Four unknowns from competing, unrelated threads:
+1. Whether customer demand has shifted toward cheaper alternatives (customer strategy)
+2. Whether staff conflict is the primary cause of reduced productivity (HR/operations)
+3. Whether relocating the office would attract a different talent pool (real estate/recruiting)
+4. Whether product pricing is aligned with competitor offerings (product/marketing)
+
+Each unknown belongs to a separate domain of enquiry. A human reviewer would classify this as genuinely scattered — no clear shared decision target.
+
+### Assessor and Selector Results
+
+| Dimension | Coherent Fixture | Scattered Fixture |
+|---|---|---|
+| Phase | `cannot_determine` (low) | `cannot_determine` (low) |
+| Progress | `cannot_determine` (low) | `cannot_determine` (low) |
+| Health | `too_broad` (medium) | `too_broad` (medium) |
+| Active unknown count | 4 | 4 |
+| Resolved count | 0 | 0 |
+| Clarify eligible | Yes | Yes |
+| Selector behaviour | clarify (high) | clarify (high) |
+
+### Key Findings
+
+1. **Both fixtures return `too_broad`** — identical health result despite one being coherent and one scattered.
+2. **Clarify becomes eligible in both** via Rule A (health === too_broad). Identical eligibility.
+3. **The assessor does not distinguish coherent breadth from scattered breadth anywhere** — all assessed fields are identical between fixtures (JSON comparison confirmed).
+4. **Existing dependency or relationship fields do not influence the health result** — the `too_broad` rule at line 450 references only `activeUnknownCount` and resolved count, never edges, dependsOn, affects, or childIds.
+5. **Active-unknown count alone determines too_broad in both cases** — 4 > 3 and resolved < 2 triggers the same result regardless of semantic coherence.
+
+### Human-Sense Review
+
+- **Coherent fixture:** `too_broad` is **questionable**. Four unknowns contributing to one decision is breadth, not confusion. The label conflates "many questions" with "scattered focus."
+- **Scattered fixture:** `too_broad` is **believable**. Four unrelated threads genuinely represent scope confusion. The label matches plain-English intuition.
+
+### Was Coherence Detected?
+
+**No.** The assessor produces identical results for both fixtures. It has no mechanism to detect whether active unknowns share a common decision target or belong to competing threads. Only the count (4) and resolution status (0) matter.
+
+### Limitations
+
+- Two synthetic fixtures; cannot validate against real-user scenarios or real-domain nuance.
+- Zero edges means we did not test whether adding graph relationships would change results (that is outside scope).
+- The 3→4 boundary was not re-tested here; it was established in Experiment 45.
+- Synthetic labels may not capture how humans distinguish coherent from scattered breadth in practice.
+
+### Conclusion
+
+**Count is useful but cannot distinguish coherence.** Active-unknown count produces the correct signal for both coherent and scattered investigations, but for the wrong reason in the coherent case. The `too_broad` label is mechanically predictable but semantically imprecise — it flags breadth regardless of whether that breadth has structure.
+
+### Questionable or Unsupported Findings
+
+1. Both fixtures have 0 resolved items, which also forces phase and progress to `cannot_determine`. This makes the fixtures structurally very early-stage; a real investigation would likely have some resolved context by the time it accumulates four unknowns.
+2. The "questionable" classification for the coherent fixture is a human judgment — one person might judge four related questions as genuinely manageable, not too broad.
+
+### Status
+
+Pending Rob's review. No production behaviour changed.
+
+### Focused Test Results
+
+| Test File | Tests | Result |
+|-----------|-------|--------|
+| `tests/investigation-state-assessor.scope-coherence.test.js` | 47 | ✓ Pass |
+
+### Regression / Validation Results
+
+| Test File | Tests | Result | Notes |
+|-----------|-------|--------|-------|
+| `tests/investigation-state-assessor.too-broad-boundary.test.js` | 32 | ✓ Pass | Zero regressions |
+| `tests/investigation-state-assessor.unclear-start.test.js` | 23 | ✓ Pass | Zero regressions |
+| `tests/investigation-state-assessor.test.js` | 51 | ✓ Pass | Zero regressions |
+| `tests/behaviour-selection.test.js` | 51 | ✓ Pass | Zero regressions |
+
+### Production Assessor Status
+
+**Unchanged.** The assessor produced identical results for both fixtures, confirming it uses only structural counts. No code was modified.