experiment: test assessor against unclear starting point

2026-08-06 18:31:15 +01:00
parent ae2d1d9c52
commit ca71e79618
3 changed files with 446 additions and 2 deletions
@@ -2717,3 +2717,100 @@ All diagnostics confirm zero Clarify eligibility across the complete set of real

 - `docs/design-evolution-log.md` — this entry
 - `docs/current-handoff.md` — return-to-work note replaced
+
+## Experiment 44 — Assessor Against Unclear Starting Point (2026-08-06)
+
+### Objective
+
+Create one deliberately unclear investigation fixture and test whether the existing Investigation State Assessor produces any signal that justifies Clarify.
+
+### Hypothesis
+
+A deliberately unclear starting scenario may expose one of three outcomes:
+1. The assessor already produces `too_broad`.
+2. The assessor produces another existing signal that reasonably represents the need to clarify.
+3. The assessor has no suitable signal for unclear framing.
+
+### Fixture Description
+
+**File:** `tests/investigation-state-assessor.unclear-start.test.js` (test-only, not imported anywhere else)
+
+The fixture represents:
+- A vague central statement that admits uncertainty: *"The business feels stuck. Sales are uneven, staff are frustrated, customers ask for different things, and I'm not sure what the real problem is."*
+- Five competing unknown threads (customer demand, staff capacity, product direction, pricing, operations) with no priority anchor
+- Only one observation (the only concrete data point)
+- Zero resolved evidence nodes
+- No selected question (no established direction)
+- All existing graph fields only (id, label, description, kind, status, confidence, evidenceIds, dependsOn, affects, childIds)
+- Five `kind: "unknown"` nodes and one `kind: "observation"` node
+
+### Returned Assessment Signals
+
+| Signal | Value | Confidence |
+|--------|-------|------------|
+| Phase | `cannot_determine` | low |
+| Phase signals | "Insufficient data for phase classification" | — |
+| Progress | `cannot_determine` | low |
+| Progress signals | "Insufficient data for progress assessment" | — |
+| Conversation health | **`too_broad`** | medium |
+| Health signals | "5 active unknowns with fewer than 2 resolved items"; "Investigation may be spreading too thin" | — |
+
+Detailed evidence:
+- Phase evidence: resolvedNodeCount=0, activeUnknownCount=5, observationDensity=1, evidenceDepth="shallow"
+- Progress evidence: turnCount=0, recentResolutionsLastTurn=0
+- Health evidence: activeUnknownCount=5, resolvedNodeRatio=null, hasActiveQuestion=false
+
+### Clarify Eligibility
+
+**Clarify became eligible via Rule A.** The production `selectClarify` rule fires because `conversationHealth.value === "too_broad"`.
+
+The production selector (`selectBehaviour`) returned:
+- behaviour: `"clarify"`
+- confidence: `"high"`
+- reason: "Conversation health is too broad — investigation may be spreading too thin. Narrow focus through a specific clarification question."
+
+### Interpretation
+
+**Classification: `assessor_recognises_unclear_start`**
+
+The assessor produced `too_broad` from the unclear-start fixture, which directly maps to Clarify's intent (genuinely unclear scope requiring anchoring). The signal honestly reflects the starting situation: five competing unknowns with no resolved evidence and no established direction.
+
+### What the Assessor Recognised
+
+1. Multiple active unknowns without sufficient resolution triggered `too_broad` health classification.
+2. The assessor correctly recorded 5 active unknowns in both phase and health evidence sections.
+3. Observation density (1) was correctly reported as shallow.
+4. Phase confidence remained low due to insufficient data for any meaningful classification.
+
+### What the Assessor Failed to Recognise
+
+1. **`orienting` phase**: Still not produced by the assessor. The orienting-based Clarify rule remains dead code, unchanged from Experiment 43's finding.
+2. **Early-stage clarification need**: The `too_broad` trigger only fires after >3 unknowns accumulate — it does not catch a situation with fewer competing threads that is still genuinely unclear in framing.
+
+### Limitations
+
+- Only one fixture was tested. Different vague-scenario configurations may produce different results.
+- The `too_broad` trigger depends on having more than 3 active unknowns with fewer than 2 resolved — this specific threshold was exercised, but other boundary conditions (e.g., exactly 4 unknowns, or 5 unknowns with 1 resolved) were not tested.
+- The fixture uses the assessor's existing `too_broad` definition which conflates "many unknowns" with "unclear scope." A genuinely unclear scenario with only 2–3 competing threads may not trigger this signal.
+
+### Status
+
+Pending Rob's review. Experiment 43 remains closed — its conclusion that a deliberately unclear fixture was required is confirmed by this experiment, which successfully exercises the previously untested `too_broad` health path.
+
+### Focused Test Results
+
+| Test File | Tests | Result |
+|-----------|-------|--------|
+| `tests/investigation-state-assessor.unclear-start.test.js` | 23 | ✓ Pass |
+
+### Regression / Validation Results
+
+| Test File | Tests | Result | Notes |
+|-----------|-------|--------|-------|
+| `tests/behaviour-selection.clarify-readiness.test.js` | 31 | ✓ Pass | Zero regressions |
+| `tests/investigation-state-assessor.test.js` | 51 | ✓ Pass | Zero regressions |
+| `tests/behaviour-selection.test.js` | 51 | ✓ Pass | Zero regressions |
+
+### Production Assessor Status
+
+**Unchanged.** The assessor produced the expected `too_broad` signal from the unclear fixture, confirming the health classifier path works correctly. No code was modified.