experiment: audit behaviour reachability and blocking

This commit is contained in:
2026-08-06 16:48:41 +01:00
parent da3d35f437
commit a4731d908f
4 changed files with 818 additions and 7 deletions
+80
View File
@@ -2324,3 +2324,83 @@ Pending Rob's review. Five behaviours are too narrow for this to be definitive,
- `docs/current-handoff.md` — return-to-work note replaced
---
## Experiment 40 — Audit Behaviour Reachability and Blocking (2026-08-06)
### Objective
Why did Clarify, Summarise, and Pause not appear during Experiment 39? Acknowledge: 5 (71%), Continue: 2 (29%), others: 0. This is a passive diagnostic — no rule changes, no engine modifications.
### Method
One test file (`tests/behaviour-selection.reachability.test.js`) containing:
- Diagnostic audit helper that evaluates every behaviour rule against one assessment object
- Real-scenario audits across the same Experiment 39 turns (8 turns total)
- Synthetic reachability checks for each behaviour in isolation
### Findings
#### Summarise — eligible_but_blocked
Eligible in 2 of 7 real turns:
- long-investigation turn 1 (resolvedNodeCount ≥ 3 + progress=steady triggers summarise rule)
- long-investigation turn 2 (phase=concluding triggers summarise rule)
In both cases, health=healthy simultaneously, so Acknowledge (priority 1) fires first. Summarise rules are met but its output is never returned because the selector returns early on priority ordering.
**Root cause: priority conflict, not assessor failure.** The phase evidence correctly identifies concluding/synthesising states; the problem is that Acknowledge's broader trigger condition (health=healthy is the most common state) fires first.
#### Clarify — never_eligible_in_tested_scenarios (reachable only in synthetic case)
Not eligible in any of 7 real turns because neither trigger condition is met:
- `health=too_broad`: requires activeUnknownCount > 3 AND resolvedNodeCount < 2 — no fixture reaches this state
- `phase=orienting + observationDensity < 3`: current assessor never produces phase=orienting for tested scenarios
Synthetic case confirms the rule fires correctly in isolation (with low-confidence phase to avoid Acknowledge blocking).
**Root cause: assessor health classification logic produces too few `too_broad` cases. The trigger condition is extremely narrow — needs activeUnknownCount > 3 AND resolved < 2 simultaneously.**
#### Pause — eligible_but_blocked
Eligible in 1 of 7 real turns:
- contradictory-evidence turn 1 (phase=focusing + progress=stalled triggers pause rule)
In this case, health=healthy simultaneously, so Acknowledge blocks it. The second pause trigger (`health=user_overloaded`) is never met because the assessor never produces that state.
**Root cause: same priority conflict as Summarise. One of two rules fires in real data but gets blocked by Acknowledge's earlier position.**
### Synthetic Reachability Confirmation
All five behaviours are independently reachable when isolated from Acknowledge:
- ✅ acknowledge — healthy + confident phase
- ✅ clarify — too_broad health (with low-confidence phase to avoid Acknowledge)
- ✅ summarise — synthesising/concluding phase (without healthy health)
- ✅ pause — focusing+stalled or user_overloaded (without healthy health)
- ✅ continue — no rules match
### Classifications
| Behaviour | Classification | Primary Cause |
|---|---|---|
| Summarise | eligible_but_blocked | Acknowledge priority 1 fires first when health=healthy |
| Clarify | never_eligible_in_tested_scenarios (reachable only in synthetic) | `too_broad` trigger too narrow for test scenarios; `orienting+low obs` not produced by assessor |
| Pause | eligible_but_blocked | Acknowledge priority 1 fires first when health=healthy; `user_overloaded` never produced |
### Impact on Prior Finding (Exp 39)
Experiment 39 concluded "the Acknowledge→Summarise priority conflict prevents Summarise from firing." Experiment 40 confirms this and adds that **Pause faces the same blocking** (1 eligible turn, blocked). Clarify's absence is fundamentally different: its rules are not triggered at all in tested scenarios.
This means any fix must address two distinct problems:
1. Priority conflict affecting Summarise AND Pause (same cause)
2. Narrow trigger conditions for Clarify and the `user_overloaded` health state
### Test Results
- `tests/behaviour-selection.reachability.test.js`: 33 passed (new diagnostic file)
- `tests/behaviour-selection.test.js`: 51 passed (no regressions)
- `tests/behaviour-selection.real-assessment.test.js`: 16 passed (shared fixtures intact)
- `tests/investigation-state-assessor.test.js`: 51 passed (assessor unchanged)
### Documents Updated
- `docs/design-evolution-log.md` — this entry
- `docs/current-handoff.md` — return-to-work note replaced