experiment: audit behaviour reachability and blocking
This commit is contained in:
@@ -2324,3 +2324,83 @@ Pending Rob's review. Five behaviours are too narrow for this to be definitive,
|
||||
- `docs/current-handoff.md` — return-to-work note replaced
|
||||
|
||||
---
|
||||
|
||||
## Experiment 40 — Audit Behaviour Reachability and Blocking (2026-08-06)
|
||||
|
||||
### Objective
|
||||
|
||||
Why did Clarify, Summarise, and Pause not appear during Experiment 39? Acknowledge: 5 (71%), Continue: 2 (29%), others: 0. This is a passive diagnostic — no rule changes, no engine modifications.
|
||||
|
||||
### Method
|
||||
|
||||
One test file (`tests/behaviour-selection.reachability.test.js`) containing:
|
||||
- Diagnostic audit helper that evaluates every behaviour rule against one assessment object
|
||||
- Real-scenario audits across the same Experiment 39 turns (8 turns total)
|
||||
- Synthetic reachability checks for each behaviour in isolation
|
||||
|
||||
### Findings
|
||||
|
||||
#### Summarise — eligible_but_blocked
|
||||
|
||||
Eligible in 2 of 7 real turns:
|
||||
- long-investigation turn 1 (resolvedNodeCount ≥ 3 + progress=steady triggers summarise rule)
|
||||
- long-investigation turn 2 (phase=concluding triggers summarise rule)
|
||||
|
||||
In both cases, health=healthy simultaneously, so Acknowledge (priority 1) fires first. Summarise rules are met but its output is never returned because the selector returns early on priority ordering.
|
||||
|
||||
**Root cause: priority conflict, not assessor failure.** The phase evidence correctly identifies concluding/synthesising states; the problem is that Acknowledge's broader trigger condition (health=healthy is the most common state) fires first.
|
||||
|
||||
#### Clarify — never_eligible_in_tested_scenarios (reachable only in synthetic case)
|
||||
|
||||
Not eligible in any of 7 real turns because neither trigger condition is met:
|
||||
- `health=too_broad`: requires activeUnknownCount > 3 AND resolvedNodeCount < 2 — no fixture reaches this state
|
||||
- `phase=orienting + observationDensity < 3`: current assessor never produces phase=orienting for tested scenarios
|
||||
|
||||
Synthetic case confirms the rule fires correctly in isolation (with low-confidence phase to avoid Acknowledge blocking).
|
||||
|
||||
**Root cause: assessor health classification logic produces too few `too_broad` cases. The trigger condition is extremely narrow — needs activeUnknownCount > 3 AND resolved < 2 simultaneously.**
|
||||
|
||||
#### Pause — eligible_but_blocked
|
||||
|
||||
Eligible in 1 of 7 real turns:
|
||||
- contradictory-evidence turn 1 (phase=focusing + progress=stalled triggers pause rule)
|
||||
|
||||
In this case, health=healthy simultaneously, so Acknowledge blocks it. The second pause trigger (`health=user_overloaded`) is never met because the assessor never produces that state.
|
||||
|
||||
**Root cause: same priority conflict as Summarise. One of two rules fires in real data but gets blocked by Acknowledge's earlier position.**
|
||||
|
||||
### Synthetic Reachability Confirmation
|
||||
|
||||
All five behaviours are independently reachable when isolated from Acknowledge:
|
||||
- ✅ acknowledge — healthy + confident phase
|
||||
- ✅ clarify — too_broad health (with low-confidence phase to avoid Acknowledge)
|
||||
- ✅ summarise — synthesising/concluding phase (without healthy health)
|
||||
- ✅ pause — focusing+stalled or user_overloaded (without healthy health)
|
||||
- ✅ continue — no rules match
|
||||
|
||||
### Classifications
|
||||
|
||||
| Behaviour | Classification | Primary Cause |
|
||||
|---|---|---|
|
||||
| Summarise | eligible_but_blocked | Acknowledge priority 1 fires first when health=healthy |
|
||||
| Clarify | never_eligible_in_tested_scenarios (reachable only in synthetic) | `too_broad` trigger too narrow for test scenarios; `orienting+low obs` not produced by assessor |
|
||||
| Pause | eligible_but_blocked | Acknowledge priority 1 fires first when health=healthy; `user_overloaded` never produced |
|
||||
|
||||
### Impact on Prior Finding (Exp 39)
|
||||
|
||||
Experiment 39 concluded "the Acknowledge→Summarise priority conflict prevents Summarise from firing." Experiment 40 confirms this and adds that **Pause faces the same blocking** (1 eligible turn, blocked). Clarify's absence is fundamentally different: its rules are not triggered at all in tested scenarios.
|
||||
|
||||
This means any fix must address two distinct problems:
|
||||
1. Priority conflict affecting Summarise AND Pause (same cause)
|
||||
2. Narrow trigger conditions for Clarify and the `user_overloaded` health state
|
||||
|
||||
### Test Results
|
||||
|
||||
- `tests/behaviour-selection.reachability.test.js`: 33 passed (new diagnostic file)
|
||||
- `tests/behaviour-selection.test.js`: 51 passed (no regressions)
|
||||
- `tests/behaviour-selection.real-assessment.test.js`: 16 passed (shared fixtures intact)
|
||||
- `tests/investigation-state-assessor.test.js`: 51 passed (assessor unchanged)
|
||||
|
||||
### Documents Updated
|
||||
- `docs/design-evolution-log.md` — this entry
|
||||
- `docs/current-handoff.md` — return-to-work note replaced
|
||||
|
||||
Reference in New Issue
Block a user