experiment: compare acknowledge priority alternatives

Experiment 41 compared two passive alternatives for reducing Acknowledge dominance:

Variant A (priority reordering): evaluate Summarise/Pause before Acknowledge
- Converges on concluding→summarise and stalled→pause correctly
- Introduces false-positive summarise at long-investigation t3

Variant B (Acknowledge exclusions): gate Acknowledge via phase/progress/health
- Converges on the same two genuine changes without false-positives
- Recommended: cleaner boundaries, preserves Acknowledge for healthy focus states

Both variants produce identical results for 2 of 7 tested turns.
Variant A diverges at long-investigation t3 (focusing phase with resolvedNodeCount=3).
Variant B correctly preserves Acknowledge there via its exclusion list.

Test files:
- tests/behaviour-selection.counterfactual.test.js (44 tests, new)
No production code changed.
This commit is contained in:
2026-08-06 17:19:02 +01:00
parent a4731d908f
commit eee8c6b1e4
4 changed files with 681 additions and 6 deletions
+90
View File
@@ -2404,3 +2404,93 @@ This means any fix must address two distinct problems:
### Documents Updated
- `docs/design-evolution-log.md` — this entry
- `docs/current-handoff.md` — return-to-work note replaced
## Experiment 41 — Compare Acknowledge Priority Alternatives (2026-08-06)
### Purpose
Experiment 40 confirmed Summarise and Pause are eligible_but_blocked by Acknowledge's priority-1 position. Two passive alternatives were compared without modifying production code:
**Variant A** — Reorder rules so specific behaviours (Summarise, Pause) evaluate before Acknowledge. The idea is that if a more specific behaviour fires first, it captures the terminal/stalled states where Acknowledge should not fire.
**Variant B** — Keep existing priority order but exclude Acknowledge from firing when phase=concluding/synthesising, progress=stalled, or health=user_overloaded. The idea is to gate Acknowledge rather than reorder everything.
### Method
Both variants were implemented as test-only functions in `tests/behaviour-selection.counterfactual.test.js`. Each variant was evaluated against the same 7 real assessment turns from Experiments 39/40 across 3 scenarios. All five behaviours confirmed independently reachable synthetically. No production rules changed.
### Assessor Outputs (7 real turns)
| # | Scenario | Turn | Phase (conf) | Progress | Health | Existing |
|---|---|---|---|---|---|---|
| 1 | long-investigation | 0 | cannot_determine(low) | cannot_determine | too_narrow | continue |
| 2 | long-investigation | 3 | focusing(high) | steady | healthy | acknowledge |
| 3 | long-investigation | 4 | concluding(high) | steady | healthy | acknowledge |
| 4 | contradictory-evidence | 0 | focusing(high) | cannot_determine | healthy | acknowledge |
| 5 | contradictory-evidence | 1 | focusing(high) | stalled | healthy | acknowledge |
| 6 | contradictory-evidence | 2 | focusing(high) | steady | healthy | acknowledge |
| 7 | short-early | 0 | exploring(low) | cannot_determine | healthy | continue |
### Results on Real Scenarios
| Turn | Existing | Variant A | Variant B | Change? |
|---|---|---|---|---|
| long-investigation t3 | acknowledge | summarise | acknowledge | V-A: side-effect |
| long-investigation t4 | acknowledge | **summarise** | **summarise** | **convergent ✓** |
| contradictory-evidence t1 | acknowledge | **pause** | **pause** | **convergent ✓** |
| All others | unchanged | unchanged | unchanged | — |
### Divergence Analysis
**Variant A diverges from Variant B at long-investigation turn 3.** Variant A produces `summarise` because its `resolvedNodeCount >= 3 && steady` rule fires at priority 1 without phase context. The assessor confirms this is a focusing-phase state (not synthesising/concluding) where the user needs acknowledgment, not compression. This is a false-positive for summarisation — a side-effect of Variant A's priority reordering.
**Variant B correctly preserves Acknowledge** at long-investigation t3 because:
1. The exclusion list only includes `synthesising`, `concluding`, `stalled`, and `user_overloaded` — not focusing
2. SummariseV2 itself has a phase gate (`phase.value === "synthesising"`) that prevents false-fire in focusing states
3. Acknowledge at priority 1 wins because no exclusion applies
### Key Findings
1. **Both variants converge on the same two genuine changes:** `concluding → summarise` and `stalled → pause`. This was the experiment's primary question, and both approaches answer it correctly.
2. **Variant A introduces a false-positive:** The `resolvedNodeCount >= 3 && steady` rule fires in focusing-phase states without phase context, causing premature summarisation when Acknowledge would be more useful.
3. **Variant B has cleaner boundaries:** Explicit exclusion conditions prevent unwanted side-effects while preserving Acknowledge's role as the default healthy-state behaviour.
4. **Distribution shift (both variants):**
- Existing: acknowledge 71%, continue 29%
- Variant A: acknowledge 29%, summarise 29%, pause 14%, continue 29%
- Variant B: acknowledge 43%, summarise 14%, pause 14%, continue 29%
- Variant B preserves more Acknowledge because it doesn't remove the default healthy-state behaviour entirely
5. **Variant B is architecturally cleaner** for this problem space because it adds a targeted gate to one rule rather than reordering five priority levels — each of which would need individual review for side-effects.
### Test Results
- `tests/behaviour-selection.counterfactual.test.js`: 44 passed (new diagnostic file)
- `tests/behaviour-selection.reachability.test.js`: 33 passed (no regressions)
- `tests/behaviour-selection.real-assessment.test.js`: 16 passed (shared fixtures intact)
- `tests/behaviour-selection.test.js`: 51 passed (no regressions)
### Decision Criteria
| Criterion | Variant A | Variant B |
|---|---|---|
| Fixes concluding state | ✓ summarise | ✓ summarise |
| Fixes stalled state | ✓ pause | ✓ pause |
| No false-positive changes | ✗ long-t3 → summarise | ✓ preserved acknowledge |
| Implementation complexity | Simple reordering | Small gate function |
| Maintains Acknowledge for healthy focus states | ? (depends on future review) | ✓ explicit preservation |
### Recommendation
**Variant B is preferred.** Both variants correctly identify the two genuine changes needed. Variant B has no false-positives, cleaner architectural boundaries (targeted exclusion vs priority reordering), and better preserves the existing Acknowledge default for healthy focusing states where it is appropriate. A recommended implementation would:
1. Keep existing priority order
2. Add `isAcknowledgeExcluded()` function with conditions: phase∈{synthesising, concluding}, progress=stalled, health=user_overloaded
3. Gate Acknowledge through this exclusion before selecting it at priority 1
### Documents Updated
- `docs/design-evolution-log.md` — this entry
- `docs/current-handoff.md` — return-to-work note replaced