experiment: compare acknowledge priority alternatives
Experiment 41 compared two passive alternatives for reducing Acknowledge dominance: Variant A (priority reordering): evaluate Summarise/Pause before Acknowledge - Converges on concluding→summarise and stalled→pause correctly - Introduces false-positive summarise at long-investigation t3 Variant B (Acknowledge exclusions): gate Acknowledge via phase/progress/health - Converges on the same two genuine changes without false-positives - Recommended: cleaner boundaries, preserves Acknowledge for healthy focus states Both variants produce identical results for 2 of 7 tested turns. Variant A diverges at long-investigation t3 (focusing phase with resolvedNodeCount=3). Variant B correctly preserves Acknowledge there via its exclusion list. Test files: - tests/behaviour-selection.counterfactual.test.js (44 tests, new) No production code changed.
This commit is contained in:
@@ -2404,3 +2404,93 @@ This means any fix must address two distinct problems:
|
||||
### Documents Updated
|
||||
- `docs/design-evolution-log.md` — this entry
|
||||
- `docs/current-handoff.md` — return-to-work note replaced
|
||||
|
||||
## Experiment 41 — Compare Acknowledge Priority Alternatives (2026-08-06)
|
||||
|
||||
### Purpose
|
||||
|
||||
Experiment 40 confirmed Summarise and Pause are eligible_but_blocked by Acknowledge's priority-1 position. Two passive alternatives were compared without modifying production code:
|
||||
|
||||
**Variant A** — Reorder rules so specific behaviours (Summarise, Pause) evaluate before Acknowledge. The idea is that if a more specific behaviour fires first, it captures the terminal/stalled states where Acknowledge should not fire.
|
||||
|
||||
**Variant B** — Keep existing priority order but exclude Acknowledge from firing when phase=concluding/synthesising, progress=stalled, or health=user_overloaded. The idea is to gate Acknowledge rather than reorder everything.
|
||||
|
||||
### Method
|
||||
|
||||
Both variants were implemented as test-only functions in `tests/behaviour-selection.counterfactual.test.js`. Each variant was evaluated against the same 7 real assessment turns from Experiments 39/40 across 3 scenarios. All five behaviours confirmed independently reachable synthetically. No production rules changed.
|
||||
|
||||
### Assessor Outputs (7 real turns)
|
||||
|
||||
| # | Scenario | Turn | Phase (conf) | Progress | Health | Existing |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 1 | long-investigation | 0 | cannot_determine(low) | cannot_determine | too_narrow | continue |
|
||||
| 2 | long-investigation | 3 | focusing(high) | steady | healthy | acknowledge |
|
||||
| 3 | long-investigation | 4 | concluding(high) | steady | healthy | acknowledge |
|
||||
| 4 | contradictory-evidence | 0 | focusing(high) | cannot_determine | healthy | acknowledge |
|
||||
| 5 | contradictory-evidence | 1 | focusing(high) | stalled | healthy | acknowledge |
|
||||
| 6 | contradictory-evidence | 2 | focusing(high) | steady | healthy | acknowledge |
|
||||
| 7 | short-early | 0 | exploring(low) | cannot_determine | healthy | continue |
|
||||
|
||||
### Results on Real Scenarios
|
||||
|
||||
| Turn | Existing | Variant A | Variant B | Change? |
|
||||
|---|---|---|---|---|
|
||||
| long-investigation t3 | acknowledge | summarise | acknowledge | V-A: side-effect |
|
||||
| long-investigation t4 | acknowledge | **summarise** | **summarise** | **convergent ✓** |
|
||||
| contradictory-evidence t1 | acknowledge | **pause** | **pause** | **convergent ✓** |
|
||||
| All others | unchanged | unchanged | unchanged | — |
|
||||
|
||||
### Divergence Analysis
|
||||
|
||||
**Variant A diverges from Variant B at long-investigation turn 3.** Variant A produces `summarise` because its `resolvedNodeCount >= 3 && steady` rule fires at priority 1 without phase context. The assessor confirms this is a focusing-phase state (not synthesising/concluding) where the user needs acknowledgment, not compression. This is a false-positive for summarisation — a side-effect of Variant A's priority reordering.
|
||||
|
||||
**Variant B correctly preserves Acknowledge** at long-investigation t3 because:
|
||||
1. The exclusion list only includes `synthesising`, `concluding`, `stalled`, and `user_overloaded` — not focusing
|
||||
2. SummariseV2 itself has a phase gate (`phase.value === "synthesising"`) that prevents false-fire in focusing states
|
||||
3. Acknowledge at priority 1 wins because no exclusion applies
|
||||
|
||||
### Key Findings
|
||||
|
||||
1. **Both variants converge on the same two genuine changes:** `concluding → summarise` and `stalled → pause`. This was the experiment's primary question, and both approaches answer it correctly.
|
||||
|
||||
2. **Variant A introduces a false-positive:** The `resolvedNodeCount >= 3 && steady` rule fires in focusing-phase states without phase context, causing premature summarisation when Acknowledge would be more useful.
|
||||
|
||||
3. **Variant B has cleaner boundaries:** Explicit exclusion conditions prevent unwanted side-effects while preserving Acknowledge's role as the default healthy-state behaviour.
|
||||
|
||||
4. **Distribution shift (both variants):**
|
||||
- Existing: acknowledge 71%, continue 29%
|
||||
- Variant A: acknowledge 29%, summarise 29%, pause 14%, continue 29%
|
||||
- Variant B: acknowledge 43%, summarise 14%, pause 14%, continue 29%
|
||||
- Variant B preserves more Acknowledge because it doesn't remove the default healthy-state behaviour entirely
|
||||
|
||||
5. **Variant B is architecturally cleaner** for this problem space because it adds a targeted gate to one rule rather than reordering five priority levels — each of which would need individual review for side-effects.
|
||||
|
||||
### Test Results
|
||||
|
||||
- `tests/behaviour-selection.counterfactual.test.js`: 44 passed (new diagnostic file)
|
||||
- `tests/behaviour-selection.reachability.test.js`: 33 passed (no regressions)
|
||||
- `tests/behaviour-selection.real-assessment.test.js`: 16 passed (shared fixtures intact)
|
||||
- `tests/behaviour-selection.test.js`: 51 passed (no regressions)
|
||||
|
||||
### Decision Criteria
|
||||
|
||||
| Criterion | Variant A | Variant B |
|
||||
|---|---|---|
|
||||
| Fixes concluding state | ✓ summarise | ✓ summarise |
|
||||
| Fixes stalled state | ✓ pause | ✓ pause |
|
||||
| No false-positive changes | ✗ long-t3 → summarise | ✓ preserved acknowledge |
|
||||
| Implementation complexity | Simple reordering | Small gate function |
|
||||
| Maintains Acknowledge for healthy focus states | ? (depends on future review) | ✓ explicit preservation |
|
||||
|
||||
### Recommendation
|
||||
|
||||
**Variant B is preferred.** Both variants correctly identify the two genuine changes needed. Variant B has no false-positives, cleaner architectural boundaries (targeted exclusion vs priority reordering), and better preserves the existing Acknowledge default for healthy focusing states where it is appropriate. A recommended implementation would:
|
||||
|
||||
1. Keep existing priority order
|
||||
2. Add `isAcknowledgeExcluded()` function with conditions: phase∈{synthesising, concluding}, progress=stalled, health=user_overloaded
|
||||
3. Gate Acknowledge through this exclusion before selecting it at priority 1
|
||||
|
||||
### Documents Updated
|
||||
- `docs/design-evolution-log.md` — this entry
|
||||
- `docs/current-handoff.md` — return-to-work note replaced
|
||||
|
||||
|
||||
Reference in New Issue
Block a user