experiment: compare acknowledge priority alternatives
Experiment 41 compared two passive alternatives for reducing Acknowledge dominance: Variant A (priority reordering): evaluate Summarise/Pause before Acknowledge - Converges on concluding→summarise and stalled→pause correctly - Introduces false-positive summarise at long-investigation t3 Variant B (Acknowledge exclusions): gate Acknowledge via phase/progress/health - Converges on the same two genuine changes without false-positives - Recommended: cleaner boundaries, preserves Acknowledge for healthy focus states Both variants produce identical results for 2 of 7 tested turns. Variant A diverges at long-investigation t3 (focusing phase with resolvedNodeCount=3). Variant B correctly preserves Acknowledge there via its exclusion list. Test files: - tests/behaviour-selection.counterfactual.test.js (44 tests, new) No production code changed.
This commit is contained in:
+10
-4
@@ -38,9 +38,15 @@ Experiment 39 resumed reasoning experiments with a passive validation of Behavio
|
||||
|
||||
Experiment 40 diagnosed the root causes: Summarise and Pause fire their rules in real data but are always blocked by Acknowledge's priority-1 position (priority conflict, not assessor failure). Clarive's triggers never activate in tested scenarios due to the `too_broad` health condition being extremely narrow. All five behaviours confirmed independently reachable in synthetic isolation. No rules changed.
|
||||
|
||||
Experiment 41 compared two passive alternatives for reducing Acknowledge dominance:
|
||||
- Variant A (priority reordering): evaluate Summarise/Pause before Acknowledge — introduces false-positive summarise in focusing phase
|
||||
- Variant B (Acknowledge exclusions): keep priority, gate Acknowledge when phase=concluding/synthesising or progress=stalled or health=user_overloaded — recommended
|
||||
- Both variants converge on the same two genuine changes: concluding→summarise and stalled→pause
|
||||
- No production code changed. Branch: `feature/user-workspace-ux-v0.7`.
|
||||
|
||||
## 5. What Remains Open
|
||||
|
||||
- Whether Acknowledge should be reordered past Summarise/Pause to resolve the priority conflict diagnosed in Exp 40;
|
||||
- Implement Variant B's acknowledgment exclusion (recommended path from Exp 41);
|
||||
- Whether the `too_broad` health trigger needs widening so Clarify fires in more typical investigations;
|
||||
- Whether `user_overloaded` health should be producible by the assessor for stalled/inconsistent evidence states.
|
||||
|
||||
@@ -88,8 +94,8 @@ Answer before continuing:
|
||||
|
||||
---
|
||||
|
||||
*Created by Experiment 34. Updated by Experiments 38, 39, 40. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
*Created by Experiment 34. Updated by Experiments 38, 39, 40, 41. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
|
||||
### Return-to-Work Note (Experiment 39)
|
||||
### Return-to-Work Note (Experiment 41)
|
||||
|
||||
Experiment 40 audited why Clarify, Summarise, and Pause were absent in Experiment 39. Summarise and Pause are eligible_but_blocked: their rule conditions fire but Acknowledge (priority 1) always wins when health=healthy — a priority conflict. Clarify is never_eligible_in_tested_scenarios: its `too_broad` trigger requires activeUnknownCount > 3 AND resolved < 2 simultaneously, which no fixture reaches; the `orienting+low obs` path is never produced by the assessor for tested domains. All five behaviours confirmed independently reachable in synthetic isolation. No rules changed. Status pending Rob's review. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `docs/current-handoff.md`, then `docs/design-evolution-log.md` entry for Experiment 40, and `tests/behaviour-selection.reachability.test.js` for diagnostic details. The next action is discussion about whether to reorder Acknowledge lower (past Summarise/Pause) or refine its trigger to exclude terminal phases.
|
||||
Experiments 39–41 diagnosed Acknowledge dominance in Behaviour Selection and compared two passive alternatives. **Experiment 41 concluded:** Variant B (Acknowledge exclusions via phase/progress/health gates) is recommended over Variant A (priority reordering, which introduces false-positive summarise). Both variants correctly identify the same two genuine changes: `concluding → summarise` and `stalled → pause`. No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `docs/current-handoff.md`, then `docs/design-evolution-log.md` entry for Experiment 41, and `tests/behaviour-selection.counterfactual.test.js` for variant details. The next action is implementing Variant B's `isAcknowledgeExcluded()` gate as a test-only function, then evaluating against a wider range of assessment scenarios.
|
||||
|
||||
@@ -46,7 +46,7 @@ These are what currently affect the working engine:
|
||||
The following were built during Experiments 18–25B. They are isolated diagnostic layers with no active integration into the user-facing investigation:
|
||||
|
||||
- Investigation-state assessment (phase and progress classification);
|
||||
- Behaviour selection from assessed state — passively evaluated in Experiment 39 against real assessment outputs; produces all valid behaviours but Acknowledge dominates when health=healthy;
|
||||
- Behaviour selection from assessed state — passively evaluated in Experiments 39–41; all five behaviours reachable but Acknowledge dominates (71% on real data); Exp 41 recommends Variant B (Acknowledge exclusions via phase/progress/health gates) as the cleaner approach;
|
||||
- Decision condition status evaluation;
|
||||
- Question-to-condition relevance scoring;
|
||||
- Evidence direction classification (support, contradict, inform);
|
||||
@@ -100,7 +100,7 @@ Historical documents are retained under `docs/archive/` and should be opened onl
|
||||
|
||||
## 8. Return-to-Work Summary
|
||||
|
||||
Engine experiments advanced to Experiment 40, a reachability diagnostic confirming Summarise and Pause fire their rules but are always blocked by Acknowledge (priority-1 conflict), while Clarive's triggers never activate in tested scenarios due to narrow `too_broad` health conditions. All five behaviours confirmed independently reachable synthetically. No rules or engine behaviour changed. Knowledge-management phase is complete: all seven criteria met. No backlog items deleted or promoted.
|
||||
Engine experiments advanced to Experiment 41, which compared two passive alternatives for reducing Acknowledge dominance. Variant B (Acknowledge exclusions via phase/progress/health gates) is recommended over Variant A (priority reordering with false-positive side effects). Both variants converge on the same two genuine changes: `concluding → summarise` and `stalled → pause`. No production code changed.
|
||||
|
||||
First document to read: `docs/current-project-state.md`. Then consult `.claude/architecture-guardrails.md` before any code changes and `docs/project-knowledge-inventory.md` for task-specific references. The full experiment history remains available in `docs/design-evolution-log.md` but is no longer default reading.
|
||||
|
||||
|
||||
@@ -2404,3 +2404,93 @@ This means any fix must address two distinct problems:
|
||||
### Documents Updated
|
||||
- `docs/design-evolution-log.md` — this entry
|
||||
- `docs/current-handoff.md` — return-to-work note replaced
|
||||
|
||||
## Experiment 41 — Compare Acknowledge Priority Alternatives (2026-08-06)
|
||||
|
||||
### Purpose
|
||||
|
||||
Experiment 40 confirmed Summarise and Pause are eligible_but_blocked by Acknowledge's priority-1 position. Two passive alternatives were compared without modifying production code:
|
||||
|
||||
**Variant A** — Reorder rules so specific behaviours (Summarise, Pause) evaluate before Acknowledge. The idea is that if a more specific behaviour fires first, it captures the terminal/stalled states where Acknowledge should not fire.
|
||||
|
||||
**Variant B** — Keep existing priority order but exclude Acknowledge from firing when phase=concluding/synthesising, progress=stalled, or health=user_overloaded. The idea is to gate Acknowledge rather than reorder everything.
|
||||
|
||||
### Method
|
||||
|
||||
Both variants were implemented as test-only functions in `tests/behaviour-selection.counterfactual.test.js`. Each variant was evaluated against the same 7 real assessment turns from Experiments 39/40 across 3 scenarios. All five behaviours confirmed independently reachable synthetically. No production rules changed.
|
||||
|
||||
### Assessor Outputs (7 real turns)
|
||||
|
||||
| # | Scenario | Turn | Phase (conf) | Progress | Health | Existing |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 1 | long-investigation | 0 | cannot_determine(low) | cannot_determine | too_narrow | continue |
|
||||
| 2 | long-investigation | 3 | focusing(high) | steady | healthy | acknowledge |
|
||||
| 3 | long-investigation | 4 | concluding(high) | steady | healthy | acknowledge |
|
||||
| 4 | contradictory-evidence | 0 | focusing(high) | cannot_determine | healthy | acknowledge |
|
||||
| 5 | contradictory-evidence | 1 | focusing(high) | stalled | healthy | acknowledge |
|
||||
| 6 | contradictory-evidence | 2 | focusing(high) | steady | healthy | acknowledge |
|
||||
| 7 | short-early | 0 | exploring(low) | cannot_determine | healthy | continue |
|
||||
|
||||
### Results on Real Scenarios
|
||||
|
||||
| Turn | Existing | Variant A | Variant B | Change? |
|
||||
|---|---|---|---|---|
|
||||
| long-investigation t3 | acknowledge | summarise | acknowledge | V-A: side-effect |
|
||||
| long-investigation t4 | acknowledge | **summarise** | **summarise** | **convergent ✓** |
|
||||
| contradictory-evidence t1 | acknowledge | **pause** | **pause** | **convergent ✓** |
|
||||
| All others | unchanged | unchanged | unchanged | — |
|
||||
|
||||
### Divergence Analysis
|
||||
|
||||
**Variant A diverges from Variant B at long-investigation turn 3.** Variant A produces `summarise` because its `resolvedNodeCount >= 3 && steady` rule fires at priority 1 without phase context. The assessor confirms this is a focusing-phase state (not synthesising/concluding) where the user needs acknowledgment, not compression. This is a false-positive for summarisation — a side-effect of Variant A's priority reordering.
|
||||
|
||||
**Variant B correctly preserves Acknowledge** at long-investigation t3 because:
|
||||
1. The exclusion list only includes `synthesising`, `concluding`, `stalled`, and `user_overloaded` — not focusing
|
||||
2. SummariseV2 itself has a phase gate (`phase.value === "synthesising"`) that prevents false-fire in focusing states
|
||||
3. Acknowledge at priority 1 wins because no exclusion applies
|
||||
|
||||
### Key Findings
|
||||
|
||||
1. **Both variants converge on the same two genuine changes:** `concluding → summarise` and `stalled → pause`. This was the experiment's primary question, and both approaches answer it correctly.
|
||||
|
||||
2. **Variant A introduces a false-positive:** The `resolvedNodeCount >= 3 && steady` rule fires in focusing-phase states without phase context, causing premature summarisation when Acknowledge would be more useful.
|
||||
|
||||
3. **Variant B has cleaner boundaries:** Explicit exclusion conditions prevent unwanted side-effects while preserving Acknowledge's role as the default healthy-state behaviour.
|
||||
|
||||
4. **Distribution shift (both variants):**
|
||||
- Existing: acknowledge 71%, continue 29%
|
||||
- Variant A: acknowledge 29%, summarise 29%, pause 14%, continue 29%
|
||||
- Variant B: acknowledge 43%, summarise 14%, pause 14%, continue 29%
|
||||
- Variant B preserves more Acknowledge because it doesn't remove the default healthy-state behaviour entirely
|
||||
|
||||
5. **Variant B is architecturally cleaner** for this problem space because it adds a targeted gate to one rule rather than reordering five priority levels — each of which would need individual review for side-effects.
|
||||
|
||||
### Test Results
|
||||
|
||||
- `tests/behaviour-selection.counterfactual.test.js`: 44 passed (new diagnostic file)
|
||||
- `tests/behaviour-selection.reachability.test.js`: 33 passed (no regressions)
|
||||
- `tests/behaviour-selection.real-assessment.test.js`: 16 passed (shared fixtures intact)
|
||||
- `tests/behaviour-selection.test.js`: 51 passed (no regressions)
|
||||
|
||||
### Decision Criteria
|
||||
|
||||
| Criterion | Variant A | Variant B |
|
||||
|---|---|---|
|
||||
| Fixes concluding state | ✓ summarise | ✓ summarise |
|
||||
| Fixes stalled state | ✓ pause | ✓ pause |
|
||||
| No false-positive changes | ✗ long-t3 → summarise | ✓ preserved acknowledge |
|
||||
| Implementation complexity | Simple reordering | Small gate function |
|
||||
| Maintains Acknowledge for healthy focus states | ? (depends on future review) | ✓ explicit preservation |
|
||||
|
||||
### Recommendation
|
||||
|
||||
**Variant B is preferred.** Both variants correctly identify the two genuine changes needed. Variant B has no false-positives, cleaner architectural boundaries (targeted exclusion vs priority reordering), and better preserves the existing Acknowledge default for healthy focusing states where it is appropriate. A recommended implementation would:
|
||||
|
||||
1. Keep existing priority order
|
||||
2. Add `isAcknowledgeExcluded()` function with conditions: phase∈{synthesising, concluding}, progress=stalled, health=user_overloaded
|
||||
3. Gate Acknowledge through this exclusion before selecting it at priority 1
|
||||
|
||||
### Documents Updated
|
||||
- `docs/design-evolution-log.md` — this entry
|
||||
- `docs/current-handoff.md` — return-to-work note replaced
|
||||
|
||||
|
||||
Reference in New Issue
Block a user