experiment: audit behaviour reachability and blocking

This commit is contained in:
2026-08-06 16:48:41 +01:00
parent da3d35f437
commit a4731d908f
4 changed files with 818 additions and 7 deletions
+7 -5
View File
@@ -36,11 +36,13 @@ Experiment 38 tested whether a genuinely cold session (no prior conversation con
Experiment 39 resumed reasoning experiments with a passive validation of Behaviour Selection against real Investigation State Assessment outputs. Seven turns across three scenarios were evaluated. Acknowledge dominated (71%) because it fires at priority 1 whenever health=healthy, even in terminal and stalled states where Summarise or Pause would be more useful. The assessor→selector contract aligns cleanly; no transformation is needed between pipeline stages. All five behaviours remain reachable but some never appear in typical scenarios (Clarify requires too_broad health which few fixtures produce). Status pending Rob's review.
Experiment 40 diagnosed the root causes: Summarise and Pause fire their rules in real data but are always blocked by Acknowledge's priority-1 position (priority conflict, not assessor failure). Clarive's triggers never activate in tested scenarios due to the `too_broad` health condition being extremely narrow. All five behaviours confirmed independently reachable in synthetic isolation. No rules changed.
## 5. What Remains Open
- Whether the Acknowledge→Summarise priority conflict can be resolved without breaking existing rule conditions;
- Whether behaviour selection produces different patterns in investigation domains other than supplier procurement, market entry, and product comparison;
- Which rules (Acknowledge, Clarify, Summarise, Pause) fire most frequently across a larger set of scenarios.
- Whether Acknowledge should be reordered past Summarise/Pause to resolve the priority conflict diagnosed in Exp 40;
- Whether the `too_broad` health trigger needs widening so Clarify fires in more typical investigations;
- Whether `user_overloaded` health should be producible by the assessor for stalled/inconsistent evidence states.
### When This Knowledge-Management Phase Is Complete
@@ -86,8 +88,8 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38, 39. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 38, 39, 40. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 39)
Experiment 39 resumed reasoning experiments with a passive validation of Behaviour Selection against real Investigation State Assessment outputs across three scenarios (long investigation, contradictory evidence, short early). No user-facing engine behaviour changed. The selector produced all valid behaviours but with skewed distribution: Acknowledge dominated at 71% because it fires first when health=healthy, even in terminal/concluding states where Summarise would be more useful — this is an acknowledged priority conflict, not a contract mismatch. The assessor→selector pipeline aligns cleanly. Clarify and Pause never fired (no too_broad health produced; other rules blocked). Status pending Rob's review. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when returning: `docs/current-handoff.md`, then `docs/design-evolution-log.md` entry for Experiment 39, and `tests/behaviour-selection.real-assessment.test.js` for evaluation details. The next action is discussion about whether to refine the selector or move on.
Experiment 40 audited why Clarify, Summarise, and Pause were absent in Experiment 39. Summarise and Pause are eligible_but_blocked: their rule conditions fire but Acknowledge (priority 1) always wins when health=healthy — a priority conflict. Clarify is never_eligible_in_tested_scenarios: its `too_broad` trigger requires activeUnknownCount > 3 AND resolved < 2 simultaneously, which no fixture reaches; the `orienting+low obs` path is never produced by the assessor for tested domains. All five behaviours confirmed independently reachable in synthetic isolation. No rules changed. Status pending Rob's review. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `docs/current-handoff.md`, then `docs/design-evolution-log.md` entry for Experiment 40, and `tests/behaviour-selection.reachability.test.js` for diagnostic details. The next action is discussion about whether to reorder Acknowledge lower (past Summarise/Pause) or refine its trigger to exclude terminal phases.