experiment: audit behaviour reachability and blocking
This commit is contained in:
@@ -36,11 +36,13 @@ Experiment 38 tested whether a genuinely cold session (no prior conversation con
|
||||
|
||||
Experiment 39 resumed reasoning experiments with a passive validation of Behaviour Selection against real Investigation State Assessment outputs. Seven turns across three scenarios were evaluated. Acknowledge dominated (71%) because it fires at priority 1 whenever health=healthy, even in terminal and stalled states where Summarise or Pause would be more useful. The assessor→selector contract aligns cleanly; no transformation is needed between pipeline stages. All five behaviours remain reachable but some never appear in typical scenarios (Clarify requires too_broad health which few fixtures produce). Status pending Rob's review.
|
||||
|
||||
Experiment 40 diagnosed the root causes: Summarise and Pause fire their rules in real data but are always blocked by Acknowledge's priority-1 position (priority conflict, not assessor failure). Clarive's triggers never activate in tested scenarios due to the `too_broad` health condition being extremely narrow. All five behaviours confirmed independently reachable in synthetic isolation. No rules changed.
|
||||
|
||||
## 5. What Remains Open
|
||||
|
||||
- Whether the Acknowledge→Summarise priority conflict can be resolved without breaking existing rule conditions;
|
||||
- Whether behaviour selection produces different patterns in investigation domains other than supplier procurement, market entry, and product comparison;
|
||||
- Which rules (Acknowledge, Clarify, Summarise, Pause) fire most frequently across a larger set of scenarios.
|
||||
- Whether Acknowledge should be reordered past Summarise/Pause to resolve the priority conflict diagnosed in Exp 40;
|
||||
- Whether the `too_broad` health trigger needs widening so Clarify fires in more typical investigations;
|
||||
- Whether `user_overloaded` health should be producible by the assessor for stalled/inconsistent evidence states.
|
||||
|
||||
### When This Knowledge-Management Phase Is Complete
|
||||
|
||||
@@ -86,8 +88,8 @@ Answer before continuing:
|
||||
|
||||
---
|
||||
|
||||
*Created by Experiment 34. Updated by Experiments 38, 39. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
*Created by Experiment 34. Updated by Experiments 38, 39, 40. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
|
||||
### Return-to-Work Note (Experiment 39)
|
||||
|
||||
Experiment 39 resumed reasoning experiments with a passive validation of Behaviour Selection against real Investigation State Assessment outputs across three scenarios (long investigation, contradictory evidence, short early). No user-facing engine behaviour changed. The selector produced all valid behaviours but with skewed distribution: Acknowledge dominated at 71% because it fires first when health=healthy, even in terminal/concluding states where Summarise would be more useful — this is an acknowledged priority conflict, not a contract mismatch. The assessor→selector pipeline aligns cleanly. Clarify and Pause never fired (no too_broad health produced; other rules blocked). Status pending Rob's review. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when returning: `docs/current-handoff.md`, then `docs/design-evolution-log.md` entry for Experiment 39, and `tests/behaviour-selection.real-assessment.test.js` for evaluation details. The next action is discussion about whether to refine the selector or move on.
|
||||
Experiment 40 audited why Clarify, Summarise, and Pause were absent in Experiment 39. Summarise and Pause are eligible_but_blocked: their rule conditions fire but Acknowledge (priority 1) always wins when health=healthy — a priority conflict. Clarify is never_eligible_in_tested_scenarios: its `too_broad` trigger requires activeUnknownCount > 3 AND resolved < 2 simultaneously, which no fixture reaches; the `orienting+low obs` path is never produced by the assessor for tested domains. All five behaviours confirmed independently reachable in synthetic isolation. No rules changed. Status pending Rob's review. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `docs/current-handoff.md`, then `docs/design-evolution-log.md` entry for Experiment 40, and `tests/behaviour-selection.reachability.test.js` for diagnostic details. The next action is discussion about whether to reorder Acknowledge lower (past Summarise/Pause) or refine its trigger to exclude terminal phases.
|
||||
|
||||
Reference in New Issue
Block a user