experiment: test assessor against unclear starting point

This commit is contained in:
2026-08-06 18:31:15 +01:00
parent ae2d1d9c52
commit ca71e79618
3 changed files with 446 additions and 2 deletions
+10 -2
View File
@@ -46,6 +46,8 @@ Experiment 42 implemented Variant B's narrow Acknowledge exclusion gate in the p
Experiment 43 audited Clarify readiness across all 10 real assessment turns in existing fixtures. Zero turns produced Clarify-eligible states. Two findings: (1) the orienting-based Clarify rule is dead code because the assessor never produces phase=orienting, and (2) the too_broad trigger requires conditions no fixture exercises. Branch: `feature/user-workspace-ux-v0.7`.
Experiment 44 created one deliberately unclear starting scenario (five competing unknowns, zero resolved evidence, vague central statement) to test whether the assessor produces a Clarify-justifying signal. The assessor returned `too_broad` conversation health — confirming the previously untested too_broad path works correctly with real data. Clarify became eligible via Rule A. No production code changed. Remaining open: whether orienting phase is needed for earlier-stage clarification, and whether 2–3 competing threads (below the >3 threshold) can represent genuine scope confusion. Status pending Rob's review.
## 5. What Remains Open
- Whether the `too_broad` health trigger needs widening so Clarify fires in more typical investigations;
@@ -57,6 +59,12 @@ Experiment 43 audited Clarify readiness across all 10 real assessment turns in e
- Whether a dedicated fixture validating the `too_broad` trigger should be created before widening its threshold;
- Whether Clarify's intent (genuinely unclear scope) maps better to an existing signal that could fire earlier.
### What Remains Open (continued from above)
- Experiment 44 confirmed the `too_broad` path fires for a genuinely unclear starting point, but only with >3 competing unknowns; scenarios with 2–3 threads may remain unflagged;
- Whether a dedicated `orienting` phase is needed to catch earlier-stage clarification needs (the orienting-based Clarify rule remains dead code);
- Whether Clarify's intent (genuinely unclear scope) maps better to an existing signal that could fire earlier.
### When This Knowledge-Management Phase Is Complete
Provisional criteria for review (all confirmed met by Experiment 38 cold-start test):
@@ -103,6 +111,6 @@ Answer before continuing:
*Created by Experiment 34. Updated by Experiments 38, 39, 40, 41, 42, 43. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 43)
### Return-to-Work Note (Experiment 44)
Experiment 43 diagnosed Clarify's absence across all real scenarios: zero eligibility in 10 turns. The orienting-based rule is dead code (assessor never produces phase=orienting); the too_broad trigger needs a dedicated fixture. Open decision: fix the dead-code path, widen the threshold, or accept both and defer. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `docs/current-handoff.md`, then `tests/behaviour-selection.clarify-readiness.test.js` for the full audit data, and `lib/behaviour-selection/behaviour-selector.js` lines 65-83 for the Clarify rules.
Experiment 44 tested one genuinely unclear starting scenario (five competing unknowns, zero resolved evidence, no clear decision target). The assessor returned `too_broad` for conversation health and Clarify became eligible via the production rule. This confirms the previously untested `too_broad` health classifier path works correctly with real data. No production code changed. What remains unresolved: whether a dedicated `orienting` phase is needed to catch earlier-stage clarification needs, and whether fewer competing threads (2–3) can still represent a genuinely unclear situation that the assessor should flag. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/investigation-state-assessor.unclear-start.test.js` for the fixture, then `docs/design-evolution-log.md` lines 2721 onward for full experiment data, then `lib/assessment/investigation-state-assessor.js` line ~450 for the `too_broad` trigger logic.