experiment: test assessor against unclear starting point
This commit is contained in:
+10
-2
@@ -46,6 +46,8 @@ Experiment 42 implemented Variant B's narrow Acknowledge exclusion gate in the p
|
||||
|
||||
Experiment 43 audited Clarify readiness across all 10 real assessment turns in existing fixtures. Zero turns produced Clarify-eligible states. Two findings: (1) the orienting-based Clarify rule is dead code because the assessor never produces phase=orienting, and (2) the too_broad trigger requires conditions no fixture exercises. Branch: `feature/user-workspace-ux-v0.7`.
|
||||
|
||||
Experiment 44 created one deliberately unclear starting scenario (five competing unknowns, zero resolved evidence, vague central statement) to test whether the assessor produces a Clarify-justifying signal. The assessor returned `too_broad` conversation health — confirming the previously untested too_broad path works correctly with real data. Clarify became eligible via Rule A. No production code changed. Remaining open: whether orienting phase is needed for earlier-stage clarification, and whether 2–3 competing threads (below the >3 threshold) can represent genuine scope confusion. Status pending Rob's review.
|
||||
|
||||
## 5. What Remains Open
|
||||
|
||||
- Whether the `too_broad` health trigger needs widening so Clarify fires in more typical investigations;
|
||||
@@ -57,6 +59,12 @@ Experiment 43 audited Clarify readiness across all 10 real assessment turns in e
|
||||
- Whether a dedicated fixture validating the `too_broad` trigger should be created before widening its threshold;
|
||||
- Whether Clarify's intent (genuinely unclear scope) maps better to an existing signal that could fire earlier.
|
||||
|
||||
### What Remains Open (continued from above)
|
||||
|
||||
- Experiment 44 confirmed the `too_broad` path fires for a genuinely unclear starting point, but only with >3 competing unknowns; scenarios with 2–3 threads may remain unflagged;
|
||||
- Whether a dedicated `orienting` phase is needed to catch earlier-stage clarification needs (the orienting-based Clarify rule remains dead code);
|
||||
- Whether Clarify's intent (genuinely unclear scope) maps better to an existing signal that could fire earlier.
|
||||
|
||||
### When This Knowledge-Management Phase Is Complete
|
||||
|
||||
Provisional criteria for review (all confirmed met by Experiment 38 cold-start test):
|
||||
@@ -103,6 +111,6 @@ Answer before continuing:
|
||||
|
||||
*Created by Experiment 34. Updated by Experiments 38, 39, 40, 41, 42, 43. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
|
||||
### Return-to-Work Note (Experiment 43)
|
||||
### Return-to-Work Note (Experiment 44)
|
||||
|
||||
Experiment 43 diagnosed Clarify's absence across all real scenarios: zero eligibility in 10 turns. The orienting-based rule is dead code (assessor never produces phase=orienting); the too_broad trigger needs a dedicated fixture. Open decision: fix the dead-code path, widen the threshold, or accept both and defer. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `docs/current-handoff.md`, then `tests/behaviour-selection.clarify-readiness.test.js` for the full audit data, and `lib/behaviour-selection/behaviour-selector.js` lines 65-83 for the Clarify rules.
|
||||
Experiment 44 tested one genuinely unclear starting scenario (five competing unknowns, zero resolved evidence, no clear decision target). The assessor returned `too_broad` for conversation health and Clarify became eligible via the production rule. This confirms the previously untested `too_broad` health classifier path works correctly with real data. No production code changed. What remains unresolved: whether a dedicated `orienting` phase is needed to catch earlier-stage clarification needs, and whether fewer competing threads (2–3) can still represent a genuinely unclear situation that the assessor should flag. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/investigation-state-assessor.unclear-start.test.js` for the fixture, then `docs/design-evolution-log.md` lines 2721 onward for full experiment data, then `lib/assessment/investigation-state-assessor.js` line ~450 for the `too_broad` trigger logic.
|
||||
|
||||
Reference in New Issue
Block a user