experiment: test too-broad assessment boundary
Experiment 45 — passive boundary experiment measuring the existing assessor's too_broad threshold from two to five competing unknowns. Key findings: - Boundary switches exactly between three and four active unknowns - Clarify eligibility follows the same boundary - Resolved-item gate works correctly (1 stays too_broad, 2 clears it) - 2–3 unknowns return cannot_determine health (not healthy or too_broad) - Boundary appears mechanically clear but conceptually uncertain No production code changed. Synthetic fixtures only.
This commit is contained in:
+8
-16
@@ -48,23 +48,15 @@ Experiment 43 audited Clarify readiness across all 10 real assessment turns in e
|
||||
|
||||
Experiment 44 created one deliberately unclear starting scenario (five competing unknowns, zero resolved evidence, vague central statement) to test whether the assessor produces a Clarify-justifying signal. The assessor returned `too_broad` conversation health — confirming the previously untested too_broad path works correctly with real data. Clarify became eligible via Rule A. No production code changed. Remaining open: whether orienting phase is needed for earlier-stage clarification, and whether 2–3 competing threads (below the >3 threshold) can represent genuine scope confusion. Status pending Rob's review.
|
||||
|
||||
Experiment 45 tested the too_broad boundary from two to five competing unknowns using identical synthetic fixtures varying only in unknown count. The assessor switched at exactly three→four active unknowns — two and three returned cannot_determine; four and five returned too_broad. Clarify eligibility followed the same boundary. Resolved-item gate works correctly: one resolved item stays too_broad, two resolves it. The boundary appears mechanically clear but conceptually uncertain — synthetic fixtures cannot confirm whether three-to-four feels right to real users. No production code changed. What remains open: whether health should default to healthy (not cannot_determine) for 2–3 unknowns with no question; whether the threshold needs widening for real-world use. Status pending Rob's review.
|
||||
|
||||
## 5. What Remains Open
|
||||
|
||||
- Whether the `too_broad` health trigger needs widening so Clarify fires in more typical investigations;
|
||||
- The `too_broad` boundary sits exactly between three and four active unknowns; it is mechanically clear but conceptually uncertain — whether it aligns with genuine user confusion requires real-scenario validation;
|
||||
- Health defaults to `cannot_determine` rather than `healthy` for 2–3 unknowns (no active question present); whether this is a bug or feature needs review;
|
||||
- Whether the `too_broad` threshold needs widening so Clarify fires in more typical investigations;
|
||||
- Whether `user_overloaded` health should be producible by the assessor for stalled/inconsistent evidence states.
|
||||
|
||||
## 5. What Remains Open
|
||||
|
||||
- Whether to fix the orienting dead-code path in the assessor or accept it as intentional design;
|
||||
- Whether a dedicated fixture validating the `too_broad` trigger should be created before widening its threshold;
|
||||
- Whether Clarify's intent (genuinely unclear scope) maps better to an existing signal that could fire earlier.
|
||||
|
||||
### What Remains Open (continued from above)
|
||||
|
||||
- Experiment 44 confirmed the `too_broad` path fires for a genuinely unclear starting point, but only with >3 competing unknowns; scenarios with 2–3 threads may remain unflagged;
|
||||
- Whether a dedicated `orienting` phase is needed to catch earlier-stage clarification needs (the orienting-based Clarify rule remains dead code);
|
||||
- Whether Clarify's intent (genuinely unclear scope) maps better to an existing signal that could fire earlier.
|
||||
|
||||
### When This Knowledge-Management Phase Is Complete
|
||||
|
||||
Provisional criteria for review (all confirmed met by Experiment 38 cold-start test):
|
||||
@@ -109,8 +101,8 @@ Answer before continuing:
|
||||
|
||||
---
|
||||
|
||||
*Created by Experiment 34. Updated by Experiments 38, 39, 40, 41, 42, 43. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
*Created by Experiment 34. Updated by Experiments 38, 39, 40, 41, 42, 43, 44, 45. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
|
||||
### Return-to-Work Note (Experiment 44)
|
||||
### Return-to-Work Note (Experiment 45)
|
||||
|
||||
Experiment 44 tested one genuinely unclear starting scenario (five competing unknowns, zero resolved evidence, no clear decision target). The assessor returned `too_broad` for conversation health and Clarify became eligible via the production rule. This confirms the previously untested `too_broad` health classifier path works correctly with real data. No production code changed. What remains unresolved: whether a dedicated `orienting` phase is needed to catch earlier-stage clarification needs, and whether fewer competing threads (2–3) can still represent a genuinely unclear situation that the assessor should flag. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/investigation-state-assessor.unclear-start.test.js` for the fixture, then `docs/design-evolution-log.md` lines 2721 onward for full experiment data, then `lib/assessment/investigation-state-assessor.js` line ~450 for the `too_broad` trigger logic.
|
||||
Experiment 45 tested the too_broad boundary from two to five competing unknowns using identical synthetic fixtures varying only in unknown count. The assessor switched at exactly three→four active unknowns; resolved-item gate works correctly (one stays too_broad, two clears it). The boundary appears mechanically clear but conceptually uncertain — real-scenario validation remains open. No production code changed. What remains: whether health should be healthy (not cannot_determine) for 2–3 unknowns with no question; whether the threshold needs widening. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `tests/investigation-state-assessor.too-broad-boundary.test.js` for results, then `docs/design-evolution-log.md` Experiment 45 section for full data.
|
||||
|
||||
Reference in New Issue
Block a user