experiment: test too-broad assessment boundary

Experiment 45 — passive boundary experiment measuring the existing
assessor's too_broad threshold from two to five competing unknowns.

Key findings:
- Boundary switches exactly between three and four active unknowns
- Clarify eligibility follows the same boundary
- Resolved-item gate works correctly (1 stays too_broad, 2 clears it)
- 2–3 unknowns return cannot_determine health (not healthy or too_broad)
- Boundary appears mechanically clear but conceptually uncertain

No production code changed. Synthetic fixtures only.
This commit is contained in:
2026-08-06 18:52:40 +01:00
parent ca71e79618
commit 1d234acd8c
3 changed files with 507 additions and 16 deletions
+114
View File
@@ -2814,3 +2814,117 @@ Pending Rob's review. Experiment 43 remains closed — its conclusion that a del
### Production Assessor Status
**Unchanged.** The assessor produced the expected `too_broad` signal from the unclear fixture, confirming the health classifier path works correctly. No code was modified.
### Closure
Experiment 44 is **closed**. Conclusion: the assessor recognises an extreme unclear start; too_broad and Clarify are reachable; the useful boundary remained unknown.
---
## Experiment 45 — Where Does "Too Broad" Begin? (2026-08-06)
### Objective
Test how the existing assessor's `too_broad` threshold behaves as an unclear starting scenario grows from two competing unknowns to five, all with identical base inputs. Passive boundary experiment only — no production code changes.
### Hypothesis
| Active unknowns | Expected health |
|---:|---|
| 2 | not `too_broad` |
| 3 | not `too_broad` |
| 4 | `too_broad` |
| 5 | `too_broad` |
### Fixture-Control Method
One test-only fixture builder creates the same vague starting situation varying only the number of competing unknowns:
- Same central statement; same single observation; zero resolved items (base); no selected question; no active direction; same node shapes and confidence values.
- Only the count of `kind: "unknown"` nodes differs.
### Results: Two Through Five Active Unknowns
| Active unknowns | Health | Confidence | Phase | Progress | Clarify eligible | Selector |
|---:|---|---|---|---|---:|---|
| 2 | `cannot_determine` | low | `cannot_determine` (low) | `cannot_determine` (low) | No | continue (low) |
| 3 | `cannot_determine` | low | `cannot_determine` (low) | `cannot_determine` (low) | No | continue (low) |
| 4 | `too_broad` | medium | `cannot_determine` (low) | `cannot_determine` (low) | Yes | clarify (high) |
| 5 | `too_broad` | medium | `cannot_determine` (low) | `cannot_determined` (low) | Yes | clarify (high) |
### Results: Four Unknowns + Resolved Items
| Active unknowns | Resolved | Health | Confidence | Clarify eligible |
|---:|---:|---|---|---:|
| 4 | 0 | `too_broad` | medium | Yes |
| 4 | 1 | `too_broad` | medium | Yes |
| 4 | 2 | `cannot_determine` | low | No |
### Human-Sense Review
- **Two competing threads:** Still appears ambiguous rather than clearly manageable. The assessor returns `cannot_determine`, not `healthy`. This is honest — two unknowns with one observation and no question genuinely leave the state unclear.
- **Three competing threads:** Appears ambiguous or already confused. The assessor still returns `cannot_determine`. This feels correct — three competing threads with minimal context is genuinely uncertain, not healthy.
- **Four competing threads:** Appears genuinely too broad. The transition from three (uncertain) to four (too_broad) feels believable — a real investigator would start losing focus at this point.
- **Five competing threads:** Clearly justifies clarification. Matches Experiment 44's result; no surprise.
- **Transition between three and four:** Understandable. Three threads with one observation is "not enough to decide"; four adds the tipping point where the spread becomes problematic.
- **Confidence language:** `too_broad` confidence is `medium` for both four and five unknowns. The signals are specific ("4 active unknowns with fewer than 2 resolved items"), so medium confidence is honest — it does not overstate certainty.
### Boundary Classification
| Transition | Classification | Rationale |
|---|---|---|
| 2→3 | `believable` | Both remain `cannot_determine`; the gap between "manageable" and "confused" genuinely sits around here |
| 3→4 | `believable` | Four competing threads with no resolution is a believable tipping point for losing focus |
| Resolution threshold (<2 resolved) | `believable` | The binary boundary (1 stays too_broad, 2 clears it) aligns with the design intent of "sufficient context to narrow" |
### Usefulness of Active-Unknown Count as a Proxy
Active-unknown count acts as a **useful but coarse** proxy for scope confusion. It works because:
1. In the tested scenarios, more unknowns directly correlates with genuine ambiguity.
2. The resolved-item gate prevents premature too_broad flags on investigations making progress.
3. It avoids subjective measurement of "how confused is the user."
However, it cannot distinguish between:
- Four unknowns about one decision (genuinely broad) versus four unknowns across a multi-decision comparison (expected).
- A well-formed investigation with natural branching versus an unfocused investigation losing its way.
### Questionable or Unsupported Findings
1. **Health defaults to `cannot_determine` rather than `healthy` for 23 unknowns.** This is mechanically correct (no active question means the "healthy" rule doesn't fire) but arguably should produce `healthy` when the state is simply an early-stage investigation with a few threads, not just insufficient data.
2. **The experiment uses synthetic boundary fixtures.** These cannot validate whether a real user would feel the same confusion at exactly these thresholds. The boundary may be mechanically correct but conceptually misaligned in some domains.
3. **All unknowns share identical labels and confidence values.** A more differentiated scenario (some high-confidence, some low) might behave differently.
### Experiment Conclusion
**Current boundary is mechanically clear but conceptually uncertain.**
The threshold sits exactly between three and four active unknowns. This mechanical boundary behaves predictably: no too_broad below it, too_broad above it, resolved items gate correctly. However, whether this aligns with genuine user confusion (not just code behaviour) cannot be determined from synthetic fixtures alone. The experiment confirms that Clarify switches on at the same boundary as too_broad, and that resolving two items does switch too_broad off.
### Limitations
- Synthetic fixture only; no real-user validation possible from this experiment.
- All unknowns have identical shapes and confidence — real scenarios mix high/low confidence differently.
- Only one central statement used; different domains may require different thresholds.
- Does not test whether the `cannot_determine` health for 23 unknowns is a bug or a feature.
### Status
Pending Rob's review. No production behaviour changed. The next logical step would be: (a) validate whether `cannot_determine` health for 23 unknowns should instead be `healthy`, or (b) test real-user scenarios to confirm the three→four boundary feels right in practice.
### Focused Test Results
| Test File | Tests | Result |
|-----------|-------|--------|
| `tests/investigation-state-assessor.too-broad-boundary.test.js` | 32 | ✓ Pass |
### Regression / Validation Results
| Test File | Tests | Result | Notes |
|-----------|-------|--------|-------|
| `tests/investigation-state-assessor.unclear-start.test.js` | 23 | ✓ Pass | Zero regressions |
| `tests/behaviour-selection.clarify-readiness.test.js` | 31 | ✓ Pass | Zero regressions |
| `tests/investigation-state-assessor.test.js` | 51 | ✓ Pass | Zero regressions |
| `tests/behaviour-selection.test.js` | 51 | ✓ Pass | Zero regressions |
### Production Assessor Status
**Unchanged.** No code was modified. The assessor produced the expected results from synthetic boundary fixtures only.