robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: test assessor against unclear starting point

Browse Source
This commit is contained in:
robbond
2026-08-06 18:31:15 +01:00
parent ae2d1d9c52
commit ca71e79618
3 changed files with 446 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+10 -2
View File
@@ -46,6 +46,8 @@ Experiment 42 implemented Variant B's narrow Acknowledge exclusion gate in the p
Experiment 43 audited Clarify readiness across all 10 real assessment turns in existing fixtures. Zero turns produced Clarify-eligible states. Two findings: (1) the orienting-based Clarify rule is dead code because the assessor never produces phase=orienting, and (2) the too_broad trigger requires conditions no fixture exercises. Branch: `feature/user-workspace-ux-v0.7`.
Experiment 44 created one deliberately unclear starting scenario (five competing unknowns, zero resolved evidence, vague central statement) to test whether the assessor produces a Clarify-justifying signal. The assessor returned `too_broad` conversation health — confirming the previously untested too_broad path works correctly with real data. Clarify became eligible via Rule A. No production code changed. Remaining open: whether orienting phase is needed for earlier-stage clarification, and whether 2–3 competing threads (below the >3 threshold) can represent genuine scope confusion. Status pending Rob's review.
## 5. What Remains Open
- Whether the `too_broad` health trigger needs widening so Clarify fires in more typical investigations;
@@ -57,6 +59,12 @@ Experiment 43 audited Clarify readiness across all 10 real assessment turns in e
- Whether a dedicated fixture validating the `too_broad` trigger should be created before widening its threshold;
- Whether Clarify's intent (genuinely unclear scope) maps better to an existing signal that could fire earlier.
### What Remains Open (continued from above)
- Experiment 44 confirmed the `too_broad` path fires for a genuinely unclear starting point, but only with >3 competing unknowns; scenarios with 2–3 threads may remain unflagged;
- Whether a dedicated `orienting` phase is needed to catch earlier-stage clarification needs (the orienting-based Clarify rule remains dead code);
- Whether Clarify's intent (genuinely unclear scope) maps better to an existing signal that could fire earlier.
### When This Knowledge-Management Phase Is Complete
Provisional criteria for review (all confirmed met by Experiment 38 cold-start test):
@@ -103,6 +111,6 @@ Answer before continuing:
*Created by Experiment 34. Updated by Experiments 38, 39, 40, 41, 42, 43. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 43)
### Return-to-Work Note (Experiment 44)
Experiment 43 diagnosed Clarify's absence across all real scenarios: zero eligibility in 10 turns. The orienting-based rule is dead code (assessor never produces phase=orienting); the too_broad trigger needs a dedicated fixture. Open decision: fix the dead-code path, widen the threshold, or accept both and defer. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `docs/current-handoff.md`, then `tests/behaviour-selection.clarify-readiness.test.js` for the full audit data, and `lib/behaviour-selection/behaviour-selector.js` lines 65-83 for the Clarify rules.
Experiment 44 tested one genuinely unclear starting scenario (five competing unknowns, zero resolved evidence, no clear decision target). The assessor returned `too_broad` for conversation health and Clarify became eligible via the production rule. This confirms the previously untested `too_broad` health classifier path works correctly with real data. No production code changed. What remains unresolved: whether a dedicated `orienting` phase is needed to catch earlier-stage clarification needs, and whether fewer competing threads (2–3) can still represent a genuinely unclear situation that the assessor should flag. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `tests/investigation-state-assessor.unclear-start.test.js` for the fixture, then `docs/design-evolution-log.md` lines 2721 onward for full experiment data, then `lib/assessment/investigation-state-assessor.js` line ~450 for the `too_broad` trigger logic.
docs/design-evolution-log.md
+97
View File
@@ -2717,3 +2717,100 @@ All diagnostics confirm zero Clarify eligibility across the complete set of real
- `docs/design-evolution-log.md` — this entry
- `docs/current-handoff.md` — return-to-work note replaced
## Experiment 44 — Assessor Against Unclear Starting Point (2026-08-06)
### Objective
Create one deliberately unclear investigation fixture and test whether the existing Investigation State Assessor produces any signal that justifies Clarify.
### Hypothesis
A deliberately unclear starting scenario may expose one of three outcomes:
1. The assessor already produces `too_broad`.
2. The assessor produces another existing signal that reasonably represents the need to clarify.
3. The assessor has no suitable signal for unclear framing.
### Fixture Description
**File:** `tests/investigation-state-assessor.unclear-start.test.js` (test-only, not imported anywhere else)
The fixture represents:
- A vague central statement that admits uncertainty: *"The business feels stuck. Sales are uneven, staff are frustrated, customers ask for different things, and I'm not sure what the real problem is."*
- Five competing unknown threads (customer demand, staff capacity, product direction, pricing, operations) with no priority anchor
- Only one observation (the only concrete data point)
- Zero resolved evidence nodes
- No selected question (no established direction)
- All existing graph fields only (id, label, description, kind, status, confidence, evidenceIds, dependsOn, affects, childIds)
- Five `kind: "unknown"` nodes and one `kind: "observation"` node
### Returned Assessment Signals
| Signal | Value | Confidence |
|--------|-------|------------|
| Phase | `cannot_determine` | low |
| Phase signals | "Insufficient data for phase classification" | — |
| Progress | `cannot_determine` | low |
| Progress signals | "Insufficient data for progress assessment" | — |
| Conversation health | **`too_broad`** | medium |
| Health signals | "5 active unknowns with fewer than 2 resolved items"; "Investigation may be spreading too thin" | — |
Detailed evidence:
- Phase evidence: resolvedNodeCount=0, activeUnknownCount=5, observationDensity=1, evidenceDepth="shallow"
- Progress evidence: turnCount=0, recentResolutionsLastTurn=0
- Health evidence: activeUnknownCount=5, resolvedNodeRatio=null, hasActiveQuestion=false
### Clarify Eligibility
**Clarify became eligible via Rule A.** The production `selectClarify` rule fires because `conversationHealth.value === "too_broad"`.
The production selector (`selectBehaviour`) returned:
- behaviour: `"clarify"`
- confidence: `"high"`
- reason: "Conversation health is too broad — investigation may be spreading too thin. Narrow focus through a specific clarification question."
### Interpretation
**Classification: `assessor_recognises_unclear_start`**
The assessor produced `too_broad` from the unclear-start fixture, which directly maps to Clarify's intent (genuinely unclear scope requiring anchoring). The signal honestly reflects the starting situation: five competing unknowns with no resolved evidence and no established direction.
### What the Assessor Recognised
1. Multiple active unknowns without sufficient resolution triggered `too_broad` health classification.
2. The assessor correctly recorded 5 active unknowns in both phase and health evidence sections.
3. Observation density (1) was correctly reported as shallow.
4. Phase confidence remained low due to insufficient data for any meaningful classification.
### What the Assessor Failed to Recognise
1. **`orienting` phase**: Still not produced by the assessor. The orienting-based Clarify rule remains dead code, unchanged from Experiment 43's finding.
2. **Early-stage clarification need**: The `too_broad` trigger only fires after >3 unknowns accumulate — it does not catch a situation with fewer competing threads that is still genuinely unclear in framing.
### Limitations
- Only one fixture was tested. Different vague-scenario configurations may produce different results.
- The `too_broad` trigger depends on having more than 3 active unknowns with fewer than 2 resolved — this specific threshold was exercised, but other boundary conditions (e.g., exactly 4 unknowns, or 5 unknowns with 1 resolved) were not tested.
- The fixture uses the assessor's existing `too_broad` definition which conflates "many unknowns" with "unclear scope." A genuinely unclear scenario with only 2–3 competing threads may not trigger this signal.
### Status
Pending Rob's review. Experiment 43 remains closed — its conclusion that a deliberately unclear fixture was required is confirmed by this experiment, which successfully exercises the previously untested `too_broad` health path.
### Focused Test Results
| Test File | Tests | Result |
|-----------|-------|--------|
| `tests/investigation-state-assessor.unclear-start.test.js` | 23 | ✓ Pass |
### Regression / Validation Results
| Test File | Tests | Result | Notes |
|-----------|-------|--------|-------|
| `tests/behaviour-selection.clarify-readiness.test.js` | 31 | ✓ Pass | Zero regressions |
| `tests/investigation-state-assessor.test.js` | 51 | ✓ Pass | Zero regressions |
| `tests/behaviour-selection.test.js` | 51 | ✓ Pass | Zero regressions |
### Production Assessor Status
**Unchanged.** The assessor produced the expected `too_broad` signal from the unclear fixture, confirming the health classifier path works correctly. No code was modified.