experiment: audit clarify readiness signals
Passive diagnostic: zero Clarify-eligible turns across 10 real-scenario assessments. Two findings — (1) orienting-based rule is dead code because assessor never produces phase=orienting, (2) too_broad trigger validly narrow but untested by any fixture. Created focused test file with 31 assertions. All regression tests pass: 51 behaviour-selection + 33 reachability + 51 assessor = 166 total.
This commit is contained in:
@@ -2582,3 +2582,138 @@ Tests were not rerun as part of this documentation-only closure. The recorded re
|
||||
|
||||
- `docs/design-evolution-log.md` — this entry
|
||||
- `docs/current-handoff.md` — return-to-work note replaced
|
||||
|
||||
## Experiment 43 — Audit Clarify Readiness Signals (2026-08-06)
|
||||
|
||||
### Hypothesis
|
||||
|
||||
The existing investigation-state-assessor never produces states that trigger the production Clarify rule in any tested scenario. Clarify is absent from Behaviour Selection not because of a selector defect but because no current fixture represents the genuinely unclear-scoped investigations that its triggers are designed for.
|
||||
|
||||
### Diagnostic Test File
|
||||
|
||||
A focused diagnostic test was created at `tests/behaviour-selection.clarify-readiness.test.js` with 31 assertions auditing every turn across all existing assessor and reachability fixtures. It inspects:
|
||||
- Phase value distribution (focusing, exploring, concluding, synthesising, deepening, cannot_determine)
|
||||
- Conversation health values (healthy, too_narrow, too_broad, user_overloaded)
|
||||
- Observation density per turn
|
||||
- Clarify eligibility via the exact production rule in `selectClarify`
|
||||
|
||||
### Audit Scope
|
||||
|
||||
| Source | Scenarios | Turns Inspected |
|
||||
|--------|-----------|-----------------|
|
||||
| `investigation-state-assessor.test.js` | 7 | 7 (one per scenario) |
|
||||
| `behaviour-selection.reachability.test.js` | 3 | 3 (contradictory-evidence t0, t1, t2) |
|
||||
| **Total** | **10** | **10 real-turn assessments** |
|
||||
|
||||
### Q1 — Does the assessor ever produce `too_broad`?
|
||||
|
||||
**No.** Zero scenarios across all test fixtures produce `conversationHealth.value === "too_broad"`.
|
||||
|
||||
The `too_broad` trigger requires `activeUnknownCount > 3 AND resolvedNodeIds.length < 2`. Every existing scenario starts with exactly one active unknown (the single unresolved question the investigation is about), and the assessor never produces a state where more than three unrelated unknowns coexist without resolution.
|
||||
|
||||
### Q2 — Does the assessor ever produce `phase.value === "orienting"`?
|
||||
|
||||
**No.** Zero scenarios produce orienting. The five phase values produced by the assessor are: concluding, synthesising, focusing, exploring, deepening, and cannot_determine. **`orienting` is not a possible output of any assessor code path.** It does not appear in `assessPhase()`.
|
||||
|
||||
### Q3 — Does orienting ever coincide with observation density < 3?
|
||||
|
||||
**Never applicable.** Since the assessor never produces orienting, this condition cannot arise in real data. The orienting-based Clarify trigger is dead code within the tested scenarios (and likely in production until a scenario change introduces orienting).
|
||||
|
||||
### Q4 — How many turns are Clarify-eligible?
|
||||
|
||||
**Zero of 10 turns.** Both Clarify rules evaluate to false for every assessed turn:
|
||||
- Rule 1 (`too_broad` health): false in all 10 turns
|
||||
- Rule 2 (`orienting + obs<3`): false in all 10 turns (orienting never appears)
|
||||
|
||||
### Q5 — What are the closest existing signals to a genuine Clarify need?
|
||||
|
||||
Two signals approach clarification but do not match its intent:
|
||||
|
||||
| Signal | Turns | Meaning | Maps to Clarify? |
|
||||
|--------|-------|---------|-----------------|
|
||||
| `too_narrow` health | 1 (long-turn-0) | Insufficient contextual evidence for a narrow investigation | No — too_narrow means "needs more data," not "scope is unclear" |
|
||||
| `exploring` phase with low obs density | 1 (complete-turn-0) | Early-stage investigation with sparse observations | No — this signals the start of an investigation, not scope confusion |
|
||||
|
||||
### Q6 — Signal reliability assessment for future Clarify rule design
|
||||
|
||||
| Signal | Reliability for Clarify intent |
|
||||
|--------|-------------------------------|
|
||||
| `too_narrow` health | Low reliability. It reliably indicates insufficient context for question formulation but conflates "too little information" with "unclear scope." The assessor's own description: "The investigation needs more contextual evidence before the current question can be answered effectively." This is about quantity, not clarity. |
|
||||
| `exploring` + low obs density | Low reliability. It reliably indicates an early-stage investigation but does not distinguish between "well-scoped investigation in early phase" and "unclear investigation needing anchoring." Both map to exploring. |
|
||||
|
||||
### Q7 — Is Clarify's absence appropriate for current fixtures?
|
||||
|
||||
**Yes.** Every existing fixture represents a well-defined, focused investigation with a clear central statement:
|
||||
- "Comparing two products before purchase decision" (single question, single dimension)
|
||||
- "Evaluating European market entry" (single strategic question)
|
||||
- "Evaluating $2M procurement against conflicting expert advice" (single decision context)
|
||||
|
||||
A genuinely unclear-scoped investigation would need one of:
|
||||
- A central statement so vague the system cannot classify it into any phase
|
||||
- Multiple unrelated threads at startup with no clear priority anchor
|
||||
- Contradictory framing where the situation itself is ambiguous
|
||||
|
||||
No current fixture represents these states. **Clarify's absence is appropriate because the existing scenarios are genuinely well-scoped, not because the selector is broken.**
|
||||
|
||||
### Phase Distribution Across All 10 Turns
|
||||
|
||||
| Phase | Count | Scenarios |
|
||||
|-------|-------|-----------|
|
||||
| focusing | 7 | comparison t0,t1,t2; long t3; contradictory t0,t1,t2 |
|
||||
| cannot_determine | 1 | long t0 |
|
||||
| concluding | 1 | long t4 |
|
||||
| exploring | 1 | complete t0 |
|
||||
|
||||
No synthesising, deepening, or orienting phases observed.
|
||||
|
||||
### Production Clarify Trigger — Exact Rule Match
|
||||
|
||||
```js
|
||||
// selectClarify (behaviour-selector.js lines 65-83)
|
||||
function selectClarify(assessment) {
|
||||
// Rule A: broad scope detected
|
||||
if (assessment.conversationHealth.value === "too_broad") return clarify;
|
||||
// Rule B: early orientation with sparse data
|
||||
if (assessment.phase.value === "orienting" && assessment.phase.evidence?.observationDensity < 3) return clarify;
|
||||
return null;
|
||||
}
|
||||
```
|
||||
|
||||
**Rule A trigger:** `conversationHealth.value === "too_broad"` — zero occurrences in tested scenarios.
|
||||
**Rule B trigger:** `phase.value === "orienting"` — never produced by assessor; **dead code path.**
|
||||
|
||||
### Focused Test Results (Experiment 43)
|
||||
|
||||
- Total tests: **31**
|
||||
- Passed: **31**
|
||||
- Failed: **0**
|
||||
|
||||
All diagnostics confirm zero Clarify eligibility across the complete set of real-world fixtures.
|
||||
|
||||
### Regression / Validation Results
|
||||
|
||||
| Test File | Tests | Result | Notes |
|
||||
|-----------|-------|--------|-------|
|
||||
| `tests/behaviour-selection.clarify-readiness.test.js` | 31 | ✓ Pass | New diagnostic file — no regression possible |
|
||||
| `tests/behaviour-selection.test.js` | 51 | ✓ Pass | Zero regressions from any prior experiments |
|
||||
| `tests/behaviour-selection.reachability.test.js` | 33 | ✓ Pass | Clarify still eligible in 0 real turns; synthetically reachable |
|
||||
| `tests/investigation-state-assessor.test.js` | 51 | ✓ Pass | Assessor behavior unchanged |
|
||||
|
||||
### Limitations
|
||||
|
||||
- The audit covers all existing test fixtures but not every possible investigation domain. Different problem domains (legal disputes, medical triage, multi-party procurement) may produce different assessor states.
|
||||
- `too_broad` requires very specific conditions (>3 active unknowns with <2 resolved) that no current fixture exercises. A fixture designed specifically to trigger it would validate the health classifier path.
|
||||
- The orienting phase was never produced by any assessor code path in the entire test suite, suggesting a design gap: either orienting was removed from the assessor without updating the selector, or it was never implemented as an active phase value.
|
||||
|
||||
### Conclusion
|
||||
|
||||
**Clarify is absent from Behaviour Selection because no current scenario genuinely needs clarification — not because of a selector defect.** The two production rules are well-formed but their trigger conditions (too_broad health and orienting phase) represent states that the assessor either cannot produce (orienting) or does not produce in any tested fixture (too_broad).
|
||||
|
||||
**Two distinct issues identified:**
|
||||
1. **Dead code path**: The orienting-based Clarify rule never activates because the assessor produces six phase values but none is `orienting`. This is a design inconsistency worth correcting — either add orienting as a real phase or remove that rule from the selector.
|
||||
2. **Narrow trigger threshold**: The too_broad condition (`activeUnknownCount > 3 AND resolvedNodeIds < 2`) is validly narrow but never exercised by any fixture. If Clarify should fire earlier in investigations, the threshold should be relaxed; if it should only fire for genuinely lost investigations, it should stay as-is and a dedicated fixture should validate it.
|
||||
|
||||
### Documents Updated
|
||||
|
||||
- `docs/design-evolution-log.md` — this entry
|
||||
- `docs/current-handoff.md` — return-to-work note replaced
|
||||
|
||||
Reference in New Issue
Block a user