experiment: audit clarify readiness signals

Passive diagnostic: zero Clarify-eligible turns across 10 real-scenario
assessments. Two findings — (1) orienting-based rule is dead code because
assessor never produces phase=orienting, (2) too_broad trigger validly narrow
but untested by any fixture. Created focused test file with 31 assertions.
All regression tests pass: 51 behaviour-selection + 33 reachability + 51
assessor = 166 total.
This commit is contained in:
2026-08-06 18:22:11 +01:00
parent fc310e77e1
commit ae2d1d9c52
4 changed files with 827 additions and 7 deletions
+12 -4
View File
@@ -42,13 +42,21 @@ Experiment 41 compared two passive alternatives for reducing Acknowledge dominan
- Variant A (priority reordering): evaluate Summarise/Pause before Acknowledge — introduces false-positive summarise in focusing phase
- Variant B (Acknowledge exclusions): keep priority, gate Acknowledge when phase=concluding/synthesising or progress=stalled or health=user_overloaded — recommended
- Both variants converge on the same two genuine changes: concluding→summarise and stalled→pause
Experiment 42 implemented Variant B's narrow Acknowledge exclusion gate in the production selector (commit `05d3d96`). Summarise now appears at conclusion; Pause now appears when stalled. All other tested turns remain unchanged. Behaviour Selection remains passive and isolated with no runtime caller — active user-facing engine behaviour did not change. Clarify remains an open question. Branch: `feature/user-workspace-ux-v0.7`.
Experiment 42 implemented Variant B's narrow Acknowledge exclusion gate in the production selector (commit `05d3d96`). Summarise now appears at conclusion; Pause now appears when stalled. All other tested turns remain unchanged. Behaviour Selection remains passive and isolated with no runtime caller — active user-facing engine behaviour did not change.
Experiment 43 audited Clarify readiness across all 10 real assessment turns in existing fixtures. Zero turns produced Clarify-eligible states. Two findings: (1) the orienting-based Clarify rule is dead code because the assessor never produces phase=orienting, and (2) the too_broad trigger requires conditions no fixture exercises. Branch: `feature/user-workspace-ux-v0.7`.
## 5. What Remains Open
- Whether the `too_broad` health trigger needs widening so Clarify fires in more typical investigations;
- Whether `user_overloaded` health should be producible by the assessor for stalled/inconsistent evidence states.
## 5. What Remains Open
- Whether to fix the orienting dead-code path in the assessor or accept it as intentional design;
- Whether a dedicated fixture validating the `too_broad` trigger should be created before widening its threshold;
- Whether Clarify's intent (genuinely unclear scope) maps better to an existing signal that could fire earlier.
### When This Knowledge-Management Phase Is Complete
Provisional criteria for review (all confirmed met by Experiment 38 cold-start test):
@@ -93,8 +101,8 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38, 39, 40, 41, 42. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 38, 39, 40, 41, 42, 43. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 42)
### Return-to-Work Note (Experiment 43)
Experiment 42 implemented the narrow Acknowledge exclusion in the production Behaviour Selection module. Summarise now appears at conclusion; Pause now appears when stalled. All other tested turns remain unchanged. Behaviour Selection remains passive and isolated with no runtime caller — active user-facing engine behaviour did not change. Clarify remains unresolved and was not modified in this experiment. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `docs/current-handoff.md`, then `lib/behaviour-selection/behaviour-selector.js` for the exclusion gate, and `tests/behaviour-selection.reachability.test.js` for the final real-turn distribution assertions.
Experiment 43 diagnosed Clarify's absence across all real scenarios: zero eligibility in 10 turns. The orienting-based rule is dead code (assessor never produces phase=orienting); the too_broad trigger needs a dedicated fixture. Open decision: fix the dead-code path, widen the threshold, or accept both and defer. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `docs/current-handoff.md`, then `tests/behaviour-selection.clarify-readiness.test.js` for the full audit data, and `lib/behaviour-selection/behaviour-selector.js` lines 65-83 for the Clarify rules.
+3 -3
View File
@@ -75,7 +75,7 @@ The following were built during Experiments 1825B. They are isolated diagnost
## 6. Work Currently Paused
- Engine experiments advanced through Experiment 42 (narrow Acknowledge exclusion gate for behaviour selection; production selector changed but Behaviour Selection remains isolated with no runtime caller).
- Engine experiments advanced through Experiment 43 (Clarify readiness diagnostic confirming zero Clarify eligibility across all real fixtures; orienting-based rule identified as dead code; too_broad trigger validly narrow but untested in fixtures).
- UI experiments are paused;
- Knowledge-management experiments are complete (confirmed by Experiment 38 cold-start validation);
- Nothing historical has been deleted or archived yet.
@@ -100,13 +100,13 @@ Historical documents are retained under `docs/archive/` and should be opened onl
## 8. Return-to-Work Summary
Engine experiments advanced through Experiment 42, which implemented Variant B's narrow Acknowledge exclusion gate in the production selector. Summarise now appears at conclusion; Pause now appears when stalled. Behaviour Selection remains passive and isolated with no runtime caller — active user-facing engine behaviour did not change. Clarify remains unresolved. No active tests rerun as part of documentation closure.
Engine experiments advanced through Experiment 43, which diagnosed Clarify's absence across all real fixtures (zero eligibility in 10 turns). The orienting-based Clarify rule is dead code — the assessor never produces phase=orienting. The too_broad trigger is validly narrow but untested by any fixture. Summarise and Pause remain operational from Exp 42. Behaviour Selection remains passive and isolated. Open decision: whether to fix the orienting dead-code path or accept it as intentional design, and whether to widen or tighten the too_broad threshold with dedicated fixtures. No active tests rerun as part of documentation closure.
First document to read: `docs/current-project-state.md`. Then consult `.claude/architecture-guardrails.md` before any code changes and `docs/project-knowledge-inventory.md` for task-specific references. The full experiment history remains available in `docs/design-evolution-log.md` but is no longer default reading.
## Verification Marker
Implementation status last checked against source: Experiment 28.
Implementation status last checked against source: Experiment 43.
The current-state document was verified as accurate by focused code inspection of API routes, orchestrator imports/calls, and cross-module traces for all passive classifiers. No corrections were required.
**Branch:** `feature/user-workspace-ux-v0.7`
+135
View File
@@ -2582,3 +2582,138 @@ Tests were not rerun as part of this documentation-only closure. The recorded re
- `docs/design-evolution-log.md` — this entry
- `docs/current-handoff.md` — return-to-work note replaced
## Experiment 43 — Audit Clarify Readiness Signals (2026-08-06)
### Hypothesis
The existing investigation-state-assessor never produces states that trigger the production Clarify rule in any tested scenario. Clarify is absent from Behaviour Selection not because of a selector defect but because no current fixture represents the genuinely unclear-scoped investigations that its triggers are designed for.
### Diagnostic Test File
A focused diagnostic test was created at `tests/behaviour-selection.clarify-readiness.test.js` with 31 assertions auditing every turn across all existing assessor and reachability fixtures. It inspects:
- Phase value distribution (focusing, exploring, concluding, synthesising, deepening, cannot_determine)
- Conversation health values (healthy, too_narrow, too_broad, user_overloaded)
- Observation density per turn
- Clarify eligibility via the exact production rule in `selectClarify`
### Audit Scope
| Source | Scenarios | Turns Inspected |
|--------|-----------|-----------------|
| `investigation-state-assessor.test.js` | 7 | 7 (one per scenario) |
| `behaviour-selection.reachability.test.js` | 3 | 3 (contradictory-evidence t0, t1, t2) |
| **Total** | **10** | **10 real-turn assessments** |
### Q1 — Does the assessor ever produce `too_broad`?
**No.** Zero scenarios across all test fixtures produce `conversationHealth.value === "too_broad"`.
The `too_broad` trigger requires `activeUnknownCount > 3 AND resolvedNodeIds.length < 2`. Every existing scenario starts with exactly one active unknown (the single unresolved question the investigation is about), and the assessor never produces a state where more than three unrelated unknowns coexist without resolution.
### Q2 — Does the assessor ever produce `phase.value === "orienting"`?
**No.** Zero scenarios produce orienting. The five phase values produced by the assessor are: concluding, synthesising, focusing, exploring, deepening, and cannot_determine. **`orienting` is not a possible output of any assessor code path.** It does not appear in `assessPhase()`.
### Q3 — Does orienting ever coincide with observation density < 3?
**Never applicable.** Since the assessor never produces orienting, this condition cannot arise in real data. The orienting-based Clarify trigger is dead code within the tested scenarios (and likely in production until a scenario change introduces orienting).
### Q4 — How many turns are Clarify-eligible?
**Zero of 10 turns.** Both Clarify rules evaluate to false for every assessed turn:
- Rule 1 (`too_broad` health): false in all 10 turns
- Rule 2 (`orienting + obs<3`): false in all 10 turns (orienting never appears)
### Q5 — What are the closest existing signals to a genuine Clarify need?
Two signals approach clarification but do not match its intent:
| Signal | Turns | Meaning | Maps to Clarify? |
|--------|-------|---------|-----------------|
| `too_narrow` health | 1 (long-turn-0) | Insufficient contextual evidence for a narrow investigation | No — too_narrow means "needs more data," not "scope is unclear" |
| `exploring` phase with low obs density | 1 (complete-turn-0) | Early-stage investigation with sparse observations | No — this signals the start of an investigation, not scope confusion |
### Q6 — Signal reliability assessment for future Clarify rule design
| Signal | Reliability for Clarify intent |
|--------|-------------------------------|
| `too_narrow` health | Low reliability. It reliably indicates insufficient context for question formulation but conflates "too little information" with "unclear scope." The assessor's own description: "The investigation needs more contextual evidence before the current question can be answered effectively." This is about quantity, not clarity. |
| `exploring` + low obs density | Low reliability. It reliably indicates an early-stage investigation but does not distinguish between "well-scoped investigation in early phase" and "unclear investigation needing anchoring." Both map to exploring. |
### Q7 — Is Clarify's absence appropriate for current fixtures?
**Yes.** Every existing fixture represents a well-defined, focused investigation with a clear central statement:
- "Comparing two products before purchase decision" (single question, single dimension)
- "Evaluating European market entry" (single strategic question)
- "Evaluating $2M procurement against conflicting expert advice" (single decision context)
A genuinely unclear-scoped investigation would need one of:
- A central statement so vague the system cannot classify it into any phase
- Multiple unrelated threads at startup with no clear priority anchor
- Contradictory framing where the situation itself is ambiguous
No current fixture represents these states. **Clarify's absence is appropriate because the existing scenarios are genuinely well-scoped, not because the selector is broken.**
### Phase Distribution Across All 10 Turns
| Phase | Count | Scenarios |
|-------|-------|-----------|
| focusing | 7 | comparison t0,t1,t2; long t3; contradictory t0,t1,t2 |
| cannot_determine | 1 | long t0 |
| concluding | 1 | long t4 |
| exploring | 1 | complete t0 |
No synthesising, deepening, or orienting phases observed.
### Production Clarify Trigger — Exact Rule Match
```js
// selectClarify (behaviour-selector.js lines 65-83)
function selectClarify(assessment) {
// Rule A: broad scope detected
if (assessment.conversationHealth.value === "too_broad") return clarify;
// Rule B: early orientation with sparse data
if (assessment.phase.value === "orienting" && assessment.phase.evidence?.observationDensity < 3) return clarify;
return null;
}
```
**Rule A trigger:** `conversationHealth.value === "too_broad"` — zero occurrences in tested scenarios.
**Rule B trigger:** `phase.value === "orienting"` — never produced by assessor; **dead code path.**
### Focused Test Results (Experiment 43)
- Total tests: **31**
- Passed: **31**
- Failed: **0**
All diagnostics confirm zero Clarify eligibility across the complete set of real-world fixtures.
### Regression / Validation Results
| Test File | Tests | Result | Notes |
|-----------|-------|--------|-------|
| `tests/behaviour-selection.clarify-readiness.test.js` | 31 | ✓ Pass | New diagnostic file — no regression possible |
| `tests/behaviour-selection.test.js` | 51 | ✓ Pass | Zero regressions from any prior experiments |
| `tests/behaviour-selection.reachability.test.js` | 33 | ✓ Pass | Clarify still eligible in 0 real turns; synthetically reachable |
| `tests/investigation-state-assessor.test.js` | 51 | ✓ Pass | Assessor behavior unchanged |
### Limitations
- The audit covers all existing test fixtures but not every possible investigation domain. Different problem domains (legal disputes, medical triage, multi-party procurement) may produce different assessor states.
- `too_broad` requires very specific conditions (>3 active unknowns with <2 resolved) that no current fixture exercises. A fixture designed specifically to trigger it would validate the health classifier path.
- The orienting phase was never produced by any assessor code path in the entire test suite, suggesting a design gap: either orienting was removed from the assessor without updating the selector, or it was never implemented as an active phase value.
### Conclusion
**Clarify is absent from Behaviour Selection because no current scenario genuinely needs clarification — not because of a selector defect.** The two production rules are well-formed but their trigger conditions (too_broad health and orienting phase) represent states that the assessor either cannot produce (orienting) or does not produce in any tested fixture (too_broad).
**Two distinct issues identified:**
1. **Dead code path**: The orienting-based Clarify rule never activates because the assessor produces six phase values but none is `orienting`. This is a design inconsistency worth correcting — either add orienting as a real phase or remove that rule from the selector.
2. **Narrow trigger threshold**: The too_broad condition (`activeUnknownCount > 3 AND resolvedNodeIds < 2`) is validly narrow but never exercised by any fixture. If Clarify should fire earlier in investigations, the threshold should be relaxed; if it should only fire for genuinely lost investigations, it should stay as-is and a dedicated fixture should validate it.
### Documents Updated
- `docs/design-evolution-log.md` — this entry
- `docs/current-handoff.md` — return-to-work note replaced