experiment: audit clarify readiness signals
Passive diagnostic: zero Clarify-eligible turns across 10 real-scenario assessments. Two findings — (1) orienting-based rule is dead code because assessor never produces phase=orienting, (2) too_broad trigger validly narrow but untested by any fixture. Created focused test file with 31 assertions. All regression tests pass: 51 behaviour-selection + 33 reachability + 51 assessor = 166 total.
This commit is contained in:
+12
-4
@@ -42,13 +42,21 @@ Experiment 41 compared two passive alternatives for reducing Acknowledge dominan
|
||||
- Variant A (priority reordering): evaluate Summarise/Pause before Acknowledge — introduces false-positive summarise in focusing phase
|
||||
- Variant B (Acknowledge exclusions): keep priority, gate Acknowledge when phase=concluding/synthesising or progress=stalled or health=user_overloaded — recommended
|
||||
- Both variants converge on the same two genuine changes: concluding→summarise and stalled→pause
|
||||
Experiment 42 implemented Variant B's narrow Acknowledge exclusion gate in the production selector (commit `05d3d96`). Summarise now appears at conclusion; Pause now appears when stalled. All other tested turns remain unchanged. Behaviour Selection remains passive and isolated with no runtime caller — active user-facing engine behaviour did not change. Clarify remains an open question. Branch: `feature/user-workspace-ux-v0.7`.
|
||||
Experiment 42 implemented Variant B's narrow Acknowledge exclusion gate in the production selector (commit `05d3d96`). Summarise now appears at conclusion; Pause now appears when stalled. All other tested turns remain unchanged. Behaviour Selection remains passive and isolated with no runtime caller — active user-facing engine behaviour did not change.
|
||||
|
||||
Experiment 43 audited Clarify readiness across all 10 real assessment turns in existing fixtures. Zero turns produced Clarify-eligible states. Two findings: (1) the orienting-based Clarify rule is dead code because the assessor never produces phase=orienting, and (2) the too_broad trigger requires conditions no fixture exercises. Branch: `feature/user-workspace-ux-v0.7`.
|
||||
|
||||
## 5. What Remains Open
|
||||
|
||||
- Whether the `too_broad` health trigger needs widening so Clarify fires in more typical investigations;
|
||||
- Whether `user_overloaded` health should be producible by the assessor for stalled/inconsistent evidence states.
|
||||
|
||||
## 5. What Remains Open
|
||||
|
||||
- Whether to fix the orienting dead-code path in the assessor or accept it as intentional design;
|
||||
- Whether a dedicated fixture validating the `too_broad` trigger should be created before widening its threshold;
|
||||
- Whether Clarify's intent (genuinely unclear scope) maps better to an existing signal that could fire earlier.
|
||||
|
||||
### When This Knowledge-Management Phase Is Complete
|
||||
|
||||
Provisional criteria for review (all confirmed met by Experiment 38 cold-start test):
|
||||
@@ -93,8 +101,8 @@ Answer before continuing:
|
||||
|
||||
---
|
||||
|
||||
*Created by Experiment 34. Updated by Experiments 38, 39, 40, 41, 42. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
*Created by Experiment 34. Updated by Experiments 38, 39, 40, 41, 42, 43. Branch: `feature/user-workspace-ux-v0.7`.*
|
||||
|
||||
### Return-to-Work Note (Experiment 42)
|
||||
### Return-to-Work Note (Experiment 43)
|
||||
|
||||
Experiment 42 implemented the narrow Acknowledge exclusion in the production Behaviour Selection module. Summarise now appears at conclusion; Pause now appears when stalled. All other tested turns remain unchanged. Behaviour Selection remains passive and isolated with no runtime caller — active user-facing engine behaviour did not change. Clarify remains unresolved and was not modified in this experiment. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `docs/current-handoff.md`, then `lib/behaviour-selection/behaviour-selector.js` for the exclusion gate, and `tests/behaviour-selection.reachability.test.js` for the final real-turn distribution assertions.
|
||||
Experiment 43 diagnosed Clarify's absence across all real scenarios: zero eligibility in 10 turns. The orienting-based rule is dead code (assessor never produces phase=orienting); the too_broad trigger needs a dedicated fixture. Open decision: fix the dead-code path, widen the threshold, or accept both and defer. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `docs/current-handoff.md`, then `tests/behaviour-selection.clarify-readiness.test.js` for the full audit data, and `lib/behaviour-selection/behaviour-selector.js` lines 65-83 for the Clarify rules.
|
||||
|
||||
@@ -75,7 +75,7 @@ The following were built during Experiments 18–25B. They are isolated diagnost
|
||||
|
||||
## 6. Work Currently Paused
|
||||
|
||||
- Engine experiments advanced through Experiment 42 (narrow Acknowledge exclusion gate for behaviour selection; production selector changed but Behaviour Selection remains isolated with no runtime caller).
|
||||
- Engine experiments advanced through Experiment 43 (Clarify readiness diagnostic confirming zero Clarify eligibility across all real fixtures; orienting-based rule identified as dead code; too_broad trigger validly narrow but untested in fixtures).
|
||||
- UI experiments are paused;
|
||||
- Knowledge-management experiments are complete (confirmed by Experiment 38 cold-start validation);
|
||||
- Nothing historical has been deleted or archived yet.
|
||||
@@ -100,13 +100,13 @@ Historical documents are retained under `docs/archive/` and should be opened onl
|
||||
|
||||
## 8. Return-to-Work Summary
|
||||
|
||||
Engine experiments advanced through Experiment 42, which implemented Variant B's narrow Acknowledge exclusion gate in the production selector. Summarise now appears at conclusion; Pause now appears when stalled. Behaviour Selection remains passive and isolated with no runtime caller — active user-facing engine behaviour did not change. Clarify remains unresolved. No active tests rerun as part of documentation closure.
|
||||
Engine experiments advanced through Experiment 43, which diagnosed Clarify's absence across all real fixtures (zero eligibility in 10 turns). The orienting-based Clarify rule is dead code — the assessor never produces phase=orienting. The too_broad trigger is validly narrow but untested by any fixture. Summarise and Pause remain operational from Exp 42. Behaviour Selection remains passive and isolated. Open decision: whether to fix the orienting dead-code path or accept it as intentional design, and whether to widen or tighten the too_broad threshold with dedicated fixtures. No active tests rerun as part of documentation closure.
|
||||
|
||||
First document to read: `docs/current-project-state.md`. Then consult `.claude/architecture-guardrails.md` before any code changes and `docs/project-knowledge-inventory.md` for task-specific references. The full experiment history remains available in `docs/design-evolution-log.md` but is no longer default reading.
|
||||
|
||||
## Verification Marker
|
||||
|
||||
Implementation status last checked against source: Experiment 28.
|
||||
Implementation status last checked against source: Experiment 43.
|
||||
The current-state document was verified as accurate by focused code inspection of API routes, orchestrator imports/calls, and cross-module traces for all passive classifiers. No corrections were required.
|
||||
|
||||
**Branch:** `feature/user-workspace-ux-v0.7`
|
||||
|
||||
@@ -2582,3 +2582,138 @@ Tests were not rerun as part of this documentation-only closure. The recorded re
|
||||
|
||||
- `docs/design-evolution-log.md` — this entry
|
||||
- `docs/current-handoff.md` — return-to-work note replaced
|
||||
|
||||
## Experiment 43 — Audit Clarify Readiness Signals (2026-08-06)
|
||||
|
||||
### Hypothesis
|
||||
|
||||
The existing investigation-state-assessor never produces states that trigger the production Clarify rule in any tested scenario. Clarify is absent from Behaviour Selection not because of a selector defect but because no current fixture represents the genuinely unclear-scoped investigations that its triggers are designed for.
|
||||
|
||||
### Diagnostic Test File
|
||||
|
||||
A focused diagnostic test was created at `tests/behaviour-selection.clarify-readiness.test.js` with 31 assertions auditing every turn across all existing assessor and reachability fixtures. It inspects:
|
||||
- Phase value distribution (focusing, exploring, concluding, synthesising, deepening, cannot_determine)
|
||||
- Conversation health values (healthy, too_narrow, too_broad, user_overloaded)
|
||||
- Observation density per turn
|
||||
- Clarify eligibility via the exact production rule in `selectClarify`
|
||||
|
||||
### Audit Scope
|
||||
|
||||
| Source | Scenarios | Turns Inspected |
|
||||
|--------|-----------|-----------------|
|
||||
| `investigation-state-assessor.test.js` | 7 | 7 (one per scenario) |
|
||||
| `behaviour-selection.reachability.test.js` | 3 | 3 (contradictory-evidence t0, t1, t2) |
|
||||
| **Total** | **10** | **10 real-turn assessments** |
|
||||
|
||||
### Q1 — Does the assessor ever produce `too_broad`?
|
||||
|
||||
**No.** Zero scenarios across all test fixtures produce `conversationHealth.value === "too_broad"`.
|
||||
|
||||
The `too_broad` trigger requires `activeUnknownCount > 3 AND resolvedNodeIds.length < 2`. Every existing scenario starts with exactly one active unknown (the single unresolved question the investigation is about), and the assessor never produces a state where more than three unrelated unknowns coexist without resolution.
|
||||
|
||||
### Q2 — Does the assessor ever produce `phase.value === "orienting"`?
|
||||
|
||||
**No.** Zero scenarios produce orienting. The five phase values produced by the assessor are: concluding, synthesising, focusing, exploring, deepening, and cannot_determine. **`orienting` is not a possible output of any assessor code path.** It does not appear in `assessPhase()`.
|
||||
|
||||
### Q3 — Does orienting ever coincide with observation density < 3?
|
||||
|
||||
**Never applicable.** Since the assessor never produces orienting, this condition cannot arise in real data. The orienting-based Clarify trigger is dead code within the tested scenarios (and likely in production until a scenario change introduces orienting).
|
||||
|
||||
### Q4 — How many turns are Clarify-eligible?
|
||||
|
||||
**Zero of 10 turns.** Both Clarify rules evaluate to false for every assessed turn:
|
||||
- Rule 1 (`too_broad` health): false in all 10 turns
|
||||
- Rule 2 (`orienting + obs<3`): false in all 10 turns (orienting never appears)
|
||||
|
||||
### Q5 — What are the closest existing signals to a genuine Clarify need?
|
||||
|
||||
Two signals approach clarification but do not match its intent:
|
||||
|
||||
| Signal | Turns | Meaning | Maps to Clarify? |
|
||||
|--------|-------|---------|-----------------|
|
||||
| `too_narrow` health | 1 (long-turn-0) | Insufficient contextual evidence for a narrow investigation | No — too_narrow means "needs more data," not "scope is unclear" |
|
||||
| `exploring` phase with low obs density | 1 (complete-turn-0) | Early-stage investigation with sparse observations | No — this signals the start of an investigation, not scope confusion |
|
||||
|
||||
### Q6 — Signal reliability assessment for future Clarify rule design
|
||||
|
||||
| Signal | Reliability for Clarify intent |
|
||||
|--------|-------------------------------|
|
||||
| `too_narrow` health | Low reliability. It reliably indicates insufficient context for question formulation but conflates "too little information" with "unclear scope." The assessor's own description: "The investigation needs more contextual evidence before the current question can be answered effectively." This is about quantity, not clarity. |
|
||||
| `exploring` + low obs density | Low reliability. It reliably indicates an early-stage investigation but does not distinguish between "well-scoped investigation in early phase" and "unclear investigation needing anchoring." Both map to exploring. |
|
||||
|
||||
### Q7 — Is Clarify's absence appropriate for current fixtures?
|
||||
|
||||
**Yes.** Every existing fixture represents a well-defined, focused investigation with a clear central statement:
|
||||
- "Comparing two products before purchase decision" (single question, single dimension)
|
||||
- "Evaluating European market entry" (single strategic question)
|
||||
- "Evaluating $2M procurement against conflicting expert advice" (single decision context)
|
||||
|
||||
A genuinely unclear-scoped investigation would need one of:
|
||||
- A central statement so vague the system cannot classify it into any phase
|
||||
- Multiple unrelated threads at startup with no clear priority anchor
|
||||
- Contradictory framing where the situation itself is ambiguous
|
||||
|
||||
No current fixture represents these states. **Clarify's absence is appropriate because the existing scenarios are genuinely well-scoped, not because the selector is broken.**
|
||||
|
||||
### Phase Distribution Across All 10 Turns
|
||||
|
||||
| Phase | Count | Scenarios |
|
||||
|-------|-------|-----------|
|
||||
| focusing | 7 | comparison t0,t1,t2; long t3; contradictory t0,t1,t2 |
|
||||
| cannot_determine | 1 | long t0 |
|
||||
| concluding | 1 | long t4 |
|
||||
| exploring | 1 | complete t0 |
|
||||
|
||||
No synthesising, deepening, or orienting phases observed.
|
||||
|
||||
### Production Clarify Trigger — Exact Rule Match
|
||||
|
||||
```js
|
||||
// selectClarify (behaviour-selector.js lines 65-83)
|
||||
function selectClarify(assessment) {
|
||||
// Rule A: broad scope detected
|
||||
if (assessment.conversationHealth.value === "too_broad") return clarify;
|
||||
// Rule B: early orientation with sparse data
|
||||
if (assessment.phase.value === "orienting" && assessment.phase.evidence?.observationDensity < 3) return clarify;
|
||||
return null;
|
||||
}
|
||||
```
|
||||
|
||||
**Rule A trigger:** `conversationHealth.value === "too_broad"` — zero occurrences in tested scenarios.
|
||||
**Rule B trigger:** `phase.value === "orienting"` — never produced by assessor; **dead code path.**
|
||||
|
||||
### Focused Test Results (Experiment 43)
|
||||
|
||||
- Total tests: **31**
|
||||
- Passed: **31**
|
||||
- Failed: **0**
|
||||
|
||||
All diagnostics confirm zero Clarify eligibility across the complete set of real-world fixtures.
|
||||
|
||||
### Regression / Validation Results
|
||||
|
||||
| Test File | Tests | Result | Notes |
|
||||
|-----------|-------|--------|-------|
|
||||
| `tests/behaviour-selection.clarify-readiness.test.js` | 31 | ✓ Pass | New diagnostic file — no regression possible |
|
||||
| `tests/behaviour-selection.test.js` | 51 | ✓ Pass | Zero regressions from any prior experiments |
|
||||
| `tests/behaviour-selection.reachability.test.js` | 33 | ✓ Pass | Clarify still eligible in 0 real turns; synthetically reachable |
|
||||
| `tests/investigation-state-assessor.test.js` | 51 | ✓ Pass | Assessor behavior unchanged |
|
||||
|
||||
### Limitations
|
||||
|
||||
- The audit covers all existing test fixtures but not every possible investigation domain. Different problem domains (legal disputes, medical triage, multi-party procurement) may produce different assessor states.
|
||||
- `too_broad` requires very specific conditions (>3 active unknowns with <2 resolved) that no current fixture exercises. A fixture designed specifically to trigger it would validate the health classifier path.
|
||||
- The orienting phase was never produced by any assessor code path in the entire test suite, suggesting a design gap: either orienting was removed from the assessor without updating the selector, or it was never implemented as an active phase value.
|
||||
|
||||
### Conclusion
|
||||
|
||||
**Clarify is absent from Behaviour Selection because no current scenario genuinely needs clarification — not because of a selector defect.** The two production rules are well-formed but their trigger conditions (too_broad health and orienting phase) represent states that the assessor either cannot produce (orienting) or does not produce in any tested fixture (too_broad).
|
||||
|
||||
**Two distinct issues identified:**
|
||||
1. **Dead code path**: The orienting-based Clarify rule never activates because the assessor produces six phase values but none is `orienting`. This is a design inconsistency worth correcting — either add orienting as a real phase or remove that rule from the selector.
|
||||
2. **Narrow trigger threshold**: The too_broad condition (`activeUnknownCount > 3 AND resolvedNodeIds < 2`) is validly narrow but never exercised by any fixture. If Clarify should fire earlier in investigations, the threshold should be relaxed; if it should only fire for genuinely lost investigations, it should stay as-is and a dedicated fixture should validate it.
|
||||
|
||||
### Documents Updated
|
||||
|
||||
- `docs/design-evolution-log.md` — this entry
|
||||
- `docs/current-handoff.md` — return-to-work note replaced
|
||||
|
||||
Reference in New Issue
Block a user