docs: close experiment 42 selector refinement

This commit is contained in:
2026-08-06 18:00:13 +01:00
parent 05d3d96014
commit fc310e77e1
3 changed files with 95 additions and 8 deletions
+5 -6
View File
@@ -36,17 +36,16 @@ Experiment 38 tested whether a genuinely cold session (no prior conversation con
Experiment 39 resumed reasoning experiments with a passive validation of Behaviour Selection against real Investigation State Assessment outputs. Seven turns across three scenarios were evaluated. Acknowledge dominated (71%) because it fires at priority 1 whenever health=healthy, even in terminal and stalled states where Summarise or Pause would be more useful. The assessor→selector contract aligns cleanly; no transformation is needed between pipeline stages. All five behaviours remain reachable but some never appear in typical scenarios (Clarify requires too_broad health which few fixtures produce). Status pending Rob's review.
Experiment 40 diagnosed the root causes: Summarise and Pause fire their rules in real data but are always blocked by Acknowledge's priority-1 position (priority conflict, not assessor failure). Clarive's triggers never activate in tested scenarios due to the `too_broad` health condition being extremely narrow. All five behaviours confirmed independently reachable in synthetic isolation. No rules changed.
Experiment 40 diagnosed the root causes: Summarise and Pause fire their rules in real data but are always blocked by Acknowledge's priority-1 position (priority conflict, not assessor failure). Clarify's triggers never activate in tested scenarios due to the `too_broad` health condition being extremely narrow. All five behaviours confirmed independently reachable in synthetic isolation. No rules changed.
Experiment 41 compared two passive alternatives for reducing Acknowledge dominance:
- Variant A (priority reordering): evaluate Summarise/Pause before Acknowledge — introduces false-positive summarise in focusing phase
- Variant B (Acknowledge exclusions): keep priority, gate Acknowledge when phase=concluding/synthesising or progress=stalled or health=user_overloaded — recommended
- Both variants converge on the same two genuine changes: concluding→summarise and stalled→pause
- No production code changed. Branch: `feature/user-workspace-ux-v0.7`.
Experiment 42 implemented Variant B's narrow Acknowledge exclusion gate in the production selector (commit `05d3d96`). Summarise now appears at conclusion; Pause now appears when stalled. All other tested turns remain unchanged. Behaviour Selection remains passive and isolated with no runtime caller — active user-facing engine behaviour did not change. Clarify remains an open question. Branch: `feature/user-workspace-ux-v0.7`.
## 5. What Remains Open
- Implement Variant B's acknowledgment exclusion (recommended path from Exp 41);
- Whether the `too_broad` health trigger needs widening so Clarify fires in more typical investigations;
- Whether `user_overloaded` health should be producible by the assessor for stalled/inconsistent evidence states.
@@ -94,8 +93,8 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38, 39, 40, 41. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 38, 39, 40, 41, 42. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 41)
### Return-to-Work Note (Experiment 42)
Experiments 3941 diagnosed Acknowledge dominance in Behaviour Selection and compared two passive alternatives. **Experiment 41 concluded:** Variant B (Acknowledge exclusions via phase/progress/health gates) is recommended over Variant A (priority reordering, which introduces false-positive summarise). Both variants correctly identify the same two genuine changes: `concluding → summarise` and `stalled → pause`. No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `docs/current-handoff.md`, then `docs/design-evolution-log.md` entry for Experiment 41, and `tests/behaviour-selection.counterfactual.test.js` for variant details. The next action is implementing Variant B's `isAcknowledgeExcluded()` gate as a test-only function, then evaluating against a wider range of assessment scenarios.
Experiment 42 implemented the narrow Acknowledge exclusion in the production Behaviour Selection module. Summarise now appears at conclusion; Pause now appears when stalled. All other tested turns remain unchanged. Behaviour Selection remains passive and isolated with no runtime caller — active user-facing engine behaviour did not change. Clarify remains unresolved and was not modified in this experiment. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `docs/current-handoff.md`, then `lib/behaviour-selection/behaviour-selector.js` for the exclusion gate, and `tests/behaviour-selection.reachability.test.js` for the final real-turn distribution assertions.
+2 -2
View File
@@ -75,7 +75,7 @@ The following were built during Experiments 1825B. They are isolated diagnost
## 6. Work Currently Paused
- Engine experiments are at Experiment 40 (behaviour reachability diagnostic — no rule changes).
- Engine experiments advanced through Experiment 42 (narrow Acknowledge exclusion gate for behaviour selection; production selector changed but Behaviour Selection remains isolated with no runtime caller).
- UI experiments are paused;
- Knowledge-management experiments are complete (confirmed by Experiment 38 cold-start validation);
- Nothing historical has been deleted or archived yet.
@@ -100,7 +100,7 @@ Historical documents are retained under `docs/archive/` and should be opened onl
## 8. Return-to-Work Summary
Engine experiments advanced to Experiment 41, which compared two passive alternatives for reducing Acknowledge dominance. Variant B (Acknowledge exclusions via phase/progress/health gates) is recommended over Variant A (priority reordering with false-positive side effects). Both variants converge on the same two genuine changes: `concluding → summarise` and `stalled → pause`. No production code changed.
Engine experiments advanced through Experiment 42, which implemented Variant B's narrow Acknowledge exclusion gate in the production selector. Summarise now appears at conclusion; Pause now appears when stalled. Behaviour Selection remains passive and isolated with no runtime caller — active user-facing engine behaviour did not change. Clarify remains unresolved. No active tests rerun as part of documentation closure.
First document to read: `docs/current-project-state.md`. Then consult `.claude/architecture-guardrails.md` before any code changes and `docs/project-knowledge-inventory.md` for task-specific references. The full experiment history remains available in `docs/design-evolution-log.md` but is no longer default reading.
+88
View File
@@ -2494,3 +2494,91 @@ Both variants were implemented as test-only functions in `tests/behaviour-select
- `docs/design-evolution-log.md` — this entry
- `docs/current-handoff.md` — return-to-work note replaced
## Experiment 41 — Conclusion
**Variant B was preferred because it changed only the two intended turns without introducing a false-positive in a focusing state. Variant A produced an early summarise in a focusing phase and was discarded. No production rule changed during Experiment 41.** The implementation of Variant B's exclusion gate is the subject of Experiment 42.
---
## Experiment 42 — Implement Narrow Acknowledge Exclusion (Variant B) (2026-08-06)
### Hypothesis
Applying a narrow exclusion gate to Acknowledge — excluding it when phase is synthesising or concluding, progress is stalled, or conversation health is user_overloaded — will reduce the two identified false-Acknowledge selections (concluding → summarise, stalled → pause) without introducing any unintended behaviour changes in other tested turns.
### Exact Exclusion Rule
`isAcknowledgeExcluded(assessment)` returns `true` when:
- `phase.value` is `synthesising` or `concluding`; OR
- `progress.value` is `stalled`; OR
- `conversationHealth.value` is `user_overloaded`.
When excluded, Acknowledge does not fire and the selector proceeds to the next priority rule. The gate qualifies the trigger; it does not replace it.
### Two Changed Turns
| Turn | Scenario | Phase | Progress | Health | Before | After |
|------|----------|-------|----------|--------|--------|-------|
| long-investigation t4 | concluding long-investigation | concluding(high) | steady | healthy | acknowledge | **summarise** |
| contradictory-evidence t1 | stalled contradictory-evidence | focusing(high) | stalled | healthy | acknowledge | **pause** |
### Five Preserved Turns
| Turn | Scenario | Phase | Progress | Health | Behaviour (unchanged) |
|------|----------|-------|----------|--------|----------------------|
| long-investigation t0 | cannot_determine(low) | cannot_determine | too_narrow | continue |
| long-investigation t3 | focusing(high) | steady | healthy | acknowledge |
| contradictory-evidence t0 | focusing(high) | cannot_determine | healthy | acknowledge |
| contradictory-evidence t2 | focusing(high) | steady | healthy | acknowledge |
| short-early t0 | exploring(low) | cannot_determine | healthy | continue |
### Final Behaviour Distribution (7 real assessment turns)
- Acknowledge: 3
- Summarise: 1
- Pause: 1
- Continue: 2
- Clarify: 0
### Integration Status
The production Behaviour Selection module (`lib/behaviour-selection/behaviour-selector.js`) was changed to include the `isAcknowledgeExcluded()` gate. However, **active user-facing engine behaviour did not change** because Behaviour Selection remains isolated with no runtime caller — it is exported but never imported by any code in the repository.
### Clarify Status
`Clarify` remains unresolved and was not modified in this experiment. Its trigger conditions (`health=too_broad` or `phase=orienting + obs<3`) require states that no tested scenario produces. This remains an open question for future work.
### Selector Output Shape
The selector output shape did not change. The exclusion gate returns `null` from `selectAcknowledge`, which is the existing early-return mechanism used when a rule does not match. No new fields, no restructuring of the return object.
### Assessor and Fixtures
Assessor logic did not change. Fixtures did not change. Priority order did not change.
### Test Results
All 151 relevant tests passed across:
- `tests/behaviour-selection.test.js`: 51 (no regressions)
- `tests/behaviour-selection.reachability.test.js`: 33 (updated for new exclusion gate)
- `tests/behaviour-selection.counterfactual.test.js`: 44 (from Exp 41, no changes)
- `tests/behaviour-selection.real-assessment.test.js`: 16 (shared fixtures intact)
Tests were not rerun as part of this documentation-only closure. The recorded result comes from the implementation commit (05d3d96).
### Limitations
- Only seven real assessment turns across three scenarios were evaluated; other investigation domains may exhibit different patterns.
- `health=user_overloaded` is excluded by rule but never produced by any current assessor fixture — it is untested in practice.
- `Clarify` remains deferred because no scenario produces the narrow trigger conditions it requires.
- The selector remains isolated with no runtime caller; there is no live user-facing validation.
### Result
**Confirmed within the tested scenarios.** Variant B correctly changes only the two intended turns and preserves all five others. No unintended side-effects were observed.
### Documents Updated
- `docs/design-evolution-log.md` — this entry
- `docs/current-handoff.md` — return-to-work note replaced