From fc310e77e1d019a4e8ff332c3629436c0fae227b Mon Sep 17 00:00:00 2001 From: robbond Date: Thu, 6 Aug 2026 18:00:13 +0100 Subject: [PATCH] docs: close experiment 42 selector refinement --- docs/current-handoff.md | 11 ++--- docs/current-project-state.md | 4 +- docs/design-evolution-log.md | 88 +++++++++++++++++++++++++++++++++++ 3 files changed, 95 insertions(+), 8 deletions(-) diff --git a/docs/current-handoff.md b/docs/current-handoff.md index ec24fe9..239fe03 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -36,17 +36,16 @@ Experiment 38 tested whether a genuinely cold session (no prior conversation con Experiment 39 resumed reasoning experiments with a passive validation of Behaviour Selection against real Investigation State Assessment outputs. Seven turns across three scenarios were evaluated. Acknowledge dominated (71%) because it fires at priority 1 whenever health=healthy, even in terminal and stalled states where Summarise or Pause would be more useful. The assessor→selector contract aligns cleanly; no transformation is needed between pipeline stages. All five behaviours remain reachable but some never appear in typical scenarios (Clarify requires too_broad health which few fixtures produce). Status pending Rob's review. -Experiment 40 diagnosed the root causes: Summarise and Pause fire their rules in real data but are always blocked by Acknowledge's priority-1 position (priority conflict, not assessor failure). Clarive's triggers never activate in tested scenarios due to the `too_broad` health condition being extremely narrow. All five behaviours confirmed independently reachable in synthetic isolation. No rules changed. +Experiment 40 diagnosed the root causes: Summarise and Pause fire their rules in real data but are always blocked by Acknowledge's priority-1 position (priority conflict, not assessor failure). Clarify's triggers never activate in tested scenarios due to the `too_broad` health condition being extremely narrow. All five behaviours confirmed independently reachable in synthetic isolation. No rules changed. Experiment 41 compared two passive alternatives for reducing Acknowledge dominance: - Variant A (priority reordering): evaluate Summarise/Pause before Acknowledge — introduces false-positive summarise in focusing phase - Variant B (Acknowledge exclusions): keep priority, gate Acknowledge when phase=concluding/synthesising or progress=stalled or health=user_overloaded — recommended - Both variants converge on the same two genuine changes: concluding→summarise and stalled→pause -- No production code changed. Branch: `feature/user-workspace-ux-v0.7`. +Experiment 42 implemented Variant B's narrow Acknowledge exclusion gate in the production selector (commit `05d3d96`). Summarise now appears at conclusion; Pause now appears when stalled. All other tested turns remain unchanged. Behaviour Selection remains passive and isolated with no runtime caller — active user-facing engine behaviour did not change. Clarify remains an open question. Branch: `feature/user-workspace-ux-v0.7`. ## 5. What Remains Open -- Implement Variant B's acknowledgment exclusion (recommended path from Exp 41); - Whether the `too_broad` health trigger needs widening so Clarify fires in more typical investigations; - Whether `user_overloaded` health should be producible by the assessor for stalled/inconsistent evidence states. @@ -94,8 +93,8 @@ Answer before continuing: --- -*Created by Experiment 34. Updated by Experiments 38, 39, 40, 41. Branch: `feature/user-workspace-ux-v0.7`.* +*Created by Experiment 34. Updated by Experiments 38, 39, 40, 41, 42. Branch: `feature/user-workspace-ux-v0.7`.* -### Return-to-Work Note (Experiment 41) +### Return-to-Work Note (Experiment 42) -Experiments 39–41 diagnosed Acknowledge dominance in Behaviour Selection and compared two passive alternatives. **Experiment 41 concluded:** Variant B (Acknowledge exclusions via phase/progress/health gates) is recommended over Variant A (priority reordering, which introduces false-positive summarise). Both variants correctly identify the same two genuine changes: `concluding → summarise` and `stalled → pause`. No production code changed. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect: `docs/current-handoff.md`, then `docs/design-evolution-log.md` entry for Experiment 41, and `tests/behaviour-selection.counterfactual.test.js` for variant details. The next action is implementing Variant B's `isAcknowledgeExcluded()` gate as a test-only function, then evaluating against a wider range of assessment scenarios. +Experiment 42 implemented the narrow Acknowledge exclusion in the production Behaviour Selection module. Summarise now appears at conclusion; Pause now appears when stalled. All other tested turns remain unchanged. Behaviour Selection remains passive and isolated with no runtime caller — active user-facing engine behaviour did not change. Clarify remains unresolved and was not modified in this experiment. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `docs/current-handoff.md`, then `lib/behaviour-selection/behaviour-selector.js` for the exclusion gate, and `tests/behaviour-selection.reachability.test.js` for the final real-turn distribution assertions. diff --git a/docs/current-project-state.md b/docs/current-project-state.md index 905c581..474e437 100644 --- a/docs/current-project-state.md +++ b/docs/current-project-state.md @@ -75,7 +75,7 @@ The following were built during Experiments 18–25B. They are isolated diagnost ## 6. Work Currently Paused -- Engine experiments are at Experiment 40 (behaviour reachability diagnostic — no rule changes). +- Engine experiments advanced through Experiment 42 (narrow Acknowledge exclusion gate for behaviour selection; production selector changed but Behaviour Selection remains isolated with no runtime caller). - UI experiments are paused; - Knowledge-management experiments are complete (confirmed by Experiment 38 cold-start validation); - Nothing historical has been deleted or archived yet. @@ -100,7 +100,7 @@ Historical documents are retained under `docs/archive/` and should be opened onl ## 8. Return-to-Work Summary -Engine experiments advanced to Experiment 41, which compared two passive alternatives for reducing Acknowledge dominance. Variant B (Acknowledge exclusions via phase/progress/health gates) is recommended over Variant A (priority reordering with false-positive side effects). Both variants converge on the same two genuine changes: `concluding → summarise` and `stalled → pause`. No production code changed. +Engine experiments advanced through Experiment 42, which implemented Variant B's narrow Acknowledge exclusion gate in the production selector. Summarise now appears at conclusion; Pause now appears when stalled. Behaviour Selection remains passive and isolated with no runtime caller — active user-facing engine behaviour did not change. Clarify remains unresolved. No active tests rerun as part of documentation closure. First document to read: `docs/current-project-state.md`. Then consult `.claude/architecture-guardrails.md` before any code changes and `docs/project-knowledge-inventory.md` for task-specific references. The full experiment history remains available in `docs/design-evolution-log.md` but is no longer default reading. diff --git a/docs/design-evolution-log.md b/docs/design-evolution-log.md index 4c8336e..d184c06 100644 --- a/docs/design-evolution-log.md +++ b/docs/design-evolution-log.md @@ -2494,3 +2494,91 @@ Both variants were implemented as test-only functions in `tests/behaviour-select - `docs/design-evolution-log.md` — this entry - `docs/current-handoff.md` — return-to-work note replaced + +## Experiment 41 — Conclusion + +**Variant B was preferred because it changed only the two intended turns without introducing a false-positive in a focusing state. Variant A produced an early summarise in a focusing phase and was discarded. No production rule changed during Experiment 41.** The implementation of Variant B's exclusion gate is the subject of Experiment 42. + +--- + +## Experiment 42 — Implement Narrow Acknowledge Exclusion (Variant B) (2026-08-06) + +### Hypothesis + +Applying a narrow exclusion gate to Acknowledge — excluding it when phase is synthesising or concluding, progress is stalled, or conversation health is user_overloaded — will reduce the two identified false-Acknowledge selections (concluding → summarise, stalled → pause) without introducing any unintended behaviour changes in other tested turns. + +### Exact Exclusion Rule + +`isAcknowledgeExcluded(assessment)` returns `true` when: +- `phase.value` is `synthesising` or `concluding`; OR +- `progress.value` is `stalled`; OR +- `conversationHealth.value` is `user_overloaded`. + +When excluded, Acknowledge does not fire and the selector proceeds to the next priority rule. The gate qualifies the trigger; it does not replace it. + +### Two Changed Turns + +| Turn | Scenario | Phase | Progress | Health | Before | After | +|------|----------|-------|----------|--------|--------|-------| +| long-investigation t4 | concluding long-investigation | concluding(high) | steady | healthy | acknowledge | **summarise** | +| contradictory-evidence t1 | stalled contradictory-evidence | focusing(high) | stalled | healthy | acknowledge | **pause** | + +### Five Preserved Turns + +| Turn | Scenario | Phase | Progress | Health | Behaviour (unchanged) | +|------|----------|-------|----------|--------|----------------------| +| long-investigation t0 | cannot_determine(low) | cannot_determine | too_narrow | continue | +| long-investigation t3 | focusing(high) | steady | healthy | acknowledge | +| contradictory-evidence t0 | focusing(high) | cannot_determine | healthy | acknowledge | +| contradictory-evidence t2 | focusing(high) | steady | healthy | acknowledge | +| short-early t0 | exploring(low) | cannot_determine | healthy | continue | + +### Final Behaviour Distribution (7 real assessment turns) + +- Acknowledge: 3 +- Summarise: 1 +- Pause: 1 +- Continue: 2 +- Clarify: 0 + +### Integration Status + +The production Behaviour Selection module (`lib/behaviour-selection/behaviour-selector.js`) was changed to include the `isAcknowledgeExcluded()` gate. However, **active user-facing engine behaviour did not change** because Behaviour Selection remains isolated with no runtime caller — it is exported but never imported by any code in the repository. + +### Clarify Status + +`Clarify` remains unresolved and was not modified in this experiment. Its trigger conditions (`health=too_broad` or `phase=orienting + obs<3`) require states that no tested scenario produces. This remains an open question for future work. + +### Selector Output Shape + +The selector output shape did not change. The exclusion gate returns `null` from `selectAcknowledge`, which is the existing early-return mechanism used when a rule does not match. No new fields, no restructuring of the return object. + +### Assessor and Fixtures + +Assessor logic did not change. Fixtures did not change. Priority order did not change. + +### Test Results + +All 151 relevant tests passed across: +- `tests/behaviour-selection.test.js`: 51 (no regressions) +- `tests/behaviour-selection.reachability.test.js`: 33 (updated for new exclusion gate) +- `tests/behaviour-selection.counterfactual.test.js`: 44 (from Exp 41, no changes) +- `tests/behaviour-selection.real-assessment.test.js`: 16 (shared fixtures intact) + +Tests were not rerun as part of this documentation-only closure. The recorded result comes from the implementation commit (05d3d96). + +### Limitations + +- Only seven real assessment turns across three scenarios were evaluated; other investigation domains may exhibit different patterns. +- `health=user_overloaded` is excluded by rule but never produced by any current assessor fixture — it is untested in practice. +- `Clarify` remains deferred because no scenario produces the narrow trigger conditions it requires. +- The selector remains isolated with no runtime caller; there is no live user-facing validation. + +### Result + +**Confirmed within the tested scenarios.** Variant B correctly changes only the two intended turns and preserves all five others. No unintended side-effects were observed. + +### Documents Updated + +- `docs/design-evolution-log.md` — this entry +- `docs/current-handoff.md` — return-to-work note replaced