robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: audit clarify readiness signals

Browse Source 
Passive diagnostic: zero Clarify-eligible turns across 10 real-scenario
assessments. Two findings — (1) orienting-based rule is dead code because
assessor never produces phase=orienting, (2) too_broad trigger validly narrow
but untested by any fixture. Created focused test file with 31 assertions.
All regression tests pass: 51 behaviour-selection + 33 reachability + 51
assessor = 166 total.
This commit is contained in:
robbond
2026-08-06 18:22:11 +01:00
parent fc310e77e1
commit ae2d1d9c52
4 changed files with 827 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+12 -4
View File
@@ -42,13 +42,21 @@ Experiment 41 compared two passive alternatives for reducing Acknowledge dominan
- Variant A (priority reordering): evaluate Summarise/Pause before Acknowledge — introduces false-positive summarise in focusing phase
- Variant B (Acknowledge exclusions): keep priority, gate Acknowledge when phase=concluding/synthesising or progress=stalled or health=user_overloaded — recommended
- Both variants converge on the same two genuine changes: concluding→summarise and stalled→pause
Experiment 42 implemented Variant B's narrow Acknowledge exclusion gate in the production selector (commit `05d3d96`). Summarise now appears at conclusion; Pause now appears when stalled. All other tested turns remain unchanged. Behaviour Selection remains passive and isolated with no runtime caller — active user-facing engine behaviour did not change. Clarify remains an open question. Branch: `feature/user-workspace-ux-v0.7`.
Experiment 42 implemented Variant B's narrow Acknowledge exclusion gate in the production selector (commit `05d3d96`). Summarise now appears at conclusion; Pause now appears when stalled. All other tested turns remain unchanged. Behaviour Selection remains passive and isolated with no runtime caller — active user-facing engine behaviour did not change.
Experiment 43 audited Clarify readiness across all 10 real assessment turns in existing fixtures. Zero turns produced Clarify-eligible states. Two findings: (1) the orienting-based Clarify rule is dead code because the assessor never produces phase=orienting, and (2) the too_broad trigger requires conditions no fixture exercises. Branch: `feature/user-workspace-ux-v0.7`.
## 5. What Remains Open
- Whether the `too_broad` health trigger needs widening so Clarify fires in more typical investigations;
- Whether `user_overloaded` health should be producible by the assessor for stalled/inconsistent evidence states.
## 5. What Remains Open
- Whether to fix the orienting dead-code path in the assessor or accept it as intentional design;
- Whether a dedicated fixture validating the `too_broad` trigger should be created before widening its threshold;
- Whether Clarify's intent (genuinely unclear scope) maps better to an existing signal that could fire earlier.
### When This Knowledge-Management Phase Is Complete
Provisional criteria for review (all confirmed met by Experiment 38 cold-start test):
@@ -93,8 +101,8 @@ Answer before continuing:
---
*Created by Experiment 34. Updated by Experiments 38, 39, 40, 41, 42. Branch: `feature/user-workspace-ux-v0.7`.*
*Created by Experiment 34. Updated by Experiments 38, 39, 40, 41, 42, 43. Branch: `feature/user-workspace-ux-v0.7`.*
### Return-to-Work Note (Experiment 42)
### Return-to-Work Note (Experiment 43)
Experiment 42 implemented the narrow Acknowledge exclusion in the production Behaviour Selection module. Summarise now appears at conclusion; Pause now appears when stalled. All other tested turns remain unchanged. Behaviour Selection remains passive and isolated with no runtime caller — active user-facing engine behaviour did not change. Clarify remains unresolved and was not modified in this experiment. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `docs/current-handoff.md`, then `lib/behaviour-selection/behaviour-selector.js` for the exclusion gate, and `tests/behaviour-selection.reachability.test.js` for the final real-turn distribution assertions.
Experiment 43 diagnosed Clarify's absence across all real scenarios: zero eligibility in 10 turns. The orienting-based rule is dead code (assessor never produces phase=orienting); the too_broad trigger needs a dedicated fixture. Open decision: fix the dead-code path, widen the threshold, or accept both and defer. Branch: `feature/user-workspace-ux-v0.7`. First file to inspect when resuming: `docs/current-handoff.md`, then `tests/behaviour-selection.clarify-readiness.test.js` for the full audit data, and `lib/behaviour-selection/behaviour-selector.js` lines 65-83 for the Clarify rules.