## Experiment 35 — Test Current Handoff Maintenance (2026-08-06) **Hypothesis:** A current handoff can remain useful if it describes only the latest stopping point, replaces stale details rather than appending history, and identifies the latest confirmed experiment and commit unambiguously. **Stale or ambiguous wording found:** - Section 1 named Experiment 33 and commit `b959cfa` as the current state — now stale after Experiments 34+35; - Section 4 described only Experiment 33's completion, giving no indication that a single handoff had been created in Experiment 34; - No explicit mention of commit `1d92aa0` anywhere in the handoff; - Footer said "Created by Experiment 34" without acknowledging this maintenance experiment. **Corrections made:** - Section 1: updated to name Experiment 34 and commit `1d92aa0`; added the maintenance principle ("replace stale details rather than appending history"); - Section 4: rewritten to describe Experiment 34's consolidation work; - Section 5: retained one genuinely open question about handoff longevity; added provisional KM completion criteria sub-section (7 criteria, marked provisional); - Footer: updated to reference Experiment 35; added Return-to-Work Note recording all current state. **Fresh-return test result:** PASS — from `current-handoff.md` and `task-context-packs.md` only, a fresh session can determine: - Latest completed KM experiment: Experiment 34 ✓ - Latest commit: `1d92aa0` ✓ - Knowledge-management active, engine/UI paused ✓ - Knowledge-Management context pack is the correct routing target ✓ - No need to open full design history ✓ - Older commits not mistaken for current stopping point ✓ **Provisional completion criteria added:** Seven criteria recorded in Section 5 (see above). Not yet declared complete — pending Rob's review. **Handoff remained concise?** Yes. 86 lines (was 68). Increase justified by the maintenance principle paragraph, updated current-state wording, and provisional completion criteria section. No historical timeline appended. **Status:** Pending Rob's review. ## Experiment 36 — Validate Reduced Context Routing **Branch:** `feature/user-workspace-ux-v0.7` ### Hypothesis The documentation system (handoff + project-state + task-context-packs) is complete enough to support normal work without silently expanding into historical documentation. A fresh session can complete representative tasks using only routing instructions. ### Initial Documents Loaded (328 lines total) 1. `docs/current-handoff.md` — 86 lines 2. `docs/current-project-state.md` — 132 lines 3. `docs/task-context-packs.md` — 110 lines ### Additional Documents Loaded | Document | Lines | Why Needed | Routing Should Include? | |---|---|---|---| | `docs/ui-mock-reference.md` | 63 | Task 2: verify mock scenarios for "long investigation" and "contradiction". Routing Test B claimed these were identifiable without loading it, but the specific scenario names do not appear in any initial document. | YES — routing defect found | | `docs/project-knowledge-inventory.md` | 215 | Task 4: confirm Engine Experiment pack's four always-read documents actually exist and understand KM phase outputs. | Debated — validated completeness but not strictly required by routing | | `docs/current-implementation-verification.md` | 111 | Cross-checked Behaviour Selection isolation against current-project-state §3. Provided corroboration but was not the sole basis for Task 1 answer. | Debated — useful corroboration; current-project-state alone sufficed | ### Tasks Completed Without Context Expansion **Task 1 — Does Behaviour Selection affect engine behaviour?** No. Current project state §3 classifies it as isolated. Handoff §2 confirms passive classifiers don't control the investigation. Task-context-packs Routing Test A corroborates (current-implementation-verification §3b). **Task 3 — Why passive classifiers are not yet in the active reasoning loop?** Passive classifiers record diagnostic signals for future use but have no integration into the turn cycle. Only investigation-state assessment is called (at 3 orchestrator sites), and its result goes into a diagnostics field — never checked by conditional branches. Others have zero callers. ### Tasks Requiring Extra Context **Task 2 — Mock scenarios for long investigation and contradictory evidence** Required `docs/ui-mock-reference.md`. Routing Test B in task-context-packs claimed these were identifiable without loading it, but the specific scenario names ("Long investigation (10–15 turns)" and "Contradiction") do not appear in any initial document. The routing claim was unverifiable until the mock reference was loaded — this is a genuine routing defect. **Task 4 — Where should a new developer begin for the next engine experiment?** Partially answered from initial documents (handoff → project-state → pack). Marginal need to verify that all four always-read pack documents actually exist, resolved by cross-referencing project-knowledge-inventory. ### Routing Failures Found One genuine failure: **Routing Test B in task-context-packs.md**. The test states that mock scenarios for long investigation and contradiction are identifiable without loading ui-mock-reference.md. This was presented as a self-evident fact but the specific scenario names only exist in ui-mock-reference.md. The routing is incomplete — it should have included the mock reference file, or at minimum acknowledged that scenario names require verification. ### Documentation Changes Made - Created `docs/context-routing-validation.md` (62 lines) — this experiment's record - Updated `docs/design-evolution-log.md` — appended Experiment 36 entry **No source code or tests changed. No archive changes.** ### Overall Assessment: Mostly ready Two of four tasks completed from initial context only. One routing defect found (Task 2; corrected by Experiment 37). After fixing Routing Test B to name ui-mock-reference.md as the scenario source, the reduced context system is ready for normal work. --- ## Experiment 37 — Validate Cross-Boundary Context Routing **Branch:** `feature/user-workspace-ux-v0.7` ### Hypothesis The context-pack system can support cross-boundary work if Claude: 1. starts with one primary pack; 2. adds a second pack only for a named boundary; 3. records why each extra document was loaded; 4. avoids loading the full history. ### Initial Documents Loaded (328 lines total) 1. `docs/current-handoff.md` — 85 lines; first return-to-work entry point 2. `docs/current-project-state.md` — 131 lines; active state and capabilities 3. `docs/task-context-packs.md` — 110 lines; routing for four work types ### Additional Documents Loaded | Document | Lines | Why Needed | Routing Should Include? | |---|---|---|---| | `docs/ui-mock-reference.md` | 62 | Cross-boundary boundary: the task requires identifying a mock scenario for workspace display. This is the second pack (UI and Mock) needed because no other loaded document names scenarios or UI fixtures. Yes — it is part of the UI/Mock pack, not an ad-hoc addition. | ### Cross-Boundary Task Result **Task:** Display passive condition-status information in the workspace for a mock investigation without changing the active reasoning loop. | Finding | Details | |---|---| | Condition-status capability | Passive: decision-condition status evaluation records signals but has no integration into the turn cycle; never controls user-facing decisions or path selection | | Active reasoning loop | Unchanged: deterministic pipeline (scenario reconstruction → graph update → unknown selection → question formulation → turn orchestration); none of these pathways are affected by passive data | | Mock scenario | "Long investigation (10–15 turns)" from `ui-mock-reference.md`; workspace can display accumulated diagnostic signals over time without interrupting the active reasoning cycle | | Implementation areas to inspect later | decision-condition-status evaluation module; evidence scope detection module; UI workspace components for passive display integration | | Both packs genuinely needed? | Yes: Engine pack identifies which capabilities are active vs passive; UI pack identifies how the workspace presents state. Neither alone suffices | | Archive or full history required? | No | **Context remained manageable:** Yes. 390 lines total (328 initial + 62 additional). Each document loaded for a specific named purpose. No blind expansion. ### Knowledge-Management Completion Criteria Review | Criterion | Status | |---|---| | 1. Fresh session can resume from handoff + one pack | met | | 2. Current state verified against implementation | met | | 3. Historical material outside default loading | met | | 4. Current principles separated from aspirational architecture | met | | 5. Task-specific routing works for engine and UI tasks | met | | 6. Cross-boundary task tested | **met** | | 7. Maintaining handoff does not require reading full history | met | All seven criteria are now met. > Knowledge-management structure is ready for Rob's review before engine experiments resume. ### Routing Defects Discovered None in this experiment. The correction to Routing Test B (naming `ui-mock-reference.md` as the scenario source) was applied before testing. No new defects found in the cross-boundary test. ### Overall Assessment: Ready The context-pack system handled a genuine engine/UI cross-boundary task by combining two packs deliberately with full documentation of each loaded document and its purpose. Context remained small (390 lines). All knowledge-management criteria are met. --- # Experiment 38 — Cold-Start Project Recovery Validation **Branch:** `feature/user-workspace-ux-v0.7` **Type:** Knowledge-management / handoff validation (final KM experiment) **Objective:** Test whether a genuinely cold session can recover the project accurately from the reduced context system alone without reading the full history or any earlier experiment reports. ## Setup Cold-start configuration: no prior conversation context, no past experiment reports loaded, repository documentation carries all context. Session was freshly created to simulate a real return-to-work scenario. Only `docs/current-handoff.md` was read first (per handoff §6 step 1), then the two documents specified by its resume instructions (§6 steps 2–3): `docs/current-project-state.md` and `docs/task-context-packs.md`. ## Documents Loaded | Document | Reason | |---|---| | `docs/current-handoff.md` | Primary entry point (handoff §6 step 1) | | `docs/current-project-state.md` | Resume instruction (§6 step 2) and routing table (§6 step 7) | | `docs/task-context-packs.md` | Pack selection (§6 step 3) and pack contents for verification | No additional documents were loaded. No blind expansion occurred. The full design-evolution log, archived documents, UI mock reference, source code, and tests were all excluded by design. ## Project-State Recovery Result The cold session correctly recovered: - What the Confidence Engine does (facilitated investigation with structured reasoning graph). - Active capabilities: deterministic reasoning pipeline, unknown selection via atomicity/answerability, question formulation, scenario API, turn cycle orchestration. - Passive capabilities: seven diagnostic layers from Experiments 18–25B, all isolated, none control user-facing investigation. - Paused work: engine experiments (after Exp 25B), UI experiments. - Why KM phase was undertaken (documentation bloat blocking session recovery). Recovery score: complete from three documents alone. No source code inspection required. ## Context-Pack Selection Result Pack 1 — Engine Experiment Work selected correctly by the cold session. The three initial documents contained sufficient information to identify the pack, its default documents, and what to exclude without reading any additional material. ## Handoff Defects Found None found in `docs/current-handoff.md`. The handoff accurately describes the stopping point, identifies all seven KM criteria as met, provides correct resume instructions, and includes accurate capability boundaries. One structural update was made: the open item "whether the handoff stays accurate after further advances" was resolved as no longer applicable (the cold-start test confirmed it is accurate). ## Completion-Criteria Result All seven knowledge-management completion criteria are confirmed met by this cold-start validation: 1. Fresh session can resume from handoff + one pack — met (Exp 38 demonstrates this) 2. Current state verified against implementation — met (Exp 28+) 3. Historical material outside default loading — met 4. Current principles separated from aspirational architecture — met 5. Task-specific routing works for engine and UI tasks — met (Exp 37) 6. Cross-boundary task tested — met (Exp 37) 7. Maintaining handoff does not require reading full history — met > The knowledge-management phase is complete enough for Rob to choose when engine experiments resume. ## Documents Updated - `docs/cold-start-validation.md` — created (this experiment's deliverable) - `docs/current-handoff.md` — Exp 38 commit placeholder, structural open-item resolution, return-to-work note replacement - `docs/current-project-state.md` — KM status update ("active" → "complete"), latest known commit correction - `docs/design-evolution-log.md` — this entry ## Overall Assessment: Ready The cold-start validation passed. A genuinely fresh session understood the project state, chose the correct context pack, verified the resume boundary, produced a valid engine-work resume brief, and found no handoff defects — all from three documents alone. No source code was read or changed. The reduced context system works for sessions that did not help create the documents. Engine and UI experiments remain paused pending Rob's review. --- ## Experiment 39 — Validate Behaviour Selection Against Real Assessment Outputs (2026-08-06) **Branch:** feature/user-workspace-ux-v0.7 ### Hypothesis The existing deterministic selector produces a useful rhythm across genuine assessment outputs without changing the active engine. If it repeatedly chooses one behaviour, chooses behaviours at the wrong time, or depends on signals the assessor does not actually produce, the experiment should expose that honestly. ### Scenarios Evaluated (from `tests/investigation-state-assessor.test.js` fixture set) 1. **Long investigation** (3 turns: early → deepening → complete terminal) 2. **Contradictory evidence** (3 turns: two conflicting consultants, 0→1→2 resolved unknowns) 3. **Short early** (1 turn: two observations, first unknown, no resolution) ### Behaviour Distribution (7 turns total) - Acknowledge: 5 (71%) - Continue: 2 (29%) - Clarify: 0 (0%) - Summarise: 0 (0%) - Pause: 0 (0%) ### Behaviour Sequence by Scenario **Long investigation:** continue → acknowledge → acknowledge - Turn 0: phase=cannot_determine, progress=cannot_determine, health=too_narrow → continue (no rule matched) - Turn 3: phase=focusing, progress=steady, health=healthy → acknowledge - Turn 4: phase=concluding, progress=steady, health=healthy → acknowledge **Contradictory evidence:** acknowledge → acknowledge → acknowledge - Turn 0: phase=focusing, progress=cannot_determine, health=healthy → acknowledge - Turn 1: phase=focusing, progress=stalled, health=healthy → acknowledge - Turn 2: phase=focusing, progress=steady, health=healthy → acknowledge **Short early:** continue - Turn 0: phase=exploring, progress=cannot_determine, health=healthy → continue ### Sensible Selections (7 of 7) All selections were classified as sensible per the selection's stated conditions. Acknowledge fires because `health=healthy AND phase confidence≠low` across most states. Continue fires when no specific rule matches (early/cannot_determine/exploring phases). ### Questionable or Inappropriate Selections **One notable pattern:** Summarise and Pause never fire, even in a concluding terminal state. This is not because the assessor fails to detect "concluding" — it does. It is because Acknowledge (priority 1) fires first when health=healthy, blocking Summarise (priority 3) from ever reaching its turn. This is an **acknowledgement/summarise priority conflict**: acknowledging a conclusion ("you've figured this out!") is not wrong, but "give me a summary" is more useful at terminal states. The current rule ordering does not distinguish "early healthy" from "concluding healthy." Clarify never fires because no test scenario produces `health=too_broad` — the assessor's "too_broad" trigger (activeUnknownCount > 3 AND resolved < 2) requires more nodes than any scenario in the fixture set has at that stage. Pause never fires because `health=user_overloaded` is never reached, and while contradictory-turn-1 has phase=focusing + progress=stalled, Acknowledge still blocks it. ### Contract Alignment Assessor → Selector contract aligns cleanly. The assessor produces all three dimensions (phase, progress, conversationHealth) with the fields the selector expects. No transformation needed between pipeline stages. ### Whether Selector Appears Useful Enough for Another Passive Experiment The existing selector works but its **behaviour variation is severely constrained by Acknowledge's priority position**. A next passive experiment should test whether reordering or refining the acknowledge condition (e.g., excluding concluding/terminal phases) produces more context-appropriate behaviour — without changing the assessor. ### Status Pending Rob's review. Five behaviours are too narrow for this to be definitive, and only three scenarios were tested. The dominant pattern (acknowledge in healthy states) may change with different investigation domains. ### Documents Updated - `docs/design-evolution-log.md` — this entry - `docs/current-handoff.md` — return-to-work note replaced --- ## Experiment 40 — Audit Behaviour Reachability and Blocking (2026-08-06) ### Objective Why did Clarify, Summarise, and Pause not appear during Experiment 39? Acknowledge: 5 (71%), Continue: 2 (29%), others: 0. This is a passive diagnostic — no rule changes, no engine modifications. ### Method One test file (`tests/behaviour-selection.reachability.test.js`) containing: - Diagnostic audit helper that evaluates every behaviour rule against one assessment object - Real-scenario audits across the same Experiment 39 turns (8 turns total) - Synthetic reachability checks for each behaviour in isolation ### Findings #### Summarise — eligible_but_blocked Eligible in 2 of 7 real turns: - long-investigation turn 1 (resolvedNodeCount ≥ 3 + progress=steady triggers summarise rule) - long-investigation turn 2 (phase=concluding triggers summarise rule) In both cases, health=healthy simultaneously, so Acknowledge (priority 1) fires first. Summarise rules are met but its output is never returned because the selector returns early on priority ordering. **Root cause: priority conflict, not assessor failure.** The phase evidence correctly identifies concluding/synthesising states; the problem is that Acknowledge's broader trigger condition (health=healthy is the most common state) fires first. #### Clarify — never_eligible_in_tested_scenarios (reachable only in synthetic case) Not eligible in any of 7 real turns because neither trigger condition is met: - `health=too_broad`: requires activeUnknownCount > 3 AND resolvedNodeCount < 2 — no fixture reaches this state - `phase=orienting + observationDensity < 3`: current assessor never produces phase=orienting for tested scenarios Synthetic case confirms the rule fires correctly in isolation (with low-confidence phase to avoid Acknowledge blocking). **Root cause: assessor health classification logic produces too few `too_broad` cases. The trigger condition is extremely narrow — needs activeUnknownCount > 3 AND resolved < 2 simultaneously.** #### Pause — eligible_but_blocked Eligible in 1 of 7 real turns: - contradictory-evidence turn 1 (phase=focusing + progress=stalled triggers pause rule) In this case, health=healthy simultaneously, so Acknowledge blocks it. The second pause trigger (`health=user_overloaded`) is never met because the assessor never produces that state. **Root cause: same priority conflict as Summarise. One of two rules fires in real data but gets blocked by Acknowledge's earlier position.** ### Synthetic Reachability Confirmation All five behaviours are independently reachable when isolated from Acknowledge: - ✅ acknowledge — healthy + confident phase - ✅ clarify — too_broad health (with low-confidence phase to avoid Acknowledge) - ✅ summarise — synthesising/concluding phase (without healthy health) - ✅ pause — focusing+stalled or user_overloaded (without healthy health) - ✅ continue — no rules match ### Classifications | Behaviour | Classification | Primary Cause | |---|---|---| | Summarise | eligible_but_blocked | Acknowledge priority 1 fires first when health=healthy | | Clarify | never_eligible_in_tested_scenarios (reachable only in synthetic) | `too_broad` trigger too narrow for test scenarios; `orienting+low obs` not produced by assessor | | Pause | eligible_but_blocked | Acknowledge priority 1 fires first when health=healthy; `user_overloaded` never produced | ### Impact on Prior Finding (Exp 39) Experiment 39 concluded "the Acknowledge→Summarise priority conflict prevents Summarise from firing." Experiment 40 confirms this and adds that **Pause faces the same blocking** (1 eligible turn, blocked). Clarify's absence is fundamentally different: its rules are not triggered at all in tested scenarios. This means any fix must address two distinct problems: 1. Priority conflict affecting Summarise AND Pause (same cause) 2. Narrow trigger conditions for Clarify and the `user_overloaded` health state ### Test Results - `tests/behaviour-selection.reachability.test.js`: 33 passed (new diagnostic file) - `tests/behaviour-selection.test.js`: 51 passed (no regressions) - `tests/behaviour-selection.real-assessment.test.js`: 16 passed (shared fixtures intact) - `tests/investigation-state-assessor.test.js`: 51 passed (assessor unchanged) ### Documents Updated - `docs/design-evolution-log.md` — this entry - `docs/current-handoff.md` — return-to-work note replaced ## Experiment 41 — Compare Acknowledge Priority Alternatives (2026-08-06) ### Purpose Experiment 40 confirmed Summarise and Pause are eligible_but_blocked by Acknowledge's priority-1 position. Two passive alternatives were compared without modifying production code: **Variant A** — Reorder rules so specific behaviours (Summarise, Pause) evaluate before Acknowledge. The idea is that if a more specific behaviour fires first, it captures the terminal/stalled states where Acknowledge should not fire. **Variant B** — Keep existing priority order but exclude Acknowledge from firing when phase=concluding/synthesising, progress=stalled, or health=user_overloaded. The idea is to gate Acknowledge rather than reorder everything. ### Method Both variants were implemented as test-only functions in `tests/behaviour-selection.counterfactual.test.js`. Each variant was evaluated against the same 7 real assessment turns from Experiments 39/40 across 3 scenarios. All five behaviours confirmed independently reachable synthetically. No production rules changed. ### Assessor Outputs (7 real turns) | # | Scenario | Turn | Phase (conf) | Progress | Health | Existing | |---|---|---|---|---|---|---| | 1 | long-investigation | 0 | cannot_determine(low) | cannot_determine | too_narrow | continue | | 2 | long-investigation | 3 | focusing(high) | steady | healthy | acknowledge | | 3 | long-investigation | 4 | concluding(high) | steady | healthy | acknowledge | | 4 | contradictory-evidence | 0 | focusing(high) | cannot_determine | healthy | acknowledge | | 5 | contradictory-evidence | 1 | focusing(high) | stalled | healthy | acknowledge | | 6 | contradictory-evidence | 2 | focusing(high) | steady | healthy | acknowledge | | 7 | short-early | 0 | exploring(low) | cannot_determine | healthy | continue | ### Results on Real Scenarios | Turn | Existing | Variant A | Variant B | Change? | |---|---|---|---|---| | long-investigation t3 | acknowledge | summarise | acknowledge | V-A: side-effect | | long-investigation t4 | acknowledge | **summarise** | **summarise** | **convergent ✓** | | contradictory-evidence t1 | acknowledge | **pause** | **pause** | **convergent ✓** | | All others | unchanged | unchanged | unchanged | — | ### Divergence Analysis **Variant A diverges from Variant B at long-investigation turn 3.** Variant A produces `summarise` because its `resolvedNodeCount >= 3 && steady` rule fires at priority 1 without phase context. The assessor confirms this is a focusing-phase state (not synthesising/concluding) where the user needs acknowledgment, not compression. This is a false-positive for summarisation — a side-effect of Variant A's priority reordering. **Variant B correctly preserves Acknowledge** at long-investigation t3 because: 1. The exclusion list only includes `synthesising`, `concluding`, `stalled`, and `user_overloaded` — not focusing 2. SummariseV2 itself has a phase gate (`phase.value === "synthesising"`) that prevents false-fire in focusing states 3. Acknowledge at priority 1 wins because no exclusion applies ### Key Findings 1. **Both variants converge on the same two genuine changes:** `concluding → summarise` and `stalled → pause`. This was the experiment's primary question, and both approaches answer it correctly. 2. **Variant A introduces a false-positive:** The `resolvedNodeCount >= 3 && steady` rule fires in focusing-phase states without phase context, causing premature summarisation when Acknowledge would be more useful. 3. **Variant B has cleaner boundaries:** Explicit exclusion conditions prevent unwanted side-effects while preserving Acknowledge's role as the default healthy-state behaviour. 4. **Distribution shift (both variants):** - Existing: acknowledge 71%, continue 29% - Variant A: acknowledge 29%, summarise 29%, pause 14%, continue 29% - Variant B: acknowledge 43%, summarise 14%, pause 14%, continue 29% - Variant B preserves more Acknowledge because it doesn't remove the default healthy-state behaviour entirely 5. **Variant B is architecturally cleaner** for this problem space because it adds a targeted gate to one rule rather than reordering five priority levels — each of which would need individual review for side-effects. ### Test Results - `tests/behaviour-selection.counterfactual.test.js`: 44 passed (new diagnostic file) - `tests/behaviour-selection.reachability.test.js`: 33 passed (no regressions) - `tests/behaviour-selection.real-assessment.test.js`: 16 passed (shared fixtures intact) - `tests/behaviour-selection.test.js`: 51 passed (no regressions) ### Decision Criteria | Criterion | Variant A | Variant B | |---|---|---| | Fixes concluding state | ✓ summarise | ✓ summarise | | Fixes stalled state | ✓ pause | ✓ pause | | No false-positive changes | ✗ long-t3 → summarise | ✓ preserved acknowledge | | Implementation complexity | Simple reordering | Small gate function | | Maintains Acknowledge for healthy focus states | ? (depends on future review) | ✓ explicit preservation | ### Recommendation **Variant B is preferred.** Both variants correctly identify the two genuine changes needed. Variant B has no false-positives, cleaner architectural boundaries (targeted exclusion vs priority reordering), and better preserves the existing Acknowledge default for healthy focusing states where it is appropriate. A recommended implementation would: 1. Keep existing priority order 2. Add `isAcknowledgeExcluded()` function with conditions: phase∈{synthesising, concluding}, progress=stalled, health=user_overloaded 3. Gate Acknowledge through this exclusion before selecting it at priority 1 ### Documents Updated - `docs/design-evolution-log.md` — this entry - `docs/current-handoff.md` — return-to-work note replaced ## Experiment 41 — Conclusion **Variant B was preferred because it changed only the two intended turns without introducing a false-positive in a focusing state. Variant A produced an early summarise in a focusing phase and was discarded. No production rule changed during Experiment 41.** The implementation of Variant B's exclusion gate is the subject of Experiment 42. ---