confidence-engine/docs/archive/experiments/vol-1-chapters/ch6/handoff-validation-behavior-selection-and-assessor-audit.md

Experiment 35 — Test Current Handoff Maintenance (2026-08-06)

Hypothesis: A current handoff can remain useful if it describes only the latest stopping point, replaces stale details rather than appending history, and identifies the latest confirmed experiment and commit unambiguously.

Stale or ambiguous wording found:

• Section 1 named Experiment 33 and commit b959cfa as the current state — now stale after Experiments 34+35;
• Section 4 described only Experiment 33's completion, giving no indication that a single handoff had been created in Experiment 34;
• No explicit mention of commit 1d92aa0 anywhere in the handoff;
• Footer said "Created by Experiment 34" without acknowledging this maintenance experiment.

Corrections made:

• Section 1: updated to name Experiment 34 and commit 1d92aa0; added the maintenance principle ("replace stale details rather than appending history");
• Section 4: rewritten to describe Experiment 34's consolidation work;
• Section 5: retained one genuinely open question about handoff longevity; added provisional KM completion criteria sub-section (7 criteria, marked provisional);
• Footer: updated to reference Experiment 35; added Return-to-Work Note recording all current state.

Fresh-return test result: PASS — from current-handoff.md and task-context-packs.md only, a fresh session can determine:

• Latest completed KM experiment: Experiment 34 ✓
• Latest commit: 1d92aa0 ✓
• Knowledge-management active, engine/UI paused ✓
• Knowledge-Management context pack is the correct routing target ✓
• No need to open full design history ✓
• Older commits not mistaken for current stopping point ✓

Provisional completion criteria added: Seven criteria recorded in Section 5 (see above). Not yet declared complete — pending Rob's review.

Handoff remained concise? Yes. 86 lines (was 68). Increase justified by the maintenance principle paragraph, updated current-state wording, and provisional completion criteria section. No historical timeline appended.

Status: Pending Rob's review.

Experiment 36 — Validate Reduced Context Routing

Branch: feature/user-workspace-ux-v0.7

Hypothesis

The documentation system (handoff + project-state + task-context-packs) is complete enough to support normal work without silently expanding into historical documentation. A fresh session can complete representative tasks using only routing instructions.

Initial Documents Loaded (328 lines total)

1. docs/current-handoff.md — 86 lines
2. docs/current-project-state.md — 132 lines
3. docs/task-context-packs.md — 110 lines

Additional Documents Loaded

Document	Lines	Why Needed	Routing Should Include?
docs/ui-mock-reference.md	63	Task 2: verify mock scenarios for "long investigation" and "contradiction". Routing Test B claimed these were identifiable without loading it, but the specific scenario names do not appear in any initial document.	YES — routing defect found
docs/project-knowledge-inventory.md	215	Task 4: confirm Engine Experiment pack's four always-read documents actually exist and understand KM phase outputs.	Debated — validated completeness but not strictly required by routing
docs/current-implementation-verification.md	111	Cross-checked Behaviour Selection isolation against current-project-state §3. Provided corroboration but was not the sole basis for Task 1 answer.	Debated — useful corroboration; current-project-state alone sufficed

Tasks Completed Without Context Expansion

Task 1 — Does Behaviour Selection affect engine behaviour? No. Current project state §3 classifies it as isolated. Handoff §2 confirms passive classifiers don't control the investigation. Task-context-packs Routing Test A corroborates (current-implementation-verification §3b).

Task 3 — Why passive classifiers are not yet in the active reasoning loop? Passive classifiers record diagnostic signals for future use but have no integration into the turn cycle. Only investigation-state assessment is called (at 3 orchestrator sites), and its result goes into a diagnostics field — never checked by conditional branches. Others have zero callers.

Tasks Requiring Extra Context

Task 2 — Mock scenarios for long investigation and contradictory evidence Required docs/ui-mock-reference.md. Routing Test B in task-context-packs claimed these were identifiable without loading it, but the specific scenario names ("Long investigation (10–15 turns)" and "Contradiction") do not appear in any initial document. The routing claim was unverifiable until the mock reference was loaded — this is a genuine routing defect.

Task 4 — Where should a new developer begin for the next engine experiment? Partially answered from initial documents (handoff → project-state → pack). Marginal need to verify that all four always-read pack documents actually exist, resolved by cross-referencing project-knowledge-inventory.

Routing Failures Found

One genuine failure: Routing Test B in task-context-packs.md. The test states that mock scenarios for long investigation and contradiction are identifiable without loading ui-mock-reference.md. This was presented as a self-evident fact but the specific scenario names only exist in ui-mock-reference.md. The routing is incomplete — it should have included the mock reference file, or at minimum acknowledged that scenario names require verification.

Documentation Changes Made

• Created docs/context-routing-validation.md (62 lines) — this experiment's record
• Updated docs/design-evolution-log.md — appended Experiment 36 entry

No source code or tests changed. No archive changes.

Overall Assessment: Mostly ready

Two of four tasks completed from initial context only. One routing defect found (Task 2; corrected by Experiment 37). After fixing Routing Test B to name ui-mock-reference.md as the scenario source, the reduced context system is ready for normal work.

---

Experiment 37 — Validate Cross-Boundary Context Routing

Branch: feature/user-workspace-ux-v0.7

Hypothesis

The context-pack system can support cross-boundary work if Claude:

1. starts with one primary pack;
2. adds a second pack only for a named boundary;
3. records why each extra document was loaded;
4. avoids loading the full history.

Initial Documents Loaded (328 lines total)

1. docs/current-handoff.md — 85 lines; first return-to-work entry point
2. docs/current-project-state.md — 131 lines; active state and capabilities
3. docs/task-context-packs.md — 110 lines; routing for four work types

Additional Documents Loaded

Document	Lines	Why Needed	Routing Should Include?
docs/ui-mock-reference.md	62	Cross-boundary boundary: the task requires identifying a mock scenario for workspace display. This is the second pack (UI and Mock) needed because no other loaded document names scenarios or UI fixtures. Yes — it is part of the UI/Mock pack, not an ad-hoc addition.

Cross-Boundary Task Result

Task: Display passive condition-status information in the workspace for a mock investigation without changing the active reasoning loop.

Finding	Details
Condition-status capability	Passive: decision-condition status evaluation records signals but has no integration into the turn cycle; never controls user-facing decisions or path selection
Active reasoning loop	Unchanged: deterministic pipeline (scenario reconstruction → graph update → unknown selection → question formulation → turn orchestration); none of these pathways are affected by passive data
Mock scenario	"Long investigation (10–15 turns)" from ui-mock-reference.md; workspace can display accumulated diagnostic signals over time without interrupting the active reasoning cycle
Implementation areas to inspect later	decision-condition-status evaluation module; evidence scope detection module; UI workspace components for passive display integration
Both packs genuinely needed?	Yes: Engine pack identifies which capabilities are active vs passive; UI pack identifies how the workspace presents state. Neither alone suffices
Archive or full history required?	No

Context remained manageable: Yes. 390 lines total (328 initial + 62 additional). Each document loaded for a specific named purpose. No blind expansion.

Knowledge-Management Completion Criteria Review

Criterion	Status
1. Fresh session can resume from handoff + one pack	met
2. Current state verified against implementation	met
3. Historical material outside default loading	met
4. Current principles separated from aspirational architecture	met
5. Task-specific routing works for engine and UI tasks	met
6. Cross-boundary task tested	met
7. Maintaining handoff does not require reading full history	met

All seven criteria are now met.

Knowledge-management structure is ready for Rob's review before engine experiments resume.

Routing Defects Discovered

None in this experiment. The correction to Routing Test B (naming ui-mock-reference.md as the scenario source) was applied before testing. No new defects found in the cross-boundary test.

Overall Assessment: Ready

The context-pack system handled a genuine engine/UI cross-boundary task by combining two packs deliberately with full documentation of each loaded document and its purpose. Context remained small (390 lines). All knowledge-management criteria are met.

---

Experiment 38 — Cold-Start Project Recovery Validation

Branch: feature/user-workspace-ux-v0.7 Type: Knowledge-management / handoff validation (final KM experiment) Objective: Test whether a genuinely cold session can recover the project accurately from the reduced context system alone without reading the full history or any earlier experiment reports.

Setup

Cold-start configuration: no prior conversation context, no past experiment reports loaded, repository documentation carries all context. Session was freshly created to simulate a real return-to-work scenario. Only docs/current-handoff.md was read first (per handoff §6 step 1), then the two documents specified by its resume instructions (§6 steps 2–3): docs/current-project-state.md and docs/task-context-packs.md.

Documents Loaded

Document	Reason
docs/current-handoff.md	Primary entry point (handoff §6 step 1)
docs/current-project-state.md	Resume instruction (§6 step 2) and routing table (§6 step 7)
docs/task-context-packs.md	Pack selection (§6 step 3) and pack contents for verification

No additional documents were loaded. No blind expansion occurred. The full design-evolution log, archived documents, UI mock reference, source code, and tests were all excluded by design.

Project-State Recovery Result

The cold session correctly recovered:

• What the Confidence Engine does (facilitated investigation with structured reasoning graph).
• Active capabilities: deterministic reasoning pipeline, unknown selection via atomicity/answerability, question formulation, scenario API, turn cycle orchestration.
• Passive capabilities: seven diagnostic layers from Experiments 18–25B, all isolated, none control user-facing investigation.
• Paused work: engine experiments (after Exp 25B), UI experiments.
• Why KM phase was undertaken (documentation bloat blocking session recovery).

Recovery score: complete from three documents alone. No source code inspection required.

Context-Pack Selection Result

Pack 1 — Engine Experiment Work selected correctly by the cold session. The three initial documents contained sufficient information to identify the pack, its default documents, and what to exclude without reading any additional material.

Handoff Defects Found

None found in docs/current-handoff.md. The handoff accurately describes the stopping point, identifies all seven KM criteria as met, provides correct resume instructions, and includes accurate capability boundaries. One structural update was made: the open item "whether the handoff stays accurate after further advances" was resolved as no longer applicable (the cold-start test confirmed it is accurate).

Completion-Criteria Result

All seven knowledge-management completion criteria are confirmed met by this cold-start validation:

1. Fresh session can resume from handoff + one pack — met (Exp 38 demonstrates this)
2. Current state verified against implementation — met (Exp 28+)
3. Historical material outside default loading — met
4. Current principles separated from aspirational architecture — met
5. Task-specific routing works for engine and UI tasks — met (Exp 37)
6. Cross-boundary task tested — met (Exp 37)
7. Maintaining handoff does not require reading full history — met

The knowledge-management phase is complete enough for Rob to choose when engine experiments resume.

Documents Updated

• docs/cold-start-validation.md — created (this experiment's deliverable)
• docs/current-handoff.md — Exp 38 commit placeholder, structural open-item resolution, return-to-work note replacement
• docs/current-project-state.md — KM status update ("active" → "complete"), latest known commit correction
• docs/design-evolution-log.md — this entry

Overall Assessment: Ready

The cold-start validation passed. A genuinely fresh session understood the project state, chose the correct context pack, verified the resume boundary, produced a valid engine-work resume brief, and found no handoff defects — all from three documents alone. No source code was read or changed. The reduced context system works for sessions that did not help create the documents.

Engine and UI experiments remain paused pending Rob's review.

---

Experiment 39 — Validate Behaviour Selection Against Real Assessment Outputs (2026-08-06)

Branch: feature/user-workspace-ux-v0.7

Hypothesis

The existing deterministic selector produces a useful rhythm across genuine assessment outputs without changing the active engine. If it repeatedly chooses one behaviour, chooses behaviours at the wrong time, or depends on signals the assessor does not actually produce, the experiment should expose that honestly.

Scenarios Evaluated (from tests/investigation-state-assessor.test.js fixture set)

1. Long investigation (3 turns: early → deepening → complete terminal)
2. Contradictory evidence (3 turns: two conflicting consultants, 0→1→2 resolved unknowns)
3. Short early (1 turn: two observations, first unknown, no resolution)

Behaviour Distribution (7 turns total)

• Acknowledge: 5 (71%)
• Continue: 2 (29%)
• Clarify: 0 (0%)
• Summarise: 0 (0%)
• Pause: 0 (0%)

Behaviour Sequence by Scenario

Long investigation: continue → acknowledge → acknowledge

• Turn 0: phase=cannot_determine, progress=cannot_determine, health=too_narrow → continue (no rule matched)
• Turn 3: phase=focusing, progress=steady, health=healthy → acknowledge
• Turn 4: phase=concluding, progress=steady, health=healthy → acknowledge

Contradictory evidence: acknowledge → acknowledge → acknowledge

• Turn 0: phase=focusing, progress=cannot_determine, health=healthy → acknowledge
• Turn 1: phase=focusing, progress=stalled, health=healthy → acknowledge
• Turn 2: phase=focusing, progress=steady, health=healthy → acknowledge

Short early: continue

• Turn 0: phase=exploring, progress=cannot_determine, health=healthy → continue

Sensible Selections (7 of 7)

All selections were classified as sensible per the selection's stated conditions. Acknowledge fires because health=healthy AND phase confidence≠low across most states. Continue fires when no specific rule matches (early/cannot_determine/exploring phases).

Questionable or Inappropriate Selections

One notable pattern: Summarise and Pause never fire, even in a concluding terminal state. This is not because the assessor fails to detect "concluding" — it does. It is because Acknowledge (priority 1) fires first when health=healthy, blocking Summarise (priority 3) from ever reaching its turn. This is an acknowledgement/summarise priority conflict: acknowledging a conclusion ("you've figured this out!") is not wrong, but "give me a summary" is more useful at terminal states. The current rule ordering does not distinguish "early healthy" from "concluding healthy."

Clarify never fires because no test scenario produces health=too_broad — the assessor's "too_broad" trigger (activeUnknownCount > 3 AND resolved < 2) requires more nodes than any scenario in the fixture set has at that stage.

Pause never fires because health=user_overloaded is never reached, and while contradictory-turn-1 has phase=focusing + progress=stalled, Acknowledge still blocks it.

Contract Alignment

Assessor → Selector contract aligns cleanly. The assessor produces all three dimensions (phase, progress, conversationHealth) with the fields the selector expects. No transformation needed between pipeline stages.

Whether Selector Appears Useful Enough for Another Passive Experiment

The existing selector works but its behaviour variation is severely constrained by Acknowledge's priority position. A next passive experiment should test whether reordering or refining the acknowledge condition (e.g., excluding concluding/terminal phases) produces more context-appropriate behaviour — without changing the assessor.

Status

Pending Rob's review. Five behaviours are too narrow for this to be definitive, and only three scenarios were tested. The dominant pattern (acknowledge in healthy states) may change with different investigation domains.

Documents Updated

• docs/design-evolution-log.md — this entry
• docs/current-handoff.md — return-to-work note replaced

---

Experiment 40 — Audit Behaviour Reachability and Blocking (2026-08-06)

Objective

Why did Clarify, Summarise, and Pause not appear during Experiment 39? Acknowledge: 5 (71%), Continue: 2 (29%), others: 0. This is a passive diagnostic — no rule changes, no engine modifications.

Method

One test file (tests/behaviour-selection.reachability.test.js) containing:

• Diagnostic audit helper that evaluates every behaviour rule against one assessment object
• Real-scenario audits across the same Experiment 39 turns (8 turns total)
• Synthetic reachability checks for each behaviour in isolation

Findings

Summarise — eligible_but_blocked

Eligible in 2 of 7 real turns:

• long-investigation turn 1 (resolvedNodeCount ≥ 3 + progress=steady triggers summarise rule)
• long-investigation turn 2 (phase=concluding triggers summarise rule)

In both cases, health=healthy simultaneously, so Acknowledge (priority 1) fires first. Summarise rules are met but its output is never returned because the selector returns early on priority ordering.

Root cause: priority conflict, not assessor failure. The phase evidence correctly identifies concluding/synthesising states; the problem is that Acknowledge's broader trigger condition (health=healthy is the most common state) fires first.

Clarify — never_eligible_in_tested_scenarios (reachable only in synthetic case)

Not eligible in any of 7 real turns because neither trigger condition is met:

• health=too_broad: requires activeUnknownCount > 3 AND resolvedNodeCount < 2 — no fixture reaches this state
• phase=orienting + observationDensity < 3: current assessor never produces phase=orienting for tested scenarios

Synthetic case confirms the rule fires correctly in isolation (with low-confidence phase to avoid Acknowledge blocking).

Root cause: assessor health classification logic produces too few too_broad cases. The trigger condition is extremely narrow — needs activeUnknownCount > 3 AND resolved < 2 simultaneously.

Pause — eligible_but_blocked

Eligible in 1 of 7 real turns:

• contradictory-evidence turn 1 (phase=focusing + progress=stalled triggers pause rule)

In this case, health=healthy simultaneously, so Acknowledge blocks it. The second pause trigger (health=user_overloaded) is never met because the assessor never produces that state.

Root cause: same priority conflict as Summarise. One of two rules fires in real data but gets blocked by Acknowledge's earlier position.

Synthetic Reachability Confirmation

All five behaviours are independently reachable when isolated from Acknowledge:

• ✅ acknowledge — healthy + confident phase
• ✅ clarify — too_broad health (with low-confidence phase to avoid Acknowledge)
• ✅ summarise — synthesising/concluding phase (without healthy health)
• ✅ pause — focusing+stalled or user_overloaded (without healthy health)
• ✅ continue — no rules match

Classifications

Behaviour	Classification	Primary Cause
Summarise	eligible_but_blocked	Acknowledge priority 1 fires first when health=healthy
Clarify	never_eligible_in_tested_scenarios (reachable only in synthetic)	too_broad trigger too narrow for test scenarios; orienting+low obs not produced by assessor
Pause	eligible_but_blocked	Acknowledge priority 1 fires first when health=healthy; user_overloaded never produced

Impact on Prior Finding (Exp 39)

Experiment 39 concluded "the Acknowledge→Summarise priority conflict prevents Summarise from firing." Experiment 40 confirms this and adds that Pause faces the same blocking (1 eligible turn, blocked). Clarify's absence is fundamentally different: its rules are not triggered at all in tested scenarios.

This means any fix must address two distinct problems:

1. Priority conflict affecting Summarise AND Pause (same cause)
2. Narrow trigger conditions for Clarify and the user_overloaded health state

Test Results

• tests/behaviour-selection.reachability.test.js: 33 passed (new diagnostic file)
• tests/behaviour-selection.test.js: 51 passed (no regressions)
• tests/behaviour-selection.real-assessment.test.js: 16 passed (shared fixtures intact)
• tests/investigation-state-assessor.test.js: 51 passed (assessor unchanged)

Documents Updated

• docs/design-evolution-log.md — this entry
• docs/current-handoff.md — return-to-work note replaced

Experiment 41 — Compare Acknowledge Priority Alternatives (2026-08-06)

Purpose

Experiment 40 confirmed Summarise and Pause are eligible_but_blocked by Acknowledge's priority-1 position. Two passive alternatives were compared without modifying production code:

Variant A — Reorder rules so specific behaviours (Summarise, Pause) evaluate before Acknowledge. The idea is that if a more specific behaviour fires first, it captures the terminal/stalled states where Acknowledge should not fire.

Variant B — Keep existing priority order but exclude Acknowledge from firing when phase=concluding/synthesising, progress=stalled, or health=user_overloaded. The idea is to gate Acknowledge rather than reorder everything.

Method

Both variants were implemented as test-only functions in tests/behaviour-selection.counterfactual.test.js. Each variant was evaluated against the same 7 real assessment turns from Experiments 39/40 across 3 scenarios. All five behaviours confirmed independently reachable synthetically. No production rules changed.

Assessor Outputs (7 real turns)

#	Scenario	Turn	Phase (conf)	Progress	Health	Existing
1	long-investigation	0	cannot_determine(low)	cannot_determine	too_narrow	continue
2	long-investigation	3	focusing(high)	steady	healthy	acknowledge
3	long-investigation	4	concluding(high)	steady	healthy	acknowledge
4	contradictory-evidence	0	focusing(high)	cannot_determine	healthy	acknowledge
5	contradictory-evidence	1	focusing(high)	stalled	healthy	acknowledge
6	contradictory-evidence	2	focusing(high)	steady	healthy	acknowledge
7	short-early	0	exploring(low)	cannot_determine	healthy	continue

Results on Real Scenarios

Turn	Existing	Variant A	Variant B	Change?
long-investigation t3	acknowledge	summarise	acknowledge	V-A: side-effect
long-investigation t4	acknowledge	summarise	summarise	convergent ✓
contradictory-evidence t1	acknowledge	pause	pause	convergent ✓
All others	unchanged	unchanged	unchanged	—

Divergence Analysis

Variant A diverges from Variant B at long-investigation turn 3. Variant A produces summarise because its resolvedNodeCount >= 3 && steady rule fires at priority 1 without phase context. The assessor confirms this is a focusing-phase state (not synthesising/concluding) where the user needs acknowledgment, not compression. This is a false-positive for summarisation — a side-effect of Variant A's priority reordering.

Variant B correctly preserves Acknowledge at long-investigation t3 because:

1. The exclusion list only includes synthesising, concluding, stalled, and user_overloaded — not focusing
2. SummariseV2 itself has a phase gate (phase.value === "synthesising") that prevents false-fire in focusing states
3. Acknowledge at priority 1 wins because no exclusion applies

Key Findings

1. Both variants converge on the same two genuine changes: concluding → summarise and stalled → pause. This was the experiment's primary question, and both approaches answer it correctly.

2. Variant A introduces a false-positive: The resolvedNodeCount >= 3 && steady rule fires in focusing-phase states without phase context, causing premature summarisation when Acknowledge would be more useful.

3. Variant B has cleaner boundaries: Explicit exclusion conditions prevent unwanted side-effects while preserving Acknowledge's role as the default healthy-state behaviour.

4. Distribution shift (both variants):

   • Existing: acknowledge 71%, continue 29%
   • Variant A: acknowledge 29%, summarise 29%, pause 14%, continue 29%
   • Variant B: acknowledge 43%, summarise 14%, pause 14%, continue 29%
   • Variant B preserves more Acknowledge because it doesn't remove the default healthy-state behaviour entirely

5. Variant B is architecturally cleaner for this problem space because it adds a targeted gate to one rule rather than reordering five priority levels — each of which would need individual review for side-effects.

Test Results

• tests/behaviour-selection.counterfactual.test.js: 44 passed (new diagnostic file)
• tests/behaviour-selection.reachability.test.js: 33 passed (no regressions)
• tests/behaviour-selection.real-assessment.test.js: 16 passed (shared fixtures intact)
• tests/behaviour-selection.test.js: 51 passed (no regressions)

Decision Criteria

Criterion	Variant A	Variant B
Fixes concluding state	✓ summarise	✓ summarise
Fixes stalled state	✓ pause	✓ pause
No false-positive changes	✗ long-t3 → summarise	✓ preserved acknowledge
Implementation complexity	Simple reordering	Small gate function
Maintains Acknowledge for healthy focus states	? (depends on future review)	✓ explicit preservation

Recommendation

Variant B is preferred. Both variants correctly identify the two genuine changes needed. Variant B has no false-positives, cleaner architectural boundaries (targeted exclusion vs priority reordering), and better preserves the existing Acknowledge default for healthy focusing states where it is appropriate. A recommended implementation would:

1. Keep existing priority order
2. Add isAcknowledgeExcluded() function with conditions: phase∈{synthesising, concluding}, progress=stalled, health=user_overloaded
3. Gate Acknowledge through this exclusion before selecting it at priority 1

Documents Updated

• docs/design-evolution-log.md — this entry
• docs/current-handoff.md — return-to-work note replaced

Experiment 41 — Conclusion

Variant B was preferred because it changed only the two intended turns without introducing a false-positive in a focusing state. Variant A produced an early summarise in a focusing phase and was discarded. No production rule changed during Experiment 41. The implementation of Variant B's exclusion gate is the subject of Experiment 42.

---