From 7849b2f215526cb472395e61c8ce1c7c0a94dd98 Mon Sep 17 00:00:00 2001 From: robbond Date: Thu, 6 Aug 2026 15:47:49 +0100 Subject: [PATCH] experiment: validate reduced context routing --- docs/context-routing-validation.md | 62 ++++++++++++++++++++++++++++++ docs/design-evolution-log.md | 55 ++++++++++++++++++++++++++ 2 files changed, 117 insertions(+) create mode 100644 docs/context-routing-validation.md diff --git a/docs/context-routing-validation.md b/docs/context-routing-validation.md new file mode 100644 index 0000000..7da0806 --- /dev/null +++ b/docs/context-routing-validation.md @@ -0,0 +1,62 @@ +# Context Routing Validation — Experiment 36 + +## Documents Initially Loaded + +1. `docs/current-handoff.md` (86 lines) — first return-to-work entry point +2. `docs/current-project-state.md` (132 lines) — active state and capabilities +3. `docs/task-context-packs.md` (110 lines) — routing for four work types + +Total: 328 lines loaded before any expansion. + +## Additional Documents Required + +### docs/ui-mock-reference.md (63 lines) +**Why:** Task 2 required identifying mock scenarios for "long investigation" and "contradictory evidence". The task-context-packs Routing Test B claimed these were identifiable without loading ui-mock-reference, but the specific scenario names were not stated in any initial document. Loading confirmed both exist: "Long investigation (10–15 turns)" and "Contradiction". +**Routing should have included it:** YES — this is a routing defect. The pack's Routing Test B presents an unverifiable claim as fact. + +### docs/project-knowledge-inventory.md (215 lines) +**Why:** Task 4 asked where a new developer should begin for engine experiments. Current handoff → project-state → task-context-packs gave the path, but inventory confirmed the Engine Experiment pack's four "always read" documents are all verifiably present in the repository. Also provided confirmation of what the knowledge-management phase created. +**Routing should have included it:** DEBATED — the inventory validates pack completeness but was not strictly necessary to answer Task 4 from routing alone. + +### docs/current-implementation-verification.md (111 lines) +**Why:** Cross-checked Behaviour Selection's isolation against current-project-state §3's classification. Found section 3b confirming `selectBehaviour` has no callers outside its module. +**Routing should have included it:** DEBATED — current-project-state already stated the same fact; this was a corroboration, not a gap fill. + +## Tasks Completed + +### Task 1 — Does Behaviour Selection affect engine behaviour? +- **Answer:** No. It is isolated — no import or call exists in any file under lib/ or app/. +- **Initial docs sufficient:** Yes (current-project-state §3 + handoff §2). +- **Expansion needed:** No. + +### Task 2 — Correct mock scenarios for long investigation and contradictory evidence? +- **Answer:** "Long investigation (10–15 turns)" and "Contradiction" from ui-mock-reference.md. +- **Initial docs sufficient:** No. Routing Test B claimed they were, but the claim was unverifiable until ui-mock-reference was loaded. +- **Expansion needed:** Yes — `docs/ui-mock-reference.md`. + +### Task 3 — Why passive classifiers are not part of active reasoning? +- **Answer:** Passive classifiers (Experiments 18–25B) record diagnostic signals for future use but have no integration into the turn cycle. Investigation-state assessment is the only one called at all, and its result goes into a diagnostics field — never checked by conditional branches. Others have zero callers. None control user-facing decisions or path selection. +- **Initial docs sufficient:** Yes (current-project-state §2–§5 + handoff §2). +- **Expansion needed:** No. + +### Task 4 — Where should a new developer begin for the next engine experiment? +- **Answer:** Read `docs/current-handoff.md` → `docs/current-project-state.md` → Engine Experiment pack from `docs/task-context-packs.md`, which directs them to four always-read documents (`current-project-state`, `current-working-principles`, `architecture-guardrails`, `current-implementation-verification`) plus the immediately previous experiment entry in the design log. The pack's "Stop and ask" rules prevent blind expansion. +- **Initial docs sufficient:** Partially — pack routing is clear but some verification required confirming documents exist (inventory). +- **Expansion needed:** Marginally — project-knowledge-inventory for pack completeness check. + +## Routing Failures Found + +**One genuine failure: Routing Test B in task-context-packs.md.** +The test states that mock scenarios for "long investigation" and "contradiction" are identifiable without loading ui-mock-reference. This was presented as a self-evident fact but could not be verified from the stated documents alone — the specific scenario names exist only in ui-mock-reference.md. The routing is incomplete; it should have included the mock reference file. + +**One questionable exclusion: project-knowledge-inventory for Task 4.** +The task-context-packs Engine Experiment pack lists four "always read" documents but does not themselves confirm all four exist. A cautious developer would load the inventory to verify, adding ~215 lines. This is acceptable cost but worth noting as a gap in the pack's self-validation. + +## Documentation Improvements Discovered + +1. **Routing Test B must include ui-mock-reference.md.** Remove the "No extra file required" claim and add the mock reference to the Engine Experiment pack's routing chain when tasks involve scenario selection. +2. **Packs should confirm their listed documents exist.** Adding a verification check (or removing unverified entries) would prevent the need for inventory cross-referencing. + +## Overall Assessment: Mostly ready + +Evidence: Three of four tasks were completed from initial context only. One routing defect was found (Task 2's claim was unverifiable without extra loading). The system works but Routing Test B demonstrates that "sufficient" claims should be evidence-based, not assumed. After fixing Test B, the reduced context system is ready for normal work. diff --git a/docs/design-evolution-log.md b/docs/design-evolution-log.md index 6a6eb13..4d66adb 100644 --- a/docs/design-evolution-log.md +++ b/docs/design-evolution-log.md @@ -2079,3 +2079,58 @@ Pending Rob's review. **Handoff remained concise?** Yes. 86 lines (was 68). Increase justified by the maintenance principle paragraph, updated current-state wording, and provisional completion criteria section. No historical timeline appended. **Status:** Pending Rob's review. + +## Experiment 36 — Validate Reduced Context Routing + +**Branch:** `feature/user-workspace-ux-v0.7` + +### Hypothesis + +The documentation system (handoff + project-state + task-context-packs) is complete enough to support normal work without silently expanding into historical documentation. A fresh session can complete representative tasks using only routing instructions. + +### Initial Documents Loaded (328 lines total) + +1. `docs/current-handoff.md` — 86 lines +2. `docs/current-project-state.md` — 132 lines +3. `docs/task-context-packs.md` — 110 lines + +### Additional Documents Loaded + +| Document | Lines | Why Needed | Routing Should Include? | +|---|---|---|---| +| `docs/ui-mock-reference.md` | 63 | Task 2: verify mock scenarios for "long investigation" and "contradiction". Routing Test B claimed these were identifiable without loading it, but the specific scenario names do not appear in any initial document. | YES — routing defect found | +| `docs/project-knowledge-inventory.md` | 215 | Task 4: confirm Engine Experiment pack's four always-read documents actually exist and understand KM phase outputs. | Debated — validated completeness but not strictly required by routing | +| `docs/current-implementation-verification.md` | 111 | Cross-checked Behaviour Selection isolation against current-project-state §3. Provided corroboration but was not the sole basis for Task 1 answer. | Debated — useful corroboration; current-project-state alone sufficed | + +### Tasks Completed Without Context Expansion + +**Task 1 — Does Behaviour Selection affect engine behaviour?** +No. Current project state §3 classifies it as isolated. Handoff §2 confirms passive classifiers don't control the investigation. Task-context-packs Routing Test A corroborates (current-implementation-verification §3b). + +**Task 3 — Why passive classifiers are not yet in the active reasoning loop?** +Passive classifiers record diagnostic signals for future use but have no integration into the turn cycle. Only investigation-state assessment is called (at 3 orchestrator sites), and its result goes into a diagnostics field — never checked by conditional branches. Others have zero callers. + +### Tasks Requiring Extra Context + +**Task 2 — Mock scenarios for long investigation and contradictory evidence** +Required `docs/ui-mock-reference.md`. Routing Test B in task-context-packs claimed these were identifiable without loading it, but the specific scenario names ("Long investigation (10–15 turns)" and "Contradiction") do not appear in any initial document. The routing claim was unverifiable until the mock reference was loaded — this is a genuine routing defect. + +**Task 4 — Where should a new developer begin for the next engine experiment?** +Partially answered from initial documents (handoff → project-state → pack). Marginal need to verify that all four always-read pack documents actually exist, resolved by cross-referencing project-knowledge-inventory. + +### Routing Failures Found + +One genuine failure: **Routing Test B in task-context-packs.md**. The test states that mock scenarios for long investigation and contradiction are identifiable without loading ui-mock-reference.md. This was presented as a self-evident fact but the specific scenario names only exist in ui-mock-reference.md. The routing is incomplete — it should have included the mock reference file, or at minimum acknowledged that scenario names require verification. + +### Documentation Changes Made + +- Created `docs/context-routing-validation.md` (62 lines) — this experiment's record +- Updated `docs/design-evolution-log.md` — appended Experiment 36 entry + +**No source code or tests changed. No archive changes.** + +### Overall Assessment: Mostly ready + +Three of four tasks completed from initial context only. One routing defect found (Task 2). After fixing Routing Test B to include ui-mock-reference.md, the reduced context system is ready for normal work. + +---