63 lines
5.4 KiB
Markdown
63 lines
5.4 KiB
Markdown
# Context Routing Validation — Experiment 36
|
||
|
||
## Documents Initially Loaded
|
||
|
||
1. `docs/current-handoff.md` (86 lines) — first return-to-work entry point
|
||
2. `docs/current-project-state.md` (132 lines) — active state and capabilities
|
||
3. `docs/task-context-packs.md` (110 lines) — routing for four work types
|
||
|
||
Total: 328 lines loaded before any expansion.
|
||
|
||
## Additional Documents Required
|
||
|
||
### docs/ui-mock-reference.md (63 lines)
|
||
**Why:** Task 2 required identifying mock scenarios for "long investigation" and "contradictory evidence". The task-context-packs Routing Test B claimed these were identifiable without loading ui-mock-reference, but the specific scenario names were not stated in any initial document. Loading confirmed both exist: "Long investigation (10–15 turns)" and "Contradiction".
|
||
**Routing should have included it:** YES — this is a routing defect. The pack's Routing Test B presents an unverifiable claim as fact.
|
||
|
||
### docs/project-knowledge-inventory.md (215 lines)
|
||
**Why:** Task 4 asked where a new developer should begin for engine experiments. Current handoff → project-state → task-context-packs gave the path, but inventory confirmed the Engine Experiment pack's four "always read" documents are all verifiably present in the repository. Also provided confirmation of what the knowledge-management phase created.
|
||
**Routing should have included it:** DEBATED — the inventory validates pack completeness but was not strictly necessary to answer Task 4 from routing alone.
|
||
|
||
### docs/current-implementation-verification.md (111 lines)
|
||
**Why:** Cross-checked Behaviour Selection's isolation against current-project-state §3's classification. Found section 3b confirming `selectBehaviour` has no callers outside its module.
|
||
**Routing should have included it:** DEBATED — current-project-state already stated the same fact; this was a corroboration, not a gap fill.
|
||
|
||
## Tasks Completed
|
||
|
||
### Task 1 — Does Behaviour Selection affect engine behaviour?
|
||
- **Answer:** No. It is isolated — no import or call exists in any file under lib/ or app/.
|
||
- **Initial docs sufficient:** Yes (current-project-state §3 + handoff §2).
|
||
- **Expansion needed:** No.
|
||
|
||
### Task 2 — Correct mock scenarios for long investigation and contradictory evidence?
|
||
- **Answer:** "Long investigation (10–15 turns)" and "Contradiction" from ui-mock-reference.md.
|
||
- **Initial docs sufficient:** No. Routing Test B claimed they were, but the claim was unverifiable until ui-mock-reference was loaded.
|
||
- **Expansion needed:** Yes — `docs/ui-mock-reference.md`.
|
||
|
||
### Task 3 — Why passive classifiers are not part of active reasoning?
|
||
- **Answer:** Passive classifiers (Experiments 18–25B) record diagnostic signals for future use but have no integration into the turn cycle. Investigation-state assessment is the only one called at all, and its result goes into a diagnostics field — never checked by conditional branches. Others have zero callers. None control user-facing decisions or path selection.
|
||
- **Initial docs sufficient:** Yes (current-project-state §2–§5 + handoff §2).
|
||
- **Expansion needed:** No.
|
||
|
||
### Task 4 — Where should a new developer begin for the next engine experiment?
|
||
- **Answer:** Read `docs/current-handoff.md` → `docs/current-project-state.md` → Engine Experiment pack from `docs/task-context-packs.md`, which directs them to four always-read documents (`current-project-state`, `current-working-principles`, `architecture-guardrails`, `current-implementation-verification`) plus the immediately previous experiment entry in the design log. The pack's "Stop and ask" rules prevent blind expansion.
|
||
- **Initial docs sufficient:** Partially — pack routing is clear but some verification required confirming documents exist (inventory).
|
||
- **Expansion needed:** Marginally — project-knowledge-inventory for pack completeness check.
|
||
|
||
## Routing Failures Found
|
||
|
||
**One genuine failure: Routing Test B in task-context-packs.md.**
|
||
The test states that mock scenarios for "long investigation" and "contradiction" are identifiable without loading ui-mock-reference. This was presented as a self-evident fact but could not be verified from the stated documents alone — the specific scenario names exist only in ui-mock-reference.md. The routing is incomplete; it should have included the mock reference file.
|
||
|
||
**One questionable exclusion: project-knowledge-inventory for Task 4.**
|
||
The task-context-packs Engine Experiment pack lists four "always read" documents but does not themselves confirm all four exist. A cautious developer would load the inventory to verify, adding ~215 lines. This is acceptable cost but worth noting as a gap in the pack's self-validation.
|
||
|
||
## Documentation Improvements Discovered
|
||
|
||
1. **Routing Test B must include ui-mock-reference.md.** Remove the "No extra file required" claim and add the mock reference to the Engine Experiment pack's routing chain when tasks involve scenario selection.
|
||
2. **Packs should confirm their listed documents exist.** Adding a verification check (or removing unverified entries) would prevent the need for inventory cross-referencing.
|
||
|
||
## Overall Assessment: Mostly ready
|
||
|
||
Evidence: Three of four tasks were completed from initial context only. One routing defect was found (Task 2's claim was unverifiable without extra loading). The system works but Routing Test B demonstrates that "sufficient" claims should be evidence-based, not assumed. After fixing Test B, the reduced context system is ready for normal work.
|