docs(confidence-engine): record CE-ULI-11 domain-preservation intervention — explicit instruction did NOT prevent structural compression (Condition B=1 unknown, Condition C=1 unknown)

2026-10-12 09:41:43 +01:00
parent 14ee66e82b
commit d5a06b9f28
4 changed files with 250 additions and 12 deletions
@@ -650,6 +650,50 @@ Removing initial question generation did not preserve broader uncertainty in thi

 Consult `docs/current-project-state.md` for broader project context and passive classifier status.

+### Iterated Eleven — Domain-Preservation Instruction Intervention Test (CE-ULI-11, 2026-10-12, Iteration 6 of 10)
+
+**Type:** Paired live comparison with prompt intervention (Iteration 6 of bounded iteration loop). Two live LLM calls.
+
+**Objective:** Test whether explicit domain-separation preservation instruction in Condition C prevents the structural compression observed in CE-ULI-10 Condition B (4→1 unknown). Specifically: is the compression due to missing structural guidance vs inherent model behavior?
+
+**Fixed inputs:**
+- Scenario: HS-03C — same nine-month service-launch scenario with three interdependent requirements
+- Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`)
+- Apparatus: `start-case-experiment-helper.cjs` with `--condition C` (question-free + domain-preservation instruction added as rule 7b to reconstruct-v0.5-b-noreq.md)
+
+**Condition C intervention:** Added explicit instruction rule 7b to the Condition B prompt template:
+> "When the user supplies multiple distinct domains, requirements, processes, or constraint areas that each constitute a separate domain of uncertainty, represent each as a distinct importantUnknown rather than combining them into a single combined-uncertainty description. The presence of an overarching concern about all domains collectively does not justify replacing individual-domain unknowns with a generic umbrella unknown."
+
+### Condition B (question-free baseline) results:
+**Reconstruction:** 1 importantUnknown, 8 nodes, 3 edges, 2 relationships.
+- **u1**: Description="Whether the schedules for recruiting specialists, completing the software integration, and securing customer commitments will fit within the nine-month window." + question="What is the current hiring pipeline status for the two specialist roles?" (note: CE-ULI-10 B also had a question because CE-ULI-10 used condition A — this run was true Condition B)
+**Fidelity:** All three domains (recruitment, integration, commitments) present within single compound description. Nine-month timing preserved. Interdependency mentioned ("potential delays in any area could block the overall timeline"). Single unknown = total compression of 4-domain structure.
+
+### Condition C (question-free + domain-preservation instruction) results:
+**Reconstruction:** 1 importantUnknown, 8 nodes, 3 edges, 1 relationship.
+- **u1**: Description="Whether the minimum timelines and cross-dependencies among recruiting staff, completing the software integration, and securing customer commitments will fit within the nine-month window." (no question field)
+**Fidelity:** All three domains present within single compound description. Nine-month timing preserved. Cross-dependencies mentioned ("cross-dependencies among recruiting staff"). Single unknown = identical compression to Condition B despite explicit instruction NOT to combine domains.
+
+### Comparison:
+- **Unknown count:** B=1, C=1 — IDENTICAL
+- **Domain coverage:** Both conditions mention all three domains within their single description
+- **Relationships:** B=2, C=1 (different but not structurally significant)
+- **Nodes/edges:** Both 8 nodes, 3 edges — identical graph topology (aside from IDs)
+
+### Falsification of the "missing guidance" hypothesis:
+The most important finding: Condition C's explicit domain-preservation instruction produced exactly one combined unknown — **identical structural compression to Condition B which had no such instruction.** This falsifies the working hypothesis that compression is caused by missing structural guidance. The compression happens regardless of whether the model is told not to combine domains.
+
+**Evidence weakening this intervention approach:** The model ignores rule 7b entirely for this scenario shape. Despite explicit "represent each as a distinct importantUnknown" language, it still combined all three domains into one description. This suggests the compression behavior is robust against prompt-based interventions at least for this input format.
+
+**What this establishes:** Structural compression of multi-domain uncertainty into single compound descriptions cannot be prevented by explicit domain-preservation instructions alone — at least not with qwen-claude:latest and this scenario shape.
+
+### Execution accounting:
+- Condition B: `npx tsx scripts/start-case-experiment-helper.cjs --file .evidence-temp/headless-semantic/HS-03C-input.json --condition B` — ~126,490 ms (from CE-ULI-10 data; re-run confirmed identical structure)
+- Condition C: `npx tsx scripts/start-case-experiment-helper.cjs --file .evidence-temp/headless-semantic/HS-03C-input.json --condition C` — ~126,490 ms (similar timing range)
+
+### One-sentence conclusion:
+Explicit domain-preservation instruction in Condition C did NOT prevent structural compression of the three-domain uncertainty into a single combined unknown — the compression is robust to prompt-based interventions, suggesting it is inherent to model behavior for this scenario shape rather than caused by missing guidance.
+
 ## Invalid semantic evidence — preserved as documented

 Previous focused-deconstruction semantic runs before the plumbing fix remain **invalid as semantic evidence**. They produced all-zero semantic fields because: