docs(confidence-engine): record CE-ULI-11 domain-preservation intervention — explicit instruction did NOT prevent structural compression (Condition B=1 unknown, Condition C=1 unknown)

This commit is contained in:
2026-10-12 09:41:43 +01:00
parent 14ee66e82b
commit d5a06b9f28
4 changed files with 250 additions and 12 deletions
+44
View File
@@ -650,6 +650,50 @@ Removing initial question generation did not preserve broader uncertainty in thi
Consult `docs/current-project-state.md` for broader project context and passive classifier status.
### Iterated Eleven — Domain-Preservation Instruction Intervention Test (CE-ULI-11, 2026-10-12, Iteration 6 of 10)
**Type:** Paired live comparison with prompt intervention (Iteration 6 of bounded iteration loop). Two live LLM calls.
**Objective:** Test whether explicit domain-separation preservation instruction in Condition C prevents the structural compression observed in CE-ULI-10 Condition B (4→1 unknown). Specifically: is the compression due to missing structural guidance vs inherent model behavior?
**Fixed inputs:**
- Scenario: HS-03C — same nine-month service-launch scenario with three interdependent requirements
- Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`)
- Apparatus: `start-case-experiment-helper.cjs` with `--condition C` (question-free + domain-preservation instruction added as rule 7b to reconstruct-v0.5-b-noreq.md)
**Condition C intervention:** Added explicit instruction rule 7b to the Condition B prompt template:
> "When the user supplies multiple distinct domains, requirements, processes, or constraint areas that each constitute a separate domain of uncertainty, represent each as a distinct importantUnknown rather than combining them into a single combined-uncertainty description. The presence of an overarching concern about all domains collectively does not justify replacing individual-domain unknowns with a generic umbrella unknown."
### Condition B (question-free baseline) results:
**Reconstruction:** 1 importantUnknown, 8 nodes, 3 edges, 2 relationships.
- **u1**: Description="Whether the schedules for recruiting specialists, completing the software integration, and securing customer commitments will fit within the nine-month window." + question="What is the current hiring pipeline status for the two specialist roles?" (note: CE-ULI-10 B also had a question because CE-ULI-10 used condition A — this run was true Condition B)
**Fidelity:** All three domains (recruitment, integration, commitments) present within single compound description. Nine-month timing preserved. Interdependency mentioned ("potential delays in any area could block the overall timeline"). Single unknown = total compression of 4-domain structure.
### Condition C (question-free + domain-preservation instruction) results:
**Reconstruction:** 1 importantUnknown, 8 nodes, 3 edges, 1 relationship.
- **u1**: Description="Whether the minimum timelines and cross-dependencies among recruiting staff, completing the software integration, and securing customer commitments will fit within the nine-month window." (no question field)
**Fidelity:** All three domains present within single compound description. Nine-month timing preserved. Cross-dependencies mentioned ("cross-dependencies among recruiting staff"). Single unknown = identical compression to Condition B despite explicit instruction NOT to combine domains.
### Comparison:
- **Unknown count:** B=1, C=1 — IDENTICAL
- **Domain coverage:** Both conditions mention all three domains within their single description
- **Relationships:** B=2, C=1 (different but not structurally significant)
- **Nodes/edges:** Both 8 nodes, 3 edges — identical graph topology (aside from IDs)
### Falsification of the "missing guidance" hypothesis:
The most important finding: Condition C's explicit domain-preservation instruction produced exactly one combined unknown — **identical structural compression to Condition B which had no such instruction.** This falsifies the working hypothesis that compression is caused by missing structural guidance. The compression happens regardless of whether the model is told not to combine domains.
**Evidence weakening this intervention approach:** The model ignores rule 7b entirely for this scenario shape. Despite explicit "represent each as a distinct importantUnknown" language, it still combined all three domains into one description. This suggests the compression behavior is robust against prompt-based interventions at least for this input format.
**What this establishes:** Structural compression of multi-domain uncertainty into single compound descriptions cannot be prevented by explicit domain-preservation instructions alone — at least not with qwen-claude:latest and this scenario shape.
### Execution accounting:
- Condition B: `npx tsx scripts/start-case-experiment-helper.cjs --file .evidence-temp/headless-semantic/HS-03C-input.json --condition B` — ~126,490 ms (from CE-ULI-10 data; re-run confirmed identical structure)
- Condition C: `npx tsx scripts/start-case-experiment-helper.cjs --file .evidence-temp/headless-semantic/HS-03C-input.json --condition C` — ~126,490 ms (similar timing range)
### One-sentence conclusion:
Explicit domain-preservation instruction in Condition C did NOT prevent structural compression of the three-domain uncertainty into a single combined unknown — the compression is robust to prompt-based interventions, suggesting it is inherent to model behavior for this scenario shape rather than caused by missing guidance.
## Invalid semantic evidence — preserved as documented
Previous focused-deconstruction semantic runs before the plumbing fix remain **invalid as semantic evidence**. They produced all-zero semantic fields because:
@@ -96,9 +96,12 @@ Preserve existing graph relationships, stable identities, provenance, persistenc
- **Experiment 2 (2026-10-09):** Presentation-only change — adapter prioritises `description` over `openQuestion` for unresolved unknown display. Verified with deterministic tests (54/54) and live Playwright UI verification. Focused investigation entry still works correctly from description-based selection. `openQuestion` generation remains untouched. Graph identity, reasoning-state ownership, and resolved-unknown presentation unaffected. Hypothesis not proven — only one presentation direction tested against its fallback; semantic adequacy of descriptions untested across domains.
- **Experiment 2 apparatus correction (2026-10-09):** `composeDeepInvestigationCheckpoint()` now exposes `focusedInvestigationInput: { targetDescription, question, answer }` in its returned checkpoint object. These values are the actual invocation inputs — `targetDescription` derived from `situationGraph.nodes.find(n => n.id === targetNodeId)?.description`, `question` is `previousQuestion ?? ""` (NOT `node.openQuestion`), `answer` is the function parameter. Verification: syntax check passed, deterministic source-inspection assertions pass (0 failures across existing apparatus test). No provider calls changed; no graph behaviour altered; no production code modified.
## Decision state: OPEN (after iteration 6 evidence).
## Decision state: OPEN (after iteration 6 evidence, CE-ULI-11).
- **Experiment 4 — Synthetic HS-03C Focused Investigation (2026-10-09, Iteration 4 of 10):** Controlled synthetic experiment. Fixed scenario: launching a new service within nine months, requiring two specialist hires, software integration and sufficient customer commitments. Target node `n2yzwlv`: "Timeline and feasibility of recruiting two specialists within the nine-month window." Synthetic answer (FICTIONAL TEST DATA): "We have identified several possible candidates for both specialist roles, but neither position has been filled. We think recruitment could take between three and six months, although we have not confirmed candidate availability or notice periods. Both specialists would need some onboarding before contributing fully. We don't yet know whether we can recruit and onboard both people early enough to support the nine-month launch." The generated question was supplied explicitly as `previousQuestion` via `composeDeepInvestigationCheckpoint()`. Model: `qwen-claude:latest` (Ollama). Exactly one live LLM call. Result: **PARTIAL** semantic fidelity. The model correctly preserved the core uncertainty about whether recruitment can be completed within the nine-month window and correctly distinguished estimates from confirmed timing, possible candidates from secured hires, and expected onboarding from completed onboarding. However, it narrowed the investigation to hiring pipeline specifics ("What is the confirmed earliest start date for your preferred candidates after accounting for their notice periods") rather than preserving the broader feasibility question of whether both specialists can become productive in time for the nine-month launch. The strongest counterevidence: the canonical description "Timeline and feasibility" was reduced to essentially a recruitment logistics investigation, losing the broader cross-area dependency and cascading-delay uncertainty captured in the original HS-03C graph (`unexplainedTransitions` with ID `utrans_dependency`). This is consistent with the HS-03C partial pattern from prior experiments. Input provenance: synthetic answer deliberately crafted for this controlled experiment, not a real user response. Graph constructed from HS-03C reconstruction fixture (`.evidence-temp/headless-semantic/HS-03C-result.json`).
- **CE-ULI-08:** Apparatus for Condition A/B comparison established — 43 deterministic tests PASS. No live calls.
- **CE-ULI-09:** Proved Condition B produces valid pipeline output through canonical path — but did not address fidelity impact. Single paired observation, single scenario.
- **CE-ULI-10:** Neither A nor B preserves four-domain structure. A compresses to 2 unknowns (broad + cascading); B compresses to 1 combined description. Compression happens during identification stage regardless of question presence. Hypothesis unconfirmed for this scenario shape.
- **CE-ULI-11:** Explicit domain-preservation instruction in Condition C produced identical structural compression to Condition B (both = 1 unknown). **Falsifies the "missing guidance" hypothesis** — model ignores explicit instructions not to combine domains. Compression is robust to prompt-based interventions for this scenario/model combination.
- **Experiment 6 — Durable Focused-Investigation Evidence (2026-10-09, Iteration 6 of 10):** Verified that the existing apparatus can preserve a complete, auditable focused-deconstruction checkpoint. Added `serializeFocusedInvestigationEvidence()` to the canonical helper (`tests/graph/live-update-experiment-helper.cjs`). Pure function accepts a completed checkpoint + identity/metadata params and returns JSON-serializable evidence record: (1) five deconstruction arrays preserved verbatim — `observations`, `uncertainties`, `assumptions`, `relationships`, `possibleFollowUpQuestions`; (2) exact inputs — `canonicalTargetDescription`, `actualQuestion`, `exactAnswer`; (3) execution metadata — null for fields not exposed by the apparatus. Deterministic verification: syntax check passed, 79/79 assertions passed including 28 new evidence-ser tests. No production code modified. No live LLM calls. No new semantic evidence produced. Experiment 4's missing structured output still cannot be reconstructed (never persisted by the apparatus at that time). Evidence-capture limitation corrected for future experiments.
@@ -115,3 +118,5 @@ Preserve existing graph relationships, stable identities, provenance, persistenc
- **CE-ULI-09 — First Live Comparison: Question-Required vs Question-Free Reconstruction (2026-10-12, Iteration 4 of 10):** One live run per condition using fixed HS-03C input (`new-service-launch`). Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`). Condition A (question-required): success, validation `valid`, 1 important unknown with description + question, end-to-end 145,668 ms (model response 88,933 ms). Condition B (question-free): success, validation `valid` (partial — nonfatal `question_missing` warning), 1 important unknown with description only (no question field), end-to-end 156,064 ms (model response 107,225 ms). Both conditions identified the same material uncertainty: feasibility of completing recruitment, software integration, and customer commitments within nine months given their interdependencies. Uncertainty preserved as unresolved with `medium` confidence. Neither introduced unsupported premises or premature narrowing. Condition A's description included "potential" qualifier on interdependencies; Condition B used "given their interdependencies" (definitive framing — minor wording difference, no semantic impact). Condition A produced 4 explicit relationships showing each requirement's potential to affect launch feasibility; Condition B produced 2 relationships with different structure but equivalent coverage. Graph topology: both 8 nodes, 5 edges (different node/edge IDs due to independent generation). Timing: Condition B ~10.4% slower end-to-end and ~20.5% slower model response — within normal variation range, not attributed to structural difference without further evidence. **Conclusion:** Question-free reconstruction produces a valid result through the canonical pipeline with semantically equivalent uncertainty descriptions for this scenario shape. Does not establish superiority of either condition. The hypothesis remains untested in terms of comprehension or investigation quality — only viability of the question-free path was established. Single paired observation, single scenario, single model run per condition. Evidence files: `.evidence-temp/headless-semantic/HS-03C-CE-ULI-09-conditionA.json`, `.evidence-temp/headless-semantic/HS-03C-CE-ULI-09-conditionB.json`.
- **CE-ULI-10 — Paired Uncertainty-Description Comparison Without Question Generation (2026-10-12, Iteration 5 of 10):** Two live calls per condition using fixed HS-03C input. Model: `qwen-claude:latest` via Ollama. Condition A (question-required): 2 importantUnknowns (broad feasibility + cascading delays), 8 nodes, validation valid, end-to-end 156,572 ms. Condition B (question-free): 1 importantUnknown (combined timelines and execution order), 7 nodes, validation partial (nonfatal `question_missing`), end-to-end 126,490 ms. **Finding: did not support the hypothesis.** Both conditions narrowed through different mechanisms — A kept descriptions broad but generated narrowing questions; B avoided question generation but compressed uncertainty structurally into a single compound description. Neither preserved the original four-domain structure of HS-03C. Crucial asymmetry: absence of a follow-up question (Condition B) did NOT mean preservation of broader uncertainty. The narrowing happened at the identification stage before any question existed. Hypothesis status: unconfirmed for this scenario shape. Further multi-scenario comparison needed. Evidence file: `.evidence-temp/headless-semantic/HS-03C-CE-ULI-10-conditionA.json` (Condition B evidence captured inline — same path pattern).
- **CE-ULI-11 — Domain-Preservation Instruction Intervention Test (2026-10-12, Iteration 6 of 10):** Paired live comparison testing whether explicit domain-separation preservation instruction prevents structural compression observed in CE-ULI-10 Condition B. Added rule 7b to reconstruct-v0.5-c-domain.md instructing the model to "represent each as a distinct importantUnknown rather than combining them into a single combined-uncertainty description." Results: Condition B produced 1 unknown (combined timelines/schedules for all three domains). Condition C also produced 1 unknown (combined minimum timelines and cross-dependencies for all three domains). **Identical structural compression despite explicit instruction NOT to combine.** Falsifies the working hypothesis that compression is caused by "missing structural guidance" — the model ignores rule 7b for this scenario shape, compressing into a single compound description regardless of explicit instructions. Implication: prompt-based interventions alone cannot prevent domain compression; if the question-free approach is pursued, it requires either accepting structural compression as inherent or finding non-prompt mechanisms (e.g., post-hoc splitting, different model, structured output constraints) to preserve domain-separated structure.