docs(confidence-engine): record CE-ULI-10 paired observation — question-free reconstruction does not preserve broader uncertainty (single paired comparison, HS-03C)

Iteration 5 of bounded iteration loop. Two live calls per condition
using fixed HS-03C input (new-service-launch scenario). Model:
qwen-claude:latest via Ollama.

Condition A (question-required): 2 importantUnknowns with broad
descriptions + narrowing questions, 8 nodes, valid validation.

Condition B (question-free): 1 importantUnknown with structurally
narrowed combined-timeline description, 7 nodes, partial validation.

Neither condition preserved the original four-domain structure of
HS-03C. Narrowing occurred through different mechanisms in each
condition — hypothesis unconfirmed for this scenario shape.

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
2026-10-12 07:54:35 +01:00
co-authored by Claude Code
parent 742d7c319b
commit 14ee66e82b
2 changed files with 94 additions and 0 deletions
@@ -113,3 +113,5 @@ Preserve existing graph relationships, stable identities, provenance, persistenc
- **CE-ULI-08 — Apparatus for Condition A/B Uncertainty Comparison (Iteration 3 of 10):** Bounded experimental apparatus implementing a two-condition treatment to test whether removing the `question` field from `importantUnknowns` in reconstruction output improves semantic fidelity. Changes: (1) `lib/reconstruction/schema-variant.js` — new `conditionBJsonSchema` using `itemSchemaV1` (id, description, confidence only; no question) and exports production `conditionAJsonSchema`; (2) `prompts/reconstruct-v0.5-b-noreq.md` — alt prompt with identical line count to production `reconstruct-v0.5.md`, Rule 7a rewritten to use description-only uncertainty representation instead of interrogative form, no question-generation language; (3) `lib/graph/orchestrator.js` — passes `generationSchema` through dependencies when present; (4) `lib/analysis.js` — accepts `opts.generationSchema ?? reconstructionGenerationJsonSchema` for experimental override; (5) `scripts/start-case-experiment-helper.cjs` — `--condition B/A` flag loads alt prompt/schema or falls back to production defaults; (6) `tests/scripts/start-case-experiment-helper.test.js` — 8 new tests verifying schema structure, prompt differences, and seam passthrough. **43 targeted deterministic tests PASS**. Full suite was interrupted and is not evidence of a complete suite pass. No live model calls. Production callers still use the original `reconstruct-v0.5.md` prompt and production schema unchanged (fallback path via `??`). Both conditions use the canonical provider invocation seam.
- **CE-ULI-09 — First Live Comparison: Question-Required vs Question-Free Reconstruction (2026-10-12, Iteration 4 of 10):** One live run per condition using fixed HS-03C input (`new-service-launch`). Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`). Condition A (question-required): success, validation `valid`, 1 important unknown with description + question, end-to-end 145,668 ms (model response 88,933 ms). Condition B (question-free): success, validation `valid` (partial — nonfatal `question_missing` warning), 1 important unknown with description only (no question field), end-to-end 156,064 ms (model response 107,225 ms). Both conditions identified the same material uncertainty: feasibility of completing recruitment, software integration, and customer commitments within nine months given their interdependencies. Uncertainty preserved as unresolved with `medium` confidence. Neither introduced unsupported premises or premature narrowing. Condition A's description included "potential" qualifier on interdependencies; Condition B used "given their interdependencies" (definitive framing — minor wording difference, no semantic impact). Condition A produced 4 explicit relationships showing each requirement's potential to affect launch feasibility; Condition B produced 2 relationships with different structure but equivalent coverage. Graph topology: both 8 nodes, 5 edges (different node/edge IDs due to independent generation). Timing: Condition B ~10.4% slower end-to-end and ~20.5% slower model response — within normal variation range, not attributed to structural difference without further evidence. **Conclusion:** Question-free reconstruction produces a valid result through the canonical pipeline with semantically equivalent uncertainty descriptions for this scenario shape. Does not establish superiority of either condition. The hypothesis remains untested in terms of comprehension or investigation quality — only viability of the question-free path was established. Single paired observation, single scenario, single model run per condition. Evidence files: `.evidence-temp/headless-semantic/HS-03C-CE-ULI-09-conditionA.json`, `.evidence-temp/headless-semantic/HS-03C-CE-ULI-09-conditionB.json`.
- **CE-ULI-10 — Paired Uncertainty-Description Comparison Without Question Generation (2026-10-12, Iteration 5 of 10):** Two live calls per condition using fixed HS-03C input. Model: `qwen-claude:latest` via Ollama. Condition A (question-required): 2 importantUnknowns (broad feasibility + cascading delays), 8 nodes, validation valid, end-to-end 156,572 ms. Condition B (question-free): 1 importantUnknown (combined timelines and execution order), 7 nodes, validation partial (nonfatal `question_missing`), end-to-end 126,490 ms. **Finding: did not support the hypothesis.** Both conditions narrowed through different mechanisms — A kept descriptions broad but generated narrowing questions; B avoided question generation but compressed uncertainty structurally into a single compound description. Neither preserved the original four-domain structure of HS-03C. Crucial asymmetry: absence of a follow-up question (Condition B) did NOT mean preservation of broader uncertainty. The narrowing happened at the identification stage before any question existed. Hypothesis status: unconfirmed for this scenario shape. Further multi-scenario comparison needed. Evidence file: `.evidence-temp/headless-semantic/HS-03C-CE-ULI-10-conditionA.json` (Condition B evidence captured inline — same path pattern).