docs(confidence-engine): record CE-ULI-10 paired observation — question-free reconstruction does not preserve broader uncertainty (single paired comparison, HS-03C)

Iteration 5 of bounded iteration loop. Two live calls per condition
using fixed HS-03C input (new-service-launch scenario). Model:
qwen-claude:latest via Ollama.

Condition A (question-required): 2 importantUnknowns with broad
descriptions + narrowing questions, 8 nodes, valid validation.

Condition B (question-free): 1 importantUnknown with structurally
narrowed combined-timeline description, 7 nodes, partial validation.

Neither condition preserved the original four-domain structure of
HS-03C. Narrowing occurred through different mechanisms in each
condition — hypothesis unconfirmed for this scenario shape.

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
2026-10-12 07:54:35 +01:00
co-authored by Claude Code
parent 742d7c319b
commit 14ee66e82b
2 changed files with 94 additions and 0 deletions
+92
View File
@@ -556,6 +556,98 @@ These results are documented as **historical experiment evidence**. The v0.61 li
| September 8 product/economics checkpoint | `docs/confidence-engine-product-checkpoint-2026-09-08.md` |
| Full experiment history (specific) | `docs/design-evolution/README.md` → relevant chapter |
## Iterated Ten — Paired Uncertainty-Description Comparison Without Initial Question Generation (CE-ULI-10, 2026-10-12, Iteration 5 of 10)
**Type:** One-paired-live-call semantic comparison (Iteration 5 of bounded iteration loop). Two live LLM calls.
**Objective:** Test whether removing initial question generation changes the fidelity of canonical uncertainty descriptions in a historical case where the original question transformation exhibited partial semantic narrowing.
**Fixed inputs:**
- Scenario: "Our company is considering launching a new service within nine months. The launch would require recruiting two specialist employees, completing a new software integration, and securing commitments from enough customers to cover the ongoing operating costs. Each requirement appears achievable individually, but delays in one area could affect the others. I need to understand whether we can realistically launch the service within nine months with all three requirements sufficiently satisfied."
- Input file: `.evidence-temp/headless-semantic/HS-03C-input.json` (scenarioId: `new-service-launch`)
- Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`)
- Apparatus: `start-case-experiment-helper.cjs` with `--condition A` (question-required, production prompt+schema) and `--condition B` (question-free, alt prompt+schema)
### Historical narrowing case (from HS-03C-result.json reconstruction):
The original reconstruction produced four importantUnknowns, each narrowed by its generated question:
1. **u1** (recruitment): Description="Timeline and feasibility of recruiting two specialists within the nine-month window" → Question narrowed to hiring pipeline status.
2. **u2** (integration): Description="Timeline for designing, testing, and deploying the software integration..." → Question narrowed to "planned development schedule."
3. **u3** (customer commitments): Description="Volume and activation timeline of customer commitments needed..." → Question narrowed to current count.
4. **u4** (cross-area dependencies): Description="Nature and magnitude of dependencies between the three requirements that could cause cascading timeline delays" → Question narrowed to specific bottlenecks.
The narrowing pattern: each question moved from broader feasibility/timing/dependency uncertainties toward present evidence, schedules, or operational bottlenecks. The historical case is about individual domain uncertainties (recruitment, integration, commitments, dependencies) being individually preserved as descriptions but narrowed by their generated questions.
### Fixture equivalence:
The HS-03C-input.json scenario represents the same nine-month service-launch case. It contains all three interdependent requirements and the explicit cross-area delay concern ("delays in one area could affect the others"). **Equivalent.**
### Condition A (question-required) results:
**Reconstruction:** 2 importantUnknowns, 8 nodes, 4 edges, 1 relationship.
- **u1**: Description="The feasibility of launching the service within nine months while satisfying recruitment, integration, and customer commitment requirements." + question="Can we realistically launch the service within nine months with recruiting, integration, and customer commitments all sufficiently satisfied?"
- **u2**: Description="How delays in one requirement could cascade to affect the feasibility of the others within the fixed deadline." + question="In what specific ways could a delay in recruiting, integration, or customer acquisition affect the timeline for the other requirements?"
**Fidelity evaluation:**
1. Relevant uncertainty identified: YES — broad launch-feasibility concern preserved as u1; cascading-delay dependency preserved as u2. However, individual domain uncertainties (recruitment-specific, integration-specific, commitment-specific) were not separately retained. They are subsumed into the broader descriptions.
2. Canonical description retained broader subject: YES — both u1 and u2 descriptions remain broad, covering combined feasibility and cascading delays without narrowing to operational specifics.
3. Qualifications/timing/conditions preserved: PARTIAL — "nine months" constraint retained; "three interdependent requirements" context partially present (via description wording); the individual-achievable-vs-combined-feasibility distinction from original scenario is implicit in summary but not explicit in descriptions.
4. Specific operational subquestion displaced broader uncertainty: NO for descriptions, YES for generated questions. The descriptions remain broad, but both generated questions narrow toward specific operational concerns (feasibility of all simultaneously; specific ways delays cascade). This is the historical HS-03C narrowing pattern reproduced via question generation.
5. Unsupported premises introduced: NONE — no unsupported causal claims or speculative elements.
6. Material uncertainty omitted: PARTIAL — individual domain uncertainties (e.g., recruitment timeline specifically, integration schedule specifically) were not separately identified, though they are subsumed under the broader u1 description. This is compression, not loss of core concern.
7. A generated a narrower question without narrowing its canonical description: YES — both questions narrowed toward operational specifics while their respective descriptions remained broadly framed about combined feasibility and cascading delays.
### Condition B (question-free) results:
**Reconstruction:** 1 importantUnknown, 7 nodes, 4 edges, 1 relationship.
- **u1**: Description="Whether the combined timelines and execution order for recruiting, software integration, and securing customer commitments will fit within the nine-month constraint without cumulative delays exceeding it." (no question field)
**Fidelity evaluation:**
1. Relevant uncertainty identified: YES — the core launch-feasibility concern across all three requirements is captured. However, cross-area dependency risk (cascading delays) is no longer a distinct uncertainty but subsumed into a single combined-timeline description.
2. Canonical description retained broader subject: PARTIAL — the description is broad in scope (all three requirements) but narrowed in framing: "combined timelines and execution order" compresses what were previously separate domain uncertainties into a single compound formulation about timing/order. The original individual-domain structure is lost.
3. Qualifications/timing/conditions preserved: PARTIAL — "nine-month constraint" retained; interdependency context present ("cumulative delays"); but the original scenario's explicit distinction between individually-achievable and collectively-difficult conditions is not captured in the uncertainty description (only implied in the summary).
4. Specific operational subquestion displaced broader uncertainty: N/A — no question was generated.
5. Unsupported premises introduced: NONE — no unsupported claims.
6. Material uncertainty omitted: PARTIAL — individual domain focus and separate cross-area dependency risk were compressed into a single combined-timeline description. The structural loss of independent domain uncertainties is notable (reduction from 4 in original HS-03C to 1 here).
7. N/A — no question generated.
8. B avoided question generation but nevertheless narrowed its description: YES — the single description ("combined timelines and execution order... without cumulative delays") is itself a narrowing from the broader four-domain structure of the original reconstruction. The narrowing occurred structurally during description generation, not via a subsequent question.
### Paired interpretation:
**Key finding:** Neither condition clearly preserves broader uncertainty better. They narrow through different mechanisms:
- Condition A keeps broad descriptions but generates narrowing follow-up questions (the historical HS-03C pattern).
- Condition B avoids question generation but produces a structurally narrower single compound description (combined timelines + execution order).
**Hypothesis status:** The hypothesis predicted that removing question generation would improve semantic fidelity. This single paired observation does NOT support that prediction. Condition B's narrowing was structural (compression during uncertainty identification) rather than question-mediated. Both conditions arrived at similar outcomes: a combined-timeline framing that subsumes individual domain uncertainties, albeit via different mechanisms.
**Evidence supporting the hypothesis:** Condition A's descriptions remained broad even while its questions narrowed — suggesting that *if* descriptions were preserved but questions eliminated, the broader framing could survive. This is the strongest point in favor of the question-free approach.
**Evidence weakening the hypothesis:** Condition B's single description was itself narrower than Condition A's two descriptions combined. The model compressed uncertainty structure during identification (not during questioning). Removing question generation alone does not prevent structural compression.
**Most important distinction (points 7 vs 8):** Condition A generated narrower questions WITHOUT narrowing its descriptions. Condition B avoided generating questions but STILL narrowed its description. This is a crucial asymmetry: the absence of a follow-up question did NOT mean preservation of broader uncertainty structure in Condition B. The narrowing happened at the identification stage, before any question existed.
### Execution accounting:
- Condition A: `npx tsx scripts/start-case-experiment-helper.cjs --file .evidence-temp/headless-semantic/HS-03C-input.json --condition A` — end-to-end 156,572 ms (model response 111,537 ms)
- Condition B: `npx tsx scripts/start-case-experiment-helper.cjs --file .evidence-temp/headless-semantic/HS-03C-input.json --condition B` — end-to-end 126,490 ms (model response 72,942 ms)
- Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`)
- Condition B was ~19% faster end-to-end and ~35% faster model response. This timing difference is within normal stochastic variation for different generation depths (2 unknowns vs 1 unknown).
### Confounds:
- **Stochastic variation:** Two independent LLM calls — results may differ on re-run even with identical inputs.
- **Input granularity mismatch:** The HS-03C-input.json scenario text ("Each requirement appears achievable individually, but delays in one area could affect the others") is a single sentence about interdependencies. It does not contain enough detail for the model to independently distinguish four domain-specific uncertainties (as seen in the original HS-03C reconstruction which presumably had more detailed input). The compression from 4 → 2 (Condition A) or 4 → 1 (Condition B) may reflect input constraints rather than model behavior.
- **Model variability:** Different LLM call times suggest different internal processing paths, making direct comparison of outputs at face value inherently limited.
### What remains uncertain:
- Whether removing question generation would improve semantic fidelity in scenarios where the model produces multiple distinct uncertainties (vs. this case where the input compressed them).
- Whether the narrowing observed is universal across scenario shapes or specific to this service-launch shape.
- Whether a single compound description (Condition B) is epistemically worse than two broad descriptions with narrow questions (Condition A) for user comprehension and investigation utility.
- Single-paired observation; single scenario shape; single model configuration.
### Return-to-work note:
CE-ULI-10 did not support the hypothesis that removing question generation preserves broader uncertainty. Both conditions narrowed through different mechanisms: A via question generation on top of broad descriptions, B via structural compression during identification. Next steps should investigate (a) whether input scenario detail level drives compression vs. model behavior, (b) multi-scenario comparison before deciding to adopt or revise the question-free approach, and (c) whether user comprehension differs between broad-description-narrow-question (A) and narrow-description-no-question (B).
### One-sentence conclusion:
Removing initial question generation did not preserve broader uncertainty in this case — Condition B compressed uncertainty structurally during identification while Condition A kept descriptions broad but generated narrowing questions, meaning neither condition clearly preserved the four-domain structure of the original HS-03C reconstruction.
Consult `docs/current-project-state.md` for broader project context and passive classifier status.
## Invalid semantic evidence — preserved as documented