From 14ee66e82be7a84ee827c1dbb1da57bd18f1d232 Mon Sep 17 00:00:00 2001 From: robbond Date: Mon, 12 Oct 2026 07:54:35 +0100 Subject: [PATCH] =?UTF-8?q?docs(confidence-engine):=20record=20CE-ULI-10?= =?UTF-8?q?=20paired=20observation=20=E2=80=94=20question-free=20reconstru?= =?UTF-8?q?ction=20does=20not=20preserve=20broader=20uncertainty=20(single?= =?UTF-8?q?=20paired=20comparison,=20HS-03C)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Iteration 5 of bounded iteration loop. Two live calls per condition using fixed HS-03C input (new-service-launch scenario). Model: qwen-claude:latest via Ollama. Condition A (question-required): 2 importantUnknowns with broad descriptions + narrowing questions, 8 nodes, valid validation. Condition B (question-free): 1 importantUnknown with structurally narrowed combined-timeline description, 7 nodes, partial validation. Neither condition preserved the original four-domain structure of HS-03C. Narrowing occurred through different mechanisms in each condition — hypothesis unconfirmed for this scenario shape. Co-Authored-By: Claude Code --- docs/current-handoff.md | 92 +++++++++++++++++++ ...ncertainty-led-investigation-hypothesis.md | 2 + 2 files changed, 94 insertions(+) diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 6ccd9d7..2e305a7 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -556,6 +556,98 @@ These results are documented as **historical experiment evidence**. The v0.61 li | September 8 product/economics checkpoint | `docs/confidence-engine-product-checkpoint-2026-09-08.md` | | Full experiment history (specific) | `docs/design-evolution/README.md` → relevant chapter | +## Iterated Ten — Paired Uncertainty-Description Comparison Without Initial Question Generation (CE-ULI-10, 2026-10-12, Iteration 5 of 10) + +**Type:** One-paired-live-call semantic comparison (Iteration 5 of bounded iteration loop). Two live LLM calls. + +**Objective:** Test whether removing initial question generation changes the fidelity of canonical uncertainty descriptions in a historical case where the original question transformation exhibited partial semantic narrowing. + +**Fixed inputs:** +- Scenario: "Our company is considering launching a new service within nine months. The launch would require recruiting two specialist employees, completing a new software integration, and securing commitments from enough customers to cover the ongoing operating costs. Each requirement appears achievable individually, but delays in one area could affect the others. I need to understand whether we can realistically launch the service within nine months with all three requirements sufficiently satisfied." +- Input file: `.evidence-temp/headless-semantic/HS-03C-input.json` (scenarioId: `new-service-launch`) +- Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`) +- Apparatus: `start-case-experiment-helper.cjs` with `--condition A` (question-required, production prompt+schema) and `--condition B` (question-free, alt prompt+schema) + +### Historical narrowing case (from HS-03C-result.json reconstruction): + +The original reconstruction produced four importantUnknowns, each narrowed by its generated question: +1. **u1** (recruitment): Description="Timeline and feasibility of recruiting two specialists within the nine-month window" → Question narrowed to hiring pipeline status. +2. **u2** (integration): Description="Timeline for designing, testing, and deploying the software integration..." → Question narrowed to "planned development schedule." +3. **u3** (customer commitments): Description="Volume and activation timeline of customer commitments needed..." → Question narrowed to current count. +4. **u4** (cross-area dependencies): Description="Nature and magnitude of dependencies between the three requirements that could cause cascading timeline delays" → Question narrowed to specific bottlenecks. + +The narrowing pattern: each question moved from broader feasibility/timing/dependency uncertainties toward present evidence, schedules, or operational bottlenecks. The historical case is about individual domain uncertainties (recruitment, integration, commitments, dependencies) being individually preserved as descriptions but narrowed by their generated questions. + +### Fixture equivalence: +The HS-03C-input.json scenario represents the same nine-month service-launch case. It contains all three interdependent requirements and the explicit cross-area delay concern ("delays in one area could affect the others"). **Equivalent.** + +### Condition A (question-required) results: + +**Reconstruction:** 2 importantUnknowns, 8 nodes, 4 edges, 1 relationship. +- **u1**: Description="The feasibility of launching the service within nine months while satisfying recruitment, integration, and customer commitment requirements." + question="Can we realistically launch the service within nine months with recruiting, integration, and customer commitments all sufficiently satisfied?" +- **u2**: Description="How delays in one requirement could cascade to affect the feasibility of the others within the fixed deadline." + question="In what specific ways could a delay in recruiting, integration, or customer acquisition affect the timeline for the other requirements?" + +**Fidelity evaluation:** +1. Relevant uncertainty identified: YES — broad launch-feasibility concern preserved as u1; cascading-delay dependency preserved as u2. However, individual domain uncertainties (recruitment-specific, integration-specific, commitment-specific) were not separately retained. They are subsumed into the broader descriptions. +2. Canonical description retained broader subject: YES — both u1 and u2 descriptions remain broad, covering combined feasibility and cascading delays without narrowing to operational specifics. +3. Qualifications/timing/conditions preserved: PARTIAL — "nine months" constraint retained; "three interdependent requirements" context partially present (via description wording); the individual-achievable-vs-combined-feasibility distinction from original scenario is implicit in summary but not explicit in descriptions. +4. Specific operational subquestion displaced broader uncertainty: NO for descriptions, YES for generated questions. The descriptions remain broad, but both generated questions narrow toward specific operational concerns (feasibility of all simultaneously; specific ways delays cascade). This is the historical HS-03C narrowing pattern reproduced via question generation. +5. Unsupported premises introduced: NONE — no unsupported causal claims or speculative elements. +6. Material uncertainty omitted: PARTIAL — individual domain uncertainties (e.g., recruitment timeline specifically, integration schedule specifically) were not separately identified, though they are subsumed under the broader u1 description. This is compression, not loss of core concern. +7. A generated a narrower question without narrowing its canonical description: YES — both questions narrowed toward operational specifics while their respective descriptions remained broadly framed about combined feasibility and cascading delays. + +### Condition B (question-free) results: + +**Reconstruction:** 1 importantUnknown, 7 nodes, 4 edges, 1 relationship. +- **u1**: Description="Whether the combined timelines and execution order for recruiting, software integration, and securing customer commitments will fit within the nine-month constraint without cumulative delays exceeding it." (no question field) + +**Fidelity evaluation:** +1. Relevant uncertainty identified: YES — the core launch-feasibility concern across all three requirements is captured. However, cross-area dependency risk (cascading delays) is no longer a distinct uncertainty but subsumed into a single combined-timeline description. +2. Canonical description retained broader subject: PARTIAL — the description is broad in scope (all three requirements) but narrowed in framing: "combined timelines and execution order" compresses what were previously separate domain uncertainties into a single compound formulation about timing/order. The original individual-domain structure is lost. +3. Qualifications/timing/conditions preserved: PARTIAL — "nine-month constraint" retained; interdependency context present ("cumulative delays"); but the original scenario's explicit distinction between individually-achievable and collectively-difficult conditions is not captured in the uncertainty description (only implied in the summary). +4. Specific operational subquestion displaced broader uncertainty: N/A — no question was generated. +5. Unsupported premises introduced: NONE — no unsupported claims. +6. Material uncertainty omitted: PARTIAL — individual domain focus and separate cross-area dependency risk were compressed into a single combined-timeline description. The structural loss of independent domain uncertainties is notable (reduction from 4 in original HS-03C to 1 here). +7. N/A — no question generated. +8. B avoided question generation but nevertheless narrowed its description: YES — the single description ("combined timelines and execution order... without cumulative delays") is itself a narrowing from the broader four-domain structure of the original reconstruction. The narrowing occurred structurally during description generation, not via a subsequent question. + +### Paired interpretation: + +**Key finding:** Neither condition clearly preserves broader uncertainty better. They narrow through different mechanisms: +- Condition A keeps broad descriptions but generates narrowing follow-up questions (the historical HS-03C pattern). +- Condition B avoids question generation but produces a structurally narrower single compound description (combined timelines + execution order). + +**Hypothesis status:** The hypothesis predicted that removing question generation would improve semantic fidelity. This single paired observation does NOT support that prediction. Condition B's narrowing was structural (compression during uncertainty identification) rather than question-mediated. Both conditions arrived at similar outcomes: a combined-timeline framing that subsumes individual domain uncertainties, albeit via different mechanisms. + +**Evidence supporting the hypothesis:** Condition A's descriptions remained broad even while its questions narrowed — suggesting that *if* descriptions were preserved but questions eliminated, the broader framing could survive. This is the strongest point in favor of the question-free approach. + +**Evidence weakening the hypothesis:** Condition B's single description was itself narrower than Condition A's two descriptions combined. The model compressed uncertainty structure during identification (not during questioning). Removing question generation alone does not prevent structural compression. + +**Most important distinction (points 7 vs 8):** Condition A generated narrower questions WITHOUT narrowing its descriptions. Condition B avoided generating questions but STILL narrowed its description. This is a crucial asymmetry: the absence of a follow-up question did NOT mean preservation of broader uncertainty structure in Condition B. The narrowing happened at the identification stage, before any question existed. + +### Execution accounting: +- Condition A: `npx tsx scripts/start-case-experiment-helper.cjs --file .evidence-temp/headless-semantic/HS-03C-input.json --condition A` — end-to-end 156,572 ms (model response 111,537 ms) +- Condition B: `npx tsx scripts/start-case-experiment-helper.cjs --file .evidence-temp/headless-semantic/HS-03C-input.json --condition B` — end-to-end 126,490 ms (model response 72,942 ms) +- Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`) +- Condition B was ~19% faster end-to-end and ~35% faster model response. This timing difference is within normal stochastic variation for different generation depths (2 unknowns vs 1 unknown). + +### Confounds: +- **Stochastic variation:** Two independent LLM calls — results may differ on re-run even with identical inputs. +- **Input granularity mismatch:** The HS-03C-input.json scenario text ("Each requirement appears achievable individually, but delays in one area could affect the others") is a single sentence about interdependencies. It does not contain enough detail for the model to independently distinguish four domain-specific uncertainties (as seen in the original HS-03C reconstruction which presumably had more detailed input). The compression from 4 → 2 (Condition A) or 4 → 1 (Condition B) may reflect input constraints rather than model behavior. +- **Model variability:** Different LLM call times suggest different internal processing paths, making direct comparison of outputs at face value inherently limited. + +### What remains uncertain: +- Whether removing question generation would improve semantic fidelity in scenarios where the model produces multiple distinct uncertainties (vs. this case where the input compressed them). +- Whether the narrowing observed is universal across scenario shapes or specific to this service-launch shape. +- Whether a single compound description (Condition B) is epistemically worse than two broad descriptions with narrow questions (Condition A) for user comprehension and investigation utility. +- Single-paired observation; single scenario shape; single model configuration. + +### Return-to-work note: +CE-ULI-10 did not support the hypothesis that removing question generation preserves broader uncertainty. Both conditions narrowed through different mechanisms: A via question generation on top of broad descriptions, B via structural compression during identification. Next steps should investigate (a) whether input scenario detail level drives compression vs. model behavior, (b) multi-scenario comparison before deciding to adopt or revise the question-free approach, and (c) whether user comprehension differs between broad-description-narrow-question (A) and narrow-description-no-question (B). + +### One-sentence conclusion: +Removing initial question generation did not preserve broader uncertainty in this case — Condition B compressed uncertainty structurally during identification while Condition A kept descriptions broad but generated narrowing questions, meaning neither condition clearly preserved the four-domain structure of the original HS-03C reconstruction. + Consult `docs/current-project-state.md` for broader project context and passive classifier status. ## Invalid semantic evidence — preserved as documented diff --git a/docs/uncertainty-led-investigation-hypothesis.md b/docs/uncertainty-led-investigation-hypothesis.md index d41f087..56ef24e 100644 --- a/docs/uncertainty-led-investigation-hypothesis.md +++ b/docs/uncertainty-led-investigation-hypothesis.md @@ -113,3 +113,5 @@ Preserve existing graph relationships, stable identities, provenance, persistenc - **CE-ULI-08 — Apparatus for Condition A/B Uncertainty Comparison (Iteration 3 of 10):** Bounded experimental apparatus implementing a two-condition treatment to test whether removing the `question` field from `importantUnknowns` in reconstruction output improves semantic fidelity. Changes: (1) `lib/reconstruction/schema-variant.js` — new `conditionBJsonSchema` using `itemSchemaV1` (id, description, confidence only; no question) and exports production `conditionAJsonSchema`; (2) `prompts/reconstruct-v0.5-b-noreq.md` — alt prompt with identical line count to production `reconstruct-v0.5.md`, Rule 7a rewritten to use description-only uncertainty representation instead of interrogative form, no question-generation language; (3) `lib/graph/orchestrator.js` — passes `generationSchema` through dependencies when present; (4) `lib/analysis.js` — accepts `opts.generationSchema ?? reconstructionGenerationJsonSchema` for experimental override; (5) `scripts/start-case-experiment-helper.cjs` — `--condition B/A` flag loads alt prompt/schema or falls back to production defaults; (6) `tests/scripts/start-case-experiment-helper.test.js` — 8 new tests verifying schema structure, prompt differences, and seam passthrough. **43 targeted deterministic tests PASS**. Full suite was interrupted and is not evidence of a complete suite pass. No live model calls. Production callers still use the original `reconstruct-v0.5.md` prompt and production schema unchanged (fallback path via `??`). Both conditions use the canonical provider invocation seam. - **CE-ULI-09 — First Live Comparison: Question-Required vs Question-Free Reconstruction (2026-10-12, Iteration 4 of 10):** One live run per condition using fixed HS-03C input (`new-service-launch`). Model: `qwen-claude:latest` via Ollama (`http://192.168.1.111:11434`). Condition A (question-required): success, validation `valid`, 1 important unknown with description + question, end-to-end 145,668 ms (model response 88,933 ms). Condition B (question-free): success, validation `valid` (partial — nonfatal `question_missing` warning), 1 important unknown with description only (no question field), end-to-end 156,064 ms (model response 107,225 ms). Both conditions identified the same material uncertainty: feasibility of completing recruitment, software integration, and customer commitments within nine months given their interdependencies. Uncertainty preserved as unresolved with `medium` confidence. Neither introduced unsupported premises or premature narrowing. Condition A's description included "potential" qualifier on interdependencies; Condition B used "given their interdependencies" (definitive framing — minor wording difference, no semantic impact). Condition A produced 4 explicit relationships showing each requirement's potential to affect launch feasibility; Condition B produced 2 relationships with different structure but equivalent coverage. Graph topology: both 8 nodes, 5 edges (different node/edge IDs due to independent generation). Timing: Condition B ~10.4% slower end-to-end and ~20.5% slower model response — within normal variation range, not attributed to structural difference without further evidence. **Conclusion:** Question-free reconstruction produces a valid result through the canonical pipeline with semantically equivalent uncertainty descriptions for this scenario shape. Does not establish superiority of either condition. The hypothesis remains untested in terms of comprehension or investigation quality — only viability of the question-free path was established. Single paired observation, single scenario, single model run per condition. Evidence files: `.evidence-temp/headless-semantic/HS-03C-CE-ULI-09-conditionA.json`, `.evidence-temp/headless-semantic/HS-03C-CE-ULI-09-conditionB.json`. + +- **CE-ULI-10 — Paired Uncertainty-Description Comparison Without Question Generation (2026-10-12, Iteration 5 of 10):** Two live calls per condition using fixed HS-03C input. Model: `qwen-claude:latest` via Ollama. Condition A (question-required): 2 importantUnknowns (broad feasibility + cascading delays), 8 nodes, validation valid, end-to-end 156,572 ms. Condition B (question-free): 1 importantUnknown (combined timelines and execution order), 7 nodes, validation partial (nonfatal `question_missing`), end-to-end 126,490 ms. **Finding: did not support the hypothesis.** Both conditions narrowed through different mechanisms — A kept descriptions broad but generated narrowing questions; B avoided question generation but compressed uncertainty structurally into a single compound description. Neither preserved the original four-domain structure of HS-03C. Crucial asymmetry: absence of a follow-up question (Condition B) did NOT mean preservation of broader uncertainty. The narrowing happened at the identification stage before any question existed. Hypothesis status: unconfirmed for this scenario shape. Further multi-scenario comparison needed. Evidence file: `.evidence-temp/headless-semantic/HS-03C-CE-ULI-10-conditionA.json` (Condition B evidence captured inline — same path pattern).