# Uncertainty Recognition Quality Investigation — Report **Date:** 2026-10-14 **Branch:** `experiment/uncertainty-led-investigation` **Starting HEAD:** `e595ee6` feat(experiments): integrate Condition D/E apparatus **Status:** CLOSED — 10-iteration programme completed and checkpointed --- ## A. Investigation Objective This investigation examined why the configured local LLM (`qwen-claude:latest` via Ollama) produced less useful uncertainty recognition through Confidence Engine's reconstruction pipeline than through a minimal direct prompt. The canonical scenario used was `RFG-03-independent-decisions-input.json`, concerning three independent decisions (replacing internal reporting software, recruiting an additional customer-support employee, refurbishing a meeting room). The user's goal is to understand what remains uncertain before deciding what to do. **Observed phenomenon:** A minimal direct-model prompt produced two outputs containing approximately 21 and 15 uncertainties with meaningful domain-specific unknowns. Confidence Engine reconstruction under Conditions D/E generally produced three generic uncertainties that largely restated the user's stated decisions without adding new insights. --- ## B. Experimental Programme (Iterations 1–10) ### Iteration 3 — Baseline Direct-Model Prompt **Status:** Complete and reproducible **Evidence:** `.evidence-temp/experiment3/` **Prompt:** Minimal raw prompt with no rule-set framing or forced JSON output. **Result:** Two model outputs: raw-run-1 (29 lines, ~15+ domain-specific uncertainties), raw-run-2 (21 lines, similar richness). Scenario SHA-256: `dc9a600f9fb1e7189dd56086a07f357893396a5b0782f4781c95d4aa29fdcf06`. ### Iteration 5 — Condition D Rule-21 Removal **Status:** Complete **Evidence:** `.evidence-temp/iteration5/` **Treatment:** Removed rule 21 from the deconstruction prompt. **Result:** Four CE uncertainties including a cross-domain "worthwhile" framework question. No material improvement over control in domain-specific discovery. Evidence: `D-no21-freeform.txt` (103 lines), `D-no21.json`, `D-original-freeform.txt`, `D-original.json`. ### Iteration 6 — Condition D Rule-23 Removal + Prompt Variant **Status:** Complete **Evidence:** `.evidence-temp/iteration6/` **Treatment:** Removed rule 23; tested with new prompt variant (`reconstruct-v0.5-d-deconstruct-no23.md`). **Result:** No material improvement. Both `run-A-original-D.json` and `run-B-no-rule-23.json` validated as freeform with zero graph nodes. ### Iteration 8 — Invalid Run + Recovery **Status:** Validated invalidity; recovery run complete **Evidence:** `.evidence-temp/iteration8/` (invalid), `.evidence-temp/iteration8-recovery/` (valid) **Invalidity cause:** Scenario filename was treated as scenario text instead of JSON content. **Recovery result:** 4 uncertainty items with modest structural enrichment (expectedStates, differences, tighter relationships) but no new domain-specific unknowns. ### Iterations 9–10 — Follow-up and Block **Iteration 9:** Partial evidence; focused-investigation dependency analysis. `.evidence-temp/iteration9/` contains a `b-variant/` subdirectory (incomplete). **Iteration 10:** BLOCKED before model execution. The existing experimental helper (`scripts/start-case-experiment-helper.cjs`) has no mechanism to pass the exact minimal raw prompt through the CE execution path without source modifications. Every invocation applies a structured prompt template. See `.evidence-temp/iteration10/final-report.txt` and `architecture-finding.txt` for the definitive architecture analysis. ### Headless-Semantic Evidence (HS-01 through HS-03C) **Status:** Extensive supporting evidence preserved **Evidence:** `.evidence-temp/headless-semantic/` (28 files, 2788 lines total) Covers: Question-free reconstruction comparison (CE-ULI-09), domain-preservation instruction tests (CE-ULI-10, CE-ULI-11), and focused-investigation evidence persistence testing. --- ## C. Established Findings These findings are directly supported by the evidence: 1. **Direct-model prompt quality gap:** Minimal direct prompts produced outputs with approximately 21 and 15 domain-specific uncertainties across three decisions, while CE reconstruction under Conditions D/E generally produced three generic restatements of the user's own decisions. 2. **Rule removal had no material effect:** Removing Rule 21 (Iteration 5) and Rule 23 (Iteration 6) did not materially improve uncertainty recognition quality. 3. **Discovery permission instruction produced modest structural enrichment:** The Iteration 8 recovery (adding explicit permission to recognise plausible unstated uncertainties) added four items including a cross-domain evaluation framework question, but no new domain-specific unknowns were discovered beyond what the user stated. 4. **CE execution path is structurally incapable of raw-prompt pass-through:** Every invocation applies a template-based prompt construction that adds rule sets and forced-JSON constraints absent from the direct-model baseline. This was definitively established in Iteration 10's architecture-finding evidence. 5. **Question-free reconstruction does not prevent structural compression:** CE-ULI-10 demonstrated that removing the question-generation requirement still produced structural compression of uncertainties into single compound descriptions for the tested scenario shape. 6. **Explicit domain-separation instructions are ignored:** CE-ULI-11 showed identical compression under both condition B and C despite explicit instruction not to combine domains. --- ## D. Unresolved Hypotheses **H1 — Model capability:** The model may be capable of richer uncertainty recognition but its output quality is constrained by the surrounding apparatus (prompt structure, output format constraints). *Untested.* Not equivalent to H2 or H3. **H2 — Prompt effectiveness:** The template-based prompt construction (rule sets, forced JSON, deconstruction framing) suppresses the model's ability to produce rich uncertainty recognition relative to a minimal direct prompt. *Supported by evidence of quality gap but not proven as causal.* No controlled experiment eliminated the structural difference while preserving all other conditions. **H3 — Confidence Engine orchestration:** Something in the CE pipeline (prompt construction, parsing, validation, graph representation, or presentation) constrains, transforms, or discards meaningful uncertainty recognition between model output and user-facing display. *Supported by evidence but not isolated.* Iteration 10 established that the CE execution path always applies templates, making controlled elimination of specific pipeline stages infeasible without source modifications. **Important:** The precise cause of the quality difference remains unresolved. The evidence demonstrates a quality gap and several plausible explanations, not definitive causation. It is not established that prompt construction, JSON formatting, or orchestration individually or collectively suppress uncertainty recognition — only that these are all uncontrolled confounders in the comparison between CE-executed requests and direct model calls. --- ## E. Experimental Apparatus Limitations 1. **Template-based prompt construction:** All CE invocation paths apply a template (v0.5, b/c/d/e variants). No raw-prompt pass-through mode exists in `lib/analysis.js` or `start-case-experiment-helper.cjs`. 2. **No raw-prompt pass-through in the helper:** Condition selector loads hardcoded template filenames. The `--experiment-instruction` option only appends — it cannot replace or suppress base structure. 3. **`--file` input-loading requirement:** Scenarios must be loaded from JSON files. Line-input and other loading modes are not supported by the helper. 4. **`--reconstruction-only` requirement:** To bypass the graph pipeline, `--reconstruction-only` is required. Its absence routes through the full production path. 5. **Generation-setting comparability limitations:** Direct-model baseline runs (experiment3) used different generation settings (no forced schema) than CE execution (forced JSON schemas). Temperature, top_p, and other parameters are not controlled or logged for direct runs. 6. **Missing evidence:** Iteration 9's b-variant evidence is incomplete (empty subdirectory). Evidence for some earlier iterations beyond condition outputs alone is unavailable. 7. **Single-model limitation:** All experiments used `qwen-claude:latest` only. Results may not generalise to other models. --- ## F. Product Principles The following principles must govern any future work on uncertainty recognition: - The user owns investigation direction. - The engine facilitates rather than steers. - Recognising an uncertainty is not inventing a fact. - Recognising an uncertainty is not selecting it. - The engine must not automatically decide what the user should investigate next. - Fidelity to the user's situation and recognition of plausible unstated unknowns can coexist. - Do not optimise for the number of uncertainty items. Optimise for meaningful, distinct, well-grounded uncertainty recognition. Richer uncertainty recognition must not be confused with engine-owned prioritisation or investigation steering. The value of additional uncertainties is in user understanding, not in expanding the investigation scope autonomously. --- ## G. Programme Conclusion The local model (`qwen-claude:latest`) demonstrated richer uncertainty recognition when invoked via a minimal direct prompt independently than when Confidence Engine mediated the invocation through its reconstruction pipeline (Conditions D/E). Confidence Engine did not reliably reproduce that behaviour across any tested condition or prompt variant. The precise limiting mechanism remains unresolved — whether it is model capability, prompt effectiveness, orchestration constraints, or their interaction. The original 10-iteration investigation programme is closed. --- ## Referenced Evidence Files | Artefact | Path | Status | |----------|------|--------| | Raw minimal prompt (experiment3) | `.evidence-temp/experiment3/raw-prompt.txt` (858B, 13 lines) | Complete | | Direct-model run 1 | `.evidence-temp/experiment3/raw-run-1-output.txt` (3520B, 29 lines) | Complete | | Direct-model run 2 | `.evidence-temp/experiment3/raw-run-2-output.txt` (1855B, 21 lines) | Complete | | Scenario SHA-256 | `.evidence-temp/iteration10/scenario-verification.txt` | `dc9a600f...fdcf06` | | Condition D / no21 evidence | `.evidence-temp/iteration5/D-no21-freeform.txt`, `D-no21.json` | Complete | | Condition D original (iter 5) | `.evidence-temp/iteration5/D-original-freeform.txt`, `D-original.json` | Complete | | Rule-23 removal prompt | `.evidence-temp/iteration6/prompt-variants/reconstruct-v0.5-d-deconstruct-no23.md` | Complete | | Run A (original D, iter 6) | `.evidence-temp/iteration6/run-A-original-D.json` | Complete | | Run B (no rule 23, iter 6) | `.evidence-temp/iteration6/run-B-no-rule-23.json` | Complete | | Iteration 8 invalid run | `.evidence-temp/iteration8/run-A-original-d.json`, `run-B-discovery-permission.json` | Invalid | | Discovery permission prompt | `.evidence-temp/iteration8/reconstruct-v0.5-d-discovery-permission.md` | Complete | | Iteration 8 recovery log | `.evidence-temp/iteration8-recovery/experiment_log.txt`, `b-comparison.txt` | Complete | | Iteration 10 final report | `.evidence-temp/iteration10/final-report.txt` | Complete (BLOCKED) | | Iteration 10 architecture finding | `.evidence-temp/iteration10/architecture-finding.txt` | Complete | | Headless-semantic series | `.evidence-temp/headless-semantic/` (28 files) | Extensive, preserved |