# Uncertainty Information Flow Audit — Programme Handoff
**Working title:** Confidence Engine — Uncertainty Information Flow Audit
**Date prepared:** 2026-10-14
**Prepared for:** Next independent investigation (following the completed 10-iteration uncertainty recognition quality investigation)
---
## Primary Question
"At which point, if any, does Confidence Engine constrain, transform, discard or fail to preserve meaningful uncertainty recognition?"
## Context
The previous 10-iteration programme demonstrated a quality gap between minimal direct-model uncertainty recognition and Confidence Engine-mediated uncertainty recognition. The precise limiting mechanism remains unresolved (see `uncertainty-recognition-investigation-report.md` sections D–G). This new programme is designed to map the actual production information flow and identify specific intervention points without presuming failure locations.
These boundaries define investigation scope, not expectations of where problems will be found.
## Recommended Approach
- Up to 10 iterations
- One bounded investigative question per iteration
- ChatGPT controls programme progression between iterations
- Claude Code executes one task per invocation and reports evidence
- Each `/rdbHandoff` invocation is a fresh session — work from repository evidence and this handoff
- Read-only investigation before any interventions
- Controlled experiments only when justified by evidence from prior iterations
- Preserve raw evidence and request provenance for every iteration
- No speculative product changes
## First Iteration Guidance
The first iteration should establish the actual production information flow and available capture boundaries:
1. Map the complete code path from scenario file input to model request construction
2. Document what data is accessible at each stage of the pipeline (raw scenario, post-parsing, pre-template, post-template, pre-request, post-response, post-parsing)
3. Identify any existing hooks, logs, or capture mechanisms for raw requests/responses
4. Determine which stages can produce evidence without source modifications
The output should be a factual map of available capture points and data accessibility, not an analysis of what is "wrong."
## Existing Evidence to Consider
-`docs/uncertainty-recognition-investigation-report.md` — Full report from the previous programme
-`.evidence-temp/iteration10/architecture-finding.txt` — Definitive CE prompt-path analysis showing template applicability at every invocation point
-`.evidence-temp/headless-semantic/` — Extensive supporting evidence from HS series experiments
## What This Programme Is Not
- A redesign of Confidence Engine
- A directive to change the model or provider configuration
- An assumption that any specific boundary is broken or deficient
- A continuation of the previous programme's quality-gap investigation (that closed at Iteration 10)
## Start Signal
Do not begin this programme from this document alone. Wait for an explicit `/rdbHandoff` invocation containing the start signal and first iteration directive.
**Status:** CLOSED — 10-iteration programme completed and checkpointed
---
## A. Investigation Objective
This investigation examined why the configured local LLM (`qwen-claude:latest` via Ollama) produced less useful uncertainty recognition through Confidence Engine's reconstruction pipeline than through a minimal direct prompt.
The canonical scenario used was `RFG-03-independent-decisions-input.json`, concerning three independent decisions (replacing internal reporting software, recruiting an additional customer-support employee, refurbishing a meeting room). The user's goal is to understand what remains uncertain before deciding what to do.
**Observed phenomenon:** A minimal direct-model prompt produced two outputs containing approximately 21 and 15 uncertainties with meaningful domain-specific unknowns. Confidence Engine reconstruction under Conditions D/E generally produced three generic uncertainties that largely restated the user's stated decisions without adding new insights.
---
## B. Experimental Programme (Iterations 1–10)
### Iteration 3 — Baseline Direct-Model Prompt
**Status:** Complete and reproducible
**Evidence:**`.evidence-temp/experiment3/`
**Prompt:** Minimal raw prompt with no rule-set framing or forced JSON output.
**Result:** Two model outputs: raw-run-1 (29 lines, ~15+ domain-specific uncertainties), raw-run-2 (21 lines, similar richness). Scenario SHA-256: `dc9a600f9fb1e7189dd56086a07f357893396a5b0782f4781c95d4aa29fdcf06`.
### Iteration 5 — Condition D Rule-21 Removal
**Status:** Complete
**Evidence:**`.evidence-temp/iteration5/`
**Treatment:** Removed rule 21 from the deconstruction prompt.
**Result:** Four CE uncertainties including a cross-domain "worthwhile" framework question. No material improvement over control in domain-specific discovery. Evidence: `D-no21-freeform.txt` (103 lines), `D-no21.json`, `D-original-freeform.txt`, `D-original.json`.
**Invalidity cause:** Scenario filename was treated as scenario text instead of JSON content.
**Recovery result:** 4 uncertainty items with modest structural enrichment (expectedStates, differences, tighter relationships) but no new domain-specific unknowns.
**Iteration 10:** BLOCKED before model execution. The existing experimental helper (`scripts/start-case-experiment-helper.cjs`) has no mechanism to pass the exact minimal raw prompt through the CE execution path without source modifications. Every invocation applies a structured prompt template. See `.evidence-temp/iteration10/final-report.txt` and `architecture-finding.txt` for the definitive architecture analysis.
### Headless-Semantic Evidence (HS-01 through HS-03C)
These findings are directly supported by the evidence:
1.**Direct-model prompt quality gap:** Minimal direct prompts produced outputs with approximately 21 and 15 domain-specific uncertainties across three decisions, while CE reconstruction under Conditions D/E generally produced three generic restatements of the user's own decisions.
2.**Rule removal had no material effect:** Removing Rule 21 (Iteration 5) and Rule 23 (Iteration 6) did not materially improve uncertainty recognition quality.
3.**Discovery permission instruction produced modest structural enrichment:** The Iteration 8 recovery (adding explicit permission to recognise plausible unstated uncertainties) added four items including a cross-domain evaluation framework question, but no new domain-specific unknowns were discovered beyond what the user stated.
4.**CE execution path is structurally incapable of raw-prompt pass-through:** Every invocation applies a template-based prompt construction that adds rule sets and forced-JSON constraints absent from the direct-model baseline. This was definitively established in Iteration 10's architecture-finding evidence.
5.**Question-free reconstruction does not prevent structural compression:** CE-ULI-10 demonstrated that removing the question-generation requirement still produced structural compression of uncertainties into single compound descriptions for the tested scenario shape.
6.**Explicit domain-separation instructions are ignored:** CE-ULI-11 showed identical compression under both condition B and C despite explicit instruction not to combine domains.
---
## D. Unresolved Hypotheses
**H1 — Model capability:** The model may be capable of richer uncertainty recognition but its output quality is constrained by the surrounding apparatus (prompt structure, output format constraints). *Untested.* Not equivalent to H2 or H3.
**H2 — Prompt effectiveness:** The template-based prompt construction (rule sets, forced JSON, deconstruction framing) suppresses the model's ability to produce rich uncertainty recognition relative to a minimal direct prompt. *Supported by evidence of quality gap but not proven as causal.* No controlled experiment eliminated the structural difference while preserving all other conditions.
**H3 — Confidence Engine orchestration:** Something in the CE pipeline (prompt construction, parsing, validation, graph representation, or presentation) constrains, transforms, or discards meaningful uncertainty recognition between model output and user-facing display. *Supported by evidence but not isolated.* Iteration 10 established that the CE execution path always applies templates, making controlled elimination of specific pipeline stages infeasible without source modifications.
**Important:** The precise cause of the quality difference remains unresolved. The evidence demonstrates a quality gap and several plausible explanations, not definitive causation. It is not established that prompt construction, JSON formatting, or orchestration individually or collectively suppress uncertainty recognition — only that these are all uncontrolled confounders in the comparison between CE-executed requests and direct model calls.
---
## E. Experimental Apparatus Limitations
1.**Template-based prompt construction:** All CE invocation paths apply a template (v0.5, b/c/d/e variants). No raw-prompt pass-through mode exists in `lib/analysis.js` or `start-case-experiment-helper.cjs`.
2.**No raw-prompt pass-through in the helper:** Condition selector loads hardcoded template filenames. The `--experiment-instruction` option only appends — it cannot replace or suppress base structure.
3.**`--file` input-loading requirement:** Scenarios must be loaded from JSON files. Line-input and other loading modes are not supported by the helper.
4.**`--reconstruction-only` requirement:** To bypass the graph pipeline, `--reconstruction-only` is required. Its absence routes through the full production path.
5.**Generation-setting comparability limitations:** Direct-model baseline runs (experiment3) used different generation settings (no forced schema) than CE execution (forced JSON schemas). Temperature, top_p, and other parameters are not controlled or logged for direct runs.
6.**Missing evidence:** Iteration 9's b-variant evidence is incomplete (empty subdirectory). Evidence for some earlier iterations beyond condition outputs alone is unavailable.
7.**Single-model limitation:** All experiments used `qwen-claude:latest` only. Results may not generalise to other models.
---
## F. Product Principles
The following principles must govern any future work on uncertainty recognition:
- The user owns investigation direction.
- The engine facilitates rather than steers.
- Recognising an uncertainty is not inventing a fact.
- Recognising an uncertainty is not selecting it.
- The engine must not automatically decide what the user should investigate next.
- Fidelity to the user's situation and recognition of plausible unstated unknowns can coexist.
- Do not optimise for the number of uncertainty items. Optimise for meaningful, distinct, well-grounded uncertainty recognition.
Richer uncertainty recognition must not be confused with engine-owned prioritisation or investigation steering. The value of additional uncertainties is in user understanding, not in expanding the investigation scope autonomously.
---
## G. Programme Conclusion
The local model (`qwen-claude:latest`) demonstrated richer uncertainty recognition when invoked via a minimal direct prompt independently than when Confidence Engine mediated the invocation through its reconstruction pipeline (Conditions D/E). Confidence Engine did not reliably reproduce that behaviour across any tested condition or prompt variant.
The precise limiting mechanism remains unresolved — whether it is model capability, prompt effectiveness, orchestration constraints, or their interaction. The original 10-iteration investigation programme is closed.
thrownewError(`Invalid treatment: ${treatment}. Must be one of: e-investigability-first`);
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
