robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
8530d954a0b9389ea48bacdbf8b3d3dda504ebdc
confidence-engine/docs/uncertainty-recognition-investigation-report.md
T

12 KiB
Raw Blame History

Uncertainty Recognition Quality Investigation — Report

Date: 2026-10-14 Branch: experiment/uncertainty-led-investigation Starting HEAD: e595ee6 feat(experiments): integrate Condition D/E apparatus Status: CLOSED — 10-iteration programme completed and checkpointed

A. Investigation Objective

This investigation examined why the configured local LLM (qwen-claude:latest via Ollama) produced less useful uncertainty recognition through Confidence Engine's reconstruction pipeline than through a minimal direct prompt.

The canonical scenario used was RFG-03-independent-decisions-input.json, concerning three independent decisions (replacing internal reporting software, recruiting an additional customer-support employee, refurbishing a meeting room). The user's goal is to understand what remains uncertain before deciding what to do.

Observed phenomenon: A minimal direct-model prompt produced two outputs containing approximately 21 and 15 uncertainties with meaningful domain-specific unknowns. Confidence Engine reconstruction under Conditions D/E generally produced three generic uncertainties that largely restated the user's stated decisions without adding new insights.

B. Experimental Programme (Iterations 1–10)

Iteration 3 — Baseline Direct-Model Prompt

Status: Complete and reproducible Evidence: .evidence-temp/experiment3/ Prompt: Minimal raw prompt with no rule-set framing or forced JSON output. Result: Two model outputs: raw-run-1 (29 lines, ~15+ domain-specific uncertainties), raw-run-2 (21 lines, similar richness). Scenario SHA-256: dc9a600f9fb1e7189dd56086a07f357893396a5b0782f4781c95d4aa29fdcf06.

Iteration 5 — Condition D Rule-21 Removal

Status: Complete Evidence: .evidence-temp/iteration5/ Treatment: Removed rule 21 from the deconstruction prompt. Result: Four CE uncertainties including a cross-domain "worthwhile" framework question. No material improvement over control in domain-specific discovery. Evidence: D-no21-freeform.txt (103 lines), D-no21.json, D-original-freeform.txt, D-original.json.

Iteration 6 — Condition D Rule-23 Removal + Prompt Variant

Status: Complete Evidence: .evidence-temp/iteration6/ Treatment: Removed rule 23; tested with new prompt variant (reconstruct-v0.5-d-deconstruct-no23.md). Result: No material improvement. Both run-A-original-D.json and run-B-no-rule-23.json validated as freeform with zero graph nodes.

Iteration 8 — Invalid Run + Recovery

Status: Validated invalidity; recovery run complete Evidence: .evidence-temp/iteration8/ (invalid), .evidence-temp/iteration8-recovery/ (valid) Invalidity cause: Scenario filename was treated as scenario text instead of JSON content. Recovery result: 4 uncertainty items with modest structural enrichment (expectedStates, differences, tighter relationships) but no new domain-specific unknowns.

Iterations 9–10 — Follow-up and Block

Iteration 9: Partial evidence; focused-investigation dependency analysis. .evidence-temp/iteration9/ contains a b-variant/ subdirectory (incomplete). Iteration 10: BLOCKED before model execution. The existing experimental helper (scripts/start-case-experiment-helper.cjs) has no mechanism to pass the exact minimal raw prompt through the CE execution path without source modifications. Every invocation applies a structured prompt template. See .evidence-temp/iteration10/final-report.txt and architecture-finding.txt for the definitive architecture analysis.

Headless-Semantic Evidence (HS-01 through HS-03C)

Status: Extensive supporting evidence preserved Evidence: .evidence-temp/headless-semantic/ (28 files, 2788 lines total) Covers: Question-free reconstruction comparison (CE-ULI-09), domain-preservation instruction tests (CE-ULI-10, CE-ULI-11), and focused-investigation evidence persistence testing.

C. Established Findings

These findings are directly supported by the evidence:

  1. Direct-model prompt quality gap: Minimal direct prompts produced outputs with approximately 21 and 15 domain-specific uncertainties across three decisions, while CE reconstruction under Conditions D/E generally produced three generic restatements of the user's own decisions.

  2. Rule removal had no material effect: Removing Rule 21 (Iteration 5) and Rule 23 (Iteration 6) did not materially improve uncertainty recognition quality.

  3. Discovery permission instruction produced modest structural enrichment: The Iteration 8 recovery (adding explicit permission to recognise plausible unstated uncertainties) added four items including a cross-domain evaluation framework question, but no new domain-specific unknowns were discovered beyond what the user stated.

  4. CE execution path is structurally incapable of raw-prompt pass-through: Every invocation applies a template-based prompt construction that adds rule sets and forced-JSON constraints absent from the direct-model baseline. This was definitively established in Iteration 10's architecture-finding evidence.

  5. Question-free reconstruction does not prevent structural compression: CE-ULI-10 demonstrated that removing the question-generation requirement still produced structural compression of uncertainties into single compound descriptions for the tested scenario shape.

  6. Explicit domain-separation instructions are ignored: CE-ULI-11 showed identical compression under both condition B and C despite explicit instruction not to combine domains.

D. Unresolved Hypotheses

H1 — Model capability: The model may be capable of richer uncertainty recognition but its output quality is constrained by the surrounding apparatus (prompt structure, output format constraints). Untested. Not equivalent to H2 or H3.

H2 — Prompt effectiveness: The template-based prompt construction (rule sets, forced JSON, deconstruction framing) suppresses the model's ability to produce rich uncertainty recognition relative to a minimal direct prompt. Supported by evidence of quality gap but not proven as causal. No controlled experiment eliminated the structural difference while preserving all other conditions.

H3 — Confidence Engine orchestration: Something in the CE pipeline (prompt construction, parsing, validation, graph representation, or presentation) constrains, transforms, or discards meaningful uncertainty recognition between model output and user-facing display. Supported by evidence but not isolated. Iteration 10 established that the CE execution path always applies templates, making controlled elimination of specific pipeline stages infeasible without source modifications.

Important: The precise cause of the quality difference remains unresolved. The evidence demonstrates a quality gap and several plausible explanations, not definitive causation. It is not established that prompt construction, JSON formatting, or orchestration individually or collectively suppress uncertainty recognition — only that these are all uncontrolled confounders in the comparison between CE-executed requests and direct model calls.

E. Experimental Apparatus Limitations

  1. Template-based prompt construction: All CE invocation paths apply a template (v0.5, b/c/d/e variants). No raw-prompt pass-through mode exists in lib/analysis.js or start-case-experiment-helper.cjs.

  2. No raw-prompt pass-through in the helper: Condition selector loads hardcoded template filenames. The --experiment-instruction option only appends — it cannot replace or suppress base structure.

  3. --file input-loading requirement: Scenarios must be loaded from JSON files. Line-input and other loading modes are not supported by the helper.

  4. --reconstruction-only requirement: To bypass the graph pipeline, --reconstruction-only is required. Its absence routes through the full production path.

  5. Generation-setting comparability limitations: Direct-model baseline runs (experiment3) used different generation settings (no forced schema) than CE execution (forced JSON schemas). Temperature, top_p, and other parameters are not controlled or logged for direct runs.

  6. Missing evidence: Iteration 9's b-variant evidence is incomplete (empty subdirectory). Evidence for some earlier iterations beyond condition outputs alone is unavailable.

  7. Single-model limitation: All experiments used qwen-claude:latest only. Results may not generalise to other models.

F. Product Principles

The following principles must govern any future work on uncertainty recognition:

  • The user owns investigation direction.
  • The engine facilitates rather than steers.
  • Recognising an uncertainty is not inventing a fact.
  • Recognising an uncertainty is not selecting it.
  • The engine must not automatically decide what the user should investigate next.
  • Fidelity to the user's situation and recognition of plausible unstated unknowns can coexist.
  • Do not optimise for the number of uncertainty items. Optimise for meaningful, distinct, well-grounded uncertainty recognition.

Richer uncertainty recognition must not be confused with engine-owned prioritisation or investigation steering. The value of additional uncertainties is in user understanding, not in expanding the investigation scope autonomously.

G. Programme Conclusion

The local model (qwen-claude:latest) demonstrated richer uncertainty recognition when invoked via a minimal direct prompt independently than when Confidence Engine mediated the invocation through its reconstruction pipeline (Conditions D/E). Confidence Engine did not reliably reproduce that behaviour across any tested condition or prompt variant.

The precise limiting mechanism remains unresolved — whether it is model capability, prompt effectiveness, orchestration constraints, or their interaction. The original 10-iteration investigation programme is closed.

Referenced Evidence Files

Artefact Path Status
Raw minimal prompt (experiment3) .evidence-temp/experiment3/raw-prompt.txt (858B, 13 lines) Complete
Direct-model run 1 .evidence-temp/experiment3/raw-run-1-output.txt (3520B, 29 lines) Complete
Direct-model run 2 .evidence-temp/experiment3/raw-run-2-output.txt (1855B, 21 lines) Complete
Scenario SHA-256 .evidence-temp/iteration10/scenario-verification.txt dc9a600f...fdcf06
Condition D / no21 evidence .evidence-temp/iteration5/D-no21-freeform.txt, D-no21.json Complete
Condition D original (iter 5) .evidence-temp/iteration5/D-original-freeform.txt, D-original.json Complete
Rule-23 removal prompt .evidence-temp/iteration6/prompt-variants/reconstruct-v0.5-d-deconstruct-no23.md Complete
Run A (original D, iter 6) .evidence-temp/iteration6/run-A-original-D.json Complete
Run B (no rule 23, iter 6) .evidence-temp/iteration6/run-B-no-rule-23.json Complete
Iteration 8 invalid run .evidence-temp/iteration8/run-A-original-d.json, run-B-discovery-permission.json Invalid
Discovery permission prompt .evidence-temp/iteration8/reconstruct-v0.5-d-discovery-permission.md Complete
Iteration 8 recovery log .evidence-temp/iteration8-recovery/experiment_log.txt, b-comparison.txt Complete
Iteration 10 final report .evidence-temp/iteration10/final-report.txt Complete (BLOCKED)
Iteration 10 architecture finding .evidence-temp/iteration10/architecture-finding.txt Complete
Headless-semantic series .evidence-temp/headless-semantic/ (28 files) Extensive, preserved
Reference in New Issue View Git Blame Copy Permalink