docs(experiments): checkpoint uncertainty recognition investigation
This commit is contained in:
@@ -0,0 +1,66 @@
|
||||
# Uncertainty Information Flow Audit — Programme Handoff
|
||||
|
||||
**Working title:** Confidence Engine — Uncertainty Information Flow Audit
|
||||
**Date prepared:** 2026-10-14
|
||||
**Prepared for:** Next independent investigation (following the completed 10-iteration uncertainty recognition quality investigation)
|
||||
|
||||
---
|
||||
|
||||
## Primary Question
|
||||
|
||||
"At which point, if any, does Confidence Engine constrain, transform, discard or fail to preserve meaningful uncertainty recognition?"
|
||||
|
||||
## Context
|
||||
|
||||
The previous 10-iteration programme demonstrated a quality gap between minimal direct-model uncertainty recognition and Confidence Engine-mediated uncertainty recognition. The precise limiting mechanism remains unresolved (see `uncertainty-recognition-investigation-report.md` sections D–G). This new programme is designed to map the actual production information flow and identify specific intervention points without presuming failure locations.
|
||||
|
||||
## Investigation Boundaries (Not Presumed Failure Points)
|
||||
|
||||
1. User input and scenario representation
|
||||
2. Prompt construction
|
||||
3. Raw LLM request and response
|
||||
4. Parsing and validation
|
||||
5. Internal graph representation
|
||||
6. User-facing presentation
|
||||
|
||||
These boundaries define investigation scope, not expectations of where problems will be found.
|
||||
|
||||
## Recommended Approach
|
||||
|
||||
- Up to 10 iterations
|
||||
- One bounded investigative question per iteration
|
||||
- ChatGPT controls programme progression between iterations
|
||||
- Claude Code executes one task per invocation and reports evidence
|
||||
- Each `/rdbHandoff` invocation is a fresh session — work from repository evidence and this handoff
|
||||
- Read-only investigation before any interventions
|
||||
- Controlled experiments only when justified by evidence from prior iterations
|
||||
- Preserve raw evidence and request provenance for every iteration
|
||||
- No speculative product changes
|
||||
|
||||
## First Iteration Guidance
|
||||
|
||||
The first iteration should establish the actual production information flow and available capture boundaries:
|
||||
|
||||
1. Map the complete code path from scenario file input to model request construction
|
||||
2. Document what data is accessible at each stage of the pipeline (raw scenario, post-parsing, pre-template, post-template, pre-request, post-response, post-parsing)
|
||||
3. Identify any existing hooks, logs, or capture mechanisms for raw requests/responses
|
||||
4. Determine which stages can produce evidence without source modifications
|
||||
|
||||
The output should be a factual map of available capture points and data accessibility, not an analysis of what is "wrong."
|
||||
|
||||
## Existing Evidence to Consider
|
||||
|
||||
- `docs/uncertainty-recognition-investigation-report.md` — Full report from the previous programme
|
||||
- `.evidence-temp/iteration10/architecture-finding.txt` — Definitive CE prompt-path analysis showing template applicability at every invocation point
|
||||
- `.evidence-temp/headless-semantic/` — Extensive supporting evidence from HS series experiments
|
||||
|
||||
## What This Programme Is Not
|
||||
|
||||
- A redesign of Confidence Engine
|
||||
- A directive to change the model or provider configuration
|
||||
- An assumption that any specific boundary is broken or deficient
|
||||
- A continuation of the previous programme's quality-gap investigation (that closed at Iteration 10)
|
||||
|
||||
## Start Signal
|
||||
|
||||
Do not begin this programme from this document alone. Wait for an explicit `/rdbHandoff` invocation containing the start signal and first iteration directive.
|
||||
Reference in New Issue
Block a user