8.0 KiB
Confidence Engine — Methodology Checkpoint: Return to Origin
Date: 18 August 2026
Status: CHECKPOINT — development paused at this methodology question
Branch: feature/decision-closure-ownership-v0.47
Build → Break → Learn → STOP. The recent selector-led work was a valuable implementation hypothesis. The experiments exposed its boundaries. Development is deliberately pausing before optimising the wrong assumption further.
Original Aim
Capture a repeatable way of thinking: take apart complicated situations, separate observation from interpretation, keep assumptions visible, admit what is not yet known, and keep moving until the next useful action becomes clear.
The core commercial ambition is to create transferable intellectual property for RDB Solutions — a methodology that helps people investigate, challenge and understand questions or decisions without depending on Rob personally being present.
The reasoning is the asset; the application is one delivery mechanism. Confidence is earned through understood evidence and manageable next actions, not through confident-sounding answers. Experiments beat opinions.
Implementation Hypothesis
The LLM reconstructs a messy situation into a SituationGraph. The graph holds knowns and unresolved uncertainties. Deterministic reasoning selects an active unknown. A graph-backed question is formulated. The user answers it. The graph updates. The next question is selected.
This was a reasonable implementation hypothesis — it made the method concrete enough to test.
What Was Built Successfully
- LLM reconstruction of messy statements into useful structure
- Explicit representation of observations, assumptions, unknowns and relationships
- Graph persistence and state mutation as understanding changes
- Decomposition of broad uncertainty into smaller investigable questions
- Question formulation, answerability checks and reasoning-pattern safeguards
- Ownership and continuation invariants that prevent silent target drift
- Captured live fixtures, browser journeys and deterministic regressions
- A disciplined experimental method: live observation → capture exact evidence → isolate first divergence → regression → diagnosis → implementation → focused verification → checkpoint
What the Experiments Exposed
The single-next-question mechanism had accumulated too much product authority.
-
Question formulation quality and investigation importance are different things. A selected uncertainty could remain the best thing to investigate even when its question wording was rejected — leading to the ownership fix that preserves the investigation target rather than silently transferring it.
-
Deterministic vocabulary weighting makes semantic priority depend on phrasing. Two materially equivalent phrasings received very different scores because one triggered fixed vocabulary rules and the other did not. Wording alone changed the selected investigation target.
-
Real users use typos, slang, abbreviations, jargon, shorthand and personal language; LLM-generated graph labels also vary between equivalent phrasings. Expanding a keyword dictionary would improve coverage but preserve a finite and brittle semantic boundary.
-
The experiments exposed that we had asked the wrong product question. Development centred on "What should the Engine ask next?" when the more useful question is "What useful open questions has the investigation exposed, and how should the person work with them?"
BUILD → BREAK → LEARN → STOP
- BUILD: Graph reconstruction, decomposition, patterns, question formulation, ownership and validation were implemented.
- BREAK: Real browser journeys and deterministic regressions exposed stale ownership, question-rejection and selection-boundary defects.
- LEARN: Question wording is not target validity; fixed vocabulary scoring is not paraphrase-invariant; next-question selection had accumulated too much authority.
- STOP: Further selector optimisation paused — before turning the response into an ever-larger dictionary, weight tuning exercise or semantic-ranking subsystem.
Return to Workspace / Notebook / Workshop
The graph should primarily organise and remember the investigation rather than force one linear route. Multiple open questions can coexist. The user can decide where they can make progress while the Engine continues to guide, challenge, connect and remember.
- Surface the open questions the LLM has already derived
- Let the user answer what they know now
- Let the user choose a question that matters most to them
- Allow questions to be deferred when evidence requires research, another person, measurement, calculation or time
- Allow the investigation to persist across minutes, days or weeks
- Keep the Engine able to challenge avoidance or highlight an unresolved issue that still materially blocks confidence
User-Directed Open Investigations
The user is not merely a respondent supplying missing fields to an automated reasoning pipeline. The user is the investigator. Choosing what to work on is itself part of the reasoning process.
Multiple investigation items can be advanced independently or in parallel. Different items can be progressed independently, and the user should be able to return when new evidence becomes available.
Asynchronous / Deferred Investigation
Real investigations do not fit neatly into one chat session. Some answers are immediate; others require documents, colleagues, calculations, measurements, research or waiting for events.
Open questions should be treated as persistent investigation items rather than failed turns. States include: Open, Answerable now, Needs investigation, Waiting for information, Partly answered, Resolved, No longer material.
What Remains Technically Valuable
- SituationGraph and structured case state
- LLM reconstruction/decomposition
- Known / assumed / unknown / evidence distinctions
- Relationships and dependencies
- Resolution and supersession state
- Question decomposition and answerability concepts
- Ownership/current-focus semantics where they represent the thread being worked on
- Validation and graph-integrity safeguards
- Persistent history and captured provenance
- Live semantic test discipline and deterministic regression workflow
What Is Paused
Further work to perfect a compulsory single-next-question selector is paused. This includes both continued keyword/dictionary optimisation and immediate replacement with an invisible semantic ranking mechanism.
No conclusion has yet been made that selection or recommendation has no role. The Engine may still recommend, challenge or identify an issue that materially blocks confidence. What is paused is the assumption that recommendation must equal compulsory routing.
Commercial Continuity
The valuable asset is not a specific selector, prompt or chat interface. Those can be replaced. The defensible value is the repeatable Confidence Engine method for turning uncertainty into an understandable investigation and helping a person build justified confidence.
That matters directly to RDB Solutions: the aim is to create products and methods that generate value without relying on Rob personally delivering every piece of reasoning. A software workspace, facilitator-led workshop, workbook, training programme or other delivery format can all express the same underlying method.
Deliberately Unanswered Next Question
Given the useful investigation structure the Engine can already derive, how should that structure be surfaced so a person can see, choose, defer, investigate and return to open questions while the Engine continues to guide and challenge their thinking toward justified confidence?
The next phase should begin from this methodology question, not from a preselected technical solution.
Source
Full methodology context: docs/Confidence_Engine_Return_to_Origin_Methodology_Context_2026-08-18.md
This is a concise repository-facing checkpoint derived from the above source, preserving all sections.