docs/confidence-engine-decomposition-and-atomic-reasoning-specification.md

@@ -0,0 +1,646 @@
+# Confidence Engine — Decomposition and Atomic Reasoning Specification
+
+**Status:** Working design specification  
+**Purpose:** Source of truth for future reasoning-engine implementation and review  
+**Audience:** Product owner, reasoning-engine developers, coding agents, testers and future methodology authors
+
+---
+
+## 1. Plain-English meaning
+
+Decomposition means breaking one difficult uncertainty into smaller uncertainties until each one can be answered directly.
+
+A large question such as:
+
+> Should we buy this business?
+
+cannot usually be answered honestly in one step. It may need to become:
+
+- Is the business profitable?
+- Are the accounts reliable?
+- Is the customer base stable?
+- Can the purchase be financed safely?
+
+If one of those questions is still too broad, it is broken down again.
+
+The engine continues until each remaining uncertainty is small enough for one focused investigation, one piece of evidence, one calculation, one observation or one direct answer to resolve it.
+
+> **Decomposition is not about breaking problems apart for its own sake. It is about shrinking uncertainty until it becomes answerable.**
+
+This reflects the wider Confidence Engine principle of breaking complicated situations into small, granular, understandable parts.
+
+---
+
+## 2. Purpose
+
+The purpose of decomposition is to prevent the engine from asking questions that are too broad, compound, vague or difficult to answer reliably.
+
+Decomposition should help the engine:
+
+1. turn large uncertainties into answerable units;
+2. preserve the relationship between each small question and the larger situation;
+3. avoid asking several things at once;
+4. reveal what evidence is actually needed;
+5. stop once further subdivision would add no useful clarity;
+6. support justified progression from uncertainty towards confidence.
+
+Decomposition is therefore a reasoning operation, not a formatting step.
+
+---
+
+## 3. Foundational principles
+
+### 3.1 One node, one uncertainty
+
+Every unknown node should represent exactly one independently investigable uncertainty.
+
+A node is valid when a person can understand what single thing is uncertain and what kind of evidence could settle it.
+
+### 3.2 One useful thing at a time
+
+The engine should ask one question whose answer can make one meaningful change to the reasoning state.
+
+### 3.3 Context belongs to the highest level where it is true
+
+Information should exist at the highest level where it first becomes true, and should not be repeated lower in the graph unless it is independently true there as well.
+
+Examples of parent-level context include:
+
+- commercial justification;
+- the overall decision being considered;
+- the user's wider objective;
+- domain framing;
+- branch-wide constraints;
+- the fact that several conditions must be considered together.
+
+Children operate within that context. They should not restate it as part of their own uncertainty.
+
+### 3.4 Every transformation must improve the reasoning
+
+A transformation is justified only when it does at least one of the following:
+
+- reduces uncertainty;
+- increases justified confidence;
+- makes an uncertainty more answerable;
+- exposes a contradiction that must be resolved;
+- separates distinct questions that were previously entangled.
+
+If decomposition creates more words but no clearer investigation path, it has failed.
+
+### 3.5 No question is preferable to an unjustified question
+
+The engine must not force progression by selecting a poor child, ignoring incompatibility or inventing an answerable-looking question.
+
+A temporary stop is better than a misleading question.
+
+### 3.6 The graph stores reasoning; the conversation exposes reasoning
+
+The graph may contain parent context, child uncertainties, dependencies and resolution state. The user-facing question should expose only the smallest justified next step.
+
+---
+
+## 4. Composite and atomic uncertainties
+
+### 4.1 Composite uncertainty
+
+A node is composite when no single investigation can resolve its uncertainty.
+
+It normally contains two or more distinct dimensions that can change independently.
+
+A node is likely composite when:
+
+- one answer can resolve part of it while leaving another part unresolved;
+- it contains separable conditions;
+- it requires several different kinds of evidence;
+- an investigator would naturally ask more than one focused question;
+- it combines a decision, criterion, explanation or relationship into one statement.
+
+A node is not composite merely because it is important, difficult or domain-specific.
+
+### 4.2 Atomic uncertainty
+
+A node is atomic when one focused investigation can settle the uncertainty it owns.
+
+An atomic node:
+
+- concerns one variable, condition or relationship;
+- has one clear semantic identity;
+- can be investigated without answering sibling questions first;
+- cannot be divided further without producing paraphrases, duplicates or trivial fragments;
+- has a recognisable resolution condition.
+
+A useful test is:
+
+> Could one focused piece of evidence or one direct answer settle this specific doubt?
+
+If yes, it is probably atomic.
+
+### 4.3 Atomic does not mean simple in subject matter
+
+An atomic question may still require specialist work.
+
+For example:
+
+> Does the unit economics produce a positive contribution margin at the projected volume?
+
+is domain-specific and may require financial modelling, but it still investigates one thing.
+
+---
+
+## 5. Information ownership
+
+### 5.1 What the parent owns
+
+The parent owns the context that makes the group of child questions meaningful.
+
+This may include:
+
+- the overall objective;
+- the decision under consideration;
+- the evaluative frame, such as commercial justification or technical feasibility;
+- branch-wide constraints;
+- the logical relationship between children;
+- the rule for aggregating child outcomes;
+- the investigation scope;
+- sibling coverage and completion state.
+
+### 5.2 What each child owns
+
+Each child owns:
+
+- one independently answerable uncertainty;
+- its own label;
+- its own scope;
+- its own answerability condition;
+- its own evidence;
+- its own resolution status;
+- its own semantic identity.
+
+### 5.3 What children must not inherit
+
+Children must not inherit parent material merely because it appeared in the parent's wording.
+
+Children should not inherit:
+
+- evaluative framing such as *commercially justified*, *viable* or *feasible*;
+- the parent's conjunction or compound structure;
+- the whole objective;
+- sibling information;
+- branch-wide constraints written as if they were child conditions;
+- wording that causes the child to become a disguised copy of the parent.
+
+### 5.4 Ownership rule
+
+> **A child should describe only the uncertainty it owns.**
+
+The parent explains why the child matters. The child states what must be investigated.
+
+---
+
+## 6. Decomposition process
+
+### Step 1 — Identify why the current uncertainty is not directly answerable
+
+Determine which independent dimensions prevent one focused investigation from resolving the node.
+
+### Step 2 — Identify the smallest distinct uncertainties
+
+Separate those dimensions into candidate children.
+
+Each candidate should correspond to one investigation path.
+
+### Step 3 — Remove inherited parent framing
+
+Rewrite each candidate so it describes only its own uncertainty.
+
+### Step 4 — Test independence
+
+Check whether each child can be investigated without needing a sibling answer.
+
+If a child depends on another child, they may not be true siblings. The dependency may require a different graph relationship.
+
+### Step 5 — Test narrowing
+
+Each child must be more specific than the parent.
+
+A child that could replace the parent without loss of meaning is not a decomposition.
+
+### Step 6 — Test uniqueness
+
+No two children should ask the same underlying question in different words.
+
+### Step 7 — Test coverage
+
+Together, the children must cover the uncertainty represented by the parent.
+
+### Step 8 — Test answerability
+
+Each child must be small enough to support one clear user-facing question or one clear evidence-gathering action.
+
+### Step 9 — Apply or reject
+
+Apply the decomposition only if it improves the reasoning state.
+
+Otherwise retain the parent as unresolved and record why decomposition failed.
+
+---
+
+## 7. Stopping rules
+
+Decomposition should stop when the earliest of the following conditions is met:
+
+1. **One investigation can settle the node.**
+2. **Further children would merely paraphrase the node.**
+3. **Candidate children overlap or duplicate one another.**
+4. **Candidate children are not narrower than the parent.**
+5. **Further subdivision would produce trivial fragments without independent investigative value.**
+6. **The required next step is evidence gathering rather than further decomposition.**
+7. **No valid lossless decomposition can be produced.**
+
+Failure to decompose is not permission to invent children.
+
+---
+
+## 8. Decomposition invariants
+
+Every accepted decomposition must satisfy all of these invariants.
+
+### 8.1 Narrowing
+
+Every child is strictly narrower than the parent.
+
+### 8.2 Atomic direction
+
+Each child moves the graph closer to an independently answerable uncertainty.
+
+A generated child must not immediately trigger the same decomposition merely because it inherited the parent's wording.
+
+### 8.3 Independence
+
+Each sibling can be investigated without requiring another sibling's answer.
+
+### 8.4 Uniqueness
+
+Each child represents a distinct semantic uncertainty.
+
+### 8.5 Coverage
+
+The children collectively cover the parent's uncertainty.
+
+### 8.6 Resolution sufficiency
+
+Resolving all required children provides enough information to derive the parent's status.
+
+If the parent remains unresolved after all children are resolved, the decomposition was incomplete or logically unsound.
+
+### 8.7 Context ownership
+
+Children do not repeat parent-level context unless that context is independently part of the child's uncertainty.
+
+### 8.8 Convergence
+
+Repeated decomposition must move towards atomic questions rather than reproducing the same semantic structure at greater depth.
+
+### 8.9 Traceability
+
+Every child remains linked to the parent so the engine can explain why the question exists.
+
+### 8.10 No invented certainty
+
+Decomposition changes structure, not truth. It must not make the parent or children appear more certain merely because they have been separated.
+
+---
+
+## 9. Quality tests for each child
+
+A candidate child should be rejected when any of the following is true:
+
+- it asks more than one primary thing;
+- it contains a compound clause that creates separable questions;
+- it is not narrower than its parent;
+- it duplicates a sibling;
+- it duplicates an already resolved uncertainty;
+- it depends on a sibling answer;
+- it restates parent framing rather than naming a distinct uncertainty;
+- it has no clear evidence or answer path;
+- its resolution would not affect the parent;
+- its meaning cannot be distinguished from another graph node;
+- it introduces unsupported domain assumptions;
+- it exists only because a template demanded a fixed number of children.
+
+---
+
+## 10. Common failure modes
+
+### 10.1 Framing contamination
+
+A child inherits evaluative or contextual language from the parent and is therefore misclassified as composite again.
+
+Example:
+
+Parent:
+
+> Is this method commercially justified?
+
+Contaminated child:
+
+> Is there commercially justified demand for this method?
+
+The child now contains its own uncertainty plus the parent's commercial evaluation frame.
+
+### 10.2 Recursive restatement
+
+Each decomposition level repeats the same uncertainty using slightly different words.
+
+This creates depth without progress.
+
+### 10.3 Compound children
+
+A child contains two or more independently answerable questions.
+
+Example:
+
+> Can the product be delivered reliably and at an acceptable cost?
+
+### 10.4 Duplicate siblings
+
+Two children describe the same uncertainty with different wording.
+
+### 10.5 Incomplete coverage
+
+All children can be resolved, but part of the parent's uncertainty remains unaddressed.
+
+### 10.6 Over-decomposition
+
+An already answerable question is broken into fragments that are too trivial or unnatural to investigate separately.
+
+### 10.7 Under-decomposition
+
+A broad or compound uncertainty is treated as atomic, producing a difficult multi-part user question.
+
+### 10.8 Template-driven decomposition
+
+Children are generated because a template expects them, rather than because the parent contains those distinct uncertainties.
+
+### 10.9 Context loss
+
+Children become independently answerable but lose their traceable relationship to why they matter.
+
+### 10.10 Dead-end after filtering
+
+Valid unresolved children exist, but all are excluded by structural, pattern or quality checks. The engine should diagnose the exact exclusion path rather than silently treating the investigation as complete.
+
+---
+
+## 11. Worked examples
+
+### 11.1 Commercial validation
+
+Parent:
+
+> Is this use case commercially justified?
+
+Poor children:
+
+- Is the use case commercially feasible with a viable pricing model?
+- Is there sufficient commercial demand?
+
+Problems:
+
+- inherited framing;
+- compound wording;
+- children remain at parent abstraction level;
+- high risk of recursive decomposition.
+
+Better children:
+
+- Who experiences the problem?
+- What cost or harm does the problem create?
+- Is the problem frequent enough to matter?
+- Will an identifiable customer pay to reduce it?
+- Can the solution be delivered at a sustainable cost?
+
+Each child owns one uncertainty. The parent retains the commercial-justification frame and combines the child outcomes.
+
+### 11.2 Vehicle fault diagnosis
+
+Parent:
+
+> Why will the car not start?
+
+Candidate children:
+
+- Does the starter motor turn?
+- Is battery voltage sufficient under load?
+- Is fuel reaching the engine?
+- Is the immobiliser preventing ignition?
+
+Each question supports a distinct investigation path.
+
+### 11.3 Agile readiness
+
+Parent:
+
+> Is this story ready to enter the sprint?
+
+Candidate children:
+
+- Is the expected outcome clear?
+- Are the acceptance conditions testable?
+- Are external dependencies resolved?
+- Is the required data available?
+- Can the team complete the work within the sprint boundary?
+
+The phrase *ready to enter the sprint* remains parent context. Each child investigates one condition contributing to readiness.
+
+### 11.4 Financial decision
+
+Parent:
+
+> Can the household safely retire at 67?
+
+Candidate children:
+
+- What annual essential spending must be covered?
+- What secure income will be available?
+- What investment assets will exist at retirement?
+- What debts will remain?
+- How resilient is the plan to lower returns or one spouse surviving longer?
+
+### 11.5 Software architecture
+
+Parent:
+
+> Should this service be separated from the monolith?
+
+Candidate children:
+
+- Does it need an independent deployment cycle?
+- Does it have a stable data boundary?
+- Would separation materially reduce operational risk?
+- Does the team have the capability to operate it independently?
+
+The parent owns the architectural decision. Children own the evidence needed to support it.
+
+---
+
+## 12. Relationship with reasoning patterns
+
+Decomposition and reasoning-pattern selection are related but distinct.
+
+- Decomposition decides whether an uncertainty is small enough to investigate directly.
+- Reasoning-pattern selection decides what kind of investigation is appropriate.
+
+The engine should not use a reasoning pattern to disguise a composite uncertainty as atomic.
+
+Equally, decomposition should not erase the parent's reasoning context. The child remains linked to the parent even though it does not repeat the parent's framing in its own wording.
+
+When a branch evolves into a different kind of reasoning, the engine may re-evaluate the active pattern using the remaining graph state. It should not bypass compatibility merely to force a next question.
+
+---
+
+## 13. Relationship with answerability
+
+Atomicity and answerability are not identical.
+
+A node may be atomic but temporarily unanswerable.
+
+Example:
+
+> What was the measured defect rate last quarter?
+
+This asks one thing, but the data may not yet exist.
+
+The correct response may be to identify an evidence-gathering action rather than decompose the question further.
+
+The engine should distinguish:
+
+- too broad to answer;
+- clear but evidence unavailable;
+- clear and directly answerable;
+- clear but requiring specialist capability.
+
+---
+
+## 14. Validation requirements for implementation
+
+An implementation should be able to demonstrate the following.
+
+### 14.1 Atomicity validation
+
+- Atomic children are not repeatedly decomposed because of inherited parent wording.
+- Genuinely compound children are still detected.
+
+### 14.2 Ownership validation
+
+- Parent framing does not appear in child labels unless independently necessary.
+- Child context remains available through graph links rather than duplicated text.
+
+### 14.3 Narrowing validation
+
+- Every accepted child has a more specific semantic scope than its parent.
+
+### 14.4 Duplicate validation
+
+- Semantic duplicates are rejected even when wording differs.
+
+### 14.5 Coverage validation
+
+- The implementation records how children collectively resolve the parent.
+
+### 14.6 Convergence validation
+
+- Repeated decomposition reaches atomic nodes or produces an explicit decomposition failure.
+- It does not oscillate or recreate the same semantic uncertainty at deeper levels.
+
+### 14.7 Question validation
+
+- Every selected child can produce one understandable user-facing question.
+- A compound child cannot escape into the user interface.
+
+### 14.8 Dead-end diagnostics
+
+When unresolved nodes remain but no question is produced, diagnostics must identify:
+
+- unresolved candidates;
+- structural eligibility;
+- pattern compatibility;
+- atomicity status;
+- decomposition result;
+- selected node, if any;
+- formulation result;
+- exact no-question reason.
+
+---
+
+## 15. Acceptance criteria for future Codex implementation
+
+A future implementation change should not be accepted unless it proves all of the following:
+
+1. Commercial parent framing no longer contaminates generated children.
+2. Generated atomic commercial children remain atomic.
+3. Genuinely composite commercial children still decompose.
+4. Child labels describe only their own uncertainty.
+5. Parent context remains preserved in the graph.
+6. Every accepted child is narrower than the parent.
+7. Duplicate and compound children remain rejected.
+8. Decomposition converges without increasing the maximum depth merely to hide recursion.
+9. Reasoning-pattern safeguards remain intact.
+10. The engine does not fall back to arbitrary unresolved candidates.
+11. Existing decision, explanation, contradiction, definition, diagnosis, comparison and prioritisation behaviours remain valid.
+12. Diagnostics clearly explain any remaining no-question state.
+
+---
+
+## 16. Guidance for coding agents
+
+When implementing this specification:
+
+1. Inspect the current repository and existing tests before proposing changes.
+2. Identify the exact observed violation of an invariant.
+3. Prefer the smallest structural correction.
+4. Do not broaden pattern compatibility to hide decomposition defects.
+5. Do not increase recursion depth as the primary fix.
+6. Do not place user-facing prose into technical graph-description functions.
+7. Keep parent context and child uncertainty as separate graph semantics.
+8. Add focused regression tests before broad refactoring.
+9. Preserve existing working reasoning families.
+10. Report which specification invariant each code change enforces.
+
+Suggested implementation prompt framing:
+
+> Read the Decomposition and Atomic Reasoning Specification. Identify where the current implementation violates its invariants. Implement the smallest correction that prevents parent framing from contaminating generated children while preserving reasoning-pattern compatibility, graph traceability and existing decomposition safeguards.
+
+---
+
+## 17. Open questions
+
+The following remain deliberately unresolved and should be answered through further experiments:
+
+- How should coverage be represented when children are sufficient but not individually necessary?
+- How should OR, AND and threshold aggregation differ?
+- When should child resolution automatically resolve the parent?
+- How should uncertain or conflicting child evidence affect parent status?
+- When should a failed decomposition trigger reframing rather than stopping?
+- How should specialist evidence-gathering actions be represented for atomic but currently unanswerable nodes?
+- How should the engine distinguish a missing child from a genuinely sufficient decomposition?
+- How should context ownership be applied to assumptions, conclusions and relationships as well as unknowns?
+
+These questions should not be answered by adding rules without observed evidence.
+
+---
+
+## 18. Summary
+
+The Confidence Engine does not decompose because smaller questions are aesthetically preferable.
+
+It decomposes because large uncertainties cannot be investigated honestly in one step.
+
+The engine should keep breaking uncertainty down until each remaining question owns one distinct doubt and can be settled by one focused investigation.
+
+The parent retains the wider context. The child owns only the uncertainty it investigates.
+
+A valid decomposition is narrower, independent, unique, complete, traceable and convergent.
+
+> **Break the complicated into small, granular, simple things — then investigate one useful thing at a time.**
+