confidence-engine/docs/reasoning-refinement-requirements.md

Reasoning Refinement Requirements — Experiment 55E Synthesis

This document captures the evidence-backed requirements emerging from the current reasoning-experiment series. It is a handoff into the next implementation pass, not a final architecture and not a replacement for the design-evolution log.

---

1. Purpose

The Confidence Engine's investigation has exposed recurring patterns where later reasoning silently converts user-supplied meaning into stronger model interpretation, flattens qualification, or over-resolves uncertainty. These experiments tested those patterns progressively across semantic separation, clarification chains, and resolution stability.

This document records what can be relied on, which requirements follow from the evidence, which regression cases should constrain the first implementation pass, and what remains genuinely unresolved.

---

Status: First implementation pass completed against A–F regression boundaries on feature/reasoning-fidelity-v0.8; remaining open design questions are not claimed solved.

---

2. What We Can Rely On So Far

Source versus interpretation

Experiments 54H–54J demonstrated that deterministic source identity (SHA-256 hashing of raw user input) can distinguish what the user supplied from what the model might infer. Multiple interpretations can share one deterministic source lineage. Automated semantic grounding (Experiment 54K/54L) showed promising directionality with one missed addition in a multi-addition case — it is viable but imperfect. These findings support the requirement to preserve source-versus-inference separation, though automated grounding alone cannot yet be relied upon for complete accuracy.

Stated meaning versus possible inference

Experiment 55D tested explicit statedMeaning / possibleInference separation across four fixed answers (weak priority, conditional trade-off, explicit hard constraint, non-answer). All four preserved statedMeaning without strengthening (4/4). Inference was cleanly separated for Cases 1 and 2; Cases 3 and 4 received unnecessary inferences (hygiene issue, not leakage). No unsupported meaning leaked into statedMeaning. This establishes that a single-call two-field output contract can separate what the user directly established from plausible model interpretation — but broader stability across other models and answer patterns remains untested.

Note: statedMeaning means meaning directly supported by the user's words, not necessarily verbatim text reproduction. Contextual framing of the clarification target in a non-answer (as seen in 55D Case 4) is acceptable; it does not constitute unsupported strengthening.

Uncertainty

Experiments 54T, 55A Case 4, and 55C Case 3 confirmed that uncertainty can be preserved when explicitly tested. A user who says "I'm not really sure" can remain uncertain without the engine forcing resolution or a leaning, supported by repeated-identical-input testing in 54T.

Evidence versus clarification

Experiment 54R showed the model can distinguish between uncertainty that observable evidence can resolve (e.g., competing delivery causes: staff capacity vs supplier lead times) and ambiguity that only the user can resolve (e.g., growth-versus-risk priority trade-off). All three fixed cases in 54R were correct. Experiment 54O/54Q showed that direct evidence-need comparison can succeed where consequence-detection alone failed (54N Case 3). These findings support distinguishing evidence-gathering need from user-clarification need, but broader reliability remains unproven.

Clarification chain

Experiments 54S through 54W tested the sequence: clarification need → clarification target → question → answer interpretation → target resolution. Each individual step worked in isolated fixed-case tests. One small chained probe (54W) produced a correct end-to-end result; upstream distortion can propagate through to downstream resolution (55C). The stages worked individually and in one small chained probe; broader reliability remains untested.

Experiments 54X–54Z showed target-specificity loss: the model sometimes broadens targets from material distinctions ("hard constraint vs preference/trade-off") to coarser priority ordering ("preferred priority between growth and risk"). With explicit answers this did not change resolved meaning (54Y); with weak answers it exposed differences between precise and broadened targets (54Z). Direction is unpredictable.

Meaning before judgement

Experiments 55B and 55C tested whether extracting meaning before making resolution decisions preserves more nuance than resolving directly. Meaning-only extraction preserved all three tested answers in 55B; carrying that meaning forward protected conditionality in 55C Case 2. However, if Stage 1 distorts the answer, Stage 2 propagates that distortion (55C Case 1). Neither two-stage approach consistently outperforms the other across all tested answer types. This supports preserving faithful early meaning when available, but does not establish a required two-call architecture.

---

3. Reasoning Requirements for the First Implementation Pass

Each requirement below is supported by at least one recorded experiment observation. Where wording is stronger than evidence, it has been weakened.

R1 — Preserve user-supplied meaning

The engine must retain what the user actually established without silently strengthening it. Supported by: 55D (4/4 statedMeaning preserved), 55C Case 1 (distortion propagated downstream). Strongest-supported requirement.

R2 — Keep inference distinguishable

Reasonable model inference must not become indistinguishable from user-supplied meaning. Supported by: 53, 55D (leakage = 0), 54J/54K (representation can separate source from interpretation).

R3 — Preserve qualification and conditionality

Language such as "might," "normally," "depends," "for the right opportunity," "I'm not sure" must not be flattened into unconditional conclusions. Supported by: 54Z, 55A Case 3, 55D Case 2.

R4 — Preserve unresolved uncertainty

A weak answer must be allowed to leave a target unresolved. Supported by: 54T (null-gating stable across three repeated calls), 55A Case 4, 55C Case 3.

R5 — Distinguish evidence need from clarification need

Do not ask the user merely because the engine lacks external evidence. Supported by: 54R (3/3 correct for tested disagreement types), 54Q (explicit evidence distinction supports consequence judgment).

R6 — Clarify the actual unresolved distinction

When clarification is needed, the target must reflect the specific user-owned ambiguity rather than only the general topic. Supported by: 54S (3/3 correct targets), 54X (broadening observed but not universal).

R7 — Avoid unnecessary clarification

If clarification is explicitly not required, do not invent a clarification target. Supported by: 54T (null-gating stable for non-required cases), 54S instability in earlier runs.

R8 — Resolution must operate on preserved meaning

Any later judgement must operate on preserved meaning rather than silently replacing it with stronger interpretation. Supported by: 55C Case 1 (distortion propagated), 55D (two-field separation prevented strengthening in the tested run). Partially supported: evidence shows operating on distorted meaning produces incorrect results, but the specific mechanism for consuming these fields has not been directly tested.

---

4. Known Failure Modes

Failure pattern	Example	What went wrong	Evidence status
Weak priority over-resolved to "not a constraint"	"Risk matters more to me." → targetResolved=true with inferred boundary meaning	Model converted relative importance into a negative hard-constraint assertion	Observed multiple times across 55A, 55B; varied across runs (run-specific)
Conditional trade-off losing qualification	"I'd normally avoid more risk, but for the right opportunity I might accept some." → flattened to "preference or trade-off"	Conditionality removed during resolution judgement	Observed in 55A Case 3 and 55B; survived with two-field separation in 55D
Target broadening replacing material distinction with priority ordering	"hard constraint vs preference/trade-off" → "preferred priority between growth and risk"	Clarification target lost the user-owned boundary type	Reproduced across runs (54X); consequence varies by answer strength (54Z)
Unnecessary possibleInference generated for explicit or non-answers	Cases 3/4 in 55D received speculative implications where none was warranted	Model tends to always provide inference content rather than null	Observed once; hygiene issue, not leakage
Stage 1 distortion propagating through Stage 2 resolution	55C Case 1: weak-priority strengthening in Stage 1 carried into Stage 2 resolution	Preserved meaning was itself distorted upstream	Observed in one run; asymmetric with conditional answers (which improved)
Uncertainty sometimes over-resolved to a leaning	"I'm not really sure" handled correctly in 55A/55C/54T, but other weak answers can produce implicit leanings	Varies by answer pattern and context	Not yet generalised; confirmed correct for non-answer in tested cases
Automated grounding missing minor additions	54K Case 2: model missed one source-supported content gap during multi-addition grounding	Grounding directionally viable but imperfect completeness	Observed once in multi-addition context

---

5. Known Good Behaviours

These have worked reliably enough in the tested runs to serve as regression expectations:

• Explicit hard constraint remains a hard constraint (55D Case 3, 54Y, 54V Case 1).
• "I'm not really sure" remains uncertain without forced resolution (54T, 55A Case 4, 55C Case 3).
• Operational cause disagreement can be evidence-resolvable without user clarification (54R Case 1, 54Q).
• Affordability ambiguity can produce a precise clarification target ("upfront cost versus long-term total cost") (54S Case 3).
• Fixed clarification target can produce one neutral question without adding meaning (54U: 3/3 cases returned correct single neutral questions).
• Explicit hard-constraint answer resolves under both precise and broadened clarification targets to materially equivalent resolved meaning (54Y).
• Weak-priority statedMeaning remains clean with two-field separation (55D Case 1: "Risk matters more" stayed as relative importance, inference kept separate).
• Deterministic source identity via SHA-256 is stable across repeated identical inputs (54H).

Be precise about scope: each of these applies only within the tested answer patterns and model configuration. Broader reliability is not yet established.

---

6. Regression Pack for the First Production Refinement

Regression A — Weak priority (over-resolution)

• Source: "I want the business to grow, but I don't want to take on more risk."
• Relevant answer: "Risk matters more to me."
• Expected preserved meaning: Risk is of greater relative importance than growth; no hard-constraint or non-hard-constraint boundary established.
• Expected uncertainty: Hard-constraint status for avoiding additional risk is unresolved.
• Must not happen: Inference that risk avoidance is "not a hard constraint" or equivalent negative assertion. The answer says nothing about whether it is a hard constraint, only about relative importance.

Regression B — Conditional trade-off

• Source: "I want the business to grow, but I don't want to take on more risk."
• Relevant answer: "I'd normally avoid more risk, but for the right opportunity I might accept some."
• Expected preserved meaning: Normal preference to avoid additional risk; conditional willingness to accept some under specific circumstances.
• Expected uncertainty: What constitutes "the right opportunity" remains undefined.
• Must not happen: Removal of the conditional qualification ("for the right opportunity"). The answer establishes a two-sided conditional, not a flat stance.

Regression C — Non-answer (uncertainty)

• Source: "I want the business to grow, but I don't want to take on more risk."
• Relevant answer: "I'm not really sure."
• Expected preserved meaning: User is uncertain about whether avoiding additional risk is a hard constraint or preference/trade-off.
• Expected uncertainty: Full — no position taken.
• Must not happen: Any leaning, inference about what the user likely prefers, or forced resolution of the underlying ambiguity. The answer may leave the existing clarification target unresolved and require further clarification.

Regression D — Explicit hard constraint

• Source: "I want the business to grow, but I don't want to take on more risk."
• Relevant answer: "It's a hard constraint. I don't want any increase in risk."
• Expected preserved meaning: Avoiding additional risk is established as non-negotiable; no exception cases supplied.
• Expected uncertainty: None regarding the constraint itself (user was explicit). Uncertainty may exist about what qualifies as "increase in risk" or how this interacts with other objectives.
• Must not happen: Downgrading to preference/trade-off, adding conditional qualifications not present in the source.

Regression E — Evidence-resolvable disagreement

• Source: Delivery delay concern.
• Relevant answer/competing causes: "Staff capacity may be the issue" / "Supplier lead times are likely responsible."
• Expected preserved meaning: Two distinct hypotheses about causation.
• Expected uncertainty: Which hypothesis is correct — resolvable by evidence gathering, not user clarification.
• Must not happen: Generating a user-facing clarification question when evidence sources can distinguish the hypotheses.

Regression F — User-owned ambiguity

• Source: "I want the business to grow, but I don't want to take on more risk."
• Relevant answer: (ambiguous statement about both)
• Expected preserved meaning: User has not specified whether avoiding additional risk is a hard constraint or a strong preference/trade-off.
• Expected uncertainty: Preference vs constraint distinction is user-owned and requires clarification.
• Must not happen: Engine-generated classification of the ambiguity as "not requiring clarification" or resolution through evidence gathering alone.

---

7. What Is Still Unproven

The following areas remain genuinely unresolved. Tomorrow's implementation should treat these as open questions, not settled decisions:

• Stability across larger case sets: Six regression cases plus supporting observations cover key patterns, but the engine has not been tested against diverse answer patterns, domains, or multiple reasoning turns.
• Behaviour across other models: All experiments used qwen-claude:latest on one instance. Different models (or different fine-tuning of the same model) may behave differently on these tasks.
• Exact production representation: Whether statedMeaning/possibleInference, a provenance annotation layer, or another mechanism is appropriate for production has not been decided.
• Graph integration: How these fields integrate into SituationGraph nodes and edges remains an open design question (Experiments 54A–54C confirmed the current graph lacks provenance fields).
• Downstream inference-field consumption: How Behaviour Selection, UI, or later reasoning steps should consume possibleInference vs statedMeaning has not been tested.
• Clarification-target precision requirements: The boundary between "materially different clarification target" and "coarser but workable framing" has been partially observed (54Y/54Z) but not generalised.
• Behaviour Selection integration: No integration tests have been performed for the clarified-meaning pipeline with Behaviour Selection.
• UI timing and wording quality: How and when clarification questions are presented to users, and whether the generated wording is appropriate, has not been tested beyond one neutral-question pass (54U).
• Performance/latency implications: Adding semantic separation, grounding, or two-field interpretation calls will add latency. This has not been measured against production requirements.
• One call versus multiple calls: The two-stage approach (55B/C) showed promise for conditionality but also propagation risk. The single-call two-field approach (55D) was cleaner but untested for resolution consumption. Which is appropriate remains undecided.

---

8. First Implementation Boundary

The first production refinement should preserve user-supported meaning, model inference, qualification, and unresolved uncertainty distinctly enough that later reasoning cannot silently convert one into another. The engine must not strengthen relative importance into constraint boundaries, flatten conditional qualifications, or over-resolve uncertainty when the answer does not supply sufficient information.

The implementation should be judged first against the regression pack above before expanding into Behaviour Selection, UI, or broader reasoning redesign.

---

9. Stop Conditions for the Next Implementation Pass

The first implementation pass should stop and reassess if:

• It requires redesigning the whole SituationGraph;
• It requires broad UI changes;
• It requires rewriting Behaviour Selection;
• It requires rereading the complete historical experiment log (this document exists so this is not needed);
• It introduces multiple new abstractions before passing the regression pack;
• The implementation cannot be explained in a few paragraphs;
• The first attempt starts changing unrelated engine behaviour.

This section is scope-control guidance, not architecture design.