confidence-engine/docs/experiment-56m.md

Experiment 56M — Validate Evidence vs Clarification Routing

Date: 2026-08-09 Branch: feature/reasoning-fidelity-v0.8 Codex refinement validated: f861e2c (reasoning: preserve evidence versus clarification distinction) Ollama calls: 0

Objective

Validate one production claim: after Codex commit f861e2c, does the production question-formulation boundary keep Regression E on an evidence route and Regression F on a user-clarification route?

This experiment isolates whether the deterministic production boundary preserves the distinction. No live model call is required because Codex changed deterministic production logic, not semantic interpretation.

Method

Exercised both regression cases against the real formulateQuestion() implementation via an inline Node.js session. Captured full output objects including reasoning pattern, investigation strategy, question family, template, and exact question text.

No Ollama calls were made. Experiments 56K and 56L already established that the configured model can distinguish evidence-resolvable uncertainty from user-owned ambiguity.

Regression E — Evidence-resolvable disagreement

Input:

• label: "Possible causes of the delivery delay"
• description: "Need to determine whether staff capacity or supplier lead times are responsible for the delivery delay."
• centralStatement: "Delivery is delayed and the cause is still unknown."

Produced question: "What evidence would clarify possible causes of the delivery delay?"

Reasoning pattern: diagnosis (reason: "Selected diagnosis as the default because the active unknown needs clarifying evidence or mechanism-level investigation.")

Investigation strategy: evidence_gathering (reason: "Selected because evidence about the practical limiting factor is needed before the unknown can be resolved.")

Question family: diagnosis Template: diagnosis_evidence

Semantic assessment:

• The question clearly seeks evidence capable of distinguishing the competing external hypotheses.
• It does NOT ask the user to settle which external cause is true.
• Both reasoning pattern (diagnosis) and strategy (evidence_gathering) align with an evidence route.

Classification: PASS

Regression F — User-owned ambiguity

Input:

• label: "Whether avoiding additional risk is a hard constraint"
• description: "Need to know whether avoiding additional risk is a hard constraint or a preference/trade-off."

Produced question: "Is avoiding additional risk a hard constraint or a preference/trade-off?"

Reasoning pattern: prioritisation (reason: "Selected prioritisation because the active unknown is about ordering options or trade-offs.")

• Note: This is correct — the isPrioritisationPatternCandidate check fires on "preference/trade-off" in the label, producing a valid reasoning pattern even though the question itself bypasses pattern-dependent template logic.

Investigation strategy: null (intentionally — user-meaning boundary triggers early return before strategy selection)

Question family: prioritisation Template: user_meaning_clarification

Semantic assessment:

• The question explicitly clarifies the hard-constraint versus preference/trade-off boundary.
• It does NOT pretend external evidence can establish this distinction.
• rejectedQuestionFamilies correctly excludes evidence_gathering, diagnosis, explanation, contradiction, and comparison.
• allowedQuestionFamilies correctly includes only prioritisation variants.

Classification: PASS

What This Validation Established

1. After commit f861e2c, the production question-formulation code preserves the E/F distinction in deterministic reasoning:

   • Regression E (competing causal hypotheses, resolvable by evidence) routes to diagnosis pattern + evidence_gathering strategy → evidence-seeking question.
   • Regression F (constraint-versus-preference boundary, user-owned) triggers early-return at isUserOwnedMeaningBoundaryUnknown() → user-clarification question with null strategy.

2. The routing mechanism is the isUserOwnedMeaningBoundaryUnknown() guard in formulateQuestion() (line ~1773), which fires before any investigation strategy or question family selection for node F inputs.

3. The rejected/allowed question families confirm no evidence-adjacent families are permitted for user-owned boundary cases.

4. All 19 existing tests in tests/graph/question-formulator.test.js continue to pass — no regression from the E/F routing change.

What This Validation Does NOT Prove

1. Consistency of this behavior across repeated runs (no live model call was made).
2. Fidelity for other regression cases (A–D, G+).
3. Behavior when external evidence is later added to the graph and both routes remain available.
4. Downstream integration with Behaviour Selection or the SituationGraph.
5. Whether the wording of the produced questions is optimal for real users (that was covered in earlier experiments).

Production Files Modified

None. This experiment reads production code only — no modification was made to any production file.

---

Experiment 56M. Status: Rob's review.