94 lines
5.1 KiB
Markdown
94 lines
5.1 KiB
Markdown
# Experiment 56M — Validate Evidence vs Clarification Routing
|
||
|
||
**Date:** 2026-08-09
|
||
**Branch:** `feature/reasoning-fidelity-v0.8`
|
||
**Codex refinement validated:** `f861e2c` (reasoning: preserve evidence versus clarification distinction)
|
||
**Ollama calls:** 0
|
||
|
||
## Objective
|
||
|
||
Validate one production claim: after Codex commit `f861e2c`, does the production question-formulation boundary keep Regression E on an evidence route and Regression F on a user-clarification route?
|
||
|
||
This experiment isolates whether the deterministic production boundary preserves the distinction. No live model call is required because Codex changed deterministic production logic, not semantic interpretation.
|
||
|
||
## Method
|
||
|
||
Exercised both regression cases against the real `formulateQuestion()` implementation via an inline Node.js session. Captured full output objects including reasoning pattern, investigation strategy, question family, template, and exact question text.
|
||
|
||
No Ollama calls were made. Experiments 56K and 56L already established that the configured model can distinguish evidence-resolvable uncertainty from user-owned ambiguity.
|
||
|
||
## Regression E — Evidence-resolvable disagreement
|
||
|
||
**Input:**
|
||
- `label`: "Possible causes of the delivery delay"
|
||
- `description`: "Need to determine whether staff capacity or supplier lead times are responsible for the delivery delay."
|
||
- `centralStatement`: "Delivery is delayed and the cause is still unknown."
|
||
|
||
**Produced question:** "What evidence would clarify possible causes of the delivery delay?"
|
||
|
||
**Reasoning pattern:** diagnosis (reason: "Selected diagnosis as the default because the active unknown needs clarifying evidence or mechanism-level investigation.")
|
||
|
||
**Investigation strategy:** `evidence_gathering` (reason: "Selected because evidence about the practical limiting factor is needed before the unknown can be resolved.")
|
||
|
||
**Question family:** diagnosis
|
||
**Template:** diagnosis_evidence
|
||
|
||
**Semantic assessment:**
|
||
- The question clearly seeks evidence capable of distinguishing the competing external hypotheses.
|
||
- It does NOT ask the user to settle which external cause is true.
|
||
- Both reasoning pattern (diagnosis) and strategy (evidence_gathering) align with an evidence route.
|
||
|
||
**Classification: PASS**
|
||
|
||
## Regression F — User-owned ambiguity
|
||
|
||
**Input:**
|
||
- `label`: "Whether avoiding additional risk is a hard constraint"
|
||
- `description`: "Need to know whether avoiding additional risk is a hard constraint or a preference/trade-off."
|
||
|
||
**Produced question:** "Is avoiding additional risk a hard constraint or a preference/trade-off?"
|
||
|
||
**Reasoning pattern:** prioritisation (reason: "Selected prioritisation because the active unknown is about ordering options or trade-offs.")
|
||
- **Note:** This is correct — the `isPrioritisationPatternCandidate` check fires on "preference/trade-off" in the label, producing a valid reasoning pattern even though the question itself bypasses pattern-dependent template logic.
|
||
|
||
**Investigation strategy:** null (intentionally — user-meaning boundary triggers early return before strategy selection)
|
||
|
||
**Question family:** prioritisation
|
||
**Template:** user_meaning_clarification
|
||
|
||
**Semantic assessment:**
|
||
- The question explicitly clarifies the hard-constraint versus preference/trade-off boundary.
|
||
- It does NOT pretend external evidence can establish this distinction.
|
||
- `rejectedQuestionFamilies` correctly excludes evidence_gathering, diagnosis, explanation, contradiction, and comparison.
|
||
- `allowedQuestionFamilies` correctly includes only prioritisation variants.
|
||
|
||
**Classification: PASS**
|
||
|
||
## What This Validation Established
|
||
|
||
1. After commit `f861e2c`, the production question-formulation code preserves the E/F distinction in deterministic reasoning:
|
||
- Regression E (competing causal hypotheses, resolvable by evidence) routes to `diagnosis` pattern + `evidence_gathering` strategy → evidence-seeking question.
|
||
- Regression F (constraint-versus-preference boundary, user-owned) triggers early-return at `isUserOwnedMeaningBoundaryUnknown()` → user-clarification question with null strategy.
|
||
|
||
2. The routing mechanism is the `isUserOwnedMeaningBoundaryUnknown()` guard in `formulateQuestion()` (line ~1773), which fires before any investigation strategy or question family selection for node F inputs.
|
||
|
||
3. The rejected/allowed question families confirm no evidence-adjacent families are permitted for user-owned boundary cases.
|
||
|
||
4. All 19 existing tests in `tests/graph/question-formulator.test.js` continue to pass — no regression from the E/F routing change.
|
||
|
||
## What This Validation Does NOT Prove
|
||
|
||
1. Consistency of this behavior across repeated runs (no live model call was made).
|
||
2. Fidelity for other regression cases (A–D, G+).
|
||
3. Behavior when external evidence is later added to the graph and both routes remain available.
|
||
4. Downstream integration with Behaviour Selection or the SituationGraph.
|
||
5. Whether the wording of the produced questions is optimal for real users (that was covered in earlier experiments).
|
||
|
||
## Production Files Modified
|
||
|
||
None. This experiment reads production code only — no modification was made to any production file.
|
||
|
||
---
|
||
|
||
*Experiment 56M. Status: Rob's review.*
|