From ec398dcec9f802bc5998ae7b2d0e18e56af29a1e Mon Sep 17 00:00:00 2001 From: robbond Date: Sun, 9 Aug 2026 16:33:14 +0100 Subject: [PATCH] experiment: validate evidence versus clarification routing --- docs/current-handoff.md | 4 ++ docs/experiment-56m.md | 93 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 97 insertions(+) create mode 100644 docs/experiment-56m.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index a222df2..b85748a 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -235,3 +235,7 @@ One live Ollama call (14,032 ms) returned `uncertaintyType: "user_clarification_ **Classification: PASS.** The model correctly identified the ambiguity as user-owned, did not introduce spurious evidence gathering, and preserved the evidence-vs-user-meaning distinction cleanly. This experiment does not prove consistency across repeated runs, fidelity for other regression cases (A–E, G+), behavior in production reasoning paths, or downstream integration with Behaviour Selection or the SituationGraph. Branch: `feature/reasoning-fidelity-v0.8`. Files: `tests/reconstruction/semantic-regression-f-user-owned-ambiguity.test.js` and `docs/experiment-56l.md`. Status pending Rob's review. + +### Experiment 56M Summary — Production evidence vs clarification routing validation + +Validated one production claim after Codex commit `f861e2c`: does the deterministic question-formulation boundary preserve the E/F distinction? No live Ollama calls were made (0). Deterministic `formulateQuestion()` was exercised with both regression fixtures. Regression E (competing delivery-delay causes: "Staff capacity may be the issue" / "Supplier lead times are likely responsible.") produced question: "What evidence would clarify possible causes of the delivery delay?" — reasoning pattern=diagnosis, strategy=evidence_gathering, template=diagnosis_evidence. PASS. Regression F (preference vs constraint ambiguity: "Whether avoiding additional risk is a hard constraint") produced question: "Is avoiding additional risk a hard constraint or a preference/trade-off?" — reasoning pattern=prioritisation, strategy=null, template=user_meaning_clarification, with rejected families correctly excluding all evidence-adjacent families. PASS. Both cases maintain their distinct routes: E on an evidence route and F on user clarification. All 19 existing tests continue to pass. Branch: `feature/reasoning-fidelity-v0.8`. File: `docs/experiment-56m.md`. Status pending Rob's review. diff --git a/docs/experiment-56m.md b/docs/experiment-56m.md new file mode 100644 index 0000000..5a4a9a9 --- /dev/null +++ b/docs/experiment-56m.md @@ -0,0 +1,93 @@ +# Experiment 56M — Validate Evidence vs Clarification Routing + +**Date:** 2026-08-09 +**Branch:** `feature/reasoning-fidelity-v0.8` +**Codex refinement validated:** `f861e2c` (reasoning: preserve evidence versus clarification distinction) +**Ollama calls:** 0 + +## Objective + +Validate one production claim: after Codex commit `f861e2c`, does the production question-formulation boundary keep Regression E on an evidence route and Regression F on a user-clarification route? + +This experiment isolates whether the deterministic production boundary preserves the distinction. No live model call is required because Codex changed deterministic production logic, not semantic interpretation. + +## Method + +Exercised both regression cases against the real `formulateQuestion()` implementation via an inline Node.js session. Captured full output objects including reasoning pattern, investigation strategy, question family, template, and exact question text. + +No Ollama calls were made. Experiments 56K and 56L already established that the configured model can distinguish evidence-resolvable uncertainty from user-owned ambiguity. + +## Regression E — Evidence-resolvable disagreement + +**Input:** +- `label`: "Possible causes of the delivery delay" +- `description`: "Need to determine whether staff capacity or supplier lead times are responsible for the delivery delay." +- `centralStatement`: "Delivery is delayed and the cause is still unknown." + +**Produced question:** "What evidence would clarify possible causes of the delivery delay?" + +**Reasoning pattern:** diagnosis (reason: "Selected diagnosis as the default because the active unknown needs clarifying evidence or mechanism-level investigation.") + +**Investigation strategy:** `evidence_gathering` (reason: "Selected because evidence about the practical limiting factor is needed before the unknown can be resolved.") + +**Question family:** diagnosis +**Template:** diagnosis_evidence + +**Semantic assessment:** +- The question clearly seeks evidence capable of distinguishing the competing external hypotheses. +- It does NOT ask the user to settle which external cause is true. +- Both reasoning pattern (diagnosis) and strategy (evidence_gathering) align with an evidence route. + +**Classification: PASS** + +## Regression F — User-owned ambiguity + +**Input:** +- `label`: "Whether avoiding additional risk is a hard constraint" +- `description`: "Need to know whether avoiding additional risk is a hard constraint or a preference/trade-off." + +**Produced question:** "Is avoiding additional risk a hard constraint or a preference/trade-off?" + +**Reasoning pattern:** prioritisation (reason: "Selected prioritisation because the active unknown is about ordering options or trade-offs.") +- **Note:** This is correct — the `isPrioritisationPatternCandidate` check fires on "preference/trade-off" in the label, producing a valid reasoning pattern even though the question itself bypasses pattern-dependent template logic. + +**Investigation strategy:** null (intentionally — user-meaning boundary triggers early return before strategy selection) + +**Question family:** prioritisation +**Template:** user_meaning_clarification + +**Semantic assessment:** +- The question explicitly clarifies the hard-constraint versus preference/trade-off boundary. +- It does NOT pretend external evidence can establish this distinction. +- `rejectedQuestionFamilies` correctly excludes evidence_gathering, diagnosis, explanation, contradiction, and comparison. +- `allowedQuestionFamilies` correctly includes only prioritisation variants. + +**Classification: PASS** + +## What This Validation Established + +1. After commit `f861e2c`, the production question-formulation code preserves the E/F distinction in deterministic reasoning: + - Regression E (competing causal hypotheses, resolvable by evidence) routes to `diagnosis` pattern + `evidence_gathering` strategy → evidence-seeking question. + - Regression F (constraint-versus-preference boundary, user-owned) triggers early-return at `isUserOwnedMeaningBoundaryUnknown()` → user-clarification question with null strategy. + +2. The routing mechanism is the `isUserOwnedMeaningBoundaryUnknown()` guard in `formulateQuestion()` (line ~1773), which fires before any investigation strategy or question family selection for node F inputs. + +3. The rejected/allowed question families confirm no evidence-adjacent families are permitted for user-owned boundary cases. + +4. All 19 existing tests in `tests/graph/question-formulator.test.js` continue to pass — no regression from the E/F routing change. + +## What This Validation Does NOT Prove + +1. Consistency of this behavior across repeated runs (no live model call was made). +2. Fidelity for other regression cases (A–D, G+). +3. Behavior when external evidence is later added to the graph and both routes remain available. +4. Downstream integration with Behaviour Selection or the SituationGraph. +5. Whether the wording of the produced questions is optimal for real users (that was covered in earlier experiments). + +## Production Files Modified + +None. This experiment reads production code only — no modification was made to any production file. + +--- + +*Experiment 56M. Status: Rob's review.*