From 5dcaed39df012357a74d5a92e3db51a66f64e03a Mon Sep 17 00:00:00 2001 From: robbond Date: Thu, 13 Aug 2026 07:15:01 +0100 Subject: [PATCH] experiment: diagnose decision sufficiency rule MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Read-only inspection of 8 files (prompt-builder.js, schema.js, utils.js, apply-proposal.js, orchestrator.js, experiment-60b1.md, experiment-60b2.md, current-handoff.md). No code changes. Key findings: - Prompt has no independent materiality/sufficiency rule (Rule 20 says null selectedQuestion when 'no consequential unresolved unknown' but doesn't define what makes an unknown non-consequential) - Validator performs structural checks only, no evidence sufficiency evaluation - No cross-option comparison logic in propagateResolvedChildEvidence - Schema has no materiality or couldChangeDecision field - 60B.1 resolved WITH 'no other material differences' cue; 60B.2 continued WITHOUT it, despite internally computing ~3.6 month payback Classification: C — NO SUFFICIENCY RULE + CONTINUATION BIAS Missing distinction: MATERIALITY / DECISION-RELEVANCE RULE --- docs/current-handoff.md | 7 +++ docs/experiment-60b3.md | 112 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 119 insertions(+) create mode 100644 docs/experiment-60b3.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 1c53b33..2f99a12 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -2664,3 +2664,10 @@ Tested whether the engine resolves the existing "Which option leaves us better o Tested whether the engine independently recognises decision sufficiency when both options have quantified material costs (£600k one-off relocation vs £2M/year stay-put) but the user does NOT say "there are no other material differences" or any equivalent stopping statement. Answer provided exact financial comparison with no explicit closure language. **Result:** Classification C — Generic Uncertainty Chasing. The engine preserved all three existing entity identities, set numeric newValue on each option (600000 on opt_relocate, 2000000 on opt_stay_put), performed an internal payback computation (~3.6 months), but treated the comparison as insufficient for decision resolution. Status changed unknown → supported (not resolved). No metric/evidence nodes created; no edges added; cost units not distinguished. Selected question "What outcome would demonstrate enough value to justify continuing?" is generic — does not identify a specific grounded missing factor. The engine demonstrated it can compute a meaningful financial comparison but does not use those computations as sufficiency triggers without an explicit user stopping cue. One live call at qwen-claude:latest on http://127.0.0.1:3000. No production code changed. + +## Experiment 60B.3 — Decision Sufficiency Rule Diagnosis (read-only) + +Tested whether the Confidence Engine can independently recognise when quantified option evidence is sufficient for decision resolution, or whether it requires explicit user cues to close a decision context. Inspection of all six primary files and two secondary files confirmed: the engine has NO independent sufficiency rule. Resolution depends on explicit user language ("no other material differences") which acts as an implicit closing signal. The validator checks structural integrity only. The schema has no materiality field. The propagateResolvedChildEvidence function handles within-decomposition sufficiency (all children resolved → parent resolves) but performs no cross-option comparison. + +**Diagnosis:** Classification C — NO SUFFICIENCY RULE + CONTINUATION BIAS +**Missing distinction:** MATERIALITY / DECISION-RELEVANCE RULE diff --git a/docs/experiment-60b3.md b/docs/experiment-60b3.md new file mode 100644 index 0000000..020ef1b --- /dev/null +++ b/docs/experiment-60b3.md @@ -0,0 +1,112 @@ +# Experiment 60B.3 — Decision Sufficiency Rule Diagnosis (read-only) + +**Branch:** `feature/decision-options-v0.25` +**Date:** 2026-08-13 +**Status:** Complete +**Type:** READ-ONLY DIAGNOSIS — Inspected code, prompt rules, schema, and experiment histories to determine whether the engine has an independent decision-sufficiency rule or depends on explicit user language. + +## Objective + +Determine whether the Confidence Engine can independently recognise when quantified option evidence is sufficient for decision resolution, or whether it requires explicit user cues (e.g., "no other material differences") to close a decision context. This experiment was designed as a zero-call diagnosis: inspect only named files and produce a comprehensive report with checkpoint answers plus documentation artifacts. + +## Inspection Scope + +Six primary files inspected in full: +1. `lib/graph/prompt-builder.js` — full 182 lines +2. `lib/graph/schema.js` — full 276 lines +3. `lib/graph/utils.js` — full 933 lines +4. `docs/experiment-60b1.md` — full 203 lines (live run WITH "no other material differences") +5. `docs/experiment-60b2.md` — full 207 lines (live run WITHOUT that phrase) +6. `docs/current-handoff.md` — first 875 of 2,667 lines + +Four additional files identified via grep and inspected: +7. `lib/graph/apply-proposal.js` — propagateResolvedChildEvidence logic +8. `lib/graph/orchestrator.js` — updateCaseWithDependencies pipeline + +Total: 8 files inspected. Zero production code changes. Zero live calls in this experiment. + +## Checkpoint Answers + +### Checkpoint 1 — Does the prompt include an explicit materiality or decision-sufficiency rule? + +**Answer: NO** + +Prompt-builder.js Rule 20 (line 123): +> "Return selectedQuestion as null only when no consequential unresolved unknown remains." + +This states *when* to return null but does NOT define what makes an unknown non-consequential. There is no materiality test, no evidence-count threshold, and no cross-option sufficiency comparison anywhere in the prompt's 32 rules or additional guidance sections. The term "consequential" appears once and is undefined. + +Prompt-builder.js Rule 5 (line 106): +> "Resolve the answered unknown first when the answer supports it." + +This refers only to the singular answered unknown — not to whether other unknowns remain consequential for the decision as a whole. No prompt rule contains: the words "materiality" or "materially", "sufficiency" or "sufficient", a test comparing option values, or a criterion for when evidence is enough to resolve a decision. + +### Checkpoint 2 — Does the validator independently judge sufficiency? + +**Answer: NO** + +From utils.js validateGraphUpdate (lines ~1-100+): +- Validates structuralActionRequired consistency with actual mutations +- Checks for duplicate node IDs +- Validates edge references to existing/new nodes +- Enforces 100KB input size limit +- Does NOT compare evidence between options +- Does NOT evaluate whether resolved nodes are sufficient to close a decision + +### Checkpoint 3 — Did apply-proposal evaluate sufficiency in 60B.1? + +**Answer: PARTIAL — Only within-decomposition, not across-option** + +From apply-proposal.js, propagateResolvedChildEvidence (lines 869-1049): +- computeParentProgressState at line 719 checks if ALL direct children of a parent unknown are resolved +- When `resolvedChildren.length === totalChildren`, it sets nextStatus: "resolved" for the parent +- This is a within-decomposition sufficiency rule (all sub-unknowns → parent resolves) +- There is NO cross-option comparison logic — no function that evaluates whether option evidence values are sufficient to close a decision node + +The engine's only automated sufficiency mechanism: "when all decomposition children of an unknown are resolved, the parent unknown resolves." This operates within a single chain of questions and answers, not across competing options. + +### Checkpoint 4 — Does schema have any materiality field? + +**Answer: NO** + +From schema.js: +- confidenceAssessmentSchema: evidenceConfidence, completenessStatus, conclusionConfidence — no materiality or couldChangeDecision +- SituationGraph: resolvedNodeIds array — no sufficiency metadata +- graphUpdateSchema: resolvedUnknownNodeIds — model proposes what to resolve but schema doesn't validate why +- confidenceAssessmentSchema.completenessStatus distinguishes empty/partial/complete locally, not globally across options + +### Checkpoint 5 — What explains the 60B.1 vs 60B.2 divergence? + +**Answer: The only difference is the presence of explicit user language ("no other material differences") which the model used as an implicit closing signal.** + +Both experiments shared identical starting graph (4 nodes, 2 edges), identical quantified comparison (£600k vs £2M/year), and identical model. The divergence was purely lexical: with "no other material differences" the engine resolved; without it, the engine defaulted to generic continuation — even though it internally computed a ~3.6 month payback and stated "relocation yields net savings." + +## Classification Choice + +**CHOSEN: C — NO SUFFICIENCY RULE + CONTINUATION BIAS** + +Evidence chain: +1. No independent sufficiency rule in prompt (Checkpoint 1: NO) +2. No validator-level sufficiency judgment (Checkpoint 2: NO) +3. No cross-option sufficiency in apply-proposal (Checkpoint 3: PARTIAL, within-decomposition only) +4. No materiality field in schema (Checkpoint 4: NO) +5. 60B.1 resolved WITH explicit cue; 60B.2 continued WITHOUT it (Checkpoint 5) + +The engine's continuation bias — defaulting to generating a question rather than proposing resolution when no explicit closing cue exists — is observable in both the prompt rules and live experiment results. The model can produce `resolved` status when given an explicit cue, but has no automated mechanism to reach that conclusion independently. + +## Missing Reasoning Distinction + +**CHOSEN: B — MATERIALITY / DECISION-RELEVANCE RULE** + +The minimal missing reasoning distinction that fixes the 60B.1 vs 60B.2 divergence is a materiality assessment rule enabling independent evaluation of which unresolved unknowns are decision-relevant versus non-material, without requiring explicit user language. Implementation options: +- New answerMeaning.resolutionGuidance value (e.g., "no_material_remaining") +- Prompt rule explaining how to assess whether option evidence constitutes sufficient comparison +- Validator-level check that when both options have quantified values, remaining unknowns should be assessed for materiality + +## Production code changed: NO +## Prompt changed: NO +## Validator changed: NO +## Harness changed: NO +## Vitest run: NO +## Ollama calls: 0 +## Direct API calls: 0