From b671681ddc701a54c920e34c99a4b6077de4a399 Mon Sep 17 00:00:00 2001 From: robbond Date: Thu, 13 Aug 2026 07:24:18 +0100 Subject: [PATCH] docs: record decision materiality rule --- docs/current-handoff.md | 4 +++ docs/experiment-60b4.md | 68 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+) create mode 100644 docs/experiment-60b4.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 2f99a12..2b23b64 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -2671,3 +2671,7 @@ Tested whether the Confidence Engine can independently recognise when quantified **Diagnosis:** Classification C — NO SUFFICIENCY RULE + CONTINUATION BIAS **Missing distinction:** MATERIALITY / DECISION-RELEVANCE RULE + +--- + +Experiment 60B.4 implemented the minimal correction: added a domain-general decision-sufficiency rule to the prompt (not schema, not validator). The rule instructs: uncertainty alone is not sufficient reason to continue; continuation requires a specific material factor that could change the preferred option; when no such factor remains, resolve the existing decision context rather than asking a generic question. Prompt-level addition only — three sentences in a new "Decision Sufficiency Rule" section in prompt-builder.js. Deterministic tests confirm all five positive and six negative criteria pass (86/86). No live regression performed. Status pending Rob's review. diff --git a/docs/experiment-60b4.md b/docs/experiment-60b4.md new file mode 100644 index 0000000..d7d6f6e --- /dev/null +++ b/docs/experiment-60b4.md @@ -0,0 +1,68 @@ +# Experiment 60B.4 — Decision Materiality Rule (prompt-only) + +**Branch:** `feature/decision-sufficiency-v0.26` +**Date:** 2026-08-13 +**Status:** Complete +**Type:** PROMPT-ONLY — Bounded instruction addition plus deterministic prompt tests. No live model calls. + +## Reasoning gap from 60B.3 + +60B.3 confirmed the engine has no independent decision-sufficiency rule. Rule 20 states: + +> "Return selectedQuestion as null only when no consequential unresolved unknown remains." + +This defines *when* to return null but does not define what makes an unknown non-consequential. The term "consequential" is undefined at decision level. Live results (60B.1 vs 60B.2) show the model defaults to generic continuation when no explicit closing cue exists — even when option evidence is quantified on both sides. + +The gap: **uncertainty remains** is always true during investigation. The prompt does not instruct the model to distinguish this from **remaining uncertainty could materially change which option is preferred**. + +## New prompt rule + +Added section "Decision Sufficiency Rule" between Proposal Rules and Decision Option Structure Rules in `lib/graph/prompt-builder.js`: + +> An unresolved decision between options should not remain open merely because some uncertainty still exists. +> +> Keep a decision context unresolved only when you can identify a specific unresolved factor that could materially change which option is preferred. +> +> If the currently supported evidence is sufficient to distinguish the options and no such material unresolved factor remains, resolve the existing decision context and do not ask a generic continuation question. + +## Why this is domain-general + +- No financial vocabulary (no £, $, payback, cost comparison) +- No relocation or industry-specific terms +- No numeric thresholds or calculation frameworks +- No keyword-based routing or taxonomic classification +- The three semantics apply to any decision between options with competing evidence: + 1. uncertainty alone is not sufficient reason to continue + 2. continuation requires a specific material factor that could change the preferred option + 3. when no such factor remains, resolve the existing decision context rather than asking a generic question + +## Focused test results + +**Command:** `npx vitest run tests/graph/prompt-builder.test.js` +**Result:** PASS (86/86) + +New materiality tests verify: +- Positive: uncertainty-alone-is-not-enough, specific-material-factor-required, could-change-preferred-criterion, resolve-when-no-material-factor, generic-continuation-discouraged, Rule-20-preserved, domain-generality +- Negative: no financial thresholds, no currency examples, no relocation examples, no automatic-resolution-when-better-looking, no new schema fields, no new node kinds + +## What remains unproven until live regression + +1. **Stability** — deterministic prompt tests confirm the instruction text is present and well-formed, but do not verify the model follows it consistently across repeated runs +2. **Cross-domain generalisation** — single-prompt-test coverage does not prove the rule works outside the test cases' structural patterns +3. **Interaction with existing rules** — no regression test confirms the materiality rule does not interfere with Rule 20, option structure rules, or the semantic-to-mutation contract +4. **Edge cases** — decisions where multiple partially-material factors exist; decisions with equal evidence across options; ambiguous factor specificity + +## Production changes + +| File | Change | +|------|--------| +| `lib/graph/prompt-builder.js` | Added "Decision Sufficiency Rule" section (3 sentences, 3 semantics) | +| No schema changes | | +| No validator changes | | +| No question-selection code changes | | +| No provider integration changes | | + +## Commit messages + +Production/tests: `feat(reasoning): add decision materiality rule` +Documentation: `docs: record decision materiality rule`