docs: record decision materiality rule
This commit is contained in:
@@ -2671,3 +2671,7 @@ Tested whether the Confidence Engine can independently recognise when quantified
|
|||||||
|
|
||||||
**Diagnosis:** Classification C — NO SUFFICIENCY RULE + CONTINUATION BIAS
|
**Diagnosis:** Classification C — NO SUFFICIENCY RULE + CONTINUATION BIAS
|
||||||
**Missing distinction:** MATERIALITY / DECISION-RELEVANCE RULE
|
**Missing distinction:** MATERIALITY / DECISION-RELEVANCE RULE
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
Experiment 60B.4 implemented the minimal correction: added a domain-general decision-sufficiency rule to the prompt (not schema, not validator). The rule instructs: uncertainty alone is not sufficient reason to continue; continuation requires a specific material factor that could change the preferred option; when no such factor remains, resolve the existing decision context rather than asking a generic question. Prompt-level addition only — three sentences in a new "Decision Sufficiency Rule" section in prompt-builder.js. Deterministic tests confirm all five positive and six negative criteria pass (86/86). No live regression performed. Status pending Rob's review.
|
||||||
|
|||||||
@@ -0,0 +1,68 @@
|
|||||||
|
# Experiment 60B.4 — Decision Materiality Rule (prompt-only)
|
||||||
|
|
||||||
|
**Branch:** `feature/decision-sufficiency-v0.26`
|
||||||
|
**Date:** 2026-08-13
|
||||||
|
**Status:** Complete
|
||||||
|
**Type:** PROMPT-ONLY — Bounded instruction addition plus deterministic prompt tests. No live model calls.
|
||||||
|
|
||||||
|
## Reasoning gap from 60B.3
|
||||||
|
|
||||||
|
60B.3 confirmed the engine has no independent decision-sufficiency rule. Rule 20 states:
|
||||||
|
|
||||||
|
> "Return selectedQuestion as null only when no consequential unresolved unknown remains."
|
||||||
|
|
||||||
|
This defines *when* to return null but does not define what makes an unknown non-consequential. The term "consequential" is undefined at decision level. Live results (60B.1 vs 60B.2) show the model defaults to generic continuation when no explicit closing cue exists — even when option evidence is quantified on both sides.
|
||||||
|
|
||||||
|
The gap: **uncertainty remains** is always true during investigation. The prompt does not instruct the model to distinguish this from **remaining uncertainty could materially change which option is preferred**.
|
||||||
|
|
||||||
|
## New prompt rule
|
||||||
|
|
||||||
|
Added section "Decision Sufficiency Rule" between Proposal Rules and Decision Option Structure Rules in `lib/graph/prompt-builder.js`:
|
||||||
|
|
||||||
|
> An unresolved decision between options should not remain open merely because some uncertainty still exists.
|
||||||
|
>
|
||||||
|
> Keep a decision context unresolved only when you can identify a specific unresolved factor that could materially change which option is preferred.
|
||||||
|
>
|
||||||
|
> If the currently supported evidence is sufficient to distinguish the options and no such material unresolved factor remains, resolve the existing decision context and do not ask a generic continuation question.
|
||||||
|
|
||||||
|
## Why this is domain-general
|
||||||
|
|
||||||
|
- No financial vocabulary (no £, $, payback, cost comparison)
|
||||||
|
- No relocation or industry-specific terms
|
||||||
|
- No numeric thresholds or calculation frameworks
|
||||||
|
- No keyword-based routing or taxonomic classification
|
||||||
|
- The three semantics apply to any decision between options with competing evidence:
|
||||||
|
1. uncertainty alone is not sufficient reason to continue
|
||||||
|
2. continuation requires a specific material factor that could change the preferred option
|
||||||
|
3. when no such factor remains, resolve the existing decision context rather than asking a generic question
|
||||||
|
|
||||||
|
## Focused test results
|
||||||
|
|
||||||
|
**Command:** `npx vitest run tests/graph/prompt-builder.test.js`
|
||||||
|
**Result:** PASS (86/86)
|
||||||
|
|
||||||
|
New materiality tests verify:
|
||||||
|
- Positive: uncertainty-alone-is-not-enough, specific-material-factor-required, could-change-preferred-criterion, resolve-when-no-material-factor, generic-continuation-discouraged, Rule-20-preserved, domain-generality
|
||||||
|
- Negative: no financial thresholds, no currency examples, no relocation examples, no automatic-resolution-when-better-looking, no new schema fields, no new node kinds
|
||||||
|
|
||||||
|
## What remains unproven until live regression
|
||||||
|
|
||||||
|
1. **Stability** — deterministic prompt tests confirm the instruction text is present and well-formed, but do not verify the model follows it consistently across repeated runs
|
||||||
|
2. **Cross-domain generalisation** — single-prompt-test coverage does not prove the rule works outside the test cases' structural patterns
|
||||||
|
3. **Interaction with existing rules** — no regression test confirms the materiality rule does not interfere with Rule 20, option structure rules, or the semantic-to-mutation contract
|
||||||
|
4. **Edge cases** — decisions where multiple partially-material factors exist; decisions with equal evidence across options; ambiguous factor specificity
|
||||||
|
|
||||||
|
## Production changes
|
||||||
|
|
||||||
|
| File | Change |
|
||||||
|
|------|--------|
|
||||||
|
| `lib/graph/prompt-builder.js` | Added "Decision Sufficiency Rule" section (3 sentences, 3 semantics) |
|
||||||
|
| No schema changes | |
|
||||||
|
| No validator changes | |
|
||||||
|
| No question-selection code changes | |
|
||||||
|
| No provider integration changes | |
|
||||||
|
|
||||||
|
## Commit messages
|
||||||
|
|
||||||
|
Production/tests: `feat(reasoning): add decision materiality rule`
|
||||||
|
Documentation: `docs: record decision materiality rule`
|
||||||
Reference in New Issue
Block a user