confidence-engine/docs/experiment-60b3.md

Experiment 60B.3 — Decision Sufficiency Rule Diagnosis (read-only)

Branch: feature/decision-options-v0.25
Date: 2026-08-13
Status: Complete
Type: READ-ONLY DIAGNOSIS — Inspected code, prompt rules, schema, and experiment histories to determine whether the engine has an independent decision-sufficiency rule or depends on explicit user language.

Objective

Determine whether the Confidence Engine can independently recognise when quantified option evidence is sufficient for decision resolution, or whether it requires explicit user cues (e.g., "no other material differences") to close a decision context. This experiment was designed as a zero-call diagnosis: inspect only named files and produce a comprehensive report with checkpoint answers plus documentation artifacts.

Inspection Scope

Six primary files inspected in full:

1. lib/graph/prompt-builder.js — full 182 lines
2. lib/graph/schema.js — full 276 lines
3. lib/graph/utils.js — full 933 lines
4. docs/experiment-60b1.md — full 203 lines (live run WITH "no other material differences")
5. docs/experiment-60b2.md — full 207 lines (live run WITHOUT that phrase)
6. docs/current-handoff.md — first 875 of 2,667 lines

Four additional files identified via grep and inspected: 7. lib/graph/apply-proposal.js — propagateResolvedChildEvidence logic 8. lib/graph/orchestrator.js — updateCaseWithDependencies pipeline

Total: 8 files inspected. Zero production code changes. Zero live calls in this experiment.

Checkpoint Answers

Checkpoint 1 — Does the prompt include an explicit materiality or decision-sufficiency rule?

Answer: NO

Prompt-builder.js Rule 20 (line 123):

"Return selectedQuestion as null only when no consequential unresolved unknown remains."

This states when to return null but does NOT define what makes an unknown non-consequential. There is no materiality test, no evidence-count threshold, and no cross-option sufficiency comparison anywhere in the prompt's 32 rules or additional guidance sections. The term "consequential" appears once and is undefined.

Prompt-builder.js Rule 5 (line 106):

"Resolve the answered unknown first when the answer supports it."

This refers only to the singular answered unknown — not to whether other unknowns remain consequential for the decision as a whole. No prompt rule contains: the words "materiality" or "materially", "sufficiency" or "sufficient", a test comparing option values, or a criterion for when evidence is enough to resolve a decision.

Checkpoint 2 — Does the validator independently judge sufficiency?

Answer: NO

From utils.js validateGraphUpdate (lines ~1-100+):

• Validates structuralActionRequired consistency with actual mutations
• Checks for duplicate node IDs
• Validates edge references to existing/new nodes
• Enforces 100KB input size limit
• Does NOT compare evidence between options
• Does NOT evaluate whether resolved nodes are sufficient to close a decision

Checkpoint 3 — Did apply-proposal evaluate sufficiency in 60B.1?

Answer: PARTIAL — Only within-decomposition, not across-option

From apply-proposal.js, propagateResolvedChildEvidence (lines 869-1049):

• computeParentProgressState at line 719 checks if ALL direct children of a parent unknown are resolved
• When resolvedChildren.length === totalChildren, it sets nextStatus: "resolved" for the parent
• This is a within-decomposition sufficiency rule (all sub-unknowns → parent resolves)
• There is NO cross-option comparison logic — no function that evaluates whether option evidence values are sufficient to close a decision node

The engine's only automated sufficiency mechanism: "when all decomposition children of an unknown are resolved, the parent unknown resolves." This operates within a single chain of questions and answers, not across competing options.

Checkpoint 4 — Does schema have any materiality field?

Answer: NO

From schema.js:

• confidenceAssessmentSchema: evidenceConfidence, completenessStatus, conclusionConfidence — no materiality or couldChangeDecision
• SituationGraph: resolvedNodeIds array — no sufficiency metadata
• graphUpdateSchema: resolvedUnknownNodeIds — model proposes what to resolve but schema doesn't validate why
• confidenceAssessmentSchema.completenessStatus distinguishes empty/partial/complete locally, not globally across options

Checkpoint 5 — What explains the 60B.1 vs 60B.2 divergence?

Answer: The only difference is the presence of explicit user language ("no other material differences") which the model used as an implicit closing signal.

Both experiments shared identical starting graph (4 nodes, 2 edges), identical quantified comparison (£600k vs £2M/year), and identical model. The divergence was purely lexical: with "no other material differences" the engine resolved; without it, the engine defaulted to generic continuation — even though it internally computed a ~3.6 month payback and stated "relocation yields net savings."

Classification Choice

CHOSEN: C — NO SUFFICIENCY RULE + CONTINUATION BIAS

Evidence chain:

1. No independent sufficiency rule in prompt (Checkpoint 1: NO)
2. No validator-level sufficiency judgment (Checkpoint 2: NO)
3. No cross-option sufficiency in apply-proposal (Checkpoint 3: PARTIAL, within-decomposition only)
4. No materiality field in schema (Checkpoint 4: NO)
5. 60B.1 resolved WITH explicit cue; 60B.2 continued WITHOUT it (Checkpoint 5)

The engine's continuation bias — defaulting to generating a question rather than proposing resolution when no explicit closing cue exists — is observable in both the prompt rules and live experiment results. The model can produce resolved status when given an explicit cue, but has no automated mechanism to reach that conclusion independently.

Missing Reasoning Distinction

CHOSEN: B — MATERIALITY / DECISION-RELEVANCE RULE

The minimal missing reasoning distinction that fixes the 60B.1 vs 60B.2 divergence is a materiality assessment rule enabling independent evaluation of which unresolved unknowns are decision-relevant versus non-material, without requiring explicit user language. Implementation options:

• New answerMeaning.resolutionGuidance value (e.g., "no_material_remaining")
• Prompt rule explaining how to assess whether option evidence constitutes sufficient comparison
• Validator-level check that when both options have quantified values, remaining unknowns should be assessed for materiality

Production code changed: NO

Prompt changed: NO

Validator changed: NO

Harness changed: NO

Vitest run: NO

Ollama calls: 0

Direct API calls: 0