robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: diagnose decision sufficiency rule

Browse Source 
Read-only inspection of 8 files (prompt-builder.js, schema.js, utils.js,
apply-proposal.js, orchestrator.js, experiment-60b1.md, experiment-60b2.md,
current-handoff.md). No code changes.

Key findings:
- Prompt has no independent materiality/sufficiency rule (Rule 20 says null
  selectedQuestion when 'no consequential unresolved unknown' but doesn't define
  what makes an unknown non-consequential)
- Validator performs structural checks only, no evidence sufficiency evaluation
- No cross-option comparison logic in propagateResolvedChildEvidence
- Schema has no materiality or couldChangeDecision field
- 60B.1 resolved WITH 'no other material differences' cue; 60B.2 continued
  WITHOUT it, despite internally computing ~3.6 month payback

Classification: C — NO SUFFICIENCY RULE + CONTINUATION BIAS
Missing distinction: MATERIALITY / DECISION-RELEVANCE RULE
This commit is contained in:
robbond
2026-08-13 07:15:01 +01:00
parent 306f8392a1
commit 5dcaed39df
2 changed files with 119 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+7
View File
@@ -2664,3 +2664,10 @@ Tested whether the engine resolves the existing "Which option leaves us better o
Tested whether the engine independently recognises decision sufficiency when both options have quantified material costs (£600k one-off relocation vs £2M/year stay-put) but the user does NOT say "there are no other material differences" or any equivalent stopping statement. Answer provided exact financial comparison with no explicit closure language.
**Result:** Classification C — Generic Uncertainty Chasing. The engine preserved all three existing entity identities, set numeric newValue on each option (600000 on opt_relocate, 2000000 on opt_stay_put), performed an internal payback computation (~3.6 months), but treated the comparison as insufficient for decision resolution. Status changed unknown → supported (not resolved). No metric/evidence nodes created; no edges added; cost units not distinguished. Selected question "What outcome would demonstrate enough value to justify continuing?" is generic — does not identify a specific grounded missing factor. The engine demonstrated it can compute a meaningful financial comparison but does not use those computations as sufficiency triggers without an explicit user stopping cue. One live call at qwen-claude:latest on http://127.0.0.1:3000. No production code changed.
## Experiment 60B.3 — Decision Sufficiency Rule Diagnosis (read-only)
Tested whether the Confidence Engine can independently recognise when quantified option evidence is sufficient for decision resolution, or whether it requires explicit user cues to close a decision context. Inspection of all six primary files and two secondary files confirmed: the engine has NO independent sufficiency rule. Resolution depends on explicit user language ("no other material differences") which acts as an implicit closing signal. The validator checks structural integrity only. The schema has no materiality field. The propagateResolvedChildEvidence function handles within-decomposition sufficiency (all children resolved → parent resolves) but performs no cross-option comparison.
**Diagnosis:** Classification C — NO SUFFICIENCY RULE + CONTINUATION BIAS
**Missing distinction:** MATERIALITY / DECISION-RELEVANCE RULE