robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: define decision closure ownership policy

Browse Source
This commit is contained in:
robbond
2026-08-15 06:21:20 +01:00
parent d49e3e8e83
commit 0e5292c46d
2 changed files with 419 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+344
View File
@@ -3735,3 +3735,347 @@ Watchman used: NO
Documentation updated: docs/experiment-60b77.md created; docs/current-handoff.md appended
Git status: will be clean after commit
---
#### Experiment 60B.78 — Decision Closure Ownership Policy (READ-ONLY DESIGN)
**Date:** 2026-08-15
**Branch:** `feature/sufficiency-decision-detection-v0.46`
**Preceded by:** Experiment 60B.77 (model-led closure proven by elimination)
**Type:** Design analysis — no implementation
## Objective
Answer: **When hasRemainingMaterialFactors(decision) === false AND isUserConfirmationOfNoRemainingUncertainty(answer) === false, should model-led parent closure be allowed, or does the deterministic sufficiency policy own whether a decision may close?**
Do not implement anything. Choose the correct ownership policy.
---
## Checkpoint 1 — Current Prompt Ownership
**Source:** `lib/graph/prompt-builder.js` lines 137–143
```
## Decision Sufficiency Rule
An unresolved decision between options should not remain open merely because some uncertainty still exists.
Keep a decision context unresolved only when you can identify a specific unresolved factor that could materially change which option is preferred.
If the currently supported evidence is sufficient to distinguish the options and no such material unresolved factor remains, resolve the existing decision context and do not ask a generic continuation question.
```
**Does the prompt explicitly require user confirmation before parent closure:**
**PARTIAL** — Rule 143 says "no such material unresolved factor remains" but does not state that *user-confirmed* absence is required. It conflates "model judges no represented factor remains" with "user confirms nothing else material."
**Does the prompt allow evidence sufficiency alone to justify parent closure:**
**YES** — Rule 143 says: "If the currently supported evidence is sufficient to distinguish the options and no such material unresolved factor remains, resolve the existing decision context." The phrase "no such material unresolved factor remains" refers to *represented* factors, not user-confirmed absence of hidden ones.
**Does the prompt distinguish "all represented factors resolved" from "user confirmed nothing else material remains":**
**NO** — Rule 143 uses a single criterion ("evidence is sufficient + no material unresolved factor") with no bifurcation between model-judged sufficiency and user-confirmed sufficiency.
---
## Checkpoint 2 — Deterministic Ownership
**Source:** `lib/graph/decision-sufficiency.js` line 224–231
```js
export function shouldCloseDecision({ decisionNodeId, graph, answer, pendingResolvedIds }) {
const remaining = countRemainingMaterialFactors(decisionNodeId, graph, pendingResolvedIds);
return remaining === 0 && isUserConfirmationOfNoRemainingUncertainty(answer);
}
```
**Source:** `lib/graph/apply-proposal.js` lines 3808–3851 — the deterministic closure gate fires for every decision with contained_in edges when shouldCloseDecision returns true. It unconditionally sets status = "resolved" and appends to resolvedUnknownNodeIds. There is no fallback path or model-override mechanism.
**Source:** `lib/graph/decision-sufficiency.js` lines 16–27 — CONFIRMATION_PHRASES includes:
```
"no other material uncertainty remains", "no further material uncertainty remains", etc.
```
**Source:** `lib/graph/decision-sufficiency.js` lines 29–32 — CONFIRMATION_PATTERNS adds regex variants.
**Source:** `lib/graph/decision-sufficiency.js` lines 10–14 — CONTRADICTION_PHRASES rejects "still ... material", "am not saying", etc.
**Choice: B — deterministic closure defines the authoritative closure policy**
**Why:** The code implements a strict two-part AND predicate (zero factors AND explicit confirmation). There is no "model may close if it judges sufficient" override. The gate unconditionally sets status = "resolved" when both conditions are met. This is an affirmative definition of *when* closure occurs, not a fallback check.
---
## Checkpoint 3 — Model-Led Closure Risk
**Scenario:** Last represented factor resolves → user does NOT say confirmation phrase → model resolves decision anyway.
**Premature-closure risk: HIGH**
**Why:** The deterministic gate requires explicit user confirmation phrases that are absent from the answer. If the model independently closes without that gate, the user's actual epistemic state ("I'm not sure nothing else matters") is ignored. This directly contradicts the conservative false-open-over-false-closed philosophy encoded in CONFIRMATION_PHRASES + CONTRADICTION_PHRASES.
**Could the user still hold an unrepresented material uncertainty: POSSIBLE** — "unrepresented" means it may not appear in hasRemainingMaterialFactors because it lacks graph structure, but the model's judgment of sufficiency could miss it. The whole purpose of explicit confirmation is to catch cases where the user has context the model does not.
**Would model-led closure bypass the conservative false-open-over-false-closed policy: YES** — The policy (encoded in the bounded confirmation phrases) exists precisely to require the extra step of asking "is anything else material?" when the raw answer doesn't self-confirm sufficiency. Model-led closure removes that step entirely.
---
## Checkpoint 4 — Model-Led Closure Benefit
If evidence clearly distinguishes options and no represented factor remains, model-led closure may reduce an unnecessary confirmation turn in cases where:
- The user's answer is so unambiguous that any reasonable person would agree nothing else matters
- The graph structure exhaustively represents the decision domain
- Both parties (model and user) share complete context
**Benefit: MEDIUM**
**Could explicit confirmation become redundant in obvious cases: YES** — In scenarios like "customer confirmed they won't sign if we launch" combined with all other factors already resolved, a reasonable user might think "that's obviously the last thing that matters." However, the system cannot reliably distinguish these from cases where the user is genuinely uncertain about other factors.
**Would requiring confirmation impose unnecessary user friction: LOW to MEDIUM** — Depends on context. In simple two-option decisions with clear resolution, asking "is anything else material?" is a single short turn but adds cognitive overhead for an answer that may already be obvious. However, the cost of missing unrepresented uncertainty is asymmetric (false closure is worse than unnecessary questioning).
---
## Checkpoint 5 — Candidate Ownership Policies
### Policy A — MODEL MAY CLOSE
Allow model to resolve whenever it judges evidence sufficient. Deterministic closure remains fallback/normalisation.
```
Preserves user meaning: LOW — model may close when user hasn't confirmed nothing else matters
Premature-closure risk: HIGH — no deterministic guard on model proposals
Unnecessary-confirmation risk: MEDIUM — eliminates confirmation entirely in model-led path
Model-compliance dependence: HIGH — relies entirely on stochastic model behavior
Deterministic clarity: LOW — two independent closure paths with unclear precedence
Schema change: NO
Prompt change: YES — would need to remove evidence-sufficiency language from prompt
Apply-proposal enforcement: MAYBE — would need to strip deterministic gate or make it conditional
Principal weakness: Undermines the entire conservative confirmation policy established by CONFIRMATION_PHRASES + CONTRADICTION_PHRASES. The model's judgment is already proven insufficient (60B.77 proved model closed when deterministic gate did not fire).
```
### Policy B — DETERMINISTIC CONFIRMATION OWNS CLOSURE
If explicit raw confirmation is absent: model-proposed parent resolution must NOT become terminal. State B question fires instead.
```
Preserves user meaning: HIGH — requires explicit user expression before closure
Premature-closure risk: LOW — deterministic AND gate prevents all false closures
Unnecessary-confirmation risk: LOW — only triggers when user hasn't self-confirmed; the confirmation phrase is short and contextually anchored
Model-compliance dependence: LOW — enforcement is deterministic, not prompt-dependent
Deterministic clarity: HIGH — single authoritative path
Schema change: NO
Prompt change: YES — would need to remove rule 143 or reword it to match deterministic gate
Apply-proposal enforcement: MAYBE — reconciliation layer may strip model proposals when they lack confirmation
Principal weakness: May ask unnecessary confirmation turns in obvious cases where user clearly intends nothing else matters. However, this is the conservative default cost.
```
### Policy C — MODEL MAY EXPRESS DIRECTION BUT NOT TERMINAL CLOSURE
Model may express "preferred option", "reasoning direction", "evidence suggests decision" but parent status remains unresolved until deterministic sufficiency confirmation is satisfied.
```
Preserves user meaning: HIGH — closure still gated by explicit confirmation
Premature-closure risk: LOW — model cannot terminate investigation without confirmation
Unnecessary-confirmation risk: LOW — same as Policy B for the confirmation question itself
Model-compliance dependence: MEDIUM — model must comply with not closing; needs prompt constraint
Deterministic clarity: HIGH — one path to closure, model can only recommend direction
Schema change: NO
Prompt change: YES — would need explicit prohibition on terminal closure in prompt
Apply-proposal enforcement: YES — must reject or strip model proposals that set parent status=resolved without confirmation
Principal weakness: Model compliance is still partly stochastic. The model may still propose terminal closure and rely on post-hoc stripping (less clean than validation rejection).
```
### Policy D — TWO LEVELS OF CLOSURE
Model can mark decision `known` while deterministic confirmation required for `resolved`. Use only if current status semantics support this distinction.
```
Preserves user meaning: MEDIUM — "known" conveys model direction without terminal commitment
Premature-closure risk: MEDIUM — "known" is a terminal status (TERMINAL_STATUSES includes "known")
Unnecessary-confirmation risk: LOW — same confirmation requirement as Policy B
Model-compliance dependence: LOW — deterministic gate still required for final resolution
Deterministic clarity: MEDIUM — introduces ambiguity about what "known" means for investigation continuation
Schema change: NO (semantics already exist in SituationStatus)
Prompt change: YES — would need to redefine known/resolved distinction in prompt
Apply-proposal enforcement: MAYBE — if "known" is treated as terminal by propagation/completeness logic, it may prevent investigation continuation
Principal weakness: "known" currently means "model has determined the value/direction" (used for resolved factors). Using it for pending decisions creates semantic confusion. TERMINAL_STATUSES includes "known" meaning it would be treated as investigated-completed in many code paths. This is not a clean distinction without broader changes.
```
### Policy E — MODEL CLOSURE ALLOWED ONLY WITH USER-SUPPORTED CONFIRMATION
Model may close only when its proposal's closure is also supported by explicit confirmation detectable from raw answer. Differs from B only in enforcement location.
```
Preserves user meaning: HIGH — same as B on outcome
Premature-closure risk: LOW — same as B on outcome
Unnecessary-confirmation risk: LOW — same as B
Model-compliance dependence: MEDIUM — relies on model checking confirmation before proposing closure
Deterministic clarity: MEDIUM — two paths (model-proposed-with-confirmation OR deterministic) both lead to same result
Schema change: NO
Prompt change: YES — would need to require model to check for confirmation phrases
Apply-proposal enforcement: MAYBE — validation layer could reject model proposals lacking confirmation when closure is proposed
Principal weakness: Adds complexity of two equivalent closure paths without clear advantage over pure deterministic gate. The prompt-level check duplicates the deterministic check.
```
---
## Checkpoint 6 — Status Semantics
**Source:** `lib/graph/schema.js` lines 24–31
```js
export const SituationStatus = {
known: "known", // directional value determined
unknown: "unknown", // needs investigation
provisional: "provisional",
supported: "supported",
weakened: "weakened",
contradicted: "contradicted",
resolved: "resolved", // investigation complete, nothing to investigate further
};
```
**TERMINAL_STATUSES (decision-sufficiency.js line 8):** `["known", "resolved", "contradicted"]`
**Does "known" mean a decision has a directional answer: YES** — "known" is used for factors where the value/direction is determined (e.g., customer confirmed X). For decisions, it would mean model has judged direction but not necessarily that investigation is complete.
**Does "resolved" mean investigation is complete: YES** — `resolved` means the decision's value is determined AND no further investigation is needed on this decision. It is the terminal closure state.
**Is there a legitimate existing distinction supporting Policy D: NO** — Both "known" and "resolved" appear in TERMINAL_STATUSES, meaning propagation/completeness logic treats both as investigated-complete. The schema does not distinguish between "model judged direction" and "user confirmed nothing else remains." Using "known" for pending decisions would incorrectly signal to downstream code that investigation is complete.
---
## Checkpoint 7 — State B Compatibility
**Policy A (Model may close):**
State B reachable: **STOCHASTIC** — depends on whether the model decides not to close. No deterministic guarantee it fires.
**Policy B (Deterministic confirmation owns closure):**
State B reachable: **YES** — deterministic gate does not fire without confirmation → decision remains open → activeUnknownNodeId is non-null → State B questioning fires.
**Policy C (Model may express direction, not terminal):**
State B reachable: **YES** — decision status remains "unknown" until deterministic confirmation → State B fires.
**Policy D (Known vs resolved):**
State B reachable: **DEPENDS** — if "known" is treated as terminal by propagation/completeness code, decision may effectively close before State B. If propagation respects the distinction, State B remains reachable. Currently UNPROVEN because TERMINAL_STATUSES includes "known".
**Policy E (Model closure only with confirmation):**
State B reachable: **YES** — without confirmation, deterministic gate doesn't fire and model can't close → decision stays open → State B fires.
---
## Checkpoint 8 — Raw Proposal Enforcement Boundary
### Boundary A — Prompt Only
Tell model not to close without explicit confirmation.
```
Deterministic guarantee: LOW — relies on stochastic model compliance
Model dependence: HIGH — model must consistently follow the rule
Semantic cleanliness: MEDIUM — prompt is clear but enforcement is implicit
Retry/rejection risk: MEDIUM — model may still propose closure, requiring downstream correction
Principal weakness: Same as the current problem: model already proposes premature closure. Adding a prompt rule without enforcement adds noise, not protection.
```
### Boundary B — Reconciliation Normalisation
If parent terminal transition lacks explicit confirmation, remove/rewrite that transition before validation.
```
Deterministic guarantee: HIGH — enforced at reconciliation layer before any mutation
Model dependence: LOW — deterministic stripping of invalid entries
Semantic cleanliness: MEDIUM — post-hoc correction is less clean than rejection but effective
Retry/rejection risk: LOW — silently corrected, no retry needed
Principal weakness: Opaque to the model. The model gets a mutated proposal it didn't propose without explanation, which may confuse future proposals.
```
### Boundary C — Proposal Compatibility Validation
Reject a model proposal that terminally closes a decision without explicit confirmation.
```
Deterministic guarantee: HIGH — validation error prevents invalid mutation
Model dependence: LOW — deterministic rejection of invalid proposals
Semantic cleanliness: HIGH — clean boundary: invalid proposals are rejected before application
Retry/rejection risk: MEDIUM — requires model to retry with different proposal; could loop if model doesn't understand why it was rejected. Mitigation: include explicit reason "explicit confirmation required before decision closure."
Principal weakness: Potential infinite loop if model keeps proposing the same closure. Must handle gracefully (e.g., degrade to recommendation-only for this turn).
```
### Boundary D — Post-Mutation Correction
Allow mutation then reopen/normalise decision.
```
Deterministic guarantee: MEDIUM — works but is reactive rather than proactive
Model dependence: LOW — deterministic reopening
Semantic cleanliness: LOW — allows invalid state briefly before correction
Retry/rejection risk: LOW — already applied, just corrected
Principal weakness: Allows brief invalid state to exist in proposalSnapshot. The model's reasoning about why it closed could be lost. Less clean than pre-mutation enforcement.
```
---
## CRITICAL DISTINCTION
**Choice: B — EXPLICIT CONFIRMATION SHOULD AUTHORITATIVELY GATE CLOSURE**
**Why:**
1. The conservative false-open-over-false-closed philosophy is encoded in the bounded confirmation phrases. This policy must not be overridable by model judgment.
2. 60B.77 proved the model closes decisions deterministically when it resolves its last factor, regardless of whether user confirmed nothing else matters. Allowing this would make the confirmation gate meaningless.
3. "Unrepresented material uncertainty" is POSSIBLE — the model's represented-factor analysis cannot capture what the user knows that isn't in the graph.
4. State B exists specifically to handle the gap between model-judged sufficiency and user-confirmed sufficiency. Removing that gap by allowing model-led closure makes State B unreachable in the critical case it was designed for.
5. The asymmetric cost profile: false closure (user had unrepresented uncertainty) is far worse than unnecessary confirmation questioning.
---
## MINIMUM CORRECTIVE BOUNDARY
**Choice: E — PROMPT CLARIFICATION + DETERMINISTIC ENFORCEMENT**
**Why:**
The prompt rule 143 currently permits evidence-only closure ("If the currently supported evidence is sufficient... resolve the decision"). This directly conflicts with the deterministic gate's requirement for explicit confirmation. The fix has two parts:
1. **Prompt clarification (B):** Reword rule 143 to require explicit user confirmation matching the deterministic gate's criteria before the model may propose parent closure. This aligns model intent with deterministic policy.
2. **Deterministic enforcement (C):** Apply a narrow validation check in apply-proposal.js that strips/rejects model proposals attempting terminal closure of unknown nodes without matching isUserConfirmationOfNoRemainingUncertainty(answer). This prevents stochastic model compliance from being the only barrier.
**Would State B remain reachable when confirmation absent: YES** — Deterministic gate does not fire → decision stays open → activeUnknownNodeId non-null → State B fires.
**Would model reasoning/direction remain preservable: YES** — Model can still express preferred option, reasoning direction, and evidence sufficiency judgment in the proposal without setting status=resolved. Only terminal closure is gated.
**Would premature terminal closure be prevented: YES** — Deterministic validation prevents model from setting parent to resolved/known without confirmation. Even if prompt is ignored, enforcement blocks it.
**Would schema remain unchanged: YES** — No new statuses or fields needed. Current SituationStatus values suffice.
---
## IMPLEMENTATION READINESS
**Choice: A — READY FOR BOUNDED IMPLEMENTATION**
If B, one unresolved question: None remaining — the analysis is complete. The two-part fix (prompt + enforcement) is well-scoped and doesn't require further design iteration.
Smallest implementation boundary:
1. Reword prompt-builder.js rule 143 to align with deterministic gate criteria (explicit confirmation required)
2. Add validation in apply-proposal.js that strips model-proposed terminal closure when isUserConfirmationOfNoRemainingUncertainty(answer) is false
---
## Execution Constraints
Production code changed: NO
Tests changed: NO
Prompt changed: NO
Schema changed: NO
Harness changed: NO
Ollama calls: 0
Live API calls: 0
Vitest run: NO
Jest run: NO
Watchman used: NO
Documentation updated: docs/experiment-60b78.md created; docs/current-handoff.md appended
Git status: will be clean after commit
docs/experiment-60b78.md
+75
View File
@@ -0,0 +1,75 @@
# Experiment 60B.78 — Decision Closure Ownership Policy (Design Only)
**Date:** 2026-08-15
**Branch:** `feature/sufficiency-decision-detection-v0.46`
**Preceded by:** Experiment 60B.77 (model-led closure proven by elimination)
**Type:** Design analysis — no implementation
## Objective
Choose the correct ownership policy for parent-decision closure when:
```text
hasRemainingMaterialFactors(decision) === false
AND
isUserConfirmationOfNoRemainingUncertainty(answer) === false
```
The question is not how to implement a fix. The question is whether model-led parent closure should be allowed in that state, or whether the deterministic sufficiency policy owns whether a decision may close.
## Key Findings
### Prompt Ownership (Prompt-Builder Rule 143)
Rule 143 states: "If the currently supported evidence is sufficient to distinguish the options and no such material unresolved factor remains, resolve the existing decision context and do not ask a generic continuation question."
This permits **evidence sufficiency alone** to justify parent closure. It does NOT explicitly require user confirmation. It conflates "model judges represented factors exhausted" with "user confirms nothing else material remains."
### Deterministic Gate (decision-sufficiency.js + apply-proposal.js)
`shouldCloseDecision` requires both:
1. `countRemainingMaterialFactors === 0`
2. `isUserConfirmationOfNoRemainingUncertainty(answer)` returns true (bounded CONFIRMATION_PHRASES/PATTERNS matching)
This is the **authoritative** closure policy — it unconditionally sets status=resolved when both conditions are met, with no override mechanism.
### Model-Led Closure Risk
60B.77 proved the model independently proposes terminal closure when resolving its last material factor. When this happens without explicit user confirmation:
- **Premature-closure risk:** HIGH — contradicts false-open-over-false-closed philosophy
- **Unrepresented uncertainty possible:** YES — model cannot capture what user knows but hasn't graphed
- **Bypasses conservative policy:** YES — removes the explicit confirmation step entirely
### Status Semantics
- `known`: directional value determined (TERMINAL_STATUSES)
- `resolved`: investigation complete, nothing to investigate further (TERMINAL_STATUSES)
- Both appear in TERMINAL_STATUSES — no existing distinction supports separating "model direction" from "user-confirmed closure" without broader changes.
## Decision
**Choice: B — EXPLICIT CONFIRMATION SHOULD AUTHORITATIVELY GATE CLOSURE**
Rationale:
1. Conservative false-open philosophy encoded in confirmation phrases must not be overridable by model judgment
2. Model-led closure was proven (60B.77) to occur without confirmation
3. Unrepresented material uncertainty is possible
4. State B exists specifically for this gap — removing it via model-led closure defeats its purpose
5. False closure cost >> unnecessary questioning cost
## Minimum Corrective Boundary
**Choice: E — PROMPT CLARIFICATION + DETERMINISTIC ENFORCEMENT**
1. Reword rule 143 to require explicit user confirmation matching deterministic gate criteria
2. Add validation in apply-proposal.js that strips model-proposed terminal closure without explicit confirmation
No schema change required. State B remains reachable when confirmation absent. Model reasoning/direction preservable (can express preferred option without setting resolved status).
## Files Modified
- `docs/experiment-60b78.md` — this file
- `docs/current-handoff.md` — appended experiment entry
No production code, tests, prompts, schema, or harness changes.