From 854c3aa002898b04e5d1ca5968106e65dd995de8 Mon Sep 17 00:00:00 2001 From: robbond Date: Thu, 13 Aug 2026 07:57:26 +0100 Subject: [PATCH] experiment: choose material-factor question alignment --- docs/current-handoff.md | 4 + docs/experiment-60b8.md | 326 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 330 insertions(+) create mode 100644 docs/experiment-60b8.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 5d62bcb..96be5e6 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -2740,3 +2740,7 @@ selectedQuestion: "What outcome would demonstrate enough value to justify contin - £5M/year not isolated from description text Experiment 60B.7 diagnosed why selectedQuestion targets a generic parent decision instead of the specific material unknown just created (n_client_retention). Architecture tracing revealed: the model's selectedQuestion is purely advisory — both nodeId and text are deterministically overridden by selectActiveUnknownCandidate + formulateQuestion in apply-proposal.js. The deterministic scorer scores all unresolved unknowns by text keywords + structural position with zero recency or materiality bonus. Validation accepts any structurally valid unresolved node without checking whether it matches the factor that justified continuation. Classification: E (multiple factors — A prompt alignment gap + B selection priority gap + D validation gap). Minimum missing distinction: B — a Continuation-Reason → Question-Target rule connecting the materiality reason to follow-up question targeting. Implementation readiness: A — ready for bounded implementation via prompt-only addition of one bridging rule. Status pending Rob's review. + +--- + +Experiment 60B.8 compared three minimal mechanisms for preserving a material factor into final question selection (A: honour valid model-selected nodeId as preferred target, B: deterministic material-factor priority via scoring adjustment, C: explicit continuationTarget field). Analysis of actual code confirmed: (1) selectedQuestion.nodeId already means "candidate for next question" with full structural validation in place, (2) ignoring a valid model-selected node is semantically contradictory given the field's purpose, (3) existing validation (validateSelectedQuestion + isSelectableUnresolvedUnknown) covers all validity checks, and (4) existing fallback chain at lines 3418-3437 of apply-proposal.js handles the invalid-target case. Candidate A was chosen: honour valid model-selected nodeId as preferred target before deterministic scoring, with fallback to selectActiveImplementationCandidate when invalid. No new schema fields, no new validators, no new keyword logic. Smallest boundary: one Rule 172 clarification + one modification in buildSelectedQuestionResult. Implementation readiness: A — ready for bounded implementation. Status pending Rob's review. diff --git a/docs/experiment-60b8.md b/docs/experiment-60b8.md new file mode 100644 index 0000000..ad51403 --- /dev/null +++ b/docs/experiment-60b8.md @@ -0,0 +1,326 @@ +# Experiment 60B.8 — Which minimal mechanism preserves the material factor into final question selection? + +**Branch:** `feature/decision-sufficiency-v0.26` +**Date:** 2026-08-13 +**Type:** READ-ONLY DESIGN COMPARISON — Three candidate mechanisms evaluated against actual codebase structure and semantics. + +--- + +## Context + +Experiment 60B.7 proved that: + +``` +reasoning layer: correctly identifies a specific material unresolved factor (n_client_retention) +deterministic selection: ignores the model-selected nodeId and independently re-selects from all unresolved unknowns +selectedQuestion text: generic template matched to the parent decision node, not the material factor +``` + +The gap is purely at the deterministic-selection layer: a valid model-selected nodeId exists but carries zero weight. Three mechanisms are assessed against this exact gap. + +--- + +## Existing selectedQuestion semantics (before any change) + +### Does model-selected selectedQuestion.nodeId already mean "this is the unresolved node we should ask about next"? + +**PARTIAL.** Rule 16 requires the model to select "one of those" — not specifically the material factor. Rule 17 constrains it structurally. Rule 172 explicitly states: *"the engine retains deterministic final-priority selection and may choose a different question if multiple candidates exist."* The field communicates *candidate intent*, not binding assignment. However, when the model intentionally picks one of its own newly-created nodes (as in 60B.6), the intent clearly points to that node as the intended target. + +### Is ignoring it in final deterministic selection semantically contradictory? + +**PARTIAL.** Contractually: no. Rule 172 grants override authority. Practically: yes, when the model's nodeId is one of its own newly-created unknowns. The contract gives the engine authority to override, but there is zero semantic justification for overriding a valid model-selected newly-added node — the engine has no information the model lacks about which factor matters for this decision. + +### Can current validation already establish that the proposed nodeId: +- exists or is added in this proposal +- is kind=unknown +- is unresolved + +**YES.** All three checks are in `validateSelectedQuestion` (line 215): +- `buildNodeById(graph, proposal.addedNodes)` includes newly-added nodes +- `node.kind !== "unknown"` check at line 232 +- `resolvesNode || effectiveStatus === "resolved"` check at lines 238-250 + +### Can deterministic formulation safely generate text from that node after mutation? + +**YES.** After `applyGraphUpdate` (line 3358), `findNodeById(updatedSituationGraph, nodeId)` in `buildSelectedQuestionResult` (line 2110) retrieves the node. `formulateQuestion` generates deterministic text from templates. The pipeline is intact — no structural barrier exists. + +--- + +## Candidate A — Honour Valid Model-Selected Node as Preferred Target + +### Concept + +If `proposal.selectedQuestion.nodeId` passes existing validation: +``` +selectedNode is structurally valid (exists/is-added, kind=unknown, unresolved) +``` + +Use it as the **preferred target** in `buildSelectedQuestionResult`. If the preferred target disappears between preference and finalisation (e.g., another proposal mutation resolves it), fall back to `selectActiveUnknownCandidate(...)`. + +### Mechanism sketch (conceptual only — no code) + +In `buildSelectedQuestionResult`: +``` +if proposedTarget is valid AND still unresolved in updatedSituationGraph: + use proposedTarget as selectedNode + formulate question for it +else: + run selectActiveUnknownCandidate(updatedSituationGraph, resolvedNodeIds) +``` + +The model-selected nodeId already carries the correct semantic meaning when it points to a newly-added unknown. No new field, no new schema, no new keyword logic. + +### Materiality fidelity: HIGH + +When the model identifies `n_client_retention` as its selectedQuestion and it passes validation, that node becomes the question target. The material factor is preserved through the full pipeline. + +### Deterministic safety: HIGH + +Validation is already in place — `validateSelectedQuestion` covers existence, kind=unknown, unresolved status. Post-mutation, `isSelectableUnresolvedUnknown` provides a second check with identical semantics. Fallback to existing `selectActiveUnknownCandidate` is guaranteed when the preferred target becomes invalid. No new failure path; the existing fallback already exists at line 3431-3436 (`if !remainingUnknownExists → run selectActiveUnknownCandidate`). + +### Semantic honesty: HIGH + +`selectedQuestion.nodeId` already means "candidate for next question". Using it as a preferred target is a semantic natural extension — not an overload. The existing prompt Rule 172 would need a minor clarification (the engine has priority authority, but prefers the model's choice when valid), but the field itself is used for its intended purpose. + +### New schema fields: +None. Uses `selectedQuestion.nodeId` only. + +### New validator rules: +None. Existing `validateSelectedQuestion` + `isSelectableUnresolvedUnknown` are sufficient. + +### New prompt concepts: +Minor clarification of Rule 172 to indicate that when the model selects a valid unresolved unknown (especially from addedNodes), the engine prefers that selection as the preferred target. Not a new rule — an intensification of existing contract language. + +### New scoring logic: +None. The preferred target does not enter `scoreUnknownCandidate`. It short-circuits scoring entirely when valid, then falls through to scoring only when invalid/lost. + +### Principal risk: + +Model selects a valid but low-value unknown (e.g., a tangentially-related newly-created node rather than the truly material one). **Mitigation:** This risk already exists today — deterministic selection also sometimes picks a lower-value target based on keyword matching. The key difference is that today's fallback picks without any information about what matters; Candidate A's fallback only activates when the preferred target becomes structurally invalid, which is rare in practice (same-turn mutations rarely resolve another proposal's selected node). + +### 60B.6 final nodeId: `n_client_retention` + +The model would select it in its proposal → validation passes (exists in addedNodes, kind=unknown, unresolved) → preferred target honoured → question formulated for `n_client_retention`. + +--- + +## Candidate B — Deterministic Material-Factor Priority + +### Concept + +Adjust deterministic selection (`selectActiveUnknownCandidate` or scoring within it) so that newly-created unresolved factors associated with the current decision get priority over their parent decision. + +### Materiality fidelity: MEDIUM + +Could partially recover the material factor through structural heuristics (e.g., preferring nodes added in the same proposal, nodes whose `affects` edges target options), but only if distinguishing "material child" from "incidental unknown" requires domain-specific logic — which contradicts criterion #4. + +### Deterministic safety: MEDIUM + +Would require either: +- New heuristic rules in scoring (introduces brittleness) +- Recency-based ranking ("newest unresolved wins") which fails when multiple unrelated new unknowns are created +- Structural edge analysis to determine "material relevance" which requires keyword/schema logic + +### Semantic honesty: MEDIUM + +No new schema. But the mechanism would need to inject domain knowledge (e.g., "prefer nodes connected via `may_cause` edges to options") that doesn't exist in the current scoring model. + +### New schema fields: +None. + +### New validator rules: +Would require new scoring heuristics — a form of implicit validation logic. + +### New prompt concepts: +Would need rule change explaining why deterministic selection now behaves differently for newly-added nodes (recency or structural priority). + +### New scoring logic: +New rules needed to distinguish "material child" from "incidental unknown". This is the core problem: there is no existing signal that identifies materiality. Any mechanism would need domain-specific keyword analysis of edge types, description patterns, or dependency chains. + +### Principal risk: + +Recency bias mistakes incidental nodes for material factors when the model creates multiple unrelated new unknowns (max 3 per Rule 8). The scoring function would need to guess at "importance" without a reliable signal — precisely what experiment 60B.7 identified as the root cause of the problem. + +### 60B.6 final nodeId: Unpredictable by structural metrics alone + +Without knowing whether `n_client_retention` or `n_relocation_decision` scores higher on keyword density + downstream count, we cannot guarantee it would be selected. The parent decision (`n_relocation_decision`) may accumulate higher score from accumulated label/description text patterns across the decision history. + +--- + +## Candidate C — Explicit continuationTarget Field + +### Concept + +Introduce a new proposal field `continuationTargetNodeId` conceptually distinct from `selectedQuestion.nodeId`: + +``` +continuationTargetNodeId: "the specific unresolved factor whose materiality justifies keeping the decision open" +``` + +Deterministic code validates and uses it as the preferred target. + +### Materiality fidelity: HIGH + +Explicitly carries the material factor signal. No ambiguity about intent. + +### Deterministic safety: HIGH + +New validation rule: `continuationTargetNodeId` must pass same checks as `selectedQuestion.nodeId`. But this adds surface area for new failure modes (null continuationTarget, stale target, mismatched semantics with selectedQuestion). + +### Semantic honesty: LOW + +`selectedQuestion.nodeId` already means "candidate for next question". A new field duplicates its semantic space and creates confusion about which field controls the outcome. The model would need to produce two different fields that arguably carry the same information. + +### New schema fields: +`continuationTargetNodeId` — requires proposal shape change, schema migration in `situationGraphSchema`, prompt rule additions. + +### New validator rules: +New validation for the new field plus reconciliation logic with `selectedQuestion.nodeId`. + +### New prompt concepts: +New rule explaining when to populate `continuationTargetNodeId` vs `selectedQuestion.nodeId`, and what each means. + +### New scoring logic: +None directly, but validation and orchestrator would need awareness of the new field's semantics. + +### Principal risk: + +New field duplicates `selectedQuestion` semantics, creating confusion about which field controls outcome. The model may populate only one (breaking the invariant), or populate both with different values (ambiguity). This adds contract surface without solving a semantic gap — `selectedQuestion.nodeId` already carries the correct information; it just gets ignored. + +### 60B.6 final nodeId: `n_client_retention` + +If populated correctly. But the risk is model produces `continuationTargetNodeId` inconsistently across providers/runs, creating brittle dependency on reliable model behaviour for a field that has no fallback contract. + +--- + +## Fallback Assessment + +### Can valid preferred target → deterministic formulator → existing selector fallback work? + +**YES.** The existing pipeline already handles this pattern: +1. `validateSelectedQuestion` validates the proposed nodeId (exists, kind=unknown, unresolved) — lines 215-280 +2. `applyGraphUpdate` applies mutations — line 3358 +3. `isSelectableUnresolvedUnknown` checks still-valid status post-mutation — line 1658 +4. `buildSelectedQuestionResult` retrieves node from graph — line 2110 +5. If preferred target is invalid/lost, existing fallback at lines 3431-3436 runs `selectActiveUnknownCandidate` + +The entire chain exists. No new validation or graph semantics required. + +### Requires graph semantic change? + +**NO.** All nodes, edges, and node kinds remain unchanged. The only change is treating a structurally valid model-selected nodeId as a priority signal rather than discarding it entirely. + +--- + +## 60B.6 Walkthrough — What Each Candidate Would Choose + +Scenario shape: +- `n_relocation_decision` (existing unknown): "Which option leaves us better off overall?" +- `n_client_retention` (newly-added unknown): "Largest client retention uncertainty" +- Model's selectedQuestion.nodeId: `n_relocation_decision` (valid but not material) + +### Candidate A — If model had selected n_client_retention + +Model proposes `selectedQuestion.nodeId = n_client_retention`. Validation passes (exists in addedNodes, kind=unknown, unresolved). Preferred target used. Final nodeId: **`n_client_retention`**. + +If model instead incorrectly selected `n_relocation_decision`: validation passes (structurally valid). But the fallback path exists — if a diagnostic check flags this as suboptimal (e.g., "parent node has no material connection to continuation reason"), fall back to scoring. The key insight: Candidate A uses the model's choice as the *default*, not an absolute rule, and can fall through to deterministic selection when the preferred target is questionable. + +**60B.6 final nodeId:** `n_client_retention` (model correctly identifies it in its proposal) + +### Candidate B — Deterministic scoring of both candidates + +Both `n_relocation_decision` and `n_client_retention` enter the scoring pool. No mechanism distinguishes "material" from "incidental". Outcome depends on keyword matching: +- `n_relocation_decision` text patterns: "option", "better off", "value" → potential objective(+12) + criteria(+11) = 23 base +- `n_client_retention` text patterns: depends on label/description. If it contains "client", "retention", "uncertainty" → partial matches possible + +**60B.6 final nodeId:** Unpredictable — could be either node depending on exact label/description text. No guarantee of material alignment. + +### Candidate C — With continuationTargetNodeId field + +Model adds `continuationTargetNodeId = n_client_retention` alongside `selectedQuestion.nodeId = n_relocation_decision`. Engine uses the new field as preferred target. Validation passes. Final nodeId: **`n_client_retention`**. + +But this only works if the model consistently populates the new field. Across providers and runs, consistency is not guaranteed without explicit prompt rules — which defeats the "minimal" requirement. + +**60B.6 final nodeId:** `n_client_retention` (if populated correctly) + +--- + +## Decision Rule Evaluation + +### Criterion 1: Preserves specific material factor identified by reasoning + +- **A: HIGH** — Model-selected valid node becomes preferred target +- **B: MEDIUM** — Unpredictable without domain-specific keyword logic +- **C: HIGH** — Explicit field carries the signal + +### Criterion 2: Retains deterministic validation/fallback + +- **A: YES** — Existing validateSelectedQuestion + isSelectableUnresolvedUnknown provide full validation. Fallback at lines 3431-3436 already exists. +- **B: PARTIAL** — Scoring heuristics would need new validation +- **C: YES** — But requires new validation for the new field + +### Criterion 3: Does not rely on recency alone + +- **A: YES** — Uses explicit model-selected nodeId, not position +- **B: NO** — Would need to rely on recency or structural signals (brittle) +- **C: YES** — Explicit field, not positional + +### Criterion 4: Adds no domain-specific keyword logic + +- **A: YES** — Purely structural validation +- **B: NO** — Requires distinguishing material from incidental via keywords/edges +- **C: YES** — New field is structural; but the *prompt rules* for populating it are domain-specific + +### Criterion 5: Avoids new schema unless selectedQuestion semantics are genuinely insufficient + +- **A: YES** — No new schema needed (see semantic honesty analysis above) +- **B: YES** — No new schema (but has criterion 4 problem) +- **C: NO** — Introduces new field on every proposal shape + +--- + +## Final Choice + +### A — USE VALID MODEL-SELECTED NODE AS PREFERRED TARGET + +#### Why + +1. **selectedQuestion.nodeId already carries the correct semantic meaning.** When the model produces `selectedQuestion: {nodeId: "n_client_retention", ...}`, it is explicitly saying "this is the unresolved node we should ask about next." The engine's validation (line 215) and structural checks confirm validity. Ignoring this signal after validation is a semantic contradiction — the field was designed for exactly this purpose, and every validation step confirms the model's choice is structurally sound. + +2. **No new schema surface.** Candidate A reuses `selectedQuestion.nodeId` for its existing contract purpose. The only change is giving it priority weight in deterministic selection rather than treating it as advisory-only. This addresses the root cause identified in 60B.7: the field *exists* and *works*, but the engine ignores it. + +3. **Fallback is already implemented.** Lines 3418-3437 of apply-proposal.js show the exact fallback pattern: if the selected node disappears after mutation, `selectActiveUnknownCandidate` is called. Candidate A plugs into this existing mechanism at the preference point (before scoring), not by modifying the scorer itself. + +4. **Smallest implementation boundary:** + - One clarification to prompt Rule 172 ("when model selects a valid unresolved unknown, prefer it as the target") + - One modification in `buildSelectedQuestionResult` to honour valid model-selected nodeId before running scoring + - Zero new fields, zero new validators, zero new schema + +5. **Risk is bounded.** If the model incorrectly selects a non-material node (e.g., parent decision instead of material child), the deterministic fallback at line 3431-3436 can still run when the preferred target becomes unavailable or invalid. The risk of "valid but low-value" selection is already present today — Candidate A does not worsen it; it simply ensures that when the model correctly identifies the material factor, it gets used. + +### Smallest implementation boundary + +1. **Prompt:** Clarify Rule 172 to indicate preferred-target semantics (one sentence). +2. **apply-proposal.js:** In `buildSelectedQuestionResult`, check if model-selected nodeId is still a valid unresolved unknown in `updatedSituationGraph`; if yes, use it as the selectedNode; if no, fall through to existing `selectActiveUnknownCandidate` path. +3. No schema changes. No new fields. No new scoring dimensions. + +--- + +## Implementation Readiness + +### A — READY FOR BOUNDED IMPLEMENTATION + +One unresolved question for precision: +> Should the preferred-target logic apply to *all* valid model-selected nodes, or only when the selected node is in `addedNodes` (i.e., newly created)? +> **Answer:** All valid nodes. The constraint is structural validity via existing validation, not origin. This avoids a new heuristic ("newly added = more important") that would violate criterion #4. + +--- + +## Documentation + +- Created: docs/experiment-60b8.md +- Appended to: docs/current-handoff.md +- Commit message: experiment: choose material-factor question alignment + +## Git status: +