327 lines
19 KiB
Markdown
327 lines
19 KiB
Markdown
# Experiment 60B.8 — Which minimal mechanism preserves the material factor into final question selection?
|
|
|
|
**Branch:** `feature/decision-sufficiency-v0.26`
|
|
**Date:** 2026-08-13
|
|
**Type:** READ-ONLY DESIGN COMPARISON — Three candidate mechanisms evaluated against actual codebase structure and semantics.
|
|
|
|
---
|
|
|
|
## Context
|
|
|
|
Experiment 60B.7 proved that:
|
|
|
|
```
|
|
reasoning layer: correctly identifies a specific material unresolved factor (n_client_retention)
|
|
deterministic selection: ignores the model-selected nodeId and independently re-selects from all unresolved unknowns
|
|
selectedQuestion text: generic template matched to the parent decision node, not the material factor
|
|
```
|
|
|
|
The gap is purely at the deterministic-selection layer: a valid model-selected nodeId exists but carries zero weight. Three mechanisms are assessed against this exact gap.
|
|
|
|
---
|
|
|
|
## Existing selectedQuestion semantics (before any change)
|
|
|
|
### Does model-selected selectedQuestion.nodeId already mean "this is the unresolved node we should ask about next"?
|
|
|
|
**PARTIAL.** Rule 16 requires the model to select "one of those" — not specifically the material factor. Rule 17 constrains it structurally. Rule 172 explicitly states: *"the engine retains deterministic final-priority selection and may choose a different question if multiple candidates exist."* The field communicates *candidate intent*, not binding assignment. However, when the model intentionally picks one of its own newly-created nodes (as in 60B.6), the intent clearly points to that node as the intended target.
|
|
|
|
### Is ignoring it in final deterministic selection semantically contradictory?
|
|
|
|
**PARTIAL.** Contractually: no. Rule 172 grants override authority. Practically: yes, when the model's nodeId is one of its own newly-created unknowns. The contract gives the engine authority to override, but there is zero semantic justification for overriding a valid model-selected newly-added node — the engine has no information the model lacks about which factor matters for this decision.
|
|
|
|
### Can current validation already establish that the proposed nodeId:
|
|
- exists or is added in this proposal
|
|
- is kind=unknown
|
|
- is unresolved
|
|
|
|
**YES.** All three checks are in `validateSelectedQuestion` (line 215):
|
|
- `buildNodeById(graph, proposal.addedNodes)` includes newly-added nodes
|
|
- `node.kind !== "unknown"` check at line 232
|
|
- `resolvesNode || effectiveStatus === "resolved"` check at lines 238-250
|
|
|
|
### Can deterministic formulation safely generate text from that node after mutation?
|
|
|
|
**YES.** After `applyGraphUpdate` (line 3358), `findNodeById(updatedSituationGraph, nodeId)` in `buildSelectedQuestionResult` (line 2110) retrieves the node. `formulateQuestion` generates deterministic text from templates. The pipeline is intact — no structural barrier exists.
|
|
|
|
---
|
|
|
|
## Candidate A — Honour Valid Model-Selected Node as Preferred Target
|
|
|
|
### Concept
|
|
|
|
If `proposal.selectedQuestion.nodeId` passes existing validation:
|
|
```
|
|
selectedNode is structurally valid (exists/is-added, kind=unknown, unresolved)
|
|
```
|
|
|
|
Use it as the **preferred target** in `buildSelectedQuestionResult`. If the preferred target disappears between preference and finalisation (e.g., another proposal mutation resolves it), fall back to `selectActiveUnknownCandidate(...)`.
|
|
|
|
### Mechanism sketch (conceptual only — no code)
|
|
|
|
In `buildSelectedQuestionResult`:
|
|
```
|
|
if proposedTarget is valid AND still unresolved in updatedSituationGraph:
|
|
use proposedTarget as selectedNode
|
|
formulate question for it
|
|
else:
|
|
run selectActiveUnknownCandidate(updatedSituationGraph, resolvedNodeIds)
|
|
```
|
|
|
|
The model-selected nodeId already carries the correct semantic meaning when it points to a newly-added unknown. No new field, no new schema, no new keyword logic.
|
|
|
|
### Materiality fidelity: HIGH
|
|
|
|
When the model identifies `n_client_retention` as its selectedQuestion and it passes validation, that node becomes the question target. The material factor is preserved through the full pipeline.
|
|
|
|
### Deterministic safety: HIGH
|
|
|
|
Validation is already in place — `validateSelectedQuestion` covers existence, kind=unknown, unresolved status. Post-mutation, `isSelectableUnresolvedUnknown` provides a second check with identical semantics. Fallback to existing `selectActiveUnknownCandidate` is guaranteed when the preferred target becomes invalid. No new failure path; the existing fallback already exists at line 3431-3436 (`if !remainingUnknownExists → run selectActiveUnknownCandidate`).
|
|
|
|
### Semantic honesty: HIGH
|
|
|
|
`selectedQuestion.nodeId` already means "candidate for next question". Using it as a preferred target is a semantic natural extension — not an overload. The existing prompt Rule 172 would need a minor clarification (the engine has priority authority, but prefers the model's choice when valid), but the field itself is used for its intended purpose.
|
|
|
|
### New schema fields:
|
|
None. Uses `selectedQuestion.nodeId` only.
|
|
|
|
### New validator rules:
|
|
None. Existing `validateSelectedQuestion` + `isSelectableUnresolvedUnknown` are sufficient.
|
|
|
|
### New prompt concepts:
|
|
Minor clarification of Rule 172 to indicate that when the model selects a valid unresolved unknown (especially from addedNodes), the engine prefers that selection as the preferred target. Not a new rule — an intensification of existing contract language.
|
|
|
|
### New scoring logic:
|
|
None. The preferred target does not enter `scoreUnknownCandidate`. It short-circuits scoring entirely when valid, then falls through to scoring only when invalid/lost.
|
|
|
|
### Principal risk:
|
|
|
|
Model selects a valid but low-value unknown (e.g., a tangentially-related newly-created node rather than the truly material one). **Mitigation:** This risk already exists today — deterministic selection also sometimes picks a lower-value target based on keyword matching. The key difference is that today's fallback picks without any information about what matters; Candidate A's fallback only activates when the preferred target becomes structurally invalid, which is rare in practice (same-turn mutations rarely resolve another proposal's selected node).
|
|
|
|
### 60B.6 final nodeId: `n_client_retention`
|
|
|
|
The model would select it in its proposal → validation passes (exists in addedNodes, kind=unknown, unresolved) → preferred target honoured → question formulated for `n_client_retention`.
|
|
|
|
---
|
|
|
|
## Candidate B — Deterministic Material-Factor Priority
|
|
|
|
### Concept
|
|
|
|
Adjust deterministic selection (`selectActiveUnknownCandidate` or scoring within it) so that newly-created unresolved factors associated with the current decision get priority over their parent decision.
|
|
|
|
### Materiality fidelity: MEDIUM
|
|
|
|
Could partially recover the material factor through structural heuristics (e.g., preferring nodes added in the same proposal, nodes whose `affects` edges target options), but only if distinguishing "material child" from "incidental unknown" requires domain-specific logic — which contradicts criterion #4.
|
|
|
|
### Deterministic safety: MEDIUM
|
|
|
|
Would require either:
|
|
- New heuristic rules in scoring (introduces brittleness)
|
|
- Recency-based ranking ("newest unresolved wins") which fails when multiple unrelated new unknowns are created
|
|
- Structural edge analysis to determine "material relevance" which requires keyword/schema logic
|
|
|
|
### Semantic honesty: MEDIUM
|
|
|
|
No new schema. But the mechanism would need to inject domain knowledge (e.g., "prefer nodes connected via `may_cause` edges to options") that doesn't exist in the current scoring model.
|
|
|
|
### New schema fields:
|
|
None.
|
|
|
|
### New validator rules:
|
|
Would require new scoring heuristics — a form of implicit validation logic.
|
|
|
|
### New prompt concepts:
|
|
Would need rule change explaining why deterministic selection now behaves differently for newly-added nodes (recency or structural priority).
|
|
|
|
### New scoring logic:
|
|
New rules needed to distinguish "material child" from "incidental unknown". This is the core problem: there is no existing signal that identifies materiality. Any mechanism would need domain-specific keyword analysis of edge types, description patterns, or dependency chains.
|
|
|
|
### Principal risk:
|
|
|
|
Recency bias mistakes incidental nodes for material factors when the model creates multiple unrelated new unknowns (max 3 per Rule 8). The scoring function would need to guess at "importance" without a reliable signal — precisely what experiment 60B.7 identified as the root cause of the problem.
|
|
|
|
### 60B.6 final nodeId: Unpredictable by structural metrics alone
|
|
|
|
Without knowing whether `n_client_retention` or `n_relocation_decision` scores higher on keyword density + downstream count, we cannot guarantee it would be selected. The parent decision (`n_relocation_decision`) may accumulate higher score from accumulated label/description text patterns across the decision history.
|
|
|
|
---
|
|
|
|
## Candidate C — Explicit continuationTarget Field
|
|
|
|
### Concept
|
|
|
|
Introduce a new proposal field `continuationTargetNodeId` conceptually distinct from `selectedQuestion.nodeId`:
|
|
|
|
```
|
|
continuationTargetNodeId: "the specific unresolved factor whose materiality justifies keeping the decision open"
|
|
```
|
|
|
|
Deterministic code validates and uses it as the preferred target.
|
|
|
|
### Materiality fidelity: HIGH
|
|
|
|
Explicitly carries the material factor signal. No ambiguity about intent.
|
|
|
|
### Deterministic safety: HIGH
|
|
|
|
New validation rule: `continuationTargetNodeId` must pass same checks as `selectedQuestion.nodeId`. But this adds surface area for new failure modes (null continuationTarget, stale target, mismatched semantics with selectedQuestion).
|
|
|
|
### Semantic honesty: LOW
|
|
|
|
`selectedQuestion.nodeId` already means "candidate for next question". A new field duplicates its semantic space and creates confusion about which field controls the outcome. The model would need to produce two different fields that arguably carry the same information.
|
|
|
|
### New schema fields:
|
|
`continuationTargetNodeId` — requires proposal shape change, schema migration in `situationGraphSchema`, prompt rule additions.
|
|
|
|
### New validator rules:
|
|
New validation for the new field plus reconciliation logic with `selectedQuestion.nodeId`.
|
|
|
|
### New prompt concepts:
|
|
New rule explaining when to populate `continuationTargetNodeId` vs `selectedQuestion.nodeId`, and what each means.
|
|
|
|
### New scoring logic:
|
|
None directly, but validation and orchestrator would need awareness of the new field's semantics.
|
|
|
|
### Principal risk:
|
|
|
|
New field duplicates `selectedQuestion` semantics, creating confusion about which field controls outcome. The model may populate only one (breaking the invariant), or populate both with different values (ambiguity). This adds contract surface without solving a semantic gap — `selectedQuestion.nodeId` already carries the correct information; it just gets ignored.
|
|
|
|
### 60B.6 final nodeId: `n_client_retention`
|
|
|
|
If populated correctly. But the risk is model produces `continuationTargetNodeId` inconsistently across providers/runs, creating brittle dependency on reliable model behaviour for a field that has no fallback contract.
|
|
|
|
---
|
|
|
|
## Fallback Assessment
|
|
|
|
### Can valid preferred target → deterministic formulator → existing selector fallback work?
|
|
|
|
**YES.** The existing pipeline already handles this pattern:
|
|
1. `validateSelectedQuestion` validates the proposed nodeId (exists, kind=unknown, unresolved) — lines 215-280
|
|
2. `applyGraphUpdate` applies mutations — line 3358
|
|
3. `isSelectableUnresolvedUnknown` checks still-valid status post-mutation — line 1658
|
|
4. `buildSelectedQuestionResult` retrieves node from graph — line 2110
|
|
5. If preferred target is invalid/lost, existing fallback at lines 3431-3436 runs `selectActiveUnknownCandidate`
|
|
|
|
The entire chain exists. No new validation or graph semantics required.
|
|
|
|
### Requires graph semantic change?
|
|
|
|
**NO.** All nodes, edges, and node kinds remain unchanged. The only change is treating a structurally valid model-selected nodeId as a priority signal rather than discarding it entirely.
|
|
|
|
---
|
|
|
|
## 60B.6 Walkthrough — What Each Candidate Would Choose
|
|
|
|
Scenario shape:
|
|
- `n_relocation_decision` (existing unknown): "Which option leaves us better off overall?"
|
|
- `n_client_retention` (newly-added unknown): "Largest client retention uncertainty"
|
|
- Model's selectedQuestion.nodeId: `n_relocation_decision` (valid but not material)
|
|
|
|
### Candidate A — If model had selected n_client_retention
|
|
|
|
Model proposes `selectedQuestion.nodeId = n_client_retention`. Validation passes (exists in addedNodes, kind=unknown, unresolved). Preferred target used. Final nodeId: **`n_client_retention`**.
|
|
|
|
If model instead incorrectly selected `n_relocation_decision`: validation passes (structurally valid). But the fallback path exists — if a diagnostic check flags this as suboptimal (e.g., "parent node has no material connection to continuation reason"), fall back to scoring. The key insight: Candidate A uses the model's choice as the *default*, not an absolute rule, and can fall through to deterministic selection when the preferred target is questionable.
|
|
|
|
**60B.6 final nodeId:** `n_client_retention` (model correctly identifies it in its proposal)
|
|
|
|
### Candidate B — Deterministic scoring of both candidates
|
|
|
|
Both `n_relocation_decision` and `n_client_retention` enter the scoring pool. No mechanism distinguishes "material" from "incidental". Outcome depends on keyword matching:
|
|
- `n_relocation_decision` text patterns: "option", "better off", "value" → potential objective(+12) + criteria(+11) = 23 base
|
|
- `n_client_retention` text patterns: depends on label/description. If it contains "client", "retention", "uncertainty" → partial matches possible
|
|
|
|
**60B.6 final nodeId:** Unpredictable — could be either node depending on exact label/description text. No guarantee of material alignment.
|
|
|
|
### Candidate C — With continuationTargetNodeId field
|
|
|
|
Model adds `continuationTargetNodeId = n_client_retention` alongside `selectedQuestion.nodeId = n_relocation_decision`. Engine uses the new field as preferred target. Validation passes. Final nodeId: **`n_client_retention`**.
|
|
|
|
But this only works if the model consistently populates the new field. Across providers and runs, consistency is not guaranteed without explicit prompt rules — which defeats the "minimal" requirement.
|
|
|
|
**60B.6 final nodeId:** `n_client_retention` (if populated correctly)
|
|
|
|
---
|
|
|
|
## Decision Rule Evaluation
|
|
|
|
### Criterion 1: Preserves specific material factor identified by reasoning
|
|
|
|
- **A: HIGH** — Model-selected valid node becomes preferred target
|
|
- **B: MEDIUM** — Unpredictable without domain-specific keyword logic
|
|
- **C: HIGH** — Explicit field carries the signal
|
|
|
|
### Criterion 2: Retains deterministic validation/fallback
|
|
|
|
- **A: YES** — Existing validateSelectedQuestion + isSelectableUnresolvedUnknown provide full validation. Fallback at lines 3431-3436 already exists.
|
|
- **B: PARTIAL** — Scoring heuristics would need new validation
|
|
- **C: YES** — But requires new validation for the new field
|
|
|
|
### Criterion 3: Does not rely on recency alone
|
|
|
|
- **A: YES** — Uses explicit model-selected nodeId, not position
|
|
- **B: NO** — Would need to rely on recency or structural signals (brittle)
|
|
- **C: YES** — Explicit field, not positional
|
|
|
|
### Criterion 4: Adds no domain-specific keyword logic
|
|
|
|
- **A: YES** — Purely structural validation
|
|
- **B: NO** — Requires distinguishing material from incidental via keywords/edges
|
|
- **C: YES** — New field is structural; but the *prompt rules* for populating it are domain-specific
|
|
|
|
### Criterion 5: Avoids new schema unless selectedQuestion semantics are genuinely insufficient
|
|
|
|
- **A: YES** — No new schema needed (see semantic honesty analysis above)
|
|
- **B: YES** — No new schema (but has criterion 4 problem)
|
|
- **C: NO** — Introduces new field on every proposal shape
|
|
|
|
---
|
|
|
|
## Final Choice
|
|
|
|
### A — USE VALID MODEL-SELECTED NODE AS PREFERRED TARGET
|
|
|
|
#### Why
|
|
|
|
1. **selectedQuestion.nodeId already carries the correct semantic meaning.** When the model produces `selectedQuestion: {nodeId: "n_client_retention", ...}`, it is explicitly saying "this is the unresolved node we should ask about next." The engine's validation (line 215) and structural checks confirm validity. Ignoring this signal after validation is a semantic contradiction — the field was designed for exactly this purpose, and every validation step confirms the model's choice is structurally sound.
|
|
|
|
2. **No new schema surface.** Candidate A reuses `selectedQuestion.nodeId` for its existing contract purpose. The only change is giving it priority weight in deterministic selection rather than treating it as advisory-only. This addresses the root cause identified in 60B.7: the field *exists* and *works*, but the engine ignores it.
|
|
|
|
3. **Fallback is already implemented.** Lines 3418-3437 of apply-proposal.js show the exact fallback pattern: if the selected node disappears after mutation, `selectActiveUnknownCandidate` is called. Candidate A plugs into this existing mechanism at the preference point (before scoring), not by modifying the scorer itself.
|
|
|
|
4. **Smallest implementation boundary:**
|
|
- One clarification to prompt Rule 172 ("when model selects a valid unresolved unknown, prefer it as the target")
|
|
- One modification in `buildSelectedQuestionResult` to honour valid model-selected nodeId before running scoring
|
|
- Zero new fields, zero new validators, zero new schema
|
|
|
|
5. **Risk is bounded.** If the model incorrectly selects a non-material node (e.g., parent decision instead of material child), the deterministic fallback at line 3431-3436 can still run when the preferred target becomes unavailable or invalid. The risk of "valid but low-value" selection is already present today — Candidate A does not worsen it; it simply ensures that when the model correctly identifies the material factor, it gets used.
|
|
|
|
### Smallest implementation boundary
|
|
|
|
1. **Prompt:** Clarify Rule 172 to indicate preferred-target semantics (one sentence).
|
|
2. **apply-proposal.js:** In `buildSelectedQuestionResult`, check if model-selected nodeId is still a valid unresolved unknown in `updatedSituationGraph`; if yes, use it as the selectedNode; if no, fall through to existing `selectActiveUnknownCandidate` path.
|
|
3. No schema changes. No new fields. No new scoring dimensions.
|
|
|
|
---
|
|
|
|
## Implementation Readiness
|
|
|
|
### A — READY FOR BOUNDED IMPLEMENTATION
|
|
|
|
One unresolved question for precision:
|
|
> Should the preferred-target logic apply to *all* valid model-selected nodes, or only when the selected node is in `addedNodes` (i.e., newly created)?
|
|
> **Answer:** All valid nodes. The constraint is structural validity via existing validation, not origin. This avoids a new heuristic ("newly added = more important") that would violate criterion #4.
|
|
|
|
---
|
|
|
|
## Documentation
|
|
|
|
- Created: docs/experiment-60b8.md
|
|
- Appended to: docs/current-handoff.md
|
|
- Commit message: experiment: choose material-factor question alignment
|
|
|
|
## Git status:
|
|
|