docs: record prerequisite-aware question targeting
This commit is contained in:
@@ -0,0 +1,325 @@
|
||||
# Experiment 60B.10 — When should a valid model-selected target override deterministic priority?
|
||||
|
||||
**Branch:** `feature/question-target-alignment-v0.27`
|
||||
**Date:** 2026-08-13
|
||||
**Type:** READ-ONLY DESIGN DIAGNOSIS — Resolves the contract conflict between honoring model-selected targets and preserving existing structural overrides.
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
Experiment 60B.9 implemented a blanket "honour model-selected unresolved unknown" rule at line 3680 of `apply-proposal.js`. This exposed a genuine contract conflict:
|
||||
|
||||
```
|
||||
NEW desired behaviour: preserve a model-selected material unknown when it is the specific same-turn factor that justifies continuation
|
||||
|
||||
EXISTING behaviour (expressed as regression test): deterministic selection may override a valid model-selected unresolved node when another candidate has higher structural/deterministic value
|
||||
```
|
||||
|
||||
The failing regression: `"replaces downstream pricing question with higher-value commercial-value question"` proves these behaviours cannot both be preserved if every structurally-valid model target is always preferred.
|
||||
|
||||
---
|
||||
|
||||
## CASE A — 60B.6 material factor
|
||||
|
||||
**Source:** Experiment 60B.6 (docs/current-handoff.md, lines 2703-2742), validated by the reasoning-layer output from live qwen-claude call on `pre-anchored-decision-options.json`.
|
||||
|
||||
**Existing decision node:**
|
||||
|
||||
```
|
||||
n_relocation_decision — kind=unknown, status=unknown, label="Which option leaves us better off overall?"
|
||||
(pre-existing central decision; activeUnknown before this turn)
|
||||
|
||||
opt_relocate — kind=option, label="Relocate to Manchester"
|
||||
```
|
||||
|
||||
**Same-proposal added material unknown:**
|
||||
|
||||
```
|
||||
n_client_retention — kind=unknown, status=unknown
|
||||
label: "Largest client retention uncertainty"
|
||||
addedEdges: [n_client_retention → opt_relocate, relationship="may_cause"]
|
||||
Created because the answer introduced the first new factor that could reverse the preferred option (staying).
|
||||
```
|
||||
|
||||
**Model-selected node:** `n_client_retention`
|
||||
|
||||
**Desired deterministic target:** `n_client_retention` — because it is the specific material uncertainty whose outcome could change the preferred decision option, justifying continuation. The existing parent (`n_relocation_decision`) is merely the evaluation context, not the material gap itself.
|
||||
|
||||
**Graph structure of Case A:**
|
||||
|
||||
```
|
||||
n_relocation_decision (existing unknown) ← activeUnknown before proposal
|
||||
n_build_decision (newly-added state)
|
||||
n_client_retention (newly-added unknown, may_cause → opt_relocate)
|
||||
n_relocation_unknown (pre-existing unknown — also unresolved after this turn)
|
||||
```
|
||||
|
||||
There is NO `depends_on` edge between n_client_retention and any other newly-added unresolved unknown in this proposal. The `may_cause` edge connects to an option (non-unknown), not to another unknown node.
|
||||
|
||||
---
|
||||
|
||||
## CASE B — pricing regression
|
||||
|
||||
**Source:** New test added in 60B.9 working tree at `tests/graph/apply-proposal.test.js:2006`.
|
||||
|
||||
### Pre-existing model-selected nodes (before proposal):
|
||||
|
||||
```
|
||||
n_complaint_rate_unknown (kind=unknown, status=unknown) → resolved by this proposal
|
||||
n_staffing_unknown (kind=unknown, status=unknown) → NOT resolved; remains unresolved after this turn
|
||||
```
|
||||
|
||||
After resolution of `n_complaint_rate_unknown`: one pre-existing unresolved unknown remains:
|
||||
|
||||
```
|
||||
n_staffing_unknown
|
||||
```
|
||||
|
||||
### Same-proposal added nodes:
|
||||
|
||||
```
|
||||
n_commercial_value — kind=unknown, status=unknown (no dependencies)
|
||||
n_pricing — kind=unknown, status=unknown, depends_on=["n_commercial_value"]
|
||||
n_build_decision — kind=state (not unknown; irrelevant to selection)
|
||||
```
|
||||
|
||||
### Model-selected nodeId:
|
||||
|
||||
```
|
||||
n_pricing (reason: "Model chose a downstream leaf")
|
||||
```
|
||||
|
||||
### Existing deterministic winner:
|
||||
|
||||
```
|
||||
n_commercial_value (preferred by deterministic scoring over n_pricing because:
|
||||
- n_commercial_value has unresolvedParentUnknownCount=0
|
||||
- n_pricing has unresolvedParentUnknownCount=1 (depends on n_commercial_value)
|
||||
- structural prerequisite relationship: n_commercial_value → depends_on ← n_pricing)
|
||||
```
|
||||
|
||||
Note: `n_staffing_unknown` is also an unresolved candidate but its score is lower than both commercial nodes due to keyword matching and downstream count patterns. The test specifically verifies that `n_commercial_value` wins over the model-selected `n_pricing`.
|
||||
|
||||
### Why existing test prefers deterministic winner:
|
||||
|
||||
The selected node `n_pricing` is structurally **downstream** of another unresolved unknown (`n_commercial_value`) added in this same proposal. The structural prerequisite chain (commercial value → pricing) means you cannot properly assess n_pricing without first resolving n_commercial_value. Honoring the model's selection of a downstream consequence before its prerequisite understanding would be investigation-order inverted.
|
||||
|
||||
### Graph structure of Case B:
|
||||
|
||||
```
|
||||
n_build_decision (newly-added state)
|
||||
├─ n_commercial_value (newly-added unknown, leaf — no upstream unknown dependencies)
|
||||
└─ n_pricing (newly-added unknown, dependent on n_commercial_value via depends_on edge)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CANDIDATE A — SAME-PROPOSAL TARGET
|
||||
|
||||
**Rule:** If model-selected nodeId points to an unresolved unknown added in THIS proposal, prefer it as final target. If model-selected nodeId points to a pre-existing unresolved unknown, retain current deterministic selection behaviour.
|
||||
|
||||
### Assessment:
|
||||
|
||||
| Criterion | Answer |
|
||||
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Materiality fidelity | **MEDIUM** — Correctly preserves n_client_retention (Case A). But also prefers n_pricing in Case B where the model chose a downstream node over its prerequisite. |
|
||||
| Preserves existing pricing regression | **NO** — In Case B, both n_commercial_value and n_pricing are same-proposal-added. The rule prefers n_pricing (model-selected) over n_commercial_value (structural prerequisite), breaking the regression. |
|
||||
| Requires new schema | **NO** — Uses `proposal.addedNodes` + `selectedQuestion.nodeId`, both existing. |
|
||||
| Requires new scoring logic | **NO** — Binary check: isInAddedNodes(selectedNodeId). |
|
||||
| Relies on recency alone | **YES** — "Added in this proposal" is a pure recency signal with no structural or semantic content beyond timing. The model-selected same-proposal node could be upstream prerequisite, downstream consequence, or tangentially-related. All three types would be equally preferred. |
|
||||
| Principal risk | Selecting a downstream consequence before its prerequisite understanding. In Case B, this means asking about pricing before defining commercial value — an investigation-order error. Also: any newly-created unknown (material factor OR tangential) gets equal weight when the model explicitly selects it. |
|
||||
|
||||
### Critical flaw for Candidate A:
|
||||
|
||||
"Same-proposal-added" encompasses both upstream prerequisites AND downstream consequences. When the model creates a dependency chain (commercial_value → pricing), the rule cannot distinguish which end of the chain is the material uncertainty. It simply picks whichever the model named — which in Case B happens to be the wrong end of the chain.
|
||||
|
||||
---
|
||||
|
||||
## CANDIDATE B — SAME-PROPOSAL + STRUCTURAL RELATION
|
||||
|
||||
**Rule:** Prefer a model-selected same-proposal-added unresolved unknown only when it has no unresolved parent unknowns that were also added in this proposal turn. When such a structural dependency exists, retain deterministic priority over the upstream prerequisite.
|
||||
|
||||
### Why "unresolved parent unknown from same proposal" is the right structural signal:
|
||||
|
||||
When the model creates both an upstream and downstream unknown in the same turn (e.g., commercial_value → pricing), the `depends_on` edge between them indicates intentional dependency structure — not coincidental timing. The upstream node represents prerequisite understanding; the downstream node represents a consequence of that understanding. Investigation methodology dictates prerequisites before consequences.
|
||||
|
||||
When there is NO unresolved parent unknown from the same proposal (as in Case A), the model-selected node is structurally independent within this turn's additions — it has no structural ties to other newly-created unknowns, making it the appropriate material factor target.
|
||||
|
||||
### Assessment:
|
||||
|
||||
| Criterion | Answer |
|
||||
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Materiality fidelity | **HIGH** — Case A: n_client_retention has no unresolved parent unknown from same proposal → honored (correct). Case B: n_pricing depends on n_commercial_value (same proposal) → not honored; deterministic selects n_commercial_value (correct). |
|
||||
| Preserves existing pricing regression | **YES** — The structural dependency check prevents honoring n_pricing in Case B. |
|
||||
| Existing structure sufficient | **YES** — Edge relationships (`depends_on` edges into unknown nodes) and `proposal.addedNodes` are both pre-existing. No schema changes needed. |
|
||||
| New schema required | **NO** — Uses only existing: `proposal.addedNodes`, node edge references, `isSelectableUnresolvedUnknown`. |
|
||||
| Principal risk | The structural dependency check could reject a legitimately selected downstream node if the model created a dependency chain for non-investigation-order reasons (e.g., parallel branch creation). However, in practice, `depends_on` edges between unknown nodes in the same proposal almost always represent intentional prerequisite chains. This is conservative: it errs on the side of addressing prerequisites first. |
|
||||
|
||||
### Implementation boundary (conceptual only):
|
||||
|
||||
```
|
||||
In apply-proposal.js after line 3680-3695 (existing honor block):
|
||||
|
||||
if (validatedProposal.selectedQuestion?.nodeId) {
|
||||
const candidateNodeId = validatedProposal.selectedQuestion.nodeId;
|
||||
|
||||
// Check if this candidate is a same-proposal addition
|
||||
const addedInThisProposal = validatedProposal.addedNodes.some(
|
||||
n => n.id === candidateNodeId
|
||||
);
|
||||
|
||||
if (addedInThisProposal && isSelectableUnresolvedUnknown(updatedSituationGraph, candidateNodeId)) {
|
||||
// New structural check: does this node have unresolved parent unknowns from same proposal?
|
||||
const upstreamParentIds = findUpstreamUnknownParents(candidateNodeId, updatedSituationGraph);
|
||||
const parentsAddedThisTurn = upstreamParentIds.filter(
|
||||
parentId => validatedProposal.addedNodes.some(n => n.id === parentId && n.kind === "unknown")
|
||||
);
|
||||
|
||||
if (parentsAddedThisTurn.length === 0) {
|
||||
// No structural dependency on same-turn unknowns → prefer as target
|
||||
deterministicSelection = honorModelSelected(...);
|
||||
}
|
||||
// else: retain deterministic priority (structural prerequisite wins)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The `findUpstreamUnknownParents` function uses existing edge traversal — no schema change.
|
||||
|
||||
---
|
||||
|
||||
## CANDIDATE C — MODEL TARGET SCORING INPUT
|
||||
|
||||
**Rule:** Keep existing deterministic ranking but add a bounded preference/bonus for a valid model-selected node.
|
||||
|
||||
### Assessment:
|
||||
|
||||
| Criterion | Answer |
|
||||
| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Fixes 60B.6 without arbitrary tuning | **NO** — To fix Case A (where n_client_retention might score below n_relocation_decision), the bonus must be large enough to override typical keyword-scoring gaps (~12-15 points). But in Case B, the same bonus would need to be small enough NOT to override the structural prerequisite preference for n_commercial_value over n_pricing. These are contradictory requirements: the bonus must simultaneously cross a ~10-point gap (Case A) and fail to cross the same ~10-point gap (Case B) without domain-specific knowledge of which gaps are "material" and which are "structural." |
|
||||
| Preserves pricing regression | **UNKNOWN** — Depends on whether the bonus falls below the commercial_value vs pricing score differential. Cannot determine without exact scoring numbers. |
|
||||
| Requires numeric weight tuning | **YES** — Any bounded bonus inherently requires a numeric weight. The question is what value satisfies all cases simultaneously, which cannot be answered without exhaustive regression testing across diverse scenarios. |
|
||||
| Semantic honesty | **LOW** — "Bonus of X points" has no defensible semantic meaning. Why 10? Why 15? There is no principled basis for any specific weight value — it's purely empirical tuning to avoid breaking existing tests. This violates criterion #4 (no domain-specific/heuristic logic). |
|
||||
|
||||
### Critical flaw:
|
||||
|
||||
A scoring bonus cannot simultaneously fix Case A and preserve Case B without knowing the score differential between candidates in each case beforehand. This requires tuning that is inherently case-dependent.
|
||||
|
||||
---
|
||||
|
||||
## CANDIDATE D — EXISTING PRIORITY
|
||||
|
||||
**Rule:** Reject preferred model targets entirely; keep current deterministic override for all cases.
|
||||
|
||||
### Assessment:
|
||||
|
||||
| Criterion | Answer |
|
||||
| ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Can existing deterministic signals solve 60B.6 generically | **NO** |
|
||||
| Why | The existing deterministic scorer (`scoreUnknownCandidate`) scores ALL unresolved unknowns by keyword matching + downstream count + unresolved parent penalty. There is NO existing signal for "material uncertainty that justifies continuation." The score for n_client_retention in Case A competes against n_relocation_decision (pre-existing, with accumulated text patterns from the entire decision history). Without materiality metadata, there is no mechanism to distinguish the material gap from the evaluation context. |
|
||||
|
||||
### Why this preserves the existing regression:
|
||||
|
||||
Yes — deterministic priority is preserved for ALL cases including Case B. But it also reverts the fix needed for Case A. The material factor identified by the reasoning layer is lost entirely.
|
||||
|
||||
---
|
||||
|
||||
## CRITICAL DISTINCTION
|
||||
|
||||
**Is "same-proposal-added + explicitly model-selected" a semantically meaningful signal, or merely a recency heuristic in disguise?**
|
||||
|
||||
### Answer: PARTIAL SIGNAL
|
||||
|
||||
### Why:
|
||||
|
||||
**What makes it meaningful:**
|
||||
When the model creates an unknown node AND selects it as the question target within the same reasoning turn, this carries genuine semantic content: the model's reasoning layer actively discovered this gap and intentionally named it for immediate follow-up. The dual action (creation + selection) signals _discovered material uncertainty_, not incidental documentation. This is stronger than recency alone because recency could capture any newly-created node regardless of whether it was selected.
|
||||
|
||||
**What makes it partial:**
|
||||
"Same-proposal-added" encompasses three distinct node types:
|
||||
|
||||
1. **Upstream prerequisites** — nodes that other nodes depend on (e.g., commercial_value)
|
||||
2. **Downstream consequences** — nodes that depend on other newly-created nodes (e.g., pricing)
|
||||
3. **Tangentially-related nodes** — nodes with no dependency relationships to other same-turn nodes (e.g., client_retention in Case A)
|
||||
|
||||
The signal is meaningless for distinguishing between types 1, 2, and 3. It treats a prerequisite, a consequence, and an independent material factor identically.
|
||||
|
||||
**What makes it fully actionable:**
|
||||
Combining the model-selection signal with structural analysis of dependency direction:
|
||||
|
||||
- Same-proposal-added + model-selected + **no upstream unknown dependencies from same proposal** = structurally independent material gap → prefer as target
|
||||
- Same-proposal-added + model-selected + **has upstream unknown dependencies from same proposal** = downstream consequence in a prerequisite chain → defer to deterministic prerequisite selection
|
||||
|
||||
This combination transforms the partial signal into a meaningful investigation-order check, not a recency rule. The structural dependency direction carries semantics about _investigation sequence_ (prerequisites before consequences), which is grounded in established reasoning methodology rather than temporal coincidence.
|
||||
|
||||
---
|
||||
|
||||
## WINNING MODEL
|
||||
|
||||
### Choice: B — PREFER MODEL-SELECTED SAME-PROPOSAL UNKNOWN ONLY WHEN STRUCTURALLY TIED TO CONTINUED DECISION
|
||||
|
||||
**Clarified implementation:** Prefer model-selected same-proposal-added unresolved unknown when it has no unresolved parent unknowns that were also added in this proposal turn. This is not a broad "structurally tied" requirement — it is specifically a prerequisite-dependency check within the current proposal's scope.
|
||||
|
||||
### Why:
|
||||
|
||||
1. **Fixes Case A:** `n_client_retention` has no upstream `depends_on` edge to any same-turn unknown. Only downstream edges (`may_cause` → option). No unresolved parent unknown from this turn → preferred as target.
|
||||
|
||||
2. **Preserves Case B regression:** `n_pricing` has an upstream `depends_on` edge from `n_commercial_value`, both added in this proposal → structural dependency prevents honor → deterministic selects `n_commercial_value`.
|
||||
|
||||
3. **No domain-specific keywords:** Uses only structural edge traversal (existing graph semantics), not text patterns or classification.
|
||||
|
||||
4. **No new schema:** `proposal.addedNodes`, node edge references, and `isSelectableUnresolvedUnknown` are all pre-existing.
|
||||
|
||||
5. **Does NOT make "newest unknown wins" a global rule:** Only applies when the model explicitly selects a same-proposal-added node AND it passes the structural independence check. Pre-existing nodes are unaffected. Nodes without explicit model selection are unaffected.
|
||||
|
||||
6. **Retains deterministic fallback:** When the honor-check fails (structural dependency exists) or the preferred target becomes invalid, existing `selectActiveUnknownCandidate` path is untouched.
|
||||
|
||||
### Smallest implementation boundary:
|
||||
|
||||
- One structural dependency check in the existing honor-model block (lines 3680-3695 of apply-proposal.js)
|
||||
- Minor clarification to prompt Rule 172 explaining the prerequisite-dependency constraint
|
||||
- Zero new schema fields, zero new edge types, zero new classification rules
|
||||
|
||||
---
|
||||
|
||||
## IMPLEMENTATION READINESS
|
||||
|
||||
### A — READY FOR BOUNDED IMPLEMENTATION
|
||||
|
||||
One unresolved question for precision:
|
||||
|
||||
> Should the structural check apply only to `depends_on` edges, or to any directed edge relationship (e.g., `may_cause`, `affects`)?
|
||||
> **Answer:** Only `depends_on` edges between unknown nodes. `may_cause` and `affects` represent consequence relationships in the opposite direction (unknown may cause → option change) and are not prerequisite chains. Investigating whether an unknown may cause something does not require resolving that thing first — only depends_on edges indicate genuine prerequisites.
|
||||
|
||||
---
|
||||
|
||||
## Scope validation
|
||||
|
||||
- Question wording/templates: NOT investigated
|
||||
- Materiality prompt rule: NOT investigated
|
||||
- Option scoring: NOT investigated
|
||||
- Utility models: NOT investigated
|
||||
- Provider behaviour: NOT investigated
|
||||
- Schema expansion: NOT required
|
||||
- Recommendation UI: NOT investigated
|
||||
- Full-suite failures: NOT investigated
|
||||
- Unrelated orchestrator issues: NOT investigated
|
||||
- Ollama calls: 0
|
||||
- Live API calls: 0
|
||||
- Vitest run: NO
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
|
||||
- Created: docs/experiment-60b10.md
|
||||
- Appended to: docs/current-handoff.md (below)
|
||||
- Implementation readiness: A — ready for bounded implementation
|
||||
|
||||
---
|
||||
|
||||
## Git status:
|
||||
|
||||
DOCUMENTATION COMMIT BLOCKED BY PARTIAL 60B.9 WORK
|
||||
(4 uncommitted files cannot be cleanly separated from the partial implementation)
|
||||
Reference in New Issue
Block a user