179 lines
4.4 KiB
Markdown
179 lines
4.4 KiB
Markdown
# Experiment 60B.11 — Prerequisite-aware preferred question targeting
|
|
|
|
**Branch:** `feature/question-target-alignment-v0.27`
|
|
**Starting HEAD:** `854c3aa`
|
|
**Date:** 2026-08-13
|
|
**Status:** Complete
|
|
|
|
---
|
|
|
|
## Why 60B.9's broad honour-rule was too wide
|
|
|
|
The partial implementation inherited from 60B.9/60B.11 was already trying to preserve a model-selected node, but the broad idea behind the earlier change was still too permissive:
|
|
|
|
```text
|
|
if model-selected node is valid and unresolved,
|
|
preserve it
|
|
```
|
|
|
|
That rule is too broad because it treats these two cases as equivalent when they are not:
|
|
|
|
1. a same-proposal-added material unknown that is ready to investigate now
|
|
2. a same-proposal-added downstream unknown that still depends on another unresolved same-turn unknown
|
|
|
|
The pricing regression proves the difference matters:
|
|
|
|
```text
|
|
n_pricing depends_on n_commercial_value
|
|
```
|
|
|
|
Preserving `n_pricing` there would invert prerequisite-first investigation order.
|
|
|
|
---
|
|
|
|
## Winning rule implemented in production
|
|
|
|
The production boundary remains narrow and unchanged outside final target selection:
|
|
|
|
```text
|
|
Prefer the model-selected target only when ALL are true:
|
|
|
|
1. proposal.selectedQuestion.nodeId exists
|
|
2. that node was added in this proposal
|
|
3. it is still a selectable unresolved unknown after mutation
|
|
4. it has NO unresolved same-proposal-added unknown prerequisite via depends_on
|
|
```
|
|
|
|
If any condition fails, the engine falls back to the existing deterministic selector unchanged.
|
|
|
|
Final wording still comes from the existing deterministic question formulator.
|
|
|
|
---
|
|
|
|
## Exact production boundary
|
|
|
|
Implemented only in the existing final-question selection path inside:
|
|
|
|
```text
|
|
lib/graph/apply-proposal.js
|
|
```
|
|
|
|
No changes were made to:
|
|
|
|
- schema
|
|
- validator contract
|
|
- selection scoring weights
|
|
- question templates
|
|
- provider integration
|
|
- harness
|
|
- materiality rule semantics
|
|
|
|
No new dependencies were added.
|
|
|
|
---
|
|
|
|
## Prerequisite definition used
|
|
|
|
Only this direct same-proposal relationship blocks preference:
|
|
|
|
```text
|
|
target --depends_on--> unresolved same-proposal-added unknown
|
|
```
|
|
|
|
The implementation checks direct `dependsOn` references and direct `depends_on` edges only.
|
|
|
|
These do **not** block preference:
|
|
|
|
- `may_cause`
|
|
- `affects`
|
|
- `causes`
|
|
- `supports`
|
|
- `measures`
|
|
- `contained_in`
|
|
- any other non-`depends_on` relationship
|
|
|
|
No transitive prerequisite planning was added.
|
|
|
|
---
|
|
|
|
## Pricing regression preservation
|
|
|
|
The established regression remains intact:
|
|
|
|
```text
|
|
model-selected: n_pricing
|
|
prerequisite: n_commercial_value
|
|
final selected node: n_commercial_value
|
|
```
|
|
|
|
This remains protected because `n_pricing` has an unresolved same-proposal-added `depends_on` prerequisite, so the preferred-target path is rejected and deterministic selection proceeds unchanged.
|
|
|
|
---
|
|
|
|
## Focused test results
|
|
|
|
### 60B.11 block
|
|
|
|
Command:
|
|
|
|
```bash
|
|
npx vitest run tests/graph/apply-proposal.test.js -t "60B.11"
|
|
```
|
|
|
|
Result:
|
|
|
|
```text
|
|
PASS — 10/10 tests
|
|
```
|
|
|
|
Covered:
|
|
|
|
- same-proposal selected target with no prerequisite is preferred
|
|
- same-proposal selected target with same-turn `depends_on` prerequisite is blocked
|
|
- pricing regression preserved
|
|
- pre-existing model target not auto-preferred
|
|
- invalid / contradicted / missing-target fallback behaviour
|
|
- deterministic wording remains authoritative
|
|
- non-prerequisite edge types do not block preference
|
|
|
|
### Focused suites
|
|
|
|
Command:
|
|
|
|
```bash
|
|
npx vitest run tests/graph/apply-proposal.test.js tests/graph/prompt-builder.test.js
|
|
```
|
|
|
|
Result:
|
|
|
|
```text
|
|
PASS — 174/174 tests
|
|
```
|
|
|
|
---
|
|
|
|
## Prompt boundary
|
|
|
|
Only the selectedQuestion guidance was clarified in the existing prompt text. The prompt now states, in bounded terms, that the engine:
|
|
|
|
- validates the model's candidate
|
|
- retains deterministic prerequisite ordering
|
|
- favours a selected same-proposal node only when no unresolved same-proposal `depends_on` prerequisite blocks it
|
|
- preserves deterministic fallback selection and deterministic formulation authority
|
|
|
|
It does **not** claim unconditional model authority.
|
|
|
|
---
|
|
|
|
## What remains unproven until live regression
|
|
|
|
The bounded implementation is covered by deterministic tests, but one thing remains unproven in live behaviour:
|
|
|
|
```text
|
|
the exact 60B.6 live continuation case,
|
|
where the model selects the newly exposed client-retention factor
|
|
and the final selected target preserves that same ready material unknown
|
|
```
|
|
|
|
That requires a live regression run against the exact live fixture path, which was intentionally out of scope here.
|