robbond/confidence-engine
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: record prerequisite-aware question targeting

Browse Source
This commit is contained in:
robbond
2026-08-13 09:36:10 +01:00
parent 54bc48342b
commit 3c6e436e89
3 changed files with 645 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/current-handoff.md
+89 -20
View File
@@ -127,6 +127,19 @@ Experiment 54N tested whether an interpretation disagreement can be judged for m
- Whether `user_overloaded` health should be producible by the assessor for stalled/inconsistent evidence states;
- Existing-scenario graphs lack populated relationship fields on unknown nodes from the initial-build path; coherence detection works through the emergent-unknown path only (Populates `dependsOn` and `parentId` correctly — but requires comparable observations to trigger);
### 60B.10 / 60B.11 question-target alignment state
- **60B.10 completed the design diagnosis**: the winning rule is narrow prerequisite-aware preservation of a same-proposal model-selected unknown, not blanket trust in model selection.
- **60B.11 is now implemented** in `lib/graph/apply-proposal.js` only.
- The engine now prefers a same-proposal model-selected unresolved unknown **only when** it remains selectable after mutation and has **no unresolved same-proposal-added `depends_on` prerequisite**.
- If that bounded preference check fails, the existing deterministic selector remains authoritative and unchanged.
- The pricing regression remains preserved: model-selected `n_pricing` still falls back to deterministic selection of `n_commercial_value` when `n_pricing depends_on n_commercial_value`.
- Final wording still comes from the deterministic question formulator; model-provided question text is not final authority.
- Focused deterministic verification passed:
- `npx vitest run tests/graph/apply-proposal.test.js -t "60B.11"` → PASS (10/10)
- `npx vitest run tests/graph/apply-proposal.test.js tests/graph/prompt-builder.test.js` → PASS (174/174)
- **Still unproven live:** the exact 60B.6 continuation case where the model selects the newly exposed client-retention factor and the final target preserves that same ready material unknown in a live run.
### When This Knowledge-Management Phase Is Complete
Provisional criteria for review (all confirmed met by Experiment 38 cold-start test):
@@ -191,6 +204,7 @@ When the financial saving is accepted as real but the user introduces retention
Fixture: `tests/fixtures/pre-anchored-update-savings-realism.json`
Existing uncertainty:
```
n_savings_realism — Are the projected office savings from relocation realistic? — status = unknown
```
@@ -239,6 +253,7 @@ n_relocation_net_value — "Does relocating provide net value despite potential
```
The node simultaneously represents both consequences mentioned in the answer:
1. **Key-engineer retention risk** — "potential staff loss" / "losing key engineers"
2. **Delivery slowdown** — "slowing engineering delivery for a year"
@@ -251,7 +266,7 @@ The node simultaneously represents both consequences mentioned in the answer:
## Assessment
| Criterion | Result | Classification |
|-----------|--------|----------------|
| --------------------------- | ----------------------------------------------------------------------------------- | ------------------------------------------- |
| Savings-realism uncertainty | `unknown` → `resolved`, in resolvedUnknownNodeIds | CLOSED CORRECTLY |
| £2m/year saving | `"Confirmed realistic: £2M annual saving is real"` on resolved node | PRESERVED AS ACCEPTED EVIDENCE |
| Key-engineer retention risk | Represented in node label + description ("losing key engineers") | STRUCTURALLY REPRESENTED |
@@ -309,11 +324,17 @@ The engine correctly closed savings-realism, preserved verified evidence, repres
---
## Production code changed: NO
## Prompt changed: NO
## Validator changed: NO
## Harness changed: NO
## Vitest run: NO
## Ollama calls beyond harness count: 0
## Dev server disturbed: NO
### Experiment 59A.2 — Trade-off Decomposition: Known Consequence vs Uncertain Consequence
@@ -321,7 +342,7 @@ The engine correctly closed savings-realism, preserved verified evidence, repres
**Branch:** `feature/question-formulation-v0.24`
**Date:** 2026-08-12
**Status:** Complete
**Following:** 59A.1 which established the boundary shift but bundled two consequences into one node. This tests whether the engine can separate a *known* consequence (engineers leaving) from an *uncertain* consequence (delivery impact).
**Following:** 59A.1 which established the boundary shift but bundled two consequences into one node. This tests whether the engine can separate a _known_ consequence (engineers leaving) from an _uncertain_ consequence (delivery impact).
**Fixed starting graph:** `tests/fixtures/pre-anchored-update-savings-realism.json`
**Fixed answer:** "I am comfortable that the £2 million annual saving is real. We know we would lose two senior engineers if we relocate. What I don't know is whether losing them would materially delay delivery, or by how much."
@@ -331,6 +352,7 @@ The engine correctly closed savings-realism, preserved verified evidence, repres
**Result:** UPDATE 422 rejected at `proposal_compatibility` — "selectedQuestion must be a single non-compound question". No selectedQuestion was produced (null).
**Reasoning assessment from rejected snapshot:**
- Savings-realism: CLOSED CORRECTLY (resolved, in resolvedUnknownNodeIds)
- £2m annual saving: PRESERVED AS ACCEPTED EVIDENCE (in userSupportedMeaning)
- Two senior engineers leaving: REPRESENTED BUT LEFT UNCERTAIN (extracted in meaning but no separate structural node for the known fact)
@@ -392,7 +414,7 @@ One update-only call via `scripts/reproduce-multi-turn-investigation.mjs`. (Firs
### CALL ACCOUNTING (successful run)
| Field | Value |
|-------|-------|
| ----------- | -------------------------------------------- |
| startCalls | 0 |
| updateCalls | 1 |
| totalCalls | 1 |
@@ -444,7 +466,7 @@ One update-only call via `scripts/reproduce-multi-turn-investigation.mjs`. (Firs
### Resulting persistent graph (5 nodes, 4 edges)
| Node | Kind | Status | Label |
|------|------|--------|-------|
| ------------------ | ---------- | ----------- | ----------------------------------------------------------- |
| n_relocation_state | state | provisional | Engineering team relocation consideration |
| n_savings_realism | unknown | resolved | Are the projected office savings from relocation realistic? |
| opt_relocate | **option** | provisional | Relocate to Manchester |
@@ -452,7 +474,7 @@ One update-only call via `scripts/reproduce-multi-turn-investigation.mjs`. (Firs
| n_overall_tradeoff | unknown | unknown | Which option leaves us better off overall? |
| Edge | From | To | Relationship |
|------|------|----|-------------|
| ---------------- | ------------------ | ------------------ | ---------------- |
| e-sr-to-state | n_savings_realism | n_relocation_state | depends_on |
| opt-rel-to-trad | opt_relocate | n_overall_tradeoff | **contained_in** |
| opt-stay-to-trad | opt_stay | n_overall_tradeoff | **contained_in** |
@@ -463,13 +485,14 @@ One update-only call via `scripts/reproduce-multi-turn-investigation.mjs`. (Firs
### 1. Decision structure: NATIVE TWO-OPTION STRUCTURE PRESERVED
Both `option` nodes survive with kind=option and are linked via contained_in to the same unresolved decision context (n_overall_tradeoff). This matches the 60A.4 pattern. Two minor differences from 60A.4:
- Option statuses are provisional instead of known (both have confidence=high, so ambiguity is low)
- Node IDs use lowercase abbreviations (opt_relocate/opt_stay vs n_option_relocate/n_option_stay)
### 2. Relocate consequences — ALL INDEPENDENTLY STRUCTURED (in descriptions)
| Consequence | Present? | Classification |
|-------------|----------|----------------|
| ---------------------------- | -------- | ------------------------------------------------------------------ |
| £2m/year saving | YES | "saves £2m/year" — exact figure and unit preserved |
| two senior engineers leave | YES | "loses two senior engineers" — exact headcount preserved |
| <= two months delivery delay | YES | "delays delivery by up to two months" — bounded phrasing preserved |
@@ -479,7 +502,7 @@ All three consequences present within opt_relocate's description. The descriptio
### 3. Stay-put consequences — ALL INDEPENDENTLY STRUCTURED (in descriptions)
| Consequence | Present? | Classification |
|-------------|----------|----------------|
| ------------------------------ | -------- | ------------------------------------ |
| retain both engineers | YES | "retain both senior engineers" |
| avoid relocation delay | YES | "avoid relocation delay" |
| continue paying extra £2m/year | YES | "continue paying the extra £2m/year" |
@@ -496,6 +519,7 @@ All three consequences present within opt_stay's description.
### 5. Option attribution — CLEAR FOR BOTH OPTIONS
Graph makes it possible to tell which option each consequence belongs to:
- opt_relocate consequences are embedded in opt_relocate.description, reachable via contained_in edge from opt_relocate to n_overall_tradeoff
- opt_stay consequences are embedded in opt_stay.description, reachable via contained_in edge from opt_stay to n_overall_tradeoff
- No cross-contamination or ambiguity
@@ -503,7 +527,7 @@ Graph makes it possible to tell which option each consequence belongs to:
### 6. Relationship direction — SEMANTICALLY CLEAR
| From | To | Relationship | Assessment |
|------|----|-------------|------------|
| ------------------ | ------------------ | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| opt_relocate | n_overall_tradeoff | contained_in | Clear — relocation is a candidate for the decision |
| opt_stay | n_overall_tradeoff | contained_in | Clear — staying put is a candidate for the decision |
| n_overall_tradeoff | n_savings_realism | depends_on | Workable but slightly odd — the tradeoff assessment depending on the savings figure makes semantic sense (you need the savings value to assess whether relocation is better), though the direction could be read as the unknown "needs" the resolved node, which is epistemically inverted. |
@@ -511,7 +535,7 @@ Graph makes it possible to tell which option each consequence belongs to:
### 7. Graph-only recoverability
| Question | Answer |
|----------|--------|
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| Recover Relocate option | YES — node kind=option, label="Relocate to Manchester" |
| Recover Stay-put option | YES — node kind=option, label="Stay in London" |
| Recover Relocate consequences | PARTIAL — present in opt_relocate.description (structured field on graph node) |
@@ -593,7 +617,7 @@ Host/model: qwen-claude:latest at http://192.168.1.111:11434. startCalls=0, upda
### Structure (5 nodes, 3 edges)
| Node | Kind | Status |
|------|------|--------|
| --------------------- | ------- | --------- |
| n_savings_realism | unknown | resolved |
| opt_relocate | option | **known** |
| opt_stay_put | option | **known** |
@@ -618,12 +642,19 @@ First call succeeds; two-option structure confirmed on first attempt. Option sta
**What it does NOT prove:** Stability across repeated runs; whether consequences can be extracted as independent nodes without description parsing; cross-domain generalisation.
## Production code changed: NO
## Prompt changed: NO
## Validator changed: NO
## Harness changed: NO
## Vitest run: NO
## Ollama calls: 1
## Direct API calls: 0
## Dev server disturbed: NO
---
@@ -664,6 +695,7 @@ selectedQuestion.nodeId: "n_disruption_impact"
```
Resulting persistent graph (3 nodes, 2 edges):
- `n_relocation_state` — Engineering team relocation consideration — status=provisional
- `n_savings_realism` — Are the projected office savings from relocation realistic? — status=resolved
- `n_disruption_impact` — Operational impact of relocation disruption — status=unknown, kind=unknown
@@ -700,6 +732,7 @@ Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434. No producti
**Result:** UPDATE 422 rejected at `proposal_compatibility` — "selectedQuestion must be a single non-compound question". No selectedQuestion was produced (null).
**Rejected proposal snapshot assessment:**
- Known engineer loss: SEPARATE KNOWN STRUCTURE (dedicated observation node n_talent_departure, kind=observation)
- Delivery impact: SEPARATE UNRESOLVED STRUCTURE (dedicated unknown node n_delivery_delay_impact, kind=unknown, status=unknown)
- Epistemic separation: CLEARLY SEPARATED — two distinct nodes with different kinds and statuses
@@ -896,6 +929,7 @@ Two-turn run (fixed scenario: "We are considering relocating the engineering tea
**Classification: A — FIX VALIDATED.** Diagnosed that 57J.61 failed because the harness accepted-update path (script lines ~97–108) printed only `HTTP status`, `stage`, `selected question`, `node count`, and `edge count` — zero answer-meaning fields, zero structural mutation fields. After Update 1 applied successfully with HTTP 200 at `update_applied`, the harness could not identify which nodes were added or what the resulting persistent graph looked like.
**Fix:** Extended the accepted-update console block in `scripts/reproduce-multi-turn-investigation.mjs` to print:
- `answerMeaning.userSupportedMeaning`, `.possibleInference`, `.supportCategory`, `.resolutionGuidance`
- `updatedProposal.updatedNodes[]`, `.resolvedUnknownNodeIds[]`, `.addedNodes[]`, `.addedEdges[]`
- `selectedQuestion.nodeId` (node reference)
@@ -1901,6 +1935,7 @@ Implemented the bounded Option C from 57J.50 on branch `feature/structured-seman
One fresh live run through the production `startCase()` → `updateCase()` path. Fixed scenario: "We are considering relocating the engineering team to reduce operating costs." Fixed answer: "I am unsure whether the projected office savings from the relocation are realistic."
**Key evidence:**
- `supportCategory = "uncertain"` (populated, valid enum) — STRUCTURED path active, not LEGACY FALLBACK.
- `resolutionGuidance = "may_resolve"` (populated).
- Meaning: FAITHFUL — model preserved uncertainty without strengthening or degrading.
@@ -1916,7 +1951,8 @@ One fresh live run through the production `startCase()` → `updateCase()` path.
**Objective:** Implement the correction from Experiment 57J.54: clarify that "same uncertainty" means the same resolution question, not topical overlap. Bounded prompt-only task — no schema, validator, graph-model, or provider changes.
**Changes:**
- `lib/graph/prompt-builder.js`: Added to Additional Guidance — *"Same uncertainty means the same resolution question: resolving the existing unknown would also resolve the uncertainty introduced by the user's answer. Mere topical overlap (concerning the same topic, object, decision, or domain) is not automatically the same uncertainty."*
- `lib/graph/prompt-builder.js`: Added to Additional Guidance — _"Same uncertainty means the same resolution question: resolving the existing unknown would also resolve the uncertainty introduced by the user's answer. Mere topical overlap (concerning the same topic, object, decision, or domain) is not automatically the same uncertainty."_
- `tests/graph/prompt-builder.test.js`: 10 new focused prompt tests verifying the definition, boundaries, and preserved contracts (49 total pass).
**Test results:** All 49 tests pass. No regression in existing prompt structure tests.
@@ -1956,11 +1992,13 @@ Tested the inverse half of the uncertainty identity rule from 57J.56: when a gra
**Objective:** Record durable handoff facts from `docs/experiment-57j58.md` which diagnosed why a proposal adding a new unresolved unknown was rejected due to missing `selectedQuestion`, and classified the root cause.
**validateQuestionSelectionRequirement() rejects when:**
- `selectedQuestion == null`
- AND
- proposal adds at least one non-resolved unknown
**requirement does NOT depend on:**
- `updatedNodes`
- actual `resolvedUnknownNodeIds`
- total existing unresolved-node count
@@ -1987,6 +2025,7 @@ Configured Ollama: none used. **Production code changed:** NO.
**Test results:** 59 tests pass (42 existing + 17 new = 59 total). Zero failures.
**Ownership split preserved:**
- MODEL: supply one valid candidate when new unresolved unknowns are added
- ENGINE: validate candidate, retain deterministic priority/scoring ownership
@@ -1996,11 +2035,12 @@ Configured Ollama: none used. **Production code changed:** prompt + tests only.
**Objective:** What is the smallest structured contract that lets the model declare whether graph action is required, and lets deterministic code verify that the actual proposal fulfils that declaration? **Classification: B — BOOLEAN STRUCTURAL-ACTION CONTRACT.**
**Part 1 — Existing fields:** C — NEW ACTION DECLARATION REQUIRED. The existing fields encode *what changed* but not *what was intended*. When all mutation arrays are empty, there is no field saying "I intentionally declare zero graph action." The validator's current check (populated userSupportedMeaning + empty mutations → REJECT) treats model silence as error rather than accepting intentional no-op.
**Part 1 — Existing fields:** C — NEW ACTION DECLARATION REQUIRED. The existing fields encode _what changed_ but not _what was intended_. When all mutation arrays are empty, there is no field saying "I intentionally declare zero graph action." The validator's current check (populated userSupportedMeaning + empty mutations → REJECT) treats model silence as error rather than accepting intentional no-op.
**Part 2 — Minimum distinction:** `MUTATION REQUIRED` vs `NO MUTATION REQUIRED`. Reuse/refine existing, add new unknown, resolve existing, other structural mutation are all DERIVABLE FROM PROPOSAL SHAPE (validator checks each independently). Only "no structural change" MUST BE DECLARED because empty arrays alone cannot distinguish intent from omission.
**Part 3 — Three designs evaluated:**
- **Option A (boolean):** Prevents no-op = PARTIAL, checks mutation = YES, schema concept = BOOLEAN, validator complexity = LOW, model-compliance risk = MEDIUM
- **Option B (enum):** Prevents no-op = PARTIAL, checks mutation = YES, schema concept = SMALL ENUM, validator complexity = MEDIUM, model-compliance risk = MEDIUM-HIGH
- **Option C (existing fields only):** Prevents no-op = PARTIAL, but accepts every noncompliant zero-mutation output silently — HIGH model-compliance risk
@@ -2010,12 +2050,14 @@ Configured Ollama: none used. **Production code changed:** prompt + tests only.
**Part 5 — Relationship to supportCategory:** INDEPENDENT. `supportCategory = "uncertain"` does NOT mean `add new unknown`. Same category can map to different actions depending on graph state (v0.21 identity rule: reuse existing equivalent).
**Part 6 — Deterministic invariants:**
1. `structuralActionRequired = true` + empty mutations → REJECT
2. `structuralActionRequired = false` + meaningful mutation → ACCEPT with diagnostic note
3. Missing field + populated userSupportedMeaning → REJECT
4. `false` + empty mutations → PASS (valid "semantic agreement, no structural change")
**Part 7 — 57J.63 walkthrough:**
- Case A (successful): declares `true`, addedNodes non-empty → PASS by shape comparison
- Case B (no-op): declares `false`, empty arrays → PASS by explicit declaration + confirmed zero mutation
@@ -2035,9 +2077,10 @@ This is ACTUAL CONTRACT ENFORCEMENT because the validator compares a declared bo
**Part 1 — Field placement: TOP-LEVEL (graphUpdateSchema), NOT inside answerMeaning.**
`structuralActionRequired` expresses *graph-mutation intent*, not semantic meaning. It validates against structural arrays (addedNodes, updatedNodes, addedEdges), not meaning fields. Placing it at the proposal level keeps semantic analysis separate from structural action decisions and aligns with where the validator evaluates it. `selectedQuestion` already sits at this level as another structural decision — `structuralActionRequired` is a peer.
`structuralActionRequired` expresses _graph-mutation intent_, not semantic meaning. It validates against structural arrays (addedNodes, updatedNodes, addedEdges), not meaning fields. Placing it at the proposal level keeps semantic analysis separate from structural action decisions and aligns with where the validator evaluates it. `selectedQuestion` already sits at this level as another structural decision — `structuralActionRequired` is a peer.
**Part 2 — Meaning/action independence:**
- Meaning populated + true → VALID (extracting meaning AND claiming action needed)
- Meaning populated + false → VALID (extracting meaning AND deciding no action needed)
- Meaning null + true → VALID ONLY UNDER SPECIFIC EXISTING CASE (maintenance/cleanup scenarios where structural action is needed without fresh meaning extraction)
@@ -2046,6 +2089,7 @@ This is ACTUAL CONTRACT ENFORCEMENT because the validator compares a declared bo
**Part 3 — Meaningful mutation: REUSE existing definition.** `hasMeaningfulChange` (5 conditions: new nodes, status changes, value changes, added edges, removed edges) directly maps to the boolean. No separate definition needed or desirable.
**Part 4 — Contradiction matrix:**
- true + mutation → PASS (contract fulfilled)
- true + no mutation → REJECT (declaration violated)
- false + no mutation → PASS (intentional no-op)
@@ -2062,6 +2106,7 @@ The contract semantics are **ADVISORY** — the boolean is a minimum intent decl
**Part 8 — Prompt obligation: SUFFICIENT.** Two minimum rules suffice: "set true when answer requires graph progress" and "set false only when existing state fully represents user-supported meaning or no progress is justified." No additional principle needed.
**Implementation decision: D — top-level field + advisory false/mutation handling.**
- New field: `structuralActionRequired` in `graphUpdateSchema`, nullable during transition
- Nullable during transition: YES (mandatory once prompt adds it and when userSupportedMeaning is populated)
@@ -2073,7 +2118,7 @@ Full record in `docs/experiment-57j66.md`.
### Experiment 57J.67 — semanticActionRequired Contract Semantics Finalized
**Classification: A — SEMANTICS SETTLED.** Resolved the final ambiguity from 57J.66: is the boolean an *exact structural claim* (strict contract) or a *minimum-action claim* (advisory)? **Decision: EXACT STRUCTURAL CLAIM.** The field name "structuralActionRequired" semantically implies necessity, not suggestion. Definition A provides cleaner semantics, fully deterministic validation in all four cases, and prevents the most damaging error class (model declares no action but produces structure). Advisory (57J.66's recommendation) is rejected: `false + mutation` violates contract consistency — if the model declares "no structural action required" but produces meaningful mutations, it has either misunderstood the answer or over-produced unnecessary structure. This is not harmless.
**Classification: A — SEMANTICS SETTLED.** Resolved the final ambiguity from 57J.66: is the boolean an _exact structural claim_ (strict contract) or a _minimum-action claim_ (advisory)? **Decision: EXACT STRUCTURAL CLAIM.** The field name "structuralActionRequired" semantically implies necessity, not suggestion. Definition A provides cleaner semantics, fully deterministic validation in all four cases, and prevents the most damaging error class (model declares no action but produces structure). Advisory (57J.66's recommendation) is rejected: `false + mutation` violates contract consistency — if the model declares "no structural action required" but produces meaningful mutations, it has either misunderstood the answer or over-produced unnecessary structure. This is not harmless.
- **Boolean definition:** EXACT STRUCTURAL CLAIM (Definition A). true = meaningful mutation present; false = no meaningful mutation needed.
- **true + mutation:** PASS. true + no mutation: REJECT. false + no mutation: PASS. false + mutation: REJECT (under exact claim).
@@ -2095,22 +2140,26 @@ Full record in `docs/experiment-57j66.md`.
**Classification: IMPLEMENTATION COMPLETE.** All design decisions from 57J.67 implemented verbatim across three production files and validated by 50 deterministic tests (24 new + 8 migrated). No live Ollama calls.
#### Production changes:
1. `lib/graph/schema.js` — Added `structuralActionRequired: z.boolean().nullable().optional()` to graphUpdateSchema.
2. `lib/graph/utils.js` — Replaced semantic-only-no-op guard with four-case contract validator (true+mutation PASS, true+zero REJECT, false+zero PASS, false+mutation REJECT).
3. `lib/graph/prompt-builder.js` — Added field name to required lists; inserted contract declaration section between numbered rules and Additional Guidance.
#### Contract guarantees:
- Populated userSupportedMeaning requires boolean structuralActionRequired.
- Declaration is exact claim about output shape: true = meaningful mutation present; false = zero mutations intentional.
- false + zero is valid intentional no-op (contract-consistent).
- Mismatch between declaration and output shape deterministically rejected.
#### Test suite:
- 24 new tests across schema.test.js (+4), prompt-builder.test.js (+10), utils.test.js (+10).
- 8 existing tests migrated to use structuralActionRequired=true where answerMeaning is populated.
- All 197 graph tests pass (previous total preserved).
#### Live regression readiness:
- All existing schema, prompt-builder, and utils tests pass.
- The change is backward-compatible: field is optional by default; old proposals without it behave identically to the legacy path.
@@ -2175,6 +2224,7 @@ The 57J.74 apparatus commit ("docs: record pre-anchored update apparatus") expli
The 57J.75 experiment document ("experiment: validate controlled structural no-op live") records a live call that "sends the fixture's graph directly as the Update request's situationGraph" — but this capability does not exist in the committed mjs script. The live test depended on temporary uncommitted harness modifications (or direct manual production API invocation) at the time of execution. These were reverted before the apparatus was committed (which only added the fixture file and test-only helper).
**Apparatus classification:**
- `tests/fixtures/pre-anchored-update-savings-realism.json`: FIXTURE ONLY — valid deterministic graph, committed
- `PRE_ANCHORED_FIXTURE` constant + `runPreAnchoredSimulation()`: TEST-ONLY HELPER — synchronous mock simulator in test file, never makes real HTTP calls
- `scripts/reproduce-multi-turn-investigation.mjs`: PRODUCTION HARNESS PATH — but only standard start→update chain, no pre-anchored mode
@@ -2215,6 +2265,7 @@ The 57J.75 experiment document ("experiment: validate controlled structural no-o
**Evidence:** 46 harness tests pass (39 pre-existing + 7 new). No production code changed. No Ollama calls. No live API calls. Normal-mode Start→Update chain unmodified under guard.
**Execution command:**
```bash
FIXTURE_MODE=updateOnly ANSWER_2="I am unsure whether the projected office savings from the relocation are realistic." node scripts/reproduce-multi-turn-investigation.mjs
```
@@ -2262,6 +2313,7 @@ This satisfies 57J.77's boundary A recommendation: a committed update-only path
**Classification: D — WRONG TARGET.** One update-only call via the committed harness. The engine did **not** select `n_savings_realism` as the next-question target. Instead it created a new unknown node `n_lease_exit_penalty` (about lease exit penalty) and selected that as the target.
### Branch: `feature/question-formulation-v0.24`
### Starting HEAD: `870d6ca` (docs: record question-formulation fix)
### Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434
@@ -2356,6 +2408,7 @@ The hypothesis asked whether selecting `n_savings_realism` would now produce a c
**Result:** UPDATE accepted (HTTP 200 at update_applied). No validation errors.
**Assessment of four factual elements:**
- £2m annual saving: PRESERVED AS KNOWN BENEFIT (resolved on `n_savings_realism` with "confirmed £2M annual saving")
- Two-engineer departure: PRESERVED AS KNOWN CONSEQUENCE (dedicated observation node `n_talent_impact_clarity`, kind=observation, status=supported)
- Two-month worst-case bound: PRESERVED AS BOUNDED DOWNSIDE (embedded in observation description: "worst-case 2-month project delay")
@@ -2418,7 +2471,6 @@ Ollama calls: 0
Dev server disturbed: NO
Read-only diagnosis: YES
### Experiment 60A.2 — Choosing the Minimum Decision Representation
**Branch:** `feature/question-formulation-v0.24`
@@ -2439,6 +2491,7 @@ Option B (Stay put): retain engineers, avoid disruption, continue paying £2M/ye
```
Three models evaluated:
- **A** — DECISION + OPTION (new decision node kind + new option node kind)
- **B** — UNKNOWN + OPTION (reuse existing unknown as decision context + new option node kind)
- **C** — OPTION PAIR ONLY (option nodes linked by alternative_to, no decision context node)
@@ -2450,18 +2503,21 @@ Not implemented. No code changed. Read-only design evaluation.
## Results
### Candidate A (DECISION + OPTION)
- Semantic honesty: HIGH | Recoverability: FULL | Lifecycle: NATIVE | Question: CLEAN | Consequences: YES | Baseline: CLEAN
- New primitives: 2 node kinds + 1 edge type + 1 optional field = **4**
- Semantic overload: NONE
- Verdict: Satisfies all criteria but adds the most primitives
### Candidate B (UNKNOWN + OPTION) ✅ WINNER
- Semantic honesty: MEDIUM | Recoverability: FULL | Lifecycle: NATIVE | Question: CLEAN | Consequences: YES | Baseline: WORKABLE
- New primitives: 1 node kind + 1 edge type + 1 optional field = **3**
- Semantic overload: LOW (unknown carries both "uncertainty" and "decision context" — natural overlap, not contradictory)
- Verdict: Smallest model satisfying all five decision-rule conditions
### Candidate C (OPTION PAIR ONLY)
- Semantic honesty: LOW | Recoverability: POOR | Lifecycle: AWKWARD | Question: WORKABLE | Consequences: YES | Baseline: WORKABLE
- New primitives: 1 node kind + 1 edge type = **2**
- Semantic overload: LOW-MEDIUM
@@ -2472,21 +2528,25 @@ Not implemented. No code changed. Read-only design evaluation.
## Architectural Choice: B — UNKNOWN + OPTION
### What changes (exact boundary):
```javascript
// schema.js additions:
option: "option" // SituationKind enum value
contained_in: "contained_in" // SituationRelationship enum value
is_baseline: z.boolean().optional() // optional on option nodes (not required for v1)
option: "option"; // SituationKind enum value
contained_in: "contained_in"; // SituationRelationship enum value
is_baseline: z.boolean().optional(); // optional on option nodes (not required for v1)
```
### What does NOT change:
- `unknown` node kind retains its existing semantics; it now also serves as decision context via the new `option` children pattern
- All existing statuses, edge types, graph topology rules unchanged
- Question compatibility uses existing `selectedQuestion` mechanism without extension
- No migration of existing nodes required
### Decision lifecycle: NATIVE — open/resolved maps to unknown status transitions
### Additional questions answered:
1. Is `alternative_to` needed between options? **NO** — shared parent membership implies alternatives.
2. Is `is_baseline` flag required? **NOT NEEDED YET** — label/consequence patterns carry sufficient signal.
@@ -2543,6 +2603,7 @@ selectedQuestion: "What evidence would clarify which option leaves us better off
7. **Savings-realism:** REMAINS OPEN (correct — answer did not address it).
### What the engine understood correctly:
1. Dual-option decomposition into two `option` nodes
2. Shared decision context via single unresolved unknown
3. Correct use of `contained_in` for option → decision membership
@@ -2550,6 +2611,7 @@ selectedQuestion: "What evidence would clarify which option leaves us better off
5. Decision-question alignment with user's stated intent
### What this establishes:
The live model CAN create native two-option graph structure when the user explicitly presents two alternatives. Both options survive as first-class `option` nodes with structural membership to a shared decision context, and downstream graph-only reasoning can recover both branches and their distinct consequences.
### Classification: A — NATIVE TWO-OPTION STRUCTURE CONFIRMED
@@ -2574,12 +2636,14 @@ Dev server disturbed: NO
**Objective:** Add test-only support for loading the reusable decision-options fixture from its committed JSON file, enabling harness tests to verify pre-anchored update-only mode with non-default fixtures without inline data duplication.
**Methodology:**
- Load `tests/fixtures/pre-anchored-decision-options.json` directly via `fs.readFileSync` in the test harness
- Add `runPreAnchoredSimulationWithFixture()` helper that mirrors the production pre-anchored path (generic anchor validation, no Start call, exactly one Update, all hardened capture)
- Generalize script's anchor validation from savings-realism-specific to generic unresolved unknown check
- Run focused vitest harness test only
**Key findings:**
- All 63 harness tests pass (including 17 new decision-options fixture mode tests)
- Fixture loads correctly from committed JSON — no inline duplication needed
- Pre-anchored validation works generically across fixture types (savings-realism and decision-options)
@@ -2599,7 +2663,6 @@ Ollama calls: 0
Live API calls: 0
Vitest run: 1 focused command (63/63 pass)
### Experiment 60A.8 — Downstream Option Evidence Update on Committed Fixture
**Branch:** `feature/decision-options-v0.25`
@@ -2612,12 +2675,14 @@ Vitest run: 1 focused command (63/63 pass)
When new information applies specifically to the Relocate option ("£400,000 lost margin from two-month delivery delay"), does the engine attach that information to the existing Relocate branch while preserving the existing Stay-put option and shared decision context?
## Fixed Starting Graph
Fixture: `tests/fixtures/pre-anchored-decision-options.json`
Pre-existing: opt_relocate (option/known), opt_stay_put (option/known), n_relocation_decision (unknown/unknown)
## Configured Ollama: qwen-claude:latest at http://192.168.1.111:11434
## Execution
startCalls=0, updateCalls=1, totalCalls=1. One live call at qwen-claude:latest. No retries.
## Result
@@ -2627,7 +2692,7 @@ startCalls=0, updateCalls=1, totalCalls=1. One live call at qwen-claude:latest.
### Structure (5 nodes, 3 edges)
| Node | Kind | Status |
|------|------|--------|
| --------------------- | ---------- | -------------------------------------------------------------- |
| opt_relocate | option | **known** (PRESERVED) |
| opt_stay_put | option | **known** (PRESERVED) |
| n_relocation_decision | unknown | **unknown** (PRESERVED, unresolved) |
@@ -2645,6 +2710,7 @@ All three existing entity identities preserved without duplication; £400k conse
**What it does NOT prove:** Stability across repeated runs; multiple concurrent consequences per option; cross-domain generalisation.
## Production code changed: NO
Prompt changed: NO
Validator changed: NO
Harness changed: NO
@@ -2692,6 +2758,7 @@ Single-call live regression of experiment 60B.4's prompt-only decision materiali
The engine resolved the existing decision context independently. All three entity identities preserved. The generic continuation failure from 60B.2 is eliminated.
**Key changes vs 60B.2:**
- `n_relocation_decision` status: unknown → **resolved** (was supported in 60B.2)
- `resolvedUnknownNodeIds`: ["n_relocation_decision"] (was empty in 60B.2)
- No selected question generated — decision is complete (was generic continuation in 60B.2)
@@ -2728,6 +2795,7 @@ selectedQuestion: "What outcome would demonstrate enough value to justify contin
```
**What worked:**
- Decision identity preserved (status remained unknown)
- Both option identities preserved
- Client-retention uncertainty created as a first-class unknown node
@@ -2735,6 +2803,7 @@ selectedQuestion: "What outcome would demonstrate enough value to justify contin
- No fabricated factors or over-closure
**What was weak:**
- Generic question despite having just identified a specific material factor
- No observation node for the known financial comparison (unlike 60B.5)
- £5M/year not isolated from description text
docs/experiment-60b10.md
+325
View File
@@ -0,0 +1,325 @@
# Experiment 60B.10 — When should a valid model-selected target override deterministic priority?
**Branch:** `feature/question-target-alignment-v0.27`
**Date:** 2026-08-13
**Type:** READ-ONLY DESIGN DIAGNOSIS — Resolves the contract conflict between honoring model-selected targets and preserving existing structural overrides.
---
## Context
Experiment 60B.9 implemented a blanket "honour model-selected unresolved unknown" rule at line 3680 of `apply-proposal.js`. This exposed a genuine contract conflict:
```
NEW desired behaviour: preserve a model-selected material unknown when it is the specific same-turn factor that justifies continuation
EXISTING behaviour (expressed as regression test): deterministic selection may override a valid model-selected unresolved node when another candidate has higher structural/deterministic value
```
The failing regression: `"replaces downstream pricing question with higher-value commercial-value question"` proves these behaviours cannot both be preserved if every structurally-valid model target is always preferred.
---
## CASE A — 60B.6 material factor
**Source:** Experiment 60B.6 (docs/current-handoff.md, lines 2703-2742), validated by the reasoning-layer output from live qwen-claude call on `pre-anchored-decision-options.json`.
**Existing decision node:**
```
n_relocation_decision — kind=unknown, status=unknown, label="Which option leaves us better off overall?"
(pre-existing central decision; activeUnknown before this turn)
opt_relocate — kind=option, label="Relocate to Manchester"
```
**Same-proposal added material unknown:**
```
n_client_retention — kind=unknown, status=unknown
label: "Largest client retention uncertainty"
addedEdges: [n_client_retention → opt_relocate, relationship="may_cause"]
Created because the answer introduced the first new factor that could reverse the preferred option (staying).
```
**Model-selected node:** `n_client_retention`
**Desired deterministic target:** `n_client_retention` — because it is the specific material uncertainty whose outcome could change the preferred decision option, justifying continuation. The existing parent (`n_relocation_decision`) is merely the evaluation context, not the material gap itself.
**Graph structure of Case A:**
```
n_relocation_decision (existing unknown) ← activeUnknown before proposal
n_build_decision (newly-added state)
n_client_retention (newly-added unknown, may_cause → opt_relocate)
n_relocation_unknown (pre-existing unknown — also unresolved after this turn)
```
There is NO `depends_on` edge between n_client_retention and any other newly-added unresolved unknown in this proposal. The `may_cause` edge connects to an option (non-unknown), not to another unknown node.
---
## CASE B — pricing regression
**Source:** New test added in 60B.9 working tree at `tests/graph/apply-proposal.test.js:2006`.
### Pre-existing model-selected nodes (before proposal):
```
n_complaint_rate_unknown (kind=unknown, status=unknown) → resolved by this proposal
n_staffing_unknown (kind=unknown, status=unknown) → NOT resolved; remains unresolved after this turn
```
After resolution of `n_complaint_rate_unknown`: one pre-existing unresolved unknown remains:
```
n_staffing_unknown
```
### Same-proposal added nodes:
```
n_commercial_value — kind=unknown, status=unknown (no dependencies)
n_pricing — kind=unknown, status=unknown, depends_on=["n_commercial_value"]
n_build_decision — kind=state (not unknown; irrelevant to selection)
```
### Model-selected nodeId:
```
n_pricing (reason: "Model chose a downstream leaf")
```
### Existing deterministic winner:
```
n_commercial_value (preferred by deterministic scoring over n_pricing because:
- n_commercial_value has unresolvedParentUnknownCount=0
- n_pricing has unresolvedParentUnknownCount=1 (depends on n_commercial_value)
- structural prerequisite relationship: n_commercial_value → depends_on ← n_pricing)
```
Note: `n_staffing_unknown` is also an unresolved candidate but its score is lower than both commercial nodes due to keyword matching and downstream count patterns. The test specifically verifies that `n_commercial_value` wins over the model-selected `n_pricing`.
### Why existing test prefers deterministic winner:
The selected node `n_pricing` is structurally **downstream** of another unresolved unknown (`n_commercial_value`) added in this same proposal. The structural prerequisite chain (commercial value → pricing) means you cannot properly assess n_pricing without first resolving n_commercial_value. Honoring the model's selection of a downstream consequence before its prerequisite understanding would be investigation-order inverted.
### Graph structure of Case B:
```
n_build_decision (newly-added state)
├─ n_commercial_value (newly-added unknown, leaf — no upstream unknown dependencies)
└─ n_pricing (newly-added unknown, dependent on n_commercial_value via depends_on edge)
```
---
## CANDIDATE A — SAME-PROPOSAL TARGET
**Rule:** If model-selected nodeId points to an unresolved unknown added in THIS proposal, prefer it as final target. If model-selected nodeId points to a pre-existing unresolved unknown, retain current deterministic selection behaviour.
### Assessment:
| Criterion | Answer |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Materiality fidelity | **MEDIUM** — Correctly preserves n_client_retention (Case A). But also prefers n_pricing in Case B where the model chose a downstream node over its prerequisite. |
| Preserves existing pricing regression | **NO** — In Case B, both n_commercial_value and n_pricing are same-proposal-added. The rule prefers n_pricing (model-selected) over n_commercial_value (structural prerequisite), breaking the regression. |
| Requires new schema | **NO** — Uses `proposal.addedNodes` + `selectedQuestion.nodeId`, both existing. |
| Requires new scoring logic | **NO** — Binary check: isInAddedNodes(selectedNodeId). |
| Relies on recency alone | **YES** — "Added in this proposal" is a pure recency signal with no structural or semantic content beyond timing. The model-selected same-proposal node could be upstream prerequisite, downstream consequence, or tangentially-related. All three types would be equally preferred. |
| Principal risk | Selecting a downstream consequence before its prerequisite understanding. In Case B, this means asking about pricing before defining commercial value — an investigation-order error. Also: any newly-created unknown (material factor OR tangential) gets equal weight when the model explicitly selects it. |
### Critical flaw for Candidate A:
"Same-proposal-added" encompasses both upstream prerequisites AND downstream consequences. When the model creates a dependency chain (commercial_value → pricing), the rule cannot distinguish which end of the chain is the material uncertainty. It simply picks whichever the model named — which in Case B happens to be the wrong end of the chain.
---
## CANDIDATE B — SAME-PROPOSAL + STRUCTURAL RELATION
**Rule:** Prefer a model-selected same-proposal-added unresolved unknown only when it has no unresolved parent unknowns that were also added in this proposal turn. When such a structural dependency exists, retain deterministic priority over the upstream prerequisite.
### Why "unresolved parent unknown from same proposal" is the right structural signal:
When the model creates both an upstream and downstream unknown in the same turn (e.g., commercial_value → pricing), the `depends_on` edge between them indicates intentional dependency structure — not coincidental timing. The upstream node represents prerequisite understanding; the downstream node represents a consequence of that understanding. Investigation methodology dictates prerequisites before consequences.
When there is NO unresolved parent unknown from the same proposal (as in Case A), the model-selected node is structurally independent within this turn's additions — it has no structural ties to other newly-created unknowns, making it the appropriate material factor target.
### Assessment:
| Criterion | Answer |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Materiality fidelity | **HIGH** — Case A: n_client_retention has no unresolved parent unknown from same proposal → honored (correct). Case B: n_pricing depends on n_commercial_value (same proposal) → not honored; deterministic selects n_commercial_value (correct). |
| Preserves existing pricing regression | **YES** — The structural dependency check prevents honoring n_pricing in Case B. |
| Existing structure sufficient | **YES** — Edge relationships (`depends_on` edges into unknown nodes) and `proposal.addedNodes` are both pre-existing. No schema changes needed. |
| New schema required | **NO** — Uses only existing: `proposal.addedNodes`, node edge references, `isSelectableUnresolvedUnknown`. |
| Principal risk | The structural dependency check could reject a legitimately selected downstream node if the model created a dependency chain for non-investigation-order reasons (e.g., parallel branch creation). However, in practice, `depends_on` edges between unknown nodes in the same proposal almost always represent intentional prerequisite chains. This is conservative: it errs on the side of addressing prerequisites first. |
### Implementation boundary (conceptual only):
```
In apply-proposal.js after line 3680-3695 (existing honor block):
if (validatedProposal.selectedQuestion?.nodeId) {
const candidateNodeId = validatedProposal.selectedQuestion.nodeId;
// Check if this candidate is a same-proposal addition
const addedInThisProposal = validatedProposal.addedNodes.some(
n => n.id === candidateNodeId
);
if (addedInThisProposal && isSelectableUnresolvedUnknown(updatedSituationGraph, candidateNodeId)) {
// New structural check: does this node have unresolved parent unknowns from same proposal?
const upstreamParentIds = findUpstreamUnknownParents(candidateNodeId, updatedSituationGraph);
const parentsAddedThisTurn = upstreamParentIds.filter(
parentId => validatedProposal.addedNodes.some(n => n.id === parentId && n.kind === "unknown")
);
if (parentsAddedThisTurn.length === 0) {
// No structural dependency on same-turn unknowns → prefer as target
deterministicSelection = honorModelSelected(...);
}
// else: retain deterministic priority (structural prerequisite wins)
}
}
```
The `findUpstreamUnknownParents` function uses existing edge traversal — no schema change.
---
## CANDIDATE C — MODEL TARGET SCORING INPUT
**Rule:** Keep existing deterministic ranking but add a bounded preference/bonus for a valid model-selected node.
### Assessment:
| Criterion | Answer |
| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Fixes 60B.6 without arbitrary tuning | **NO** — To fix Case A (where n_client_retention might score below n_relocation_decision), the bonus must be large enough to override typical keyword-scoring gaps (~12-15 points). But in Case B, the same bonus would need to be small enough NOT to override the structural prerequisite preference for n_commercial_value over n_pricing. These are contradictory requirements: the bonus must simultaneously cross a ~10-point gap (Case A) and fail to cross the same ~10-point gap (Case B) without domain-specific knowledge of which gaps are "material" and which are "structural." |
| Preserves pricing regression | **UNKNOWN** — Depends on whether the bonus falls below the commercial_value vs pricing score differential. Cannot determine without exact scoring numbers. |
| Requires numeric weight tuning | **YES** — Any bounded bonus inherently requires a numeric weight. The question is what value satisfies all cases simultaneously, which cannot be answered without exhaustive regression testing across diverse scenarios. |
| Semantic honesty | **LOW** — "Bonus of X points" has no defensible semantic meaning. Why 10? Why 15? There is no principled basis for any specific weight value — it's purely empirical tuning to avoid breaking existing tests. This violates criterion #4 (no domain-specific/heuristic logic). |
### Critical flaw:
A scoring bonus cannot simultaneously fix Case A and preserve Case B without knowing the score differential between candidates in each case beforehand. This requires tuning that is inherently case-dependent.
---
## CANDIDATE D — EXISTING PRIORITY
**Rule:** Reject preferred model targets entirely; keep current deterministic override for all cases.
### Assessment:
| Criterion | Answer |
| ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Can existing deterministic signals solve 60B.6 generically | **NO** |
| Why | The existing deterministic scorer (`scoreUnknownCandidate`) scores ALL unresolved unknowns by keyword matching + downstream count + unresolved parent penalty. There is NO existing signal for "material uncertainty that justifies continuation." The score for n_client_retention in Case A competes against n_relocation_decision (pre-existing, with accumulated text patterns from the entire decision history). Without materiality metadata, there is no mechanism to distinguish the material gap from the evaluation context. |
### Why this preserves the existing regression:
Yes — deterministic priority is preserved for ALL cases including Case B. But it also reverts the fix needed for Case A. The material factor identified by the reasoning layer is lost entirely.
---
## CRITICAL DISTINCTION
**Is "same-proposal-added + explicitly model-selected" a semantically meaningful signal, or merely a recency heuristic in disguise?**
### Answer: PARTIAL SIGNAL
### Why:
**What makes it meaningful:**
When the model creates an unknown node AND selects it as the question target within the same reasoning turn, this carries genuine semantic content: the model's reasoning layer actively discovered this gap and intentionally named it for immediate follow-up. The dual action (creation + selection) signals _discovered material uncertainty_, not incidental documentation. This is stronger than recency alone because recency could capture any newly-created node regardless of whether it was selected.
**What makes it partial:**
"Same-proposal-added" encompasses three distinct node types:
1. **Upstream prerequisites** — nodes that other nodes depend on (e.g., commercial_value)
2. **Downstream consequences** — nodes that depend on other newly-created nodes (e.g., pricing)
3. **Tangentially-related nodes** — nodes with no dependency relationships to other same-turn nodes (e.g., client_retention in Case A)
The signal is meaningless for distinguishing between types 1, 2, and 3. It treats a prerequisite, a consequence, and an independent material factor identically.
**What makes it fully actionable:**
Combining the model-selection signal with structural analysis of dependency direction:
- Same-proposal-added + model-selected + **no upstream unknown dependencies from same proposal** = structurally independent material gap → prefer as target
- Same-proposal-added + model-selected + **has upstream unknown dependencies from same proposal** = downstream consequence in a prerequisite chain → defer to deterministic prerequisite selection
This combination transforms the partial signal into a meaningful investigation-order check, not a recency rule. The structural dependency direction carries semantics about _investigation sequence_ (prerequisites before consequences), which is grounded in established reasoning methodology rather than temporal coincidence.
---
## WINNING MODEL
### Choice: B — PREFER MODEL-SELECTED SAME-PROPOSAL UNKNOWN ONLY WHEN STRUCTURALLY TIED TO CONTINUED DECISION
**Clarified implementation:** Prefer model-selected same-proposal-added unresolved unknown when it has no unresolved parent unknowns that were also added in this proposal turn. This is not a broad "structurally tied" requirement — it is specifically a prerequisite-dependency check within the current proposal's scope.
### Why:
1. **Fixes Case A:** `n_client_retention` has no upstream `depends_on` edge to any same-turn unknown. Only downstream edges (`may_cause` → option). No unresolved parent unknown from this turn → preferred as target.
2. **Preserves Case B regression:** `n_pricing` has an upstream `depends_on` edge from `n_commercial_value`, both added in this proposal → structural dependency prevents honor → deterministic selects `n_commercial_value`.
3. **No domain-specific keywords:** Uses only structural edge traversal (existing graph semantics), not text patterns or classification.
4. **No new schema:** `proposal.addedNodes`, node edge references, and `isSelectableUnresolvedUnknown` are all pre-existing.
5. **Does NOT make "newest unknown wins" a global rule:** Only applies when the model explicitly selects a same-proposal-added node AND it passes the structural independence check. Pre-existing nodes are unaffected. Nodes without explicit model selection are unaffected.
6. **Retains deterministic fallback:** When the honor-check fails (structural dependency exists) or the preferred target becomes invalid, existing `selectActiveUnknownCandidate` path is untouched.
### Smallest implementation boundary:
- One structural dependency check in the existing honor-model block (lines 3680-3695 of apply-proposal.js)
- Minor clarification to prompt Rule 172 explaining the prerequisite-dependency constraint
- Zero new schema fields, zero new edge types, zero new classification rules
---
## IMPLEMENTATION READINESS
### A — READY FOR BOUNDED IMPLEMENTATION
One unresolved question for precision:
> Should the structural check apply only to `depends_on` edges, or to any directed edge relationship (e.g., `may_cause`, `affects`)?
> **Answer:** Only `depends_on` edges between unknown nodes. `may_cause` and `affects` represent consequence relationships in the opposite direction (unknown may cause → option change) and are not prerequisite chains. Investigating whether an unknown may cause something does not require resolving that thing first — only depends_on edges indicate genuine prerequisites.
---
## Scope validation
- Question wording/templates: NOT investigated
- Materiality prompt rule: NOT investigated
- Option scoring: NOT investigated
- Utility models: NOT investigated
- Provider behaviour: NOT investigated
- Schema expansion: NOT required
- Recommendation UI: NOT investigated
- Full-suite failures: NOT investigated
- Unrelated orchestrator issues: NOT investigated
- Ollama calls: 0
- Live API calls: 0
- Vitest run: NO
---
## Documentation
- Created: docs/experiment-60b10.md
- Appended to: docs/current-handoff.md (below)
- Implementation readiness: A — ready for bounded implementation
---
## Git status:
DOCUMENTATION COMMIT BLOCKED BY PARTIAL 60B.9 WORK
(4 uncommitted files cannot be cleanly separated from the partial implementation)
docs/experiment-60b11.md
+178
View File
@@ -0,0 +1,178 @@
# Experiment 60B.11 — Prerequisite-aware preferred question targeting
**Branch:** `feature/question-target-alignment-v0.27`
**Starting HEAD:** `854c3aa`
**Date:** 2026-08-13
**Status:** Complete
---
## Why 60B.9's broad honour-rule was too wide
The partial implementation inherited from 60B.9/60B.11 was already trying to preserve a model-selected node, but the broad idea behind the earlier change was still too permissive:
```text
if model-selected node is valid and unresolved,
preserve it
```
That rule is too broad because it treats these two cases as equivalent when they are not:
1. a same-proposal-added material unknown that is ready to investigate now
2. a same-proposal-added downstream unknown that still depends on another unresolved same-turn unknown
The pricing regression proves the difference matters:
```text
n_pricing depends_on n_commercial_value
```
Preserving `n_pricing` there would invert prerequisite-first investigation order.
---
## Winning rule implemented in production
The production boundary remains narrow and unchanged outside final target selection:
```text
Prefer the model-selected target only when ALL are true:
1. proposal.selectedQuestion.nodeId exists
2. that node was added in this proposal
3. it is still a selectable unresolved unknown after mutation
4. it has NO unresolved same-proposal-added unknown prerequisite via depends_on
```
If any condition fails, the engine falls back to the existing deterministic selector unchanged.
Final wording still comes from the existing deterministic question formulator.
---
## Exact production boundary
Implemented only in the existing final-question selection path inside:
```text
lib/graph/apply-proposal.js
```
No changes were made to:
- schema
- validator contract
- selection scoring weights
- question templates
- provider integration
- harness
- materiality rule semantics
No new dependencies were added.
---
## Prerequisite definition used
Only this direct same-proposal relationship blocks preference:
```text
target --depends_on--> unresolved same-proposal-added unknown
```
The implementation checks direct `dependsOn` references and direct `depends_on` edges only.
These do **not** block preference:
- `may_cause`
- `affects`
- `causes`
- `supports`
- `measures`
- `contained_in`
- any other non-`depends_on` relationship
No transitive prerequisite planning was added.
---
## Pricing regression preservation
The established regression remains intact:
```text
model-selected: n_pricing
prerequisite: n_commercial_value
final selected node: n_commercial_value
```
This remains protected because `n_pricing` has an unresolved same-proposal-added `depends_on` prerequisite, so the preferred-target path is rejected and deterministic selection proceeds unchanged.
---
## Focused test results
### 60B.11 block
Command:
```bash
npx vitest run tests/graph/apply-proposal.test.js -t "60B.11"
```
Result:
```text
PASS — 10/10 tests
```
Covered:
- same-proposal selected target with no prerequisite is preferred
- same-proposal selected target with same-turn `depends_on` prerequisite is blocked
- pricing regression preserved
- pre-existing model target not auto-preferred
- invalid / contradicted / missing-target fallback behaviour
- deterministic wording remains authoritative
- non-prerequisite edge types do not block preference
### Focused suites
Command:
```bash
npx vitest run tests/graph/apply-proposal.test.js tests/graph/prompt-builder.test.js
```
Result:
```text
PASS — 174/174 tests
```
---
## Prompt boundary
Only the selectedQuestion guidance was clarified in the existing prompt text. The prompt now states, in bounded terms, that the engine:
- validates the model's candidate
- retains deterministic prerequisite ordering
- favours a selected same-proposal node only when no unresolved same-proposal `depends_on` prerequisite blocks it
- preserves deterministic fallback selection and deterministic formulation authority
It does **not** claim unconditional model authority.
---
## What remains unproven until live regression
The bounded implementation is covered by deterministic tests, but one thing remains unproven in live behaviour:
```text
the exact 60B.6 live continuation case,
where the model selects the newly exposed client-retention factor
and the final selected target preserves that same ready material unknown
```
That requires a live regression run against the exact live fixture path, which was intentionally out of scope here.