From 865565b7af1dec9b1f0f2064330b1de1d4879f92 Mon Sep 17 00:00:00 2001 From: robbond Date: Fri, 14 Aug 2026 08:51:19 +0100 Subject: [PATCH] experiment: define active selector terminal guard --- docs/current-handoff.md | 4 + docs/experiment-60b41.md | 174 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 178 insertions(+) create mode 100644 docs/experiment-60b41.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index a399d9c..87dd149 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -2964,3 +2964,7 @@ Experiment 60B.38 was a read-only diagnosis of why `n_product_launch_decision` b --- Experiment 60B.40 performed read-only post-mutation guard location diagnosis for same-turn terminal-target survival (known status). **Classification: B — PREFERRED TARGET NEEDS EXPLICIT POST-MUTATION REVALIDATION.** Traced all five post-mutation sources that can supply the final selectedQuestion node after applyGraphUpdate (line 3671): Source A (selectActiveUnknownCandidate, line 3722) has zero status filtering; Source B (isSelectableUnresolvedUnknown preservation at line 3764) excludes ["resolved", "contradicted"] but not "known"; Source C (model-selection honour at line 3984) has same gap; Source D (remainingUnknownExists inline check at line 3710) checks only kind + resolvedNodeIds, no status; Source E/F (selectPatternCompatibleUnknownCandidate and listUnresolvedUnknownCandidates) share the same ["resolved", "contradicted"] exclusion gap. Earliest safe guard point: line 3704 — updatedSituationGraph exists, proposal accepted, selected target can still be discarded, question not yet finalized. Fallback behaviour confirmed as C (both A+B): selectAnotherUnresolvedCandidate when one exists, return null when none remain. Minimum corrective boundary: E (minimum combination) — add "known" to isSelectableUnresolvedUnknown exclusion list AND add known exclusion to remainingUnknownExists inline check at line 3710-3711. Leaves residual gap in selectActiveUnknownCandidate if all candidates cascade to known status. One unresolved question requires clarification before readiness. Full analysis in docs/experiment-60b40.md. No production code changed. 0 Ollama calls. Pure code inspection. + +--- + +Experiment 60B.41 was a read-only diagnosis of whether `selectActiveUnknownCandidate` must independently exclude terminal-status nodes (known/resolved/contradicted) for the 60B.37 closure path to be correct and fallback selection to remain semantically sound. **Classification: A — SELECTOR ITSELF MUST FILTER TERMINAL STATUS.** The selector at utils.js:593 filters only `kind === "unknown"` and `!resolvedNodeIds.includes(n.id)`. Zero status filtering exists. Key findings: (1) Scoring (`scoreUnknownCandidate`) is completely blind to node.status — it uses text classification, downstream count, and upstream unresolved dependency count, none of which inspect status. A known-status unknown-kind node scores identically to an unknown-status one. (2) In 60B.37's exact scenario: if no other genuine unresolved unknown remains, the selector would return the known decision node as the sole candidate, recreating the stale-target bug post-guard. (3) Adding terminal-status filtering does NOT affect genuine fallback candidates or prerequisite-first ordering because filtering happens before scoring and all genuine unknown-status nodes pass unchanged. (4) Five independent locations share the same gap pattern (`["resolved", "contradicted"]` exclusion without "known"): isSelectableUnresolvedUnknown, listUnresolvedUnknownCandidates, carriedActiveUnknownStillUnresolved, remainingUnknownExists inline check, and selectPatternCompatibleUnknownCandidate. This is PARTIAL overlap — different contracts serving different operational contexts but sharing the same systematic gap. **Critical distinction: D — SHARED ELIGIBILITY CONTRACT IS REQUIRED.** **Minimum implementation model: A — add terminal-status filter to selectActiveUnknownCandidate.** The selector must independently guard because it has zero caller-enforced input protection and known-status nodes bypass resolvedNodeIds entirely (they transition via updatedNodes.newStatus). Implementation boundary: one line change at utils.js:596 — add `!["known", "resolved", "contradicted"].includes(n.status)` to the existing filter. Ready for bounded implementation. No production code changed. 0 Ollama calls. Pure code inspection. Full trace in docs/experiment-60b41.md. diff --git a/docs/experiment-60b41.md b/docs/experiment-60b41.md new file mode 100644 index 0000000..138befc --- /dev/null +++ b/docs/experiment-60b41.md @@ -0,0 +1,174 @@ +# Experiment 60B.41 — Does `selectActiveUnknownCandidate` need its own known-status guard? + +**Date:** 2026-08-14 +**Branch:** `feature/known-target-exclusion-v0.36` +**Objective:** Determine whether `selectActiveUnknownCandidate` must independently exclude `status = known` (and other terminal states) for the 60B.37 closure path to be correct and for fallback selection to remain semantically sound. + +--- + +## Checkpoint 1 — Exact selector contract + +``` +function: selectActiveUnknownCandidate(graph, resolvedNodeIds) +location: lib/graph/utils.js:593-642 +candidate source: graph.nodes (all nodes in the graph) + +kind filter: node.kind === "unknown" +status filter: NONE — zero status filtering. The inline filter is: + (n) => n.kind === "unknown" && !resolvedNodeIds.includes(n.id) +resolvedNodeIds filter: !resolvedNodeIds.includes(n.id) + +other eligibility filter: none — purely kind + resolvedNodeIds +scoring happens after filtering: YES — scoreUnknownCandidate runs on the already-filtered unresolved set at line 603 +``` + +**Scoring function analysis** (`scoreUnknownCandidate`, utils.js:332-361): +- `collectNodeText(node)` — node label/description text only +- `classifyUnknownPriority(text)` — keyword classification on text +- `findDependentNodes(graph, node.id).length` — downstream edge count +- `countIncomingUnknownDependencies(graph, node.id, resolvedNodeIds)` — upstream dep count + +None of these inspect `node.status`. A node's status field is completely invisible to scoring. + +``` +Can status=known enter scoring: YES +Can status=resolved enter scoring if absent from resolvedNodeIds: YES (theoretically possible via a bug in caller, but practically blocked by caller passing the correct resolvedNodeIds) +Can status=contradicted enter scoring if absent from resolvedNodeIds: YES (same theoretical possibility as resolved) +``` + +--- + +## Checkpoint 2 — 60B.37 fallback reconstruction + +**Scenario:** `n_product_launch_decision` becomes terminal (`status=known`) during the same mutation turn. No other genuine unresolved unknown remains in the graph. The post-mutation guard correctly discards it as proposal target, then the fallback path runs: + +``` +selectActiveUnknownCandidate(updatedSituationGraph, resolvedNodeIds) +``` + +**Candidates seen:** +- All `kind === "unknown"` nodes that are NOT in `resolvedNodeIds` +- `n_product_launch_decision` has `kind === "unknown"` and is NOT in `resolvedNodeIds` (known-status nodes use `updatedNodes.newStatus`, not `resolvedUnknownNodeIds`) +- Therefore `n_product_launch_decision` appears as the sole candidate + +**Would `n_product_launch_decision` still qualify:** YES — passes both filters: kind="unknown" ✓, not in resolvedNodeIds ✓ + +**Would it be returned:** YES — with no other candidates to compete against, it scores highest by default (only candidate). Without terminal-status filtering, its status is invisible to scoring and classification. + +**Would final selectedQuestion become non-null again:** YES — `newActiveUnknownNodeId` would be set to the known node's ID at line 3716-3719, and this would propagate through deterministicSelection → finalSelectedQuestion → result.selectedQuestion, recreating the 60B.37 stale-target bug exactly. + +--- + +## Checkpoint 3 — Genuine fallback case + +**Existing test/case:** `reproduce-multi-turn-investigation.harness.test.js:1388` (pre-anchored product-launch customer-signing fixture) +- Nodes: `n_product_launch_decision` (kind=unknown, status=unknown), `n_enterprise_customer_signing` (kind=unknown, status=unknown) +- This is a genuine two-candidate scenario + +**Remaining unresolved candidate:** `n_enterprise_customer_signing` (status=unknown, kind=unknown, not resolved) + +**Would known-status exclusion affect it:** NO — this node has `status === "unknown"`, so adding terminal-status filtering to the selector would still let it pass all filters. Its scoring is identical because status doesn't enter scoring logic. + +**Would prerequisite-first ordering change:** NO — prerequisite blocking depends on `hasUnresolvedSameProposalDependsOnPrerequisite` (apply-proposal.js:2209) which checks node.kind membership in `proposal.dependsOn`. This is independent of node status. No known-status exclusion could alter prerequisite-first ordering because it operates at the kind+resolved boundary, not the prerequisite boundary. + +--- + +## Checkpoint 4 — Duplicated eligibility logic + +**Choice:** PARTIAL — OVERLAPPING BUT DIFFERENT CONTRACTS + +**Why:** `isSelectableUnresolvedUnknown` and `selectActiveUnknownCandidate` share the same *intent* (find unresolved unknown nodes) but differ in their terminal-state handling: the predicate excludes `["resolved", "contradicted"]` while the selector has zero status filtering. However, they also serve different operational contexts — the predicate validates a single node ID by reference (used for preservation checks), while the selector enumerates and ranks all candidates from the graph. `listUnresolvedUnknownCandidates` shares the predicate's exclusion list. `carriedActiveUnknownStillUnresolved` mirrors the predicate's pattern inline. None of these functions treat "known" as terminal, creating a systematic gap across all five locations rather than a pure duplication. + +--- + +## Checkpoint 5 — Canonical rule placement + +### Candidate A — PATCH SELECTOR ONLY + +Add terminal-status exclusion directly inside `selectActiveUnknownCandidate`. + +``` +60B.37 safe: YES — The fallback candidate would exclude known/resolved/contradicted, preventing stale target re-selection. +Can return terminal nodes elsewhere: YES — isSelectableUnresolvedUnknown (line 1663), remainingUnknownExists inline (line 3710-3711), listUnresolvedUnknownCandidates (line 1677), carriedActiveUnknownStillUnresolved (line 3850) all have the same gap. +Preserves existing scoring: YES — status filtering is applied before scoring; nodes that already pass kind+resolved filters retain their scores unchanged. Adding one more filter cannot change relative ordering. +Semantic-drift risk: MEDIUM — fixes only one of five locations; other gaps remain silently active. +Implementation scope: SMALL — one line change inside the existing filter at utils.js:596. +``` + +### Candidate B — REUSE CANONICAL PREDICATE + +Make selector candidate eligibility equivalent to `isSelectableUnresolvedUnknown` or a shared helper with the same terminal-state semantics. + +``` +60B.37 safe: YES — Same correctness as Candidate A, but also fixes Sources D, E, F, G identified in 60B.40. +Can return terminal nodes elsewhere: NO — all five locations converge on the same canonical rule. +Preserves existing scoring: YES — filtering scope expands uniformly; no node's relative score changes. +Semantic-drift risk: LOW — eliminates the systematic gap across all paths, establishing a single source of truth for unresolved unknown eligibility. +Implementation scope: MEDIUM — requires changes to utils.js (selector) AND apply-proposal.js (remainingUnknownExists, carriedActiveUnknownStillUnresolved, listUnresolvedUnknownCandidates), plus updating isSelectableUnresolvedUnknown to include "known". +``` + +### Candidate C — LEAVE SELECTOR UNCHANGED + +Rely on callers/eligible-candidate chains to protect it. + +``` +60B.37 safe: PARTIAL — Would work only if remainingUnknownExists at line 3710-3711 is also fixed AND no other code path reaches the selector with a known candidate in its filter set. But the 60B.40 analysis (Sources D, F, G) shows multiple inline checks also have the gap. +Can return terminal nodes elsewhere: YES — Sources B (isSelectableUnresolvedUnknown), F (selectPatternCompatibleUnknownCandidate), and G (listUnresolvedUnknownCandidates) all pass known-status through. +Preserves existing scoring: LIKELY — unchanged selector preserves current behavior; risk is in unguarded callers, not the selector itself. +Semantic-drift risk: HIGH — relies on fragile assumption that callers always provide correct filtered input. No defense-in-depth. +Implementation scope: SMALL (selector side) / LARGE (to actually fix — would require fixing all callers). +``` + +--- + +## Critical distinction + +**Choice: D — SHARED ELIGIBILITY CONTRACT IS REQUIRED** + +Why: The gap (`known` not treated as terminal) exists across five independent locations with identical filtering logic. Fixing only one is a band-aid; the remaining four continue to silently accept known-status nodes as eligible unresolved unknowns. A shared predicate eliminates the systematic inconsistency at its root rather than treating each symptom individually. + +--- + +## Minimum implementation model + +**Choice: A — add terminal-status filter to selectActiveUnknownCandidate** + +Why: For the specific question of this experiment (does 60B.41 require a fix to the selector itself?), the answer is definitively YES. The selector MUST independently exclude terminal statuses because: +1. It has zero status filtering today — the only filters are kind and resolvedNodeIds +2. Known-status nodes bypass resolvedNodeIds (they use updatedNodes.newStatus, not resolvedUnknownNodeIds) +3. No caller guarantees filtered input before reaching the selector +4. Adding `status !== "known" && status !== "resolved" && status !== "contradicted"` to the filter prevents 60B.37 without affecting any genuine unresolved candidate + +**Would genuine unresolved fallback still work:** YES — genuine unknown-status nodes pass all filters unchanged. Their scoring is identical (status doesn't enter scoring). Prerequisite-first ordering is unaffected. + +**Would prerequisite-first scoring remain unchanged:** YES — filtering adds a gate before scoring, not during it. No node's score or rank changes; only the candidate set shrinks by removing terminal nodes that would have been invisible to scoring anyway. + +**Would valid closure proposals remain accepted:** YES — `validateSelectedQuestion` (pre-mutation validation) is untouched. The filter only applies post-mutation selection. A customer-signing closure proposal that was valid pre-mutation still passes all filters post-mutation because the node's status hasn't changed. + +--- + +## Implementation readiness + +**Choice: A — READY FOR BOUNDED IMPLEMENTATION** + +The question is answered definitively. The selector must add terminal-status filtering. One unresolved follow-on question remains for separate treatment: whether `isSelectableUnresolvedUnknown` and other predicate functions also need `"known"` added to their exclusion lists (they do, but that is a scope decision beyond 60B.41). + +**Smallest implementation boundary:** +Add status filter to `selectActiveUnknownCandidate` at utils.js:596. Change line 596 from: +```js +(n) => n.kind === "unknown" && !resolvedNodeIds.includes(n.id) +``` +to: +```js +(n) => n.kind === "unknown" && + !["known", "resolved", "contradicted"].includes(n.status) && + !resolvedNodeIds.includes(n.id) +``` + +Production code changed: NO +Tests changed: NO +Prompt changed: NO +Schema changed: NO +Ollama calls: 0 +Live API calls: 0 +Vitest run: NO