From 29d565372b68fd286d797b18b19864778d517d03 Mon Sep 17 00:00:00 2001 From: robbond Date: Fri, 14 Aug 2026 06:48:24 +0100 Subject: [PATCH] experiment: diagnose runtime question formulation path --- docs/current-handoff.md | 4 ++ docs/experiment-60b32.md | 148 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 152 insertions(+) create mode 100644 docs/experiment-60b32.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index bc06158..34c05fd 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -2915,3 +2915,7 @@ Experiment 60B.27 ran the exact product-launch live regression on the post-60B.2 --- Experiment 60B.31 was a live regression of 60B.29 on the post-60B.30 branch to verify that `Uncertainty over whether...` now preserves the full proposition end-to-end live. **Classification: B — FULL PROPOSITION PRESERVED BUT DIFFERENT PREFIX EXERCISED.** The model produced `n_enterprise_customer_signing` (kind=unknown, status=unknown) with a depends_on edge. Decision remains unresolved. However, the node description started with `Whether...` not `Uncertainty over whether...`, meaning 60B.30's targeted normalization was NOT exercised live. The final question ("What outcome would demonstrate enough value to justify launching?") is generic justification interrogative — same shape as 60B.29, full proposition lost from the question text but preserved in the source node description (source graph meaning intact). No rationale contamination in the final question. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed. The `Uncertainty over whether...` normalization is deterministic-proven (39/39 tests) but never exercised against the live model producing this exact prefix — follow-up needed to force that prefix live. + +--- + +Experiment 60B.32 was a code-path diagnosis of why the full runtime produces generic decision-justification questions for nodes whose descriptions start with bare "Whether..." — despite deterministic tests producing correct evidence-framed propositions. **Classification: D — ACTIVE CONTEXT DRIFT.** Three compounding factors in question-formulator.js cause the divergence: (1) extractMeaning's proposition detection requires label keywords (likelihood/status/probability/chance/risk/uncertainty) that "Enterprise customer signing decision" lacks, so the meaning strips "Whether" and isWhetherPropositionMeaning returns false; (2) hasDecisionValueLanguage becomes true because the description contains "value" within a parent-driven decision context; (3) extractActionPhrase scans ALL related nodes including the parent product-launch node, which supplies an action phrase ("launch") embedded in the generic template. The focused test avoids all three because its graph has no parent nodes and its label "Supplier renewal likelihood" contains the keyword "likelihood". Minimum corrective boundary: make extractMeaning recognize bare "Whether..." descriptions regardless of label keywords, so isWhetherPropositionMeaning returns true and blocks decision_threshold override. No production code changed. 0 Ollama calls. Pure code inspection. diff --git a/docs/experiment-60b32.md b/docs/experiment-60b32.md new file mode 100644 index 0000000..a78febb --- /dev/null +++ b/docs/experiment-60b32.md @@ -0,0 +1,148 @@ +# Experiment 60B.32 — Runtime Question Formulation Path Diagnosis + +**Branch:** `feature/proposition-prefix-over-v0.32` +**Starting HEAD:** clean (after 60B.31) +**Date:** 2026-08-14 +**Status:** COMPLETE — Classification: **D — ACTIVE CONTEXT DRIFT** + +## Objective + +Answer exactly: + +> Where does the full runtime diverge from the deterministic question-formulator path, causing the correct selected node to end with a generic decision-justification question? + +## Fixed Input (60B.31) + +``` +The revenue and launch-cost estimates are good enough for the decision. The remaining issue is one prospective enterprise customer. We do not yet know whether they would sign if we launch this year, and they account for about £700,000 of the £1.2 million expected annual revenue. +``` + +Live node added: +- **id:** `n_enterprise_customer_signing` +- **label:** `Enterprise customer signing decision` (or variant with "decision" at end) +- **description:** `Whether the prospective enterprise customer will commit this year, because resolving this uncertainty is needed to decide if launching this year provides superior net value over waiting twelve months.` + +## 60B.32 Findings + +### Root Cause: extractMeaning proposition detection depends on label keywords + +In `question-formulator.js` line 120-127 of `extractMeaning`: + +```js +if ( + /\b(status|likelihood|probability|chance|risk|uncertainty)\b/i.test( + String(node?.label || ""), + ) && + /^whether\s+/i.test(strippedDescription) +) { + return sentenceCase(extractWhetherProposition(strippedDescription)); +} +``` + +The proposition-extraction path requires the **label** to contain one of: status, likelihood, probability, chance, risk, uncertainty. + +The focused test (line 896-914) uses label `"Supplier renewal likelihood"` — contains "likelihood" ✓ → meaning starts with "Whether..." → `isWhetherPropositionMeaning(meaning)` = true. + +The live 60B.31 node uses label `"Enterprise customer signing decision"` — contains none of those keywords ✗ → falls through to line 129-136 which strips "Whether" → meaning does NOT start with "Whether..." → `isWhetherPropositionMeaning(meaning)` = false. + +### Root Cause: Parent context bleeds into child formulation via extractActionPhrase + +In `selectInvestigationStrategy` (line 1593): + +```js +const actionPhrase = extractActionPhrase([ + ...resolvedValues, + ...relatedNodes.map((relatedNode) => relatedNode.value), + ...relatedNodes.map((relatedNode) => relatedNode.label), + ...relatedNodes.map((relatedNode) => relatedNode.description), + graph?.centralStatement, +]); +``` + +`extractActionPhrase` iterates over ALL related nodes including the parent `n_product_launch_decision`. The regex `\b(build|launch|adopt|buy|continue|proceed|invest in|fund)\s+([^.,;:]+)/i` matches words like "launch" in the parent's label/description, returning an action phrase from the **parent node**. + +This means the child node's question text embeds the parent's decision vocabulary ("launching"), not the child's own proposition. + +### Root Cause: hasDecisionValueLanguage wins over proposition semantics + +At line 1680-1695: + +```js +if ( + !selectedStrategy && + (hasCriteriaLanguage || + (hasDecisionValueLanguage && !isWhetherPropositionMeaning(meaning))) +) { + selectedStrategy = buildInvestigationStrategy({ + key: "decision_threshold", + ... + }); +} +``` + +Three conditions conspire: +1. `hasDecisionContext` is true (parent product-launch node exists) +2. `hasDecisionValueLanguage` is true ("value" in description text within decision context) +3. `!isWhetherPropositionMeaning(meaning)` is true (extractMeaning stripped "Whether") + +All three are true → selects `decision_threshold` strategy over evidence gathering. + +### Generic question origin + +**Function:** `buildQuestionFromStrategy` at line 1759 of `question-formulator.js` +**Pattern:** `"decision_threshold"` +**Family:** `"decision_threshold"` +**Template:** Uses `strategy.actionPhrase` from parent node's "launch" keyword +**Trigger:** `actionPhrase != null` (from parent context) → interpolates gerund form + +```js +return strategy.actionPhrase + ? `What outcome would demonstrate enough value to justify ${toGerundPhrase(strategy.actionPhrase)}?` + : "What outcome would be sufficient to justify this decision?"; +``` + +**Why it wins:** The decision_threshold condition (line 1680-1695) fires before evidence_gathering conditions (line 1726-1742). `hasDecisionValueLanguage` combines with `!isWhetherPropositionMeaning(meaning)` as a gate — and because extractMeaning didn't produce a "Whether..." meaning for this node, the gate passes. + +### Critical distinction between test and live + +The focused test's graph (via `makeGraphFor`) contains ONLY the single unknown node. No parent nodes exist. Therefore: +- `collectRelatedNodes` returns no ancestors with decision keywords +- `hasDecisionContext` checks only the single node + centralStatement → false (centralStatement defaults to "Decision context" which doesn't match `\b(whether to|build|launch|continue...)`) +- `actionPhrase` scans nothing relevant → null + +The live graph contains: parent product-launch decision node + child customer-signing unknown node. The parent provides both hasDecisionContext and actionPhrase via collectRelatedNodes. + +### Context comparison + +| Input | Focused deterministic test | Live runtime (60B.31) | +|---|---|---| +| node label | "Supplier renewal likelihood" | "Enterprise customer signing decision" | +| node description | "Whether the supplier will renew the contract." | "Whether the prospective enterprise customer will commit this year, because resolving this uncertainty is needed to decide if launching this year provides superior net value over waiting twelve months." | +| label keywords match | YES ("likelihood") | NO (none of status/likelihood/probability/chance/risk/uncertainty) | +| extracted meaning starts with "Whether" | YES | NO | +| isWhetherPropositionMeaning | true | false | +| parent node exists | NO | YES (n_product_launch_decision) | +| hasDecisionContext | false | true | +| decisionContext flag in selectInvestigationStrategy | false | true | +| actionPhrase source | null (nothing to scan) | parent node's "launch" keyword | +| hasDecisionValueLanguage | false ("value"/"justify" not in text) | true (description contains "value", context is decision) | +| selectedStrategy key | evidence_gathering | decision_threshold | +| reasoningPattern | diagnosis (default, no decision context) | decision (parent triggers it) | + +### Minimum corrective boundary + +**Choice: D — REMOVE/CHANGE POST-FORMULATION OVERRIDE** (more precisely: make proposition semantics override decision-context heuristics) + +The fix must ensure that when a node description starts with "Whether..." (bare proposition), the proposition extraction in extractMeaning does NOT depend on label keywords. The description-level "Whether" itself is sufficient evidence of an unresolved proposition. + +Specifically, line 120-127 of question-formulator.js should be augmented: +- Either remove the label keyword requirement when description starts with "Whether..." +- Or add a separate extraction path that checks bare "Whether..." in description regardless of label + +### Would this preserve generic decision questions when the decision node itself is selected? + +YES — because only nodes whose **description** starts with "Whether" (not just any node with "decision" in its label) would get the proposition extraction boost. A product launch decision node has a different description format. + +### Would it preserve 60B.20 direct interrogative behaviour? + +LIKELY — because `isDirectInterrogativeMeaning` is checked at line 92 first, before any "Whether" handling. Direct interrogatives already bypass all the Whether-stripping logic.