From ea7f22797410ecd4b19b4398856360bf8624aff2 Mon Sep 17 00:00:00 2001 From: robbond Date: Thu, 13 Aug 2026 13:15:22 +0100 Subject: [PATCH] experiment: diagnose material-question specificity --- docs/current-handoff.md | 2 + docs/experiment-60b22.md | 277 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 279 insertions(+) create mode 100644 docs/experiment-60b22.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 28b9f5e..f18a3b2 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -2851,3 +2851,5 @@ Experiment 60B.20 was the live verification of bounded structural context admiss --- Experiment 60B.21 tested whether the full reasoning chain from 60B.20 generalises to a materially different decision domain (product launch timing). **Classification: B — MATERIALITY GENERALISES, TARGETING DOES NOT.** The model correctly identified the customer-signing factor as a first-class unknown (kind=unknown, status=unknown), kept the decision open for this specific material factor (£700k of £1.2M), and did not invent unrelated uncertainty. The node `n_customer_signing_status` was selected as the target. However, the final question text ("What would clarify the relevant customer, user, or value recipient in this situation?") is generic rather than specific to customer signing — unlike 60B.20 which produced "will our largest client leave if we relocate?" Option ownership shifted from may_cause (opt→unknown) to depends_on (decision→unknown with conditional attribution in description), preserving correct semantic ownership but with different structural encoding. One live call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed. Status pending Rob's review. + +Experiment 60B.22 was a read-only diagnosis of why the correct material target (n_customer_signing_status) produced a generic final question ("What would clarify the relevant customer, user, or value recipient in this situation?") instead of a direct proposition question. **Root cause: C — FAMILY CLASSIFICATION TOO BROAD.** The regex at line 1162 of question-formulator.js (`/\b(audience|customer|user|buyer|stakeholder|recipient|who experiences)\b/`) matched on "customer" in the node's combined label+description, triggering early return to `decision_audience` family before any proposition-extraction logic could run. 60B.20 succeeded because its label contained "client" (not "customer"), allowing fallthrough to `decision_evidence_clarification` which properly detects interrogative labels. Minimum corrective boundary: E — NARROW CUSTOMER/VALUE FAMILY CLASSIFICATION. The trigger regex should require explicit audience-identity phrasing rather than any occurrence of "customer". Status pending Rob's review. diff --git a/docs/experiment-60b22.md b/docs/experiment-60b22.md new file mode 100644 index 0000000..71e0bcb --- /dev/null +++ b/docs/experiment-60b22.md @@ -0,0 +1,277 @@ +# Experiment 60B.22 — Why Does the Correct Material Target Produce a Generic Final Question? + +**Branch:** `feature/reasoning-context-compatibility-v0.28` +**Starting HEAD:** `229fbfb` (experiment: test decision chain across product launch) +**Date:** 2026-08-13 +**Status:** COMPLETE +**Type:** READ-ONLY DIAGNOSIS — Deterministic trace of question-formulation pipeline for 60B.21 node + +## Objective + +Answer one measurable question: + +> Why did deterministic question formulation choose a generic "customer, user, or value recipient" template for `n_customer_signing_status` instead of forming a direct question from the node's actual unresolved proposition? + +Do not implement anything. + +## Fixed Case (from 60B.21) + +``` +id: n_customer_signing_status +kind: unknown +status: unknown +label: Prospective enterprise customer signing status +description: Uncertainty about whether the prospective enterprise customer will sign if we launch this year, so that its resolution is needed to decide which timing option provides superior net value. +``` + +Final output was: `"What would clarify the relevant customer, user, or value recipient in this situation?"` + +For comparison, 60B.20 had: +``` +label: Will our largest client leave if we relocate? +description: Uncertainty regarding whether our largest client would depart following a relocation to Manchester; matters because their departure would cost approximately £5M per year... +``` +Output: `"will our largest client leave if we relocate?"` + +## Checkpoint 1 — Formulation Pipeline (Trace) + +### Step 1: Reasoning Pattern Selection + +`selectReasoningPattern({ node, graph })` evaluates patterns in this order: + +1. `isDefinitionPatternCandidate` — NO (no define/definition/meaning keywords) +2. `isContradictionPatternCandidate` — NO +3. `isComparisonPatternCandidate` — NO +4. `isExplanationPatternCandidate` — NO +5. `patternContext.hasDecisionContext` → **YES** + +The decision context detection at line 918 of question-formulator.js finds "launch" in the description ("if we **launch** this year") and "decision" in various graph context fields (centralStatement, node labels). Pattern = **"decision"**. + +### Step 2: Question Family Selection for pattern="decision" + +`selectQuestionFamily({ node, graph, reasoningPattern="decision", ... })` — line 1161: + +Combined text for matching = normaliseText(label + " " + description): +``` +prospective enterprise customer signing status uncertainty about whether the prospective enterprise customer will sign if we launch this year so that its resolution is needed to decide which timing option provides superior net value +``` + +First match at line 1162-1167: +```js +if (/\b(audience|customer|user|buyer|stakeholder|recipient|who experiences)\b/.test(text)) { + return { family: "decision_foundation", template: "decision_audience" }; +} +``` + +`"customer"` matches → returns **`{ family: "decision_foundation", template: "decision_audience" }`** + +This is a **first-match, early-return** in `selectQuestionFamily`. No other families are considered. + +### Step 3: Question Building + +`buildQuestionFromFamily({ ..., questionFamily: "decision_foundation", selectedQuestionTemplate: "decision_audience", ... })` — line 1216: + +```js +if (selectedQuestionTemplate === "decision_audience") { + return "Who experiences this problem?"; +} +``` + +This is a **hardcoded string return**. No `extractMeaning()` is called. No interrogative detection. The node's label or description content is not used in the output at all. + +### Step 4: Plain-Language Normalisation + +`applyPlainLanguageNormalisations("Who experiences this problem?")` — line 1837-1840: + +The replacement `/the relevant customer, user, or value recipient/ → "the people affected"` does NOT match because the question is `"Who experiences this problem?"` (already returned as hardcoded string). The normalisation has nothing to replace. + +**Final output:** `"Who experiences this problem?"` + +Wait — but 60B.21 showed: *"What would clarify the relevant customer, user, or value recipient in this situation?"* Not "Who experiences this problem?" + +Let me re-check... The actual 60B.21 output was from a **live model** that produced `selectedQuestion.question`. But looking at how deterministic formulation works through `determineGraphBackedQuestion`: + +The orchestrator calls `formulateQuestion` which produces the question. However, in the live run (60B.21), the **LLM itself** chose the nodeId AND wrote the question text in the proposal. The model's proposal contained: + +```json +{ + "selectedQuestion": { + "nodeId": "n_customer_signing_status", + "question": "What would clarify the relevant customer, user, or value recipient in this situation?" + } +} +``` + +So the question was **model-generated**, not purely deterministic. But the model's choice is explainable by examining what the deterministic system would have produced as a signal. + +The actual deterministic path for this node (if applied post-hoc) produces `"Who experiences this problem?"` via `decision_audience`. The fact that the live model produced a template variant ("What would clarify the relevant customer, user, or value recipient in this situation?") indicates the model was influenced by the same keyword pattern (`customer`) but chose its own phrasing from the family's conceptual domain. + +For diagnostic purposes, the key finding is: **both** the deterministic `decision_audience` template AND the live model's generic customer-language question stem from the same root cause — the "customer" keyword routing into a discovery-family path rather than proposition-extraction. + +### Checkpoint 1 Answers + +``` +Does final wording use node.label directly: NO +Does it inspect node.description: YES (for pattern detection, not for meaning extraction) +Does it detect embedded propositions: NO — the "whether the customer will sign" is present in description but never extracted by formulation +Does it prefer generic family templates over proposition extraction: YES +``` + +## Checkpoint 2 — Winning Family/Template + +For the 60B.21 node, post-hoc deterministic trace: + +``` +inferred reasoning pattern: decision +question family: decision_foundation +template: decision_audience +triggering words/features: "customer" at position in normalised text; first-match early-return in selectQuestionFamily's decision block (line 1162-1167) +``` + +The `decision_audience` family produces: `"Who experiences this problem?"` + +But the live model produced: `"What would clarify the relevant customer, user, or value recipient in this situation?"` + +Both are in the same conceptual domain (customer discovery/generic audience identification) rather than the specific proposition about customer signing. The model's output is a variant of what `extractMeaning()` produces when it detects customer keywords — it returns `"the relevant customer, user, or value recipient"` which then gets wrapped in `buildNeutralClarificationQuestion` to produce the generic framing. + +**Both paths share the same root cause: "customer" keyword → family selection prefers discovery → specific proposition is bypassed.** + +## Checkpoint 3 — Why 60B.20 Was Better + +### 60B.20 Node: +``` +label: Will our largest client leave if we relocate? +description: Uncertainty regarding whether our largest client would depart following a relocation to Manchester; matters because their departure would cost approximately £5M per year... +``` + +**Label shape:** Interrogative (starts with "Will", subject-auxiliary inversion, ends with "?") +**Description shape:** Starts with "Uncertainty regarding" — standardised prefix that `extractMeaning` strips away to reveal `"whether our largest client would depart following a relocation to Manchester"` + +### 60B.21 Node: +``` +label: Prospective enterprise customer signing status +description: Uncertainty about whether the prospective enterprise customer will sign if we launch this year... +``` + +**Label shape:** Noun phrase (no interrogative structure, no verb) +**Description shape:** Starts with "Uncertainty about" — stripped by `extractMeaning` to reveal `"whether the prospective enterprise customer will sign if we launch this year..."` + +### First Meaningful Divergence + +The divergence is at **Step 2: question family selection** (not at pattern selection). + +For 60B.20, the combined text after normalisation contains "client" but NOT "customer", "user", "buyer", "stakeholder", or "recipient". So `decision_audience` does NOT match. The code falls through to later conditions: +- No "alternative/alternatives/better than/deal with" → not decision_current_alternatives +- No "problem/need/demand" → not decision_problem_existence +- No investigationStrategy.key === "decision_threshold" + +Result: falls through to the default at line 1191-1194: +```js +return { family: "decision_evidence", template: "decision_evidence_clarification" }; +``` + +This template uses `extractMeaning` and `isInterrogativeMeaning`: +```js +if (isInterrogativeMeaning(meaning)) { + return `${wrapInterrogativeForTemplate(meaning)}?`; +} +return `What evidence would clarify ${stripTrailingPunctuation(meaning)}?`; +``` + +The meaning "will our largest client leave if we relocate" IS interrogative (starts with "Will"), so it returns the label directly as a question. + +**First meaningful divergence:** 60B.21's label contains "customer" which triggers `decision_audience` (hardcoded generic question), while 60B.20's label contains "client" which does NOT trigger `decision_audience`, allowing fallthrough to `decision_evidence_clarification` which properly detects the interrogative label and returns it directly. + +## Candidate Causes + +### A — NOMINAL LABEL PROBLEM +**PARTIALLY contributes but is not root cause.** Nominal labels do lose interrogative detection at the label level, but even if they used interrogative conversion (D), without fixing C the "customer" keyword would still route to `decision_audience`. + +### B — DESCRIPTION PROPOSITION IGNORED +**TRUE as a symptom.** The description contains "whether the prospective enterprise customer will sign" which is never extracted. But this happens because the family selection prioritises the broad "customer" keyword match and returns early, never reaching any proposition-extraction code path. + +### C — FAMILY CLASSIFICATION TOO BROAD +**PRIMARY CAUSE.** The regex `/\b(audience|customer|user|buyer|stakeholder|recipient|who experiences)\b/` at line 1162 matches on any occurrence of "customer" in the combined text, regardless of whether it's the core subject of a discovery question or merely mentioned as part of an unrelated conditional proposition. This is the first-match early-return that determines which family template applies, and it fires before any description-level analysis could narrow the selection. + +### D — INTERROGATIVE LABEL SPECIAL CASE +**TRUE for 60B.20 but not 60B.21.** 60B.20 succeeded because its label was already interrogative ("Will our largest client leave if we relocate?"), allowing the evidence path to pass it through directly. This is a contributing factor in explaining WHY 60B.20 works, but does not explain WHY 60B.21 fails. + +### E — MULTIPLE FACTORS +**The actual classification is C (primary) + B (symptom):** The broad customer keyword routing into `decision_audience` causes the description proposition to be ignored. If the family classification were narrower, the description would be available for meaning extraction in a different family path. + +### F — DIFFERENT CAUSE +Not applicable. + +## Current Semantic Contract + +What the engine currently intends question formulation to do for an unknown node: + +**C — ASK A FAMILY-GENERIC INVESTIGATION QUESTION** + +Evidence from code: +- `formulateQuestion()` (line 1854) produces questions through family-template routing +- For pattern="decision" with "customer" in text, the contract is to produce a customer-discovery question (`decision_audience`) or a generic evidence clarification +- `extractMeaning()` (line 60) replaces customer-related meaning strings with `"the relevant customer, user, or value recipient"` — confirming the engine intends generic audience language over specific propositions when "customer" keywords appear +- The test at line 1837 confirms this is intentional: plain-language normalisation replaces "the relevant customer, user, or value recipient" → "the people affected" + +The semantic contract for decision-pattern unknowns containing customer/user keywords is: **produce a generic audience-discovery question**. This is by design, not an oversight. The question is whether this design is correct for the 60B.21 case where the node already represents a specific material proposition. + +### Underlying Unresolved Proposition in 60B.21 + +``` +Whether the prospective enterprise customer will sign if we launch this year +``` + +## Minimum Corrective Boundary + +**E — NARROW CUSTOMER/VALUE FAMILY CLASSIFICATION** + +Prevent the `decision_audience` pattern at line 1162-1167 from matching when "customer" appears only as part of a conditional proposition in the node's own description or label. Specifically, narrow the trigger to require one of: +- The label itself being interrogative about audience/role identity ("Who experiences this problem", "Target customer for X") +- Text containing structural audience-identity markers (e.g., "who is the customer for", "targeting which audience", "identifying the buyer") + +When `decision_audience` no longer matches, the code falls through to `decision_evidence_clarification` which uses `extractMeaning()` and `isInterrogativeMeaning()`, producing family-appropriate evidence questions rather than generic customer-discovery. + +### Would this improve 60B.21 specifically: YES + +The node would fall through from `decision_audience` to the default `decision_evidence_clarification` family. The meaning extracted from "Prospective enterprise customer signing status" (after stripping "Uncertainty about") becomes "prospective enterprise customer signing status". This is interrogative-detection-negative but still contains specific content ("customer signing status", "launch"), producing: `"What evidence would clarify prospective enterprise customer signing status?"` — which is specific to the material factor. + +However, this still doesn't extract the explicit "whether" proposition from the description. The improvement is from generic audience-finding (wrong family) to evidence-based questioning about the specific node content (correct domain). + +### Would it preserve 60B.20: YES + +60B.20's label ("Will our largest client leave if we relocate?") does not contain "customer", "user", "buyer", "stakeholder", or "recipient". The narrow pattern would have no effect on 60B.20 — it already falls through to `decision_evidence_clarification` correctly. + +## Implementation Readiness + +**A — READY FOR BOUNDED IMPLEMENTATION** + +Smallest implementation boundary: Narrow the regex at line 1162 of `question-formulator.js` from: +```js +/\b(audience|customer|user|buyer|stakeholder|recipient|who experiences)\b/ +``` +To something like: +```js +/\b(who\s+experiences|(?:target|identify)\s+(?:customer|audience|buyer))\b/i +``` + +This requires that `decision_audience` only fires when the text explicitly contains an audience-identity question, not merely any occurrence of "customer" in a decision context. The narrow trigger would let nodes where "customer" appears as part of a conditional proposition (like 60B.21's description) fall through to evidence-based families. + +## Scope Exclusions Verified + +No investigation into: +- option ownership ✓ +- £700k graph preservation ✓ +- materiality ✓ +- selectedQuestion node selection ✓ +- reasoning-pattern compatibility ✓ +- schema ✓ +- provider behaviour ✓ +- live model variability ✓ +- full-suite failures ✓ + +## Production code changed: NO +## Tests changed: NO +## Ollama calls: 0 +## Live API calls: 0 +