diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 347488f..650c112 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -2869,3 +2869,7 @@ Experiment 60B.22 was a read-only diagnosis of why the correct material target ( --- Experiment 60B.24 ran the exact product-launch live regression (same fixture as 60B.21) on the post-60B.23 branch to verify the proposition-specificity fix end-to-end. **Classification: A — LIVE QUESTION-SPECIFICITY FIX CONFIRMED.** The model produced `uncertain_enterprise_customer_signing` (kind=unknown, status=unknown) with depends_on edge from n_product_launch_decision and affects edge to opt_launch_this_year, preserving £700k significance in the node description. The final selected question ("whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k of the £1.2M expected annual revenue...") is proposition-specific — not generic audience wording. Core reasoning chain preserved, decision remains unresolved, no unrelated uncertainty invented. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed. + +--- + +Experiment 60B.25 was a read-only diagnosis of why the proposition-specific question produced "whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k..." (proposition-plus-rationale) instead of a clean conversational question. **Classification: C — BOTH A + B.** Two independent defects in `extractMeaning` (lib/graph/question-formulator.js): (A) **Rationale extraction too broad** at line 108 — when label contains "status" and description starts with "Whether", the function returns `sentenceCase(strippedDescription)` which includes the full description text after the semicolon, capturing explanatory rationale (~£700k context clause). No internal delimiter logic exists. (B) **No whether→direct-question conversion** — `isInterrogativeMeaning` at line 189 treats "whether" clauses as already interrogative and passes them through unchanged via `wrapInterrogativeForTemplate`, so the output is proposition+rationale with "?" appended, not a converted question. The first meaningful divergence from 60B.20 (which produced clean "will our largest client leave if we relocate?") is at extractMeaning line 108: 60B.20's interrogative label bypasses the status+whether branch entirely, while 60B.24's nominal "signing status" label triggers it, pulling in full description including rationale. **Smallest corrective boundary:** A+C — strip rationale on extraction (split on first semicolon), then let existing evidence framing produce "What evidence would clarify whether X?" or keep direct interrogative form via decision path line 1272-1273. No provider rewrite, no target selection change, no schema change needed. Implementation readiness: A. Full trace documented in docs/experiment-60b25.md. diff --git a/docs/experiment-60b25.md b/docs/experiment-60b25.md new file mode 100644 index 0000000..f3616fe --- /dev/null +++ b/docs/experiment-60b25.md @@ -0,0 +1,232 @@ +# Experiment 60B.25 — Diagnosis: Proposition-Plus-Rationale Instead of Clean Question + +**Branch:** `feature/question-family-specificity-v0.29` +**Date:** 2026-08-13 +**Status:** COMPLETE +**Type:** READ-ONLY DIAGNOSIS (no code changes) + +## Objective + +Answer one measurable question: + +> Why does deterministic formulation preserve the whole proposition-plus-rationale string instead of converting the unresolved proposition into a concise interrogative question? + +Input node (from 60B.24): +``` +label: "Prospective enterprise customer signing status" +description: "Whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k of the £1.2M expected annual revenue, so that resolving their intent is needed to assess the financial impact of launching now." +``` + +Produced question: +``` +whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k of the £1.2M expected annual revenue, so that resolving their intent is needed to assess the financial impact of launching now? +``` + +## Checkpoint 1 — Extraction Behaviour (full trace) + +### `extractMeaning(node)` trace for the fixed node: + +**Line 86:** `raw = "Prospective enterprise customer signing status Whether one prospective..."` + +**Line 87-89:** `meaning = stripTrailingPunctuation("Prospective enterprise customer signing status")` +→ `"Prospective enterprise customer signing status"` (no trailing punctuation to strip) + +**Line 92:** `hasAudienceIdentityQuestion(lowered)` → **NO** +None of the patterns match: "who is the customer", "target customer", "identifying the customer", etc. The word "enterprise customer" does not match any pattern — it's a noun modifier, not an audience-identity construct. + +**Line 96-97:** `strippedDescription = stripTrailingPunctuation(description)` +→ Full description text with no trailing punctuation change (it ends with period which gets stripped). + +**Lines 102-108 — THE KEY BRANCH:** +```js +if ( + /\b(status|likelihood|probability|chance|risk|uncertainty)\b/i.test("Prospective enterprise customer signing status") && // MATCHES "status" ✓ + /^whether\s+/i.test(strippedDescription) // MATCHES "Whether..." ✓ +) { + return sentenceCase(strippedDescription); // ← ALL text returned +} +``` + +Both conditions match. **This branch is taken.** + +`sentenceCase("Whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k...")` → +**lowercases first char, preserves everything else including rationale after semicolon** + +### Checkpoint 1 Answers: + +| Question | Answer | +|---|---| +| label considered? | **YES** — used to trigger the status regex | +| description considered? | **YES** — full text passed to sentenceCase on line 108 | +| returned meaning | `"whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k of the £1.2M expected annual revenue, so that resolving their intent is needed to assess the financial impact of launching now."` (first char lowered) | +| rationale stripped? | **NO** — line 108 returns full `strippedDescription` | +| semicolon boundary recognized? | **NO** — no split logic exists for description extraction | +| "so that" rationale recognized? | **NO** — no rationale marker detection in extractMeaning | + +## Checkpoint 2 — Interrogative Conversion (full trace) + +### `isInterrogativeMeaning(meaning)` trace: + +Input: `"whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k..."` + +- Line 170 (`wh-questions`): No match — starts with "whether", not who/what/where/when/how +- Lines 179-186 (aux inversion): No match — first word is "whether", not a modal auxiliary +- **Line 189: `/^whether\b/i.test(trimmed)` → YES ✓** + +Returns `true`. The engine recognises the meaning as already question-shaped. + +### `wrapInterrogativeForTemplate(meaning)` trace: + +Input: lowercased meaning string +- Line 198 (wh-questions): No match — starts with "whether" +- **Line 202: `isInterrogativeMeaning` → true** +- Returns: `stripTrailingPunctuation(meaning).trim()` = full proposition + rationale, no trailing punctuation + +### Decision evidence path trace (line 1271-1273): + +```js +if (isInterrogativeMeaning(meaning)) { // YES ✓ + return `${wrapInterrogativeForTemplate(meaning)}?`; +} +``` + +Result: `"whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k of the £1.2M expected annual revenue, so that resolving their intent is needed to assess the financial impact of launching now?"` + +### Checkpoint 2 Answers: + +| Question | Answer | +|---|---| +| Does engine recognise "whether..." as unresolved proposition? | **YES** — `isInterrogativeMeaning` returns true at line 189 | +| Does it convert "whether X..." into "Will/Does/Is X...?" | **NO** — no conversion logic exists; "whether" is treated as already interrogative | +| Does it merely append "?" | **YES** — direct from the full extracted string including rationale | +| Evidence framing applied? | **CONDITIONAL** — `buildEvidenceFallbackQuestion` would add "What evidence would confirm or rule out...", but in the decision reasoning path (line 1272-1273), interrogative means bypass the evidence template and go straight to append "?" | + +## Checkpoint 3 — Why 60B.20 Looked Better + +### 60B.20 source shape: +``` +label: "Will our largest client leave if we relocate?" +description: "Uncertainty regarding whether our largest client would depart following a relocation to Manchester; matters because their departure would cost approximately £5M per year..." +``` + +**Label is already interrogative:** "Will our largest client leave if we relocate?" + +### 60B.24 source shape: +``` +label: "Prospective enterprise customer signing status" +description: "Whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k of the £1.2M expected annual revenue, so that resolving their intent is needed to assess the financial impact of launching now." +``` + +**Label is nominal (noun phrase); proposition in description starting with "Whether"** + +### First meaningful divergence: `extractMeaning` line 108 + +In 60B.24, the condition at lines 102-107 fires because the label contains "status" AND the description starts with "Whether". This causes `extractMeaning` to return the **full** `strippedDescription` (proposition + rationale after semicolon). + +In 60B.20, the label is already interrogive ("Will our largest client..."). The condition at lines 102-107 does NOT fire because: +- Label contains no status/probability words (no "status" in "Will our largest client leave if we relocate?") +- Even though it has no "status", the label itself IS interrogative + +The extracted meaning for 60B.20 is the **label** ("Will our largest client leave if we relocate?"), not the description. This is already a clean question, so the template simply appends "?" to produce a valid output. + +### Quality difference cause: `description extraction` + `rationale contamination` + +The root divergence is in `extractMeaning`: +1. **60B.20** — Label is interrogative → meaning = label (clean) → output = label + "?" ✓ +2. **60B.24** — Label triggers status condition → meaning = full description including rationale after semicolon → output = proposition+rationale + "?" ✗ + +The quality difference comes from **description extraction capturing rationale** and the **absence of "whether→direct-question conversion"**. + +## Cause Classification: C (Both A + B) + +### A — RATIONALE EXTRACTION TOO BROAD +`extractMeaning` line 108 returns `sentenceCase(strippedDescription)` which includes everything after the semicolon. There is no internal delimiter logic for separating proposition from explanatory rationale. + +### B — NO WHETHER→QUESTION CONVERSION +The engine recognises "whether X" as already interrogative (line 189) and passes it through unchanged. No conversion to "Will/Does/Is X?" exists in the codebase. The `isInterrogativeMeaning` function treats "whether" clauses as complete interrogatives rather than treating them as unresolved propositions that need conversion. + +## Current Semantic Contract + +**For a selected unknown whose explicit meaning is "whether X", what should deterministic formulation ideally represent?** + +### C — Direct Interrogative: "Will/Does/Is X?" + +The current code's intent (lines 170-189, 202-203) is: +- If the extracted meaning is already interrogative (wh- question, aux inversion, or whether-clause), pass it through unchanged. +- The rationale for line 189 treating "whether" as complete interrogative was to prevent double-wrapping ("What would clarify are..."). + +However, this conflates two distinct semantic states: +1. **Direct interrogative** (e.g., "Will X happen?") — ready as a question +2. **Indirect interrogative / unresolved proposition** (e.g., "whether X will happen") — needs conversion + +The current contract treats both identically, which is why 60B.24's output preserves the indirect form with rationale contamination. + +## Candidate Evaluations + +### Candidate A — Strip Rationale Only + +Extract only: `"Whether one prospective enterprise customer will sign if we launch this year"` (before semicolon). Then preserve existing formulation behaviour. + +| Assessment | Value | +|---|---| +| Improves concision | **HIGH** — removes the entire explanatory clause | +| Produces conversational question | **NO** — "Whether one prospective enterprise customer will sign if we launch this year?" is still an indirect question (embedded/yes-no proposition form), not natural conversational English. The user would expect "Will one...?" | +| Risk of losing context | **LOW** — rationale is explanatory, not material. Materiality lives in the graph structure (£700k on edge/unknown node) | + +### Candidate B — Deterministic Whether→Interrogative Conversion + +Convert simple explicit propositions: +- Input: `"whether the customer will sign"` +- Output: `"Will the customer sign?"` + +| Assessment | Value | +|---|---| +| Semantic robustness | **MEDIUM** — works for straightforward propositions but fails on complex conditionals ("whether we should launch if X AND Y") | +| Grammar complexity | **HIGH** — requires subject-auxiliary inversion, pronoun mapping, tense preservation, conditional clause handling | +| Meaning-change risk | **LOW** — "whether X" is semantically equivalent to "Will/Does/Is X?" in decision context | + +### Candidate C — Clean Proposition + Evidence Framing + +Strip rationale → formulate: `"What evidence would clarify whether the customer will sign if we launch this year?"` + +| Assessment | Value | +|---|---| +| Semantic robustness | **HIGH** — "whether" is preserved as-is (no conversion needed); framing adapts to any proposition form | +| Conversational quality | **MEDIUM** — more formal than direct questions but still natural and decision-relevant. Standard in decision analysis literature | +| Consistency with existing `decision_evidence` family | **HIGH** — aligns with the evidence-gathering intent of the family (see line 1275 template) | + +### Candidate D — Minimum Combination + +**A + C**: Strip rationale first (Candidate A's extraction fix), then let existing evidence framing apply (producing Candidate C output). This avoids Candidate B's grammar complexity entirely. + +## Decision Criteria Assessment + +| Criterion | A | B | C | D (A+C) | +|---|---|---|---|---| +| 1. Remove explanatory rationale from question | PARTIAL | NO | YES | **YES** ✓ | +| 2. Preserve unresolved proposition | YES | YES | YES | **YES** ✓ | +| 3. Remains deterministic | YES | PARTIAL | YES | **YES** ✓ | +| 4. No provider rewriting | YES | YES | YES | **YES** ✓ | +| 5. No target selection change | YES | YES | YES | **YES** ✓ | +| 6. Preserve direct interrogative cases like 60B.20 | NO (breaks label path) | PARTIAL | YES | **YES** ✓ | +| 7. Avoid domain-specific grammar rules | YES | NO | YES | **YES** ✓ | + +## Final Choice: D — MINIMUM COMBINATION + +### Smallest implementation boundary + +**One change to `extractMeaning`:** +On line 108, instead of returning the full `strippedDescription`, split on semicolons and return only the first segment (the proposition), before applying `sentenceCase`. + +``` +Before: return sentenceCase(strippedDescription); +After: return sentenceCase(strippedDescription.split(/;|[,]\s*(so\s+that|which\s+means)/i)[0].trim()); +``` + +**No other changes required.** The existing decision-evidence formulation path (line 1272-1273) will then receive a clean proposition, and the `isInterrogativeMeaning` detection on line 189 will still correctly handle "whether" clauses as interrogative. + +**Alternative boundary:** If you want cleaner output than "Whether X?" for all cases, also modify the decision path (line 1272-1273) to use `buildEvidenceFallbackQuestion` instead of direct append-for-interrogative-meaning when the meaning starts with "whether". This produces "What evidence would clarify whether X?" which is both natural and consistent with the evidence family. + +## Implementation Readiness: A — READY FOR BOUNDED IMPLEMENTATION + +One line change to `extractMeaning` at line 108 plus (optionally) one additional refinement in the decision path formatting logic.