233 lines
13 KiB
Markdown
233 lines
13 KiB
Markdown
# Experiment 60B.25 — Diagnosis: Proposition-Plus-Rationale Instead of Clean Question
|
|
|
|
**Branch:** `feature/question-family-specificity-v0.29`
|
|
**Date:** 2026-08-13
|
|
**Status:** COMPLETE
|
|
**Type:** READ-ONLY DIAGNOSIS (no code changes)
|
|
|
|
## Objective
|
|
|
|
Answer one measurable question:
|
|
|
|
> Why does deterministic formulation preserve the whole proposition-plus-rationale string instead of converting the unresolved proposition into a concise interrogative question?
|
|
|
|
Input node (from 60B.24):
|
|
```
|
|
label: "Prospective enterprise customer signing status"
|
|
description: "Whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k of the £1.2M expected annual revenue, so that resolving their intent is needed to assess the financial impact of launching now."
|
|
```
|
|
|
|
Produced question:
|
|
```
|
|
whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k of the £1.2M expected annual revenue, so that resolving their intent is needed to assess the financial impact of launching now?
|
|
```
|
|
|
|
## Checkpoint 1 — Extraction Behaviour (full trace)
|
|
|
|
### `extractMeaning(node)` trace for the fixed node:
|
|
|
|
**Line 86:** `raw = "Prospective enterprise customer signing status Whether one prospective..."`
|
|
|
|
**Line 87-89:** `meaning = stripTrailingPunctuation("Prospective enterprise customer signing status")`
|
|
→ `"Prospective enterprise customer signing status"` (no trailing punctuation to strip)
|
|
|
|
**Line 92:** `hasAudienceIdentityQuestion(lowered)` → **NO**
|
|
None of the patterns match: "who is the customer", "target customer", "identifying the customer", etc. The word "enterprise customer" does not match any pattern — it's a noun modifier, not an audience-identity construct.
|
|
|
|
**Line 96-97:** `strippedDescription = stripTrailingPunctuation(description)`
|
|
→ Full description text with no trailing punctuation change (it ends with period which gets stripped).
|
|
|
|
**Lines 102-108 — THE KEY BRANCH:**
|
|
```js
|
|
if (
|
|
/\b(status|likelihood|probability|chance|risk|uncertainty)\b/i.test("Prospective enterprise customer signing status") && // MATCHES "status" ✓
|
|
/^whether\s+/i.test(strippedDescription) // MATCHES "Whether..." ✓
|
|
) {
|
|
return sentenceCase(strippedDescription); // ← ALL text returned
|
|
}
|
|
```
|
|
|
|
Both conditions match. **This branch is taken.**
|
|
|
|
`sentenceCase("Whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k...")` →
|
|
**lowercases first char, preserves everything else including rationale after semicolon**
|
|
|
|
### Checkpoint 1 Answers:
|
|
|
|
| Question | Answer |
|
|
|---|---|
|
|
| label considered? | **YES** — used to trigger the status regex |
|
|
| description considered? | **YES** — full text passed to sentenceCase on line 108 |
|
|
| returned meaning | `"whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k of the £1.2M expected annual revenue, so that resolving their intent is needed to assess the financial impact of launching now."` (first char lowered) |
|
|
| rationale stripped? | **NO** — line 108 returns full `strippedDescription` |
|
|
| semicolon boundary recognized? | **NO** — no split logic exists for description extraction |
|
|
| "so that" rationale recognized? | **NO** — no rationale marker detection in extractMeaning |
|
|
|
|
## Checkpoint 2 — Interrogative Conversion (full trace)
|
|
|
|
### `isInterrogativeMeaning(meaning)` trace:
|
|
|
|
Input: `"whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k..."`
|
|
|
|
- Line 170 (`wh-questions`): No match — starts with "whether", not who/what/where/when/how
|
|
- Lines 179-186 (aux inversion): No match — first word is "whether", not a modal auxiliary
|
|
- **Line 189: `/^whether\b/i.test(trimmed)` → YES ✓**
|
|
|
|
Returns `true`. The engine recognises the meaning as already question-shaped.
|
|
|
|
### `wrapInterrogativeForTemplate(meaning)` trace:
|
|
|
|
Input: lowercased meaning string
|
|
- Line 198 (wh-questions): No match — starts with "whether"
|
|
- **Line 202: `isInterrogativeMeaning` → true**
|
|
- Returns: `stripTrailingPunctuation(meaning).trim()` = full proposition + rationale, no trailing punctuation
|
|
|
|
### Decision evidence path trace (line 1271-1273):
|
|
|
|
```js
|
|
if (isInterrogativeMeaning(meaning)) { // YES ✓
|
|
return `${wrapInterrogativeForTemplate(meaning)}?`;
|
|
}
|
|
```
|
|
|
|
Result: `"whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k of the £1.2M expected annual revenue, so that resolving their intent is needed to assess the financial impact of launching now?"`
|
|
|
|
### Checkpoint 2 Answers:
|
|
|
|
| Question | Answer |
|
|
|---|---|
|
|
| Does engine recognise "whether..." as unresolved proposition? | **YES** — `isInterrogativeMeaning` returns true at line 189 |
|
|
| Does it convert "whether X..." into "Will/Does/Is X...?" | **NO** — no conversion logic exists; "whether" is treated as already interrogative |
|
|
| Does it merely append "?" | **YES** — direct from the full extracted string including rationale |
|
|
| Evidence framing applied? | **CONDITIONAL** — `buildEvidenceFallbackQuestion` would add "What evidence would confirm or rule out...", but in the decision reasoning path (line 1272-1273), interrogative means bypass the evidence template and go straight to append "?" |
|
|
|
|
## Checkpoint 3 — Why 60B.20 Looked Better
|
|
|
|
### 60B.20 source shape:
|
|
```
|
|
label: "Will our largest client leave if we relocate?"
|
|
description: "Uncertainty regarding whether our largest client would depart following a relocation to Manchester; matters because their departure would cost approximately £5M per year..."
|
|
```
|
|
|
|
**Label is already interrogative:** "Will our largest client leave if we relocate?"
|
|
|
|
### 60B.24 source shape:
|
|
```
|
|
label: "Prospective enterprise customer signing status"
|
|
description: "Whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k of the £1.2M expected annual revenue, so that resolving their intent is needed to assess the financial impact of launching now."
|
|
```
|
|
|
|
**Label is nominal (noun phrase); proposition in description starting with "Whether"**
|
|
|
|
### First meaningful divergence: `extractMeaning` line 108
|
|
|
|
In 60B.24, the condition at lines 102-107 fires because the label contains "status" AND the description starts with "Whether". This causes `extractMeaning` to return the **full** `strippedDescription` (proposition + rationale after semicolon).
|
|
|
|
In 60B.20, the label is already interrogive ("Will our largest client..."). The condition at lines 102-107 does NOT fire because:
|
|
- Label contains no status/probability words (no "status" in "Will our largest client leave if we relocate?")
|
|
- Even though it has no "status", the label itself IS interrogative
|
|
|
|
The extracted meaning for 60B.20 is the **label** ("Will our largest client leave if we relocate?"), not the description. This is already a clean question, so the template simply appends "?" to produce a valid output.
|
|
|
|
### Quality difference cause: `description extraction` + `rationale contamination`
|
|
|
|
The root divergence is in `extractMeaning`:
|
|
1. **60B.20** — Label is interrogative → meaning = label (clean) → output = label + "?" ✓
|
|
2. **60B.24** — Label triggers status condition → meaning = full description including rationale after semicolon → output = proposition+rationale + "?" ✗
|
|
|
|
The quality difference comes from **description extraction capturing rationale** and the **absence of "whether→direct-question conversion"**.
|
|
|
|
## Cause Classification: C (Both A + B)
|
|
|
|
### A — RATIONALE EXTRACTION TOO BROAD
|
|
`extractMeaning` line 108 returns `sentenceCase(strippedDescription)` which includes everything after the semicolon. There is no internal delimiter logic for separating proposition from explanatory rationale.
|
|
|
|
### B — NO WHETHER→QUESTION CONVERSION
|
|
The engine recognises "whether X" as already interrogative (line 189) and passes it through unchanged. No conversion to "Will/Does/Is X?" exists in the codebase. The `isInterrogativeMeaning` function treats "whether" clauses as complete interrogatives rather than treating them as unresolved propositions that need conversion.
|
|
|
|
## Current Semantic Contract
|
|
|
|
**For a selected unknown whose explicit meaning is "whether X", what should deterministic formulation ideally represent?**
|
|
|
|
### C — Direct Interrogative: "Will/Does/Is X?"
|
|
|
|
The current code's intent (lines 170-189, 202-203) is:
|
|
- If the extracted meaning is already interrogative (wh- question, aux inversion, or whether-clause), pass it through unchanged.
|
|
- The rationale for line 189 treating "whether" as complete interrogative was to prevent double-wrapping ("What would clarify are...").
|
|
|
|
However, this conflates two distinct semantic states:
|
|
1. **Direct interrogative** (e.g., "Will X happen?") — ready as a question
|
|
2. **Indirect interrogative / unresolved proposition** (e.g., "whether X will happen") — needs conversion
|
|
|
|
The current contract treats both identically, which is why 60B.24's output preserves the indirect form with rationale contamination.
|
|
|
|
## Candidate Evaluations
|
|
|
|
### Candidate A — Strip Rationale Only
|
|
|
|
Extract only: `"Whether one prospective enterprise customer will sign if we launch this year"` (before semicolon). Then preserve existing formulation behaviour.
|
|
|
|
| Assessment | Value |
|
|
|---|---|
|
|
| Improves concision | **HIGH** — removes the entire explanatory clause |
|
|
| Produces conversational question | **NO** — "Whether one prospective enterprise customer will sign if we launch this year?" is still an indirect question (embedded/yes-no proposition form), not natural conversational English. The user would expect "Will one...?" |
|
|
| Risk of losing context | **LOW** — rationale is explanatory, not material. Materiality lives in the graph structure (£700k on edge/unknown node) |
|
|
|
|
### Candidate B — Deterministic Whether→Interrogative Conversion
|
|
|
|
Convert simple explicit propositions:
|
|
- Input: `"whether the customer will sign"`
|
|
- Output: `"Will the customer sign?"`
|
|
|
|
| Assessment | Value |
|
|
|---|---|
|
|
| Semantic robustness | **MEDIUM** — works for straightforward propositions but fails on complex conditionals ("whether we should launch if X AND Y") |
|
|
| Grammar complexity | **HIGH** — requires subject-auxiliary inversion, pronoun mapping, tense preservation, conditional clause handling |
|
|
| Meaning-change risk | **LOW** — "whether X" is semantically equivalent to "Will/Does/Is X?" in decision context |
|
|
|
|
### Candidate C — Clean Proposition + Evidence Framing
|
|
|
|
Strip rationale → formulate: `"What evidence would clarify whether the customer will sign if we launch this year?"`
|
|
|
|
| Assessment | Value |
|
|
|---|---|
|
|
| Semantic robustness | **HIGH** — "whether" is preserved as-is (no conversion needed); framing adapts to any proposition form |
|
|
| Conversational quality | **MEDIUM** — more formal than direct questions but still natural and decision-relevant. Standard in decision analysis literature |
|
|
| Consistency with existing `decision_evidence` family | **HIGH** — aligns with the evidence-gathering intent of the family (see line 1275 template) |
|
|
|
|
### Candidate D — Minimum Combination
|
|
|
|
**A + C**: Strip rationale first (Candidate A's extraction fix), then let existing evidence framing apply (producing Candidate C output). This avoids Candidate B's grammar complexity entirely.
|
|
|
|
## Decision Criteria Assessment
|
|
|
|
| Criterion | A | B | C | D (A+C) |
|
|
|---|---|---|---|---|
|
|
| 1. Remove explanatory rationale from question | PARTIAL | NO | YES | **YES** ✓ |
|
|
| 2. Preserve unresolved proposition | YES | YES | YES | **YES** ✓ |
|
|
| 3. Remains deterministic | YES | PARTIAL | YES | **YES** ✓ |
|
|
| 4. No provider rewriting | YES | YES | YES | **YES** ✓ |
|
|
| 5. No target selection change | YES | YES | YES | **YES** ✓ |
|
|
| 6. Preserve direct interrogative cases like 60B.20 | NO (breaks label path) | PARTIAL | YES | **YES** ✓ |
|
|
| 7. Avoid domain-specific grammar rules | YES | NO | YES | **YES** ✓ |
|
|
|
|
## Final Choice: D — MINIMUM COMBINATION
|
|
|
|
### Smallest implementation boundary
|
|
|
|
**One change to `extractMeaning`:**
|
|
On line 108, instead of returning the full `strippedDescription`, split on semicolons and return only the first segment (the proposition), before applying `sentenceCase`.
|
|
|
|
```
|
|
Before: return sentenceCase(strippedDescription);
|
|
After: return sentenceCase(strippedDescription.split(/;|[,]\s*(so\s+that|which\s+means)/i)[0].trim());
|
|
```
|
|
|
|
**No other changes required.** The existing decision-evidence formulation path (line 1272-1273) will then receive a clean proposition, and the `isInterrogativeMeaning` detection on line 189 will still correctly handle "whether" clauses as interrogative.
|
|
|
|
**Alternative boundary:** If you want cleaner output than "Whether X?" for all cases, also modify the decision path (line 1272-1273) to use `buildEvidenceFallbackQuestion` instead of direct append-for-interrogative-meaning when the meaning starts with "whether". This produces "What evidence would clarify whether X?" which is both natural and consistent with the evidence family.
|
|
|
|
## Implementation Readiness: A — READY FOR BOUNDED IMPLEMENTATION
|
|
|
|
One line change to `extractMeaning` at line 108 plus (optionally) one additional refinement in the decision path formatting logic.
|