experiment: validate clean proposition question live

2026-08-13 17:54:04 +01:00
parent 802eb1cc16
commit 4e66e1ffbf
2 changed files with 174 additions and 0 deletions
@@ -2883,3 +2883,5 @@ Experiment 60B.24 ran the exact product-launch live regression (same fixture as
 ---

 Experiment 60B.25 was a read-only diagnosis of why the proposition-specific question produced "whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k..." (proposition-plus-rationale) instead of a clean conversational question. **Classification: C — BOTH A + B.** Two independent defects in `extractMeaning` (lib/graph/question-formulator.js): (A) **Rationale extraction too broad** at line 108 — when label contains "status" and description starts with "Whether", the function returns `sentenceCase(strippedDescription)` which includes the full description text after the semicolon, capturing explanatory rationale (~£700k context clause). No internal delimiter logic exists. (B) **No whether→direct-question conversion** — `isInterrogativeMeaning` at line 189 treats "whether" clauses as already interrogative and passes them through unchanged via `wrapInterrogativeForTemplate`, so the output is proposition+rationale with "?" appended, not a converted question. The first meaningful divergence from 60B.20 (which produced clean "will our largest client leave if we relocate?") is at extractMeaning line 108: 60B.20's interrogative label bypasses the status+whether branch entirely, while 60B.24's nominal "signing status" label triggers it, pulling in full description including rationale. **Smallest corrective boundary:** A+C — strip rationale on extraction (split on first semicolon), then let existing evidence framing produce "What evidence would clarify whether X?" or keep direct interrogative form via decision path line 1272-1273. No provider rewrite, no target selection change, no schema change needed. Implementation readiness: A. Full trace documented in docs/experiment-60b25.md.
+
+Experiment 60B.27 ran the exact product-launch live regression on the post-60B.26 branch to verify rationale-stripping + evidence-framing end-to-end. **Classification: A — LIVE CLEAN-QUESTION FIX CONFIRMED.** The model produced `unc_customer_signing_likelihood` (kind=unknown, status=unknown) with may_cause edge to opt_launch_this_year. Core reasoning chain preserved: decision unresolved, customer-signing survives as first-class unknown, no unrelated uncertainty invented. Final question ("What evidence would clarify prospective enterprise customer signing likelihood?") uses correct evidence framing and contains ZERO rationale contamination — no £700k, £1.2M, expected revenue, or financial impact leaked into user-facing text. Source graph description preserves full rationale intact. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed. Minor gap identified: extractMeaning description-start check (`/^whether\s+/i`) doesn't match "Unknown whether..." descriptions, producing nominalized output instead of a full whether-clause — functionally equivalent but less natural English. Status pending Rob's review.