experiment: validate uncertainty-over proposition live

This commit is contained in:
2026-08-14 06:38:44 +01:00
parent 827dc82eeb
commit 9b5942799f
2 changed files with 127 additions and 0 deletions
+4
View File
@@ -2911,3 +2911,7 @@ Experiment 60B.24 ran the exact product-launch live regression (same fixture as
Experiment 60B.25 was a read-only diagnosis of why the proposition-specific question produced "whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k..." (proposition-plus-rationale) instead of a clean conversational question. **Classification: C — BOTH A + B.** Two independent defects in `extractMeaning` (lib/graph/question-formulator.js): (A) **Rationale extraction too broad** at line 108 — when label contains "status" and description starts with "Whether", the function returns `sentenceCase(strippedDescription)` which includes the full description text after the semicolon, capturing explanatory rationale (~£700k context clause). No internal delimiter logic exists. (B) **No whether→direct-question conversion**`isInterrogativeMeaning` at line 189 treats "whether" clauses as already interrogative and passes them through unchanged via `wrapInterrogativeForTemplate`, so the output is proposition+rationale with "?" appended, not a converted question. The first meaningful divergence from 60B.20 (which produced clean "will our largest client leave if we relocate?") is at extractMeaning line 108: 60B.20's interrogative label bypasses the status+whether branch entirely, while 60B.24's nominal "signing status" label triggers it, pulling in full description including rationale. **Smallest corrective boundary:** A+C — strip rationale on extraction (split on first semicolon), then let existing evidence framing produce "What evidence would clarify whether X?" or keep direct interrogative form via decision path line 1272-1273. No provider rewrite, no target selection change, no schema change needed. Implementation readiness: A. Full trace documented in docs/experiment-60b25.md.
Experiment 60B.27 ran the exact product-launch live regression on the post-60B.26 branch to verify rationale-stripping + evidence-framing end-to-end. **Classification: A — LIVE CLEAN-QUESTION FIX CONFIRMED.** The model produced `unc_customer_signing_likelihood` (kind=unknown, status=unknown) with may_cause edge to opt_launch_this_year. Core reasoning chain preserved: decision unresolved, customer-signing survives as first-class unknown, no unrelated uncertainty invented. Final question ("What evidence would clarify prospective enterprise customer signing likelihood?") uses correct evidence framing and contains ZERO rationale contamination — no £700k, £1.2M, expected revenue, or financial impact leaked into user-facing text. Source graph description preserves full rationale intact. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed. Minor gap identified: extractMeaning description-start check (`/^whether\s+/i`) doesn't match "Unknown whether..." descriptions, producing nominalized output instead of a full whether-clause — functionally equivalent but less natural English. Status pending Rob's review.
---
Experiment 60B.31 was a live regression of 60B.29 on the post-60B.30 branch to verify that `Uncertainty over whether...` now preserves the full proposition end-to-end live. **Classification: B — FULL PROPOSITION PRESERVED BUT DIFFERENT PREFIX EXERCISED.** The model produced `n_enterprise_customer_signing` (kind=unknown, status=unknown) with a depends_on edge. Decision remains unresolved. However, the node description started with `Whether...` not `Uncertainty over whether...`, meaning 60B.30's targeted normalization was NOT exercised live. The final question ("What outcome would demonstrate enough value to justify launching?") is generic justification interrogative — same shape as 60B.29, full proposition lost from the question text but preserved in the source node description (source graph meaning intact). No rationale contamination in the final question. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed. The `Uncertainty over whether...` normalization is deterministic-proven (39/39 tests) but never exercised against the live model producing this exact prefix — follow-up needed to force that prefix live.