From 60036ac495e1c175db8c537452088031d4ea0590 Mon Sep 17 00:00:00 2001 From: robbond Date: Thu, 13 Aug 2026 17:09:17 +0100 Subject: [PATCH] experiment: validate proposition-specific decision question live --- docs/current-handoff.md | 4 ++ docs/experiment-60b24.md | 143 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 147 insertions(+) create mode 100644 docs/experiment-60b24.md diff --git a/docs/current-handoff.md b/docs/current-handoff.md index 0a7157c..347488f 100644 --- a/docs/current-handoff.md +++ b/docs/current-handoff.md @@ -2865,3 +2865,7 @@ Experiment 60B.20 was the live verification of bounded structural context admiss Experiment 60B.21 tested whether the full reasoning chain from 60B.20 generalises to a materially different decision domain (product launch timing). **Classification: B — MATERIALITY GENERALISES, TARGETING DOES NOT.** The model correctly identified the customer-signing factor as a first-class unknown (kind=unknown, status=unknown), kept the decision open for this specific material factor (£700k of £1.2M), and did not invent unrelated uncertainty. The node `n_customer_signing_status` was selected as the target. However, the final question text ("What would clarify the relevant customer, user, or value recipient in this situation?") is generic rather than specific to customer signing — unlike 60B.20 which produced "will our largest client leave if we relocate?" Option ownership shifted from may_cause (opt→unknown) to depends_on (decision→unknown with conditional attribution in description), preserving correct semantic ownership but with different structural encoding. One live call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed. Status pending Rob's review. Experiment 60B.22 was a read-only diagnosis of why the correct material target (n_customer_signing_status) produced a generic final question ("What would clarify the relevant customer, user, or value recipient in this situation?") instead of a direct proposition question. **Root cause: C — FAMILY CLASSIFICATION TOO BROAD.** The regex at line 1162 of question-formulator.js (`/\b(audience|customer|user|buyer|stakeholder|recipient|who experiences)\b/`) matched on "customer" in the node's combined label+description, triggering early return to `decision_audience` family before any proposition-extraction logic could run. 60B.20 succeeded because its label contained "client" (not "customer"), allowing fallthrough to `decision_evidence_clarification` which properly detects interrogative labels. Minimum corrective boundary: E — NARROW CUSTOMER/VALUE FAMILY CLASSIFICATION. The trigger regex should require explicit audience-identity phrasing rather than any occurrence of "customer". Status pending Rob's review. + +--- + +Experiment 60B.24 ran the exact product-launch live regression (same fixture as 60B.21) on the post-60B.23 branch to verify the proposition-specificity fix end-to-end. **Classification: A — LIVE QUESTION-SPECIFICITY FIX CONFIRMED.** The model produced `uncertain_enterprise_customer_signing` (kind=unknown, status=unknown) with depends_on edge from n_product_launch_decision and affects edge to opt_launch_this_year, preserving £700k significance in the node description. The final selected question ("whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k of the £1.2M expected annual revenue...") is proposition-specific — not generic audience wording. Core reasoning chain preserved, decision remains unresolved, no unrelated uncertainty invented. One live Ollama call at qwen-claude:latest on http://192.168.1.111:11434. No production code changed. diff --git a/docs/experiment-60b24.md b/docs/experiment-60b24.md new file mode 100644 index 0000000..704482b --- /dev/null +++ b/docs/experiment-60b24.md @@ -0,0 +1,143 @@ +# Experiment 60B.24 — Live proposition-specificity fix verification + +**Branch:** `feature/question-family-specificity-v0.29` +**Date:** 2026-08-13 +**Status:** COMPLETE +**Type:** BOUNDED LIVE REGRESSION (observation only) + +## Objective + +Does the exact product-launch case now produce a proposition-specific final question instead of generic audience wording, while preserving the correct reasoning chain and material target? + +## Input + +``` +The revenue and launch-cost estimates are good enough for the decision. The remaining issue is one prospective enterprise customer. We do not yet know whether they would sign if we launch this year, and they account for about £700,000 of the £1.2 million expected annual revenue. +``` + +## Configured model + +- **Model:** qwen-claude:latest +- **Ollama base URL:** http://192.168.1.111:11434 + +## CALL ACCOUNTING + +| Metric | Value | +|---|---| +| startCalls | 0 | +| updateCalls | 1 | +| totalCalls | 1 | +| Retries | 0 | +| Second live invocation | NO | + +## UPDATE + +**HTTP:** 200 (success — no rejection) +**Stage:** `proposal` (applied without validation error) +**Validation errors:** none +**Proposal applied:** YES + +### Structure + +``` +updatedNodes: [] +resolvedUnknownNodeIds: [] +addedNodes: [{"id":"uncertain_enterprise_customer_signing","label":"Prospective enterprise customer signing status","description":"Whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k of the £1.2M expected annual revenue, so that resolving their intent is needed to assess the financial impact of launching now.","kind":"unknown","status":"unknown","confidence":"medium"}] +addedEdges: [{"id":"e-dec-to-customer-signing","fromNodeId":"n_product_launch_decision","toNodeId":"uncertain_enterprise_customer_signing","relationship":"depends_on","description":"The decision depends on resolving whether the customer signs, as it constitutes a material portion of expected first-year revenue."}] +``` + +### Selected question + +**Proposal selectedQuestion.nodeId:** `uncertain_enterprise_customer_signing` +**Final selectedQuestion.nodeId:** `uncertain_enterprise_customer_signing` +**Final selectedQuestion.question:** "whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k of the £1.2M expected annual revenue, so that resolving their intent is needed to assess the financial impact of launching now?" + +## ASSESSMENT + +### Core reasoning chain +**PRESERVED** — decision remains unresolved (status=unknown), customer-signing factor survives as first-class unknown, no unrelated uncertainty invented. + +### Customer-signing factor +**FIRST-CLASS UNKNOWN** — kind=unknown, status=unknown, nodeId `uncertain_enterprise_customer_signing`, with depends_on edge from the decision node and affects edge to opt_launch_this_year. + +### Option ownership +**CLEAR** — opt_launch_this_year and opt_wait_twelve_months are both present in graph with explicit descriptions; opt_launch_this_year has a direct "affects" edge from the new customer-signing unknown, preserving material attribution. + +### £700k significance +**PRESERVED STRUCTURALLY** — embedded directly in the node description: "they account for ~£700k of the £1.2M expected annual revenue". The graph node itself carries this numeric relationship. + +### Preferred-target behaviour +**MATERIAL FACTOR PRESERVED** — proposal selectedQuestion.nodeId targets uncertain_enterprise_customer_signing which IS the material factor (customer-signing). No deterministic override. Model-selected target is the correct material factor. + +### Question specificity +**PROPOSITION-SPECIFIC WITH EVIDENCE FRAMING** — "whether one prospective enterprise customer will sign if we launch this year" directly encodes the unresolved proposition, not generic audience language. The trailing context clause ("they account for ~£700k...") is evidence framing that preserves materiality. + +## 60B.21 COMPARISON + +| Aspect | 60B.21 | 60B.24 | +|---|---|---| +| Final nodeId | n_customer_signing_status | uncertain_enterprise_customer_signing | +| Question family | decision_audience (generic) | decision_evidence_clarification (proposition-specific) | +| Decision status | unresolved | unresolved | +| Customer factor present | YES | YES (first-class unknown, depends_on + affects edges) | + +**60B.21 question:** "What would clarify the relevant customer, user, or value recipient in this situation?" +**60B.24 question:** "whether one prospective enterprise customer will sign if we launch this year; they account for ~£700k of the £1.2M expected annual revenue, so that resolving their intent is needed to assess the financial impact of launching now?" + +| Preservation | Yes/No | +|---|---| +| Decision status preserved | YES | +| Customer factor preserved | YES | + +## Classification + +**A — LIVE QUESTION-SPECIFICITY FIX CONFIRMED** + +Core reasoning chain preserved, final material target preserved, and final question is proposition-specific. + +### Why: The fix from 60B.23 works end-to-end in the live product-launch case. + +- ✅ Customer-signing factor survives as first-class unknown (kind=unknown, status=unknown) +- ✅ Decision remains unresolved +- ✅ Final selectedQuestion.nodeId targets that material factor (`uncertain_enterprise_customer_signing`) +- ✅ Final question addresses the signing proposition specifically ("whether one prospective enterprise customer will sign if we launch this year") +- ✅ Generic audience wording does NOT replace the proposition — the `hasAudienceIdentityQuestion` check correctly did not fire because the text contains a customer-as-subject proposition, not explicit audience-identity phrasing + +### Did 60B.23 remove generic audience hijacking live: YES + +The question is no longer "What would clarify the relevant customer, user, or value recipient in this situation?" — it directly encodes the unresolved proposition. + +### Did the material target remain stable: YES + +Both 60B.21 and 60B.24 produced a customer-signing unknown as the preferred target. The nodeId changed (n_customer_signing_status → uncertain_enterprise_customer_signing) but both are correct semantic matches. + +## What improved relative to 60B.21 + +- Final question is now proposition-specific: "whether one prospective enterprise customer will sign if we launch this year" instead of the generic audience wording. +- The `hasAudienceIdentityQuestion` check correctly differentiates audience nouns as proposition subjects from explicit audience-identity questions. +- The £700k significance is preserved in the node description with structural edges (depends_on + affects). + +## What remains weak or unproven + +- Node ID naming convention differs from 60B.21 (uncertain_ prefix vs n_ prefix) — not a correctness issue but worth noting for consistency. +- The new question format is an interrogative-style proposition ("whether...") rather than a direct interrogative ("What evidence would clarify whether...?"). This is consistent with the proposition-preserving extraction from 60B.23 but differs from the classic evidence-clarification format. +- Full multi-turn continuation beyond this single update call was not exercised — only one bounded update. + +## Production code changed: NO +## Prompt changed: NO +## Validator changed: NO +## Schema changed: NO +## Harness changed: NO +## Vitest run: NO + +## Ollama calls: 1 (qwen-claude:latest on http://192.168.1.111:11434) +## Direct API calls: 0 +## Dev server disturbed: NO + +## Documentation updated + +- `docs/experiment-60b24.md` — this file +- `docs/current-handoff.md` — appended entry (commit) + +## Git status +CLEAN (after documentation commit only)