docs/design-evolution-log.md

@@ -1101,6 +1101,26 @@ Further evidence is still required if the classifier is to be considered viable.
 
 ---
 
+### Experiment 20 — Conclusion
+
+The hypothesis was not confirmed by this evaluation.
+
+**What happened:**
+
+- The passive classifier collapsed to a single category (`incidental`) across the long-investigation scenario.
+- Three independent factors caused the collapse: no downstream dependencies, missed decision-text patterns (regex required "whether to" but questions used "Whether [clause]"), and zero graph edges on active unknowns.
+- The keyword-only approach produced technically correct but practically useless classifications.
+
+**What this means:**
+
+Question importance cannot be judged in isolation from the decision being investigated. A question like "Do we have competitive differentiation?" is only important when compared against a clear decision target. Without that target, keyword matching and local graph structure are insufficient signals.
+
+**Decision:**
+
+The Experiment 20 classifier has not been accepted into the active engine. Its rules remain unchanged (do not expand them). The next step is Experiment 21: testing whether providing an explicit decision target allows a simple deterministic classifier to produce useful distinctions.
+
+---
+
 ## Phase Transition
 
 Record that the project has moved from:
@@ -1139,6 +1159,45 @@ The objective is to make the investigation feel like a natural facilitated conve
 
 ---
 
+### Experiment 21 — Question Relevance Against Decision Target
+
+#### Hypothesis
+
+Does giving the classifier an explicit decision target allow it to distinguish questions that could change the decision from questions that are merely useful or incidental?
+
+This is one question. Nothing else matters until this is answered.
+
+#### Scope
+
+A pure function `assessQuestionRelevanceToDecision({ decisionTarget, unknown, graph })` implementing four deterministic rules:
+
+1. **could_change_decision** — The question directly mirrors the decision's core action (e.g., "whether to enter", "should we launch", "whether there is [demand/market/need]") AND the decision target contains a matching action keyword. Answering could reasonably reverse the proposed action.
+2. **supports_decision** — Necessary precondition (e.g., compliance, cost feasibility) OR supporting context (e.g., differentiation, competitive position). The answer would improve confidence or evidence but is less likely to reverse the decision alone.
+3. **unlikely_to_change_decision** — Background detail or comparative reference that does not affect the decision conditions.
+4. **cannot_determine** — Decision target or unknown is missing, empty, or too unclear to compare honestly.
+
+The classifier is passive — validated only against mock scenario fixtures. No changes to: graph construction, question importance classifier, unknown selection, question selection, prompts, Ollama integration, APIs, UI, state assessment, behaviour selection, conversation output, or engine behaviour in any way.
+
+#### Decision Target
+
+For the long-investigation scenario, use an explicit target from the fixture:
+
+> Should we enter the European market with our SaaS analytics platform?
+
+Do not attempt to discover the decision target automatically. For this experiment, the decision target is supplied by the test fixture.
+
+#### Evaluation
+
+Run the classifier passively across the same long-investigation turns used in Experiment 20 (turns 0–3). Record per-turn classification. Compare with Experiment 20 results. Expect at least two distinct categories — not a collapse to one.
+
+#### Questions
+
+- Does providing an explicit decision target enable more useful distinctions than keyword-only matching?
+- Do the four categories map intuitively to how a human evaluator would judge relevance?
+- Or does the deterministic rule set still miss cases that appear obviously important?
+
+---
+
 ## Current Open Questions
 
 The following are active explorations rather than decisions.

lib/graph/question-decision-relevance.js

@@ -0,0 +1,115 @@
+/**
+ * Question Relevance to Decision Target — passive classifier for Experiment 21.
+ *
+ * Classifies an unresolved unknown's relevance against an explicit decision target.
+ * Importance is not an isolated property of a question; it is a relationship between
+ * the question and the decision the investigation is trying to support.
+ *
+ * Classification categories:
+ *   could_change_decision        — Answering could reasonably reverse the proposed action.
+ *   supports_decision            — Answer improves confidence/evidence, less likely to reverse alone.
+ *   unlikely_to_change_decision  — Answer may be interesting but unlikely to materially affect the decision.
+ *   cannot_determine             — Decision target or unknown is missing / empty / too unclear.
+ *
+ * No LLM calls. No new graph fields. Pure function. No engine mutation.
+ */
+
+/* ── Helper: normalise text for matching ─────────────────────── */
+
+function normalise(value) {
+  return String(value || "").toLowerCase().replace(/[^a-z0-9]+/g, " ").trim();
+}
+
+/* ── Rule 1: Direct action — question asks whether the decision's core action should happen ─ */
+
+const DECISION_REVERSAL_PATTERNS = [
+  // Direct: "whether to enter/launch/build/proceed..."
+  /whether to (proceed|enter|launch|build|stop|abandon|drop|cancel|shelve)/i,
+  // Direct: "we should/must/can [action]..."
+  /\bwe\s+(should|must|can|need)\s+(to\s+)?(enter|launch|build|stop|proceed|pursue)\b/i,
+  // Fundamental existence check: "whether there is/are [any words] demand/market/need..."
+  /whether there (is|are).*\b(demand|market|need|interest|customers|audience|users)\b/i,
+];
+
+/* ── Rule 2: Necessary precondition — must be true for the decision to proceed ─ */
+
+const PRECONDITION_PATTERNS = [
+  // "whether our product is/has ..."
+  /\bour product (is|has|supports|meets|handles)\b.*\b(compliance|regulation|legal|required|mandatory|suitable)\b/i,
+];
+
+/* ── Rule 3a: Feasibility / cost-justify — supports but not decisive alone ─ */
+
+const FEASIBILITY_PATTERNS = [
+  /\b(cost (of|versus|vs|and)\s+(\w+)|justified\s+(by|with|through))/i,
+];
+
+/* ── Rule 3: Supporting context — informative but not decisive ─ */
+
+const SUPPORTING_CONTEXT_PATTERNS = [
+  /\bdifferentiat.*\b(against|versus|existing)\b/i,
+  /\bcompetitive (differentiation|advantage|landscape|position)\b/i,
+  /\boption|approach|path|way\b/i,
+];
+
+/* ── Rule 4: Incidental — background or comparative detail ─ */
+
+const INCIDENTAL_PATTERNS = [
+  /\b(history|background|general|typically|usually|generally)\b/i,
+  /\bcustomer (segment|base|profile|persona)\b/i,
+  /\bbenchmark(s)?|standard\s+(reference|example|case\s*study)\b/i,
+];
+
+/* ── Core classification function ───────────────────────────── */
+
+export function assessQuestionRelevanceToDecision(input) {
+  const { decisionTarget, unknown: node, graph } = input || {};
+
+  // Missing or invalid inputs → cannot_determine
+  if (!decisionTarget || !node || typeof node.kind !== "string") {
+    return { relevance: "cannot_determine", reason: "missing_input" };
+  }
+
+  const text = `${normalise(node.label)} ${normalise(node.description)}`.trim();
+
+  if (!text) {
+    return { relevance: "cannot_determine", reason: "empty_node_text" };
+  }
+
+  const decisionText = normalise(decisionTarget);
+
+  // Rule 1: Direct action — question mirrors the decision's core action
+  const hasActionKeyword = /\b(enter|launch|build|stop|abandon)\b/.test(decisionText);
+  const directMatch = DECISION_REVERSAL_PATTERNS.some((p) => p.test(text));
+
+  if (directMatch && hasActionKeyword) {
+    return { relevance: "could_change_decision", reason: "question directly mirrors the decision's core action" };
+  }
+
+  // Rule 2: Necessary precondition — tests a condition that must be true for the decision
+  const precondMatch = PRECONDITION_PATTERNS.some((p) => p.test(text));
+  if (precondMatch) {
+    return { relevance: "supports_decision", reason: "question tests a necessary precondition for the decision" };
+  }
+
+  // Rule 3a: Feasibility / cost-justify — supports but not decisive alone
+  const feasibilityMatch = FEASIBILITY_PATTERNS.some((p) => p.test(text));
+  if (feasibilityMatch) {
+    return { relevance: "supports_decision", reason: "question assesses feasibility or cost justification of the decision" };
+  }
+
+  // Rule 3: Supporting context — informative but not decisive on its own
+  const supportingMatch = SUPPORTING_CONTEXT_PATTERNS.some((p) => p.test(text));
+  if (supportingMatch) {
+    return { relevance: "supports_decision", reason: "question provides supporting context rather than a go/no-go condition" };
+  }
+
+  // Rule 4: Unlikely to change the decision — background detail
+  const incidentalMatch = INCIDENTAL_PATTERNS.some((p) => p.test(text));
+  if (incidentalMatch) {
+    return { relevance: "unlikely_to_change_decision", reason: "question concerns background detail rather than decision conditions" };
+  }
+
+  // Fallback — text is too generic to judge against the decision target
+  return { relevance: "cannot_determine", reason: "text does not clearly relate to or contrast with the decision target" };
+}

tests/graph/question-decision-relevance.test.js

@@ -0,0 +1,308 @@
+/**
+ * Experiment 21 — Question Relevance to Decision Target.
+ *
+ * Tests assessQuestionRelevanceToDecision against explicit decision targets
+ * and the long-investigation scenario fixture.
+ */
+
+import { describe, expect, it } from "vitest";
+import { assessQuestionRelevanceToDecision } from "@/lib/graph/question-decision-relevance.js";
+import { buildScenarioFixture } from "@/lib/mocks/scenarios.js";
+
+/* ── Helpers ─────────────────────────────────────────────────── */
+
+const DECISION_TARGET = "Should we enter the European market with our SaaS analytics platform?";
+
+function makeUnknown(id, label) {
+  return {
+    id, label: label || `Unknown ${id}`, description: label || `Unknown ${id}`,
+    kind: "unknown", status: "unknown", confidence: "low",
+    value: null, unit: null, evidenceIds: [], dependsOn: [], affects: [], childIds: [],
+  };
+}
+
+function makeGraphFor(nodes) {
+  return {
+    nodes, edges: [], resolvedNodeIds: [], activeUnknownNodeId: null,
+    centralStatement: "Test", currentSummary: "Test", reasoningState: null,
+  };
+}
+
+/* ── Could change decision ───────────────────────────────────── */
+
+describe("assessQuestionRelevanceToDecision — could_change_decision", () => {
+  it("classifies a direct go/no-go question against the decision target", () => {
+    const result = assessQuestionRelevanceToDecision({
+      decisionTarget: DECISION_TARGET,
+      unknown: makeUnknown("g1", "Whether to proceed with European market entry"),
+    });
+
+    expect(result.relevance).toBe("could_change_decision");
+    expect(typeof result.reason).toBe("string");
+    expect(result.reason.length).toBeGreaterThan(0);
+  });
+
+  it("classifies a 'whether we should enter' question as could_change_decision", () => {
+    const result = assessQuestionRelevanceToDecision({
+      decisionTarget: DECISION_TARGET,
+      unknown: makeUnknown("g2", "Whether we should enter the European market"),
+    });
+
+    expect(result.relevance).toBe("could_change_decision");
+  });
+
+  it("is deterministic for identical inputs", () => {
+    const node = makeUnknown("g3", "Whether to proceed with expansion into Europe");
+    const input = { decisionTarget: DECISION_TARGET, unknown: node };
+    const r1 = assessQuestionRelevanceToDecision(input);
+    const r2 = assessQuestionRelevanceToDecision(input);
+    expect(r1).toEqual(r2);
+  });
+});
+
+/* ── Supports decision ───────────────────────────────────────── */
+
+describe("assessQuestionRelevanceToDecision — supports_decision", () => {
+  it("classifies a compliance question as supports_decision (necessary precondition)", () => {
+    const result = assessQuestionRelevanceToDecision({
+      decisionTarget: DECISION_TARGET,
+      unknown: makeUnknown("c1", "Whether our product is suitable for European compliance requirements"),
+    });
+
+    expect(result.relevance).toBe("supports_decision");
+  });
+
+  it("classifies a cost justification question as supports_decision (feasibility)", () => {
+    const result = assessQuestionRelevanceToDecision({
+      decisionTarget: DECISION_TARGET,
+      unknown: makeUnknown("c2", "Whether the cost of achieving compliance is justified by the market size"),
+    });
+
+    expect(result.relevance).toBe("supports_decision");
+  });
+
+  it("classifies a differentiation question as supports_decision (supporting context)", () => {
+    const result = assessQuestionRelevanceToDecision({
+      decisionTarget: DECISION_TARGET,
+      unknown: makeUnknown("c3", "Whether we have competitive differentiation against existing European players"),
+    });
+
+    expect(result.relevance).toBe("supports_decision");
+  });
+
+  it("classifies a feasibility question as supports_decision even without precondition keywords", () => {
+    const result = assessQuestionRelevanceToDecision({
+      decisionTarget: DECISION_TARGET,
+      unknown: makeUnknown("c4", "Is the investment of $500K and six months justified by potential returns"),
+    });
+
+    expect(result.relevance).toBe("supports_decision");
+  });
+});
+
+/* ── Unlikely to change decision ─────────────────────────────── */
+
+describe("assessQuestionRelevanceToDecision — unlikely_to_change_decision", () => {
+  it("classifies a background comparison as unlikely_to_change_decision", () => {
+    const result = assessQuestionRelevanceToDecision({
+      decisionTarget: DECISION_TARGET,
+      unknown: makeUnknown("b1", "What benchmarks do other SaaS companies use for market sizing"),
+    });
+
+    expect(result.relevance).toBe("unlikely_to_change_decision");
+  });
+});
+
+/* ── Cannot determine — missing input ────────────────────────── */
+
+describe("assessQuestionRelevanceToDecision — cannot_determine (missing input)", () => {
+  it("returns cannot_determine when decisionTarget is missing", () => {
+    const result = assessQuestionRelevanceToDecision({
+      unknown: makeUnknown("m1", "Whether to enter Europe"),
+    });
+
+    expect(result.relevance).toBe("cannot_determine");
+  });
+
+  it("returns cannot_determine when decisionTarget is empty string", () => {
+    const result = assessQuestionRelevanceToDecision({
+      decisionTarget: "",
+      unknown: makeUnknown("m2", "Whether to enter Europe"),
+    });
+
+    expect(result.relevance).toBe("cannot_determine");
+  });
+
+  it("returns cannot_determine when unknown is missing", () => {
+    const result = assessQuestionRelevanceToDecision({
+      decisionTarget: DECISION_TARGET,
+    });
+
+    expect(result.relevance).toBe("cannot_determine");
+  });
+
+  it("returns cannot_determine when unknown has empty label and description", () => {
+    const node = makeUnknown("m3", "");
+    node.label = "";
+    node.description = "";
+
+    const result = assessQuestionRelevanceToDecision({
+      decisionTarget: DECISION_TARGET,
+      unknown: node,
+    });
+
+    expect(result.relevance).toBe("cannot_determine");
+  });
+
+  it("returns cannot_determine when no input object is provided", () => {
+    const result = assessQuestionRelevanceToDecision(null);
+    expect(result.relevance).toBe("cannot_determine");
+  });
+});
+
+/* ── Input immutability ──────────────────────────────────────── */
+
+describe("assessQuestionRelevanceToDecision — input immutability", () => {
+  it("does not mutate the unknown node", () => {
+    const node = makeUnknown("i1", "Whether to proceed with European market entry");
+    const snapshot = JSON.parse(JSON.stringify(node));
+    assessQuestionRelevanceToDecision({ decisionTarget: DECISION_TARGET, unknown: node });
+    expect(node).toEqual(snapshot);
+  });
+
+  it("does not mutate the graph (when provided)", () => {
+    const node = makeUnknown("i2", "Whether to proceed with European market entry");
+    const graph = makeGraphFor([node]);
+    const originalNodesJSON = JSON.stringify(graph.nodes);
+    assessQuestionRelevanceToDecision({ decisionTarget: DECISION_TARGET, unknown: node, graph });
+    expect(JSON.stringify(graph.nodes)).toBe(originalNodesJSON);
+  });
+
+  it("does not mutate the decisionTarget string", () => {
+    const dt = "Should we enter the European market with our SaaS analytics platform?";
+    assessQuestionRelevanceToDecision({ decisionTarget: dt, unknown: makeUnknown("i3", "test") });
+    expect(dt).toBe("Should we enter the European market with our SaaS analytics platform?");
+  });
+});
+
+/* ── Complete long-investigation sequence ─────────────────────── */
+
+describe("assessQuestionRelevanceToDecision — long investigation sequence", () => {
+  const scenarioName = "long";
+  const turnCount = 5;
+
+  const allResults = [];
+
+  for (let t = 0; t < turnCount; t++) {
+    const fixture = buildScenarioFixture(scenarioName, t);
+    expect(fixture).not.toBeNull(`Turn ${t} should have a valid fixture`);
+
+    // Use the centralStatement as decision target (present in every turn)
+    const decisionTarget = fixture.situationGraph.centralStatement;
+    const resolvedIds = new Set(fixture.situationGraph.resolvedNodeIds || []);
+
+    for (const node of fixture.situationGraph.nodes) {
+      if (node.kind !== "unknown") continue;
+      if (resolvedIds.has(node.id)) continue;
+
+      const result = assessQuestionRelevanceToDecision({
+        decisionTarget,
+        unknown: node,
+        graph: fixture.situationGraph,
+      });
+
+      allResults.push({ turn: t, nodeId: node.id, label: node.label, relevance: result.relevance, reason: result.reason });
+    }
+  }
+
+  it("every unresolved unknown across all turns receives a valid relevance classification", () => {
+    const validCategories = ["could_change_decision", "supports_decision", "unlikely_to_change_decision", "cannot_determine"];
+    for (const r of allResults) {
+      expect(validCategories).toContain(r.relevance);
+    }
+  });
+
+  it("the complete long-investigation sequence runs without error and produces results", () => {
+    expect(allResults.length).toBeGreaterThan(0);
+  });
+
+  it("produces more than one distinct relevance category across the sequence", () => {
+    const distinct = new Set(allResults.map((r) => r.relevance));
+    expect(distinct.size).toBeGreaterThanOrEqual(2);
+  });
+
+  it("category distribution across the full sequence", () => {
+    const categories = {};
+    for (const r of allResults) {
+      categories[r.relevance] = (categories[r.relevance] || 0) + 1;
+    }
+    console.log("\n=== Experiment 21 — Classification Results ===");
+    console.log(`Total unresolved unknowns classified: ${allResults.length}`);
+    console.log("Category distribution:", JSON.stringify(categories, null, 2));
+    expect(allResults.length).toBeGreaterThan(0);
+  });
+
+  it("detailed per-turn classifications", () => {
+    const byTurn = {};
+    for (const r of allResults) {
+      if (!byTurn[r.turn]) byTurn[r.turn] = [];
+      byTurn[r.turn].push(r);
+    }
+    for (const [turn, results] of Object.entries(byTurn)) {
+      console.log(`\n--- Turn ${turn} (${results.length} unresolved unknowns) ---`);
+      for (const r of results) {
+        const shortLabel = r.label?.length > 60 ? r.label.slice(0, 60) + "..." : r.label;
+        console.log(`  [${r.relevance}] ${shortLabel}`);
+      }
+    }
+    expect(allResults.length).toBeGreaterThan(0);
+  });
+
+  it("Turn 0 (market demand question) is classified as could_change_decision", () => {
+    const turn0 = allResults.find((r) => r.turn === 0);
+    expect(turn0).toBeDefined();
+    expect(turn0.relevance).toBe("could_change_decision");
+  });
+
+  it("Turns 1-3 (compliance, cost, differentiation) are classified as supports_decision", () => {
+    const turns1to3 = allResults.filter((r) => r.turn >= 1 && r.turn <= 3);
+    for (const r of turns1to3) {
+      expect(r.relevance).toBe("supports_decision");
+    }
+  });
+
+  it("all classified questions include a non-empty reason", () => {
+    for (const r of allResults) {
+      if (r.relevance !== "cannot_determine") {
+        expect(typeof r.reason).toBe("string");
+        expect(r.reason.length).toBeGreaterThan(0);
+      }
+    }
+  });
+});
+
+/* ── Comparison with Experiment 20: different categories produced ─ */
+
+describe("assessQuestionRelevanceToDecision — comparison with Experiment 20 collapse", () => {
+  it("does not collapse entirely to one category across the long investigation", () => {
+    const scenarioName = "long";
+    const results = [];
+
+    for (let t = 0; t < 5; t++) {
+      const fixture = buildScenarioFixture(scenarioName, t);
+      if (!fixture) continue;
+      const decisionTarget = fixture.situationGraph.centralStatement;
+      const resolvedIds = new Set(fixture.situationGraph.resolvedNodeIds || []);
+
+      for (const node of fixture.situationGraph.nodes) {
+        if (node.kind !== "unknown") continue;
+        if (resolvedIds.has(node.id)) continue;
+
+        results.push(assessQuestionRelevanceToDecision({ decisionTarget, unknown: node }));
+      }
+    }
+
+    const distinct = new Set(results.map((r) => r.relevance));
+    expect(distinct.size).toBeGreaterThan(1);
+  });
+});