diff --git a/docs/design-evolution-log.md b/docs/design-evolution-log.md index b044585..22bd9b3 100644 --- a/docs/design-evolution-log.md +++ b/docs/design-evolution-log.md @@ -1037,6 +1037,70 @@ Run the classifier passively against existing mock scenarios (comparison, contra --- +### Long-Investigation Evaluation — Full Sequence Results + +**Test file:** `tests/graph/question-importance.long-investigation.test.js` +**Fixture:** `longTurns` from `lib/mocks/scenarios.js` (5 turns, sequential mock mode) +**Method:** Ran `assessQuestionImportance` against every unresolved unknown at each turn. No rule changes before evaluation. + +#### Category distribution + +| Total | important | helpful | incidental | cannot_determine | +|-------|-----------|---------|------------|-------------------| +| 4 | 0 | 0 | 4 | 0 | + +The classifier collapsed to a single category: **`incidental`**. + +#### Per-turn detail + +| Turn | Unknown ID | Label (short) | Classification | +|------|------------|---------------|----------------| +| 0 | u-1 | Whether there is genuine demand for our category in Europe | incidental | +| 1 | u-2 | Whether our product is suitable for European compliance requirements | incidental | +| 2 | u-3 | Whether the cost of achieving compliance is justified by the market size | incidental | +| 3 | u-4 | Whether we have competitive differentiation against existing European players | incidental | + +Turn 4 had zero unresolved unknowns (all resolved). + +#### Analysis of collapse to `incidental` + +All four unresolved unknowns in the long-investigation sequence were classified as `incidental`. Three independent factors caused this: + +1. **No downstream dependencies.** No unresolved unknown has another unresolved unknown depending on it via `dependsOn` or edges — each question is a leaf in its turn's dependency graph. The downstream-dependency rule (Rule 1, first clause) never triggers. + +2. **Decision-text patterns missed.** The DECISION_PATTERNS regex requires `"whether to"` (the word "to" must follow "whether"). None of the four unknown labels contain "whether to" — they all use the structure "Whether [subject] [verb]" rather than "Whether to [verb]". Similarly, none contain "build", "launch", "proceed", or "continue.*develop". Rule 1's text-match clause (second disjunct) requires both a pattern match AND ≥1 graph connection — the pattern fails first. + +3. **No direct graph edges.** The long-investigation fixture's edges connect observations to state nodes and resolved unknowns, but the active unknown in each turn has zero incident edges (`collectConnectedIds` returns an empty set). Without connections, the threshold-based rules (≥1 for important, ≥2 for helpful) never trigger regardless of text content. + +#### Evidence that appears correct + +- Turn 0, u-1: "Whether there is genuine demand for our category in Europe" → `incidental`. This is questionable. The question frames the entire strategic decision ("should we enter Europe?"), yet no pattern matches because the edge from obs-2 to u-1 (market size evidence) only appears starting at turn 1 — at turn 0, u-1 genuinely has zero connections and no text match. + +#### Evidence that appears questionable + +- Turn 3, u-4: "Whether we have competitive differentiation against existing European players" → `incidental`. This is arguably a central question in the investigation, yet it is classified as incidental because it has zero graph edges and no decision-context keyword ("whether" alone does not match). The graph structure (edge from obs-5 to u-4) only connects observations to unknowns — but those connections exist on the source side, not the target. + +- Turn 2, u-3: "Whether the cost of achieving compliance is justified by the market size" → `incidental`. The word "cost" does not match EVIDENCE_PATTERNS and the node has zero direct edges. A human evaluator would classify this as important (it is the last financial feasibility gate before a go/no-go decision). + +#### Do questions change category across turns? + +No. All four resolved to `incidental`. There is no meaningful variation. This is not because the unknowns are identical — they address distinctly different strategic dimensions (market existence, compliance, cost, differentiation) — but because the classifier's two rule families (dependency detection and keyword matching) do not fire for any of them. + +#### Does the result appear useful enough to keep passive? + +**No.** A classifier that tags every unresolved unknown in a realistic long investigation as `incidental` provides no discrimination signal. It is technically correct under its own rules, but those rules are too narrow for the investigation structure as it currently exists. The collapse reveals a structural gap: active unknowns in this scenario have zero direct edges, and their labels use "Whether [clause]" phrasing rather than "Whether to [verb]" or other decision keywords. + +Further evidence is still required if the classifier is to be considered viable. Options include: +- Expanding DECISION_PATTERNS to capture broader question structures (not just "whether to" + keyword combos). +- Adjusting how graph connections are counted for target nodes vs source nodes in edges. +- Testing against scenarios where unknowns have direct observation→unknown edges. + +#### Evaluation status + +**Incomplete.** The classifier did not produce useful variation across the long-investigation sequence. It passed determinism and immutability checks, but failed to discriminate between questions that clearly have different strategic importance. The hypothesis is not yet supported by this evaluation. Further evidence or rule refinement (not on this branch) is required before the classifier can be considered viable as a passive tool. + +--- + ## Phase Transition Record that the project has moved from: diff --git a/tests/graph/question-importance.long-investigation.test.js b/tests/graph/question-importance.long-investigation.test.js new file mode 100644 index 0000000..677065d --- /dev/null +++ b/tests/graph/question-importance.long-investigation.test.js @@ -0,0 +1,160 @@ +/** + * Experiment 20 — Question Importance: long-investigation evaluation test. + * + * Runs the passive classifier across every unresolved unknown at each turn of + * the long-investigation fixture (longTurns). Captures per-turn classifications + * and produces a category distribution summary. Validates correctness requirements + * from the experiment brief. + */ + +import { describe, expect, it } from "vitest"; +import { assessQuestionImportance } from "@/lib/graph/question-importance.js"; +import { buildScenarioFixture } from "@/lib/mocks/scenarios.js"; + +/* ── The long-investigation fixture (5 turns) ─────────── */ + +describe("assessQuestionImportance — long investigation evaluation", () => { + const scenarioName = "long"; + const turnCount = 5; + + /* Run the full sequence and collect results */ + const allResults = []; + + for (let t = 0; t < turnCount; t++) { + const fixture = buildScenarioFixture(scenarioName, t); + expect(fixture).not.toBeNull(`Turn ${t} should have a valid fixture`); + + const graph = fixture.situationGraph; + const resolvedIds = new Set(graph.resolvedNodeIds || []); + + /* Evaluate every unresolved unknown at this turn */ + for (const node of graph.nodes) { + if (node.kind !== "unknown") continue; + if (resolvedIds.has(node.id)) continue; + + const result = assessQuestionImportance({ node, graph, resolvedNodeIds: graph.resolvedNodeIds }); + + allResults.push({ turn: t, nodeId: node.id, label: node.label, category: result.category }); + } + } + + /* ── Assertion: the sequence runs without error and every unknown gets a valid classification */ + + it("every unresolved unknown across all turns receives a valid classification", () => { + const validCategories = ["important", "helpful", "incidental", "cannot_determine"]; + for (const r of allResults) { + expect(validCategories).toContain(r.category); + } + }); + + it("the complete long-investigation sequence runs without error and produces results", () => { + expect(allResults.length).toBeGreaterThan(0); + }); + + /* ── Category distribution summary ───────────────────── */ + + const categories = {}; + for (const r of allResults) { + categories[r.category] = (categories[r.category] || 0) + 1; + } + + it("category distribution across the full sequence", () => { + console.log("\n=== Long-Investigation Classification Results ==="); + console.log(`Total unresolved unknowns classified: ${allResults.length}`); + console.log("Category distribution:", JSON.stringify(categories, null, 2)); + expect(allResults.length).toBeGreaterThan(0); + }); + + /* ── Per-turn detail (for evaluation review) ─────────── */ + + it("detailed per-turn classifications", () => { + const byTurn = {}; + for (const r of allResults) { + if (!byTurn[r.turn]) byTurn[r.turn] = []; + byTurn[r.turn].push(r); + } + for (const [turn, results] of Object.entries(byTurn)) { + console.log(`\n--- Turn ${turn} (${results.length} unresolved unknowns) ---`); + for (const r of results) { + const shortLabel = r.label?.length > 60 ? r.label.slice(0, 60) + "..." : r.label; + console.log(` [${r.category}] ${shortLabel}`); + } + } + expect(allResults.length).toBeGreaterThan(0); + }); + + /* ── Requirement: at least two distinct categories or explicit collapse ───── */ + + it("produces more than one meaningful category (or explicitly records collapse)", () => { + const distinct = new Set(allResults.map((r) => r.category)); + if (distinct.size < 2) { + console.log("\nNote: classification collapsed to a single category:", [...distinct][0]); + } + expect(distinct.size).toBeGreaterThanOrEqual(1); + }); + + /* ── Determinism ─────────────────────────────────────── */ + + it("identical turn data produces identical results", () => { + const fixture = buildScenarioFixture(scenarioName, 2); + const graph = fixture.situationGraph; + + for (const node of graph.nodes) { + if (node.kind !== "unknown") continue; + if ((graph.resolvedNodeIds || []).includes(node.id)) continue; + + const r1 = assessQuestionImportance({ node, graph }); + const deepCopy = JSON.parse(JSON.stringify(graph)); + const r2 = assessQuestionImportance({ node: { ...node }, graph: deepCopy }); + expect(r1).toEqual(r2); + } + }); + + /* ── Input immutability ─────────────────────────────── */ + + it("does not mutate input nodes or graphs during the sequence", () => { + const fixture = buildScenarioFixture(scenarioName, 3); + const originalNodesJSON = JSON.stringify(fixture.situationGraph.nodes); + const originalEdgesJSON = JSON.stringify(fixture.situationGraph.edges); + + for (const node of fixture.situationGraph.nodes) { + if (node.kind !== "unknown") continue; + if ((fixture.situationGraph.resolvedNodeIds || []).includes(node.id)) continue; + const nodeSnapshot = JSON.parse(JSON.stringify(node)); + assessQuestionImportance({ node, graph: fixture.situationGraph }); + + /* original node must be unchanged */ + expect(node).toEqual(nodeSnapshot); + } + + expect(JSON.stringify(fixture.situationGraph.nodes)).toBe(originalNodesJSON); + expect(JSON.stringify(fixture.situationGraph.edges)).toBe(originalEdgesJSON); + }); + + /* ── Evaluation: key questions for manual review ─────── */ + + it("key classification examples for evaluation", () => { + const important = []; + const helpful = []; + const incidental = []; + const cannot_determine = []; + + for (const r of allResults) { + const entry = { turn: r.turn, id: r.nodeId, label: r.label, category: r.category }; + if (r.category === "important") important.push(entry); + else if (r.category === "helpful") helpful.push(entry); + else if (r.category === "incidental") incidental.push(entry); + else cannot_determine.push(entry); + } + + console.log("\n=== Detailed Evaluation Data ==="); + console.log("IMPORTANT:", JSON.stringify(important, null, 2)); + console.log("HELPFUL:", JSON.stringify(helpful, null, 2)); + console.log("INCIDENTAL:", JSON.stringify(incidental, null, 2)); + console.log("CANNOT_DETERMINE:", JSON.stringify(cannot_determine, null, 2)); + + expect(important.length).toBeGreaterThanOrEqual(0); + expect(helpful.length).toBeGreaterThanOrEqual(0); + expect(incidental.length).toBeGreaterThanOrEqual(0); + }); +});