experiment: add passive question importance test

2026-08-05 19:48:11 +01:00
parent fda3c9c02d
commit 445aaa7b37
3 changed files with 452 additions and 0 deletions
@@ -1001,6 +1001,42 @@ If none of these can be evaluated after 2–3 real investigations with v0.1, the

 ---

+### Experiment 20 — Passive Question Importance Classification
+
+#### Hypothesis
+
+Does a passive classifier that tags unresolved unknowns as `important`, `helpful`, `incidental`, or `cannot_determine` (using only existing graph fields, no scoring, no weights) produce coherent importance patterns across normal investigations?
+
+This is one question. Nothing else matters until this is answered.
+
+#### Scope
+
+A pure function `assessQuestionImportance({ node, graph })` implementing three deterministic rules:
+
+1. **important** — Other unresolved unknown(s) depend on this one (via `dependsOn` or edges); OR text contains decision-context patterns ("whether to", "build", "launch") AND has ≥1 graph connection.
+2. **helpful** — Text contains evidence-related patterns ("evidence", "metric", "measure", "criteria"); OR has ≥2 total connections in the graph.
+3. **incidental** — Default when neither important nor helpful conditions are met.
+4. **cannot_determine** — Node label and description are both empty/null (fallback for empty input).
+
+The classifier is passive — validated only against mock scenario fixtures. No changes to: graph construction, unknown selection, question selection, prompts, Ollama integration, APIs, UI, state assessment, behaviour selection, or conversation output.
+
+#### Validation
+
+Run the classifier passively against existing mock scenarios (comparison, contradictory, missing-evidence, decision, long investigation, complete) and verify at least three classifications align with intuitive expectations:
+
+- The "decision" scenario's build/commercial unknown → `important`
+- An evidence-gathering unknown from the comparison scenario → `helpful`
+- A minor formatting or cosmetic unknown → `incidental`
+
+#### Open Questions
+
+- Which importance category appears most frequently across normal investigations?
+- Does the downstream-dependency rule align with how the engine currently prioritises (score-based selection)?
+- Are decision-context text patterns ("whether to", "build") capturing the right signal, or is this too coarse-grained?
+- Can a future experiment use these categories to influence question phrasing (not priority) without breaking existing selection?
+
+---
+
 ## Phase Transition

 Record that the project has moved from: