35 KiB
Experiment 07 — Lightweight Starting Observation
Hypothesis
A smaller initial input will make beginning an investigation feel easier and will communicate that the engine needs only a concise observation rather than a complete analysis.
Questions
- Does the input feel like a conversation starter rather than a report form?
- Is three to four visible lines sufficient?
- Does the facilitator panel and input area feel better balanced?
- Does the user understand that further detail will be gathered through questions?
- Does reducing the input height make the Analyse action easier to notice?
Evaluation
Pending visual review.
Result
Confirmed.
Four visible rows better communicates a starting observation than six.
Input size communicates expected effort.
"What have you noticed?" reinforces observational thinking.
Users are encouraged to begin rather than compose.
The facilitator and workspace now feel more balanced.
This interaction principle should continue throughout the investigation rather than existing only on the landing page.
Decision
Retain the smaller landing input.
Proceed to investigate consistency between the landing experience and investigation responses.
Experiment 09 — Investigation Rhythm
Result
Partially confirmed.
What did we learn?
- Moving History directly beneath Response improves the sense of conversational continuity.
- The sequence Question → Response → History is cognitively coherent.
- History behaves like the growing notebook of the investigation, not general reference material.
- Allowing History to span the full workspace breaks the wider spatial model.
- Situation and Investigation Map should remain stable supporting artefacts rather than moving down as the notebook grows.
- The conversation needs a dedicated vertical lane.
Decision
Keep History directly connected to Response.
Refine the desktop workspace into a stable conversation lane and a stable supporting lane.
Do not rewrite previous experiments.
Experiment 08 — Consistent Investigation Responses
Hypothesis
Every answer given during an investigation should feel like an observation, not a report.
The response component should therefore communicate the same expected effort as the initial scenario input.
Questions
- Does a smaller response area reduce perceived effort?
- Does the investigation feel more conversational?
- Does consistency improve confidence?
- Does the workspace become visually calmer?
- Does the current investigation remain the dominant focus?
Result
Confirmed.
Consistent interaction patterns reduce cognitive effort.
Users should not have to learn different behaviours between the landing page and investigation.
Smaller response areas reinforce concise observations.
The engine appears more conversational when each answer feels lightweight.
Consistency is becoming a stronger design tool than decoration.
Decision
Retain consistent input sizing across both contexts.
Experiment 10 — Stable Conversation Column
Hypothesis
A persistent two-thirds conversation column beside a one-third supporting column will allow the investigation notebook to grow without moving the shared reference artefacts.
Questions
- Does the left column feel like one continuous investigation?
- Does History grow naturally beneath Response?
- Do Situation and Investigation Map remain easy to reference?
- Does the interface feel spatially stable as turns accumulate?
- Does showing full question text improve readability now that sufficient width exists?
Evaluation
Visual review completed.
Status
Closed.
Result
Partially confirmed.
What did we learn?
-
The investigation workspace is beginning to feel like a genuine facilitated investigation rather than a document.
-
The two-column workspace (conversation on the left, reference material on the right) is proving to be a stronger mental model than previous layouts.
-
Keeping Situation and Investigation Map fixed while History grows vertically feels more natural.
-
The investigation question, response and history now read as one continuous conversation.
-
Developer Details have become extremely valuable.
-
The graph produced by the reasoning engine is far richer than previously realised. The graph now contains structured concepts including:
- observations
- unknowns
- assumptions
- relationships
- metrics
- state
This suggests the UI should increasingly become a human-friendly projection of the graph rather than inventing separate state.
The current "Investigation in progress" panel exposes developer-oriented statistics (nodes, edges, unknowns etc.) which are useful during development but are not the most helpful representation for an end user.
Emerging Direction — Graph as Source of Truth
The reasoning graph is becoming the shared source of truth for multiple UI views.
Different interfaces may project the same graph for different audiences:
- Version A — compact technical progress;
- Version B — detailed graph inspection;
- Version C — user-facing facilitator view;
- Developer Details — complete diagnostics;
- Investigation Map — future spatial projection;
- Current Question — active uncertainty projection.
The UI should not maintain separate invented summaries where the graph already contains the underlying information.
This is an emerging direction, not a final architecture decision.
Emerging Direction — Facilitator Translation Layer
The UI should progressively become a translation layer over the reasoning graph rather than maintaining separate duplicated summaries. Internal graph concepts should remain available for developers, while end users see a facilitator-style explanation of what is currently understood and what remains uncertain.
The current technical progress panel (nodes, edges, unknowns, assumptions) exposes developer-oriented statistics. These are valuable during development but not the most helpful representation for an end user.
The next direction is to explore presenting the same underlying graph data as a facilitator's notebook — what is known, what remains uncertain, and a quiet summary of the reasoning state underneath.
Experiment 11 — Facilitator Progress Panel (Version B)
Hypothesis
The same underlying reasoning graph can be presented in a much more human-friendly way without changing the reasoning engine, API contracts, or graph generation.
A facilitator-style panel should communicate:
- what is known (resolved nodes and observations)
- what remains uncertain (unresolved unknowns and assumptions)
- a quiet summary of the reasoning state underneath
Questions
- Can the same graph data be translated into a facilitator-style view that end users understand more naturally?
- Does separating "known" from "still investigating" reduce cognitive load compared to node/edge counts?
- Is a quiet reasoning summary sufficient, or does it need more context?
- Does the translation-layer principle hold — presenting the graph as a notebook rather than raw data?
Result
Partially confirmed.
What did we learn?
- Version B proved that the reasoning graph contains substantially more useful information than Version A exposes.
- The graph already contains observations, unknowns, assumptions, metrics, relationships and state.
- The graph is rich enough to support multiple UI projections.
- Exposing the graph almost verbatim overwhelms the user.
- Technical categories are useful for development but do not directly communicate investigation progress.
- The user needs a translation of the graph rather than a graph browser.
- Developer Details should remain the place for complete technical inspection.
- A user-facing view needs filtering, prioritisation, deduplication and clear epistemic labels.
Decision
Keep Version A and Version B available for comparison.
Proceed with a Version C facilitator view built from the same graph.
Experiment 12 — Facilitator View (Version C)
Hypothesis
The existing reasoning graph can be deterministically translated into a concise facilitator view that helps the user understand:
- what is currently known;
- what remains uncertain;
- what may explain the situation;
- why the investigation is continuing.
Questions
- Can the graph produce a useful human-facing summary without another LLM call?
- Can observations, unknowns and assumptions be clearly distinguished?
- Can duplicate or low-value graph content be filtered reliably?
- Does a concise projection improve understanding without exposing implementation detail?
- Does the panel remain useful across mocks and live Ollama output?
- Can the same view work during early, middle and terminal investigation states?
Evaluation
Completed. Visual and live-data review performed.
Result
Confirmed.
What did we learn?
- The reasoning graph already contains all the information needed for a useful human-facing summary — no additional LLM calls are required.
- Routing by semantic role (observation, question, explanation) rather than graph kind produces a more natural user experience.
- Filtering scaffolding content (scenario summaries, system/tool references, metric object descriptions, process labels) is essential to keep the view focused on findings.
- Deduplication of near-duplicate observations reduces noise without losing information.
- Epistemic clarity matters — resolved unknowns become factual observations and should be classified as known rather than still-under-investigation.
- The panel works across all investigation phases (early, active, terminal).
Decision
Close Experiment 12 as confirmed. Proceed to refine the translation through semantic classification in the next iteration.
Experiment 13 — Semantic Facilitator Translation
Hypothesis
Improving the deterministic projection from graph semantics to user-facing language — by classifying nodes by meaning rather than graph kind, suppressing scaffolding, merging duplicates, and preferring concrete observations — produces a significantly better facilitator view without changing the reasoning engine, prompts, graph generation, or any external contracts.
Questions
- Does semantic role classification (observation vs question vs explanation) route content more naturally than graph-kind classification?
- Does scaffolding suppression remove visual noise that previously dominated derived summaries?
- Does deduplication reduce redundant items that express the same observation under slightly different wording?
- Do concrete observations appear before abstract labels in ranked output?
- Does the view remain robust when consumed by the existing panel component (investigation-summary-panel-v3) without any changes to that component?
Evaluation
Completed. Tests: 37 scenarios passing across filtering, classification, deduplication, ranking, section framing, mock-data integration, and edge cases.
Result
Confirmed.
What did we learn?
- Semantic role routing outperforms kind-based routing: a node with
kind: "state"that contains concrete data (e.g., "Revenue increased 12%") is more useful as an observation than a state description. - Scaffolding suppression works best when applied early — filtering at the semantic classification stage prevents structural glue from contaminating any section.
- Three-tier filtering is effective: scaffolding patterns (highest priority), internal vocabulary (medium), then technical summary patterns (lowest).
- Deduplication by normalised text removes meaningful noise. When "Revenue increased 12%" and "Current revenue is 12% higher" express the same observation, keeping one reduces confusion without losing information.
- Resolved unknowns and assumptions are factual answers to previously unanswered questions — they should appear in the known section with an epistemic label ("Not yet established" / "To be tested") if their status hasn't been explicitly set.
- The translation adapter is the right place for this work: it is a single deterministic function, testable in isolation, and its output contracts are stable.
Result
Confirmed.
What did we learn?
- Semantic filtering significantly improved Version C.
- The remaining limitations are architectural rather than visual.
- Graph nodes still do not naturally map to facilitator language.
- Users think in investigation progress rather than graph structure.
- Version C proved the need for an intermediate narrative model.
Decision
Keep the semantic projection approach.
Do not continue improving graph projection indefinitely.
Proceed to designing an Investigation Narrative layer. Experiment 13 is closed.
Experiment 14 — Investigation Narrative Layer
Hypothesis
The graph should remain the internal reasoning model.
A separate narrative model should become the presentation model.
The facilitator UI should consume narrative state rather than graph nodes.
Questions
- What information belongs in a narrative?
- What belongs only in the graph?
- Which narrative elements can be derived deterministically?
- What should remain hidden?
- Can every facilitator panel consume the same narrative object?
Status
Architectural experiment.
Evaluation
Pending.
Emerging Direction — Investigation Narrative
The Confidence Engine architecture is becoming:
User
↓
Facilitated Conversation
↓
Reasoning Graph
↓
Investigation Narrative
↓
Workspace Projection
↓
User
The reasoning graph becomes the machine representation.
The investigation narrative becomes the human representation.
The UI simply renders whichever projection is appropriate.
This is an emerging architectural direction.
It is intentionally recorded before implementation so future experiments remain aligned.
Experiment 15 — Facilitator Behaviour Specification
Hypothesis
An expert consultant does not have a script. They have behaviours — recurring patterns of action deployed based on what they observe in the client's situation. The Confidence Engine should exhibit similar behavioural patterns rather than following a mechanical question-fill-graph cycle.
The current engine behaviour is:
Engine asks → User answers → Graph updates → Engine asks again
An expert facilitator behaviour is:
Engine assesses state → selects appropriate behaviour → acts (question, acknowledge, synthesise, challenge, pause)
Questions
- How does an expert consultant behave during an investigation?
- Which behaviours recur across investigations?
- What triggers each behaviour?
- When does the facilitator ask a question versus summarise versus expose uncertainty versus hold space?
- What distinguishes guided thinking from mechanical Q&A?
Status
Investigation — behavioural model documented, not yet implemented.
Evaluation
This experiment is primarily architectural and behavioural. No code changes are required at this stage. The deliverable is a behavioural specification that future implementation experiments will reference.
Result
Confirmed as the correct next direction.
What did we learn?
- Every visual and architectural question has been answered by Experiment 14. Further visual iteration yields diminishing returns.
- The remaining gap is not visual — it is behavioural.
- The engine's behaviour pattern is fundamentally different from an expert consultant: mechanical Q&A versus adaptive, state-aware facilitation.
- The graph captures state but not behaviour. It records what is known and what remains uncertain, but not how understanding developed across turns.
- Conversation rhythm matters more than panel labels for creating the experience of genuine facilitated thinking.
- 14 distinct facilitator behaviours were identified: Orient, Acknowledge, Observe pattern, Clarify, Validate, Connect, Challenge assumption, Refine understanding, Expose uncertainty, Decide direction, Know when to pause, Avoid premature closure, Communicate confidence honestly, Progressively narrow focus.
- Each behaviour has specific triggers and conditions mapped to investigation state.
- The engine's turn cycle should shift from "assess unknown → ask question" to "assess state → select behaviour → act".
Decision
Commit the behavioural specification. Do not implement yet. Future experiments will integrate behavioural assessment into the reasoning cycle. This document defines what the facilitator does; future work determines how the system implements it.
Status: Closed. The behavioural model is established and documented. The gap it identified — that behaviours need a decision process operating on investigation state rather than graph structure — becomes the focus of Experiment 16.
Experiment 16 — Investigation State Assessment
Hypothesis
The facilitator should never inspect the graph directly when deciding what to do next.
Instead it should act upon an assessment of the investigation — its phase, progress, evidence quality, understanding trajectory, uncertainty trend, conversation health, and behaviour readiness.
This assessment is distinct from both:
- The reasoning graph (which captures what is known)
- The investigation narrative (which translates what is known into human language)
The assessment answers: Given where we are, what kind of help is most appropriate right now?
No reasoning changes.
No prompt changes.
No UI changes.
This is an architectural experiment.
Status
Architectural.
Evaluation
Confirmed.
What did we learn?
Document observations such as:
- Investigation state is distinct from behaviour.
- Behaviour should consume assessment rather than graph structure.
- State assessment provides a stable contract between reasoning and facilitation.
- The architecture is becoming layered rather than procedural.
Decision:
Proceed to documenting the investigation turn cycle.
Experiment 16 — Emerging Architecture Observation
The Confidence Engine architecture is becoming:
User
↓
Facilitated Conversation (where behaviour lives)
↓
Behaviour Selection (consumes assessment output)
↓
Investigation State Assessment (describes investigation)
↓
Investigation Narrative (human representation of state)
↓
Reasoning Graph (machine representation)
↓
LLM / Ollama / Reasoning Engine
↓
User
This is not a final design. It is an observation emerging from 16 experiments.
What is becoming clear:
- The reasoning graph is the machine representation.
- The investigation narrative is the human representation.
- The investigation state assessment is the decision representation — it translates state into readiness signals for behaviour selection.
- Behaviour selection determines what kind of help to deploy.
- Facilitated Conversation is where that help is delivered.
Each layer has a single responsibility. Each feeds the next. No layer inspects another's implementation details.
This architecture emerged from observation, not top-down design. It may still change as future experiments test it.
Experiment 17 — Investigation Turn Cycle
Hypothesis
A complete investigation can be described as a repeating turn cycle in which every architectural layer has a single responsibility.
Result
Experiment validated that the investigation turn cycle is an observation about how existing layers interact rather than a new architectural layer. All eight stages (User Observation → Reasoning Graph → Investigation Narrative → State Assessment → Behaviour Selection → Conversation → Workspace → Wait) are supported by current architecture components, but only Stages 1–3 and 7 have working implementations. Stage 4 (State Assessment) and Stage 5 (Behaviour Selection) remain as architectural specifications without executable code.
What did we learn?
- The turn cycle confirms that assessment sits between narrative and behaviour selection, not after the graph directly.
- Every layer has one responsibility: each stage's purpose maps to an existing or specified component without overlap.
- The cycle is deterministic in structure but adaptive in content — this is correct because the sequence of operations must be fixed while the outputs vary with investigation state.
- Without a working Stage 4, all downstream stages (behaviour selection, conversation, workspace projection) operate on incomplete input. Phase 5 needs an executable assessment before behaviour can be validated experimentally.
Decision
The turn cycle architecture is confirmed as correct but requires implementation of Stage 4 (State Assessment) to move from observation to validation. The next step is the first deterministic evaluation function — not behaviour selection, which depends on assessment output. This becomes Experiment 18: First Executable Slice.
Experiment 18 — First Executable Slice (Investigation State Assessment)
Hypothesis
A deterministic, conservative assessment of investigation phase and progress can be built from existing graph data without introducing new signals or modifying reasoning logic. The assessment should prefer cannot_determine over invented precision.
Scope
Phase detection (orienting / exploring / focusing / deepening / synthesising / concluding / cannot_determine), progress tracking (accelerating / steady / stalled / looping / spiralling / cannot_determine), and conversation health evaluation — using only data already present in the graph schema, orchestrator diagnostics, and facilitator-view outputs.
Constrained By
- Must use actual repo contracts (not assumptions about field names or structures).
- Must be pure function — no network, LLM, mutation, or side effects.
- Must handle missing fields gracefully — safe with absent data.
- Must produce versioned assessment objects for future compatibility.
- Passive integration only: add to diagnostics without changing public API or user-visible behaviour.
Questions
- Can phase be reliably classified from node composition (kind/status ratio) alone?
- Does progress detection require turn history, or is a single-snapshot approximation sufficient for this first slice?
- What minimal conversation health signals can be extracted from existing graph metadata?
Evaluation
- Deterministic output across identical inputs.
- Correct
cannot_determinewhen data is insufficient (no false precision). - Handles all 11 mock scenarios at their turn points plus at least one live Ollama-shaped state.
- Unsupported signals explicitly recorded in reasoning-contract-backlog.md.
Status
Closed. The assessment is implemented, tested, and validated. See investigation-state-assessment-contract.md and lib/assessment/investigation-state-assessor.js.
Enabled for Behaviour Selection
Experiment 18 proved three things that make Experiment 19 possible:
-
Phase detection works. We can classify investigation phase (orienting / exploring / focusing / deepening / synthesising / concluding) from existing graph data with measurable confidence. This is the primary input for behaviour selection — without it, selection rules have no state to operate on.
-
Progress tracking works. Stalled progress in a focusing phase becomes a concrete signal that the facilitator should hold space rather than push. Previously this was an architectural idea; now it's observable data.
-
Conversation health is measurable. Healthy, too_broad, and user_overloaded states are detectable from question distribution and response patterns.
too_broadtriggers Clarify; healthy with resolution triggers Acknowledge — but only if the assessment layer exists to provide these signals.
Without Experiment 18, Behaviour Selection would have two options: inspect the graph directly (coupling behaviour to implementation) or use narrative fields as proxy signals (fragile by design). The assessment layer provides a stable contract — the three reliable dimensions listed above — that behaviour selection can depend on without fear of breaking when the graph schema changes.
Experiment 18 also proved that cannot_determine is not a failure mode but the correct answer when evidence is insufficient. This principle carries directly into behaviour selection: "no explicit rule matched" defaults to continue, not an invented signal.
Experiment 19 — Passive Behaviour Selection
Hypothesis
Does selecting from a small set of five behaviours (Acknowledge, Clarify, Summarise, Continue, Pause) — instead of always asking — make the investigation feel more like guided thinking and less like automated Q&A?
This is one question. Nothing else matters until this is answered.
Scope
A deterministic selector that maps investigation state assessment output to exactly one of five behaviours per turn:
- Acknowledge — when conversation health is healthy AND phase confidence is not low
- Clarify — when health is
too_broadOR (phase is orienting AND observations < 3) - Summarise — when phase is synthesising/concluding OR (≥ 3 resolved with steady progress)
- Pause — when phase is focusing AND progress is stalled; also user_overloaded health
- Continue — default when no rule matches
Selection uses priority ordering: Acknowledge > Clarify > Summarise > Pause > Continue. No scoring, no weighting, no convergence thresholds. First matching rule wins.
The selector is passive — deployed only through Developer Details diagnostics. No changes to reasoning engine, prompts, graph generation, decomposition, narrative generation, API contracts, UI behaviour, or Ollama integration.
Evaluation Criteria
- Behaviour diversity: Does the system deploy at least 3 different behaviours across a normal investigation, or does it default to Continue most of the time?
- Acknowledge appears: Does Acknowledge fire whenever new information resolves an uncertainty? If not, the trigger condition is wrong — fix it, don't abandon selection.
- Pause feels like relief, not delay: When Pause fires, does the user experience it as a natural break rather than a system failure to produce a question?
- Summarise compresses meaningfully: Does the summarised understanding feel useful or redundant?
- Conversation rhythm changes: Is there a perceptible difference between "engine always asking" and "engine sometimes acknowledging/summarising/pausing first"?
If none of these can be evaluated after 2–3 real investigations with v0.1, the experiment was too small to answer the question.
Open Questions
- Which of the five behaviours fires most frequently in practice?
- Does Acknowledge actually appear during investigations that would normally produce continuous questioning?
- Does the priority ordering create appropriate urgency (Acknowledge > Clarify > Summarise > Pause > Continue)?
- Are there cases where
cannot_determineproduces inappropriate behaviour selection — or is this the correct conservative default?
Experiment 20 — Passive Question Importance Classification
Hypothesis
Does a passive classifier that tags unresolved unknowns as important, helpful, incidental, or cannot_determine (using only existing graph fields, no scoring, no weights) produce coherent importance patterns across normal investigations?
This is one question. Nothing else matters until this is answered.
Scope
A pure function assessQuestionImportance({ node, graph }) implementing three deterministic rules:
- important — Other unresolved unknown(s) depend on this one (via
dependsOnor edges); OR text contains decision-context patterns ("whether to", "build", "launch") AND has ≥1 graph connection. - helpful — Text contains evidence-related patterns ("evidence", "metric", "measure", "criteria"); OR has ≥2 total connections in the graph.
- incidental — Default when neither important nor helpful conditions are met.
- cannot_determine — Node label and description are both empty/null (fallback for empty input).
The classifier is passive — validated only against mock scenario fixtures. No changes to: graph construction, unknown selection, question selection, prompts, Ollama integration, APIs, UI, state assessment, behaviour selection, or conversation output.
Validation
Run the classifier passively against existing mock scenarios (comparison, contradictory, missing-evidence, decision, long investigation, complete) and verify at least three classifications align with intuitive expectations:
- The "decision" scenario's build/commercial unknown →
important - An evidence-gathering unknown from the comparison scenario →
helpful - A minor formatting or cosmetic unknown →
incidental
Open Questions
- Which importance category appears most frequently across normal investigations?
- Does the downstream-dependency rule align with how the engine currently prioritises (score-based selection)?
- Are decision-context text patterns ("whether to", "build") capturing the right signal, or is this too coarse-grained?
- Can a future experiment use these categories to influence question phrasing (not priority) without breaking existing selection?
Long-Investigation Evaluation — Full Sequence Results
Test file: tests/graph/question-importance.long-investigation.test.js
Fixture: longTurns from lib/mocks/scenarios.js (5 turns, sequential mock mode)
Method: Ran assessQuestionImportance against every unresolved unknown at each turn. No rule changes before evaluation.
Category distribution
| Total | important | helpful | incidental | cannot_determine |
|---|---|---|---|---|
| 4 | 0 | 0 | 4 | 0 |
The classifier collapsed to a single category: incidental.
Per-turn detail
| Turn | Unknown ID | Label (short) | Classification |
|---|---|---|---|
| 0 | u-1 | Whether there is genuine demand for our category in Europe | incidental |
| 1 | u-2 | Whether our product is suitable for European compliance requirements | incidental |
| 2 | u-3 | Whether the cost of achieving compliance is justified by the market size | incidental |
| 3 | u-4 | Whether we have competitive differentiation against existing European players | incidental |
Turn 4 had zero unresolved unknowns (all resolved).
Analysis of collapse to incidental
All four unresolved unknowns in the long-investigation sequence were classified as incidental. Three independent factors caused this:
-
No downstream dependencies. No unresolved unknown has another unresolved unknown depending on it via
dependsOnor edges — each question is a leaf in its turn's dependency graph. The downstream-dependency rule (Rule 1, first clause) never triggers. -
Decision-text patterns missed. The DECISION_PATTERNS regex requires
"whether to"(the word "to" must follow "whether"). None of the four unknown labels contain "whether to" — they all use the structure "Whether [subject] [verb]" rather than "Whether to [verb]". Similarly, none contain "build", "launch", "proceed", or "continue.*develop". Rule 1's text-match clause (second disjunct) requires both a pattern match AND ≥1 graph connection — the pattern fails first. -
No direct graph edges. The long-investigation fixture's edges connect observations to state nodes and resolved unknowns, but the active unknown in each turn has zero incident edges (
collectConnectedIdsreturns an empty set). Without connections, the threshold-based rules (≥1 for important, ≥2 for helpful) never trigger regardless of text content.
Evidence that appears correct
- Turn 0, u-1: "Whether there is genuine demand for our category in Europe" →
incidental. This is questionable. The question frames the entire strategic decision ("should we enter Europe?"), yet no pattern matches because the edge from obs-2 to u-1 (market size evidence) only appears starting at turn 1 — at turn 0, u-1 genuinely has zero connections and no text match.
Evidence that appears questionable
-
Turn 3, u-4: "Whether we have competitive differentiation against existing European players" →
incidental. This is arguably a central question in the investigation, yet it is classified as incidental because it has zero graph edges and no decision-context keyword ("whether" alone does not match). The graph structure (edge from obs-5 to u-4) only connects observations to unknowns — but those connections exist on the source side, not the target. -
Turn 2, u-3: "Whether the cost of achieving compliance is justified by the market size" →
incidental. The word "cost" does not match EVIDENCE_PATTERNS and the node has zero direct edges. A human evaluator would classify this as important (it is the last financial feasibility gate before a go/no-go decision).
Do questions change category across turns?
No. All four resolved to incidental. There is no meaningful variation. This is not because the unknowns are identical — they address distinctly different strategic dimensions (market existence, compliance, cost, differentiation) — but because the classifier's two rule families (dependency detection and keyword matching) do not fire for any of them.
Does the result appear useful enough to keep passive?
No. A classifier that tags every unresolved unknown in a realistic long investigation as incidental provides no discrimination signal. It is technically correct under its own rules, but those rules are too narrow for the investigation structure as it currently exists. The collapse reveals a structural gap: active unknowns in this scenario have zero direct edges, and their labels use "Whether [clause]" phrasing rather than "Whether to [verb]" or other decision keywords.
Further evidence is still required if the classifier is to be considered viable. Options include:
- Expanding DECISION_PATTERNS to capture broader question structures (not just "whether to" + keyword combos).
- Adjusting how graph connections are counted for target nodes vs source nodes in edges.
- Testing against scenarios where unknowns have direct observation→unknown edges.
Evaluation status
Incomplete. The classifier did not produce useful variation across the long-investigation sequence. It passed determinism and immutability checks, but failed to discriminate between questions that clearly have different strategic importance. The hypothesis is not yet supported by this evaluation. Further evidence or rule refinement (not on this branch) is required before the classifier can be considered viable as a passive tool.
Experiment 20 — Conclusion
The hypothesis was not confirmed by this evaluation.
What happened:
- The passive classifier collapsed to a single category (
incidental) across the long-investigation scenario. - Three independent factors caused the collapse: no downstream dependencies, missed decision-text patterns (regex required "whether to" but questions used "Whether [clause]"), and zero graph edges on active unknowns.
- The keyword-only approach produced technically correct but practically useless classifications.
What this means:
Question importance cannot be judged in isolation from the decision being investigated. A question like "Do we have competitive differentiation?" is only important when compared against a clear decision target. Without that target, keyword matching and local graph structure are insufficient signals.
Decision:
The Experiment 20 classifier has not been accepted into the active engine. Its rules remain unchanged (do not expand them). The next step is Experiment 21: testing whether providing an explicit decision target allows a simple deterministic classifier to produce useful distinctions.