1543 lines
68 KiB
Markdown
1543 lines
68 KiB
Markdown
# Design Evolution Log
|
||
|
||
A chronological record of why significant design decisions were made. This is NOT a changelog. It records the product's evolution of thinking.
|
||
|
||
This document records discoveries, not decisions. Every entry represents our best understanding at that point in time and may later be superseded by a better model.
|
||
|
||
---
|
||
|
||
## Phase 1
|
||
|
||
Simple conversational investigation
|
||
|
||
Question → Answer interaction.
|
||
|
||
Purpose:
|
||
Prove the reasoning loop.
|
||
|
||
Learning:
|
||
Conversation alone does not provide sufficient context during longer investigations.
|
||
|
||
---
|
||
|
||
## Phase 2
|
||
|
||
Persistent investigation notebook
|
||
|
||
Added:
|
||
|
||
- current understanding
|
||
- original situation
|
||
- investigation history
|
||
|
||
Learning:
|
||
Users need persistent context rather than remembering previous answers.
|
||
|
||
---
|
||
|
||
## Phase 3
|
||
|
||
Document workspace
|
||
|
||
Created a coherent workspace with:
|
||
|
||
- investigation status
|
||
- current investigation
|
||
- response
|
||
- understanding
|
||
- investigation map placeholder
|
||
- situation
|
||
- history
|
||
|
||
Learning:
|
||
The interface became usable but still behaved like a document rather than a workspace.
|
||
|
||
---
|
||
|
||
## Phase 4 (Current Exploration)
|
||
|
||
Facilitated Investigation Workshop
|
||
|
||
Status:
|
||
Experimental.
|
||
|
||
Hypothesis:
|
||
|
||
The Confidence Engine is not:
|
||
|
||
- a chatbot
|
||
- a dashboard
|
||
- a form
|
||
|
||
It is a facilitated investigation workspace.
|
||
|
||
The interface should resemble the environment in which structured thinking happens.
|
||
|
||
Record discoveries rather than conclusions.
|
||
|
||
Leave room for future phases.
|
||
|
||
---
|
||
|
||
## Phase 4 — Guiding Principles
|
||
|
||
The Confidence Engine is a workspace, not a document.
|
||
|
||
People think in multiple directions simultaneously.
|
||
|
||
Useful context should be visible together.
|
||
|
||
The interface should favour thinking over scrolling.
|
||
|
||
The workspace should feel like a large desk or workshop rather than a narrow report.
|
||
|
||
The engine facilitates thinking.
|
||
|
||
The user contributes evidence.
|
||
|
||
The workspace captures shared understanding.
|
||
|
||
### Experiment 01 — Wider canvas
|
||
|
||
Hypothesis:
|
||
The document-like feeling is caused partly by the narrow outer container.
|
||
|
||
Change:
|
||
Increase the available desktop workspace width without rearranging any components.
|
||
|
||
Result:
|
||
Confirmed.
|
||
|
||
Learning:
|
||
Increasing the outer workspace width reduced the narrow-document feeling and made better use of large displays.
|
||
|
||
Unexpected learning:
|
||
Width alone did not create a workshop. The wider canvas exposed that the interface still behaves as a collection of independent cards, with supporting artefacts unsure how to use the available space.
|
||
|
||
Decision:
|
||
Keep the wider desktop canvas.
|
||
|
||
Next question:
|
||
Can grouping the interface into cognitive work zones make the wider canvas feel like a coherent investigation surface?
|
||
|
||
### Experiment 02 — Cognitive work zones
|
||
|
||
Hypothesis:
|
||
A workspace organised around what the investigator is doing will feel more coherent than one organised around equal cards or equal columns.
|
||
|
||
Result:
|
||
Partially confirmed.
|
||
|
||
Learning:
|
||
|
||
The workspace feels more coherent when organised into cognitive work zones rather than a simple document stack.
|
||
|
||
However, another distinction emerged that is more important than the zones themselves.
|
||
|
||
The interface naturally separates into two different modes:
|
||
|
||
• the active conversation between investigator and facilitator
|
||
|
||
and
|
||
|
||
• the shared workspace describing the current understanding.
|
||
|
||
Unexpected learning:
|
||
|
||
History feels incorrect when treated as reference information.
|
||
|
||
History is actually the continuation of the investigator's conversation.
|
||
|
||
Every response immediately becomes history.
|
||
|
||
The notebook should therefore grow naturally from the Response area.
|
||
|
||
The Investigation Status card currently competes with the Current Investigation card.
|
||
|
||
The current question is the primary focus.
|
||
|
||
Status is supporting context.
|
||
|
||
Decision:
|
||
|
||
Keep the cognitive-zone concept.
|
||
|
||
Refine the zones around conversational flow instead of card grouping.
|
||
|
||
Next question:
|
||
Can the workspace clearly separate conversation from shared understanding?
|
||
|
||
### Experiment 03 — Conversation versus Workspace
|
||
|
||
Hypothesis
|
||
|
||
Investigators think in two simultaneous modes.
|
||
|
||
Mode 1:
|
||
The conversation.
|
||
|
||
Question
|
||
↓
|
||
|
||
Response
|
||
↓
|
||
|
||
History
|
||
|
||
Mode 2:
|
||
The shared workspace.
|
||
|
||
Status
|
||
|
||
Understanding
|
||
|
||
Situation
|
||
|
||
Map
|
||
|
||
Separating these should make the interface feel more like a facilitated investigation than a collection of cards.
|
||
|
||
Evaluation:
|
||
Partially confirmed.
|
||
|
||
Learning:
|
||
|
||
The workspace feels more coherent when organised into cognitive zones rather than a simple document stack.
|
||
|
||
However, another distinction emerged that is more important than the zones themselves.
|
||
|
||
The interface naturally separates into two different modes:
|
||
|
||
• the active conversation between investigator and facilitator
|
||
|
||
and
|
||
|
||
• the shared workspace describing the current understanding.
|
||
|
||
Unexpected learning:
|
||
|
||
History feels incorrect when treated as reference information.
|
||
|
||
History is actually the continuation of the investigator's conversation.
|
||
|
||
Every response immediately becomes history.
|
||
|
||
The notebook should therefore grow naturally from the Response area.
|
||
|
||
The Investigation Status card currently competes with the Current Investigation card.
|
||
|
||
The current question is the primary focus.
|
||
|
||
Status is supporting context.
|
||
|
||
Decision:
|
||
|
||
Keep the cognitive-zone concept.
|
||
|
||
Refine the zones around conversational flow instead of card grouping.
|
||
|
||
Next question:
|
||
Can the workspace clearly separate conversation from shared understanding?
|
||
|
||
### Experiment 04 — Facilitated Workshop Introduction
|
||
|
||
Hypothesis
|
||
|
||
Beginning with a facilitator-style introduction will create more confidence than presenting an empty workspace.
|
||
|
||
Questions
|
||
|
||
- Does the interface feel more welcoming?
|
||
- Does reducing the visual weight of the textarea improve the first experience?
|
||
- Does separating "starting" from "investigating" feel natural?
|
||
- Does the transition into the investigation workspace feel meaningful?
|
||
|
||
Status:
|
||
Experimental.
|
||
|
||
Result:
|
||
Partially confirmed.
|
||
|
||
Learning:
|
||
|
||
The facilitator introduction reduced the intimidation of the first screen.
|
||
|
||
Replacing the empty landing page with a guided introduction improved the emotional tone.
|
||
|
||
However, stacking the introduction above the input still gives the introduction excessive visual prominence.
|
||
|
||
Repeat users may not want to repeatedly read the same introduction.
|
||
|
||
Orientation should remain available without dominating the workflow.
|
||
|
||
Decision:
|
||
|
||
Keep the introduction concept but change its spatial relationship to the workspace — move it from above to beside, making it optional rather than mandatory.
|
||
|
||
Next question:
|
||
Does a horizontal facilitator/workspace layout feel more natural?
|
||
|
||
### Experiment 05 — Facilitator Panel and Adaptive Landing Workspace
|
||
|
||
Hypothesis
|
||
|
||
Placing the facilitator beside the working area will feel more like entering a facilitated workshop than stacking instructional content above the workspace.
|
||
|
||
Allowing the user to dismiss the facilitator will reduce friction for returning users while preserving onboarding for new users.
|
||
|
||
Questions
|
||
|
||
- Does a horizontal facilitator/workspace layout feel more natural?
|
||
- Does the user's eye move naturally from facilitator to workspace?
|
||
- Does the workspace become the primary focus?
|
||
- Does "Don't show again" feel preferable to automatically hiding the introduction?
|
||
- Should the facilitator panel become an optional workspace companion rather than mandatory onboarding?
|
||
|
||
Status:
|
||
Completed.
|
||
|
||
Findings:
|
||
|
||
- A horizontal facilitator/workspace arrangement feels more natural than stacked onboarding.
|
||
- The workspace becomes the visual destination rather than the introduction.
|
||
- User-controlled dismissal is preferable to automatic hiding.
|
||
- The facilitator feels useful but visually too passive.
|
||
- Remaining issues are now visual hierarchy rather than layout architecture.
|
||
|
||
### Experiment 06 — Focused Investigation
|
||
|
||
Hypothesis
|
||
|
||
The interface should gently guide attention towards the current task without hiding supporting information.
|
||
|
||
Reducing competition between panels may improve concentration more than introducing additional colour or decoration.
|
||
|
||
Questions
|
||
|
||
- Does visual emphasis naturally guide the eye?
|
||
- Can supporting panels become quieter without disappearing?
|
||
- Does the investigation question become the obvious focal point?
|
||
- Does the workspace feel calmer?
|
||
- Are we approaching a professional investigation environment?
|
||
|
||
Status:
|
||
Closed.
|
||
|
||
## Result
|
||
|
||
Partially confirmed.
|
||
|
||
## What did we learn?
|
||
|
||
- Stronger visual hierarchy can direct attention without rearranging the interface.
|
||
- The facilitator briefing became easier to distinguish.
|
||
- Colour and tint improved separation only modestly.
|
||
- Meaning must not depend on colour.
|
||
- Areas and intent should remain distinguishable through structure, spacing, typography, borders, shape and placement.
|
||
- The initial textarea still implies that the user should provide a detailed report.
|
||
- The size of an input communicates the amount of information expected.
|
||
|
||
## Decision
|
||
|
||
Retain the useful hierarchy refinements provisionally.
|
||
|
||
Do not increase reliance on colour.
|
||
|
||
Defer dark mode and broader palette work.
|
||
|
||
The next experiment should test whether a smaller starting input better communicates that the user only needs to provide an initial observation.
|
||
|
||
Do not rewrite previous experiments.
|
||
|
||
---
|
||
|
||
### Experiment 07 — Lightweight Starting Observation
|
||
|
||
#### Hypothesis
|
||
|
||
A smaller initial input will make beginning an investigation feel easier and will communicate that the engine needs only a concise observation rather than a complete analysis.
|
||
|
||
#### Questions
|
||
|
||
- Does the input feel like a conversation starter rather than a report form?
|
||
- Is three to four visible lines sufficient?
|
||
- Does the facilitator panel and input area feel better balanced?
|
||
- Does the user understand that further detail will be gathered through questions?
|
||
- Does reducing the input height make the Analyse action easier to notice?
|
||
|
||
#### Evaluation
|
||
|
||
Pending visual review.
|
||
|
||
#### Result
|
||
|
||
Confirmed.
|
||
|
||
Four visible rows better communicates a starting observation than six.
|
||
|
||
Input size communicates expected effort.
|
||
|
||
"What have you noticed?" reinforces observational thinking.
|
||
|
||
Users are encouraged to begin rather than compose.
|
||
|
||
The facilitator and workspace now feel more balanced.
|
||
|
||
This interaction principle should continue throughout the investigation rather than existing only on the landing page.
|
||
|
||
#### Decision
|
||
|
||
Retain the smaller landing input.
|
||
|
||
Proceed to investigate consistency between the landing experience and investigation responses.
|
||
|
||
---
|
||
|
||
### Experiment 09 — Investigation Rhythm
|
||
|
||
#### Result
|
||
|
||
Partially confirmed.
|
||
|
||
#### What did we learn?
|
||
|
||
- Moving History directly beneath Response improves the sense of conversational continuity.
|
||
- The sequence Question → Response → History is cognitively coherent.
|
||
- History behaves like the growing notebook of the investigation, not general reference material.
|
||
- Allowing History to span the full workspace breaks the wider spatial model.
|
||
- Situation and Investigation Map should remain stable supporting artefacts rather than moving down as the notebook grows.
|
||
- The conversation needs a dedicated vertical lane.
|
||
|
||
#### Decision
|
||
|
||
Keep History directly connected to Response.
|
||
|
||
Refine the desktop workspace into a stable conversation lane and a stable supporting lane.
|
||
|
||
Do not rewrite previous experiments.
|
||
|
||
---
|
||
|
||
### Experiment 08 — Consistent Investigation Responses
|
||
|
||
#### Hypothesis
|
||
|
||
Every answer given during an investigation should feel like an observation, not a report.
|
||
|
||
The response component should therefore communicate the same expected effort as the initial scenario input.
|
||
|
||
#### Questions
|
||
|
||
- Does a smaller response area reduce perceived effort?
|
||
- Does the investigation feel more conversational?
|
||
- Does consistency improve confidence?
|
||
- Does the workspace become visually calmer?
|
||
- Does the current investigation remain the dominant focus?
|
||
|
||
#### Result
|
||
|
||
Confirmed.
|
||
|
||
Consistent interaction patterns reduce cognitive effort.
|
||
|
||
Users should not have to learn different behaviours between the landing page and investigation.
|
||
|
||
Smaller response areas reinforce concise observations.
|
||
|
||
The engine appears more conversational when each answer feels lightweight.
|
||
|
||
Consistency is becoming a stronger design tool than decoration.
|
||
|
||
#### Decision
|
||
|
||
Retain consistent input sizing across both contexts.
|
||
|
||
---
|
||
|
||
### Experiment 10 — Stable Conversation Column
|
||
|
||
#### Hypothesis
|
||
|
||
A persistent two-thirds conversation column beside a one-third supporting column will allow the investigation notebook to grow without moving the shared reference artefacts.
|
||
|
||
#### Questions
|
||
|
||
- Does the left column feel like one continuous investigation?
|
||
- Does History grow naturally beneath Response?
|
||
- Do Situation and Investigation Map remain easy to reference?
|
||
- Does the interface feel spatially stable as turns accumulate?
|
||
- Does showing full question text improve readability now that sufficient width exists?
|
||
|
||
#### Evaluation
|
||
|
||
Visual review completed.
|
||
|
||
#### Status
|
||
|
||
Closed.
|
||
|
||
## Result
|
||
|
||
Partially confirmed.
|
||
|
||
## What did we learn?
|
||
|
||
- The investigation workspace is beginning to feel like a genuine facilitated investigation rather than a document.
|
||
- The two-column workspace (conversation on the left, reference material on the right) is proving to be a stronger mental model than previous layouts.
|
||
- Keeping Situation and Investigation Map fixed while History grows vertically feels more natural.
|
||
- The investigation question, response and history now read as one continuous conversation.
|
||
- Developer Details have become extremely valuable.
|
||
- The graph produced by the reasoning engine is far richer than previously realised. The graph now contains structured concepts including:
|
||
|
||
- observations
|
||
- unknowns
|
||
- assumptions
|
||
- relationships
|
||
- metrics
|
||
- state
|
||
|
||
This suggests the UI should increasingly become a human-friendly projection of the graph rather than inventing separate state.
|
||
|
||
The current "Investigation in progress" panel exposes developer-oriented statistics (nodes, edges, unknowns etc.) which are useful during development but are not the most helpful representation for an end user.
|
||
|
||
---
|
||
|
||
## Emerging Direction — Graph as Source of Truth
|
||
|
||
The reasoning graph is becoming the shared source of truth for multiple UI views.
|
||
|
||
Different interfaces may project the same graph for different audiences:
|
||
|
||
- Version A — compact technical progress;
|
||
- Version B — detailed graph inspection;
|
||
- Version C — user-facing facilitator view;
|
||
- Developer Details — complete diagnostics;
|
||
- Investigation Map — future spatial projection;
|
||
- Current Question — active uncertainty projection.
|
||
|
||
The UI should not maintain separate invented summaries where the graph already contains the underlying information.
|
||
|
||
This is an emerging direction, not a final architecture decision.
|
||
|
||
---
|
||
|
||
## Emerging Direction — Facilitator Translation Layer
|
||
|
||
> The UI should progressively become a translation layer over the reasoning graph rather than maintaining separate duplicated summaries. Internal graph concepts should remain available for developers, while end users see a facilitator-style explanation of what is currently understood and what remains uncertain.
|
||
|
||
The current technical progress panel (nodes, edges, unknowns, assumptions) exposes developer-oriented statistics. These are valuable during development but not the most helpful representation for an end user.
|
||
|
||
The next direction is to explore presenting the same underlying graph data as a facilitator's notebook — what is known, what remains uncertain, and a quiet summary of the reasoning state underneath.
|
||
|
||
---
|
||
|
||
### Experiment 11 — Facilitator Progress Panel (Version B)
|
||
|
||
#### Hypothesis
|
||
|
||
The same underlying reasoning graph can be presented in a much more human-friendly way without changing the reasoning engine, API contracts, or graph generation.
|
||
|
||
A facilitator-style panel should communicate:
|
||
|
||
- what is known (resolved nodes and observations)
|
||
- what remains uncertain (unresolved unknowns and assumptions)
|
||
- a quiet summary of the reasoning state underneath
|
||
|
||
#### Questions
|
||
|
||
- Can the same graph data be translated into a facilitator-style view that end users understand more naturally?
|
||
- Does separating "known" from "still investigating" reduce cognitive load compared to node/edge counts?
|
||
- Is a quiet reasoning summary sufficient, or does it need more context?
|
||
- Does the translation-layer principle hold — presenting the graph as a notebook rather than raw data?
|
||
|
||
#### Result
|
||
|
||
Partially confirmed.
|
||
|
||
#### What did we learn?
|
||
|
||
- Version B proved that the reasoning graph contains substantially more useful information than Version A exposes.
|
||
- The graph already contains observations, unknowns, assumptions, metrics, relationships and state.
|
||
- The graph is rich enough to support multiple UI projections.
|
||
- Exposing the graph almost verbatim overwhelms the user.
|
||
- Technical categories are useful for development but do not directly communicate investigation progress.
|
||
- The user needs a translation of the graph rather than a graph browser.
|
||
- Developer Details should remain the place for complete technical inspection.
|
||
- A user-facing view needs filtering, prioritisation, deduplication and clear epistemic labels.
|
||
|
||
#### Decision
|
||
|
||
Keep Version A and Version B available for comparison.
|
||
|
||
Proceed with a Version C facilitator view built from the same graph.
|
||
|
||
---
|
||
|
||
### Experiment 12 — Facilitator View (Version C)
|
||
|
||
#### Hypothesis
|
||
|
||
The existing reasoning graph can be deterministically translated into a concise facilitator view that helps the user understand:
|
||
|
||
- what is currently known;
|
||
- what remains uncertain;
|
||
- what may explain the situation;
|
||
- why the investigation is continuing.
|
||
|
||
#### Questions
|
||
|
||
- Can the graph produce a useful human-facing summary without another LLM call?
|
||
- Can observations, unknowns and assumptions be clearly distinguished?
|
||
- Can duplicate or low-value graph content be filtered reliably?
|
||
- Does a concise projection improve understanding without exposing implementation detail?
|
||
- Does the panel remain useful across mocks and live Ollama output?
|
||
- Can the same view work during early, middle and terminal investigation states?
|
||
|
||
#### Evaluation
|
||
|
||
Completed. Visual and live-data review performed.
|
||
|
||
#### Result
|
||
|
||
Confirmed.
|
||
|
||
#### What did we learn?
|
||
|
||
- The reasoning graph already contains all the information needed for a useful human-facing summary — no additional LLM calls are required.
|
||
- Routing by semantic role (observation, question, explanation) rather than graph kind produces a more natural user experience.
|
||
- Filtering scaffolding content (scenario summaries, system/tool references, metric object descriptions, process labels) is essential to keep the view focused on findings.
|
||
- Deduplication of near-duplicate observations reduces noise without losing information.
|
||
- Epistemic clarity matters — resolved unknowns become factual observations and should be classified as known rather than still-under-investigation.
|
||
- The panel works across all investigation phases (early, active, terminal).
|
||
|
||
#### Decision
|
||
|
||
Close Experiment 12 as confirmed. Proceed to refine the translation through semantic classification in the next iteration.
|
||
|
||
---
|
||
|
||
### Experiment 13 — Semantic Facilitator Translation
|
||
|
||
#### Hypothesis
|
||
|
||
Improving the deterministic projection from graph semantics to user-facing language — by classifying nodes by *meaning* rather than *graph kind*, suppressing scaffolding, merging duplicates, and preferring concrete observations — produces a significantly better facilitator view without changing the reasoning engine, prompts, graph generation, or any external contracts.
|
||
|
||
#### Questions
|
||
|
||
- Does semantic role classification (observation vs question vs explanation) route content more naturally than graph-kind classification?
|
||
- Does scaffolding suppression remove visual noise that previously dominated derived summaries?
|
||
- Does deduplication reduce redundant items that express the same observation under slightly different wording?
|
||
- Do concrete observations appear before abstract labels in ranked output?
|
||
- Does the view remain robust when consumed by the existing panel component (investigation-summary-panel-v3) without any changes to that component?
|
||
|
||
#### Evaluation
|
||
|
||
Completed. Tests: 37 scenarios passing across filtering, classification, deduplication, ranking, section framing, mock-data integration, and edge cases.
|
||
|
||
#### Result
|
||
|
||
Confirmed.
|
||
|
||
#### What did we learn?
|
||
|
||
- Semantic role routing outperforms kind-based routing: a node with `kind: "state"` that contains concrete data (e.g., "Revenue increased 12%") is more useful as an observation than a state description.
|
||
- Scaffolding suppression works best when applied early — filtering at the semantic classification stage prevents structural glue from contaminating any section.
|
||
- Three-tier filtering is effective: scaffolding patterns (highest priority), internal vocabulary (medium), then technical summary patterns (lowest).
|
||
- Deduplication by normalised text removes meaningful noise. When "Revenue increased 12%" and "Current revenue is 12% higher" express the same observation, keeping one reduces confusion without losing information.
|
||
- Resolved unknowns and assumptions are factual answers to previously unanswered questions — they should appear in the known section with an epistemic label ("Not yet established" / "To be tested") if their status hasn't been explicitly set.
|
||
- The translation adapter is the right place for this work: it is a single deterministic function, testable in isolation, and its output contracts are stable.
|
||
|
||
#### Result
|
||
|
||
Confirmed.
|
||
|
||
#### What did we learn?
|
||
|
||
- Semantic filtering significantly improved Version C.
|
||
- The remaining limitations are architectural rather than visual.
|
||
- Graph nodes still do not naturally map to facilitator language.
|
||
- Users think in investigation progress rather than graph structure.
|
||
- Version C proved the need for an intermediate narrative model.
|
||
|
||
#### Decision
|
||
|
||
Keep the semantic projection approach.
|
||
|
||
Do not continue improving graph projection indefinitely.
|
||
|
||
Proceed to designing an Investigation Narrative layer. Experiment 13 is closed.
|
||
|
||
---
|
||
|
||
### Experiment 14 — Investigation Narrative Layer
|
||
|
||
#### Hypothesis
|
||
|
||
The graph should remain the internal reasoning model.
|
||
|
||
A separate narrative model should become the presentation model.
|
||
|
||
The facilitator UI should consume narrative state rather than graph nodes.
|
||
|
||
#### Questions
|
||
|
||
- What information belongs in a narrative?
|
||
- What belongs only in the graph?
|
||
- Which narrative elements can be derived deterministically?
|
||
- What should remain hidden?
|
||
- Can every facilitator panel consume the same narrative object?
|
||
|
||
#### Status
|
||
|
||
Architectural experiment.
|
||
|
||
#### Evaluation
|
||
|
||
Pending.
|
||
|
||
---
|
||
|
||
## Emerging Direction — Investigation Narrative
|
||
|
||
The Confidence Engine architecture is becoming:
|
||
|
||
User
|
||
|
||
↓
|
||
|
||
Facilitated Conversation
|
||
|
||
↓
|
||
|
||
Reasoning Graph
|
||
|
||
↓
|
||
|
||
Investigation Narrative
|
||
|
||
↓
|
||
|
||
Workspace Projection
|
||
|
||
↓
|
||
|
||
User
|
||
|
||
The reasoning graph becomes the machine representation.
|
||
|
||
The investigation narrative becomes the human representation.
|
||
|
||
The UI simply renders whichever projection is appropriate.
|
||
|
||
This is an emerging architectural direction.
|
||
|
||
It is intentionally recorded before implementation so future experiments remain aligned.
|
||
|
||
---
|
||
|
||
### Experiment 15 — Facilitator Behaviour Specification
|
||
|
||
#### Hypothesis
|
||
|
||
An expert consultant does not have a script. They have behaviours — recurring patterns of action deployed based on what they observe in the client's situation. The Confidence Engine should exhibit similar behavioural patterns rather than following a mechanical question-fill-graph cycle.
|
||
|
||
The current engine behaviour is:
|
||
|
||
> Engine asks → User answers → Graph updates → Engine asks again
|
||
|
||
An expert facilitator behaviour is:
|
||
|
||
> Engine assesses state → selects appropriate behaviour → acts (question, acknowledge, synthesise, challenge, pause)
|
||
|
||
#### Questions
|
||
|
||
- How does an expert consultant behave during an investigation?
|
||
- Which behaviours recur across investigations?
|
||
- What triggers each behaviour?
|
||
- When does the facilitator ask a question versus summarise versus expose uncertainty versus hold space?
|
||
- What distinguishes guided thinking from mechanical Q&A?
|
||
|
||
#### Status
|
||
|
||
Investigation — behavioural model documented, not yet implemented.
|
||
|
||
#### Evaluation
|
||
|
||
This experiment is primarily architectural and behavioural. No code changes are required at this stage. The deliverable is a behavioural specification that future implementation experiments will reference.
|
||
|
||
#### Result
|
||
|
||
Confirmed as the correct next direction.
|
||
|
||
#### What did we learn?
|
||
|
||
- Every visual and architectural question has been answered by Experiment 14. Further visual iteration yields diminishing returns.
|
||
- The remaining gap is not visual — it is behavioural.
|
||
- The engine's behaviour pattern is fundamentally different from an expert consultant: mechanical Q&A versus adaptive, state-aware facilitation.
|
||
- The graph captures *state* but not *behaviour*. It records what is known and what remains uncertain, but not how understanding developed across turns.
|
||
- Conversation rhythm matters more than panel labels for creating the experience of genuine facilitated thinking.
|
||
- 14 distinct facilitator behaviours were identified: Orient, Acknowledge, Observe pattern, Clarify, Validate, Connect, Challenge assumption, Refine understanding, Expose uncertainty, Decide direction, Know when to pause, Avoid premature closure, Communicate confidence honestly, Progressively narrow focus.
|
||
- Each behaviour has specific triggers and conditions mapped to investigation state.
|
||
- The engine's turn cycle should shift from "assess unknown → ask question" to "assess state → select behaviour → act".
|
||
|
||
#### Decision
|
||
|
||
Commit the behavioural specification. Do not implement yet. Future experiments will integrate behavioural assessment into the reasoning cycle. This document defines what the facilitator does; future work determines how the system implements it.
|
||
|
||
**Status: Closed.** The behavioural model is established and documented. The gap it identified — that behaviours need a decision process operating on investigation state rather than graph structure — becomes the focus of Experiment 16.
|
||
|
||
---
|
||
|
||
### Experiment 16 — Investigation State Assessment
|
||
|
||
#### Hypothesis
|
||
|
||
The facilitator should never inspect the graph directly when deciding what to do next.
|
||
|
||
Instead it should act upon an assessment of the investigation — its phase, progress, evidence quality, understanding trajectory, uncertainty trend, conversation health, and behaviour readiness.
|
||
|
||
This assessment is distinct from both:
|
||
|
||
- The reasoning graph (which captures *what* is known)
|
||
- The investigation narrative (which translates *what is known* into human language)
|
||
|
||
The assessment answers: *Given where we are, what kind of help is most appropriate right now?*
|
||
|
||
No reasoning changes.
|
||
|
||
No prompt changes.
|
||
|
||
No UI changes.
|
||
|
||
This is an architectural experiment.
|
||
|
||
#### Status
|
||
|
||
Architectural.
|
||
|
||
#### Evaluation
|
||
|
||
Confirmed.
|
||
|
||
---
|
||
|
||
## What did we learn?
|
||
|
||
Document observations such as:
|
||
|
||
- Investigation state is distinct from behaviour.
|
||
- Behaviour should consume assessment rather than graph structure.
|
||
- State assessment provides a stable contract between reasoning and facilitation.
|
||
- The architecture is becoming layered rather than procedural.
|
||
|
||
Decision:
|
||
|
||
Proceed to documenting the investigation turn cycle.
|
||
|
||
---
|
||
|
||
### Experiment 16 — Emerging Architecture Observation
|
||
|
||
The Confidence Engine architecture is becoming:
|
||
|
||
User
|
||
|
||
↓
|
||
|
||
Facilitated Conversation (where behaviour lives)
|
||
|
||
↓
|
||
|
||
Behaviour Selection (consumes assessment output)
|
||
|
||
↓
|
||
|
||
Investigation State Assessment (describes investigation)
|
||
|
||
↓
|
||
|
||
Investigation Narrative (human representation of state)
|
||
|
||
↓
|
||
|
||
Reasoning Graph (machine representation)
|
||
|
||
↓
|
||
|
||
LLM / Ollama / Reasoning Engine
|
||
|
||
↓
|
||
|
||
User
|
||
|
||
This is not a final design. It is an observation emerging from 16 experiments.
|
||
|
||
What is becoming clear:
|
||
|
||
- The reasoning graph is the machine representation.
|
||
- The investigation narrative is the human representation.
|
||
- The investigation state assessment is the decision representation — it translates state into readiness signals for behaviour selection.
|
||
- Behaviour selection determines what kind of help to deploy.
|
||
- Facilitated Conversation is where that help is delivered.
|
||
|
||
Each layer has a single responsibility. Each feeds the next. No layer inspects another's implementation details.
|
||
|
||
This architecture emerged from observation, not top-down design. It may still change as future experiments test it.
|
||
|
||
---
|
||
|
||
### Experiment 17 — Investigation Turn Cycle
|
||
|
||
#### Hypothesis
|
||
|
||
A complete investigation can be described as a repeating turn cycle in which every architectural layer has a single responsibility.
|
||
|
||
Result
|
||
|
||
Experiment validated that the investigation turn cycle is an *observation* about how existing layers interact rather than a new architectural layer. All eight stages (User Observation → Reasoning Graph → Investigation Narrative → State Assessment → Behaviour Selection → Conversation → Workspace → Wait) are supported by current architecture components, but only Stages 1–3 and 7 have working implementations. Stage 4 (State Assessment) and Stage 5 (Behaviour Selection) remain as architectural specifications without executable code.
|
||
|
||
What did we learn?
|
||
|
||
- The turn cycle confirms that assessment sits between narrative and behaviour selection, not after the graph directly.
|
||
- Every layer has one responsibility: each stage's purpose maps to an existing or specified component without overlap.
|
||
- The cycle is deterministic in structure but adaptive in content — this is correct because the *sequence* of operations must be fixed while the *outputs* vary with investigation state.
|
||
- Without a working Stage 4, all downstream stages (behaviour selection, conversation, workspace projection) operate on incomplete input. Phase 5 needs an executable assessment before behaviour can be validated experimentally.
|
||
|
||
Decision
|
||
|
||
The turn cycle architecture is confirmed as correct but requires implementation of Stage 4 (State Assessment) to move from observation to validation. The next step is the first deterministic evaluation function — not behaviour selection, which depends on assessment output. This becomes Experiment 18: First Executable Slice.
|
||
|
||
---
|
||
|
||
### Experiment 18 — First Executable Slice (Investigation State Assessment)
|
||
|
||
#### Hypothesis
|
||
|
||
A deterministic, conservative assessment of investigation phase and progress can be built from existing graph data without introducing new signals or modifying reasoning logic. The assessment should prefer `cannot_determine` over invented precision.
|
||
|
||
#### Scope
|
||
|
||
Phase detection (orienting / exploring / focusing / deepening / synthesising / concluding / cannot_determine), progress tracking (accelerating / steady / stalled / looping / spiralling / cannot_determine), and conversation health evaluation — using only data already present in the graph schema, orchestrator diagnostics, and facilitator-view outputs.
|
||
|
||
#### Constrained By
|
||
|
||
- Must use actual repo contracts (not assumptions about field names or structures).
|
||
- Must be pure function — no network, LLM, mutation, or side effects.
|
||
- Must handle missing fields gracefully — safe with absent data.
|
||
- Must produce versioned assessment objects for future compatibility.
|
||
- Passive integration only: add to diagnostics without changing public API or user-visible behaviour.
|
||
|
||
#### Questions
|
||
|
||
1. Can phase be reliably classified from node composition (kind/status ratio) alone?
|
||
2. Does progress detection require turn history, or is a single-snapshot approximation sufficient for this first slice?
|
||
3. What minimal conversation health signals can be extracted from existing graph metadata?
|
||
|
||
#### Evaluation
|
||
|
||
- Deterministic output across identical inputs.
|
||
- Correct `cannot_determine` when data is insufficient (no false precision).
|
||
- Handles all 11 mock scenarios at their turn points plus at least one live Ollama-shaped state.
|
||
- Unsupported signals explicitly recorded in reasoning-contract-backlog.md.
|
||
|
||
#### Status
|
||
|
||
**Closed.** The assessment is implemented, tested, and validated. See `investigation-state-assessment-contract.md` and `lib/assessment/investigation-state-assessor.js`.
|
||
|
||
#### Enabled for Behaviour Selection
|
||
|
||
Experiment 18 proved three things that make Experiment 19 possible:
|
||
|
||
1. **Phase detection works.** We can classify investigation phase (orienting / exploring / focusing / deepening / synthesising / concluding) from existing graph data with measurable confidence. This is the primary input for behaviour selection — without it, selection rules have no state to operate on.
|
||
|
||
2. **Progress tracking works.** Stalled progress in a focusing phase becomes a concrete signal that the facilitator should hold space rather than push. Previously this was an architectural idea; now it's observable data.
|
||
|
||
3. **Conversation health is measurable.** Healthy, too_broad, and user_overloaded states are detectable from question distribution and response patterns. `too_broad` triggers Clarify; healthy with resolution triggers Acknowledge — but only if the assessment layer exists to provide these signals.
|
||
|
||
Without Experiment 18, Behaviour Selection would have two options: inspect the graph directly (coupling behaviour to implementation) or use narrative fields as proxy signals (fragile by design). The assessment layer provides a stable contract — the three reliable dimensions listed above — that behaviour selection can depend on without fear of breaking when the graph schema changes.
|
||
|
||
Experiment 18 also proved that `cannot_determine` is not a failure mode but the correct answer when evidence is insufficient. This principle carries directly into behaviour selection: "no explicit rule matched" defaults to continue, not an invented signal.
|
||
|
||
---
|
||
|
||
### Experiment 19 — Passive Behaviour Selection
|
||
|
||
#### Hypothesis
|
||
|
||
Does selecting from a small set of five behaviours (Acknowledge, Clarify, Summarise, Continue, Pause) — instead of always asking — make the investigation feel more like guided thinking and less like automated Q&A?
|
||
|
||
This is one question. Nothing else matters until this is answered.
|
||
|
||
#### Scope
|
||
|
||
A deterministic selector that maps investigation state assessment output to exactly one of five behaviours per turn:
|
||
|
||
1. **Acknowledge** — when conversation health is healthy AND phase confidence is not low
|
||
2. **Clarify** — when health is `too_broad` OR (phase is orienting AND observations < 3)
|
||
3. **Summarise** — when phase is synthesising/concluding OR (≥ 3 resolved with steady progress)
|
||
4. **Pause** — when phase is focusing AND progress is stalled; also user_overloaded health
|
||
5. **Continue** — default when no rule matches
|
||
|
||
Selection uses priority ordering: Acknowledge > Clarify > Summarise > Pause > Continue. No scoring, no weighting, no convergence thresholds. First matching rule wins.
|
||
|
||
The selector is passive — deployed only through Developer Details diagnostics. No changes to reasoning engine, prompts, graph generation, decomposition, narrative generation, API contracts, UI behaviour, or Ollama integration.
|
||
|
||
#### Evaluation Criteria
|
||
|
||
1. **Behaviour diversity:** Does the system deploy at least 3 different behaviours across a normal investigation, or does it default to Continue most of the time?
|
||
2. **Acknowledge appears:** Does Acknowledge fire whenever new information resolves an uncertainty? If not, the trigger condition is wrong — fix it, don't abandon selection.
|
||
3. **Pause feels like relief, not delay:** When Pause fires, does the user experience it as a natural break rather than a system failure to produce a question?
|
||
4. **Summarise compresses meaningfully:** Does the summarised understanding feel useful or redundant?
|
||
5. **Conversation rhythm changes:** Is there a perceptible difference between "engine always asking" and "engine sometimes acknowledging/summarising/pausing first"?
|
||
|
||
If none of these can be evaluated after 2–3 real investigations with v0.1, the experiment was too small to answer the question.
|
||
|
||
#### Open Questions
|
||
|
||
- Which of the five behaviours fires most frequently in practice?
|
||
- Does Acknowledge actually appear during investigations that would normally produce continuous questioning?
|
||
- Does the priority ordering create appropriate urgency (Acknowledge > Clarify > Summarise > Pause > Continue)?
|
||
- Are there cases where `cannot_determine` produces inappropriate behaviour selection — or is this the correct conservative default?
|
||
|
||
---
|
||
|
||
### Experiment 20 — Passive Question Importance Classification
|
||
|
||
#### Hypothesis
|
||
|
||
Does a passive classifier that tags unresolved unknowns as `important`, `helpful`, `incidental`, or `cannot_determine` (using only existing graph fields, no scoring, no weights) produce coherent importance patterns across normal investigations?
|
||
|
||
This is one question. Nothing else matters until this is answered.
|
||
|
||
#### Scope
|
||
|
||
A pure function `assessQuestionImportance({ node, graph })` implementing three deterministic rules:
|
||
|
||
1. **important** — Other unresolved unknown(s) depend on this one (via `dependsOn` or edges); OR text contains decision-context patterns ("whether to", "build", "launch") AND has ≥1 graph connection.
|
||
2. **helpful** — Text contains evidence-related patterns ("evidence", "metric", "measure", "criteria"); OR has ≥2 total connections in the graph.
|
||
3. **incidental** — Default when neither important nor helpful conditions are met.
|
||
4. **cannot_determine** — Node label and description are both empty/null (fallback for empty input).
|
||
|
||
The classifier is passive — validated only against mock scenario fixtures. No changes to: graph construction, unknown selection, question selection, prompts, Ollama integration, APIs, UI, state assessment, behaviour selection, or conversation output.
|
||
|
||
#### Validation
|
||
|
||
Run the classifier passively against existing mock scenarios (comparison, contradictory, missing-evidence, decision, long investigation, complete) and verify at least three classifications align with intuitive expectations:
|
||
|
||
- The "decision" scenario's build/commercial unknown → `important`
|
||
- An evidence-gathering unknown from the comparison scenario → `helpful`
|
||
- A minor formatting or cosmetic unknown → `incidental`
|
||
|
||
#### Open Questions
|
||
|
||
- Which importance category appears most frequently across normal investigations?
|
||
- Does the downstream-dependency rule align with how the engine currently prioritises (score-based selection)?
|
||
- Are decision-context text patterns ("whether to", "build") capturing the right signal, or is this too coarse-grained?
|
||
- Can a future experiment use these categories to influence question phrasing (not priority) without breaking existing selection?
|
||
|
||
---
|
||
|
||
### Long-Investigation Evaluation — Full Sequence Results
|
||
|
||
**Test file:** `tests/graph/question-importance.long-investigation.test.js`
|
||
**Fixture:** `longTurns` from `lib/mocks/scenarios.js` (5 turns, sequential mock mode)
|
||
**Method:** Ran `assessQuestionImportance` against every unresolved unknown at each turn. No rule changes before evaluation.
|
||
|
||
#### Category distribution
|
||
|
||
| Total | important | helpful | incidental | cannot_determine |
|
||
|-------|-----------|---------|------------|-------------------|
|
||
| 4 | 0 | 0 | 4 | 0 |
|
||
|
||
The classifier collapsed to a single category: **`incidental`**.
|
||
|
||
#### Per-turn detail
|
||
|
||
| Turn | Unknown ID | Label (short) | Classification |
|
||
|------|------------|---------------|----------------|
|
||
| 0 | u-1 | Whether there is genuine demand for our category in Europe | incidental |
|
||
| 1 | u-2 | Whether our product is suitable for European compliance requirements | incidental |
|
||
| 2 | u-3 | Whether the cost of achieving compliance is justified by the market size | incidental |
|
||
| 3 | u-4 | Whether we have competitive differentiation against existing European players | incidental |
|
||
|
||
Turn 4 had zero unresolved unknowns (all resolved).
|
||
|
||
#### Analysis of collapse to `incidental`
|
||
|
||
All four unresolved unknowns in the long-investigation sequence were classified as `incidental`. Three independent factors caused this:
|
||
|
||
1. **No downstream dependencies.** No unresolved unknown has another unresolved unknown depending on it via `dependsOn` or edges — each question is a leaf in its turn's dependency graph. The downstream-dependency rule (Rule 1, first clause) never triggers.
|
||
|
||
2. **Decision-text patterns missed.** The DECISION_PATTERNS regex requires `"whether to"` (the word "to" must follow "whether"). None of the four unknown labels contain "whether to" — they all use the structure "Whether [subject] [verb]" rather than "Whether to [verb]". Similarly, none contain "build", "launch", "proceed", or "continue.*develop". Rule 1's text-match clause (second disjunct) requires both a pattern match AND ≥1 graph connection — the pattern fails first.
|
||
|
||
3. **No direct graph edges.** The long-investigation fixture's edges connect observations to state nodes and resolved unknowns, but the active unknown in each turn has zero incident edges (`collectConnectedIds` returns an empty set). Without connections, the threshold-based rules (≥1 for important, ≥2 for helpful) never trigger regardless of text content.
|
||
|
||
#### Evidence that appears correct
|
||
|
||
- Turn 0, u-1: "Whether there is genuine demand for our category in Europe" → `incidental`. This is questionable. The question frames the entire strategic decision ("should we enter Europe?"), yet no pattern matches because the edge from obs-2 to u-1 (market size evidence) only appears starting at turn 1 — at turn 0, u-1 genuinely has zero connections and no text match.
|
||
|
||
#### Evidence that appears questionable
|
||
|
||
- Turn 3, u-4: "Whether we have competitive differentiation against existing European players" → `incidental`. This is arguably a central question in the investigation, yet it is classified as incidental because it has zero graph edges and no decision-context keyword ("whether" alone does not match). The graph structure (edge from obs-5 to u-4) only connects observations to unknowns — but those connections exist on the source side, not the target.
|
||
|
||
- Turn 2, u-3: "Whether the cost of achieving compliance is justified by the market size" → `incidental`. The word "cost" does not match EVIDENCE_PATTERNS and the node has zero direct edges. A human evaluator would classify this as important (it is the last financial feasibility gate before a go/no-go decision).
|
||
|
||
#### Do questions change category across turns?
|
||
|
||
No. All four resolved to `incidental`. There is no meaningful variation. This is not because the unknowns are identical — they address distinctly different strategic dimensions (market existence, compliance, cost, differentiation) — but because the classifier's two rule families (dependency detection and keyword matching) do not fire for any of them.
|
||
|
||
#### Does the result appear useful enough to keep passive?
|
||
|
||
**No.** A classifier that tags every unresolved unknown in a realistic long investigation as `incidental` provides no discrimination signal. It is technically correct under its own rules, but those rules are too narrow for the investigation structure as it currently exists. The collapse reveals a structural gap: active unknowns in this scenario have zero direct edges, and their labels use "Whether [clause]" phrasing rather than "Whether to [verb]" or other decision keywords.
|
||
|
||
Further evidence is still required if the classifier is to be considered viable. Options include:
|
||
- Expanding DECISION_PATTERNS to capture broader question structures (not just "whether to" + keyword combos).
|
||
- Adjusting how graph connections are counted for target nodes vs source nodes in edges.
|
||
- Testing against scenarios where unknowns have direct observation→unknown edges.
|
||
|
||
#### Evaluation status
|
||
|
||
**Incomplete.** The classifier did not produce useful variation across the long-investigation sequence. It passed determinism and immutability checks, but failed to discriminate between questions that clearly have different strategic importance. The hypothesis is not yet supported by this evaluation. Further evidence or rule refinement (not on this branch) is required before the classifier can be considered viable as a passive tool.
|
||
|
||
---
|
||
|
||
### Experiment 20 — Conclusion
|
||
|
||
The hypothesis was not confirmed by this evaluation.
|
||
|
||
**What happened:**
|
||
|
||
- The passive classifier collapsed to a single category (`incidental`) across the long-investigation scenario.
|
||
- Three independent factors caused the collapse: no downstream dependencies, missed decision-text patterns (regex required "whether to" but questions used "Whether [clause]"), and zero graph edges on active unknowns.
|
||
- The keyword-only approach produced technically correct but practically useless classifications.
|
||
|
||
**What this means:**
|
||
|
||
Question importance cannot be judged in isolation from the decision being investigated. A question like "Do we have competitive differentiation?" is only important when compared against a clear decision target. Without that target, keyword matching and local graph structure are insufficient signals.
|
||
|
||
**Decision:**
|
||
|
||
The Experiment 20 classifier has not been accepted into the active engine. Its rules remain unchanged (do not expand them). The next step is Experiment 21: testing whether providing an explicit decision target allows a simple deterministic classifier to produce useful distinctions.
|
||
|
||
---
|
||
|
||
## Phase Transition
|
||
|
||
Record that the project has moved from:
|
||
|
||
Interface Design
|
||
→
|
||
Facilitated Investigation
|
||
→
|
||
Behavioural Architecture
|
||
→
|
||
System Architecture
|
||
|
||
Future work should validate these layers rather than introduce new ones.
|
||
|
||
---
|
||
|
||
## Emerging Direction — Graph as Source of Truth
|
||
|
||
The first UX experiments focused on workspace structure.
|
||
|
||
The next series will focus on investigation rhythm and behaviour.
|
||
|
||
Future experiments should explore:
|
||
|
||
- how conversations unfold (behavioural, not visual)
|
||
- how understanding evolves across turns
|
||
- how the facilitator selects its behavioural response
|
||
- how confidence is gradually built through action, not description
|
||
- what state assessment enables better question selection
|
||
|
||
The objective is no longer to arrange cards or translate panels.
|
||
|
||
The objective is to make each turn of the investigation feel like a natural step in a guided thinking process.
|
||
|
||
The objective is to make the investigation feel like a natural facilitated conversation.
|
||
|
||
---
|
||
|
||
### Experiment 21 — Question Relevance Against Decision Target
|
||
|
||
#### Hypothesis
|
||
|
||
Does giving the classifier an explicit decision target allow it to distinguish questions that could change the decision from questions that are merely useful or incidental?
|
||
|
||
This is one question. Nothing else matters until this is answered.
|
||
|
||
#### Scope
|
||
|
||
A pure function `assessQuestionRelevanceToDecision({ decisionTarget, unknown, graph })` implementing four deterministic rules:
|
||
|
||
1. **could_change_decision** — The question directly mirrors the decision's core action (e.g., "whether to enter", "should we launch", "whether there is [demand/market/need]") AND the decision target contains a matching action keyword. Answering could reasonably reverse the proposed action.
|
||
2. **supports_decision** — Necessary precondition (e.g., compliance, cost feasibility) OR supporting context (e.g., differentiation, competitive position). The answer would improve confidence or evidence but is less likely to reverse the decision alone.
|
||
3. **unlikely_to_change_decision** — Background detail or comparative reference that does not affect the decision conditions.
|
||
4. **cannot_determine** — Decision target or unknown is missing, empty, or too unclear to compare honestly.
|
||
|
||
The classifier is passive — validated only against mock scenario fixtures. No changes to: graph construction, question importance classifier, unknown selection, question selection, prompts, Ollama integration, APIs, UI, state assessment, behaviour selection, conversation output, or engine behaviour in any way.
|
||
|
||
#### Decision Target
|
||
|
||
For the long-investigation scenario, use an explicit target from the fixture:
|
||
|
||
> Should we enter the European market with our SaaS analytics platform?
|
||
|
||
Do not attempt to discover the decision target automatically. For this experiment, the decision target is supplied by the test fixture.
|
||
|
||
#### Evaluation
|
||
|
||
Run the classifier passively across the same long-investigation turns used in Experiment 20 (turns 0–3). Record per-turn classification. Compare with Experiment 20 results. Expect at least two distinct categories — not a collapse to one.
|
||
|
||
#### Questions
|
||
|
||
- Does providing an explicit decision target enable more useful distinctions than keyword-only matching?
|
||
- Do the four categories map intuitively to how a human evaluator would judge relevance?
|
||
- Or does the deterministic rule set still miss cases that appear obviously important?
|
||
|
||
---
|
||
|
||
### Experiment 22 — Question Relevance Against Explicit Decision Conditions
|
||
|
||
Explicit decision conditions were supplied:
|
||
|
||
1. Credible customer demand exists in Europe
|
||
2. European compliance is achievable
|
||
3. The expected market value justifies the cost of entry
|
||
4. The product offers sufficient competitive differentiation
|
||
|
||
Each long-investigation unknown matched a different deciding condition. All four correctly classified as `tests_deciding_condition`.
|
||
|
||
Category variety is not automatically a measure of quality — here, uniformity (all four as decisive) is correct because each question directly tests a required condition.
|
||
|
||
The classifier remains passive and is not in the active reasoning path.
|
||
|
||
---
|
||
|
||
## Experiment 23 — Decision Condition Status Assessment
|
||
|
||
**Status:** Concluded (passive layer)
|
||
|
||
### Hypothesis
|
||
|
||
Given resolved graph evidence, we can determine which explicit decision conditions are `established`, `contradicted`, `unresolved`, or `cannot_determine` using only existing node fields and simple keyword matching — no scoring, no weights, no LLM calls.
|
||
|
||
### Scope
|
||
|
||
- Pure passive classifier: reads `resolvedNodeIds`, `nodes[].label`, `nodes[].description`, `nodes[].status`
|
||
- Four-state classification with contradiction-precedence-over-support rule
|
||
- Uses the same concept groups that power Experiment 22's question relevance (demand, compliance, value_cost, differentiation)
|
||
- Returns evidence node IDs alongside status for traceability
|
||
|
||
### Implementation
|
||
|
||
File: `lib/graph/decision-condition-status.js`
|
||
|
||
Classification rules (evaluated in order):
|
||
|
||
1. **cannot_determine** — missing condition text or incomplete graph
|
||
2. **contradicted** — resolved evidence contains a contradiction phrase (e.g. "does not support", "not achievable")
|
||
3. **established** — resolved evidence supports the condition AND no contradiction found
|
||
4. **unresolved** — condition is relevant but no resolved evidence establishes or contradicts it
|
||
|
||
Contradiction detection uses universal phrases applied to ALL resolved node texts, regardless of condition category. This keeps the system robust: any observation with "does not support" weakens any relevant condition.
|
||
|
||
Support detection first determines which concept categories a condition text matches (from its keywords), then checks whether any resolved node text contains supporting keywords from those matched categories.
|
||
|
||
### Evaluation method
|
||
|
||
- 39 focused tests: established (5), contradicted (4), unresolved (4), cannot_determine (6), precedence (3), immutability (2), long-investigation sequence (15)
|
||
- Long-investigation sequence tested across turns 0–4 of the "long" scenario fixture
|
||
|
||
### Observed status transitions (long investigation)
|
||
|
||
| Turn | Resolved nodes | Demand | Compliance | Value/cost | Differentiation |
|
||
|------|------------------|---------------|------------------|-----------------|-----------------|
|
||
| 0 | — | unresolved | unresolved | unresolved | unresolved |
|
||
| 1 | u-1 | established | unresolved | unresolved | unresolved |
|
||
| 2 | u-1, u-2 | established | established | unresolved | unresolved |
|
||
| 3 | u-1, u-2, u-3 | established | established | established | unresolved |
|
||
| 4 | u-1, u-2, u-3, u-4 | established | established | established | established |
|
||
|
||
Note: Observation nodes (obs-*) are NEVER in `resolvedNodeIds` — they remain "known" observations. Only unknowns become resolved during investigation turns. This means contradiction phrases in observations don't trigger detection with the current implementation.
|
||
|
||
### Limitations
|
||
|
||
- Contradiction detection only works on resolved node labels/descriptions, not on observation notes (which is a deliberate design choice to avoid false positives from unverified data)
|
||
- Absent conditions are `unresolved`, never `contradicted` — absence of evidence ≠ evidence of absence
|
||
- No handling for partially established conditions (e.g. some sub-conditions met, others not)
|
||
- Keyword matching is case-insensitive substring only; no stemming or semantic understanding
|
||
|
||
### Conclusion
|
||
|
||
The assessment works correctly across all test cases: 39/39 passing. It provides a useful passive layer showing which conditions have been addressed by the investigation without any engine mutation or new graph structure. The long-investigation sequence shows natural progression from `unresolved` to `established` as evidence accumulates, confirming the system behaves as intended during an investigation's lifecycle.
|
||
|
||
---
|
||
|
||
## Experiment 24A — Evidence Direction Classification
|
||
|
||
**Status:** Completed (passive layer)
|
||
|
||
### Hypothesis
|
||
|
||
Answer evidence can be distinguished from resolved-question wording and classified by whether it supports, contradicts or merely informs a decision condition.
|
||
|
||
### What was implemented
|
||
|
||
A passive deterministic evidence-direction classifier (`lib/graph/evidence-direction.js`) that reads existing evidence text directly — not the resolved-question label — and classifies each piece of resolved evidence as `supports`, `contradicts`, `informs`, or `cannot_determine` relative to an explicit decision condition. Concept groups (demand, compliance, value_cost, differentiation) are defined locally within the classifier file, removing avoidable coupling from the mock fixture library.
|
||
|
||
### Observed results
|
||
|
||
- market evidence (`"European analytics SaaS market valued at approximately €8B and growing 15% annually"`) → `supports` demand condition
|
||
- missing EU data residency (`"Our platform does not currently support EU data residency requirements"`) → `contradicts` compliance condition
|
||
- cost evidence (`"Achieving compliance would require approximately 6 months and $500K engineering investment"`) → `informs` value-versus-cost condition
|
||
- unique capability evidence (`"Our real-time collaboration feature has no direct European equivalent"`) → `supports` differentiation condition
|
||
|
||
### What was learned
|
||
|
||
- Resolving a question is not the same as establishing its condition.
|
||
- Answer evidence must be inspected directly, not inferred from resolved-question wording.
|
||
- Relevant evidence may inform without proving.
|
||
- Contradiction must remain attached to the condition it concerns.
|
||
|
||
### Focused test results
|
||
|
||
22 focused tests pass (supports × 2, contradicts × 1, informs × 2, cannot_determine × 7, determinism × 2, immutability × 2, long-investigation examples × 4, unrelated evidence × 2).
|
||
|
||
### Cleanup performed
|
||
|
||
- Moved `EVIDENCE_DIRECTION_GROUPS` from `lib/mocks/scenarios.js` into `lib/graph/evidence-direction.js`.
|
||
- Removed unused `DECISION_CONDITIONS` and `CONTRADICTION_KEYWORDS` exports from `lib/mocks/scenarios.js`.
|
||
- Removed the cross-module import that coupled evidence-direction to the mock library.
|
||
|
||
### Experiment 23 compatibility
|
||
|
||
`decision-condition-status.test.js` (39 tests) and `question-decision-conditions.test.js` (40 tests) both continue to pass. No behaviour change in Experiment 23 or 22 classifiers.
|
||
|
||
### Next steps
|
||
|
||
Do not yet integrate evidence direction into active reasoning. That belongs to a separate follow-on experiment. Do not amend Experiment 23 condition statuses here.
|
||
|
||
---
|
||
|
||
## Experiment 24B — Derive Condition Status from Answer Evidence
|
||
|
||
**Status:** Completed (passive layer)
|
||
|
||
### Hypothesis
|
||
|
||
Decision condition status should be derived from linked answer evidence (supports/contradicts/informs), not from the resolved-question label. When mapped unknowns and linked observations exist, use `assessEvidenceDirection`. When no mapped unknown or linked evidence exists, fall back to conservative keyword inspection of resolved nodes.
|
||
|
||
### What was implemented
|
||
|
||
Two assessment paths in `lib/graph/decision-condition-status.js`:
|
||
|
||
**Path 1 — Linked evidence path:** when a resolved unknown and linked observation/evidence nodes exist via edges, invoke `assessEvidenceDirection` for each linked observation; derive status from the classified direction (supports → established, contradicts → contradicted, informs → unresolved). Condition text is now passed as `{ text: condition }` to avoid the string-to-object mismatch that caused all directions to return `cannot_determine`.
|
||
|
||
**Path 2 — Conservative fallback:** when no mapped unknown or linked evidence exists (focused tests use deliberately minimal graphs with resolved nodes but no edge structure), inspect all resolved evidence-like nodes for contradiction phrases first, then check the matched unknown's label plus any linked observations for category-specific support keywords. Generic cost/investment phrases are excluded from value_cost support detection to prevent classifying contextual compliance data as proof of value justification.
|
||
|
||
### Corrected long-investigation statuses
|
||
|
||
| Condition | Status | Rationale |
|
||
|---|---|---|
|
||
| Demand → established | Linked evidence (`€8B market, 15% growing`) supports the demand condition |
|
||
| Compliance → contradicted | Linked evidence ("does not support EU data residency") contains compliance negation phrase |
|
||
| Value versus cost → unresolved | Cost evidence ("6 months, $500K engineering investment") is contextual; does not prove value justifies cost |
|
||
| Differentiation → established | Linked evidence ("no direct European equivalent") supports differentiation |
|
||
|
||
### Focused test changes
|
||
|
||
- Generic cost/investment evidence (`$500K investment`) now correctly returns **unresolved** for value_cost (was erroneously established) — updated two focused tests and their descriptions.
|
||
- Single-node contradiction tests now accept fallback resolved unknowns when pattern keywords don't match the node label (na-1 → "not achievable" → contradicted).
|
||
- EvidenceNodeIds test adjusted: unresolved conditions may retain linked observation IDs when the unknown was resolved but evidence was contextual only.
|
||
|
||
### What was learned
|
||
|
||
- Linked answer evidence controls condition status; resolved-question labels are not proof.
|
||
- Minimal-graph tests require a conservative resolved-evidence fallback path that inspects matched unknown + linked observations for support, all resolved nodes for contradiction.
|
||
- Generic cost phrases must not establish value_cost — value justification requires explicit supporting language.
|
||
- The classifier remains passive: no scores, weights, graph fields, or LLM calls.
|
||
|
||
### Focused test results
|
||
|
||
36 focused tests pass (established × 5, contradicted × 2, unresolved × 3, long-investigation sequence × 19, edge-case + determinism × 7).
|
||
22 evidence-direction tests pass.
|
||
40 question-decision-conditions tests pass.
|
||
|
||
### Experiment 24A unchanged
|
||
|
||
Evidence-direction classifier (`evidence-direction.js`) is untouched. All 22 tests pass. The fix was only in `decision-condition-status.js` and test expectations.
|
||
|
||
### Active engine behaviour unchanged
|
||
|
||
No changes to the active reasoning loop, prompt generation, or question-selection logic. This layer reads graph state only.
|
||
|
||
---
|
||
|
||
### Experiment 25A — Evidence-Condition Scope Comparison
|
||
|
||
**Status:** Completed (passive layer)
|
||
|
||
#### Hypothesis
|
||
|
||
Before evidence can support or contradict a condition, the engine must establish that both refer to the same:
|
||
|
||
- subject;
|
||
- timeframe;
|
||
- type of claim.
|
||
|
||
A small deterministic check distinguishes direct evidence from evidence that is relevant but answers a different question. Experiment 24B works mechanically, but the compliance example exposed a remaining question about whether the evidence and condition refer to the same claim and timeframe.
|
||
|
||
#### The Present-State Versus Future-Feasibility Distinction
|
||
|
||
The engine has observed this ambiguity repeatedly:
|
||
|
||
> Condition: *European compliance is achievable*
|
||
> Evidence: *Our platform does not currently support EU data residency requirements*
|
||
|
||
The evidence proves the platform is not compliant now. It does not prove that compliance cannot be achieved. Treating this as a direct contradiction may be too strong without first confirming scope alignment.
|
||
|
||
#### Implementation Scope
|
||
|
||
A pure function `assessEvidenceConditionScope({ condition, evidenceNode })` implementing four deterministic rules using small explicit language patterns:
|
||
|
||
1. **present_state** — Both the condition and evidence describe a current, existing situation (keywords: "currently", "does not support", "is", "has", "supports", "compliant").
|
||
2. **future_feasibility** — The condition concerns future achievability or feasibility while the evidence describes present state (keywords for future: "can be achieved", "is achievable", "will", "would require").
|
||
3. **subject_mismatch** — The evidence and condition address different subjects (e.g., compliance vs market demand). Detected via shared category from evidence-direction concept groups.
|
||
4. **cannot_determine** — Either input is missing or too unclear to compare honestly.
|
||
|
||
No LLM calls, no scoring, no weights, no graph schema changes, no mutation.
|
||
|
||
#### Evaluated Examples
|
||
|
||
| Condition | Evidence | Expected Scope |
|
||
|---|---|---|
|
||
| The platform currently supports EU data residency requirements | Our platform does not currently support EU data residency requirements | `direct_match` |
|
||
| European compliance can be achieved within an acceptable time and cost | Our platform does not currently support EU data residency requirements | `different_timeframe` |
|
||
| European compliance can be achieved within an acceptable time and cost | Achieving compliance would require approximately six months and $500K | `partial_match` |
|
||
| Credible customer demand exists in Europe | The European analytics SaaS market is valued at approximately €8B and growing 15% annually | `direct_match` |
|
||
|
||
#### Findings
|
||
|
||
- Present-state conditions versus present-state evidence produce clean `direct_match` signals.
|
||
- Future-feasibility conditions versus current-evidence observations correctly produce `different_timeframe`.
|
||
- The compliance example now has a documented scope classification that explains *why* it is a contradiction at the evidence level but not necessarily at the condition level.
|
||
- Subject-mismatch detection via shared concept categories works reliably for the four established categories (demand, compliance, value_cost, differentiation).
|
||
|
||
#### Phrase list additions
|
||
|
||
The future-feasibility phrase list was extended from `"can be achieved"` to also include `"can achieve"`, `"be achieved"`, and `"is achievable"`. These address cases where present-state evidence ("Our team currently has no EU regulatory expertise") and future-feasibility conditions ("We can achieve European compliance within 12 months" / "European compliance is achievable") must be recognised as referring to different timeframes.
|
||
|
||
#### Limitations
|
||
|
||
- Present-state evidence and future-feasibility conditions can refer to different timeframes; scope detection must check both inputs independently.
|
||
- Timeframe detection relies on explicit keyword patterns. It does not attempt general tense parsing or natural-language understanding. The phrase handling is provisional — not a finished language-understanding system.
|
||
- Subject matching uses substring keyword overlap from existing concept groups; it may miss evidence that is semantically relevant but uses different terminology.
|
||
- `partial_match` is a heuristic classification based on presence of feasibility-related keywords in the evidence rather than a deep analysis of partial claim coverage.
|
||
- The function does not call or depend on the evidence-direction classifier (experiments remain isolated).
|
||
|
||
#### Experiment 25B — Scope-Aware Condition Status With Actual Fixture Wording
|
||
|
||
**Status:** Completed (passive layer)
|
||
|
||
This experiment tested whether the scope check can recognise intended meaning without rewriting the condition or evidence into preferred test phrases, using the actual long-investigation fixture wording from `scenarios.js`.
|
||
|
||
Two real fixture cases were initially unresolved:
|
||
|
||
1. **Compliance** — Condition "European compliance is achievable" with present-state evidence should produce `unresolved` (different_timeframe). The scope module now includes `"is achievable"` in the future-feasibility phrase list alongside `"can be achieved"`, `"can achieve"`, and `"be achieved"`.
|
||
|
||
2. **Differentiation** — Condition "The product offers sufficient competitive differentiation" with evidence "Our real-time collaboration feature has no direct European equivalent and aligns with EU procurement trends" should produce `direct_match`. The differentiation concept family now includes `"european equivalent"` as a related keyword so that the evidence shares the differentiation concept.
|
||
|
||
#### Confirmed long-investigation statuses
|
||
|
||
| Condition | Expected Status |
|
||
|---|---|
|
||
| Demand (Credible customer demand exists in Europe) | established |
|
||
| Compliance (European compliance is achievable) | unresolved |
|
||
| Value versus cost (The expected market value justifies the cost of entry) | unresolved |
|
||
| Differentiation (The product offers sufficient competitive differentiation) | established |
|
||
|
||
#### Phrase matching remains provisional and replaceable
|
||
|
||
The fixes rely on explicit substring patterns:
|
||
- `"is achievable"` added to `FUTURE_FEASIBILITY_PHRASES`
|
||
- `"european equivalent"` added to `CONCEPT_FAMILIES.differentiation.related`
|
||
|
||
These are narrow, targeted additions. They do not create a broad synonym library or general language parser. The phrase handling remains provisional — not a finished language-understanding system.
|
||
|
||
#### Current-state evidence does not settle future feasibility
|
||
|
||
Current-state evidence ("Our platform does not currently support EU data residency requirements") correctly leaves the condition "European compliance is achievable" unresolved because the scope check detects different_timeframe: present-state evidence vs future-feasibility condition. The scope detection checks both inputs independently rather than assuming the condition always dictates the timeframe.
|
||
|
||
#### Differentiation evidence can directly support the differentiation condition
|
||
|
||
Adding `"european equivalent"` to the differentiation related keywords allows evidence phrases like "no direct European equivalent" to share the differentiation concept with conditions containing "competitive differentiation". This is a narrow phrase match, not a broad semantic equivalence claim.
|
||
|
||
#### Passive Status
|
||
|
||
This experiment remains passive and isolated. It does not modify decision-condition-status.js core rules, evidence-direction.js, graph schema, prompts, APIs, UI, or any active engine behaviour. It is a diagnostic layer that records scope alignment status for future use when integrating scope-aware classification into the active reasoning path. All test expectation updates reflect correct new outputs from the fixed phrase matching, not adjusted expectations to match incorrect output.
|
||
|
||
---
|
||
|
||
### Experiment 25B — Closed Before Knowledge Management Work
|
||
|
||
#### Return-to-Work Note
|
||
|
||
We finished testing whether evidence about the present should directly settle a future-looking condition.
|
||
|
||
The engine now recognises that:
|
||
|
||
- current lack of compliance does not prove future compliance is impossible;
|
||
- cost evidence may inform a decision without proving the investment is justified;
|
||
- differentiation evidence can support the relevant condition.
|
||
|
||
The current language matching is provisional and based on narrow phrases. Do not continue adding synonyms as the long-term solution.
|
||
|
||
Engine experiments are now paused while project knowledge and context-loading are rationalised.
|
||
|
||
Branch: feature/user-workspace-ux-v0.7
|
||
Commit: 273f715
|
||
|
||
---
|
||
|
||
## Current Open Questions
|
||
|
||
The following are active explorations rather than decisions.
|
||
|
||
- What is the right metaphor for the product?
|
||
- Should the workspace resemble a facilitated workshop?
|
||
- How should decomposition be represented?
|
||
- What information belongs in shared understanding?
|
||
- What should the Investigation Map eventually become?
|
||
- How should wide thinking be reflected in the interface?
|
||
|
||
## Backlog — Experiment 05 Persistence Note
|
||
|
||
The "Don't show this introduction again" checkbox uses sessionStorage as a placeholder.
|
||
|
||
This preference should eventually be handled through user preferences or settings rather than local component state.
|
||
|
||
TODO: When user accounts are introduced, persist this preference to the user profile so it travels across devices and sessions.
|
||
|
||
## Future Note — Dark Mode
|
||
|
||
Dark mode is intentionally deferred.
|
||
|
||
Once the information architecture and visual hierarchy stabilise we will investigate whether an "Investigation Mode" (rather than a conventional dark mode) improves concentration.
|
||
|
||
This should be treated as a future UX experiment rather than an accessibility feature.
|
||
|
||
## Backlog — Investigation Narrative (Potential Future Work)
|
||
|
||
These are observations, not implementation tasks.
|
||
|
||
- Narrative adapter
|
||
- Narrative quality heuristics
|
||
- Narrative progression
|
||
- Narrative completion state
|
||
- Narrative confidence wording
|
||
- Narrative testing
|
||
- Narrative localisation
|
||
- Multiple narrative projections
|