1117 lines
40 KiB
Markdown
1117 lines
40 KiB
Markdown
# Design Evolution Log
|
||
|
||
A chronological record of why significant design decisions were made. This is NOT a changelog. It records the product's evolution of thinking.
|
||
|
||
This document records discoveries, not decisions. Every entry represents our best understanding at that point in time and may later be superseded by a better model.
|
||
|
||
---
|
||
|
||
## Phase 1
|
||
|
||
Simple conversational investigation
|
||
|
||
Question → Answer interaction.
|
||
|
||
Purpose:
|
||
Prove the reasoning loop.
|
||
|
||
Learning:
|
||
Conversation alone does not provide sufficient context during longer investigations.
|
||
|
||
---
|
||
|
||
## Phase 2
|
||
|
||
Persistent investigation notebook
|
||
|
||
Added:
|
||
|
||
- current understanding
|
||
- original situation
|
||
- investigation history
|
||
|
||
Learning:
|
||
Users need persistent context rather than remembering previous answers.
|
||
|
||
---
|
||
|
||
## Phase 3
|
||
|
||
Document workspace
|
||
|
||
Created a coherent workspace with:
|
||
|
||
- investigation status
|
||
- current investigation
|
||
- response
|
||
- understanding
|
||
- investigation map placeholder
|
||
- situation
|
||
- history
|
||
|
||
Learning:
|
||
The interface became usable but still behaved like a document rather than a workspace.
|
||
|
||
---
|
||
|
||
## Phase 4 (Current Exploration)
|
||
|
||
Facilitated Investigation Workshop
|
||
|
||
Status:
|
||
Experimental.
|
||
|
||
Hypothesis:
|
||
|
||
The Confidence Engine is not:
|
||
|
||
- a chatbot
|
||
- a dashboard
|
||
- a form
|
||
|
||
It is a facilitated investigation workspace.
|
||
|
||
The interface should resemble the environment in which structured thinking happens.
|
||
|
||
Record discoveries rather than conclusions.
|
||
|
||
Leave room for future phases.
|
||
|
||
---
|
||
|
||
## Phase 4 — Guiding Principles
|
||
|
||
The Confidence Engine is a workspace, not a document.
|
||
|
||
People think in multiple directions simultaneously.
|
||
|
||
Useful context should be visible together.
|
||
|
||
The interface should favour thinking over scrolling.
|
||
|
||
The workspace should feel like a large desk or workshop rather than a narrow report.
|
||
|
||
The engine facilitates thinking.
|
||
|
||
The user contributes evidence.
|
||
|
||
The workspace captures shared understanding.
|
||
|
||
### Experiment 01 — Wider canvas
|
||
|
||
Hypothesis:
|
||
The document-like feeling is caused partly by the narrow outer container.
|
||
|
||
Change:
|
||
Increase the available desktop workspace width without rearranging any components.
|
||
|
||
Result:
|
||
Confirmed.
|
||
|
||
Learning:
|
||
Increasing the outer workspace width reduced the narrow-document feeling and made better use of large displays.
|
||
|
||
Unexpected learning:
|
||
Width alone did not create a workshop. The wider canvas exposed that the interface still behaves as a collection of independent cards, with supporting artefacts unsure how to use the available space.
|
||
|
||
Decision:
|
||
Keep the wider desktop canvas.
|
||
|
||
Next question:
|
||
Can grouping the interface into cognitive work zones make the wider canvas feel like a coherent investigation surface?
|
||
|
||
### Experiment 02 — Cognitive work zones
|
||
|
||
Hypothesis:
|
||
A workspace organised around what the investigator is doing will feel more coherent than one organised around equal cards or equal columns.
|
||
|
||
Result:
|
||
Partially confirmed.
|
||
|
||
Learning:
|
||
|
||
The workspace feels more coherent when organised into cognitive work zones rather than a simple document stack.
|
||
|
||
However, another distinction emerged that is more important than the zones themselves.
|
||
|
||
The interface naturally separates into two different modes:
|
||
|
||
• the active conversation between investigator and facilitator
|
||
|
||
and
|
||
|
||
• the shared workspace describing the current understanding.
|
||
|
||
Unexpected learning:
|
||
|
||
History feels incorrect when treated as reference information.
|
||
|
||
History is actually the continuation of the investigator's conversation.
|
||
|
||
Every response immediately becomes history.
|
||
|
||
The notebook should therefore grow naturally from the Response area.
|
||
|
||
The Investigation Status card currently competes with the Current Investigation card.
|
||
|
||
The current question is the primary focus.
|
||
|
||
Status is supporting context.
|
||
|
||
Decision:
|
||
|
||
Keep the cognitive-zone concept.
|
||
|
||
Refine the zones around conversational flow instead of card grouping.
|
||
|
||
Next question:
|
||
Can the workspace clearly separate conversation from shared understanding?
|
||
|
||
### Experiment 03 — Conversation versus Workspace
|
||
|
||
Hypothesis
|
||
|
||
Investigators think in two simultaneous modes.
|
||
|
||
Mode 1:
|
||
The conversation.
|
||
|
||
Question
|
||
↓
|
||
|
||
Response
|
||
↓
|
||
|
||
History
|
||
|
||
Mode 2:
|
||
The shared workspace.
|
||
|
||
Status
|
||
|
||
Understanding
|
||
|
||
Situation
|
||
|
||
Map
|
||
|
||
Separating these should make the interface feel more like a facilitated investigation than a collection of cards.
|
||
|
||
Evaluation:
|
||
Partially confirmed.
|
||
|
||
Learning:
|
||
|
||
The workspace feels more coherent when organised into cognitive zones rather than a simple document stack.
|
||
|
||
However, another distinction emerged that is more important than the zones themselves.
|
||
|
||
The interface naturally separates into two different modes:
|
||
|
||
• the active conversation between investigator and facilitator
|
||
|
||
and
|
||
|
||
• the shared workspace describing the current understanding.
|
||
|
||
Unexpected learning:
|
||
|
||
History feels incorrect when treated as reference information.
|
||
|
||
History is actually the continuation of the investigator's conversation.
|
||
|
||
Every response immediately becomes history.
|
||
|
||
The notebook should therefore grow naturally from the Response area.
|
||
|
||
The Investigation Status card currently competes with the Current Investigation card.
|
||
|
||
The current question is the primary focus.
|
||
|
||
Status is supporting context.
|
||
|
||
Decision:
|
||
|
||
Keep the cognitive-zone concept.
|
||
|
||
Refine the zones around conversational flow instead of card grouping.
|
||
|
||
Next question:
|
||
Can the workspace clearly separate conversation from shared understanding?
|
||
|
||
### Experiment 04 — Facilitated Workshop Introduction
|
||
|
||
Hypothesis
|
||
|
||
Beginning with a facilitator-style introduction will create more confidence than presenting an empty workspace.
|
||
|
||
Questions
|
||
|
||
- Does the interface feel more welcoming?
|
||
- Does reducing the visual weight of the textarea improve the first experience?
|
||
- Does separating "starting" from "investigating" feel natural?
|
||
- Does the transition into the investigation workspace feel meaningful?
|
||
|
||
Status:
|
||
Experimental.
|
||
|
||
Result:
|
||
Partially confirmed.
|
||
|
||
Learning:
|
||
|
||
The facilitator introduction reduced the intimidation of the first screen.
|
||
|
||
Replacing the empty landing page with a guided introduction improved the emotional tone.
|
||
|
||
However, stacking the introduction above the input still gives the introduction excessive visual prominence.
|
||
|
||
Repeat users may not want to repeatedly read the same introduction.
|
||
|
||
Orientation should remain available without dominating the workflow.
|
||
|
||
Decision:
|
||
|
||
Keep the introduction concept but change its spatial relationship to the workspace — move it from above to beside, making it optional rather than mandatory.
|
||
|
||
Next question:
|
||
Does a horizontal facilitator/workspace layout feel more natural?
|
||
|
||
### Experiment 05 — Facilitator Panel and Adaptive Landing Workspace
|
||
|
||
Hypothesis
|
||
|
||
Placing the facilitator beside the working area will feel more like entering a facilitated workshop than stacking instructional content above the workspace.
|
||
|
||
Allowing the user to dismiss the facilitator will reduce friction for returning users while preserving onboarding for new users.
|
||
|
||
Questions
|
||
|
||
- Does a horizontal facilitator/workspace layout feel more natural?
|
||
- Does the user's eye move naturally from facilitator to workspace?
|
||
- Does the workspace become the primary focus?
|
||
- Does "Don't show again" feel preferable to automatically hiding the introduction?
|
||
- Should the facilitator panel become an optional workspace companion rather than mandatory onboarding?
|
||
|
||
Status:
|
||
Completed.
|
||
|
||
Findings:
|
||
|
||
- A horizontal facilitator/workspace arrangement feels more natural than stacked onboarding.
|
||
- The workspace becomes the visual destination rather than the introduction.
|
||
- User-controlled dismissal is preferable to automatic hiding.
|
||
- The facilitator feels useful but visually too passive.
|
||
- Remaining issues are now visual hierarchy rather than layout architecture.
|
||
|
||
### Experiment 06 — Focused Investigation
|
||
|
||
Hypothesis
|
||
|
||
The interface should gently guide attention towards the current task without hiding supporting information.
|
||
|
||
Reducing competition between panels may improve concentration more than introducing additional colour or decoration.
|
||
|
||
Questions
|
||
|
||
- Does visual emphasis naturally guide the eye?
|
||
- Can supporting panels become quieter without disappearing?
|
||
- Does the investigation question become the obvious focal point?
|
||
- Does the workspace feel calmer?
|
||
- Are we approaching a professional investigation environment?
|
||
|
||
Status:
|
||
Closed.
|
||
|
||
## Result
|
||
|
||
Partially confirmed.
|
||
|
||
## What did we learn?
|
||
|
||
- Stronger visual hierarchy can direct attention without rearranging the interface.
|
||
- The facilitator briefing became easier to distinguish.
|
||
- Colour and tint improved separation only modestly.
|
||
- Meaning must not depend on colour.
|
||
- Areas and intent should remain distinguishable through structure, spacing, typography, borders, shape and placement.
|
||
- The initial textarea still implies that the user should provide a detailed report.
|
||
- The size of an input communicates the amount of information expected.
|
||
|
||
## Decision
|
||
|
||
Retain the useful hierarchy refinements provisionally.
|
||
|
||
Do not increase reliance on colour.
|
||
|
||
Defer dark mode and broader palette work.
|
||
|
||
The next experiment should test whether a smaller starting input better communicates that the user only needs to provide an initial observation.
|
||
|
||
Do not rewrite previous experiments.
|
||
|
||
---
|
||
|
||
### Experiment 07 — Lightweight Starting Observation
|
||
|
||
#### Hypothesis
|
||
|
||
A smaller initial input will make beginning an investigation feel easier and will communicate that the engine needs only a concise observation rather than a complete analysis.
|
||
|
||
#### Questions
|
||
|
||
- Does the input feel like a conversation starter rather than a report form?
|
||
- Is three to four visible lines sufficient?
|
||
- Does the facilitator panel and input area feel better balanced?
|
||
- Does the user understand that further detail will be gathered through questions?
|
||
- Does reducing the input height make the Analyse action easier to notice?
|
||
|
||
#### Evaluation
|
||
|
||
Pending visual review.
|
||
|
||
#### Result
|
||
|
||
Confirmed.
|
||
|
||
Four visible rows better communicates a starting observation than six.
|
||
|
||
Input size communicates expected effort.
|
||
|
||
"What have you noticed?" reinforces observational thinking.
|
||
|
||
Users are encouraged to begin rather than compose.
|
||
|
||
The facilitator and workspace now feel more balanced.
|
||
|
||
This interaction principle should continue throughout the investigation rather than existing only on the landing page.
|
||
|
||
#### Decision
|
||
|
||
Retain the smaller landing input.
|
||
|
||
Proceed to investigate consistency between the landing experience and investigation responses.
|
||
|
||
---
|
||
|
||
### Experiment 09 — Investigation Rhythm
|
||
|
||
#### Result
|
||
|
||
Partially confirmed.
|
||
|
||
#### What did we learn?
|
||
|
||
- Moving History directly beneath Response improves the sense of conversational continuity.
|
||
- The sequence Question → Response → History is cognitively coherent.
|
||
- History behaves like the growing notebook of the investigation, not general reference material.
|
||
- Allowing History to span the full workspace breaks the wider spatial model.
|
||
- Situation and Investigation Map should remain stable supporting artefacts rather than moving down as the notebook grows.
|
||
- The conversation needs a dedicated vertical lane.
|
||
|
||
#### Decision
|
||
|
||
Keep History directly connected to Response.
|
||
|
||
Refine the desktop workspace into a stable conversation lane and a stable supporting lane.
|
||
|
||
Do not rewrite previous experiments.
|
||
|
||
---
|
||
|
||
### Experiment 08 — Consistent Investigation Responses
|
||
|
||
#### Hypothesis
|
||
|
||
Every answer given during an investigation should feel like an observation, not a report.
|
||
|
||
The response component should therefore communicate the same expected effort as the initial scenario input.
|
||
|
||
#### Questions
|
||
|
||
- Does a smaller response area reduce perceived effort?
|
||
- Does the investigation feel more conversational?
|
||
- Does consistency improve confidence?
|
||
- Does the workspace become visually calmer?
|
||
- Does the current investigation remain the dominant focus?
|
||
|
||
#### Result
|
||
|
||
Confirmed.
|
||
|
||
Consistent interaction patterns reduce cognitive effort.
|
||
|
||
Users should not have to learn different behaviours between the landing page and investigation.
|
||
|
||
Smaller response areas reinforce concise observations.
|
||
|
||
The engine appears more conversational when each answer feels lightweight.
|
||
|
||
Consistency is becoming a stronger design tool than decoration.
|
||
|
||
#### Decision
|
||
|
||
Retain consistent input sizing across both contexts.
|
||
|
||
---
|
||
|
||
### Experiment 10 — Stable Conversation Column
|
||
|
||
#### Hypothesis
|
||
|
||
A persistent two-thirds conversation column beside a one-third supporting column will allow the investigation notebook to grow without moving the shared reference artefacts.
|
||
|
||
#### Questions
|
||
|
||
- Does the left column feel like one continuous investigation?
|
||
- Does History grow naturally beneath Response?
|
||
- Do Situation and Investigation Map remain easy to reference?
|
||
- Does the interface feel spatially stable as turns accumulate?
|
||
- Does showing full question text improve readability now that sufficient width exists?
|
||
|
||
#### Evaluation
|
||
|
||
Visual review completed.
|
||
|
||
#### Status
|
||
|
||
Closed.
|
||
|
||
## Result
|
||
|
||
Partially confirmed.
|
||
|
||
## What did we learn?
|
||
|
||
- The investigation workspace is beginning to feel like a genuine facilitated investigation rather than a document.
|
||
- The two-column workspace (conversation on the left, reference material on the right) is proving to be a stronger mental model than previous layouts.
|
||
- Keeping Situation and Investigation Map fixed while History grows vertically feels more natural.
|
||
- The investigation question, response and history now read as one continuous conversation.
|
||
- Developer Details have become extremely valuable.
|
||
- The graph produced by the reasoning engine is far richer than previously realised. The graph now contains structured concepts including:
|
||
|
||
- observations
|
||
- unknowns
|
||
- assumptions
|
||
- relationships
|
||
- metrics
|
||
- state
|
||
|
||
This suggests the UI should increasingly become a human-friendly projection of the graph rather than inventing separate state.
|
||
|
||
The current "Investigation in progress" panel exposes developer-oriented statistics (nodes, edges, unknowns etc.) which are useful during development but are not the most helpful representation for an end user.
|
||
|
||
---
|
||
|
||
## Emerging Direction — Graph as Source of Truth
|
||
|
||
The reasoning graph is becoming the shared source of truth for multiple UI views.
|
||
|
||
Different interfaces may project the same graph for different audiences:
|
||
|
||
- Version A — compact technical progress;
|
||
- Version B — detailed graph inspection;
|
||
- Version C — user-facing facilitator view;
|
||
- Developer Details — complete diagnostics;
|
||
- Investigation Map — future spatial projection;
|
||
- Current Question — active uncertainty projection.
|
||
|
||
The UI should not maintain separate invented summaries where the graph already contains the underlying information.
|
||
|
||
This is an emerging direction, not a final architecture decision.
|
||
|
||
---
|
||
|
||
## Emerging Direction — Facilitator Translation Layer
|
||
|
||
> The UI should progressively become a translation layer over the reasoning graph rather than maintaining separate duplicated summaries. Internal graph concepts should remain available for developers, while end users see a facilitator-style explanation of what is currently understood and what remains uncertain.
|
||
|
||
The current technical progress panel (nodes, edges, unknowns, assumptions) exposes developer-oriented statistics. These are valuable during development but not the most helpful representation for an end user.
|
||
|
||
The next direction is to explore presenting the same underlying graph data as a facilitator's notebook — what is known, what remains uncertain, and a quiet summary of the reasoning state underneath.
|
||
|
||
---
|
||
|
||
### Experiment 11 — Facilitator Progress Panel (Version B)
|
||
|
||
#### Hypothesis
|
||
|
||
The same underlying reasoning graph can be presented in a much more human-friendly way without changing the reasoning engine, API contracts, or graph generation.
|
||
|
||
A facilitator-style panel should communicate:
|
||
|
||
- what is known (resolved nodes and observations)
|
||
- what remains uncertain (unresolved unknowns and assumptions)
|
||
- a quiet summary of the reasoning state underneath
|
||
|
||
#### Questions
|
||
|
||
- Can the same graph data be translated into a facilitator-style view that end users understand more naturally?
|
||
- Does separating "known" from "still investigating" reduce cognitive load compared to node/edge counts?
|
||
- Is a quiet reasoning summary sufficient, or does it need more context?
|
||
- Does the translation-layer principle hold — presenting the graph as a notebook rather than raw data?
|
||
|
||
#### Result
|
||
|
||
Partially confirmed.
|
||
|
||
#### What did we learn?
|
||
|
||
- Version B proved that the reasoning graph contains substantially more useful information than Version A exposes.
|
||
- The graph already contains observations, unknowns, assumptions, metrics, relationships and state.
|
||
- The graph is rich enough to support multiple UI projections.
|
||
- Exposing the graph almost verbatim overwhelms the user.
|
||
- Technical categories are useful for development but do not directly communicate investigation progress.
|
||
- The user needs a translation of the graph rather than a graph browser.
|
||
- Developer Details should remain the place for complete technical inspection.
|
||
- A user-facing view needs filtering, prioritisation, deduplication and clear epistemic labels.
|
||
|
||
#### Decision
|
||
|
||
Keep Version A and Version B available for comparison.
|
||
|
||
Proceed with a Version C facilitator view built from the same graph.
|
||
|
||
---
|
||
|
||
### Experiment 12 — Facilitator View (Version C)
|
||
|
||
#### Hypothesis
|
||
|
||
The existing reasoning graph can be deterministically translated into a concise facilitator view that helps the user understand:
|
||
|
||
- what is currently known;
|
||
- what remains uncertain;
|
||
- what may explain the situation;
|
||
- why the investigation is continuing.
|
||
|
||
#### Questions
|
||
|
||
- Can the graph produce a useful human-facing summary without another LLM call?
|
||
- Can observations, unknowns and assumptions be clearly distinguished?
|
||
- Can duplicate or low-value graph content be filtered reliably?
|
||
- Does a concise projection improve understanding without exposing implementation detail?
|
||
- Does the panel remain useful across mocks and live Ollama output?
|
||
- Can the same view work during early, middle and terminal investigation states?
|
||
|
||
#### Evaluation
|
||
|
||
Completed. Visual and live-data review performed.
|
||
|
||
#### Result
|
||
|
||
Confirmed.
|
||
|
||
#### What did we learn?
|
||
|
||
- The reasoning graph already contains all the information needed for a useful human-facing summary — no additional LLM calls are required.
|
||
- Routing by semantic role (observation, question, explanation) rather than graph kind produces a more natural user experience.
|
||
- Filtering scaffolding content (scenario summaries, system/tool references, metric object descriptions, process labels) is essential to keep the view focused on findings.
|
||
- Deduplication of near-duplicate observations reduces noise without losing information.
|
||
- Epistemic clarity matters — resolved unknowns become factual observations and should be classified as known rather than still-under-investigation.
|
||
- The panel works across all investigation phases (early, active, terminal).
|
||
|
||
#### Decision
|
||
|
||
Close Experiment 12 as confirmed. Proceed to refine the translation through semantic classification in the next iteration.
|
||
|
||
---
|
||
|
||
### Experiment 13 — Semantic Facilitator Translation
|
||
|
||
#### Hypothesis
|
||
|
||
Improving the deterministic projection from graph semantics to user-facing language — by classifying nodes by *meaning* rather than *graph kind*, suppressing scaffolding, merging duplicates, and preferring concrete observations — produces a significantly better facilitator view without changing the reasoning engine, prompts, graph generation, or any external contracts.
|
||
|
||
#### Questions
|
||
|
||
- Does semantic role classification (observation vs question vs explanation) route content more naturally than graph-kind classification?
|
||
- Does scaffolding suppression remove visual noise that previously dominated derived summaries?
|
||
- Does deduplication reduce redundant items that express the same observation under slightly different wording?
|
||
- Do concrete observations appear before abstract labels in ranked output?
|
||
- Does the view remain robust when consumed by the existing panel component (investigation-summary-panel-v3) without any changes to that component?
|
||
|
||
#### Evaluation
|
||
|
||
Completed. Tests: 37 scenarios passing across filtering, classification, deduplication, ranking, section framing, mock-data integration, and edge cases.
|
||
|
||
#### Result
|
||
|
||
Confirmed.
|
||
|
||
#### What did we learn?
|
||
|
||
- Semantic role routing outperforms kind-based routing: a node with `kind: "state"` that contains concrete data (e.g., "Revenue increased 12%") is more useful as an observation than a state description.
|
||
- Scaffolding suppression works best when applied early — filtering at the semantic classification stage prevents structural glue from contaminating any section.
|
||
- Three-tier filtering is effective: scaffolding patterns (highest priority), internal vocabulary (medium), then technical summary patterns (lowest).
|
||
- Deduplication by normalised text removes meaningful noise. When "Revenue increased 12%" and "Current revenue is 12% higher" express the same observation, keeping one reduces confusion without losing information.
|
||
- Resolved unknowns and assumptions are factual answers to previously unanswered questions — they should appear in the known section with an epistemic label ("Not yet established" / "To be tested") if their status hasn't been explicitly set.
|
||
- The translation adapter is the right place for this work: it is a single deterministic function, testable in isolation, and its output contracts are stable.
|
||
|
||
#### Result
|
||
|
||
Confirmed.
|
||
|
||
#### What did we learn?
|
||
|
||
- Semantic filtering significantly improved Version C.
|
||
- The remaining limitations are architectural rather than visual.
|
||
- Graph nodes still do not naturally map to facilitator language.
|
||
- Users think in investigation progress rather than graph structure.
|
||
- Version C proved the need for an intermediate narrative model.
|
||
|
||
#### Decision
|
||
|
||
Keep the semantic projection approach.
|
||
|
||
Do not continue improving graph projection indefinitely.
|
||
|
||
Proceed to designing an Investigation Narrative layer. Experiment 13 is closed.
|
||
|
||
---
|
||
|
||
### Experiment 14 — Investigation Narrative Layer
|
||
|
||
#### Hypothesis
|
||
|
||
The graph should remain the internal reasoning model.
|
||
|
||
A separate narrative model should become the presentation model.
|
||
|
||
The facilitator UI should consume narrative state rather than graph nodes.
|
||
|
||
#### Questions
|
||
|
||
- What information belongs in a narrative?
|
||
- What belongs only in the graph?
|
||
- Which narrative elements can be derived deterministically?
|
||
- What should remain hidden?
|
||
- Can every facilitator panel consume the same narrative object?
|
||
|
||
#### Status
|
||
|
||
Architectural experiment.
|
||
|
||
#### Evaluation
|
||
|
||
Pending.
|
||
|
||
---
|
||
|
||
## Emerging Direction — Investigation Narrative
|
||
|
||
The Confidence Engine architecture is becoming:
|
||
|
||
User
|
||
|
||
↓
|
||
|
||
Facilitated Conversation
|
||
|
||
↓
|
||
|
||
Reasoning Graph
|
||
|
||
↓
|
||
|
||
Investigation Narrative
|
||
|
||
↓
|
||
|
||
Workspace Projection
|
||
|
||
↓
|
||
|
||
User
|
||
|
||
The reasoning graph becomes the machine representation.
|
||
|
||
The investigation narrative becomes the human representation.
|
||
|
||
The UI simply renders whichever projection is appropriate.
|
||
|
||
This is an emerging architectural direction.
|
||
|
||
It is intentionally recorded before implementation so future experiments remain aligned.
|
||
|
||
---
|
||
|
||
### Experiment 15 — Facilitator Behaviour Specification
|
||
|
||
#### Hypothesis
|
||
|
||
An expert consultant does not have a script. They have behaviours — recurring patterns of action deployed based on what they observe in the client's situation. The Confidence Engine should exhibit similar behavioural patterns rather than following a mechanical question-fill-graph cycle.
|
||
|
||
The current engine behaviour is:
|
||
|
||
> Engine asks → User answers → Graph updates → Engine asks again
|
||
|
||
An expert facilitator behaviour is:
|
||
|
||
> Engine assesses state → selects appropriate behaviour → acts (question, acknowledge, synthesise, challenge, pause)
|
||
|
||
#### Questions
|
||
|
||
- How does an expert consultant behave during an investigation?
|
||
- Which behaviours recur across investigations?
|
||
- What triggers each behaviour?
|
||
- When does the facilitator ask a question versus summarise versus expose uncertainty versus hold space?
|
||
- What distinguishes guided thinking from mechanical Q&A?
|
||
|
||
#### Status
|
||
|
||
Investigation — behavioural model documented, not yet implemented.
|
||
|
||
#### Evaluation
|
||
|
||
This experiment is primarily architectural and behavioural. No code changes are required at this stage. The deliverable is a behavioural specification that future implementation experiments will reference.
|
||
|
||
#### Result
|
||
|
||
Confirmed as the correct next direction.
|
||
|
||
#### What did we learn?
|
||
|
||
- Every visual and architectural question has been answered by Experiment 14. Further visual iteration yields diminishing returns.
|
||
- The remaining gap is not visual — it is behavioural.
|
||
- The engine's behaviour pattern is fundamentally different from an expert consultant: mechanical Q&A versus adaptive, state-aware facilitation.
|
||
- The graph captures *state* but not *behaviour*. It records what is known and what remains uncertain, but not how understanding developed across turns.
|
||
- Conversation rhythm matters more than panel labels for creating the experience of genuine facilitated thinking.
|
||
- 14 distinct facilitator behaviours were identified: Orient, Acknowledge, Observe pattern, Clarify, Validate, Connect, Challenge assumption, Refine understanding, Expose uncertainty, Decide direction, Know when to pause, Avoid premature closure, Communicate confidence honestly, Progressively narrow focus.
|
||
- Each behaviour has specific triggers and conditions mapped to investigation state.
|
||
- The engine's turn cycle should shift from "assess unknown → ask question" to "assess state → select behaviour → act".
|
||
|
||
#### Decision
|
||
|
||
Commit the behavioural specification. Do not implement yet. Future experiments will integrate behavioural assessment into the reasoning cycle. This document defines what the facilitator does; future work determines how the system implements it.
|
||
|
||
**Status: Closed.** The behavioural model is established and documented. The gap it identified — that behaviours need a decision process operating on investigation state rather than graph structure — becomes the focus of Experiment 16.
|
||
|
||
---
|
||
|
||
### Experiment 16 — Investigation State Assessment
|
||
|
||
#### Hypothesis
|
||
|
||
The facilitator should never inspect the graph directly when deciding what to do next.
|
||
|
||
Instead it should act upon an assessment of the investigation — its phase, progress, evidence quality, understanding trajectory, uncertainty trend, conversation health, and behaviour readiness.
|
||
|
||
This assessment is distinct from both:
|
||
|
||
- The reasoning graph (which captures *what* is known)
|
||
- The investigation narrative (which translates *what is known* into human language)
|
||
|
||
The assessment answers: *Given where we are, what kind of help is most appropriate right now?*
|
||
|
||
No reasoning changes.
|
||
|
||
No prompt changes.
|
||
|
||
No UI changes.
|
||
|
||
This is an architectural experiment.
|
||
|
||
#### Status
|
||
|
||
Architectural.
|
||
|
||
#### Evaluation
|
||
|
||
Confirmed.
|
||
|
||
---
|
||
|
||
## What did we learn?
|
||
|
||
Document observations such as:
|
||
|
||
- Investigation state is distinct from behaviour.
|
||
- Behaviour should consume assessment rather than graph structure.
|
||
- State assessment provides a stable contract between reasoning and facilitation.
|
||
- The architecture is becoming layered rather than procedural.
|
||
|
||
Decision:
|
||
|
||
Proceed to documenting the investigation turn cycle.
|
||
|
||
---
|
||
|
||
### Experiment 16 — Emerging Architecture Observation
|
||
|
||
The Confidence Engine architecture is becoming:
|
||
|
||
User
|
||
|
||
↓
|
||
|
||
Facilitated Conversation (where behaviour lives)
|
||
|
||
↓
|
||
|
||
Behaviour Selection (consumes assessment output)
|
||
|
||
↓
|
||
|
||
Investigation State Assessment (describes investigation)
|
||
|
||
↓
|
||
|
||
Investigation Narrative (human representation of state)
|
||
|
||
↓
|
||
|
||
Reasoning Graph (machine representation)
|
||
|
||
↓
|
||
|
||
LLM / Ollama / Reasoning Engine
|
||
|
||
↓
|
||
|
||
User
|
||
|
||
This is not a final design. It is an observation emerging from 16 experiments.
|
||
|
||
What is becoming clear:
|
||
|
||
- The reasoning graph is the machine representation.
|
||
- The investigation narrative is the human representation.
|
||
- The investigation state assessment is the decision representation — it translates state into readiness signals for behaviour selection.
|
||
- Behaviour selection determines what kind of help to deploy.
|
||
- Facilitated Conversation is where that help is delivered.
|
||
|
||
Each layer has a single responsibility. Each feeds the next. No layer inspects another's implementation details.
|
||
|
||
This architecture emerged from observation, not top-down design. It may still change as future experiments test it.
|
||
|
||
---
|
||
|
||
### Experiment 17 — Investigation Turn Cycle
|
||
|
||
#### Hypothesis
|
||
|
||
A complete investigation can be described as a repeating turn cycle in which every architectural layer has a single responsibility.
|
||
|
||
Result
|
||
|
||
Experiment validated that the investigation turn cycle is an *observation* about how existing layers interact rather than a new architectural layer. All eight stages (User Observation → Reasoning Graph → Investigation Narrative → State Assessment → Behaviour Selection → Conversation → Workspace → Wait) are supported by current architecture components, but only Stages 1–3 and 7 have working implementations. Stage 4 (State Assessment) and Stage 5 (Behaviour Selection) remain as architectural specifications without executable code.
|
||
|
||
What did we learn?
|
||
|
||
- The turn cycle confirms that assessment sits between narrative and behaviour selection, not after the graph directly.
|
||
- Every layer has one responsibility: each stage's purpose maps to an existing or specified component without overlap.
|
||
- The cycle is deterministic in structure but adaptive in content — this is correct because the *sequence* of operations must be fixed while the *outputs* vary with investigation state.
|
||
- Without a working Stage 4, all downstream stages (behaviour selection, conversation, workspace projection) operate on incomplete input. Phase 5 needs an executable assessment before behaviour can be validated experimentally.
|
||
|
||
Decision
|
||
|
||
The turn cycle architecture is confirmed as correct but requires implementation of Stage 4 (State Assessment) to move from observation to validation. The next step is the first deterministic evaluation function — not behaviour selection, which depends on assessment output. This becomes Experiment 18: First Executable Slice.
|
||
|
||
---
|
||
|
||
### Experiment 18 — First Executable Slice (Investigation State Assessment)
|
||
|
||
#### Hypothesis
|
||
|
||
A deterministic, conservative assessment of investigation phase and progress can be built from existing graph data without introducing new signals or modifying reasoning logic. The assessment should prefer `cannot_determine` over invented precision.
|
||
|
||
#### Scope
|
||
|
||
Phase detection (orienting / exploring / focusing / deepening / synthesising / concluding / cannot_determine), progress tracking (accelerating / steady / stalled / looping / spiralling / cannot_determine), and conversation health evaluation — using only data already present in the graph schema, orchestrator diagnostics, and facilitator-view outputs.
|
||
|
||
#### Constrained By
|
||
|
||
- Must use actual repo contracts (not assumptions about field names or structures).
|
||
- Must be pure function — no network, LLM, mutation, or side effects.
|
||
- Must handle missing fields gracefully — safe with absent data.
|
||
- Must produce versioned assessment objects for future compatibility.
|
||
- Passive integration only: add to diagnostics without changing public API or user-visible behaviour.
|
||
|
||
#### Questions
|
||
|
||
1. Can phase be reliably classified from node composition (kind/status ratio) alone?
|
||
2. Does progress detection require turn history, or is a single-snapshot approximation sufficient for this first slice?
|
||
3. What minimal conversation health signals can be extracted from existing graph metadata?
|
||
|
||
#### Evaluation
|
||
|
||
- Deterministic output across identical inputs.
|
||
- Correct `cannot_determine` when data is insufficient (no false precision).
|
||
- Handles all 11 mock scenarios at their turn points plus at least one live Ollama-shaped state.
|
||
- Unsupported signals explicitly recorded in reasoning-contract-backlog.md.
|
||
|
||
#### Status
|
||
|
||
**Closed.** The assessment is implemented, tested, and validated. See `investigation-state-assessment-contract.md` and `lib/assessment/investigation-state-assessor.js`.
|
||
|
||
#### Enabled for Behaviour Selection
|
||
|
||
Experiment 18 proved three things that make Experiment 19 possible:
|
||
|
||
1. **Phase detection works.** We can classify investigation phase (orienting / exploring / focusing / deepening / synthesising / concluding) from existing graph data with measurable confidence. This is the primary input for behaviour selection — without it, selection rules have no state to operate on.
|
||
|
||
2. **Progress tracking works.** Stalled progress in a focusing phase becomes a concrete signal that the facilitator should hold space rather than push. Previously this was an architectural idea; now it's observable data.
|
||
|
||
3. **Conversation health is measurable.** Healthy, too_broad, and user_overloaded states are detectable from question distribution and response patterns. `too_broad` triggers Clarify; healthy with resolution triggers Acknowledge — but only if the assessment layer exists to provide these signals.
|
||
|
||
Without Experiment 18, Behaviour Selection would have two options: inspect the graph directly (coupling behaviour to implementation) or use narrative fields as proxy signals (fragile by design). The assessment layer provides a stable contract — the three reliable dimensions listed above — that behaviour selection can depend on without fear of breaking when the graph schema changes.
|
||
|
||
Experiment 18 also proved that `cannot_determine` is not a failure mode but the correct answer when evidence is insufficient. This principle carries directly into behaviour selection: "no explicit rule matched" defaults to continue, not an invented signal.
|
||
|
||
---
|
||
|
||
### Experiment 19 — Passive Behaviour Selection
|
||
|
||
#### Hypothesis
|
||
|
||
Does selecting from a small set of five behaviours (Acknowledge, Clarify, Summarise, Continue, Pause) — instead of always asking — make the investigation feel more like guided thinking and less like automated Q&A?
|
||
|
||
This is one question. Nothing else matters until this is answered.
|
||
|
||
#### Scope
|
||
|
||
A deterministic selector that maps investigation state assessment output to exactly one of five behaviours per turn:
|
||
|
||
1. **Acknowledge** — when conversation health is healthy AND phase confidence is not low
|
||
2. **Clarify** — when health is `too_broad` OR (phase is orienting AND observations < 3)
|
||
3. **Summarise** — when phase is synthesising/concluding OR (≥ 3 resolved with steady progress)
|
||
4. **Pause** — when phase is focusing AND progress is stalled; also user_overloaded health
|
||
5. **Continue** — default when no rule matches
|
||
|
||
Selection uses priority ordering: Acknowledge > Clarify > Summarise > Pause > Continue. No scoring, no weighting, no convergence thresholds. First matching rule wins.
|
||
|
||
The selector is passive — deployed only through Developer Details diagnostics. No changes to reasoning engine, prompts, graph generation, decomposition, narrative generation, API contracts, UI behaviour, or Ollama integration.
|
||
|
||
#### Evaluation Criteria
|
||
|
||
1. **Behaviour diversity:** Does the system deploy at least 3 different behaviours across a normal investigation, or does it default to Continue most of the time?
|
||
2. **Acknowledge appears:** Does Acknowledge fire whenever new information resolves an uncertainty? If not, the trigger condition is wrong — fix it, don't abandon selection.
|
||
3. **Pause feels like relief, not delay:** When Pause fires, does the user experience it as a natural break rather than a system failure to produce a question?
|
||
4. **Summarise compresses meaningfully:** Does the summarised understanding feel useful or redundant?
|
||
5. **Conversation rhythm changes:** Is there a perceptible difference between "engine always asking" and "engine sometimes acknowledging/summarising/pausing first"?
|
||
|
||
If none of these can be evaluated after 2–3 real investigations with v0.1, the experiment was too small to answer the question.
|
||
|
||
#### Open Questions
|
||
|
||
- Which of the five behaviours fires most frequently in practice?
|
||
- Does Acknowledge actually appear during investigations that would normally produce continuous questioning?
|
||
- Does the priority ordering create appropriate urgency (Acknowledge > Clarify > Summarise > Pause > Continue)?
|
||
- Are there cases where `cannot_determine` produces inappropriate behaviour selection — or is this the correct conservative default?
|
||
|
||
---
|
||
|
||
### Experiment 20 — Passive Question Importance Classification
|
||
|
||
#### Hypothesis
|
||
|
||
Does a passive classifier that tags unresolved unknowns as `important`, `helpful`, `incidental`, or `cannot_determine` (using only existing graph fields, no scoring, no weights) produce coherent importance patterns across normal investigations?
|
||
|
||
This is one question. Nothing else matters until this is answered.
|
||
|
||
#### Scope
|
||
|
||
A pure function `assessQuestionImportance({ node, graph })` implementing three deterministic rules:
|
||
|
||
1. **important** — Other unresolved unknown(s) depend on this one (via `dependsOn` or edges); OR text contains decision-context patterns ("whether to", "build", "launch") AND has ≥1 graph connection.
|
||
2. **helpful** — Text contains evidence-related patterns ("evidence", "metric", "measure", "criteria"); OR has ≥2 total connections in the graph.
|
||
3. **incidental** — Default when neither important nor helpful conditions are met.
|
||
4. **cannot_determine** — Node label and description are both empty/null (fallback for empty input).
|
||
|
||
The classifier is passive — validated only against mock scenario fixtures. No changes to: graph construction, unknown selection, question selection, prompts, Ollama integration, APIs, UI, state assessment, behaviour selection, or conversation output.
|
||
|
||
#### Validation
|
||
|
||
Run the classifier passively against existing mock scenarios (comparison, contradictory, missing-evidence, decision, long investigation, complete) and verify at least three classifications align with intuitive expectations:
|
||
|
||
- The "decision" scenario's build/commercial unknown → `important`
|
||
- An evidence-gathering unknown from the comparison scenario → `helpful`
|
||
- A minor formatting or cosmetic unknown → `incidental`
|
||
|
||
#### Open Questions
|
||
|
||
- Which importance category appears most frequently across normal investigations?
|
||
- Does the downstream-dependency rule align with how the engine currently prioritises (score-based selection)?
|
||
- Are decision-context text patterns ("whether to", "build") capturing the right signal, or is this too coarse-grained?
|
||
- Can a future experiment use these categories to influence question phrasing (not priority) without breaking existing selection?
|
||
|
||
---
|
||
|
||
## Phase Transition
|
||
|
||
Record that the project has moved from:
|
||
|
||
Interface Design
|
||
→
|
||
Facilitated Investigation
|
||
→
|
||
Behavioural Architecture
|
||
→
|
||
System Architecture
|
||
|
||
Future work should validate these layers rather than introduce new ones.
|
||
|
||
---
|
||
|
||
## Emerging Direction — Graph as Source of Truth
|
||
|
||
The first UX experiments focused on workspace structure.
|
||
|
||
The next series will focus on investigation rhythm and behaviour.
|
||
|
||
Future experiments should explore:
|
||
|
||
- how conversations unfold (behavioural, not visual)
|
||
- how understanding evolves across turns
|
||
- how the facilitator selects its behavioural response
|
||
- how confidence is gradually built through action, not description
|
||
- what state assessment enables better question selection
|
||
|
||
The objective is no longer to arrange cards or translate panels.
|
||
|
||
The objective is to make each turn of the investigation feel like a natural step in a guided thinking process.
|
||
|
||
The objective is to make the investigation feel like a natural facilitated conversation.
|
||
|
||
---
|
||
|
||
## Current Open Questions
|
||
|
||
The following are active explorations rather than decisions.
|
||
|
||
- What is the right metaphor for the product?
|
||
- Should the workspace resemble a facilitated workshop?
|
||
- How should decomposition be represented?
|
||
- What information belongs in shared understanding?
|
||
- What should the Investigation Map eventually become?
|
||
- How should wide thinking be reflected in the interface?
|
||
|
||
## Backlog — Experiment 05 Persistence Note
|
||
|
||
The "Don't show this introduction again" checkbox uses sessionStorage as a placeholder.
|
||
|
||
This preference should eventually be handled through user preferences or settings rather than local component state.
|
||
|
||
TODO: When user accounts are introduced, persist this preference to the user profile so it travels across devices and sessions.
|
||
|
||
## Future Note — Dark Mode
|
||
|
||
Dark mode is intentionally deferred.
|
||
|
||
Once the information architecture and visual hierarchy stabilise we will investigate whether an "Investigation Mode" (rather than a conventional dark mode) improves concentration.
|
||
|
||
This should be treated as a future UX experiment rather than an accessibility feature.
|
||
|
||
## Backlog — Investigation Narrative (Potential Future Work)
|
||
|
||
These are observations, not implementation tasks.
|
||
|
||
- Narrative adapter
|
||
- Narrative quality heuristics
|
||
- Narrative progression
|
||
- Narrative completion state
|
||
- Narrative confidence wording
|
||
- Narrative testing
|
||
- Narrative localisation
|
||
- Multiple narrative projections
|