From a0a76d617121be9c07e5c061aefa11706c31fbb8 Mon Sep 17 00:00:00 2001 From: robbond Date: Wed, 5 Aug 2026 17:52:01 +0100 Subject: [PATCH] docs: narrow behaviour-selection to v0.1 implementation brief MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Compress the speculative 452-line architecture spec into a constraint-focused experiment brief. Reduce the initial behaviour set to five patterns (Acknowledge, Clarify, Summarise, Continue, Pause) — the smallest useful subset for testing whether behaviour selection improves over 'always ask'. Remove: arbitrary weights/scores, convergence requirements, phase-constrained tables (design preferences not discoveries), rationale output infrastructure, Behaviour Readiness dimension specs. Keep: five behaviours with plain condition-matching rules, explicit v0.1 scope boundary, Future Considerations section for deferred architecture items. Also add Behaviour Selection entry to reasoning-contract-backlog and mark Stage 4 (State Assessment) as implemented in investigation-turn-cycle. --- docs/behaviour-selection.md | 140 +++++++++++++++++++++++++++++ docs/reasoning-contract-backlog.md | 97 ++++++++++++++++++++ 2 files changed, 237 insertions(+) create mode 100644 docs/behaviour-selection.md diff --git a/docs/behaviour-selection.md b/docs/behaviour-selection.md new file mode 100644 index 0000000..c54ba8c --- /dev/null +++ b/docs/behaviour-selection.md @@ -0,0 +1,140 @@ +# Behaviour Selection — v0.1 Implementation Brief + +> **Status: Design only.** Experiment 19 pending. This brief is a constraint on the experiment, not an architecture. + +--- + +## The Problem (Discovered) + +Experiments 1–14 proved that the workspace layout is stable and the reasoning engine works. What they revealed but could not fix: + +> The current engine behaviour is: **ask → wait → ask → wait**. Every turn produces a question. This makes the investigation feel like automated Q&A rather than guided thinking. + +The user's framing from Exp 15: *"An expert consultant does not have a script. They have behaviours — recurring patterns of action that they deploy based on what they observe."* + +This experiment tests whether adding **behaviour selection** between assessment and conversation changes that pattern in a meaningful way. + +--- + +## What We Can Measure Now (From Exp 18) + +The assessor produces three reliable dimensions: + +| Dimension | What it tells us | Available now? | +|-----------|-----------------|----------------| +| Phase | Where the investigation is (orienting → concluding) | ✓ | +| Progress | Whether understanding is advancing (accelerating/steady/stalled) | ✓ | +| Conversation Health | Whether the interaction pattern is productive (healthy/too_broad/too_narrow) | ✓ | + +These are sufficient for a first test. We do not need evidence quality, uncertainty trend, or understanding trajectory yet. + +--- + +## v0.1 Behaviour Set: Five Patterns + +The smallest useful set that covers the gap between "always asking" and "facilitated thinking": + +| Behaviour | When to deploy | What it does | +|-----------|---------------|--------------| +| **Acknowledge** | Any turn where user provided useful information (at least one resolved node) | State what was learned; do not immediately ask a new question | +| **Clarify** | Conversation health is `too_broad` or phase is `orienting` with insufficient data | Ask for a single specific piece of context, not an unknown-node query | +| **Summarise** | Phase is `synthesising` or `concluding`; or ≥3 turns have passed without summarisation | Restate current understanding; compress without losing detail | +| **Continue** | Default — no other behaviour matches | Ask the next useful question (current behaviour, but made explicit) | +| **Pause** | Phase is `focusing` with stalled progress | Hold space; acknowledge what was learned; invite reflection rather than asking for more | + +Every turn must select exactly one of these. No combinations, no secondary actions. The test is: does *choosing* change the pattern? + +--- + +## Selection Rules (One Rule Per Behaviour) + +These are plain conditions with no scoring, no weights, no convergence: + +1. **Acknowledge triggers** if `conversation health == healthy` AND at least one node was resolved this turn +2. **Clarify triggers** if `conversation health == too_broad` OR `phase == orienting` AND observations < 3 +3. **Summarise triggers** if `phase == synthesising` OR `phase == concluding` OR (turns ≥ 3 AND no summarisation in recent turns) +4. **Pause triggers** if `phase == focusing` AND `progress == stalled` +5. **Continue** is the default — use it when none of the above match + +If multiple rules fire simultaneously, priority is: Acknowledge > Clarify > Summarise > Pause > Continue. No convergence required. If two conditions are equally relevant, pick the one that adds *information* rather than the one that asks for more input. + +--- + +## What v0.1 Does NOT Do + +These are intentional exclusions — not deferred features: + +- **No scoring or weighting.** A condition either matches or it doesn't. +- **No "convergence" threshold.** If two dimensions trigger, pick by the priority rule. +- **No evidence quality or uncertainty trend integration.** We don't have that data yet, and we don't need it for this test. +- **No stable behaviour pairing.** Acknowledge replaces "acknowledge + communicate confidence." One action per turn. +- **No rationale output or developer view.** That's infrastructure, not signal. +- **No phase-constrained allow/block tables.** The rules above *are* the constraints. + +--- + +## Success Criteria + +How we know whether behaviour selection is worth continuing: + +1. **Behaviour diversity:** Does the system deploy at least 3 different behaviours across a normal investigation, or does it default to Continue (Continue) most of the time? +2. **Acknowledge appears:** Does Acknowledge fire whenever new information resolves an uncertainty? If not, the trigger condition is wrong — fix it, don't abandon selection. +3. **Pause feels like relief, not delay:** When Pause fires, does the user experience it as a natural break rather than a system failure to produce a question? +4. **Summarise compresses meaningfully:** Does the summarised understanding feel useful (new synthesis) or redundant (restatement of what's already on screen)? +5. **Conversation rhythm changes:** Is there a perceptible difference between "engine always asking" and "engine sometimes acknowledging/summarising/pausing first"? + +If none of these can be evaluated after 2-3 real investigations with v0.1, the experiment was too small to answer the question. Expand the behaviour set or extend the test — don't abandon selection. + +--- + +## Future Considerations (Not In v0.1) + +| Concept | Status | Why deferred | +|---------|--------|-------------| +| Signal weighting / scoring formula | Speculative | No observed basis; inventing numbers | +| Convergence requirement | Speculative | Design preference, not discovery | +| Behaviour Readiness derived dimension | Architecture housekeeping | Useful later if v0.1 validates the approach | +| Full 14-behaviour inventory | Available but not tested | Expand only after v0.1 proves the mechanism works | +| Rationale output for developer view | Infrastructure | Not signal; can be added post-validation | +| Multi-dimensional contradiction detection | Complex, unneeded yet | One rule per behaviour is simpler and testable | +| Phase-specific allow/block tables | Invented constraints | Rules above *are* the constraints | + +--- + +## Evaluation Criteria for Behaviour Selection + +How we know a behaviour is working? Not through visual metrics, but through conversational quality: + +1. **Does each turn feel like it builds on the previous one?** (Continuity) +2. **Does the user understand why they are being asked what they are being asked?** (Purpose) +3. **Does the investigation feel guided rather than mechanical?** (Direction) +4. **Does the user feel understood, not just processed?** (Respect) +5. **Does uncertainty feel honest, not manufactured?** (Trust) +6. **Does progress feel real, not illusory?** (Substance) + +These six criteria come directly from `facilitator-behaviour.md` (Experiment 14). They apply to all experiments that touch conversation behaviour. + +--- + +## Relationship to Assessment Layer + +Behaviour Selection does not replace the assessor. It *consumes* it. + +| Assessment Dimension | How Selection Uses It | +|---------------------|----------------------| +| Phase | Determines which behaviours are available (orienting → acknowledge/clarify; synthesising → summarise) | +| Progress | Stalled progress in focusing phase triggers Pause instead of Continue | +| Conversation Health | `too_broad` triggers Clarify; healthy with resolution triggers Acknowledge | + +If the assessment contract changes, selection rules that read those dimensions must be reviewed. The mechanism (plain condition matching, no scoring) is stable regardless. + +--- + +## What This Experiment Proves + +One question: **Does selecting from a small set of behaviours — instead of always asking — make the investigation feel more like guided thinking and less like automated Q&A?** + +If yes: expand the behaviour set and test refinement in v0.2. +If no: either the five behaviours are wrong (not selection itself) or the engine's core loop needs a deeper change than this layer can provide. + +Nothing else matters until this is answered. diff --git a/docs/reasoning-contract-backlog.md b/docs/reasoning-contract-backlog.md index ca72614..5e9738a 100644 --- a/docs/reasoning-contract-backlog.md +++ b/docs/reasoning-contract-backlog.md @@ -146,6 +146,103 @@ The current adapter (`lib/map/investigation-map-adapter.js`) uses generic placeh --- +## Behaviour Selection (Experiment 19) + +**Goal:** Test whether selecting from a small set of behaviours — instead of always asking a question — makes the investigation feel like guided thinking rather than automated Q&A. + +### v0.1 Behaviour Set (5 Patterns) + +| Behaviour | When to deploy | What it does | +|-----------|---------------|--------------| +| **Acknowledge** | Any turn where user provided useful info (≥1 resolved node) | State what was learned; do not immediately ask | +| **Clarify** | Conversation health is `too_broad` OR phase is `orienting` with insufficient data | Ask for a single specific piece of context | +| **Summarise** | Phase is `synthesising`/`concluding`; or ≥3 turns without summarisation | Restate current understanding; compress without losing detail | +| **Continue** | Default — no other behaviour matches | Ask the next useful question (current engine behaviour) | +| **Pause** | Phase is `focusing` with stalled progress | Hold space; acknowledge what was learned; invite reflection | + +### Selection Rules (No Scoring, No Weights) + +Plain conditions. If multiple fire, priority is: Acknowledge > Clarify > Summarise > Pause > Continue. + +1. Acknowledge if conversation health is healthy AND at least one node resolved +2. Clarify if conversation health is too_broad OR phase is orienting with <3 observations +3. Summarise if phase is synthesising/concluding OR ≥3 turns without summarisation +4. Pause if phase is focusing AND progress is stalled +5. Continue as default + +### Why v0.1 Is Deliberately Narrow + +- No scoring or weighting (invented numbers, not observed signals) +- No convergence requirements (design preference, not discovery) +- Only the three assessment dimensions currently available (phase, progress, conversation health) +- One behaviour per turn — no combinations, no stable pairing +- No rationale output or developer view infrastructure (signal first, display later) + +### Future Considerations (Not In v0.1) + +See `docs/behaviour-selection.md` section "Future Considerations" for: signal weighting, convergence thresholds, full 14-behaviour inventory, Behaviour Readiness derived dimension, rationale output. + +--- + +## Investigation State Assessment (Experiment 18) + +The investigation state assessment layer introduces three new assessed dimensions that feed into behaviour selection: **phase**, **progress**, and **conversationHealth**. Each dimension has its own value enum, confidence level, descriptive signals, and evidence object. The overall assessment uses the minimum confidence across all dimensions. + +### Phase Assessment + +| Feature | UI need | Temporary mock | Desired reasoning output | Likely stage | Notes | +|---------|---------|---------------|------------------------|-------------|-------| +| Behaviour Selection (phase gate) | Determine which behaviours are appropriate now | Heuristic based on resolved node count + hasActiveQuestion | `assessment.phase: { value, confidence, signals[], evidence }` where value ∈ `"orienting"`, `"exploring"`, `"focusing"`, `"deepening"`, `"synthesising"`, `"concluding"`, `"cannot_determine"` | Per-turn assessment | Deterministic thresholds: conclusive (active=0, resolved≥2), synthesising (active≤1, ratio>0.5), focusing (active=1, observations≥3), exploring (observations≥2, ratio<0.4), deepening (active>1, resolved≥3) | + +### Progress Assessment + +| Feature | UI need | Temporary mock | Desired reasoning output | Likely stage | Notes | +|---------|---------|---------------|------------------------|-------------|-------| +| Behaviour Selection (urgency gate) | Determine whether investigation is moving forward and at what velocity | Heuristic based on resolved count trend | `assessment.progress: { value, confidence, signals[], evidence }` where value ∈ `"accelerating"`, `"steady"`, `"stalled"`, `"looping"`, `"spiralling"`, `"cannot_determine"` | Per-turn assessment | Single-snapshot approximation in v0.1 using resolution ratio thresholds: accelerating (>0.6), steady (0.2-0.6), stalled (<0.2 with ≥1 resolved) | + +### Conversation Health Assessment + +| Feature | UI need | Temporary mock | Desired reasoning output | Likely stage | Notes | +|---------|---------|---------------|------------------------|-------------|-------| +| Behaviour Selection (interruption gate) | Determine whether the interaction pattern needs adjustment regardless of investigation state | Heuristic based on question type and unknown count | `assessment.conversationHealth: { value, confidence, signals[], evidence }` where value ∈ `"healthy"`, `"repetitive"`, `"too_broad"`, `"too_narrow"`, `"user_overloaded"`, `"user_under_informed"`, `"cannot_determine"` | Per-turn assessment | v0.1 rules: healthy (has unknown + has question), too_broad (>3 active, <2 resolved), too_narrow (≤1 observation with question) | + +### Confidence Aggregation + +| Feature | UI need | Temporary mock | Desired reasoning output | Likely stage | Notes | +|---------|---------|---------------|------------------------|-------------|-------| +| Overall assessment trustworthiness | How much should we trust any individual dimension? | N/A — no prior equivalent | `assessment.confidence` = min(phase.confidence, progress.confidence, conversationHealth.confidence) where "high" < "medium" < "low" < "cannot_determine" | Per-turn assessment | Conservative: if ANY dimension is low/cannot_determine, overall drops. This prevents false precision in behaviour selection. | + +### Evidence Objects (Source Mapping) + +All evidence fields are derived from the situation graph and orchestrator diagnostics without direct graph traversal by the behaviour layer: + +| Evidence Field | Source | Available In | +|---------------|--------|-------------| +| `resolvedNodeCount` | count of nodes with status `"resolved"` or in `resolvedNodeIds` | Every turn | +| `activeUnknownCount` | count of unknown-kinded unresolved nodes + active node fallback | Every turn | +| `unknownResolutionRatio` | resolvedNodeCount / totalNonEmptyNodes (null if total ≤ 0) | Every turn | +| `observationDensity` | observations = observation-kind known/resolved + high-confidence non-unknown non-state | Every turn | +| `evidenceDepth` | `"shallow"` (<2), `"moderate"` (2-3), `"deep"` (≥4 observations) | Every turn | +| `turnCount` | approximated as `floor(totalNodes / 3)` | Every turn | +| `hasActiveQuestion` | Boolean: `selectedQuestion?.nodeId` exists | Every turn | +| `summaryLength` | Length of `situationGraph.currentSummary` | Every turn | + +### Current Limitations (Experiment 18 v0.1) + +These are acknowledged constraints of the current implementation, not change requests: + +- **Single-snapshot progress**: v0.1 uses a resolution ratio from the current snapshot only. Multi-turn trend detection (looping, spiralling) is planned but requires turn history data not yet available in the contract. +- **No evidence quality dimension**: This is specified in the architecture doc but requires per-node evidence confidence scoring across multiple sources — not yet implementable. +- **No understanding trajectory dimension**: Requires comparing narrative complexity across turns; depends on future narrative evolution. +- **No uncertainty trend dimension**: Requires tracking which unknowns resolve by what pattern across turns. +- **No behaviour readiness layer**: The final synthesis of all dimensions into behaviour signals is deferred to the behaviour selection experiment. + +### Implementation Status + +**Implemented.** The assessor (`lib/assessment/investigation-state-assessor.js`) produces a deterministic assessment object matching this contract at v0.1 schema version. Integration call sites in `lib/graph/orchestrator.js` (lines ~552, ~904, ~1013) pass correctly shaped input to `assessInvestigationState()`. The 51-test suite validates all classification rules and edge cases. + +--- + ## Facilitator View Projection (Experiment 12) Version C derives its content from existing graph fields without requiring new backend data. The following fields are used as inputs: