10 KiB
Failure Modes — Architecture Experiment 17
This is a design document only. Do not implement yet. Record failures as they are identified, do not solve them here. Solving these failures becomes future experimental work.
Failure Mode 1 — Graph Incomplete at Narrative Stage
Sequence
Graph incomplete → Narrative weak → Assessment unreliable → Behaviour inappropriate → Poor question → User confidence falls
Description
If the reasoning graph fails to capture a critical observation (node missing, relationship not established, status incorrectly marked), the narrative produced from it will be structurally weaker than reality. The assessment will evaluate a distorted picture. The behaviour selection will deploy based on incorrect state. The conversation response will ask about something already known or miss something urgently uncertain.
Observable Signal
Narrative contains gaps that the user immediately recognises. "But I already told you that" or "You're asking about X when Y was just established."
Do Not Solve Here
This is a graph integrity problem, not a narrative or behaviour problem. The solution belongs in graph reasoning validation, not in these layers.
Failure Mode 2 — Narrative Over-translates Graph
Sequence
Narrative invents synthesis → Assessment receives fabricated coherence → Behaviour selects confident action → User perceives overconfidence → Trust erodes
Description
The narrative layer may produce a coherent-sounding summary by connecting graph nodes that were never connected by the reasoning engine. The assessment then evaluates this artificially coherent picture. The behaviour becomes overconfident because the narrative shows strong convergence where none actually exists in the graph.
Observable Signal
User feels the facilitator is "seeing patterns that aren't there" or "connecting things I didn't connect." Confidence in the system decreases.
Do Not Solve Here
This is a narrative translation constraint problem. The solution belongs in the narrative composition rules (Experiment 14), specifically the "Never invent facts" principle.
Failure Mode 3 — Assessment Reaches False Precision
Sequence
Assessment assigns definitive values → Behaviour reads confidence that doesn't exist → Response over-commits to interpretation → User corrects → Investigation backtracks
Description
The assessment may assign values (e.g., "Evidence Quality: Strong") that appear certain but are based on insufficient or noisy data. When the behaviour layer treats this as high-confidence input, it deploys actions that assume more certainty than actually exists. The user must then correct the facilitator's overconfidence.
Observable Signal
User explicitly states uncertainty in areas where the system acts confidently. "That doesn't seem right yet" or "We're not at that stage."
Do Not Solve Here
This is an assessment precision problem. Principle 6 of the state assessment architecture (Assessment Is Stateful Across Turns, Uncertainty About Assessment Is Itself Assessable) partially addresses this but does not solve it.
Failure Mode 4 — Behaviour Selection Overlaps or Conflicts
Sequence
Multiple dimensions signal different behaviours → Selection picks one → Other needs unmet → Investigation feels inconsistent → User confused about direction
Description
Different assessment dimensions may converge on different behaviours (e.g., Evidence Quality: Contradictory suggests Challenge assumption, while Understanding Trajectory: Consolidating favours Refine understanding). The selection layer must choose one, leaving the other need unaddressed. Over turns this creates an inconsistent investigative personality.
Observable Signal
The facilitator feels like different people on different turns — sometimes synthesising when the user needs precision, sometimes questioning when the user needs acknowledgment.
Do Not Solve Here
This is a behaviour selection priority problem. The decision matrix in Experiment 16 captures some patterns but does not resolve conflicts between equally urgent signals.
Failure Mode 5 — Conversation Response Does Not Match Selected Behaviour
Sequence
Behaviour correctly selected → Response generation diverges from behaviour intent → User receives wrong type of interaction → Investigation rhythm disrupted
Description
The selected behaviour may be correct (e.g., "Pause" to hold space after a significant insight), but the conversation response layer may generate an active question instead of holding space. The behavioural intention is lost in translation between abstract selection and concrete text generation.
Observable Signal
User says "I was still thinking about that" or "Don't ask me another question yet." The facilitator pushes forward when it should have held.
Do Not Solve Here
This is a response generation alignment problem. It requires ensuring the text generation layer respects the behavioural intention constraint — but the mechanics of that alignment are not solved here.
Failure Mode 6 — Workspace Projects Stale State
Sequence
State changes during turn → Workspace renders before update complete → User sees contradiction between conversation and panels → Cognitive dissonance
Description
If the workspace projection renders from a narrative state that has not yet been fully updated (e.g., a node was just resolved but the panel still shows it as unknown), the user receives conflicting information across interface areas.
Observable Signal
Facilitator says "We now know X" while the panel still lists X as an active unknown. Or vice versa.
Do Not Solve Here
This is a synchronisation and rendering timing problem, not an architectural layer problem.
Failure Mode 7 — User Withholds Information Due to Misreading Facilitator State
Sequence
Facilitator appears confident → User withholds doubt → Graph receives incomplete data → Assessment evaluates distorted picture → Investigation proceeds on false foundation
Description
If the workspace or conversation makes the facilitator appear more confident or certain than it should be, the user may stop providing information they think is irrelevant. The graph then operates on an incomplete picture. Subsequent turns compound the error because each new turn starts from a fundamentally wrong state.
Observable Signal
User's contributions become shorter over time. They answer questions but do not volunteer context they clearly possess. "That's everything I know" when they clearly know more.
Do Not Solve Here
This is a confidence signalling and user psychology problem. It touches every layer because a false sense of completeness can be communicated through narrative language, assessment presentation, or conversation tone.
Failure Mode 8 — Assessment Accumulates Error Across Turns
Sequence
Assessment slight overconfidence → Behaviour slightly wrong → Graph updates based on that wrong behaviour → Next assessment builds on wrong graph → Error compounds
Description
The assessment layer (Principle 5: Assessment Is Stateful Across Turns) accumulates state. If early turns establish an incorrect trajectory, each subsequent turn's assessment inherits and amplifies that error. The investigation enters a state where it cannot self-correct because every layer is built on the same mistaken foundation.
Observable Signal
The investigation feels like it is "going in circles" but for a reason the user cannot articulate. Turns happen frequently but understanding does not advance.
Do Not Solve Here
This is a systemic error accumulation problem. It may require periodic re-evaluation of earlier turns, or explicit anomaly detection when assessment signals contradict across dimensions.
Failure Mode 9 — Behaviour Readiness Produces Conservative Paralysis
Sequence
Assessment shows multiple uncertain dimensions → Readiness narrows behaviour options to safe defaults → Facilitator only acknowledges and clarifies → Investigation stalls because no new direction is ever proposed
Description
When the assessment correctly identifies uncertainty across most dimensions, the behaviour readiness layer may produce a narrow set of "safe" behaviours (Acknowledge, Clarify). The facilitator becomes overly conservative, repeatedly acknowledging without advancing. The investigation stalls because no bold action was ever taken.
Observable Signal
The user feels stuck in an acknowledgment loop. "I know you've heard me — what should I think about next?"
Do Not Solve Below
This is a risk tolerance problem at the behaviour selection layer. The system needs permission to act with partial information — but the threshold for that permission is not defined here.
Failure Mode 10 — Narrative Becomes Too Complex for User Cognitive Load
Sequence
Graph grows dense → Narrative includes all available sections → User sees too much simultaneously → Investigation Map becomes incomprehensible → User disengages
Description
As the investigation deepens, more narrative sections become populated. If all are presented simultaneously, the workspace may overwhelm rather than clarify. The user cannot find what matters because everything is equally visible.
Observable Signal
User stops reading panels. They focus only on the conversation lane and ignore status, understanding, or evidence displays entirely.
Do Not Solve Here
This is a presentation selection problem — not all narrative sections need to be visible at all times. Progressive disclosure should be governed by investigation phase and user state, but this is an implementation detail.
Recording Note
These failure modes are observations, not requirements for solutions. Each identifies where the architecture may fail when implemented. Which of these actually occur — and in what order — can only be determined through experimentation with working implementations.
This document exists so that future experiments know what to watch for.