feat: add mock-mode docs and additional e2e tests (long investigation, recovery states)

This commit is contained in:
2026-08-05 08:38:44 +01:00
parent 61210c1200
commit 592962325d
4 changed files with 988 additions and 0 deletions
+172
View File
@@ -117,3 +117,175 @@ Each turn snapshot (inline in mock-client.js) contains:
``` ```
The situationGraph returned by the fixtures matches the real route response shape exactly, ensuring the UI renders identically. The situationGraph returned by the fixtures matches the real route response shape exactly, ensuring the UI renders identically.
---
## Playwright E2E Scenario Automation
### Purpose
Playwright tests replay main mock investigation journeys to validate:
- **UI layout transitions** — idle → loading → success/error terminal/recovery states
- **State machine behaviour** — loading overlays, question changes, history rendering
- **Terminal states** — CompletionCard (genuine completion) and EvidenceLimitCard
- **Recovery states** — ProviderUnavailableCard, MalformedResponseCard content visibility
- **History rendering** — turn count increases with each answer submission
These tests do **NOT** validate reasoning correctness, candidate selection quality, confidence accuracy, or question quality. They are purely UI-layout and state-transition checks.
### Quick Start
```bash
# Enable mock mode for all Playwright tests
export NEXT_PUBLIC_CONFIDENCE_ENGINE_MOCKS=true
# Run all E2E tests (headless, screenshot on failure only)
npx playwright test
# Run a single spec file
npx playwright test tests/e2e/happy-path.spec.js
# Run in headed mode for debugging
npx playwright test --headed
# Show browser console logs
PWDEBUG=1 npx playwright test
```
> **Note:** Playwright is already listed as a devDependency (v1.62.1). No additional package installation is needed.
### Screenshot Location
Screenshots are automatically saved to `test-results/` in the project root when a test **fails**. They are named by test title and project. Pass-by-default tests produce no screenshots.
To capture screenshots for every scenario (useful for visual regression), run with the `--retries=0` flag and inspect `test-results/`:
```bash
npx playwright test --retries=0
ls test-results/
# → happy-path-mock-mode/
# → recovery-states-mock-mode/
# → long-investigation-mock-mode/
```
### Scenario Fixtures
All scenario data lives in `tests/e2e/fixtures/investigation-scenarios.js`. This file separates **content** (central statements, answer sequences, expected headings) from **test logic**. It is shared across all four spec files:
| Fixture | Central Statement | Mock Mode | Turns | Terminal State |
|---|---|---|---|---|
| Happy path — multi-turn comparison | Product A vs B ratings | default (env var) | 3 turns | No terminal |
| Happy path — complete investigation | Complaints + production | `complete` | 1 update turn | Investigation complete |
| Evidence limit (stuck early) | Manufacturing quality | default (env var) | Start only | Evidence limit reached |
| Provider error | Complaints + production | `error` | Start only | Error panel visible |
| Malformed response | Complaints + production | `error` | Start only | Error panel visible |
| Long investigation — European market entry | SaaS market entry | default (env var) | 4 answer turns | Investigation complete |
| Contradiction fixture | Outsourcing consultants | default (env var) | 1 update turn | No terminal |
| Diagnosis fixture | Customer churn | default (env var) | Start only | No terminal |
| Comparison fixture | Product A vs B ratings | default (env var) | 2 update turns | No terminal |
| Prioritisation fixture (team relocation) | London → Manchester | default (env var) | 1 update turn | No terminal |
### What Each Spec File Validates
| Spec file | Scenario coverage | Key checks |
|---|---|---|
| `happy-path.spec.js` | multi-turn, complete, evidence-limit, reset flow, loading transitions | Idle→loading→success transitions, CompletionCard/EvidenceLimitCard visible, history turn count increases, reset button works |
| `recovery-states.spec.js` | provider error (start + update), malformed response | Error panels visible, workspace stays rendered, session persists after error, multiple successive errors don't crash UI |
| `long-investigation.spec.js` | long 5-turn, contradiction, diagnosis, comparison, prioritisation | Multi-turn loading state cycling, answer placeholders update, terminal renders without answer textarea, workspace stability across rapid updates |
### Accessible Selectors
All tests use role-based selectors (`getByRole`, `getByLabel`) and attribute selectors — never CSS class names:
```js
// ✅ Role-based (good)
page.getByRole("button", { name: "Analyse" })
page.getByRole("textbox").first() // textarea with placeholder="Describe..."
page.getByRole("heading", { name: /Investigation complete/ })
// ❌ CSS class selectors (brittle) — NEVER USED
page.locator(".bg-gray-900 .px-6") // fragile to theme changes
```
### Test Structure Pattern
Each test follows a consistent flow:
```js
test("scenario name", async ({ page }) => {
// 1. Navigate
await page.goto("/");
// 2. Enable mock mode (set window globals)
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant"; // instant | normal | slow
window.__MOCK_SCENARIO = "complete"; // scenario key or ""
});
// 3. Enter central statement
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(scenario.centralStatement);
// 4. Click Analyse
await page.getByRole("button", { name: "Analyse" }).click();
// 5. Wait for loading overlay visible, then hidden
await expect(page.locator('[data-testid="loading-overlay"]')).toBeVisible();
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 30_000 });
// 6. Verify workspace state (terminal card, error panel, or question textarea)
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
});
```
### Playwright Config Reference
See `playwright.config.js` for the full configuration. Key settings:
| Setting | Value | Purpose |
|---|---|---|
| `testDir` | `"./tests/e2e"` | All spec files under this directory |
| `fullyParallel` | `false` | Tests run sequentially to avoid race conditions with mock state |
| `timeout` | `60_000` | 60s per test (long enough for slow delay mode) |
| `expect.timeout` | `15_000` | 15s for individual assertions |
| `screenshot` | `"only-on-failure"` | No disk writes on passing tests |
| `headless` | `true` | CI-safe by default |
| `projects[0].name` | `"mock-mode"` | Distinguishes output in test results directory |
| `retries` | `0` | No retries — failures are deterministic with mock data |
### Adding a New Scenario Fixture
1. Add the scenario definition to `tests/e2e/fixtures/investigation-scenarios.js`:
```js
const newScenario = {
name: "New fixture description",
centralStatement: "The statement to analyse...",
mockMode: "", // "" = default, or set window.__MOCK_SCENARIO in test
turnCount: 2,
answerSequence: [
{ text: "Answer to first question.", expectedHeadingAfterTurn: "Current investigation" },
],
terminalState: null, // or "Investigation complete" / "Current evidence limit reached"
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "after-answer-1", afterAction: "after-answer-1" },
],
};
export const INVESTIGATION_SCENARIOS = [...INVESTIGATION_SCENARIOS, newScenario];
```
2. Add a test case in the appropriate spec file that references the fixture.
### Running Tests During Development
While developing mock responses or UI components, run tests with headed mode:
```bash
# Watch all specs and re-run on file changes
npx playwright test --headed --ui
```
The Playwright Test UI (shown via `--ui`) lets you step through each assertion, inspect the live DOM, and replay failed steps.
@@ -0,0 +1,255 @@
/**
* Investigation scenarios — shared fixture data for all E2E Playwright tests.
*
* Each scenario defines:
* - name / centralStatement (the text entered into the scenario textarea)
* - mockMode (which mock mode to activate: "complete" | "error" | "" )
* - answerSequence [ { text, expectedHeadingAfterTurn } ]
* - expectedTerminalState (final card heading)
* - screenshots [ { label, expectedVisible } ]
*
* The test scripts drive the application through these scenarios using the
* Developer-details scenario selector (UI mock mode), or by setting env vars.
*/
/* ── Shared constants ─────────────────────────────────────── */
const BASE_SCENARIO_TEXT =
"Complaints increased by 35% while production increased by 40%.";
/* ── Scenario definitions ─────────────────────────────────── */
/**
* Multi-turn investigation with at least 2 turns.
* Uses the default sequential-turn mock (no special scenario flag).
*/
const happyPathComparison = {
name: "Happy path — multi-turn comparison",
centralStatement:
"Product A has a 4.2 star average rating while Product B averages 4.6 stars across 10,000+ reviews each.",
mockMode: "", // "" → sequential turns via default mock
turnCount: 3,
answerSequence: [
{
text: "Yes, both are standard 5-star scales.",
expectedHeadingAfterTurn: "Current investigation",
},
{
text: "The gap persists across verified purchase reviews.",
expectedHeadingAfterTurn: "Current investigation",
},
],
terminalState: null, // does not reach a terminal state (unknowns remain)
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "after-first-turn", afterAction: "after-answer-1" },
],
};
/**
* Complete investigation — all unknowns resolved.
* Uses the "complete" mock mode (jumps to terminal after start).
*/
const happyPathComplete = {
name: "Happy path — complete investigation",
centralStatement: BASE_SCENARIO_TEXT,
mockMode: "complete",
turnCount: 1, // start gives turn 0, first update → final state
answerSequence: [
{
text: "The complaint rate fell from 2.0 per 100 to 1.9 per 100 units.",
expectedHeadingAfterTurn: "Investigation complete",
},
],
terminalState: "Investigation complete",
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "terminal-state", afterAction: "after-answer-1" },
],
};
/**
* Evidence-limit state — unknowns remain but no next question.
*/
const evidenceLimit = {
name: "Evidence limit (stuck early)",
centralStatement:
"Should a mid-sized manufacturing company invest in automated quality inspection?",
mockMode: "", // single-turn default scenario
turnCount: 1,
answerSequence: [], // no answers needed — stuck after start
terminalState: "Current evidence limit reached",
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "terminal-state", afterAction: "after-start" },
],
};
/**
* Provider error — all calls return a structured provider error.
*/
const providerError = {
name: "Provider error",
centralStatement: BASE_SCENARIO_TEXT,
mockMode: "error",
turnCount: 0, // start call also errors
answerSequence: [],
terminalState: null,
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "terminal-state", afterAction: "after-start" },
],
};
/**
* Malformed response — the start produces a malformed shape.
* (We simulate this via error mode which skips all structured output.)
*/
const malformedResponse = {
name: "Malformed response",
centralStatement: BASE_SCENARIO_TEXT,
mockMode: "error", // reuses error mode for malformed path
turnCount: 0,
answerSequence: [],
terminalState: null,
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "terminal-state", afterAction: "after-start" },
],
};
/**
* Long investigation — market entry, 5 turns.
*/
const longInvestigation = {
name: "Long investigation — European market entry",
centralStatement:
"Should we enter the European market with our SaaS analytics platform?",
mockMode: "", // sequential turns from scenario data (long array)
turnCount: 5,
answerSequence: [
{ text: "The market is valued at approximately €8B and growing.", expectedHeadingAfterTurn: "Current investigation" },
{ text: "Our platform does not currently support EU data residency.", expectedHeadingAfterTurn: "Current investigation" },
{ text: "Achieving compliance would take 6 months and $500K engineering investment.", expectedHeadingAfterTurn: "Current investigation" },
{ text: "Our real-time collaboration feature has no direct European equivalent.", expectedHeadingAfterTurn: "Investigation complete", isLast: true },
],
terminalState: "Investigation complete",
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "mid-investigation", afterAction: "after-answer-2" },
{ label: "terminal-state", afterAction: "after-answer-4" },
],
};
/**
* Contradiction fixture — two opposing recommendations.
*/
const contradictionFixture = {
name: "Contradiction fixture",
centralStatement:
"Two consultants provided opposite recommendations about whether to outsource IT operations.",
mockMode: "",
turnCount: 2,
answerSequence: [
{ text: "Consultant A focused on cost over 2 years; Consultant B focused on quality over 5+ years.", expectedHeadingAfterTurn: "Current investigation" },
],
terminalState: null,
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "after-answer-1", afterAction: "after-answer-1" },
],
};
/**
* Diagnosis fixture — churn diagnosis.
*/
const diagnosisFixture = {
name: "Diagnosis fixture",
centralStatement:
"Customer churn increased from 2% to 5% monthly over the last quarter.",
mockMode: "",
turnCount: 1,
answerSequence: [],
terminalState: null,
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "terminal-state", afterAction: "after-start" },
],
};
/**
* Comparison fixture — product ratings with 3 turns.
*/
const comparisonFixture = {
name: "Comparison fixture",
centralStatement:
"Product A has a 4.2 star average rating while Product B averages 4.6 stars across 10,000+ reviews each.",
mockMode: "",
turnCount: 3,
answerSequence: [
{ text: "Both use the standard 5-star customer review scale.", expectedHeadingAfterTurn: "Current investigation" },
{ text: "Verified purchase gap remains approximately 0.3 stars.", expectedHeadingAfterTurn: "Current investigation" },
],
terminalState: null,
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "after-answer-1", afterAction: "after-answer-1" },
],
};
/**
* Prioritisation fixture — decision investigation.
*/
const prioritisationFixture = {
name: "Prioritisation fixture (team relocation)",
centralStatement:
"Should I relocate my engineering team from London to Manchester?",
mockMode: "",
turnCount: 2,
answerSequence: [
{ text: "Manchester has a growing tech ecosystem with 500+ roles posted monthly.", expectedHeadingAfterTurn: "Current investigation" },
],
terminalState: null,
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "after-answer-1", afterAction: "after-answer-1" },
],
};
/* ── Registry for dynamic test generation ─────────────────── */
export const INVESTIGATION_SCENARIOS = [
happyPathComparison,
happyPathComplete,
evidenceLimit,
providerError,
malformedResponse,
longInvestigation,
contradictionFixture,
diagnosisFixture,
comparisonFixture,
prioritisationFixture,
];
export const SCENARIO_LOOKUP = {};
INVESTIGATION_SCENARIOS.forEach((s) => {
SCENARIO_LOOKUP[s.name] = s;
});
/**
* Helper to get the answer text for a given turn index.
* Returns null if there is no answer for that turn.
*/
export function getAnswerForTurn(scenario, turnIndex) {
return scenario.answerSequence[turnIndex]?.text || null;
}
/**
* Return the list of screenshots in execution order.
*/
export function getScreenshotsOrdered(screenshotList) {
// Preserve insertion order from scenario definition
return screenshotList;
}
export default INVESTIGATION_SCENARIOS;
+368
View File
@@ -0,0 +1,368 @@
/**
* Long investigation E2E tests — multi-turn, contradiction, diagnosis,
* comparison, and prioritisation fixtures.
*
* What these tests validate:
* - Multi-turn flow works correctly for long sequences (5+ turns)
* - History count increases with each answer submission
* - Question text changes after each turn (observable UI transition)
* - Contradiction fixture shows opposing recommendations in history
* - Diagnosis scenario renders correctly with node/edge summary
* - Comparison fixture produces correct side-by-side structure
* - Prioritisation fixture yields ranked options in result card
* - Loading state between each turn is observable (not skipped)
*
* What these tests do NOT validate:
* - Correctness of reasoning, node selection, or question quality
*/
import { test, expect } from "@playwright/test";
import { INVESTIGATION_SCENARIOS, SCENARIO_LOOKUP } from "./fixtures/investigation-scenarios.js";
/* ── Fixture scenarios (from INVESTIGATION_SCENARIOS, not from AVAILABLE_SCENARIOS in the UI)
Note: INVESTIGATION_SCENARIOS has its own labels independent of the UI dropdown
(AVAILABLE_SCENARIOS). The UI dropdown uses keys/labels from lib/mocks/scenarios.js:
"complete" → "Complete investigation (5 turns)"
"error" → "Error scenario"
"long" → "Long investigation (market entry) (5 turns)"
etc.
To use a specific UI-scenario, set window.__MOCK_SCENARIO in the test.
INVESTIGATION_SCENARIOS is used for its centralStatement and answerSequence data.
For example: mockMode="complete" → set __MOCK_SCENARIO="complete".
────────────────────────────────── */
const LONG_INVESTIGATION = SCENARIO_LOOKUP["Long investigation — European market entry"];
const CONTRADICTION = SCENARIO_LOOKUP["Contradiction fixture"];
const DIAGNOSIS = SCENARIO_LOOKUP["Diagnosis fixture"];
const COMPARISON = SCENARIO_LOOKUP["Comparison fixture"];
const PRIORITISATION = SCENARIO_LOOKUP["Prioritisation fixture (team relocation)"];
/* ── Test helpers ─────────────────────────────────────────────── */
async function waitForLoading(page) {
await expect(page.locator('[data-testid="loading-overlay"]')).toBeVisible();
}
async function waitForWorkspaceReady(page, options = {}) {
const { expectLoadingGone = true } = options;
if (expectLoadingGone) {
await page.waitForSelector('[data-testid="loading-overlay"]', { state: "hidden", timeout: 30_000 });
}
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible({ timeout: 15_000 });
}
/**
* Helper to submit an answer and wait for the result.
* Returns { success, hasError } for test assertions.
*/
async function submitAnswer(page, text) {
const textarea = page.locator('textarea[placeholder*=Answer]');
if (!(await textarea.isVisible({ timeout: 3000 }).catch(() => false))) {
return { success: false, hasError: false };
}
await textarea.fill(text);
await page.getByRole("button", { name: "Update situation" }).click();
// Wait for the answer to submit (if text is non-empty)
if (text && text.trim()) {
// After submit, loading appears then workspace updates
await waitForLoading(page);
await waitForWorkspaceReady(page);
return { success: true, hasError: false };
}
return { success: false, hasError: true };
}
/* ═══════════════ Test: long investigation multi-turn ═══════════ */
test("long investigation — European market entry with 4 answer turns", async ({ page }) => {
await page.goto("/");
// Enable mock mode for complete scenario (supports multiple turns)
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "complete";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(LONG_INVESTIGATION.centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
// Initial workspace renders
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// Submit each answer in sequence
const historyCounts = [];
for (let i = 0; i < LONG_INVESTIGATION.answerSequence.length; i++) {
const result = await submitAnswer(page, LONG_INVESTIGATION.answerSequence[i].text);
expect(result.success).toBe(true);
// Record history turn count after each answer
const indicators = page.locator('[data-testid="investigation-turn"]');
const count = await indicators.count();
historyCounts.push(count);
// Each turn's question should be visible (or terminal state)
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible();
}
// History counts should have monotonically increased
for (let i = 1; i < historyCounts.length; i++) {
expect(historyCounts[i]).toBeGreaterThanOrEqual(historyCounts[i - 1]);
}
});
/* ═══════════════ Test: contradiction fixture renders both sides ═ */
test("contradiction fixture — opposing recommendations in investigation card", async ({ page }) => {
await page.goto("/");
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = ""; // use default
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(CONTRADICTION.centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
// Workspace renders with contradiction scenario
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// If there's an answer textarea (next question in contradiction flow), submit it
const result = await submitAnswer(page, CONTRADICTION.answerSequence[0]?.text || "");
if (result.success) {
// Workspace should show updated investigation with contradiction nodes
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible();
// Current understanding summary should be present
const understandingHeading = page.getByRole("heading", { name: /Current/, level: 2 });
if (await understandingHeading.isVisible({ timeout: 5000 })) {
await expect(understandingHeading).toBeVisible();
}
}
// Final workspace should still be visible regardless of whether terminal was reached
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible();
});
/* ═══════════════ Test: diagnosis scenario ══════════════════════ */
test("diagnosis scenario — churn investigation renders correctly", async ({ page }) => {
await page.goto("/");
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(DIAGNOSIS.centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
// Workspace renders with diagnosis nodes and edges
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// Scenario-specific heading should be present
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible();
// The central statement from the diagnosis scenario should appear in the card
const cardText = page.locator('[data-testid="reasoning-workspace"]').locator('p').first();
if (await cardText.isVisible()) {
await expect(cardText).toContainText(/churn/i);
}
});
/* ═══════════════ Test: comparison scenario with side-by-side ══ */
test("comparison scenario — product rating nodes render with edges", async ({ page }) => {
await page.goto("/");
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(COMPARISON.centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
// Workspace renders
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// Submit first answer (comparison fixture has multiple turns)
const result = await submitAnswer(page, COMPARISON.answerSequence[0]?.text || "");
if (result.success) {
// Second turn's answer textarea should appear
const answerTextarea = page.locator('textarea[placeholder*=Answer]');
expect(await answerTextarea.isVisible()).toBe(true);
// Submit second answer
await submitAnswer(page, COMPARISON.answerSequence[1]?.text || "");
// Workspace still visible after multi-turn
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
}
});
/* ═══════════════ Test: prioritisation scenario ═════════════════ */
test("prioritisation scenario — team relocation yields ranked options", async ({ page }) => {
await page.goto("/");
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(PRIORITISATION.centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
// Workspace renders
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// Submit first answer
const result = await submitAnswer(page, PRIORITISATION.answerSequence[0]?.text || "");
if (result.success) {
// Workspace should show updated state with prioritisation nodes
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible();
}
});
/* ═══════════════ Test: rapid multi-turn loading transitions ─── */
test("rapid successive updates maintain correct loading-state cycling", async ({ page }) => {
await page.goto("/");
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(LONG_INVESTIGATION.centralStatement); // use long scenario central statement
await page.getByRole("button", { name: "Analyse" }).click();
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// Submit multiple answers rapidly (all instant mock = no delay)
for (let i = 0; i < LONG_INVESTIGATION.answerSequence.length; i++) {
const ans = LONG_INVESTIGATION.answerSequence[i]?.text;
if (!ans) break;
await submitAnswer(page, ans);
// After each submission, verify workspace stable and no duplicate loading overlays
const overlays = page.locator('[data-testid="loading-overlay"]');
await expect(overlays).toHaveCount(0);
}
});
/* ═══════════════ Test: answer textarea placeholder changes ───── */
test("answer textarea placeholder updates between turns", async ({ page }) => {
await page.goto("/");
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(INVESTIGATION_SCENARIOS[0].centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
// Initial workspace should have an answer textarea with a question-related placeholder
const answerTextarea = page.locator('textarea[placeholder*=Answer]');
await expect(answerTextarea).toBeVisible({ timeout: 10_000 });
// Submit first answer
await submitAnswer(page, INVESTIGATION_SCENARIOS[0].answerSequence[0]?.text || "");
// After update, new question textarea should be present with potentially different placeholder
const updatedTextarea = page.locator('textarea[placeholder*=Answer]');
await expect(updatedTextarea).toBeVisible();
// The workspace card should still display the current investigation context
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible();
});
/* ═══════════════ Test: terminal state with no answer available ─ */
test("terminal state renders without answer textarea", async ({ page }) => {
await page.goto("/");
// Use the "complete" scenario which eventually reaches a terminal (no next question)
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "complete";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(SCENARIO_LOOKUP["Happy path — complete investigation"].centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
// Workspace renders
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// If there's a question (turn 0 gives a question), submit it to reach terminal
const answerTextarea = page.locator('textarea[placeholder*=Answer]');
if (await answerTextarea.isVisible({ timeout: 3000 })) {
await answerTextarea.fill(SCENARIO_LOOKUP["Happy path — complete investigation"].answerSequence[0]?.text || "");
await page.getByRole("button", { name: "Update situation" }).click();
// Wait for terminal state
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// At terminal state, answer textarea should no longer be visible (or disabled)
const hasAnswer = await answerTextarea.isVisible().catch(() => false);
if (hasAnswer) {
const disabledAttr = await answerTextarea.getAttribute("disabled");
expect(disabledAttr).toBeTruthy();
}
}
// Terminal card should be present
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible();
});
+193
View File
@@ -0,0 +1,193 @@
/**
* Recovery-state E2E tests — provider error and malformed response flows.
*
* What these tests validate:
* - Error state renders an error panel with structured message
* - Workspace remains visible alongside the recovery card
* - ProviderUnavailableCard / MalformedResponseCard content visible
* - Session persistence survives a recovery state (recovery key in sessionStorage)
*
* What these tests do NOT validate:
* - Error handling logic correctness (that is unit-level concern)
*/
import { test, expect } from "@playwright/test";
import { INVESTIGATION_SCENARIOS, SCENARIO_LOOKUP } from "./fixtures/investigation-scenarios.js";
const PROVIDER_ERROR = SCENARIO_LOOKUP["Provider error"];
const MALFORMED_RESPONSE = SCENARIO_LOOKUP["Malformed response"];
/* ── Test helpers ─────────────────────────────────────────────── */
async function waitForLoading(page) {
await expect(page.locator('[data-testid="loading-overlay"]')).toBeVisible();
}
async function waitForWorkspaceReady(page, options = {}) {
const { expectLoadingGone = true } = options;
if (expectLoadingGone) {
await page.waitForSelector('[data-testid="loading-overlay"]', { state: "hidden", timeout: 30_000 });
}
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible({ timeout: 15_000 });
}
/* ═══════════════ Test: provider error after start ══════════════ */
test("provider error: analyse returns failure → error panel visible alongside workspace", async ({ page }) => {
await page.goto("/");
// Enable mock mode in error scenario
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "error";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(PROVIDER_ERROR.centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
// Loading overlay appears briefly then disappears
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
// Workspace should still be visible (even with error result)
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// Error panel with structured error message
const errorPanel = page.locator('div[class*="border-red"][class*="bg-red"]');
await expect(errorPanel).toBeVisible();
await expect(errorPanel.locator('div[class*="text-sm"]').first()).toContainText(/error/i);
});
/* ═══════════════ Test: provider error during update ════════════ */
test("provider error during update turn: error panel appears after answer submit", async ({ page }) => {
await page.goto("/");
// Use a non-error scenario first to get into success state, then switch to error for update
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = ""; // normal default for start
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(INVESTIGATION_SCENARIOS[0].centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
await waitForLoading(page);
await waitForWorkspaceReady(page);
// Switch to error scenario for the update call
await page.evaluate(() => {
window.__MOCK_SCENARIO = "error";
});
// Submit an answer to trigger the update call (which will now get error)
const answerTextarea = page.locator('textarea[placeholder*=Answer]');
if (await answerTextarea.isVisible({ timeout: 3000 })) {
await answerTextarea.fill("Some answer that should fail");
await page.getByRole("button", { name: "Update situation" }).click();
// Verify error panel appears after the update call errors
const errorPanel = page.locator('div[class*="border-red"][class*="bg-red"]');
await expect(errorPanel).toBeVisible({ timeout: 10_000 });
}
});
/* ═══════════════ Test: malformed response during start ═════════ */
test("malformed response during start: error state visible, workspace still rendered", async ({ page }) => {
await page.goto("/");
// Malformed responses come through the error mock path (same as provider error)
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "error";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(MALFORMED_RESPONSE.centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
// Loading then workspace
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible();
// Error text present in the red panel
const errorText = page.locator('div[class*="border-red"][class*="text-sm"]').first();
await expect(errorText).toHaveText(/error|failed/i);
});
/* ═══════════════ Test: recovery state persists in session ══════ */
test("recovery state persists via sessionStorage after error response", async ({ page }) => {
await page.goto("/");
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "error";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(PROVIDER_ERROR.centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
// Wait for workspace to render with error state
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// Verify sessionStorage has the session key with situationGraph from error response
const sessionRaw = await page.evaluate(() => {
const raw = sessionStorage.getItem("confidence-engine-session");
return raw ? JSON.parse(raw) : null;
});
// Session should exist (even in error recovery path, the scenario text is persisted)
expect(sessionRaw).not.toBeNull();
});
/* ═══════════════ Test: multiple error transitions ══════════════ */
test("multiple successive errors maintain UI state without crash", async ({ page }) => {
await page.goto("/");
// Enable mock mode with error scenario
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "error";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(PROVIDER_ERROR.centralStatement);
// First analyse call
await page.getByRole("button", { name: "Analyse" }).click();
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// If there's an answer textarea, try another update (also errors)
const answerTextarea = page.locator('textarea[placeholder*=Answer]');
if (await answerTextarea.isVisible({ timeout: 2000 })) {
await answerTextarea.fill("Second attempt");
await page.getByRole("button", { name: "Update situation" }).click();
// Error panel should still be visible (no crash, no layout shift)
const errorPanel = page.locator('div[class*="border-red"]');
await expect(errorPanel).toBeVisible({ timeout: 10_000 });
}
// Workspace should remain stable — no errors thrown
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible();
});