feat: add mock-mode docs and additional e2e tests (long investigation, recovery states)

This commit is contained in:
2026-08-05 08:38:44 +01:00
parent 61210c1200
commit 592962325d
4 changed files with 988 additions and 0 deletions
+172
View File
@@ -117,3 +117,175 @@ Each turn snapshot (inline in mock-client.js) contains:
```
The situationGraph returned by the fixtures matches the real route response shape exactly, ensuring the UI renders identically.
---
## Playwright E2E Scenario Automation
### Purpose
Playwright tests replay main mock investigation journeys to validate:
- **UI layout transitions** — idle → loading → success/error terminal/recovery states
- **State machine behaviour** — loading overlays, question changes, history rendering
- **Terminal states** — CompletionCard (genuine completion) and EvidenceLimitCard
- **Recovery states** — ProviderUnavailableCard, MalformedResponseCard content visibility
- **History rendering** — turn count increases with each answer submission
These tests do **NOT** validate reasoning correctness, candidate selection quality, confidence accuracy, or question quality. They are purely UI-layout and state-transition checks.
### Quick Start
```bash
# Enable mock mode for all Playwright tests
export NEXT_PUBLIC_CONFIDENCE_ENGINE_MOCKS=true
# Run all E2E tests (headless, screenshot on failure only)
npx playwright test
# Run a single spec file
npx playwright test tests/e2e/happy-path.spec.js
# Run in headed mode for debugging
npx playwright test --headed
# Show browser console logs
PWDEBUG=1 npx playwright test
```
> **Note:** Playwright is already listed as a devDependency (v1.62.1). No additional package installation is needed.
### Screenshot Location
Screenshots are automatically saved to `test-results/` in the project root when a test **fails**. They are named by test title and project. Pass-by-default tests produce no screenshots.
To capture screenshots for every scenario (useful for visual regression), run with the `--retries=0` flag and inspect `test-results/`:
```bash
npx playwright test --retries=0
ls test-results/
# → happy-path-mock-mode/
# → recovery-states-mock-mode/
# → long-investigation-mock-mode/
```
### Scenario Fixtures
All scenario data lives in `tests/e2e/fixtures/investigation-scenarios.js`. This file separates **content** (central statements, answer sequences, expected headings) from **test logic**. It is shared across all four spec files:
| Fixture | Central Statement | Mock Mode | Turns | Terminal State |
|---|---|---|---|---|
| Happy path — multi-turn comparison | Product A vs B ratings | default (env var) | 3 turns | No terminal |
| Happy path — complete investigation | Complaints + production | `complete` | 1 update turn | Investigation complete |
| Evidence limit (stuck early) | Manufacturing quality | default (env var) | Start only | Evidence limit reached |
| Provider error | Complaints + production | `error` | Start only | Error panel visible |
| Malformed response | Complaints + production | `error` | Start only | Error panel visible |
| Long investigation — European market entry | SaaS market entry | default (env var) | 4 answer turns | Investigation complete |
| Contradiction fixture | Outsourcing consultants | default (env var) | 1 update turn | No terminal |
| Diagnosis fixture | Customer churn | default (env var) | Start only | No terminal |
| Comparison fixture | Product A vs B ratings | default (env var) | 2 update turns | No terminal |
| Prioritisation fixture (team relocation) | London → Manchester | default (env var) | 1 update turn | No terminal |
### What Each Spec File Validates
| Spec file | Scenario coverage | Key checks |
|---|---|---|
| `happy-path.spec.js` | multi-turn, complete, evidence-limit, reset flow, loading transitions | Idle→loading→success transitions, CompletionCard/EvidenceLimitCard visible, history turn count increases, reset button works |
| `recovery-states.spec.js` | provider error (start + update), malformed response | Error panels visible, workspace stays rendered, session persists after error, multiple successive errors don't crash UI |
| `long-investigation.spec.js` | long 5-turn, contradiction, diagnosis, comparison, prioritisation | Multi-turn loading state cycling, answer placeholders update, terminal renders without answer textarea, workspace stability across rapid updates |
### Accessible Selectors
All tests use role-based selectors (`getByRole`, `getByLabel`) and attribute selectors — never CSS class names:
```js
// ✅ Role-based (good)
page.getByRole("button", { name: "Analyse" })
page.getByRole("textbox").first() // textarea with placeholder="Describe..."
page.getByRole("heading", { name: /Investigation complete/ })
// ❌ CSS class selectors (brittle) — NEVER USED
page.locator(".bg-gray-900 .px-6") // fragile to theme changes
```
### Test Structure Pattern
Each test follows a consistent flow:
```js
test("scenario name", async ({ page }) => {
// 1. Navigate
await page.goto("/");
// 2. Enable mock mode (set window globals)
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant"; // instant | normal | slow
window.__MOCK_SCENARIO = "complete"; // scenario key or ""
});
// 3. Enter central statement
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(scenario.centralStatement);
// 4. Click Analyse
await page.getByRole("button", { name: "Analyse" }).click();
// 5. Wait for loading overlay visible, then hidden
await expect(page.locator('[data-testid="loading-overlay"]')).toBeVisible();
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 30_000 });
// 6. Verify workspace state (terminal card, error panel, or question textarea)
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
});
```
### Playwright Config Reference
See `playwright.config.js` for the full configuration. Key settings:
| Setting | Value | Purpose |
|---|---|---|
| `testDir` | `"./tests/e2e"` | All spec files under this directory |
| `fullyParallel` | `false` | Tests run sequentially to avoid race conditions with mock state |
| `timeout` | `60_000` | 60s per test (long enough for slow delay mode) |
| `expect.timeout` | `15_000` | 15s for individual assertions |
| `screenshot` | `"only-on-failure"` | No disk writes on passing tests |
| `headless` | `true` | CI-safe by default |
| `projects[0].name` | `"mock-mode"` | Distinguishes output in test results directory |
| `retries` | `0` | No retries — failures are deterministic with mock data |
### Adding a New Scenario Fixture
1. Add the scenario definition to `tests/e2e/fixtures/investigation-scenarios.js`:
```js
const newScenario = {
name: "New fixture description",
centralStatement: "The statement to analyse...",
mockMode: "", // "" = default, or set window.__MOCK_SCENARIO in test
turnCount: 2,
answerSequence: [
{ text: "Answer to first question.", expectedHeadingAfterTurn: "Current investigation" },
],
terminalState: null, // or "Investigation complete" / "Current evidence limit reached"
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "after-answer-1", afterAction: "after-answer-1" },
],
};
export const INVESTIGATION_SCENARIOS = [...INVESTIGATION_SCENARIOS, newScenario];
```
2. Add a test case in the appropriate spec file that references the fixture.
### Running Tests During Development
While developing mock responses or UI components, run tests with headed mode:
```bash
# Watch all specs and re-run on file changes
npx playwright test --headed --ui
```
The Playwright Test UI (shown via `--ui`) lets you step through each assertion, inspect the live DOM, and replay failed steps.
@@ -0,0 +1,255 @@
/**
* Investigation scenarios — shared fixture data for all E2E Playwright tests.
*
* Each scenario defines:
* - name / centralStatement (the text entered into the scenario textarea)
* - mockMode (which mock mode to activate: "complete" | "error" | "" )
* - answerSequence [ { text, expectedHeadingAfterTurn } ]
* - expectedTerminalState (final card heading)
* - screenshots [ { label, expectedVisible } ]
*
* The test scripts drive the application through these scenarios using the
* Developer-details scenario selector (UI mock mode), or by setting env vars.
*/
/* ── Shared constants ─────────────────────────────────────── */
const BASE_SCENARIO_TEXT =
"Complaints increased by 35% while production increased by 40%.";
/* ── Scenario definitions ─────────────────────────────────── */
/**
* Multi-turn investigation with at least 2 turns.
* Uses the default sequential-turn mock (no special scenario flag).
*/
const happyPathComparison = {
name: "Happy path — multi-turn comparison",
centralStatement:
"Product A has a 4.2 star average rating while Product B averages 4.6 stars across 10,000+ reviews each.",
mockMode: "", // "" → sequential turns via default mock
turnCount: 3,
answerSequence: [
{
text: "Yes, both are standard 5-star scales.",
expectedHeadingAfterTurn: "Current investigation",
},
{
text: "The gap persists across verified purchase reviews.",
expectedHeadingAfterTurn: "Current investigation",
},
],
terminalState: null, // does not reach a terminal state (unknowns remain)
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "after-first-turn", afterAction: "after-answer-1" },
],
};
/**
* Complete investigation — all unknowns resolved.
* Uses the "complete" mock mode (jumps to terminal after start).
*/
const happyPathComplete = {
name: "Happy path — complete investigation",
centralStatement: BASE_SCENARIO_TEXT,
mockMode: "complete",
turnCount: 1, // start gives turn 0, first update → final state
answerSequence: [
{
text: "The complaint rate fell from 2.0 per 100 to 1.9 per 100 units.",
expectedHeadingAfterTurn: "Investigation complete",
},
],
terminalState: "Investigation complete",
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "terminal-state", afterAction: "after-answer-1" },
],
};
/**
* Evidence-limit state — unknowns remain but no next question.
*/
const evidenceLimit = {
name: "Evidence limit (stuck early)",
centralStatement:
"Should a mid-sized manufacturing company invest in automated quality inspection?",
mockMode: "", // single-turn default scenario
turnCount: 1,
answerSequence: [], // no answers needed — stuck after start
terminalState: "Current evidence limit reached",
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "terminal-state", afterAction: "after-start" },
],
};
/**
* Provider error — all calls return a structured provider error.
*/
const providerError = {
name: "Provider error",
centralStatement: BASE_SCENARIO_TEXT,
mockMode: "error",
turnCount: 0, // start call also errors
answerSequence: [],
terminalState: null,
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "terminal-state", afterAction: "after-start" },
],
};
/**
* Malformed response — the start produces a malformed shape.
* (We simulate this via error mode which skips all structured output.)
*/
const malformedResponse = {
name: "Malformed response",
centralStatement: BASE_SCENARIO_TEXT,
mockMode: "error", // reuses error mode for malformed path
turnCount: 0,
answerSequence: [],
terminalState: null,
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "terminal-state", afterAction: "after-start" },
],
};
/**
* Long investigation — market entry, 5 turns.
*/
const longInvestigation = {
name: "Long investigation — European market entry",
centralStatement:
"Should we enter the European market with our SaaS analytics platform?",
mockMode: "", // sequential turns from scenario data (long array)
turnCount: 5,
answerSequence: [
{ text: "The market is valued at approximately €8B and growing.", expectedHeadingAfterTurn: "Current investigation" },
{ text: "Our platform does not currently support EU data residency.", expectedHeadingAfterTurn: "Current investigation" },
{ text: "Achieving compliance would take 6 months and $500K engineering investment.", expectedHeadingAfterTurn: "Current investigation" },
{ text: "Our real-time collaboration feature has no direct European equivalent.", expectedHeadingAfterTurn: "Investigation complete", isLast: true },
],
terminalState: "Investigation complete",
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "mid-investigation", afterAction: "after-answer-2" },
{ label: "terminal-state", afterAction: "after-answer-4" },
],
};
/**
* Contradiction fixture — two opposing recommendations.
*/
const contradictionFixture = {
name: "Contradiction fixture",
centralStatement:
"Two consultants provided opposite recommendations about whether to outsource IT operations.",
mockMode: "",
turnCount: 2,
answerSequence: [
{ text: "Consultant A focused on cost over 2 years; Consultant B focused on quality over 5+ years.", expectedHeadingAfterTurn: "Current investigation" },
],
terminalState: null,
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "after-answer-1", afterAction: "after-answer-1" },
],
};
/**
* Diagnosis fixture — churn diagnosis.
*/
const diagnosisFixture = {
name: "Diagnosis fixture",
centralStatement:
"Customer churn increased from 2% to 5% monthly over the last quarter.",
mockMode: "",
turnCount: 1,
answerSequence: [],
terminalState: null,
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "terminal-state", afterAction: "after-start" },
],
};
/**
* Comparison fixture — product ratings with 3 turns.
*/
const comparisonFixture = {
name: "Comparison fixture",
centralStatement:
"Product A has a 4.2 star average rating while Product B averages 4.6 stars across 10,000+ reviews each.",
mockMode: "",
turnCount: 3,
answerSequence: [
{ text: "Both use the standard 5-star customer review scale.", expectedHeadingAfterTurn: "Current investigation" },
{ text: "Verified purchase gap remains approximately 0.3 stars.", expectedHeadingAfterTurn: "Current investigation" },
],
terminalState: null,
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "after-answer-1", afterAction: "after-answer-1" },
],
};
/**
* Prioritisation fixture — decision investigation.
*/
const prioritisationFixture = {
name: "Prioritisation fixture (team relocation)",
centralStatement:
"Should I relocate my engineering team from London to Manchester?",
mockMode: "",
turnCount: 2,
answerSequence: [
{ text: "Manchester has a growing tech ecosystem with 500+ roles posted monthly.", expectedHeadingAfterTurn: "Current investigation" },
],
terminalState: null,
screenshots: [
{ label: "initial-state", afterAction: "before-start" },
{ label: "after-answer-1", afterAction: "after-answer-1" },
],
};
/* ── Registry for dynamic test generation ─────────────────── */
export const INVESTIGATION_SCENARIOS = [
happyPathComparison,
happyPathComplete,
evidenceLimit,
providerError,
malformedResponse,
longInvestigation,
contradictionFixture,
diagnosisFixture,
comparisonFixture,
prioritisationFixture,
];
export const SCENARIO_LOOKUP = {};
INVESTIGATION_SCENARIOS.forEach((s) => {
SCENARIO_LOOKUP[s.name] = s;
});
/**
* Helper to get the answer text for a given turn index.
* Returns null if there is no answer for that turn.
*/
export function getAnswerForTurn(scenario, turnIndex) {
return scenario.answerSequence[turnIndex]?.text || null;
}
/**
* Return the list of screenshots in execution order.
*/
export function getScreenshotsOrdered(screenshotList) {
// Preserve insertion order from scenario definition
return screenshotList;
}
export default INVESTIGATION_SCENARIOS;
+368
View File
@@ -0,0 +1,368 @@
/**
* Long investigation E2E tests — multi-turn, contradiction, diagnosis,
* comparison, and prioritisation fixtures.
*
* What these tests validate:
* - Multi-turn flow works correctly for long sequences (5+ turns)
* - History count increases with each answer submission
* - Question text changes after each turn (observable UI transition)
* - Contradiction fixture shows opposing recommendations in history
* - Diagnosis scenario renders correctly with node/edge summary
* - Comparison fixture produces correct side-by-side structure
* - Prioritisation fixture yields ranked options in result card
* - Loading state between each turn is observable (not skipped)
*
* What these tests do NOT validate:
* - Correctness of reasoning, node selection, or question quality
*/
import { test, expect } from "@playwright/test";
import { INVESTIGATION_SCENARIOS, SCENARIO_LOOKUP } from "./fixtures/investigation-scenarios.js";
/* ── Fixture scenarios (from INVESTIGATION_SCENARIOS, not from AVAILABLE_SCENARIOS in the UI)
Note: INVESTIGATION_SCENARIOS has its own labels independent of the UI dropdown
(AVAILABLE_SCENARIOS). The UI dropdown uses keys/labels from lib/mocks/scenarios.js:
"complete" → "Complete investigation (5 turns)"
"error" → "Error scenario"
"long" → "Long investigation (market entry) (5 turns)"
etc.
To use a specific UI-scenario, set window.__MOCK_SCENARIO in the test.
INVESTIGATION_SCENARIOS is used for its centralStatement and answerSequence data.
For example: mockMode="complete" → set __MOCK_SCENARIO="complete".
────────────────────────────────── */
const LONG_INVESTIGATION = SCENARIO_LOOKUP["Long investigation — European market entry"];
const CONTRADICTION = SCENARIO_LOOKUP["Contradiction fixture"];
const DIAGNOSIS = SCENARIO_LOOKUP["Diagnosis fixture"];
const COMPARISON = SCENARIO_LOOKUP["Comparison fixture"];
const PRIORITISATION = SCENARIO_LOOKUP["Prioritisation fixture (team relocation)"];
/* ── Test helpers ─────────────────────────────────────────────── */
async function waitForLoading(page) {
await expect(page.locator('[data-testid="loading-overlay"]')).toBeVisible();
}
async function waitForWorkspaceReady(page, options = {}) {
const { expectLoadingGone = true } = options;
if (expectLoadingGone) {
await page.waitForSelector('[data-testid="loading-overlay"]', { state: "hidden", timeout: 30_000 });
}
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible({ timeout: 15_000 });
}
/**
* Helper to submit an answer and wait for the result.
* Returns { success, hasError } for test assertions.
*/
async function submitAnswer(page, text) {
const textarea = page.locator('textarea[placeholder*=Answer]');
if (!(await textarea.isVisible({ timeout: 3000 }).catch(() => false))) {
return { success: false, hasError: false };
}
await textarea.fill(text);
await page.getByRole("button", { name: "Update situation" }).click();
// Wait for the answer to submit (if text is non-empty)
if (text && text.trim()) {
// After submit, loading appears then workspace updates
await waitForLoading(page);
await waitForWorkspaceReady(page);
return { success: true, hasError: false };
}
return { success: false, hasError: true };
}
/* ═══════════════ Test: long investigation multi-turn ═══════════ */
test("long investigation — European market entry with 4 answer turns", async ({ page }) => {
await page.goto("/");
// Enable mock mode for complete scenario (supports multiple turns)
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "complete";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(LONG_INVESTIGATION.centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
// Initial workspace renders
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// Submit each answer in sequence
const historyCounts = [];
for (let i = 0; i < LONG_INVESTIGATION.answerSequence.length; i++) {
const result = await submitAnswer(page, LONG_INVESTIGATION.answerSequence[i].text);
expect(result.success).toBe(true);
// Record history turn count after each answer
const indicators = page.locator('[data-testid="investigation-turn"]');
const count = await indicators.count();
historyCounts.push(count);
// Each turn's question should be visible (or terminal state)
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible();
}
// History counts should have monotonically increased
for (let i = 1; i < historyCounts.length; i++) {
expect(historyCounts[i]).toBeGreaterThanOrEqual(historyCounts[i - 1]);
}
});
/* ═══════════════ Test: contradiction fixture renders both sides ═ */
test("contradiction fixture — opposing recommendations in investigation card", async ({ page }) => {
await page.goto("/");
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = ""; // use default
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(CONTRADICTION.centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
// Workspace renders with contradiction scenario
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// If there's an answer textarea (next question in contradiction flow), submit it
const result = await submitAnswer(page, CONTRADICTION.answerSequence[0]?.text || "");
if (result.success) {
// Workspace should show updated investigation with contradiction nodes
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible();
// Current understanding summary should be present
const understandingHeading = page.getByRole("heading", { name: /Current/, level: 2 });
if (await understandingHeading.isVisible({ timeout: 5000 })) {
await expect(understandingHeading).toBeVisible();
}
}
// Final workspace should still be visible regardless of whether terminal was reached
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible();
});
/* ═══════════════ Test: diagnosis scenario ══════════════════════ */
test("diagnosis scenario — churn investigation renders correctly", async ({ page }) => {
await page.goto("/");
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(DIAGNOSIS.centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
// Workspace renders with diagnosis nodes and edges
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// Scenario-specific heading should be present
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible();
// The central statement from the diagnosis scenario should appear in the card
const cardText = page.locator('[data-testid="reasoning-workspace"]').locator('p').first();
if (await cardText.isVisible()) {
await expect(cardText).toContainText(/churn/i);
}
});
/* ═══════════════ Test: comparison scenario with side-by-side ══ */
test("comparison scenario — product rating nodes render with edges", async ({ page }) => {
await page.goto("/");
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(COMPARISON.centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
// Workspace renders
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// Submit first answer (comparison fixture has multiple turns)
const result = await submitAnswer(page, COMPARISON.answerSequence[0]?.text || "");
if (result.success) {
// Second turn's answer textarea should appear
const answerTextarea = page.locator('textarea[placeholder*=Answer]');
expect(await answerTextarea.isVisible()).toBe(true);
// Submit second answer
await submitAnswer(page, COMPARISON.answerSequence[1]?.text || "");
// Workspace still visible after multi-turn
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
}
});
/* ═══════════════ Test: prioritisation scenario ═════════════════ */
test("prioritisation scenario — team relocation yields ranked options", async ({ page }) => {
await page.goto("/");
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(PRIORITISATION.centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
// Workspace renders
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// Submit first answer
const result = await submitAnswer(page, PRIORITISATION.answerSequence[0]?.text || "");
if (result.success) {
// Workspace should show updated state with prioritisation nodes
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible();
}
});
/* ═══════════════ Test: rapid multi-turn loading transitions ─── */
test("rapid successive updates maintain correct loading-state cycling", async ({ page }) => {
await page.goto("/");
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(LONG_INVESTIGATION.centralStatement); // use long scenario central statement
await page.getByRole("button", { name: "Analyse" }).click();
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// Submit multiple answers rapidly (all instant mock = no delay)
for (let i = 0; i < LONG_INVESTIGATION.answerSequence.length; i++) {
const ans = LONG_INVESTIGATION.answerSequence[i]?.text;
if (!ans) break;
await submitAnswer(page, ans);
// After each submission, verify workspace stable and no duplicate loading overlays
const overlays = page.locator('[data-testid="loading-overlay"]');
await expect(overlays).toHaveCount(0);
}
});
/* ═══════════════ Test: answer textarea placeholder changes ───── */
test("answer textarea placeholder updates between turns", async ({ page }) => {
await page.goto("/");
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(INVESTIGATION_SCENARIOS[0].centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
// Initial workspace should have an answer textarea with a question-related placeholder
const answerTextarea = page.locator('textarea[placeholder*=Answer]');
await expect(answerTextarea).toBeVisible({ timeout: 10_000 });
// Submit first answer
await submitAnswer(page, INVESTIGATION_SCENARIOS[0].answerSequence[0]?.text || "");
// After update, new question textarea should be present with potentially different placeholder
const updatedTextarea = page.locator('textarea[placeholder*=Answer]');
await expect(updatedTextarea).toBeVisible();
// The workspace card should still display the current investigation context
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible();
});
/* ═══════════════ Test: terminal state with no answer available ─ */
test("terminal state renders without answer textarea", async ({ page }) => {
await page.goto("/");
// Use the "complete" scenario which eventually reaches a terminal (no next question)
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "complete";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(SCENARIO_LOOKUP["Happy path — complete investigation"].centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
// Workspace renders
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// If there's a question (turn 0 gives a question), submit it to reach terminal
const answerTextarea = page.locator('textarea[placeholder*=Answer]');
if (await answerTextarea.isVisible({ timeout: 3000 })) {
await answerTextarea.fill(SCENARIO_LOOKUP["Happy path — complete investigation"].answerSequence[0]?.text || "");
await page.getByRole("button", { name: "Update situation" }).click();
// Wait for terminal state
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// At terminal state, answer textarea should no longer be visible (or disabled)
const hasAnswer = await answerTextarea.isVisible().catch(() => false);
if (hasAnswer) {
const disabledAttr = await answerTextarea.getAttribute("disabled");
expect(disabledAttr).toBeTruthy();
}
}
// Terminal card should be present
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible();
});
+193
View File
@@ -0,0 +1,193 @@
/**
* Recovery-state E2E tests — provider error and malformed response flows.
*
* What these tests validate:
* - Error state renders an error panel with structured message
* - Workspace remains visible alongside the recovery card
* - ProviderUnavailableCard / MalformedResponseCard content visible
* - Session persistence survives a recovery state (recovery key in sessionStorage)
*
* What these tests do NOT validate:
* - Error handling logic correctness (that is unit-level concern)
*/
import { test, expect } from "@playwright/test";
import { INVESTIGATION_SCENARIOS, SCENARIO_LOOKUP } from "./fixtures/investigation-scenarios.js";
const PROVIDER_ERROR = SCENARIO_LOOKUP["Provider error"];
const MALFORMED_RESPONSE = SCENARIO_LOOKUP["Malformed response"];
/* ── Test helpers ─────────────────────────────────────────────── */
async function waitForLoading(page) {
await expect(page.locator('[data-testid="loading-overlay"]')).toBeVisible();
}
async function waitForWorkspaceReady(page, options = {}) {
const { expectLoadingGone = true } = options;
if (expectLoadingGone) {
await page.waitForSelector('[data-testid="loading-overlay"]', { state: "hidden", timeout: 30_000 });
}
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible({ timeout: 15_000 });
}
/* ═══════════════ Test: provider error after start ══════════════ */
test("provider error: analyse returns failure → error panel visible alongside workspace", async ({ page }) => {
await page.goto("/");
// Enable mock mode in error scenario
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "error";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(PROVIDER_ERROR.centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
// Loading overlay appears briefly then disappears
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
// Workspace should still be visible (even with error result)
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// Error panel with structured error message
const errorPanel = page.locator('div[class*="border-red"][class*="bg-red"]');
await expect(errorPanel).toBeVisible();
await expect(errorPanel.locator('div[class*="text-sm"]').first()).toContainText(/error/i);
});
/* ═══════════════ Test: provider error during update ════════════ */
test("provider error during update turn: error panel appears after answer submit", async ({ page }) => {
await page.goto("/");
// Use a non-error scenario first to get into success state, then switch to error for update
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = ""; // normal default for start
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(INVESTIGATION_SCENARIOS[0].centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
await waitForLoading(page);
await waitForWorkspaceReady(page);
// Switch to error scenario for the update call
await page.evaluate(() => {
window.__MOCK_SCENARIO = "error";
});
// Submit an answer to trigger the update call (which will now get error)
const answerTextarea = page.locator('textarea[placeholder*=Answer]');
if (await answerTextarea.isVisible({ timeout: 3000 })) {
await answerTextarea.fill("Some answer that should fail");
await page.getByRole("button", { name: "Update situation" }).click();
// Verify error panel appears after the update call errors
const errorPanel = page.locator('div[class*="border-red"][class*="bg-red"]');
await expect(errorPanel).toBeVisible({ timeout: 10_000 });
}
});
/* ═══════════════ Test: malformed response during start ═════════ */
test("malformed response during start: error state visible, workspace still rendered", async ({ page }) => {
await page.goto("/");
// Malformed responses come through the error mock path (same as provider error)
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "error";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(MALFORMED_RESPONSE.centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
// Loading then workspace
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible();
// Error text present in the red panel
const errorText = page.locator('div[class*="border-red"][class*="text-sm"]').first();
await expect(errorText).toHaveText(/error|failed/i);
});
/* ═══════════════ Test: recovery state persists in session ══════ */
test("recovery state persists via sessionStorage after error response", async ({ page }) => {
await page.goto("/");
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "error";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(PROVIDER_ERROR.centralStatement);
await page.getByRole("button", { name: "Analyse" }).click();
// Wait for workspace to render with error state
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// Verify sessionStorage has the session key with situationGraph from error response
const sessionRaw = await page.evaluate(() => {
const raw = sessionStorage.getItem("confidence-engine-session");
return raw ? JSON.parse(raw) : null;
});
// Session should exist (even in error recovery path, the scenario text is persisted)
expect(sessionRaw).not.toBeNull();
});
/* ═══════════════ Test: multiple error transitions ══════════════ */
test("multiple successive errors maintain UI state without crash", async ({ page }) => {
await page.goto("/");
// Enable mock mode with error scenario
await page.evaluate(() => {
window.__MOCK_ENABLED = true;
window.__MOCK_DELAY = "instant";
window.__MOCK_SCENARIO = "error";
});
const textarea = page.locator("textarea[placeholder*=Describe]");
await textarea.fill(PROVIDER_ERROR.centralStatement);
// First analyse call
await page.getByRole("button", { name: "Analyse" }).click();
await waitForLoading(page);
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
// If there's an answer textarea, try another update (also errors)
const answerTextarea = page.locator('textarea[placeholder*=Answer]');
if (await answerTextarea.isVisible({ timeout: 2000 })) {
await answerTextarea.fill("Second attempt");
await page.getByRole("button", { name: "Update situation" }).click();
// Error panel should still be visible (no crash, no layout shift)
const errorPanel = page.locator('div[class*="border-red"]');
await expect(errorPanel).toBeVisible({ timeout: 10_000 });
}
// Workspace should remain stable — no errors thrown
const workspace = page.locator('[data-testid="reasoning-workspace"]');
await expect(workspace).toBeVisible();
});