feat: add mock-mode docs and additional e2e tests (long investigation, recovery states)

2026-08-05 08:38:44 +01:00
parent 61210c1200
commit 592962325d
4 changed files with 988 additions and 0 deletions
@@ -117,3 +117,175 @@ Each turn snapshot (inline in mock-client.js) contains:
 ```

 The situationGraph returned by the fixtures matches the real route response shape exactly, ensuring the UI renders identically.
+
+---
+
+## Playwright E2E Scenario Automation
+
+### Purpose
+
+Playwright tests replay main mock investigation journeys to validate:
+- **UI layout transitions** — idle → loading → success/error terminal/recovery states
+- **State machine behaviour** — loading overlays, question changes, history rendering
+- **Terminal states** — CompletionCard (genuine completion) and EvidenceLimitCard
+- **Recovery states** — ProviderUnavailableCard, MalformedResponseCard content visibility
+- **History rendering** — turn count increases with each answer submission
+
+These tests do **NOT** validate reasoning correctness, candidate selection quality, confidence accuracy, or question quality. They are purely UI-layout and state-transition checks.
+
+### Quick Start
+
+```bash
+# Enable mock mode for all Playwright tests
+export NEXT_PUBLIC_CONFIDENCE_ENGINE_MOCKS=true
+
+# Run all E2E tests (headless, screenshot on failure only)
+npx playwright test
+
+# Run a single spec file
+npx playwright test tests/e2e/happy-path.spec.js
+
+# Run in headed mode for debugging
+npx playwright test --headed
+
+# Show browser console logs
+PWDEBUG=1 npx playwright test
+```
+
+> **Note:** Playwright is already listed as a devDependency (v1.62.1). No additional package installation is needed.
+
+### Screenshot Location
+
+Screenshots are automatically saved to `test-results/` in the project root when a test **fails**. They are named by test title and project. Pass-by-default tests produce no screenshots.
+
+To capture screenshots for every scenario (useful for visual regression), run with the `--retries=0` flag and inspect `test-results/`:
+
+```bash
+npx playwright test --retries=0
+ls test-results/
+# → happy-path-mock-mode/
+# → recovery-states-mock-mode/
+# → long-investigation-mock-mode/
+```
+
+### Scenario Fixtures
+
+All scenario data lives in `tests/e2e/fixtures/investigation-scenarios.js`. This file separates **content** (central statements, answer sequences, expected headings) from **test logic**. It is shared across all four spec files:
+
+| Fixture | Central Statement | Mock Mode | Turns | Terminal State |
+|---|---|---|---|---|
+| Happy path — multi-turn comparison | Product A vs B ratings | default (env var) | 3 turns | No terminal |
+| Happy path — complete investigation | Complaints + production | `complete` | 1 update turn | Investigation complete |
+| Evidence limit (stuck early) | Manufacturing quality | default (env var) | Start only | Evidence limit reached |
+| Provider error | Complaints + production | `error` | Start only | Error panel visible |
+| Malformed response | Complaints + production | `error` | Start only | Error panel visible |
+| Long investigation — European market entry | SaaS market entry | default (env var) | 4 answer turns | Investigation complete |
+| Contradiction fixture | Outsourcing consultants | default (env var) | 1 update turn | No terminal |
+| Diagnosis fixture | Customer churn | default (env var) | Start only | No terminal |
+| Comparison fixture | Product A vs B ratings | default (env var) | 2 update turns | No terminal |
+| Prioritisation fixture (team relocation) | London → Manchester | default (env var) | 1 update turn | No terminal |
+
+### What Each Spec File Validates
+
+| Spec file | Scenario coverage | Key checks |
+|---|---|---|
+| `happy-path.spec.js` | multi-turn, complete, evidence-limit, reset flow, loading transitions | Idle→loading→success transitions, CompletionCard/EvidenceLimitCard visible, history turn count increases, reset button works |
+| `recovery-states.spec.js` | provider error (start + update), malformed response | Error panels visible, workspace stays rendered, session persists after error, multiple successive errors don't crash UI |
+| `long-investigation.spec.js` | long 5-turn, contradiction, diagnosis, comparison, prioritisation | Multi-turn loading state cycling, answer placeholders update, terminal renders without answer textarea, workspace stability across rapid updates |
+
+### Accessible Selectors
+
+All tests use role-based selectors (`getByRole`, `getByLabel`) and attribute selectors — never CSS class names:
+
+```js
+// ✅ Role-based (good)
+page.getByRole("button", { name: "Analyse" })
+page.getByRole("textbox").first()  // textarea with placeholder="Describe..."
+page.getByRole("heading", { name: /Investigation complete/ })
+
+// ❌ CSS class selectors (brittle) — NEVER USED
+page.locator(".bg-gray-900 .px-6")  // fragile to theme changes
+```
+
+### Test Structure Pattern
+
+Each test follows a consistent flow:
+
+```js
+test("scenario name", async ({ page }) => {
+  // 1. Navigate
+  await page.goto("/");
+
+  // 2. Enable mock mode (set window globals)
+  await page.evaluate(() => {
+    window.__MOCK_ENABLED = true;
+    window.__MOCK_DELAY = "instant";      // instant | normal | slow
+    window.__MOCK_SCENARIO = "complete";   // scenario key or ""
+  });
+
+  // 3. Enter central statement
+  const textarea = page.locator("textarea[placeholder*=Describe]");
+  await textarea.fill(scenario.centralStatement);
+
+  // 4. Click Analyse
+  await page.getByRole("button", { name: "Analyse" }).click();
+
+  // 5. Wait for loading overlay visible, then hidden
+  await expect(page.locator('[data-testid="loading-overlay"]')).toBeVisible();
+  await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 30_000 });
+
+  // 6. Verify workspace state (terminal card, error panel, or question textarea)
+  await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
+});
+```
+
+### Playwright Config Reference
+
+See `playwright.config.js` for the full configuration. Key settings:
+
+| Setting | Value | Purpose |
+|---|---|---|
+| `testDir` | `"./tests/e2e"` | All spec files under this directory |
+| `fullyParallel` | `false` | Tests run sequentially to avoid race conditions with mock state |
+| `timeout` | `60_000` | 60s per test (long enough for slow delay mode) |
+| `expect.timeout` | `15_000` | 15s for individual assertions |
+| `screenshot` | `"only-on-failure"` | No disk writes on passing tests |
+| `headless` | `true` | CI-safe by default |
+| `projects[0].name` | `"mock-mode"` | Distinguishes output in test results directory |
+| `retries` | `0` | No retries — failures are deterministic with mock data |
+
+### Adding a New Scenario Fixture
+
+1. Add the scenario definition to `tests/e2e/fixtures/investigation-scenarios.js`:
+
+```js
+const newScenario = {
+  name: "New fixture description",
+  centralStatement: "The statement to analyse...",
+  mockMode: "",                     // "" = default, or set window.__MOCK_SCENARIO in test
+  turnCount: 2,
+  answerSequence: [
+    { text: "Answer to first question.", expectedHeadingAfterTurn: "Current investigation" },
+  ],
+  terminalState: null,              // or "Investigation complete" / "Current evidence limit reached"
+  screenshots: [
+    { label: "initial-state", afterAction: "before-start" },
+    { label: "after-answer-1", afterAction: "after-answer-1" },
+  ],
+};
+
+export const INVESTIGATION_SCENARIOS = [...INVESTIGATION_SCENARIOS, newScenario];
+```
+
+2. Add a test case in the appropriate spec file that references the fixture.
+
+### Running Tests During Development
+
+While developing mock responses or UI components, run tests with headed mode:
+
+```bash
+# Watch all specs and re-run on file changes
+npx playwright test --headed --ui
+```
+
+The Playwright Test UI (shown via `--ui`) lets you step through each assertion, inspect the live DOM, and replay failed steps.