feat: add mock-mode docs and additional e2e tests (long investigation, recovery states)
This commit is contained in:
@@ -117,3 +117,175 @@ Each turn snapshot (inline in mock-client.js) contains:
|
||||
```
|
||||
|
||||
The situationGraph returned by the fixtures matches the real route response shape exactly, ensuring the UI renders identically.
|
||||
|
||||
---
|
||||
|
||||
## Playwright E2E Scenario Automation
|
||||
|
||||
### Purpose
|
||||
|
||||
Playwright tests replay main mock investigation journeys to validate:
|
||||
- **UI layout transitions** — idle → loading → success/error terminal/recovery states
|
||||
- **State machine behaviour** — loading overlays, question changes, history rendering
|
||||
- **Terminal states** — CompletionCard (genuine completion) and EvidenceLimitCard
|
||||
- **Recovery states** — ProviderUnavailableCard, MalformedResponseCard content visibility
|
||||
- **History rendering** — turn count increases with each answer submission
|
||||
|
||||
These tests do **NOT** validate reasoning correctness, candidate selection quality, confidence accuracy, or question quality. They are purely UI-layout and state-transition checks.
|
||||
|
||||
### Quick Start
|
||||
|
||||
```bash
|
||||
# Enable mock mode for all Playwright tests
|
||||
export NEXT_PUBLIC_CONFIDENCE_ENGINE_MOCKS=true
|
||||
|
||||
# Run all E2E tests (headless, screenshot on failure only)
|
||||
npx playwright test
|
||||
|
||||
# Run a single spec file
|
||||
npx playwright test tests/e2e/happy-path.spec.js
|
||||
|
||||
# Run in headed mode for debugging
|
||||
npx playwright test --headed
|
||||
|
||||
# Show browser console logs
|
||||
PWDEBUG=1 npx playwright test
|
||||
```
|
||||
|
||||
> **Note:** Playwright is already listed as a devDependency (v1.62.1). No additional package installation is needed.
|
||||
|
||||
### Screenshot Location
|
||||
|
||||
Screenshots are automatically saved to `test-results/` in the project root when a test **fails**. They are named by test title and project. Pass-by-default tests produce no screenshots.
|
||||
|
||||
To capture screenshots for every scenario (useful for visual regression), run with the `--retries=0` flag and inspect `test-results/`:
|
||||
|
||||
```bash
|
||||
npx playwright test --retries=0
|
||||
ls test-results/
|
||||
# → happy-path-mock-mode/
|
||||
# → recovery-states-mock-mode/
|
||||
# → long-investigation-mock-mode/
|
||||
```
|
||||
|
||||
### Scenario Fixtures
|
||||
|
||||
All scenario data lives in `tests/e2e/fixtures/investigation-scenarios.js`. This file separates **content** (central statements, answer sequences, expected headings) from **test logic**. It is shared across all four spec files:
|
||||
|
||||
| Fixture | Central Statement | Mock Mode | Turns | Terminal State |
|
||||
|---|---|---|---|---|
|
||||
| Happy path — multi-turn comparison | Product A vs B ratings | default (env var) | 3 turns | No terminal |
|
||||
| Happy path — complete investigation | Complaints + production | `complete` | 1 update turn | Investigation complete |
|
||||
| Evidence limit (stuck early) | Manufacturing quality | default (env var) | Start only | Evidence limit reached |
|
||||
| Provider error | Complaints + production | `error` | Start only | Error panel visible |
|
||||
| Malformed response | Complaints + production | `error` | Start only | Error panel visible |
|
||||
| Long investigation — European market entry | SaaS market entry | default (env var) | 4 answer turns | Investigation complete |
|
||||
| Contradiction fixture | Outsourcing consultants | default (env var) | 1 update turn | No terminal |
|
||||
| Diagnosis fixture | Customer churn | default (env var) | Start only | No terminal |
|
||||
| Comparison fixture | Product A vs B ratings | default (env var) | 2 update turns | No terminal |
|
||||
| Prioritisation fixture (team relocation) | London → Manchester | default (env var) | 1 update turn | No terminal |
|
||||
|
||||
### What Each Spec File Validates
|
||||
|
||||
| Spec file | Scenario coverage | Key checks |
|
||||
|---|---|---|
|
||||
| `happy-path.spec.js` | multi-turn, complete, evidence-limit, reset flow, loading transitions | Idle→loading→success transitions, CompletionCard/EvidenceLimitCard visible, history turn count increases, reset button works |
|
||||
| `recovery-states.spec.js` | provider error (start + update), malformed response | Error panels visible, workspace stays rendered, session persists after error, multiple successive errors don't crash UI |
|
||||
| `long-investigation.spec.js` | long 5-turn, contradiction, diagnosis, comparison, prioritisation | Multi-turn loading state cycling, answer placeholders update, terminal renders without answer textarea, workspace stability across rapid updates |
|
||||
|
||||
### Accessible Selectors
|
||||
|
||||
All tests use role-based selectors (`getByRole`, `getByLabel`) and attribute selectors — never CSS class names:
|
||||
|
||||
```js
|
||||
// ✅ Role-based (good)
|
||||
page.getByRole("button", { name: "Analyse" })
|
||||
page.getByRole("textbox").first() // textarea with placeholder="Describe..."
|
||||
page.getByRole("heading", { name: /Investigation complete/ })
|
||||
|
||||
// ❌ CSS class selectors (brittle) — NEVER USED
|
||||
page.locator(".bg-gray-900 .px-6") // fragile to theme changes
|
||||
```
|
||||
|
||||
### Test Structure Pattern
|
||||
|
||||
Each test follows a consistent flow:
|
||||
|
||||
```js
|
||||
test("scenario name", async ({ page }) => {
|
||||
// 1. Navigate
|
||||
await page.goto("/");
|
||||
|
||||
// 2. Enable mock mode (set window globals)
|
||||
await page.evaluate(() => {
|
||||
window.__MOCK_ENABLED = true;
|
||||
window.__MOCK_DELAY = "instant"; // instant | normal | slow
|
||||
window.__MOCK_SCENARIO = "complete"; // scenario key or ""
|
||||
});
|
||||
|
||||
// 3. Enter central statement
|
||||
const textarea = page.locator("textarea[placeholder*=Describe]");
|
||||
await textarea.fill(scenario.centralStatement);
|
||||
|
||||
// 4. Click Analyse
|
||||
await page.getByRole("button", { name: "Analyse" }).click();
|
||||
|
||||
// 5. Wait for loading overlay visible, then hidden
|
||||
await expect(page.locator('[data-testid="loading-overlay"]')).toBeVisible();
|
||||
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 30_000 });
|
||||
|
||||
// 6. Verify workspace state (terminal card, error panel, or question textarea)
|
||||
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
|
||||
});
|
||||
```
|
||||
|
||||
### Playwright Config Reference
|
||||
|
||||
See `playwright.config.js` for the full configuration. Key settings:
|
||||
|
||||
| Setting | Value | Purpose |
|
||||
|---|---|---|
|
||||
| `testDir` | `"./tests/e2e"` | All spec files under this directory |
|
||||
| `fullyParallel` | `false` | Tests run sequentially to avoid race conditions with mock state |
|
||||
| `timeout` | `60_000` | 60s per test (long enough for slow delay mode) |
|
||||
| `expect.timeout` | `15_000` | 15s for individual assertions |
|
||||
| `screenshot` | `"only-on-failure"` | No disk writes on passing tests |
|
||||
| `headless` | `true` | CI-safe by default |
|
||||
| `projects[0].name` | `"mock-mode"` | Distinguishes output in test results directory |
|
||||
| `retries` | `0` | No retries — failures are deterministic with mock data |
|
||||
|
||||
### Adding a New Scenario Fixture
|
||||
|
||||
1. Add the scenario definition to `tests/e2e/fixtures/investigation-scenarios.js`:
|
||||
|
||||
```js
|
||||
const newScenario = {
|
||||
name: "New fixture description",
|
||||
centralStatement: "The statement to analyse...",
|
||||
mockMode: "", // "" = default, or set window.__MOCK_SCENARIO in test
|
||||
turnCount: 2,
|
||||
answerSequence: [
|
||||
{ text: "Answer to first question.", expectedHeadingAfterTurn: "Current investigation" },
|
||||
],
|
||||
terminalState: null, // or "Investigation complete" / "Current evidence limit reached"
|
||||
screenshots: [
|
||||
{ label: "initial-state", afterAction: "before-start" },
|
||||
{ label: "after-answer-1", afterAction: "after-answer-1" },
|
||||
],
|
||||
};
|
||||
|
||||
export const INVESTIGATION_SCENARIOS = [...INVESTIGATION_SCENARIOS, newScenario];
|
||||
```
|
||||
|
||||
2. Add a test case in the appropriate spec file that references the fixture.
|
||||
|
||||
### Running Tests During Development
|
||||
|
||||
While developing mock responses or UI components, run tests with headed mode:
|
||||
|
||||
```bash
|
||||
# Watch all specs and re-run on file changes
|
||||
npx playwright test --headed --ui
|
||||
```
|
||||
|
||||
The Playwright Test UI (shown via `--ui`) lets you step through each assertion, inspect the live DOM, and replay failed steps.
|
||||
|
||||
@@ -0,0 +1,255 @@
|
||||
/**
|
||||
* Investigation scenarios — shared fixture data for all E2E Playwright tests.
|
||||
*
|
||||
* Each scenario defines:
|
||||
* - name / centralStatement (the text entered into the scenario textarea)
|
||||
* - mockMode (which mock mode to activate: "complete" | "error" | "" )
|
||||
* - answerSequence [ { text, expectedHeadingAfterTurn } ]
|
||||
* - expectedTerminalState (final card heading)
|
||||
* - screenshots [ { label, expectedVisible } ]
|
||||
*
|
||||
* The test scripts drive the application through these scenarios using the
|
||||
* Developer-details scenario selector (UI mock mode), or by setting env vars.
|
||||
*/
|
||||
|
||||
/* ── Shared constants ─────────────────────────────────────── */
|
||||
|
||||
const BASE_SCENARIO_TEXT =
|
||||
"Complaints increased by 35% while production increased by 40%.";
|
||||
|
||||
/* ── Scenario definitions ─────────────────────────────────── */
|
||||
|
||||
/**
|
||||
* Multi-turn investigation with at least 2 turns.
|
||||
* Uses the default sequential-turn mock (no special scenario flag).
|
||||
*/
|
||||
const happyPathComparison = {
|
||||
name: "Happy path — multi-turn comparison",
|
||||
centralStatement:
|
||||
"Product A has a 4.2 star average rating while Product B averages 4.6 stars across 10,000+ reviews each.",
|
||||
mockMode: "", // "" → sequential turns via default mock
|
||||
turnCount: 3,
|
||||
answerSequence: [
|
||||
{
|
||||
text: "Yes, both are standard 5-star scales.",
|
||||
expectedHeadingAfterTurn: "Current investigation",
|
||||
},
|
||||
{
|
||||
text: "The gap persists across verified purchase reviews.",
|
||||
expectedHeadingAfterTurn: "Current investigation",
|
||||
},
|
||||
],
|
||||
terminalState: null, // does not reach a terminal state (unknowns remain)
|
||||
screenshots: [
|
||||
{ label: "initial-state", afterAction: "before-start" },
|
||||
{ label: "after-first-turn", afterAction: "after-answer-1" },
|
||||
],
|
||||
};
|
||||
|
||||
/**
|
||||
* Complete investigation — all unknowns resolved.
|
||||
* Uses the "complete" mock mode (jumps to terminal after start).
|
||||
*/
|
||||
const happyPathComplete = {
|
||||
name: "Happy path — complete investigation",
|
||||
centralStatement: BASE_SCENARIO_TEXT,
|
||||
mockMode: "complete",
|
||||
turnCount: 1, // start gives turn 0, first update → final state
|
||||
answerSequence: [
|
||||
{
|
||||
text: "The complaint rate fell from 2.0 per 100 to 1.9 per 100 units.",
|
||||
expectedHeadingAfterTurn: "Investigation complete",
|
||||
},
|
||||
],
|
||||
terminalState: "Investigation complete",
|
||||
screenshots: [
|
||||
{ label: "initial-state", afterAction: "before-start" },
|
||||
{ label: "terminal-state", afterAction: "after-answer-1" },
|
||||
],
|
||||
};
|
||||
|
||||
/**
|
||||
* Evidence-limit state — unknowns remain but no next question.
|
||||
*/
|
||||
const evidenceLimit = {
|
||||
name: "Evidence limit (stuck early)",
|
||||
centralStatement:
|
||||
"Should a mid-sized manufacturing company invest in automated quality inspection?",
|
||||
mockMode: "", // single-turn default scenario
|
||||
turnCount: 1,
|
||||
answerSequence: [], // no answers needed — stuck after start
|
||||
terminalState: "Current evidence limit reached",
|
||||
screenshots: [
|
||||
{ label: "initial-state", afterAction: "before-start" },
|
||||
{ label: "terminal-state", afterAction: "after-start" },
|
||||
],
|
||||
};
|
||||
|
||||
/**
|
||||
* Provider error — all calls return a structured provider error.
|
||||
*/
|
||||
const providerError = {
|
||||
name: "Provider error",
|
||||
centralStatement: BASE_SCENARIO_TEXT,
|
||||
mockMode: "error",
|
||||
turnCount: 0, // start call also errors
|
||||
answerSequence: [],
|
||||
terminalState: null,
|
||||
screenshots: [
|
||||
{ label: "initial-state", afterAction: "before-start" },
|
||||
{ label: "terminal-state", afterAction: "after-start" },
|
||||
],
|
||||
};
|
||||
|
||||
/**
|
||||
* Malformed response — the start produces a malformed shape.
|
||||
* (We simulate this via error mode which skips all structured output.)
|
||||
*/
|
||||
const malformedResponse = {
|
||||
name: "Malformed response",
|
||||
centralStatement: BASE_SCENARIO_TEXT,
|
||||
mockMode: "error", // reuses error mode for malformed path
|
||||
turnCount: 0,
|
||||
answerSequence: [],
|
||||
terminalState: null,
|
||||
screenshots: [
|
||||
{ label: "initial-state", afterAction: "before-start" },
|
||||
{ label: "terminal-state", afterAction: "after-start" },
|
||||
],
|
||||
};
|
||||
|
||||
/**
|
||||
* Long investigation — market entry, 5 turns.
|
||||
*/
|
||||
const longInvestigation = {
|
||||
name: "Long investigation — European market entry",
|
||||
centralStatement:
|
||||
"Should we enter the European market with our SaaS analytics platform?",
|
||||
mockMode: "", // sequential turns from scenario data (long array)
|
||||
turnCount: 5,
|
||||
answerSequence: [
|
||||
{ text: "The market is valued at approximately €8B and growing.", expectedHeadingAfterTurn: "Current investigation" },
|
||||
{ text: "Our platform does not currently support EU data residency.", expectedHeadingAfterTurn: "Current investigation" },
|
||||
{ text: "Achieving compliance would take 6 months and $500K engineering investment.", expectedHeadingAfterTurn: "Current investigation" },
|
||||
{ text: "Our real-time collaboration feature has no direct European equivalent.", expectedHeadingAfterTurn: "Investigation complete", isLast: true },
|
||||
],
|
||||
terminalState: "Investigation complete",
|
||||
screenshots: [
|
||||
{ label: "initial-state", afterAction: "before-start" },
|
||||
{ label: "mid-investigation", afterAction: "after-answer-2" },
|
||||
{ label: "terminal-state", afterAction: "after-answer-4" },
|
||||
],
|
||||
};
|
||||
|
||||
/**
|
||||
* Contradiction fixture — two opposing recommendations.
|
||||
*/
|
||||
const contradictionFixture = {
|
||||
name: "Contradiction fixture",
|
||||
centralStatement:
|
||||
"Two consultants provided opposite recommendations about whether to outsource IT operations.",
|
||||
mockMode: "",
|
||||
turnCount: 2,
|
||||
answerSequence: [
|
||||
{ text: "Consultant A focused on cost over 2 years; Consultant B focused on quality over 5+ years.", expectedHeadingAfterTurn: "Current investigation" },
|
||||
],
|
||||
terminalState: null,
|
||||
screenshots: [
|
||||
{ label: "initial-state", afterAction: "before-start" },
|
||||
{ label: "after-answer-1", afterAction: "after-answer-1" },
|
||||
],
|
||||
};
|
||||
|
||||
/**
|
||||
* Diagnosis fixture — churn diagnosis.
|
||||
*/
|
||||
const diagnosisFixture = {
|
||||
name: "Diagnosis fixture",
|
||||
centralStatement:
|
||||
"Customer churn increased from 2% to 5% monthly over the last quarter.",
|
||||
mockMode: "",
|
||||
turnCount: 1,
|
||||
answerSequence: [],
|
||||
terminalState: null,
|
||||
screenshots: [
|
||||
{ label: "initial-state", afterAction: "before-start" },
|
||||
{ label: "terminal-state", afterAction: "after-start" },
|
||||
],
|
||||
};
|
||||
|
||||
/**
|
||||
* Comparison fixture — product ratings with 3 turns.
|
||||
*/
|
||||
const comparisonFixture = {
|
||||
name: "Comparison fixture",
|
||||
centralStatement:
|
||||
"Product A has a 4.2 star average rating while Product B averages 4.6 stars across 10,000+ reviews each.",
|
||||
mockMode: "",
|
||||
turnCount: 3,
|
||||
answerSequence: [
|
||||
{ text: "Both use the standard 5-star customer review scale.", expectedHeadingAfterTurn: "Current investigation" },
|
||||
{ text: "Verified purchase gap remains approximately 0.3 stars.", expectedHeadingAfterTurn: "Current investigation" },
|
||||
],
|
||||
terminalState: null,
|
||||
screenshots: [
|
||||
{ label: "initial-state", afterAction: "before-start" },
|
||||
{ label: "after-answer-1", afterAction: "after-answer-1" },
|
||||
],
|
||||
};
|
||||
|
||||
/**
|
||||
* Prioritisation fixture — decision investigation.
|
||||
*/
|
||||
const prioritisationFixture = {
|
||||
name: "Prioritisation fixture (team relocation)",
|
||||
centralStatement:
|
||||
"Should I relocate my engineering team from London to Manchester?",
|
||||
mockMode: "",
|
||||
turnCount: 2,
|
||||
answerSequence: [
|
||||
{ text: "Manchester has a growing tech ecosystem with 500+ roles posted monthly.", expectedHeadingAfterTurn: "Current investigation" },
|
||||
],
|
||||
terminalState: null,
|
||||
screenshots: [
|
||||
{ label: "initial-state", afterAction: "before-start" },
|
||||
{ label: "after-answer-1", afterAction: "after-answer-1" },
|
||||
],
|
||||
};
|
||||
|
||||
/* ── Registry for dynamic test generation ─────────────────── */
|
||||
|
||||
export const INVESTIGATION_SCENARIOS = [
|
||||
happyPathComparison,
|
||||
happyPathComplete,
|
||||
evidenceLimit,
|
||||
providerError,
|
||||
malformedResponse,
|
||||
longInvestigation,
|
||||
contradictionFixture,
|
||||
diagnosisFixture,
|
||||
comparisonFixture,
|
||||
prioritisationFixture,
|
||||
];
|
||||
|
||||
export const SCENARIO_LOOKUP = {};
|
||||
INVESTIGATION_SCENARIOS.forEach((s) => {
|
||||
SCENARIO_LOOKUP[s.name] = s;
|
||||
});
|
||||
|
||||
/**
|
||||
* Helper to get the answer text for a given turn index.
|
||||
* Returns null if there is no answer for that turn.
|
||||
*/
|
||||
export function getAnswerForTurn(scenario, turnIndex) {
|
||||
return scenario.answerSequence[turnIndex]?.text || null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Return the list of screenshots in execution order.
|
||||
*/
|
||||
export function getScreenshotsOrdered(screenshotList) {
|
||||
// Preserve insertion order from scenario definition
|
||||
return screenshotList;
|
||||
}
|
||||
|
||||
export default INVESTIGATION_SCENARIOS;
|
||||
@@ -0,0 +1,368 @@
|
||||
/**
|
||||
* Long investigation E2E tests — multi-turn, contradiction, diagnosis,
|
||||
* comparison, and prioritisation fixtures.
|
||||
*
|
||||
* What these tests validate:
|
||||
* - Multi-turn flow works correctly for long sequences (5+ turns)
|
||||
* - History count increases with each answer submission
|
||||
* - Question text changes after each turn (observable UI transition)
|
||||
* - Contradiction fixture shows opposing recommendations in history
|
||||
* - Diagnosis scenario renders correctly with node/edge summary
|
||||
* - Comparison fixture produces correct side-by-side structure
|
||||
* - Prioritisation fixture yields ranked options in result card
|
||||
* - Loading state between each turn is observable (not skipped)
|
||||
*
|
||||
* What these tests do NOT validate:
|
||||
* - Correctness of reasoning, node selection, or question quality
|
||||
*/
|
||||
|
||||
import { test, expect } from "@playwright/test";
|
||||
import { INVESTIGATION_SCENARIOS, SCENARIO_LOOKUP } from "./fixtures/investigation-scenarios.js";
|
||||
|
||||
/* ── Fixture scenarios (from INVESTIGATION_SCENARIOS, not from AVAILABLE_SCENARIOS in the UI)
|
||||
|
||||
Note: INVESTIGATION_SCENARIOS has its own labels independent of the UI dropdown
|
||||
(AVAILABLE_SCENARIOS). The UI dropdown uses keys/labels from lib/mocks/scenarios.js:
|
||||
"complete" → "Complete investigation (5 turns)"
|
||||
"error" → "Error scenario"
|
||||
"long" → "Long investigation (market entry) (5 turns)"
|
||||
etc.
|
||||
|
||||
To use a specific UI-scenario, set window.__MOCK_SCENARIO in the test.
|
||||
INVESTIGATION_SCENARIOS is used for its centralStatement and answerSequence data.
|
||||
For example: mockMode="complete" → set __MOCK_SCENARIO="complete".
|
||||
────────────────────────────────── */
|
||||
|
||||
const LONG_INVESTIGATION = SCENARIO_LOOKUP["Long investigation — European market entry"];
|
||||
const CONTRADICTION = SCENARIO_LOOKUP["Contradiction fixture"];
|
||||
const DIAGNOSIS = SCENARIO_LOOKUP["Diagnosis fixture"];
|
||||
const COMPARISON = SCENARIO_LOOKUP["Comparison fixture"];
|
||||
const PRIORITISATION = SCENARIO_LOOKUP["Prioritisation fixture (team relocation)"];
|
||||
|
||||
/* ── Test helpers ─────────────────────────────────────────────── */
|
||||
|
||||
async function waitForLoading(page) {
|
||||
await expect(page.locator('[data-testid="loading-overlay"]')).toBeVisible();
|
||||
}
|
||||
|
||||
async function waitForWorkspaceReady(page, options = {}) {
|
||||
const { expectLoadingGone = true } = options;
|
||||
if (expectLoadingGone) {
|
||||
await page.waitForSelector('[data-testid="loading-overlay"]', { state: "hidden", timeout: 30_000 });
|
||||
}
|
||||
const workspace = page.locator('[data-testid="reasoning-workspace"]');
|
||||
await expect(workspace).toBeVisible({ timeout: 15_000 });
|
||||
}
|
||||
|
||||
/**
|
||||
* Helper to submit an answer and wait for the result.
|
||||
* Returns { success, hasError } for test assertions.
|
||||
*/
|
||||
async function submitAnswer(page, text) {
|
||||
const textarea = page.locator('textarea[placeholder*=Answer]');
|
||||
if (!(await textarea.isVisible({ timeout: 3000 }).catch(() => false))) {
|
||||
return { success: false, hasError: false };
|
||||
}
|
||||
|
||||
await textarea.fill(text);
|
||||
await page.getByRole("button", { name: "Update situation" }).click();
|
||||
|
||||
// Wait for the answer to submit (if text is non-empty)
|
||||
if (text && text.trim()) {
|
||||
// After submit, loading appears then workspace updates
|
||||
await waitForLoading(page);
|
||||
await waitForWorkspaceReady(page);
|
||||
return { success: true, hasError: false };
|
||||
}
|
||||
|
||||
return { success: false, hasError: true };
|
||||
}
|
||||
|
||||
/* ═══════════════ Test: long investigation multi-turn ═══════════ */
|
||||
|
||||
test("long investigation — European market entry with 4 answer turns", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
|
||||
// Enable mock mode for complete scenario (supports multiple turns)
|
||||
await page.evaluate(() => {
|
||||
window.__MOCK_ENABLED = true;
|
||||
window.__MOCK_DELAY = "instant";
|
||||
window.__MOCK_SCENARIO = "complete";
|
||||
});
|
||||
|
||||
const textarea = page.locator("textarea[placeholder*=Describe]");
|
||||
await textarea.fill(LONG_INVESTIGATION.centralStatement);
|
||||
await page.getByRole("button", { name: "Analyse" }).click();
|
||||
|
||||
// Initial workspace renders
|
||||
await waitForLoading(page);
|
||||
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
|
||||
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
|
||||
|
||||
// Submit each answer in sequence
|
||||
const historyCounts = [];
|
||||
|
||||
for (let i = 0; i < LONG_INVESTIGATION.answerSequence.length; i++) {
|
||||
const result = await submitAnswer(page, LONG_INVESTIGATION.answerSequence[i].text);
|
||||
expect(result.success).toBe(true);
|
||||
|
||||
// Record history turn count after each answer
|
||||
const indicators = page.locator('[data-testid="investigation-turn"]');
|
||||
const count = await indicators.count();
|
||||
historyCounts.push(count);
|
||||
|
||||
// Each turn's question should be visible (or terminal state)
|
||||
const workspace = page.locator('[data-testid="reasoning-workspace"]');
|
||||
await expect(workspace).toBeVisible();
|
||||
}
|
||||
|
||||
// History counts should have monotonically increased
|
||||
for (let i = 1; i < historyCounts.length; i++) {
|
||||
expect(historyCounts[i]).toBeGreaterThanOrEqual(historyCounts[i - 1]);
|
||||
}
|
||||
});
|
||||
|
||||
/* ═══════════════ Test: contradiction fixture renders both sides ═ */
|
||||
|
||||
test("contradiction fixture — opposing recommendations in investigation card", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
|
||||
await page.evaluate(() => {
|
||||
window.__MOCK_ENABLED = true;
|
||||
window.__MOCK_DELAY = "instant";
|
||||
window.__MOCK_SCENARIO = ""; // use default
|
||||
});
|
||||
|
||||
const textarea = page.locator("textarea[placeholder*=Describe]");
|
||||
await textarea.fill(CONTRADICTION.centralStatement);
|
||||
await page.getByRole("button", { name: "Analyse" }).click();
|
||||
|
||||
// Workspace renders with contradiction scenario
|
||||
await waitForLoading(page);
|
||||
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
|
||||
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
|
||||
|
||||
// If there's an answer textarea (next question in contradiction flow), submit it
|
||||
const result = await submitAnswer(page, CONTRADICTION.answerSequence[0]?.text || "");
|
||||
|
||||
if (result.success) {
|
||||
// Workspace should show updated investigation with contradiction nodes
|
||||
const workspace = page.locator('[data-testid="reasoning-workspace"]');
|
||||
await expect(workspace).toBeVisible();
|
||||
|
||||
// Current understanding summary should be present
|
||||
const understandingHeading = page.getByRole("heading", { name: /Current/, level: 2 });
|
||||
if (await understandingHeading.isVisible({ timeout: 5000 })) {
|
||||
await expect(understandingHeading).toBeVisible();
|
||||
}
|
||||
}
|
||||
|
||||
// Final workspace should still be visible regardless of whether terminal was reached
|
||||
const workspace = page.locator('[data-testid="reasoning-workspace"]');
|
||||
await expect(workspace).toBeVisible();
|
||||
});
|
||||
|
||||
/* ═══════════════ Test: diagnosis scenario ══════════════════════ */
|
||||
|
||||
test("diagnosis scenario — churn investigation renders correctly", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
|
||||
await page.evaluate(() => {
|
||||
window.__MOCK_ENABLED = true;
|
||||
window.__MOCK_DELAY = "instant";
|
||||
window.__MOCK_SCENARIO = "";
|
||||
});
|
||||
|
||||
const textarea = page.locator("textarea[placeholder*=Describe]");
|
||||
await textarea.fill(DIAGNOSIS.centralStatement);
|
||||
await page.getByRole("button", { name: "Analyse" }).click();
|
||||
|
||||
// Workspace renders with diagnosis nodes and edges
|
||||
await waitForLoading(page);
|
||||
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
|
||||
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
|
||||
|
||||
// Scenario-specific heading should be present
|
||||
const workspace = page.locator('[data-testid="reasoning-workspace"]');
|
||||
await expect(workspace).toBeVisible();
|
||||
|
||||
// The central statement from the diagnosis scenario should appear in the card
|
||||
const cardText = page.locator('[data-testid="reasoning-workspace"]').locator('p').first();
|
||||
if (await cardText.isVisible()) {
|
||||
await expect(cardText).toContainText(/churn/i);
|
||||
}
|
||||
});
|
||||
|
||||
/* ═══════════════ Test: comparison scenario with side-by-side ══ */
|
||||
|
||||
test("comparison scenario — product rating nodes render with edges", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
|
||||
await page.evaluate(() => {
|
||||
window.__MOCK_ENABLED = true;
|
||||
window.__MOCK_DELAY = "instant";
|
||||
window.__MOCK_SCENARIO = "";
|
||||
});
|
||||
|
||||
const textarea = page.locator("textarea[placeholder*=Describe]");
|
||||
await textarea.fill(COMPARISON.centralStatement);
|
||||
await page.getByRole("button", { name: "Analyse" }).click();
|
||||
|
||||
// Workspace renders
|
||||
await waitForLoading(page);
|
||||
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
|
||||
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
|
||||
|
||||
// Submit first answer (comparison fixture has multiple turns)
|
||||
const result = await submitAnswer(page, COMPARISON.answerSequence[0]?.text || "");
|
||||
|
||||
if (result.success) {
|
||||
// Second turn's answer textarea should appear
|
||||
const answerTextarea = page.locator('textarea[placeholder*=Answer]');
|
||||
expect(await answerTextarea.isVisible()).toBe(true);
|
||||
|
||||
// Submit second answer
|
||||
await submitAnswer(page, COMPARISON.answerSequence[1]?.text || "");
|
||||
|
||||
// Workspace still visible after multi-turn
|
||||
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
|
||||
}
|
||||
});
|
||||
|
||||
/* ═══════════════ Test: prioritisation scenario ═════════════════ */
|
||||
|
||||
test("prioritisation scenario — team relocation yields ranked options", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
|
||||
await page.evaluate(() => {
|
||||
window.__MOCK_ENABLED = true;
|
||||
window.__MOCK_DELAY = "instant";
|
||||
window.__MOCK_SCENARIO = "";
|
||||
});
|
||||
|
||||
const textarea = page.locator("textarea[placeholder*=Describe]");
|
||||
await textarea.fill(PRIORITISATION.centralStatement);
|
||||
await page.getByRole("button", { name: "Analyse" }).click();
|
||||
|
||||
// Workspace renders
|
||||
await waitForLoading(page);
|
||||
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
|
||||
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
|
||||
|
||||
// Submit first answer
|
||||
const result = await submitAnswer(page, PRIORITISATION.answerSequence[0]?.text || "");
|
||||
|
||||
if (result.success) {
|
||||
// Workspace should show updated state with prioritisation nodes
|
||||
const workspace = page.locator('[data-testid="reasoning-workspace"]');
|
||||
await expect(workspace).toBeVisible();
|
||||
}
|
||||
});
|
||||
|
||||
/* ═══════════════ Test: rapid multi-turn loading transitions ─── */
|
||||
|
||||
test("rapid successive updates maintain correct loading-state cycling", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
|
||||
await page.evaluate(() => {
|
||||
window.__MOCK_ENABLED = true;
|
||||
window.__MOCK_DELAY = "instant";
|
||||
window.__MOCK_SCENARIO = "";
|
||||
});
|
||||
|
||||
const textarea = page.locator("textarea[placeholder*=Describe]");
|
||||
await textarea.fill(LONG_INVESTIGATION.centralStatement); // use long scenario central statement
|
||||
await page.getByRole("button", { name: "Analyse" }).click();
|
||||
|
||||
await waitForLoading(page);
|
||||
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
|
||||
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
|
||||
|
||||
// Submit multiple answers rapidly (all instant mock = no delay)
|
||||
for (let i = 0; i < LONG_INVESTIGATION.answerSequence.length; i++) {
|
||||
const ans = LONG_INVESTIGATION.answerSequence[i]?.text;
|
||||
if (!ans) break;
|
||||
|
||||
await submitAnswer(page, ans);
|
||||
|
||||
// After each submission, verify workspace stable and no duplicate loading overlays
|
||||
const overlays = page.locator('[data-testid="loading-overlay"]');
|
||||
await expect(overlays).toHaveCount(0);
|
||||
}
|
||||
});
|
||||
|
||||
/* ═══════════════ Test: answer textarea placeholder changes ───── */
|
||||
|
||||
test("answer textarea placeholder updates between turns", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
|
||||
await page.evaluate(() => {
|
||||
window.__MOCK_ENABLED = true;
|
||||
window.__MOCK_DELAY = "instant";
|
||||
window.__MOCK_SCENARIO = "";
|
||||
});
|
||||
|
||||
const textarea = page.locator("textarea[placeholder*=Describe]");
|
||||
await textarea.fill(INVESTIGATION_SCENARIOS[0].centralStatement);
|
||||
await page.getByRole("button", { name: "Analyse" }).click();
|
||||
|
||||
// Initial workspace should have an answer textarea with a question-related placeholder
|
||||
const answerTextarea = page.locator('textarea[placeholder*=Answer]');
|
||||
await expect(answerTextarea).toBeVisible({ timeout: 10_000 });
|
||||
|
||||
// Submit first answer
|
||||
await submitAnswer(page, INVESTIGATION_SCENARIOS[0].answerSequence[0]?.text || "");
|
||||
|
||||
// After update, new question textarea should be present with potentially different placeholder
|
||||
const updatedTextarea = page.locator('textarea[placeholder*=Answer]');
|
||||
await expect(updatedTextarea).toBeVisible();
|
||||
|
||||
// The workspace card should still display the current investigation context
|
||||
const workspace = page.locator('[data-testid="reasoning-workspace"]');
|
||||
await expect(workspace).toBeVisible();
|
||||
});
|
||||
|
||||
/* ═══════════════ Test: terminal state with no answer available ─ */
|
||||
|
||||
test("terminal state renders without answer textarea", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
|
||||
// Use the "complete" scenario which eventually reaches a terminal (no next question)
|
||||
await page.evaluate(() => {
|
||||
window.__MOCK_ENABLED = true;
|
||||
window.__MOCK_DELAY = "instant";
|
||||
window.__MOCK_SCENARIO = "complete";
|
||||
});
|
||||
|
||||
const textarea = page.locator("textarea[placeholder*=Describe]");
|
||||
await textarea.fill(SCENARIO_LOOKUP["Happy path — complete investigation"].centralStatement);
|
||||
await page.getByRole("button", { name: "Analyse" }).click();
|
||||
|
||||
// Workspace renders
|
||||
await waitForLoading(page);
|
||||
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
|
||||
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
|
||||
|
||||
// If there's a question (turn 0 gives a question), submit it to reach terminal
|
||||
const answerTextarea = page.locator('textarea[placeholder*=Answer]');
|
||||
if (await answerTextarea.isVisible({ timeout: 3000 })) {
|
||||
await answerTextarea.fill(SCENARIO_LOOKUP["Happy path — complete investigation"].answerSequence[0]?.text || "");
|
||||
await page.getByRole("button", { name: "Update situation" }).click();
|
||||
|
||||
// Wait for terminal state
|
||||
await waitForLoading(page);
|
||||
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
|
||||
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
|
||||
|
||||
// At terminal state, answer textarea should no longer be visible (or disabled)
|
||||
const hasAnswer = await answerTextarea.isVisible().catch(() => false);
|
||||
if (hasAnswer) {
|
||||
const disabledAttr = await answerTextarea.getAttribute("disabled");
|
||||
expect(disabledAttr).toBeTruthy();
|
||||
}
|
||||
}
|
||||
|
||||
// Terminal card should be present
|
||||
const workspace = page.locator('[data-testid="reasoning-workspace"]');
|
||||
await expect(workspace).toBeVisible();
|
||||
});
|
||||
@@ -0,0 +1,193 @@
|
||||
/**
|
||||
* Recovery-state E2E tests — provider error and malformed response flows.
|
||||
*
|
||||
* What these tests validate:
|
||||
* - Error state renders an error panel with structured message
|
||||
* - Workspace remains visible alongside the recovery card
|
||||
* - ProviderUnavailableCard / MalformedResponseCard content visible
|
||||
* - Session persistence survives a recovery state (recovery key in sessionStorage)
|
||||
*
|
||||
* What these tests do NOT validate:
|
||||
* - Error handling logic correctness (that is unit-level concern)
|
||||
*/
|
||||
|
||||
import { test, expect } from "@playwright/test";
|
||||
import { INVESTIGATION_SCENARIOS, SCENARIO_LOOKUP } from "./fixtures/investigation-scenarios.js";
|
||||
|
||||
const PROVIDER_ERROR = SCENARIO_LOOKUP["Provider error"];
|
||||
const MALFORMED_RESPONSE = SCENARIO_LOOKUP["Malformed response"];
|
||||
|
||||
/* ── Test helpers ─────────────────────────────────────────────── */
|
||||
|
||||
async function waitForLoading(page) {
|
||||
await expect(page.locator('[data-testid="loading-overlay"]')).toBeVisible();
|
||||
}
|
||||
|
||||
async function waitForWorkspaceReady(page, options = {}) {
|
||||
const { expectLoadingGone = true } = options;
|
||||
if (expectLoadingGone) {
|
||||
await page.waitForSelector('[data-testid="loading-overlay"]', { state: "hidden", timeout: 30_000 });
|
||||
}
|
||||
const workspace = page.locator('[data-testid="reasoning-workspace"]');
|
||||
await expect(workspace).toBeVisible({ timeout: 15_000 });
|
||||
}
|
||||
|
||||
/* ═══════════════ Test: provider error after start ══════════════ */
|
||||
|
||||
test("provider error: analyse returns failure → error panel visible alongside workspace", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
|
||||
// Enable mock mode in error scenario
|
||||
await page.evaluate(() => {
|
||||
window.__MOCK_ENABLED = true;
|
||||
window.__MOCK_DELAY = "instant";
|
||||
window.__MOCK_SCENARIO = "error";
|
||||
});
|
||||
|
||||
const textarea = page.locator("textarea[placeholder*=Describe]");
|
||||
await textarea.fill(PROVIDER_ERROR.centralStatement);
|
||||
await page.getByRole("button", { name: "Analyse" }).click();
|
||||
|
||||
// Loading overlay appears briefly then disappears
|
||||
await waitForLoading(page);
|
||||
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
|
||||
|
||||
// Workspace should still be visible (even with error result)
|
||||
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
|
||||
|
||||
// Error panel with structured error message
|
||||
const errorPanel = page.locator('div[class*="border-red"][class*="bg-red"]');
|
||||
await expect(errorPanel).toBeVisible();
|
||||
await expect(errorPanel.locator('div[class*="text-sm"]').first()).toContainText(/error/i);
|
||||
});
|
||||
|
||||
/* ═══════════════ Test: provider error during update ════════════ */
|
||||
|
||||
test("provider error during update turn: error panel appears after answer submit", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
|
||||
// Use a non-error scenario first to get into success state, then switch to error for update
|
||||
await page.evaluate(() => {
|
||||
window.__MOCK_ENABLED = true;
|
||||
window.__MOCK_DELAY = "instant";
|
||||
window.__MOCK_SCENARIO = ""; // normal default for start
|
||||
});
|
||||
|
||||
const textarea = page.locator("textarea[placeholder*=Describe]");
|
||||
await textarea.fill(INVESTIGATION_SCENARIOS[0].centralStatement);
|
||||
await page.getByRole("button", { name: "Analyse" }).click();
|
||||
|
||||
await waitForLoading(page);
|
||||
await waitForWorkspaceReady(page);
|
||||
|
||||
// Switch to error scenario for the update call
|
||||
await page.evaluate(() => {
|
||||
window.__MOCK_SCENARIO = "error";
|
||||
});
|
||||
|
||||
// Submit an answer to trigger the update call (which will now get error)
|
||||
const answerTextarea = page.locator('textarea[placeholder*=Answer]');
|
||||
if (await answerTextarea.isVisible({ timeout: 3000 })) {
|
||||
await answerTextarea.fill("Some answer that should fail");
|
||||
await page.getByRole("button", { name: "Update situation" }).click();
|
||||
|
||||
// Verify error panel appears after the update call errors
|
||||
const errorPanel = page.locator('div[class*="border-red"][class*="bg-red"]');
|
||||
await expect(errorPanel).toBeVisible({ timeout: 10_000 });
|
||||
}
|
||||
});
|
||||
|
||||
/* ═══════════════ Test: malformed response during start ═════════ */
|
||||
|
||||
test("malformed response during start: error state visible, workspace still rendered", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
|
||||
// Malformed responses come through the error mock path (same as provider error)
|
||||
await page.evaluate(() => {
|
||||
window.__MOCK_ENABLED = true;
|
||||
window.__MOCK_DELAY = "instant";
|
||||
window.__MOCK_SCENARIO = "error";
|
||||
});
|
||||
|
||||
const textarea = page.locator("textarea[placeholder*=Describe]");
|
||||
await textarea.fill(MALFORMED_RESPONSE.centralStatement);
|
||||
await page.getByRole("button", { name: "Analyse" }).click();
|
||||
|
||||
// Loading then workspace
|
||||
await waitForLoading(page);
|
||||
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
|
||||
|
||||
const workspace = page.locator('[data-testid="reasoning-workspace"]');
|
||||
await expect(workspace).toBeVisible();
|
||||
|
||||
// Error text present in the red panel
|
||||
const errorText = page.locator('div[class*="border-red"][class*="text-sm"]').first();
|
||||
await expect(errorText).toHaveText(/error|failed/i);
|
||||
});
|
||||
|
||||
/* ═══════════════ Test: recovery state persists in session ══════ */
|
||||
|
||||
test("recovery state persists via sessionStorage after error response", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
|
||||
await page.evaluate(() => {
|
||||
window.__MOCK_ENABLED = true;
|
||||
window.__MOCK_DELAY = "instant";
|
||||
window.__MOCK_SCENARIO = "error";
|
||||
});
|
||||
|
||||
const textarea = page.locator("textarea[placeholder*=Describe]");
|
||||
await textarea.fill(PROVIDER_ERROR.centralStatement);
|
||||
await page.getByRole("button", { name: "Analyse" }).click();
|
||||
|
||||
// Wait for workspace to render with error state
|
||||
await waitForLoading(page);
|
||||
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
|
||||
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
|
||||
|
||||
// Verify sessionStorage has the session key with situationGraph from error response
|
||||
const sessionRaw = await page.evaluate(() => {
|
||||
const raw = sessionStorage.getItem("confidence-engine-session");
|
||||
return raw ? JSON.parse(raw) : null;
|
||||
});
|
||||
|
||||
// Session should exist (even in error recovery path, the scenario text is persisted)
|
||||
expect(sessionRaw).not.toBeNull();
|
||||
});
|
||||
|
||||
/* ═══════════════ Test: multiple error transitions ══════════════ */
|
||||
|
||||
test("multiple successive errors maintain UI state without crash", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
|
||||
// Enable mock mode with error scenario
|
||||
await page.evaluate(() => {
|
||||
window.__MOCK_ENABLED = true;
|
||||
window.__MOCK_DELAY = "instant";
|
||||
window.__MOCK_SCENARIO = "error";
|
||||
});
|
||||
|
||||
const textarea = page.locator("textarea[placeholder*=Describe]");
|
||||
await textarea.fill(PROVIDER_ERROR.centralStatement);
|
||||
|
||||
// First analyse call
|
||||
await page.getByRole("button", { name: "Analyse" }).click();
|
||||
await waitForLoading(page);
|
||||
await expect(page.locator('[data-testid="loading-overlay"]')).toBeHidden({ timeout: 5_000 });
|
||||
await expect(page.locator('[data-testid="reasoning-workspace"]')).toBeVisible();
|
||||
|
||||
// If there's an answer textarea, try another update (also errors)
|
||||
const answerTextarea = page.locator('textarea[placeholder*=Answer]');
|
||||
if (await answerTextarea.isVisible({ timeout: 2000 })) {
|
||||
await answerTextarea.fill("Second attempt");
|
||||
await page.getByRole("button", { name: "Update situation" }).click();
|
||||
|
||||
// Error panel should still be visible (no crash, no layout shift)
|
||||
const errorPanel = page.locator('div[class*="border-red"]');
|
||||
await expect(errorPanel).toBeVisible({ timeout: 10_000 });
|
||||
}
|
||||
|
||||
// Workspace should remain stable — no errors thrown
|
||||
const workspace = page.locator('[data-testid="reasoning-workspace"]');
|
||||
await expect(workspace).toBeVisible();
|
||||
});
|
||||
Reference in New Issue
Block a user