pedwfrontend/context/runbook.md

Engineering Runbook (PEDW FrontEnd)

Purpose

Provide a repeatable operational playbook for safe delivery and incident-aware change management.

Pre-flight: run GUARDRAILS.md checklist before significant implementation or merge.

Standard Change Workflow

1. Clarify scope, constraints, and affected flows (public search/case, account/auth, portal/admin).
2. Identify risk level:
   • High: auth/session, Prisma schema, middleware/CSP, uploads/documents, notifications
   • Medium: routing/i18n rewrites, Redux hydration/persistence, PDF generation
   • Low: isolated UI or content-only updates
3. Implement smallest viable change with clear rollback path.
4. Validate (minimum):
   • npm run lint
   • targeted manual checks for touched routes/APIs
   • EN/CY parity checks
   • accessibility smoke (keyboard, focus, labels, headings)
   • negative-path checks for sensitive logic
5. Record outcomes in memory-bank/change-log.md.

Pre-Release Checklist

• Scope and non-goals documented
• Risk notes cover auth/data/i18n/a11y impacts
• Validation evidence captured (command output + manual matrix)
• Rollback steps prepared
• Memory-bank updated for decisions/pitfalls/open questions

Incident Triage (Quick)

1. Classify impact: service unavailable, degraded flow, data/security concern.
2. Identify blast radius: which routes/APIs/locales/users are affected.
3. Check telemetry and logs (server + app insights) for errors around deploy window.
4. Apply mitigation:
   • rollback/revert high-risk change,
   • or ship minimal hotfix with guarded scope.
5. Validate recovery on EN/CY critical journeys.
6. Capture post-incident notes in memory-bank/change-log.md and memory-bank/pitfalls.md.

High-Risk Flow Verification Matrix

• Auth: sign-in, verify-request, callback redirect safety, session continuity
• Portal/account: protected route access and profile/account updates
• Uploads/docs: invalid file type/size handling, unauthorized access protection
• Notifications: template/language selection and failure handling
• Routing/i18n: rewritten Welsh routes land on expected handlers

Relay Hardening Rollout Playbook (TASK22239)

Use this when changing shared relay forwarding policy (timeouts/retries/logging) or deploying relay policy updates.

Pre-Merge Governance Gate

• PR scope states what changed and what did not (endpoint contracts vs relay internals)
• Risk notes include auth/data/i18n/a11y impact and relay behavior impact
• Validation evidence attached:
  • relay hardening targeted tests
  • endpoint contract regression tests
  • lint status
• Rollback steps documented (config rollback + commit revert)
• Memory-bank updated (change-log, and where relevant decisions/patterns)

Non-Prod Smoke Matrix (Required)

Run against a representative non-production environment:

1. Deterministic auth/client failures
   • force/verify 401 and 403
   • expected: no retries, immediate handled failure
2. Deterministic validation failures
   • force/verify 400 or 404
   • expected: no retries
3. Transient upstream failures
   • force/verify 503 / 429
   • expected: bounded retries + bounded backoff
4. Timeout behavior
   • force latency above timeout threshold
   • expected: bounded failure path and no retry storm
5. Operational logging behavior
   • expected: structured redacted retry/failure events
   • expected: no duplicate endpoint-layer error spam for already-logged relay failures

Progressive Runtime Rollout

1. Deploy with conservative retry settings.
2. Verify service stability and log volume for first release window.
3. Tune only one variable at a time (timeout, then retry count, then delays).

Recommended starting posture:

• RELAY_RETRY_MAX in low range (e.g. 1 or 2)
• bounded delay values aligned with user-facing latency tolerance
• avoid simultaneous increases of timeout and retries unless incident evidence requires it

Monitoring Checks (Day 1 / Day 3)

• Relay retry rate trend
• Timeout/error ratio and top status buckets
• Upstream latency impact on citizen-facing journeys
• Log volume increase/decrease and duplicate-error noise

Fast Rollback / Mitigation

1. Immediate mitigation: set RELAY_RETRY_MAX=0 (disables retries without code rollback).
2. If needed, reduce timeout and delay knobs to baseline-safe values.
3. Full rollback path: revert relay hardening commit set and redeploy.
4. Record incident + mitigation outcome in memory-bank/change-log.md and memory-bank/pitfalls.md.

PR Evidence Snapshot Template (copy into PR description)

Use this template when shipping relay policy updates:

Relay policy smoke evidence (non-prod)

- [ ] 401/403 deterministic auth failures -> no retries observed
- [ ] 400/404 deterministic validation failures -> no retries observed
- [ ] 429/503 transient failures -> bounded retries/backoff observed
- [ ] timeout path -> bounded failure path, no retry storm
- [ ] structured redacted logs emitted (`relay_request_retrying` / `relay_request_failed`)
- [ ] duplicate endpoint-layer error logging suppressed for already-logged relay failures

Runtime knobs used during validation:
- RELAY_TIMEOUT_MS=
- RELAY_RETRY_MAX=
- RELAY_RETRY_BASE_DELAY_MS=
- RELAY_RETRY_MAX_DELAY_MS=

Rollback readiness:
- [ ] quick mitigation validated: RELAY_RETRY_MAX=0
- [ ] commit-revert path identified and documented

Definition of Ready for AI-Assisted Tasks

• Clear acceptance criteria
• Named affected files/flows
• Risk classification assigned
• Validation plan agreed (including manual checks)