144 lines
5.6 KiB
Markdown
144 lines
5.6 KiB
Markdown
# Engineering Runbook (PEDW FrontEnd)
|
|
|
|
## Purpose
|
|
|
|
Provide a repeatable operational playbook for safe delivery and incident-aware change management.
|
|
|
|
> Pre-flight: run `GUARDRAILS.md` checklist before significant implementation or merge.
|
|
|
|
## Standard Change Workflow
|
|
|
|
1. Clarify scope, constraints, and affected flows (public search/case, account/auth, portal/admin).
|
|
2. Identify risk level:
|
|
- **High:** auth/session, Prisma schema, middleware/CSP, uploads/documents, notifications
|
|
- **Medium:** routing/i18n rewrites, Redux hydration/persistence, PDF generation
|
|
- **Low:** isolated UI or content-only updates
|
|
3. Implement smallest viable change with clear rollback path.
|
|
4. Validate (minimum):
|
|
- `npm run lint`
|
|
- targeted manual checks for touched routes/APIs
|
|
- EN/CY parity checks
|
|
- accessibility smoke (keyboard, focus, labels, headings)
|
|
- negative-path checks for sensitive logic
|
|
5. Record outcomes in `memory-bank/change-log.md`.
|
|
|
|
## Pre-Release Checklist
|
|
|
|
- [ ] Scope and non-goals documented
|
|
- [ ] Risk notes cover auth/data/i18n/a11y impacts
|
|
- [ ] Validation evidence captured (command output + manual matrix)
|
|
- [ ] Rollback steps prepared
|
|
- [ ] Memory-bank updated for decisions/pitfalls/open questions
|
|
|
|
## Incident Triage (Quick)
|
|
|
|
1. Classify impact: service unavailable, degraded flow, data/security concern.
|
|
2. Identify blast radius: which routes/APIs/locales/users are affected.
|
|
3. Check telemetry and logs (server + app insights) for errors around deploy window.
|
|
4. Apply mitigation:
|
|
- rollback/revert high-risk change,
|
|
- or ship minimal hotfix with guarded scope.
|
|
5. Validate recovery on EN/CY critical journeys.
|
|
6. Capture post-incident notes in `memory-bank/change-log.md` and `memory-bank/pitfalls.md`.
|
|
|
|
## High-Risk Flow Verification Matrix
|
|
|
|
- **Auth:** sign-in, verify-request, callback redirect safety, session continuity
|
|
- **Portal/account:** protected route access and profile/account updates
|
|
- **Uploads/docs:** invalid file type/size handling, unauthorized access protection
|
|
- **Notifications:** template/language selection and failure handling
|
|
- **Routing/i18n:** rewritten Welsh routes land on expected handlers
|
|
|
|
## Relay Hardening Rollout Playbook (TASK22239)
|
|
|
|
Use this when changing shared relay forwarding policy (timeouts/retries/logging) or deploying relay policy updates.
|
|
|
|
### Pre-Merge Governance Gate
|
|
|
|
- [ ] PR scope states what changed and what did not (endpoint contracts vs relay internals)
|
|
- [ ] Risk notes include auth/data/i18n/a11y impact and relay behavior impact
|
|
- [ ] Validation evidence attached:
|
|
- relay hardening targeted tests
|
|
- endpoint contract regression tests
|
|
- lint status
|
|
- [ ] Rollback steps documented (config rollback + commit revert)
|
|
- [ ] Memory-bank updated (`change-log`, and where relevant `decisions`/`patterns`)
|
|
|
|
### Non-Prod Smoke Matrix (Required)
|
|
|
|
Run against a representative non-production environment:
|
|
|
|
1. **Deterministic auth/client failures**
|
|
- force/verify `401` and `403`
|
|
- expected: no retries, immediate handled failure
|
|
2. **Deterministic validation failures**
|
|
- force/verify `400` or `404`
|
|
- expected: no retries
|
|
3. **Transient upstream failures**
|
|
- force/verify `503` / `429`
|
|
- expected: bounded retries + bounded backoff
|
|
4. **Timeout behavior**
|
|
- force latency above timeout threshold
|
|
- expected: bounded failure path and no retry storm
|
|
5. **Operational logging behavior**
|
|
- expected: structured redacted retry/failure events
|
|
- expected: no duplicate endpoint-layer error spam for already-logged relay failures
|
|
|
|
### Progressive Runtime Rollout
|
|
|
|
1. Deploy with conservative retry settings.
|
|
2. Verify service stability and log volume for first release window.
|
|
3. Tune only one variable at a time (`timeout`, then `retry count`, then delays).
|
|
|
|
Recommended starting posture:
|
|
|
|
- `RELAY_RETRY_MAX` in low range (e.g. `1` or `2`)
|
|
- bounded delay values aligned with user-facing latency tolerance
|
|
- avoid simultaneous increases of timeout and retries unless incident evidence requires it
|
|
|
|
### Monitoring Checks (Day 1 / Day 3)
|
|
|
|
- Relay retry rate trend
|
|
- Timeout/error ratio and top status buckets
|
|
- Upstream latency impact on citizen-facing journeys
|
|
- Log volume increase/decrease and duplicate-error noise
|
|
|
|
### Fast Rollback / Mitigation
|
|
|
|
1. Immediate mitigation: set `RELAY_RETRY_MAX=0` (disables retries without code rollback).
|
|
2. If needed, reduce timeout and delay knobs to baseline-safe values.
|
|
3. Full rollback path: revert relay hardening commit set and redeploy.
|
|
4. Record incident + mitigation outcome in `memory-bank/change-log.md` and `memory-bank/pitfalls.md`.
|
|
|
|
### PR Evidence Snapshot Template (copy into PR description)
|
|
|
|
Use this template when shipping relay policy updates:
|
|
|
|
```
|
|
Relay policy smoke evidence (non-prod)
|
|
|
|
- [ ] 401/403 deterministic auth failures -> no retries observed
|
|
- [ ] 400/404 deterministic validation failures -> no retries observed
|
|
- [ ] 429/503 transient failures -> bounded retries/backoff observed
|
|
- [ ] timeout path -> bounded failure path, no retry storm
|
|
- [ ] structured redacted logs emitted (`relay_request_retrying` / `relay_request_failed`)
|
|
- [ ] duplicate endpoint-layer error logging suppressed for already-logged relay failures
|
|
|
|
Runtime knobs used during validation:
|
|
- RELAY_TIMEOUT_MS=
|
|
- RELAY_RETRY_MAX=
|
|
- RELAY_RETRY_BASE_DELAY_MS=
|
|
- RELAY_RETRY_MAX_DELAY_MS=
|
|
|
|
Rollback readiness:
|
|
- [ ] quick mitigation validated: RELAY_RETRY_MAX=0
|
|
- [ ] commit-revert path identified and documented
|
|
```
|
|
|
|
## Definition of Ready for AI-Assisted Tasks
|
|
|
|
- Clear acceptance criteria
|
|
- Named affected files/flows
|
|
- Risk classification assigned
|
|
- Validation plan agreed (including manual checks)
|