Incident Response¶
Last audited: 2026-05-30.
Severity¶
SEV1: production unavailable, payment/subscription corruption, broad data exposure, or destructive data loss.SEV2: major feature broken for many users, webhook/payment processing delayed, repeated 5xx, or production deploy regression with workaround.SEV3: limited user impact, noisy degradation, non-critical job failure, or integration-only incident.
Ownership¶
The first responder owns the incident until they explicitly hand off. For SEV1 and SEV2, assign:
- Incident owner: coordinates decisions and timeline.
- Technical lead: investigates and fixes.
- Communications owner: updates users or stakeholders.
- Scribe: records timestamps, evidence, and actions.
Acknowledgement Targets¶
SEV1: acknowledge within 10 minutes.SEV2: acknowledge within 30 minutes.SEV3: acknowledge during the same working day.
Initial Triage¶
- Confirm environment and start time.
- Check newest deploy, GitLab pipeline, New Relic incidents and Discord notification delivery, and CloudWatch backend logs. Confirm the alert reached the intended channel before treating an alarm as acknowledged.
- Capture
requestId,correlationId,spawnId, Stripe event IDs, user/account references needed for support, and release SHA. Do not copy raw secrets, tokens, payment payloads, message bodies, or unnecessary personal data into the incident record. - Decide whether to roll back, disable a workflow, replay a webhook, or continue diagnosis.
Rollback Triggers¶
Rollback or redeploy last known good when:
- 5xx or health alarms start within 30 minutes of deploy.
- New Relic shows new or regressed production errors tied to the release SHA.
- Payment, subscription, login, or upload flows fail broadly.
- The team cannot identify a safer mitigation within the acknowledgement target.
Customer Communications¶
Communicate for SEV1, user-visible SEV2, payment/subscription incidents, or data integrity risk. Include what is affected, current workaround, next update time, and final resolution.
Post-Incident¶
Within two working days, record:
- Timeline and detection source
- User/account impact
- Root cause and contributing factors
- What worked and what slowed diagnosis
- Follow-up owners and due dates