Skip to content

Incident Triage

Current monitoring stack: New Relic and CloudWatch. Use Warsaw time for incident windows and record the corresponding UTC boundaries.

Symptom First place Corroborating evidence
Backend ERROR or unhandled exception New Relic Backend ERROR logs, APM errors and traces Structured CloudWatch logs and stack trace; release and request correlation
API unavailable or slow New Relic backend health, ALB/ECS or integration host metrics Public health endpoint, CloudWatch logs, deployment status
Mobile crash New Relic mobile occurrences and crash condition Exact build/release, readable frames, native diagnostics
Admin JavaScript error New Relic browser errors and source maps Exact served build and private source-map verification
Failed payment or webhook New Relic critical workflow and Stripe backlog/dead-event metrics Stripe event status and backend CloudWatch logs
Infrastructure lifecycle failure New Relic MamNajemAwsEvent and AWS polling metrics ECS/EventBridge/SNS/Lambda CloudWatch evidence
Failed delivery or ingestion New Relic notification records and telemetry freshness Forwarder Lambda logs, workflow/destination status and IAM scope
Failed deployment or scheduled maintenance Exact GitLab pipeline and failed job Release marker and post-deploy New Relic/CloudWatch window
  1. Take a current read-only snapshot using the local mamnajem-alert-triage skill. Query recovered incidents as well as currently open issues; follow pagination.
  2. Check release/deployment timestamps and correlate each activation with logs. A closed issue demonstrates recovery, not a permanent fix.
  3. Identify expected tests separately. Routine checks should use an isolated policy or scoped maintenance muting; preserve detection of genuine errors.
  4. Classify the finding, reproduce its source and recommend an exact fix. Implement only when remediation is authorized.
  5. After an authorized fix, verify New Relic receipt, condition behavior, Discord provider delivery records and the post-deploy CloudWatch window. Follow guarded delivery.

Backend log coverage

Both environments have a critical Backend ERROR logs condition:

SELECT count(*) FROM Log
WHERE `entity.name` = 'mamnajem-integration-backend' AND level = 'ERROR'

Use the production application name for production. Count above zero qualifies, evaluated at least once in a 300-second duration with 60-second aggregation. The generic exception handler logs at ERROR with its throwable. A WARN stack trace or a trace printed outside the logger is not covered by this severity filter; compare CloudWatch with New Relic before declaring full coverage. The separate APM condition excludes expected HTTP 4xx responses and retains server/status-less failures.

See observability and alerts for live coverage and remaining verification gates. Physical iPhone production crash verification remains deferred by the owner.