Skip to content

Observability And Alerts

Monitoring status: 2026-10-05.

New Relic account 8573192 (EU) is the selected destination for operational metrics, logs, distributed traces, backend errors, admin JavaScript errors and mobile crashes. Firebase remains the optional product analytics provider.

Product Outcomes reporting

Product reports use shared backend aggregate definitions and a separate private Product Outcomes dashboard. They require a dedicated runtime reporting query identity; the workstation administration key must never be injected into the backend. Publication stays disabled until runtime identity, effective retention, and billing are confirmed. The current change set prepares collection/history, admin reports, aggregate publication, and isolated non-paging warnings; it does not extend the deployed operational cutover evidence below into reporting acceptance. See the implementation brief and reporting runbook.

New Relic setup and cutover

The integration backend is deployed with Spring Boot 4.0.8 and source revision 34c777248f73bfcb51746f1327e0344aafa4bb7b. Pipeline 2910938493 passed its security, build, API compatibility and deployment gates. Local Maven verification passed 6,660 tests with no failures or errors. Public health, liveness, readiness and subscription plans returned HTTP 200 after deployment.

New Relic receipt is verified for backend transactions, distributed spans, application logs, business metrics and health (3=UP), host metrics, system, Caddy and PostgreSQL logs, and backup/deployment success (1). A real PostgreSQL backup reached S3 and its success metric was read back. New Relic recorded successful Discord notifications for a real failed deployment and its recovery. The agent requires Fluent Bit as an explicit dependency because the bootstrap uses --no-install-recommends. Caddy forwarding covers both the bootstrap and TLS log paths; Java agent logs are queried by entity.name because they do not carry the infrastructure collector's environment attribute.

All 49 MamNajem CloudWatch metric alarms were removed after their New Relic replacements and Discord delivery were verified: 17 integration and 32 production. The New Relic policies contain 30 integration and 39 production conditions. AWS log groups, metric filters, SNS topics and ECS EventBridge rules remain; deleting the alarms removes their recurring alarm charges.

The existing event-forwarder Lambda now sends safe infrastructure events to New Relic, which evaluates the alerts and delivers Discord notifications. A labelled MAMNAJEM_MIGRATION_TEST published to each environment's SNS topic was received as MamNajemAwsEvent, and each workflow recorded a successful activated notification. This check uses the real SNS subscription and Lambda without failing or restarting a production task.

Coverage includes integration and production Cognito/Lambda errors and throttles, SNS delivery failures, ECS deployment failures and unexpected task exits, backend restart bursts, and account notification errors. A restart burst means more than one backend startup within a rolling 15-minute window; a single expected deployment startup is allowed. Stale export processing already increments data.export.archive.failed, so the export-failure condition covers it without relying on an unpublished stale gauge.

Backend, admin and mobile use New Relic in integration and production. Deployment and runtime verification have passed for the authorized cutover; the owner deferred the physical iPhone production crash check.

Surface Current result Remaining gate
Integration backend and host Live New Relic telemetry, backup/deployment metrics and Discord delivery verified No remaining deployment gate
Admin Revision f591a69d1d9eefcee26620b3138fbeeec28aa7a5 deployed to both environments; live login form, marker, served asset, security headers and authenticated activation reset verified; production page view and scrubbed JavaScript error received in New Relic No remaining deployment gate
React Native Revision d836145f passed all four NVIDIA CI shards; signed integration build 991 and production build 997 delivered; production Android crash and ten readable framework frames received, with successful Discord delivery Owner confirmed readable Android JavaScript file/function names; physical iPhone production crash verification deferred by the owner
Production Backend pipeline 2910968044 and mobile root 2911478424 passed; live backend and admin monitoring, cloud polling and Discord verified; Android build 997 uploaded to its internal track, iOS build 997 observed as VALID in its separate production TestFlight app No remaining deployment gate; physical iPhone production crash verification deferred by the owner

Admin integration pipeline 2907572052 passed build, security, source-map, smoke and deployment checks. The live verifier independently confirmed the sealed source revision, runtime-contract checksum and security headers. A real protected GitLab job authenticated with the reset controller before AWS writes; its cleanup disabled integration activation and cleared the scoped activation SHA. Production activation remained disabled. The temporary inactive schedule was removed after the one-shot deployment; its pipeline evidence is retained.

The controller runs on the existing integration host using architecture revision 9f6cc425. Its GitLab key is restricted to the admin project and the read/update operations needed to verify jobs and disarm activation. The service reads its protected systemd credential from the read-only credential mount. It rejects unsafe credential paths or permissions.

A production startup check found that the admin client still rejected production environment settings before authentication and diagnostics could initialize. Revision f591a69d binds each environment to its exact admin and API URLs and retains rejection of cross-environment settings. The new production acceptance regression failed before the fix. After the fix, 134 application tests, 49 delivery helper tests, 34 Python checks, a production build and a local browser login check passed. Integration pipeline 2911418673 and production pipeline 2911448639 passed. Both live login forms, build markers, served assets, security headers and authenticated activation resets were verified. An anonymous production browser session produced a PageView and a scrubbed MamNajemMigrationProbe JavaScript error, both tagged with production and release mamnajem-admin@f591a69d1d9eefcee26620b3138fbeeec28aa7a5.

A fresh anonymous browser session produced PageView and JavaScriptError events for integration browser application 538909644. The controlled MamNajemMigrationProbe was received with the deployed release, environment integration and message Diagnostic error, confirming scrubbing. This probe verifies browser ingestion; it is not evidence of a mobile crash or production receipt. Source maps are uploaded privately and read back against the exact asset URL, release and build commit. A duplicate upload is accepted only after that read-back; the source-map listing API does not expose a content checksum.

React Native verification

The owner confirmed readable JavaScript file/function names in a New Relic occurrence from integration Android build 1.1.16+2000000908. This closes the UI-only JavaScript stack check. Physical iPhone production crash verification was deferred by the owner on October 4, 2026; it has not been demonstrated.

Integration uses the New Relic React Native SDK for native crashes, handled exceptions and JavaScript errors. SDK 1.9.1 reports JavaScript errors through MobileJSError. Crash and handled-error alert queries include that event and separate fatal from nonfatal errors. Architecture revision 7c85f0d3 passed pipeline 2907765835; its New Relic query configuration was applied to both environments. This does not establish a production application rollout.

The obsolete mobile EXPO_PUBLIC_NEW_RELIC_LICENSE_KEY CI variable was removed from the global, integration-build and integration-e2e-device scopes, with absence confirmed by read-back. Historical New Relic events remain available. Backend, admin and mobile project-level New Relic DSN, organization and project variables were subsequently removed and absence verified. Shared credentials and historical New Relic projects were preserved.

Signed Android build 1.1.16+2000000904, source 88dd18ba, produced a controlled native crash after diagnostic consent was enabled. New Relic received the crash for application 538909505; occurrence read-back returned ten readable Android framework frames. Notification history confirmed successful delivery of its incident to mamnajem-integration-discord. The shell-induced framework crash proves native ingestion and readable framework frames; it does not prove application ProGuard deobfuscation. This release is unminified, and its artifact contract records that no ProGuard map is required.

Signed Android build 1.1.16+2000000908, source 68d39fdc, produced controlled JavaScript network errors with bundle identity [email protected]+2000000908. New Relic received MobileJSError events and successfully delivered the handled-error incident to Discord. Its exact JavaScript source map was uploaded privately and read back. The mobile JavaScript errors view shows the ApiFailure and Error groups, but readable JavaScript frames were confirmed readable by the owner in the UI occurrence. The classic exception API returned no frames and did not establish that result. Connectivity was restored, diagnostic consent withdrawn, fixtures cleaned and the owned emulator stopped after the runtime probe.

Symbol publication is bound to each signed artifact's native version and bundle identity. Android requires a ProGuard map only for obfuscated bundles. iOS requires its dSYM archive and JavaScript map. Build, upload and diagnostic-job success alone do not prove runtime symbolication. The physical iPhone production crash and readable-stack check is deferred by the owner.

Production Android build 1.1.16+2000000997, source d836145f, was installed from the exact signed CI APK after its checksum and release identity were verified. On an owned disposable NVIDIA emulator, its anonymous login screen rendered and diagnostic consent was seeded only in local fixture storage. A controlled Android platform crash reached New Relic application 538909741 with the production environment and exact JavaScript bundle identity. Occurrence read-back returned ten readable Android framework frames, and a production mobile session was received. New Relic notification history also confirmed successful delivery of its real crash incident to mamnajem-production-discord. This verifies platform Java crash ingestion, not NDK or application ProGuard symbolication. The APK is unminified. Consent was disabled and verified before the owned emulator was removed; no production user or backend data was created or changed.

Android delivery gate

Mobile revision 536a545107b5083204dd126d8c4bd69c389b57f8 passed the complete integration delivery in pipeline 2910646711. The root, workload, Android release and iOS release pipelines all succeeded. JUnit verification confirmed all 220 required selectors in their assigned shards: 210 passed, ten expected skips and zero failures. All 5,388 application tests passed across 746 suites; quality, security and compatibility checks passed. Android 1.1.16+2000000965 was observed on the integration Play alpha track, and iOS 1.1.16 (965) completed TestFlight processing.

Before retrying that existing commit, a complete local run used the exact application and instrumentation APKs from its earlier CI attempt, four fresh NVIDIA GPU emulators and the tested runner storage settings. It passed the same 220-selector inventory with the same ten expected skips. The current application tests, runner checks and quality checks also passed locally. This retry required no additional mobile code push.

The local MamNajem runner 54582069 has a limit of four and two Docker tmpfs mounts: /opt/android-avd uses rw,size=10g, and /tmp/android-unknown uses rw,size=10g,mode=0700. The second mount holds QEMU's writable disk overlays; putting only the AVD directory in RAM leaves those writes on the host filesystem. A local storage reproduction produced a provider-guard cleanup timeout. The complete local and CI runs passed with both mounts in RAM. All four live CI containers were checked for these mounts, NVIDIA GLES rendering and host GPU processes; software-rendering fallback was disabled. The runner setting is in its private workstation configuration and must be preserved during runner reconfiguration.

Each shard uses a distinct emulator port and nine distinct owned fixture identities. All 36 identities were disjoint in the successful CI run. Bounded fixture cleanup passed in all four shards, and all owned emulators stopped. Future mobile changes require the complete local gate before an integration push, then terminal CI verification and the applicable signed release checks.

After the owner installed NVIDIA driver 595.91.07, the exact Android CI image booted its emulator locally with usable KVM and the NVIDIA RTX 4080 SUPER OpenGL renderer. The desktop now uses display :0; the protected project variable DISPLAY=:0, scoped to integration-e2e-device, overrides the older job default. The private Xauthority mirror was refreshed, and its existing desktop-session watcher remains enabled. MamNajem GPU runner 54582069 is online and unpaused, with both RAM mounts and hardware-only rendering preserved. The existing integration pipeline was retried at the same tested mobile revision, without another source push. Workload pipeline 2911399249 passed all four device shards and its release quality gate: 223 selectors, 213 passes, ten expected skips, zero failures and zero errors. Each shard ran on local runner 54582069, and each live emulator reported the NVIDIA RTX 4080 SUPER OpenGL renderer. The root, workload, Android release and iOS release pipelines all succeeded. Android 1.1.16+2000000991 was observed on the integration Play alpha track, and iOS 1.1.16 (991) was observed as VALID in the separate integration TestFlight app. The owner confirmed readable Android JavaScript frames. A green build does not establish physical iOS crash reporting; the owner deferred that production check.

Both iOS environments now use the protected project setting IOS_CLEAN_DERIVED_DATA=1. Integration initially restored cached Xcode output and failed the exact JavaScript source-map selection check after a successful archive. Its clean build at the same source revision passed the New Relic source-map upload. Production already used this clean-build setting. Keep the exact-map guard enabled; archive success alone is insufficient evidence.

The dependency gate verifies the pinned node-forge fix for GHSA-86w9-cpqp-85rv and the pinned upstream braces depth-validation patch for GHSA-vfj7-8cjw-p6xm. These are validated patches, not advisory suppressions. Patch, lockfile, installed runtime-file hashes and focused regressions are bound into the audit receipt; altered evidence or another unresolved high/critical finding fails the gate.

Architecture revision 9f6cc425 installed the separate protected docs reset controller on the existing integration host. Docs revision 4cd5f8ea passed pipeline 2908731679 and publication verification. Its authenticated deployment receipt proves that activation was disabled and its source SHA cleared after deployment; the controller seal remains bound to the deployed source. Production deployment is not established by these receipts.

The production admin runtime contract binds browser application 538909745 to EU account 8573192, with protected file and checksum variables verified by read-back. Its controller environment, endpoint and audience are configured. Backend, architecture and admin revisions are now promoted and verified in production. Mobile revision d836145f was promoted by fast-forward merge request !6 after both its integration and merge-request checks passed. Production root 2911478424, Android release 2911484352 and iOS release 2911483906 all succeeded. Signed Android 1.1.16+2000000997 is uploaded to the production internal track; signed iOS 1.1.16 (997) was observed as VALID in the production TestFlight app.

Empty panels are not evidence of application health or crash-free operation.

Environment Application prefix Dashboard Alert policy
Integration mamnajem-integration- mamnajem-integration-observability mamnajem-integration
Production mamnajem-production- mamnajem-production-observability mamnajem-production

Application suffixes are backend, admin, mobile-android and mobile-ios. The integration host is mamnajem-integration-api. Open the private integration overview for the verified backend and host surfaces. Production rollout remains manual.

New Relic alert delivery goes directly through an environment-filtered workflow and secure webhook destination to the existing Discord channel. Notifications cover open, acknowledged and resolved issues, respect fully muted issues and disable Discord mentions. The integration notification test was delivered.

From architecture, inspect then apply the environment configuration:

node scripts/configure-newrelic-discord.cjs --environment integration
node scripts/configure-newrelic-discord.cjs --environment integration --apply
node scripts/configure-newrelic-observability.cjs --environment integration
node scripts/configure-newrelic-observability.cjs --environment integration --apply
node scripts/configure-newrelic-aws.cjs
node scripts/configure-newrelic-aws.cjs --apply

The commands reconcile named resources and refuse ambiguous duplicates. The Discord helper reads the existing AWS webhook secret without printing it. Add --test only when one explicitly labelled Discord notification is intended. Preparing New Relic production resources does not deploy production applications.

Private credentials live in ~/.config/mamnajem/newrelic.env, owned by the operator with mode 0600. NEW_RELIC_LICENSE_KEY is the ingest key for agents; NEW_RELIC_API_KEY is the User API key for NerdGraph and symbol uploads. CI names the latter NEW_RELIC_USER_API_KEY. Mobile app tokens are platform-specific, protected environment inputs. Never put private keys in source or terminal output.

Useful integration queries:

FROM SystemSample SELECT count(*) WHERE entityName = 'mamnajem-integration-api' SINCE 20 minutes ago
FROM Log SELECT count(*) WHERE environment = 'integration' OR `entity.name` = 'mamnajem-integration-backend' SINCE 20 minutes ago
FROM Log SELECT count(*) WHERE environment = 'integration' AND service = 'postgres-backup' SINCE 30 hours ago
FROM Metric SELECT latest(mamnajem.backup.success) WHERE environment = 'integration' SINCE 30 hours ago
FROM Transaction SELECT count(*) WHERE appName = 'mamnajem-integration-backend' SINCE 20 minutes ago
FROM Span SELECT count(*) WHERE appName = 'mamnajem-integration-backend' SINCE 20 minutes ago
FROM MamNajemMetric SELECT latest(value) WHERE environment = 'integration' AND metricName = 'health' SINCE 20 minutes ago

Health values are 3=UP, 2=OUT_OF_SERVICE, 1=DOWN, 0=UNKNOWN. Host and backend health alerts detect loss of a previously received signal after five minutes; the daily backup allows 30 hours. Loss-of-signal detection needs an initial sample, so validate first receipt as part of deployment.

The dashboard covers requests, latency, errors, mobile crashes, admin errors, logs, spans, host CPU/memory/disk, backup/deployment and account ingest. The integration paging policy uses critical thresholds only. The integration host memory condition is disabled; host memory remains visible on the dashboard. The account-wide ingest budget alert uses NrMTDConsumption and pages above 80% of the reported monthly free allowance: above 80 GB while the allowance is 100 GB. It divides the latest monthly GigabytesIngested by the reported freeLimit; hourly release/E2E bursts alone do not page. Consumption is reported hourly and can lag by approximately three hours; the 10-minute event timer can further delay notifications. This leaves 20% allowance headroom and does not impose a billing hard cap. See New Relic usage queries. The owner confirmed the Free plan with no payment method in the account UI; the API cannot independently verify that billing state. Continue checking calendar-month usage and billing settings separately; consumption reporting can lag incoming data.

Mobile diagnostics require consent. Session replay, automatic HTTP-body capture, web-view instrumentation, mobile log forwarding, offline storage and background reporting are disabled. Diagnostic messages and stack URLs are sanitized; opt-out stops the native agent and clears identity. Re-enable after shutdown requires an app restart. CI binds JavaScript maps, Android mappings and iOS dSYMs to the exact release artifact before store gates. An opted-in device crash with readable symbols remains required runtime evidence.

The admin agent disables cookies, replay, automatic Ajax collection, session traces and log forwarding. It sanitizes errors, uploads source maps to the EU endpoint, reads them back by release and URL, and excludes maps from public ZIPs. The Java agent obfuscates SQL, strips exception messages, excludes sensitive attributes and limits logs/spans. Host samples run every 60 seconds and process samples are disabled. Avoid forwarding Java logs again through a second collector.

Cutover verification covers deployed backend transactions, spans, logs and custom metrics, browser errors, native Android crashes and Discord delivery. Production ECS, ALB, RDS and Lambda coverage comes from the linked New Relic AWS integration; the integration Lightsail agent covers its own host. The production ingest secret and execution-role grant are provisioned and the backend rollout is verified. Historical CloudWatch logs and New Relic events are retained. Platform Lambda and RDS logs remain in AWS; cloud polling supplies their metrics and inventory rather than forwarding all platform logs. Physical iPhone production crash verification is deferred by the owner.

RDS creates /aws/rds/instance/mamnajem-production-postgres/postgresql itself when log export starts, with no expiry. Terraform does not manage it because the guarded apply rejects imported resources, so its 30-day retention was set directly with aws logs put-retention-policy on 2026-11-08. A recreated database or log group needs the same command again.

Manual production preparation

architecture/terraform/aws-production now declares the retained mamnajem-production-new-relic-license-key secret container. Its value is never managed by Terraform. A dedicated new_relic_ingest inline policy grants the existing execution role access to this secret only, leaving its existing runtime secret grant unchanged during monitoring preparation. new_relic_enabled defaults to true after authorized provisioning of its AWSCURRENT value outside Terraform. Use the new_relic_license_secret_arn output for protected backend NEW_RELIC_LICENSE_SECRET_ARN.

new_relic_aws_polling_enabled now defaults to true, preserving the applied role. The authorized scoped apply created a metrics and inventory role with no credential or application data access. Its trust is restricted to New Relic AWS account 754728514883 and external ID 8573192. Link new_relic_polling_role_arn in New Relic EU, enable ECS, RDS and ALB polling at 300 seconds or longer, then verify received service samples before adding their alerts. Polling still uses AWS CloudWatch metrics and may incur AWS API charges; New Relic Free does not remove those charges.

The owner authorized production deployments on 2026-10-04. The reviewed preparation was applied as four creates: the ingest-secret container, its exact execution-role grant, the New Relic polling role and its metrics policy. The license was provisioned privately into AWSCURRENT; protected backend NEW_RELIC_LICENSE_SECRET_ARN points to that container. No compute capacity, updates, replacements or deletions were included.

A separate guarded one-create apply adds read-only Lambda inventory permissions for the retained Cognito functions. The IAM-role mirror normalization accepts only its inline-policy projection and null-to-empty tags; trust, role name, session limits and all other attributes remain exact. Lambda polling is scoped to functions tagged Application=mamnajem in eu-west-1, covering both environments. Linked New Relic cloud account 71700 polls ECS, RDS, ALB, Lambda and SNS metrics every 900 seconds and inventory every 3,600 seconds. Each poll is a billed CloudWatch GetMetricData read: at 300 seconds it cost about 14 dollars a month, so on 2026-11-08 polling moved to 900 seconds (about a third of that). The alert windows for these metrics are 900 seconds to match, so AWS-metric alerts react up to 15 minutes later; ALB unhealthy targets needs three consecutive polls, while a full outage still pages on the first poll. SNS extended inventory is disabled. Its metadata permissions are restricted to the two MamNajem alert topics. Production ECS/RDS/ALB metrics and Cognito function inventory have been received; dormant functions need no invented zero metrics.

Production Discord workflow delivery was tested successfully. The production admin build, asset, headers and authenticated activation reset are verified. Backend revision c953f0cc passed integration after a production packaging failure: CI now transfers the New Relic agent directory with the JAR. A local Docker build using the transferred-artifact layout passed, and all 139 CI helper tests passed. Container vulnerability scanning then blocked 13 high and one critical finding in Jackson 2/3, Bouncy Castle and SCRAM. Revision 34c77724 patches those dependencies; its full local Maven gate passed 6,660 tests with zero failures, API compatibility passed and the patched container scan returned zero high or critical findings. Production pipeline 2910968044 subsequently passed deployment and runtime verification. New Relic receives transactions, spans, logs, health (3=UP) and business metrics from the production backend. Instrumented startup took approximately 164 seconds on the existing 0.5 vCPU/1 GB tier. ECS health-check grace is now 300 seconds; desired count remains one and the deployment circuit breaker still rolls back failures. Architecture revision ad7ac8a4 passed both integration and production CI.

Signed production Android build 1.1.16+2000000973 and its New Relic diagnostic artifacts passed verification and the build was uploaded to the production internal track. Owner Google access allowed the missing production publisher quota and Remote Config permissions to be repaired through a scoped four-create IAM apply. Both environments' IAM grants were read back; no compute resources were created. The production iOS build 1.1.16(973) passed signing and diagnostic verification after its scoped Firebase Analytics setting was corrected and derived-data cache was cleared. Its source map and dSYM upload passed, and TestFlight upload succeeded, and Apple processing returned VALID. Integration uses bundle pl.mamnajem.integration, App Store Connect app 6801274157; production uses pl.mamnajem, app 6801273572. Both branches deliver to their own TestFlight app. Release observation checks the exact processed TestFlight build. Public required-update publication is a separate opt-in action requiring the exact publicly distributed build before changing Firebase Remote Config.

Mobile revision d836145facc87f169787f75b6c3d7dfee451bc44 adds that production publication distinction. Its local gate passed lint, types, formatting, 5,397 application tests, 1,189 general CI helper tests (three expected skips), and 443 Android helper tests (one expected skip). Its integration root 2911196168, workload 2911399249, Android release 2911407696 and iOS release 2911407698 all passed, including the current 223-selector device inventory. The exact revision then passed merge-request checks and was promoted by fast-forward MR !6. Production root 2911478424, Android release 2911484352 and iOS release 2911483906 all passed. Both signed production platforms delivered build 997: Android to pl.mamnajem internal testing and iOS to the separate production TestFlight app, with Apple processing VALID. Their New Relic source-map checks passed; iOS also passed its exact dSYM check. Public store launch and mandatory-update publication remain separate guarded actions. The production Android runtime crash and Discord delivery are verified above. Physical iPhone production crash reporting and readable-stack verification are deferred by the owner.

The alert retirement adds no compute capacity or secret containers. It reuses the account ingest secret and the existing event-forwarder Lambda. The earlier ingest-secret container retains its AWS storage charge, and cloud polling, CloudWatch metrics/logs, SNS and Lambda use can still incur AWS charges. All 49 billed CloudWatch alarm resources are removed, rather than merely disabling their actions. Their historical log data and source metrics remain; deleted alarm resources no longer expose their alarm history.

The forwarder can decrypt the ingest secret through Secrets Manager only; its KMS grant is restricted to that secret encryption context and the existing key.

New Relic subscription cost remains $0 within the Free plan's 100 GB monthly allowance. On October 5, observed collection suggested 30–50 GB per month, with about 40 GB as the planning estimate; sustained release/test bursts would increase this. Check actual calendar-month consumption. Free-plan ingestion can stop at the allowance, so usage alerts provide warning rather than a billing cap or a guarantee of uninterrupted monitoring.

References: New Relic pricing, infrastructure log forwarding, NRQL alerts.

Retained AWS collection and historical data

Applications and alerts use New Relic. Historical New Relic projects and events remain available, while their application configuration is removed. CloudWatch service metrics remain the AWS polling source, and log groups retain their configured history. All MamNajem CloudWatch metric alarm resources are removed. The sections below describe the current investigation paths.

Primary Alert And Error Surfaces

Service/tool What it catches What to look for
New Relic errors Backend exceptions, admin JavaScript errors, mobile crashes, handled mobile errors and release regressions. Issues by environment, release, route, workflow, exception type, requestId, correlationId, and spawnId where present.
AWS CloudWatch Logs Backend logs, Caddy logs, PostgreSQL logs, system logs, Lambda logs, ECS logs. Request failures, startup errors, webhook failures, DB errors, proxy errors, and tracing IDs in backend application logs.
AWS CloudWatch Metrics Health, route metrics, Stripe webhook metrics, deploy metrics, ECS/ALB/RDS metrics. 5xx spikes, latency, unhealthy targets, dead webhooks, retry age, deploy failures.
New Relic alerts Alert rules over collected application, host and AWS signals. Incidents, condition queries, thresholds, loss of signal and notification delivery.
AWS event forwarder / New Relic workflow Existing Lambda forwards safe SNS infrastructure events into New Relic; the New Relic workflow sends Discord notifications. MamNajemAwsEvent, Lambda failures, SNS failures and New Relic notification status.
AWS SNS Infrastructure event delivery path. Terraform and backend config also contain Android SNS mobile-push resources, but current backend notification publishing uses ExpoPushSender. Failed infrastructure event delivery or event-forwarder subscription issues. Do not assume app push uses SNS without checking current code.
GitLab CI/CD Build, test, deploy, release, and security failures. Failed jobs in Maven, tests, dependency check, Gitleaks, Semgrep, deploy, Android release, iOS release, Terraform plan/apply.
Spring Actuator Backend health endpoint. /actuator/health, readiness after deploy, missing health response.
Docker/ECS health checks Container/backend process health. Unhealthy containers, ECS task restarts, ALB target health failures.
Docker Compose on the integration laptop Integration backend, PostgreSQL and Caddy containers; deployer state per component. Container restarts, docker compose logs, failed JAR startup or readiness, failed or rolled-back deployer runs.
Stripe Dashboard Webhook delivery, payment errors, disputes, checkout/payment state. Failed webhook attempts, payment intent failures, disputes, Connect issues.
Firebase Analytics / DebugView Mobile and web product telemetry. Missing events, failed workflow events, required-update events, privacy-safe event properties.
Firebase Remote Config Required app update config behavior. Bad update rules, unexpected app blocking, stale minimum version config.
Google Play / App Store Connect Mobile release, in-app purchase, TestFlight/internal release problems. Rejected builds, failed uploads, subscription/product configuration issues.
PostgreSQL logs / DB metrics Query, runtime, and database health failures. Connection exhaustion, disk/storage pressure, slow or failed DB operations.
Caddy logs Integration reverse proxy behavior. TLS/domain/proxy failures, edge 4xx/5xx responses.

Alert Flow

Infrastructure event alert:

ECS deployment failure or unexpected stopped task
-> EventBridge rule
-> existing SNS alert topic
-> existing event-forwarder Lambda
-> New Relic MamNajemAwsEvent
-> New Relic alert condition and environment workflow
-> Discord notification

Collected metrics and logs alert directly through New Relic conditions. NrIntegrationError with category=NotificationError identifies destination failures; check NrAiNotification for notification status. If Discord is unavailable, its failure remains visible in New Relic.

Application error alert:

Application exception or crash
-> New Relic error or mobile crash event
-> New Relic issue/alert condition
-> investigate by environment, release, route, workflow, breadcrumbs, and tracing IDs

GitLab pipeline alert:

Push, schedule, release, or manual job
-> GitLab pipeline
-> failed stage/job
-> inspect whether failure is test, contract, security, deploy, release, or infrastructure

Stripe operational issue:

Stripe event/payment/dispute
-> Stripe webhook delivery
-> backend unified webhook flow
-> backend logs, New Relic metrics and errors, DB state

Request Tracing IDs

The backend and mobile app use three privacy-safe HTTP tracing identifiers:

Header / field Meaning Where to look
x-request-id / requestId The normalized ID attached to the current backend request. The backend currently aligns it to the accepted correlation ID. Backend response headers, backend logs, New Relic context, mobile API telemetry.
x-correlation-id / correlationId The logical workflow trace. Mobile sends it on API requests; backend accepts a safe incoming value, falls back to x-request-id, or generates one. Mobile API telemetry, backend logs, New Relic, outbound backend WebClient calls.
x-spawn-id / spawnId A per-client or per-backend spawned operation ID for separating related calls inside the same correlated workflow. Mobile API telemetry, backend logs, New Relic, outbound backend WebClient calls.

Backend handling:

  • Incoming IDs must match the safe ID format before being reused.
  • Missing or unsafe IDs are replaced with generated backend IDs.
  • The accepted IDs are returned as response headers.
  • The IDs are added to MDC/log context and Reactor context.
  • Injected backend WebClient.Builder instances propagate the IDs on outbound calls.
  • Diagnostic privacy filtering allows safe tracing IDs, but still scrubs raw user, payment, lease, property, tenant, message, token, and secret data.

Mobile handling:

  • API requests include x-request-id, x-correlation-id, and x-spawn-id.
  • API success and failure telemetry includes requestId, correlationId, and spawnId.
  • Diagnostic privacy filtering treats safe tracing identifiers as diagnostic metadata.

What Each Tool Is Best For

Tool Best question
New Relic errors What exception happened, in what release, on what route, workflow, or tracing ID?
CloudWatch Logs What did the backend, proxy, database, Lambda, or host actually log for the tracing ID or timestamp?
New Relic metrics/alerts Is the system unhealthy, degraded, or trending badly?
GitLab CI/CD Did tests, deploy, security, or release fail before users saw it?
Stripe Dashboard Did Stripe send the event, and what is the payment/subscription/dispute state?
Firebase DebugView Is mobile telemetry being emitted correctly by an integration build?
Cognito Is login, signup, token, redirect, or identity-provider configuration broken?
SNS/event forwarder and New Relic notifications Did infrastructure events reach New Relic and did alerts reach Discord?
Expo Push sender Did backend push publication to Expo fail, and did the stored token format match what the sender expects?

Important Names

Item Value
Integration API https://api.mamnajem.win/v1
Integration web https://www.mamnajem.win
Integration admin https://admin.mamnajem.win
Integration docs https://docs.mamnajem.win
Integration web app (web-frontend) https://app.mamnajem.win
Production API https://api.mamnajem.pl
Production web https://mamnajem.pl
Backend health /actuator/health
Stripe webhook /v1/webhooks/stripe
Integration CloudWatch namespace MamNajem/integration
Production CloudWatch namespace MamNajem/production
Integration backend runtime backend service of the mamnajem-integration Docker Compose project on the integration laptop

First Checks By Area

Backend/API issue:

  1. Check GitLab pipeline for a recent failed deploy or test.
  2. Check /actuator/health.
  3. Check New Relic alerts and backend logs.
  4. Check New Relic backend issues by environment, release, route, and tracing IDs.

Payment issue:

  1. Check Stripe Dashboard event/payment state.
  2. Check Stripe webhook delivery status.
  3. Check backend webhook logs.
  4. Check New Relic Stripe webhook metrics and dead-webhook conditions.
  5. Check database payment attempt/request state.

Mobile issue:

  1. Check New Relic mobile errors or crashes by release and environment.
  2. Check Firebase DebugView for integration builds.
  3. Check API request telemetry and backend logs by requestId, correlationId, or spawnId.
  4. Check app environment variables and telemetry-disabled flags.
  5. For integration URL problems, verify whether the failing build was configured from the mobile .pl defaults or the architecture .pro web/API values.

Auth issue:

  1. Check Cognito user pool/client configuration.
  2. Check redirect/logout URLs.
  3. Check Amplify configuration.
  4. Check backend auth errors and New Relic.

Deploy issue:

  1. Check the GitLab deploy job.
  2. Check CloudWatch DeploySuccess metric where applicable.
  3. Check /actuator/health.
  4. For integration, check the laptop deployer state for the commit and the backend container's readiness and logs.
  5. For production, check ECS service deployment and ALB target health.

Backend ERROR alert coverage

Both environments have enabled critical Backend ERROR logs conditions, scoped to entity.name = mamnajem-<environment>-backend and level = ERROR. One matching forwarded log qualifies; the 60-second aggregation is evaluated with AT_LEAST_ONCE in a 300-second duration. Five minutes without a matching signal closes the issue without opening a new silence issue. Stack traces logged at ERROR are covered through the log record; bare stdout/stderr or WARN traces require separate receipt/severity investigation. The APM Backend errors condition excludes expected HTTP 4xx responses while keeping server/status-less exceptions.

Use only New Relic dashboards, metrics, incidents and notification records plus CloudWatch logs for operational alert audits. Current private access instructions are in the Codex workflow; source and installed skill copies are maintained by the workspace bootstrap.