Product Outcomes reporting¶
Updated: 2026-10-06. Publication is enabled and exactly reconciled in integration and production. New installations default to disabled publication; collection and persisted aggregate calculation are independent of New Relic delivery.
Use the Product Outcomes implementation and review brief for application context, the twelve metric definitions, recurring work, and historical limits. This runbook explains how to operate those definitions without hiding uncertainty or changing development data.
Ownership and source¶
The backend owns evidence, product-outcomes-v2, immutable aggregate runs, and delivery verification. Version 2 separates unknown activation timing in monthly observed work: setup first observed during a month cannot prove first activation in that month; observing it earlier proves setup already existed before the work month. Persisted version 1 results retain their original version and payload. Admin provides audited account/pilot classification and report refresh. Architecture owns private runtime secret references, one private New Relic dashboard, and isolated reporting warning policies. Individual customer investigation remains in admin.
| Repository | Source |
|---|---|
| Backend | db/changelog/0016-product-evidence.sql and 0017-product-report-runs.sql under src/main/resources |
| Backend | ProductReportCalculator, ProductReportSource, ProductReportExport, ProductReportNewRelicClient, ProductReportStore, and ProductReportService under src/main/java/pl/mamnajem/service |
| Backend | src/main/java/pl/mamnajem/controller/AdminProductReportsController.java; config/ProductReportProperties.java |
| Backend | .gitlab/ci/deploy-integration-host.sh: private integration query-secret reference and on-host runtime handoff |
| Backend | .gitlab/ci/deploy-production-ecs.sh, .gitlab/ci/verify-production-product-report-variable-metadata.sh, and .gitlab/ci/product-report-runtime.md: dedicated manual production ECS handoff |
| Admin | src/ProductReports.tsx, src/App.tsx, src/lib/api.ts |
| Architecture | scripts/configure-newrelic-product-report.cjs; terraform/aws-integration/product-report.tf; production product-report.tf, storage.tf, and runtime.tf |
Each environment has a separate managed IAM read policy scoped to its exact query-secret ARN. Integration attaches it to the runtime user without exceeding the inline-policy quota; production attaches it to the task execution role independently of database/payment resources.
Architecture provisioning details also live in its docs/runbooks/newrelic-product-report.md. Paths refer to source; local files or passed tests alone do not certify deployment.
Defaults and publication prerequisites¶
| Runtime input | Default/requirement |
|---|---|
PRODUCT_REPORT_ENABLED |
true in integration/production configuration; enables initial/daily reporting. Local tests disable scheduled reporting. |
PRODUCT_REPORT_PUBLICATION_ENABLED |
false; turn on only after the prerequisites below are satisfied. |
PRODUCT_REPORT_SCHEDULING_ENABLED |
true; tests can disable scheduled reporting independently of aggregate APIs. |
PRODUCT_REPORT_ACCOUNT_ID |
8573192 only. |
PRODUCT_REPORT_REGION |
EU only. |
PRODUCT_REPORT_QUERY_API_KEY |
Owner-authorized private query credential for account 8573192 EU; the current reuse decision is recorded below. |
NEW_RELIC_LICENSE_KEY |
Existing private ingest credential; never use it as the query identity. |
Load the existing private New Relic credentials through architecture privateCredentials(). Transfer the authorized query value directly through private secret storage; keep it outside command arguments, local files, logs, source and Terraform state.
On 2026-10-06 the owner confirmed the New Relic Free plan and explicitly authorized reuse of the existing administrator User API key for backend reporting verification. This replaces the separate-identity requirement for this deployment. The credential retains administrator permissions; runtime queries stay fixed to EU account 8573192. Its value is delivered privately through the two existing mamnajem-{environment}-product-report-query-key Secrets Manager containers, separately from the existing ingest license. No additional user, paid upgrade, custom group, or paid feature is required. The Free allowance is 100 GB/month; live preceding-30-day ingestion was 7.799118108 GB, and effective Default custom-event retention remains 30 days. Never print, commit, or include credential values in Terraform state or evidence.
Before infrastructure reads, run aws sts get-caller-identity with workstation/default credentials and confirm the intended account. Never select unrelated credentials. Integration and production query-secret containers are named mamnajem-{environment}-product-report-query-key; their non-secret reference is Terraform output product_report_query_secret_arn. The integration runtime receives an exact Secrets Manager read grant. Production ECS query-secret injection defaults to disabled and remains a manual rollout decision. The owner-authorized current CI rollout enables it; set Terraform product_report_publication_enabled=true when reviewing a later enabled runtime plan. Populate secret values privately outside Terraform; an empty container is not a working identity.
The integration handoff is implemented in backend/.gitlab/ci/deploy-integration-host.sh. Set the Terraform output's full ARN as a protected, raw, exactly integration-scoped backend CI variable named INTEGRATION_PRODUCT_REPORT_QUERY_SECRET_ARN. The accepted reference is the dedicated mamnajem-integration-product-report-query-key secret in account 550435500873, region eu-west-1, including its six-character Secrets Manager suffix. Other accounts, regions, production secrets, and ingest-key references are rejected. Put only the owner-authorized query credential's text API key in the secret's AWSCURRENT version through private identity management outside Terraform.
INTEGRATION_PRODUCT_REPORT_PUBLICATION_ENABLED defaults to false. After authorized query access, effective retention and the account plan are verified, an integration rollout may set this protected, raw, exactly integration-scoped variable to true. The deploy script then resolves the exact ARN's AWSCURRENT value on the integration host, using the installed mamnajem-integration-runtime AWS profile. Under the existing deployment lock, it installs PRODUCT_REPORT_QUERY_API_KEY in /opt/mamnajem-integration/shared/app.env with ownership root:mamnajem and mode 0640. CI receives the reference only; key values and AWS response bodies remain outside deployment logs. Missing, denied, or malformed secrets fail an enabled deployment before replacing the runtime file. Enabled deployments reread the current secret; if CI omits the reference, the installed reference is retained. With publication disabled, deployment makes no query-secret lookup and clears any installed query key while preserving the reference. The architecture reporting runbook contains the full provisioning and handoff procedure. Both disabled runtime clearing and the owner-authorized enabled handoff are verified. The current integration runtime has publication enabled and exact query reconciliation.
The production handoff is implemented separately in backend/.gitlab/ci/deploy-production-ecs.sh. Its optional inputs are PRODUCTION_PRODUCT_REPORT_PUBLICATION_ENABLED (true or false, default false) and PRODUCTION_PRODUCT_REPORT_QUERY_SECRET_ARN. Both must be protected, raw env_var variables scoped exactly to mamnajem/production, the backend production job's environment. Plain production and wildcard scopes are rejected. When either input is present, verify-production-product-report-variable-metadata.sh checks canonical project 84799688, the protected production branch, exact metadata scope, and matching effective values using private GitLab access. Unavailable or invalid metadata stops delivery without revealing private responses.
When supplying either production input, provision a protected, masked GITLAB_ACCESS_TOKEN in that exact environment with read access to backend project variable metadata. The current rollout reuses the existing backend CI metadata credential in protected, masked and hidden form. The earlier absent-credential preflight failure was resolved without weakening the metadata verifier or creating another token. Hidden variables return redacted API values; verify their metadata and the authenticated job rather than comparing the redacted response to the credential.
An enabled production deployment accepts only arn:aws:secretsmanager:eu-west-1:550435500873:secret:mamnajem-production-product-report-query-key-<six-character-suffix> and injects that reference as ECS secret PRODUCT_REPORT_QUERY_API_KEY. ECS resolves its current version using the task execution role; the deploy script never retrieves the query-key value. A later enabled deployment may reuse one unambiguous installed reference with that exact identity. Missing, ambiguous, or foreign references stop task registration. Disabling publication, including omitting both optional inputs, removes stale query-key environment and ECS secret entries and requires no query-secret permission. Supply the ARN again when re-enabling after a disabled deployment; a supplied malformed ARN is rejected even when publication is false. Keep production Terraform product_report_publication_enabled aligned with the manual CI opt-in. The script preserves the scanned commit image, unrelated environment/secrets, and other containers. See backend .gitlab/ci/product-report-runtime.md for the delivery contract; the owner-approved credential decision and the verified Free-plan/retention checks above satisfy the current rollout prerequisites.
From the architecture repository, inspect definitions without credentials or mutation:
node scripts/configure-newrelic-product-report.cjs --environment integration
Inspect account query access, effective custom-event retention, and ingest usage read-only:
node scripts/configure-newrelic-product-report.cjs --environment integration --inspect
The --inspect helper reports conservative operator confirmation flags, not live backend publication state. Read the current backend result and authoritative New Relic marker to establish runtime status. Before enabling a new deployment, verify the authorized runtime credential's actual account/query access, effective custom-event retention, and the account plan. Reconcile persisted rows against runtime-queryable rows after deployment; operator query success alone does not establish runtime delivery. Record the approved credential provenance without recording its value.
The earlier empty-secret and restricted-group provisioning investigation is superseded by the owner-authorized reuse above. The existing administrator key was verified through NerdGraph; both runtime containers now have an AWSCURRENT version. Free does not support creating custom groups, and the current API-key UI offers User and Ingest keys. The existing publication, exact verification, retry, current-run and non-paging warning contracts remain unchanged.
Classification and historical evidence¶
Existing accounts and intake records are UNKNOWN. In admin, explicitly set REAL, INTERNAL, TEST, or UNKNOWN with an operator reason. Matched pilot participants use their account classification; unmatched intake records use intake classification. Matching alone never classifies an account. Account editing can record controlled OWNER, ASSISTED, IMPORTED, or UNKNOWN setup provenance without changing the preserved first milestones.
The owner account editor also records Trial access evidence as UNKNOWN, GENUINE, or PILOT_EXTENSION with an audited operator reason. Genuine effective native/provider trials and ordinary no-card trials of up to thirty elapsed days count as trial owners, excluding paid-term owners. Explicit PILOT_EXTENSION evidence selects the separate extension category. Extended no-card trials longer than thirty elapsed days remain UNKNOWN until an operator establishes genuine-trial or pilot-extension provenance. Existing records default to UNKNOWN; an account match or a long expiry alone does not establish extension evidence. This reporting assessment labels existing access without changing its entitlement or dates.
First qualification is captured by successful backend profile/role changes under icp-v1. Legacy observations stay FIRST_OBSERVED and unsupported anchors stay UNKNOWN. No operator should invent historical first-ever dates to increase coverage. Valid setup already present at qualification counts immediately; Q7 still requires the full seven elapsed days before entering a conversion denominator. Pending leases remain visible.
Q7 uses qualification-window setup evidence separately from the first global activation. A property or connected setup that was removed before qualification cannot count toward Q7 merely because an earlier first milestone exists. The backend records setup that is valid at qualification at that qualification timestamp, then preserves readiness achieved after qualification through the inclusive seven-day deadline. For example, setup first activated in September and removed before an October qualification does not satisfy October Q7; if usable connected setup exists at the October qualification, it counts then. The first global activation remains September for recurring-work and next-full-month return cohorts. Later qualification-window readiness never resets that global activation anchor.
Legacy account/setup observations remain FIRST_OBSERVED when dependable first-ever tracking is absent. A later profile edit or initial setup scan cannot manufacture a known first qualification, Q7 setup timestamps, or a measurable retention anchor for that history. Such owners remain visible in first-observed, unknown, and unmeasurable populations.
Connected setup uses the same lease-specific rules for ordinary activation, aggregate pilot readiness, and individual pilot evidence. A bank-transfer schedule must select a live bank account owned by the property's owner, using its schedule account or lease payee account; an account from another owner cannot satisfy readiness. Stored schedule/account IBAN evidence can be encrypted, so readiness does not require plaintext decryption. Bank selection and account changes trigger transactional evidence capture. Pilot document readiness additionally requires a LEASE_DOC attached to that same connected ACTIVE lease. A document on an unrelated lease cannot complete another lease's setup; tenant acceptance and documents remain optional for general activation.
Approved monthly work flags come only from successful trusted owner transactions: receipt creation, actual paid/unpaid transitions, handover confirmation, and settlement confirmation. Do not classify admin corrections, automated jobs, tenant work, setup alone, retries, no-ops, failures, or rolled-back writes as owner return. Flags cover the current and twelve preceding Warsaw months; account-linked evidence follows account deletion. Persisted aggregates are independent of New Relic retention.
Payment normalization uses existing processed signature-verified provider evidence without new provider lookups. Currently supported proof is Stripe subscription invoice.paid with one complete positive non-prorated recurring line linked to the current subscription, a known term/environment, and supported legacy in-band or modern paid PaymentIntent evidence, plus available refund/dispute reversal evidence. Manual, one-off, prorated, truncated, multiple-line, or incomplete evidence stays unsupported. Unsupported store or incomplete provider proof remains UNKNOWN. Cancellation of renewal alone does not remove an effective proven paid term.
Schedule, job claims, and admin refresh¶
The backend performs an initial run when reporting is enabled and schedules daily calculation at 03:00 Europe/Warsaw. A bounded read-only REPEATABLE_READ snapshot supplies one database cutoff; the source limit is 100,000 rows per source and sixty seconds. A failed bound must fail the run, not publish sampled data.
The database reserves refresh through the singleton job row under FOR UPDATE with a fifteen-minute job lease. Overlapping admin or scheduled requests coalesce into the reserved run. Results go to immutable product_report_runs. Delivery state, expected/queryable counts, claim/retry times, and exact publication payloads go to product_report_publications. These lifecycles are separate; a retry does not recalculate or rewrite an immutable result.
The latest-results response and publication-status retries preserve the immutable run's stored definition version and environment. A later application deployment cannot relabel an older reporting result with its current definition or runtime environment.
Failed computation-status delivery also has an automatic retry path when calculation produced no immutable result or publication row. The minute-level monitor atomically claims only a current FAILED computation whose retry is due, using a two-minute database lease. It redelivers the same persisted status payload and timestamp with a one-minute delivery bound, then schedules another failed attempt after fifteen minutes. Release and delivery remain guarded by the current run; a new reservation clears obsolete marker-retry state. Missing result/publication rows therefore do not abandon current-status delivery or permit a superseded run to reappear.
Admin uses:
GET /v1/admin/product-reports/latest: current results and definition, cutoff, maturity/coverage, health, and publication status. HTTP 204 means no run exists.POST /v1/admin/product-reports/refresh: authenticated audited request returning HTTP 202 and a run ID. The UI displays acceptance separately from computation/publication completion.
Use Load latest to inspect current persisted results, and Recompute report after classification or other source corrections. The default view shows snapshot/latest periods; the business-period selector exposes the published historical cohorts. Definitions & data quality explains Q7, P, cumulative support, observation maturity, and coverage. An unavailable rate has value=null and available=false; do not render zero as its substitute.
Publication and exact verification¶
Publish only allowlisted MamNajemProductReportSnapshot rows and MamNajemProductReportRun markers. Snapshot context includes runId, revision, snapshotKey, publicationKey, environment, cutoff, definition version, business period, unit, counts/rate availability, and controlled dimensions. No user IDs or hashes, emails, contact details, addresses, encrypted customer content, or arbitrary free text may leave the backend.
The current plus twelve preceding business months are republished with the run's current event timestamp and explicit historical periods. These are cohort/month snapshots, not a time series of past ingestion. New Relic accepts timestamps only within 24 hours of its server time and does not store null attributes; availability is therefore explicit. The payload is bounded and compressed. See the custom-event requirements.
HTTP acceptance is only the start of asynchronous delivery. The backend queries account 8573192 EU using the separate runtime identity and compares every expected logical key and allowlisted value with the persisted snapshot. It polls every thirty seconds for up to ten minutes. Missing keys, extra/duplicate logical keys, conflicting values, and reconciliation differences fail verification. Identical delivery retries do not create a different logical result. Verify the authoritative current marker after row reconciliation. See the Event API documentation.
Publication work uses an eleven-minute database claim. Retry failed publication every fifteen minutes only while the run remains current. Reuse the exact stored snapshot payload and timestamp; do not regenerate them from ingestion time. Superseded runs cannot publish an authoritative current marker. After the timestamp is older than 24 hours, mark the current publication INVALID instead of resending an unsupported timestamp; await the next daily run or request a fresh run.
Dashboard configuration and current status¶
After reviewing the architecture definition, the authorized configuration step creates/updates the private dashboard and the selected environment's warning policy:
node scripts/configure-newrelic-product-report.cjs --environment integration --apply
This command does not enable backend publication or populate runtime secrets. Production preparation uses its environment argument only during the authorized manual rollout. No production application deployment follows automatically.
The single private MamNajem Product Outcomes dashboard has Overview and Definitions and data quality pages. Its environment selector defaults to production; integration is available for verification. Snapshot widgets ignore the global time picker and use explicit business periods. Tables/bars show cohort and month results rather than treating ingestion timestamps as historical activity.
Live apply configured this one PRIVATE, two-page dashboard, GUID ODU3MzE5MnxWSVp8REFTSEJPQVJEfGRhOjI3NjI4Mzk, for both environments. Integration non-paging reporting policy 1744041 has conditions 15850430 / 15850431; production policy 1744042 has conditions 15850440 / 15850441. Publication is now enabled in both runtimes and all eleven live query widgets return current verified data for each environment. The exact-row and authenticated admin evidence appears below. New Relic UI visual inspection remains unverified because the available browser profiles redirect to sign-in.
Snapshots carry publicationKey=runId|VERIFIED; run markers carry runId|status. Select the latest current marker first, then use it for every snapshot query. For example:
SELECT latest(value)
FROM MamNajemProductReportSnapshot
WHERE environment = 'integration'
AND environment IN (
SELECT latest(environment) FROM (
SELECT latest(value) AS publicationEnabled,
latest(environment) AS environment
FROM MamNajemMetric
WHERE environment = 'integration' AND metricName = 'productReportEnabled'
) WHERE publicationEnabled = 1 SINCE 5 minutes ago
)
AND publicationKey IN (
SELECT latest(publicationKey)
FROM MamNajemProductReportRun
WHERE environment = 'integration' AND isCurrent = true
SINCE 30 days ago
)
FACET snapshotKey
LIMIT MAX
SINCE 30 days ago
The run-status widget uses the same newest marker without filtering to VERIFIED first. Snapshot and status queries also require the newest independent productReportEnabled heartbeat within five minutes. Disabled or missing enablement makes retained VERIFIED events unavailable. The runtime availability widget shows the latest flag before filtering it. Metric tables retain unavailable rows and their counts, including zero denominators. A current PENDING, FAILED, INVALID, or DISABLED marker matches no valid snapshot rows; an older successful run cannot silently reappear. Invalid current results have unavailable metrics. The definitions page includes unavailable/pending rows and expected/queryable comparison.
Non-paging reporting warnings¶
The existing minute-level monitoring loop supplies freshness/failure-age gauges even between daily reporting runs. config/NewRelicMetricsConfig.java allowlists all three reporting gauges explicitly; other unapproved metric names remain denied. While publication is disabled, the age gauges are zero. Do not derive freshness only from a once-daily event that might never arrive.
Freshness uses the most recent verified publication across runs. Before any publication has verified, it ages from the persisted observation start, so daily replacement of an unverified run cannot keep the freshness gauge at zero. The database also preserves a continuous failure streak across new run claims and failed retries. Reserving another run or retrying publication does not reset that streak; successful verification clears it, and disabled publication clears reporting failure state and suppresses the age gauges.
When the authoritative persisted failureSince exists, the failure-age gauge uses that timestamp exclusively. The run's earlier computation cutoff is a fallback only when no recorded failure timestamp exists; it cannot make a newly observed failure trigger the fifteen-minute warning prematurely.
| Backend gauge | MamNajemMetric.metricName |
Warning |
|---|---|---|
product.report.last.verified.age.seconds |
productReportLastVerifiedAgeSeconds |
Above 108000 seconds, thirty hours. |
product.report.failure.age.seconds |
productReportFailureAgeSeconds |
Above 899 seconds, fifteen complete minutes. |
product.report.enabled |
productReportEnabled |
Enablement evidence, no condition. |
Policies are mamnajem-{environment}-product-reporting-warnings with WARNING terms. They create no notification workflow or destination. Configuration refuses attachment if an enabled workflow could match the policy; existing environment-specific operational Discord routing remains separate. Report job/publication failures at WARN without incrementing customer-facing failure counters. Do not alert on adoption or retention percentages.
Failure handling¶
| Observation | Response |
|---|---|
| No report / HTTP 204 | Check reporting enablement, initial/scheduled job, and database claim before requesting refresh. Do not invent metrics. |
| Source bound/time failure | Preserve data, inspect the source/error and job claim, then request a fresh run after correction. Do not silently raise bounds or sample. |
| HTTP ingest accepted but rows not queryable | Allow the bounded polling window; inspect query permission and exact key/value comparison. Acceptance is not verification. |
| Missing/conflicting rows or failed reconciliation | Preserve immutable result and exact payload; expose FAILED/INVALID current status and unavailable data. Compare the same definition/cutoff, not a different report. |
| Current failed run with valid timestamp | Retry after fifteen minutes under the publication claim; preserve payload/timestamp. |
| Failed computation without a result/publication row | The minute monitor retries its persisted status marker after fifteen minutes under the dedicated current-run lease. Preserve its payload/timestamp; correct the source problem and request a fresh calculation separately. |
| Superseded retry | Stop its current-marker publication. Verify the newer current run. |
| Timestamp older than 24 hours | Invalidate without ingest and await/request a new reporting run. |
| Account plan, retention, or authorized runtime query access unconfirmed | Keep publication disabled; collection and backend aggregate history can continue. |
| Restricted reporting group creation denied on Free | Use the explicitly authorized existing-identity handoff recorded above. Creating a second key for the same user does not reduce its permissions. A future separate-identity rollout requires its own approved account/login and private key handoff. |
| Enabled integration secret lookup missing, denied, or malformed | Deployment fails before replacing the runtime file. Check the exact protected integration ARN, AWSCURRENT text value, and runtime Secrets Manager grant privately; keep publication disabled until the prerequisites are verified. |
| Enabled production metadata or installed secret reference invalid | Stop before task registration. Check protected raw variables scoped exactly to mamnajem/production, their effective values, the dedicated production ARN, and task execution-role permission privately. Keep publication disabled until prerequisites are verified. |
| Freshness/failure warning | Inspect current persisted state, WARN logs, New Relic rows, and CloudWatch backend logs through the existing monitoring workflow. |
Do not reveal an older healthy publication to hide a current failure. Never print credentials in diagnoses. Unsupported history remains unavailable. This runbook authorizes no destructive reset, data deletion, or arbitrary backfill.
Delivery and acceptance evidence¶
Run backend mvn verify, admin checks/build and focused tests, architecture build/tests, and documentation checks/build. Exercise qualification/connected setup boundaries, trusted owner work and rollback/no-ops, Warsaw/DST/maturity, paid proof and reversals, pilot deduplication/offers/support coverage, concurrent claims, ingestion delay, key/value mismatch, supersession, invalid-current status, allowlisting, and admin authorization.
Current verification uses unrestricted access and includes actual database-backed backend scenarios. Backend mvn verify passes: 6,790 tests with zero failures, errors, or skips, coverage passed, and SpotBugs reported zero findings. Final evidence is /tmp/mamnajem-product-activation-timing-full-verify-2.log, terminal BUILD SUCCESS after 5 minutes 15 seconds.
Architecture build and all 41 suites / 451 tests pass. Admin local verification passes 232 tests (149 UI, 49 Node, 34 Python), the integration fixture build, and build-output checks. Docs pass all 98 tests in a canonical CI checkout with matching source, modes, dependencies, and workspace layout; strict build, changed-page rendering, and artifact identity scan also pass. The original workspace remains 97 / 98 only because preserved refs/heads/codex/remove-mobile-stripe-20261005 violates the scanner's explicit active-ref namespace policy. Its existing regression confirms that policy; no scanner weakening or branch deletion/rename was performed. Current docs check evidence is /tmp/mamnajem-product-docs-final-check-7.log; deployment pipelines also run the strict build and exact artifact verification.
All 162 backend CI-script Python tests also pass, including the production delivery metadata and secret-reference cases. Architecture evidence is /tmp/mamnajem-product-activation-timing-architecture-test.log and /tmp/mamnajem-product-activation-timing-architecture-build.log; CI-script evidence is /tmp/mamnajem-product-final-ci-python.log.
Socket-free PostgreSQL 16 verification also applied all 17 current migrations and passed 126 SQL assertions across 41 groups, with no ERROR, FATAL, or PANIC entries, recorded in /tmp/mamnajem-product-pg-delivery-evidence.txt. It covers qualification-window versus global activation, legacy uncertainty, bank/account selection, matching lease documents, payment evidence, continuous reporting failure state, and failed computation-marker retry claims. These assertions supplement the restored database-backed test phase.
Publication was enabled on 2026-10-06 after the owner confirmed the Free plan and explicitly authorized reusing the existing administrator User API key. Both managed runtime query-secret containers now have an AWSCURRENT version. The existing backend CI metadata credential is protected, masked and hidden in the exact production environment; the metadata check remains enabled. No new New Relic user, custom group, paid upgrade or paid secret-store service was created.
The first enabled acceptance runs use product-outcomes-v2, with 524 expected and 524 queryable rows, exact reconciliation and no reporting failure state:
| Environment | Run | Revision | Cutoff (UTC) |
|---|---|---|---|
| Integration | ba8a9b82-daaf-496a-a316-e1fa6eb16f19 |
8 | 2026-10-06T07:49:28.616486Z |
| Production | d92c6108-143d-4c40-99d3-1cd655584424 |
4 | 2026-10-06T07:56:19.202590Z |
These immutable acceptance runs were produced by backend 29e6b74ad8ff0903ce2c731a8014024299ae667e. Integration's exact deployment retry succeeded through job 16960487050 in pipeline 2915601096. Production's exact deployment retry succeeded through job 16960582118 in pipeline 2915666448, followed by successful deployment verification pipeline 2916691643. ECS task definition 20 completed its rollout with one desired/running task, no pending task, publication enabled and the private query-secret reference installed. The checked production startup stream contains no application ERROR or reporting WARN. Later daily or authorized refresh runs have new run IDs; inspect the current authoritative marker rather than treating this evidence table as the current run selector.
Authenticated integration admin results match the persisted report: 13 history months, 13 unknown-activation-timing rows, and 372 UNKNOWN owners. An independent comparison matched all 524 exact snapshot keys and 13,624 allowlisted attribute values against the backend API result. The runtime's automatic exact key/value comparison independently verified both environments. Both enabled heartbeats and zero reporting-failure ages are queryable. All eleven live dashboard NRQL widgets in each environment return results and use the current VERIFIED run; the private dashboard remains PRIVATE and its query widgets ignore the global time picker.
The available New Relic browser profiles redirect to sign-in. Configuration and all 22 live widget queries are verified through NerdGraph, but visual inspection inside the authenticated New Relic UI remains unverified. This does not require another API key. Authenticated admin report/definitions rendering was verified independently. Unsupported historical anchors remain unavailable, and existing UNKNOWN accounts require explicit operator classification before contributing to REAL adoption metrics.
Deterministic TEST fixture acceptance also verified preserved first qualification after profile retries, PENDING-to-ACTIVE connected setup, usable recurring RENT, successful owner receipt activity, idempotent receipt retry, tenant denial and exclusion from REAL metrics. Supported app deletion and Cognito cleanup removed the disposable fixtures and restored the original population. Initial disabled-publication runs established collection and admin behavior; the enabled acceptance above supersedes those earlier delivery limitations.
Deliver on integration through the guarded push workflow, monitor every pushed pipeline to terminal state, and verify affected deployed workflows and this published page. Record the run ID, definition, cutoff and exact reconciliation without customer identifiers. Production remains a manual fast-forward rollout. No destructive backfill or database reset is part of this delivery.
For production documentation, first confirm that the exact integration candidate can fast-forward production; stop on divergence. Promote only through the authorized guarded workflow. Create a pipeline on protected production with input activation_requested=true and activation_request_sha equal to the full candidate SHA. After brand_residual_barrier and docs:build pass, run the blocking manual deploy_production_docs job. The job requires MAMNAJEM_ACTIVATION_ENABLED=true, matching candidate values in MAMNAJEM_ACTIVATION_SHA and MAMNAJEM_SEALED_RELEASE_MATRIX_SHA, the checksummed MAMNAJEM_PRODUCTION_ARCHITECTURE_CONTRACT_FILE and MAMNAJEM_PRODUCTION_ARCHITECTURE_CONTRACT_SHA256, and working production reset-controller inputs. It deploys the verified artifact using its OIDC role, waits for Amplify, checks https://docs.mamnajem.pl/.mamnajem-build.json against that SHA, and resets activation after completion. Verify this published runbook and the changed review brief separately from the build marker.