Service Incident - December 11, 2025 - Support | All Pods - API Usage dashboard was down

Summary

From December 11, 2025 at 14:37 UTC until December 15, 2025 at 16:40 UTC, customers using the Admin Center API Usage dashboard were unable to view API usage data for a specific time period.

Timeline

December 11, 2025 15:41 UTC | December 11, 2025 07:41 PT

We have confirmed an issue affecting the API usage dashboard, causing it to not refresh as expected. Our engineers are engaged and actively working to restore functionality. We apologize for any disruption and will provide updates as they become available.

December 12, 2025 02:43 UTC | December 11, 2025 18:43 PT

We have identified data ingestion issues impacting the underlying service that supports the API Usage Dashboard. Our teams are working through options to restore the dashboard functionality. We will provide another update when we have more to share.

December 12, 2025 14:24 UTC | December 12, 2025 06:24 PT

Our engineering team has deployed a fix for the delays impacting data refresh on the API dashboards. Due to the large volume of data, it may take some time for all information to update fully. We appreciate your patience as we work to resolve this promptly and will keep you informed with updates as they become available.

December 15, 2025 04:02 UTC | December 14, 2025 20:02 PT

We would like to provide an update regarding the resolution of the delays affecting data refresh on the API dashboards. Our engineering team is actively coordinating and finalizing the necessary steps to address this incident. While the majority of the data is now current, some files still require reprocessing. We apologize for any inconvenience this may have caused and appreciate your patience and understanding.

December 15, 2025 17:09 UTC | December 15, 2025 09:09 PT

Our engineering team has completed a backfill of missing data to resolve the data refresh problems with the API dashboards. Data from December 4, 2025, 01:00 UTC through December 5, 2025, 08:00 UTC is unfortunately lost and cannot be retrieved. We appreciate your patience during this process.

Root Cause Analysis

The incident was caused by the unintentional stoppage of the data feed that the API Usage dashboard relies on while decommissioning a legacy data process as part of planned end-of-life work. The legacy system’s output was still being used behind the scenes by our newer analytics platform, and that dependency wasn’t visible during the change.

Resolution

To resolve the issue, we restored the data feed and backfilled the missing period so usage data reappeared in the dashboard. However, due to data retention limits in our logging system, data between December 4, 2025, 01:00 UTC and December 5, 2025, 08:00 UTC could not be backfilled.

Remediation Items

Restore event routing service

Improve API Usage pipeline observability

a. Add a verification check for up to date data on the API Usage pipeline.

b. Enable freshness checks using our Data Quality system.

Build the API Usage dataset via the upgraded data system