Service Incident - November 14, 2024 - Network errors accessing Explore dashboards and reports

SUMMARY

On November 14, 2024 from 14:15 UTC to 23:20 UTC, customers using Explore experienced network errors accessing reports and dashboards.

Timeline

November 14, 2024 06:11 PM UTC | November 14, 2024 10:11 AM PT
We are happy to report that the issue causing "Network Error" messages when loading Explore dashboards or reports has been solved. Thank you for your patience during our investigation.

November 14, 2024 03:18 PM UTC | November 14, 2024 07:18 AM PT
Some Explore users may see a "Network Error" message when loading dashboards or reports. If this happens, clearing your browser’s cookies and cache should resolve the issue. We apologise for the inconvenience.

POST-MORTEM

Root Cause Analysis

This incident was caused by a network change that increased the number of headers sent to the Explore service beyond its configured limit. This led to the service failing silently, resulting in the dashboards not loading for customers.

Resolution

To resolve this issue, the network team was paged to investigate the changes. Upon identifying the problem, they reverted the changes that added excessive headers, which restored normal functionality for the Explore Dashboards.

Remediation Items

Improve existing implementation tools: Review and enhance the tools used for managing header limits in API requests to prevent similar incidents.

Create additional alerts: Set up alerts for monitoring header counts and other critical metrics to detect issues before they impact customers.

Add connection limits on specific applications: Implement connection limits on APIs to ensure they do not exceed operational thresholds, reducing the risk of future incidents.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.