Service Incident - March 11, 2025 - Support | Pod 13 - "Service Unavailable" error

Summary

On March 11, 2025 from 13:58 UTC to 14:06 UTC, Zendesk Support customers on Pod 13 may have encountered a "Service Unavailable" error message preventing them from using the product.

Timeline

March 11, 2025 02:38 PM UTC | March 11, 2025 07:38 AM PT
We are investigating reports of our Support customers on Pod 13 receiving a "Service unavailable" error between 14:00 and 14:05 UTC.

March 11, 2025 02:56 PM UTC | March 11, 2025 07:56 AM PT
We are pleased to inform you that our team has completed the investigation into the issues that caused our Support customers to receive a "Service unavailable" message on Pod 13. We have confirmed that this issue was contained between 14:00 and 14:05 UTC. Our team has been actively monitoring the situation and is now happy to confirm that the issue has been resolved.

Root Cause Analysis

This incident was caused by an inadvertent reduction in capacity during the deployment of the network change, which led to significant CPU throttling on an internal service.  As part of migrating network traffic to a new proxy service, the process triggered a series of rapid container restarts, overwhelming the system and causing increased error rates.

Resolution

This issue was promptly resolved by redirecting traffic to our backup proxy infrastructure.

Remediation Items

Develop a more graceful restart mechanism for our proxy server restarts.

FOR MORE INFORMATION

For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.