Service Incident - March 31, 2025 - Talk | All Pods - Dropped calls

Summary

On March 31, 2025 from 15:02 UTC to 16:44 UTC, Zendesk Talk customers experienced a high volume of dropped calls across all pods.

Timeline

March 31, 2025 03:58 PM UTC | March 31, 2025 08:58 AM PT
We are receiving reports of dropped calls across multiple pods and our team is investigating. Further updates will be posted shortly.

March 31, 2025 04:12 PM UTC | March 31, 2025 09:12 AM PT
We have confirmed an issue with our Talk partner provider causing calls to drop when accessing some greetings within Talk. We are working with that provider and will post further information soon.

March 31, 2025 06:09 PM UTC | March 31, 2025 11:09 AM PT
We are happy to report that the impact to the Talk product has been resolved working with our partner provider and all calls should be connecting as expected at this time. Thank you for your patience during our investigation.

Root Cause Analysis

This incident was caused by a failure in our service provider's caching proxy that fetches call greeting files, resulting in system timeouts that dropped inbound calls.

Resolution

To fix this issue, our service provider implemented several remediation measures, including adding additional capacity to handle the high load, and a fix patch to prevent recurrence of the issue. These actions helped restore normal service and mitigate the impact on customers.

Remediation Items

Update runbooks with call greeting flow for easier debugging.

Update Talk SLO monitoring dashboards.

Review and update service provider escalation process and documentation.

FOR MORE INFORMATION

For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.