Service Incident - July 31, 2025 - All Pods | Zendesk Sales - Performance degradation and failing calls

Summary

Between July 27, 2025, 08:30 UTC and August 08, 2025, 06:25 UTC, we received 48 customer reports regarding performance degradation impacting the Zendesk Sales (Sell) service. Upon investigation, it was determined that the user account “Persimmon” was generating an unusually high volume of appointment messages, which contributed significantly to the service degradation.

Timeline

July 31, 2025 17:50 UTC | July 31, 2025 10:50 PT

We are investigating performance issues for customers using Zendesk Sales (Sell) across all pods. Impact includes failing calls, slow performance, and issues with imports/exports. Our engineers are adding resources and continue to investigate and monitor the situation.

July 31, 2025 6:51 UTC | July 31, 2025 11:51 PT

Issues with Zendesk Sales Voice and live data services are now resolved. We continue to address performance issues with other services of Zendesk Sales. Some searches may not return recently created items. We will update when this data has been processed.

August 04, 2025 15:30 UTC | August 04, 2025 8:30 AM PT

Our engineering team has deployed a targeted update to address the recent performance degradation affecting Zendesk Sell services. This fix is now live and under close monitoring. If you continue to experience any performance issues with Zendesk Sell, please contact us promptly so we can provide immediate assistance.Thank you for your patience and understanding.

August 05, 2025 12:55 UTC | August 05, 2025 05:55 AM PT

Our engineers continue to closely monitor system activity following the deployment of a fix for the recent performance degradation affecting Zendesk Sell. We would appreciate your feedback on whether the issue has been resolved on your end. Your input will enable us to assist you more effectively. Thank you for your cooperation.

August 06, 2025 15:39 UTC | August 06, 2025 8:39 AM PT

As our engineers continue to work on the performance issues related to the Zendesk Sales product, we have finished processing a backlog of records that contributed to the issues related to inaccurate or "stale" reporting data. We observe that no further issues are present in reporting and we continue to monitor to ensure full recovery. Please let us know if you continue to experience any issues in either live data or historical data accuracy and product functionality.

August 08, 2025 06:06 UTC | August 07, 2025 11:06 PM PT

We are pleased to inform you that the performance issues with Zendesk Sales (Sell) have been resolved. Thank you for your patience and cooperation while we worked to address this matter.

Root Cause Analysis

The root cause of this incident was identified as an unusually high volume of appointment messages generated by the user account “Persimmon,” which significantly contributed to the degradation of service performance.

Resolution

To resolve the issue, engineers transferred the messages to the RabbitMQ broker and cleared nearly 170,000 “appointment” messages originating from the affected user account.

Remediation Items

Revisit all RMQ-related alerts to ensure comprehensive coverage including restarts, crashes, error rate increases, and disk-related metrics per queue. Verify alert thresholds and update playbook links accordingly.

Update and refine playbooks for RabbitMQ (RMQ) incidents, including specific procedures for the RabbitQOS service and guidelines for handling offenders.

Review services that upscaled during the incident, such as the sell-hoarder service. Correct and update resource limits, consider implementing or calibrating Horizontal Pod Autoscaling (HPA), and adjust existing resource requests as necessary.

Perform broker deletions and other necessary cleanup work. Investigate the possibility of fixing payload issues originating from the sell-appointment service to improve system stability.

FOR MORE INFORMATION

For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.