Service Incident - March 26, 2025 - Chat, Support (Agent Workspace) | Pod 18 - Chat and Messaging performance issues

Summary

On March 26, 2025 from 06:13 UTC to 07:12 UTC, some Zendesk Chat and Support (Agent Workspace - Messaging) customers on Pod 18 experienced chat disconnections, login issues and other errors while using the products.

Timeline

March 26, 2025 7:26 AM UTC | March 26, 2025 12:26 AM PT
We have received reports of issues in Chat and Support (Agent Workspace - Messaging) for Pod 18 customers. Our team has identified and mitigated the issue. We have observed the recovery of our services. We will continue to monitoring for a short time before we post our final update.

March 26, 2025 8:15 AM UTC | March 26, 2025 1:15 AM PT

We are happy to report that the issue affecting Chat and Agent Workspace Messaging on Pod 18 has now been resolved. We thank you for your patience as our team worked to resolve the issue today.

Root Cause Analysis

This incident was caused by a hardware failure in a single host in our chat infrastructure. A subset of customers whose accounts communicate through this host were impacted by this issue.

Resolution

To fix this issue, our team identified the problematic host and replaced it with a healthy instance. This led to quick recovery of our chat services for the affected customers.

Remediation Items

Investigate hardware failure root cause with service provider.

Improve alerting for internal monitoring dashboards.

Refine LC bootup process.

FOR MORE INFORMATION

For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.