Post-mortem - November 26, 2025 - All pods | Zendesk QA - Customers are unable to access Zendesk QA

Summary

On November 26, 2025, several customers experienced issues accessing the Zendesk QA environment. The majority of users encountered a persistent loading screen, while some received 503 errors. Both customer-facing and internal environments were impacted.

Timeline

November 26, 2025 02:53 UTC | November 25, 2025 06:53 PM PT

We are currently aware of an issue affecting Zendesk QA, which is causing customers to experience prolonged loading times and an inability to access the QA product. Our engineering team is actively investigating the issue, and we will provide an update within the next 30 minutes. We appreciate your patience and understanding.

November 26, 2025 03:21 UTC | November 25, 2025 07:21 PM PT

We are following up on our previous communication regarding the issue affecting the Zendesk QA. The majority of reports indicate that users are encountering a "No Healthy Upstream" error. Our team continues to actively investigate the root cause and is working towards a resolution. We will provide further updates as soon as they become available. We apologize for any inconvenience this may have caused and appreciate your patience and understanding.

November 26, 2025 04:04 UTC | November 25, 2025 08:04 PM PT

Our engineering team has mitigated the issue in the Zendesk QA environment that was preventing customers from accessing the system. Initial internal testing has been successful. Could you please verify if the issue persists on your end? Thank you for your patience and cooperation. Thank you for your patience and cooperation.

November 26, 2025 06:44 UTC | November 25, 2025 10:44 PM PT

Our engineering team has closely monitored the Zendesk QA following the mitigation. We are able to confirm that the QA environment is now stable. Thank you for your patience and understanding throughout this process.

November 26, 2025 15:43 UTC | November 26, 2025 07:43 AM PT

We are pleased to inform you that the issue preventing customer access to the Zendesk QA environment has been resolved. We sincerely appreciate your patience as our team worked diligently to address this matter.

Root Cause Analysis

The incident was caused by a configuration error where production settings were mistakenly set to staging values, leading to routing issues that caused traffic to overwhelm certain systems. A lack of validation and monitoring allowed this misconfiguration to go undetected.

Resolution

To fix this issue, the engineering team disabled the frontend health check and increased CPU allocation for the service experiencing issues. Internal testing verified that access to Zendesk QA has been restored across the board. Current system metrics indicate that CPU load and monitoring/alerting mechanisms are operating within expected parameters.

Remediation Items

Increased monitoring priority for the application and set it to alert the on-call engineer promptly.

Improved system stability by raising CPU limits and temporarily disabling certain checks to prevent unnecessary restarts.

Expanded testing and reporting to cover all relevant accounts to catch issues earlier.

Updated the deployment guide to reflect these changes and improve future processes.

FOR MORE INFORMATION

For current system status information about Zendesk and specific impacts to your account, visit our system status page. If you have additional questions about this incident, contact Zendesk customer support.