Post-mortem - December 08, 2025 - All pods | Quality Assurance - Zendesk QA loading issues

Summary

On December 8, 2025, between 12:00 UTC and 14:08 UTC, customers across all pods experienced difficulties accessing the Zendesk QA landing page. Investigation revealed the root cause to be a recent database migration that blocked the application from fetching metadata related to logged-in users.

Timeline

December 08, 2025 12:46 UTC | December 08, 2025 04:46 AM PT

We are aware of the performance issues some users have experienced in Zendesk QA across multiple pods. Our Engineering team has taken steps to address the problem and is closely monitoring the situation. If you continue to experience any difficulties, please let us know. We will provide the next update in 30 minutes.

December 08, 2025 13:10 UTC | December 08, 2025 05:10 AM PT

We are pleased to inform you that our engineering team has successfully resolved the issue affecting the QA system. We sincerely apologize for any inconvenience this may have caused. If you continue to experience any performance issues, please do not hesitate to reach out. We appreciate your patience and cooperation.

December 08, 2025 14:10 UTC | December 08, 2025 06:10 AM PT

We are happy to report that, after careful monitoring, our engineers have confirmed that Zendesk QA is now stable and operating as expected. We truly appreciate your patience and understanding.

Root Cause Analysis

This incident was caused by a database schema migration that unexpectedly locked the main account records, preventing the Zendesk QA application from retrieving logged-in user information during a routine code release.

Resolution

To resolve the issue, our engineers canceled the database migration query that was causing the blockage, immediately restoring successful application access for all customers.

Remediation Items

Rewrite the query that updates user sessions to improve efficiency and stability.

Adjust how the smoke test triggers incident severity to better reflect actual impact.

Reduce the session endpoint timeout from 2 minutes to 15 seconds for faster response.

Refine monitoring priorities to better detect errors and delays in the system gateway.

Create a clear guide for making schema changes safely and effectively.

Share best practices for managing MySQL databases to improve overall system performance.

Configure timeouts for faster response on critical endpoints.

FOR MORE INFORMATION

For current system status information about Zendesk and specific impacts to your account, visit our system status page. If you have additional questions about this incident, contact Zendesk customer support.