Service Incident - December 9, 2024 - All Pods | Issues with Multi-Brand Ticket Creation

SUMMARY

On December 9, 2024 from 8:08 UTC to 13:03 UTC, customers using the multi-brand functionality experienced errors when trying to create tickets or update ticket titles, but changes were saved.

Timeline

December 09, 2024 10:21 PM UTC | 02:21 PM PT

We are happy to inform you that the recent change has been successfully reverted, and the issue with multi-brand ticket creation has now been resolved.
If you are still experiencing any issues, please try performing a hard refresh (Ctrl + F5) or clear your browser cache and cookies, as this can help resolve any lingering problems.
Feel free to reach out if you continue to encounter difficulties. Thank you for your patience!

December 09, 2024 12:58 PM UTC | 04:58 AM PT
We're currently experiencing issues with multi-brand ticket creation. Our engineering team is aware of the problem and is actively working on resolving it as quickly as possible.
We will provide updates as soon as more information becomes available. Thank you for your patience and understanding!

POST-MORTEM

Root Cause Analysis

The root cause of the incident was a flaw in the ticket initialization logic. When the requester ID was undefined, the system attempted to retrieve it based on the ticket key, leading to errors whenever any field change occurred in the ticket UI. The introduction of new logic to read the user ID from the user profile page and set it to the requester ID inadvertently set the requester object to undefined, which broke the expected functionality.

Resolution

To resolve the issue, the problematic update was reverted.

Remediation Items

Fix flawed code: Ensure that the system correctly sets requester data when creating tickets to prevent blank entries.

Add automatic testing: Create a test to check that the ticket save process correctly handles requester information.

Confirm Manual Testing: Require deployers to manually test changes on “canary” PODs and confirm that everything works before deployment.

Improve Monitoring: Set up monitoring to alert on browser errors, such as “something went wrong,” to quickly identify issues.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.