Service Incident - March 26, 2026 - Sunshine Platform | All Pods - Intermittent 404 responses on submitted requests to Sunshine Custom Objects

Summary

On March 26, 2026 from 13:45 UTC to 16:41 UTC, and again on March 27, 2026 from 05:20 UTC to 09:06 UTC, Sunshine Platform (Custom Objects) customers in All Pods experienced intermittent errors accessing legacy custom object records, which caused some workflows to fail or only work after repeated retries.

Timeline

March 26, 2026 16:09 UTC | March 26, 2026 09:09 AM PT

We have received reports of intermittent 404 errors occurring when customers submit requests to Sunshine Custom Objects. Our engineers are aware of the issue and are actively investigating. We apologize for any inconvenience caused and appreciate your patience as we work toward a resolution. Updates will be provided as soon as they become available.

March 26, 2026 16:55 UTC | March 26, 2026 09:55 AM PT (Initial stable time)

Our engineers have identified the root cause and have implemented a change. After this change, we have observed recovery and system stability. Please reach out back to us if you continue to have issues with accessing the custom object records.

March 27, 2026 07:06 UTC | March 26, 2026 12:06 AM PT (Reoccurrence)

We are aware of the recurring intermittent 404 errors when customer submit requests to Sunshine Objects. Our engineering team is actively investigating the issue. We apologize for any inconvenience this may be causing and will provide updates as soon as more information is available. Thank you for your patience and understanding.

March 27, 2026 09:05 UTC | March 27, 2026 02:05 AM PT

Following the deployment of a fix by our engineering team, we have observed signs of recovery and are actively monitoring the system to ensure continued stability. We will keep you informed with updates as more information becomes available. We sincerely appreciate your patience as we work to address this matter swiftly.

March 27, 2026 20:38 UTC | March 27, 2026 01:38 PM PT

We are happy to report after extensive monitoring from our team that the issue causing 404 errors in Sunshine Custom Objects is now resolved. Thank you for your patience and understanding while we worked through this incident.

Root Cause Analysis

This incident was caused by an internal credential rotation that wasn’t fully applied across all service instances, which temporarily prevented part of our routing layer from loading required configuration and led to intermittent 404 errors for some Custom Objects requests.

The problem reoccurred when the old credential was removed and some instances needed to be restarted to fully pick up the updated configuration.

Resolution

To fix this issue, we restarted and redeployed the affected services to ensure all instances picked up the updated credentials and rebuilt the correct routing configuration.

After redeployment completed across all impacted pods, request success rates and traffic levels returned to normal, and Custom Objects errors have ceased.

Remediation Items

Set up automatic health checks that alert us if there’s a loss of connection before customers notice any issues.

Update deployment processes to ensure recent key usage is verified before disabling old credentials, preventing deletion if the key was used recently.

Consider switching from long-lasting access keys to temporary credentials to reduce risks during key changes.

Add clear checks to confirm new credentials are working everywhere before removing any old keys.

Develop and implement a clear, shared key rotation process with defined ownership, verification, and communication steps.

FOR MORE INFORMATION

For current system status information about Zendesk and specific impacts to your account, visit our system status page. If you have additional questions about this incident, contact Zendesk customer support.