Service Incident - December 11, 2024 - Sunshine Conversations | All Pods - Messenger, Whatsapp and Instagram message delivery issues

Summary

On December 11, 2024 from 18:06 UTC to 22:57 UTC, Sunshine Conversations customers experienced message delivery issues in WhatsApp, Instagram and Messenger communication channels.

Timeline

December 11, 2024 06:15 PM UTC | December 11, 2024 10:14 AM PT
We are aware of a partner provider outage affecting WhatsApp, Messenger, and Instagram. We are working with the provider to restore services and will provide further updates soon.

December 11, 2024 06:35 PM UTC | December 11, 2024 10:35 AM PT

Our team continues their work with our partner provider to restore service to WhatsApp, Messenger, and Instagram. We will provide another update in one hour or when we have new information to share.

December 11, 2024 07:37 PM UTC | December 11, 2024 11:37 AM PT
We are still working with our partner provider to restore service to WhatsApp, Messenger, and Instagram channels. We will provide additional updates when we have new information to share.

December 11, 2024 11:01 PM UTC | December 11, 2024 3:01 PM PT

We are beginning to see some recovery from the provider outage affecting WhatsApp, Messenger, and Instagram. Our partner provider is still competing their work, and as such we will continue to monitor our systems to ensure that we have fully recovered. Please let us know if you continue to experience any issues.

December 12, 2024 02:00 AM UTC | December 11, 2024 6:00 PM PT

We are happy to report that the provider outage affecting WhatsApp, Messenger, and Instagram has been resolved. Thanks for your patience while we worked through today's issue.

Root Cause Analysis

This incident was caused by an issue with Meta's Cloud API services impacting Facebook, Messenger, WhatsApp, and Instagram. The exact root cause was not disclosed by Meta, but they acknowledged the disruption and worked on restoring services. During this time, Sunshine Conversations' monitoring systems detected a high occurrence of errors when attempting to send messages through the affected channels.

Resolution

The resolution of the outage was entirely dependent on Meta's recovery efforts. Sunshine Conversations engineers monitored the situation closely and the Zendesk incident team provided updates to customers as they became available. Once Meta announced that services were restored, we verified that outbound traffic resumed to normal levels and that messages were being successfully delivered through the social messaging channels.

Remediation Items

Review and improve Meta edge case monitoring.

Update Meta escalation procedures.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, contact Zendesk customer support.