Service Incident - October 29, 2025 - Multiple Pods | AI Agents - Import and retrieval operations degradation issues

Summary

On October 29, 2025, from 07:34 UTC to November 11, 2025, 13:46 UTC, customers using AI Agents in the EU region experienced delays and failures in importing and retrieving knowledge base data. Customers were affected, specifically those with bots created before October 9, 2025.

Timeline

October 29, 2025 14:30 UTC | October 29, 2025 07:30 AM PT

We have received reports regarding degraded performance with one of our vendor platforms, causing delays in import and data retrieval operations in AI Agents for customers. Our engineering team is currently conducting a thorough investigation to identify and resolve the cause. We apologize for any inconvenience this may cause and will share further updates as they become available.

October 29, 2025 15:35 UTC | October 29, 2025 08:35 AM PT

Our engineering team has deployed a mitigation aimed at improving the performance of the search and observability suite that was causing delays in importing and retrieving data within AI Agents. We continue to monitor the system actively and appreciate your understanding and patience. Further updates will be shared as they become available.

October 29, 2025 16:42 UTC | October 29, 2025 09:42 AM PT

Our engineers have confirmed that the Search and Observability service is now stable, and the issue causing delays in data import and retrieval within AI Agents has been resolved. We sincerely appreciate your patience as our team diligently worked to address this matter.

October 31, 2025 12:59 UTC | October 31, 2025 05:59 AM PT

While our engineers have addressed the data import and retrieval delays affecting AI Agents, they continue working on a permanent solution. As a temporary workaround, creating new bots has been tested and shown to help. We apologize for any inconvenience caused and appreciate your ongoing patience. We will share updates as soon as more information becomes available.

October 31, 2025 12:59 UTC | October 31, 2025 05:59 AM PT

Our engineering team will continue their work into next week to address the issue affecting AI Agents. In the meantime, we recommend the temporary workaround creating new bots. We apologize for any interruption and thank you for your continued patience.

November 04, 2025 13:17 UTC | November 04, 2025 05:17 AM PT

Our engineers continue to work on resolving the issue affecting AI Agents. As noted previously, we recommend using the workaround of creating a new bot or using the US dashboard for imports until the issue is fully resolved. We apologize for the inconvenience and assure you that our team is diligently addressing the matter. Updates will be provided as they become available.

November 04, 2025 16:25 UTC | November 04, 2025 08:25 AM PT

Our engineering team has provided an update regarding the AI Agents disruption. Imports remain temporarily disabled for most EU bots created prior to October 9, 2025, to protect system stability, and no immediate workaround is available. Imports for US bots and EU bots created after this date are functioning as expected. We advise against EU customers creating US bots due to compliance issues. For urgent cases, please contact Customer Support. We regret any inconvenience caused and will continue to provide updates via the Help Center.

November 05, 2025 12:38 UTC | November 05, 2025 04:38 AM PT

Our engineers have completed successful testing of the remediation in the development environment and will begin implementing the changes today to address the AI Agents issue. The fix for manual imports is expected to be available by tomorrow through a phased rollout. We appreciate your patience as our team works to fully resolve this matter and will provide updates as they become available.

November 06, 2025 14:37 UTC | November 06, 2025 06:37 AM PT

Our engineers have completed the phased rollout of manual imports and will now proceed with enabling imports on a randomized basis for remaining customers. To ensure successful deployment of the fix, the team will temporarily pause accepting additional requests. We appreciate your patience as we work diligently to fully resolve this issue and will provide updates as they become available.

November 06, 2025 16:58 UTC | November 06, 2025 08:58 AM PT

Our engineering team has restored manual import functionality for all AI Agents bots affected by the incident. We apologize for the disruption and sincerely appreciate your patience while we continue working toward a complete resolution. We will continue to provide updates until the incident is fully resolved.

November 07, 2025 16:22 UTC | November 07, 2025 08:22 AM PT

As part of our ongoing efforts to resolve the issue impacting customers using AI Agents, our engineers will perform emergency maintenance on Monday, November 10, 2025. The maintenance is expected to last approximately 4 hours, from 11:00 to 15:00 UTC. During this period imports will be unavailable for customers operating on the US cluster. We will update when the maintenance is complete or if we have new information. We appreciate your understanding and apologize for any inconvenience this may cause.

November 10, 2025 14:02 UTC | November 10, 2025 06:02 AM PT

We would like to inform you that the emergency maintenance window has been extended from 15:00 UTC to 17:00 UTC to allow our engineers to fully restore import functionality for customers in the US cluster. We apologize for any inconvenience this may cause and appreciate your patience as we continue to work diligently. Updates will be provided as they become available.

November 10, 2025 15:27 UTC | November 10, 2025 07:27 AM PT

We are pleased to announce that the emergency maintenance addressing import functionality for AI Agents customers in the US is now complete and we have reenabled manual knowledge imports for all affected customers. We appreciate your patience during this time and urge you to contact us if the issue persists.

November 11, 2025 13:54 UTC | November 11, 2025 05:54 AM PT

We are pleased to inform you that the service issues impacting AI Agents import and retrieval operations have been resolved. Our engineers have successfully restored all import capabilities, and we have observed a successful automated import run on EU bots. We sincerely appreciate your patience as our team diligently worked to address this matter.

Root Cause Analysis

This incident was caused by the search system being overloaded with too much data, which affected its performance and caused delays in importing and retrieving information.

Resolution

To fix this issue, new search clusters were added to balance the workload, manual imports were resumed, and unused data was cleared. Automated imports restarted once the system stabilized. Some system settings were also updated to help maintain stability and avoid similar problems in the future.

Remediation Items

Set up regular monitoring and alerts to catch early signs of system overload.

Create clear procedures for handling capacity issues.

Plan ahead for system upgrades and improvements to better manage growth and performance.

Put limits on how many knowledge bases customers can create to manage resource use.

Streamline import processes to reduce unnecessary work and ease system load.

FOR MORE INFORMATION

For current system status information about Zendesk and specific impacts to your account, visit our system status page. You can follow this article to be notified when our post-mortem report is published. If you have additional questions about this incident, contact Zendesk customer support.