Announcing web crawler advanced URL filters and custom scheduler

Announced on | Rollout starts | Rollout ends
August 20, 2026 | August 20, 2026 | August 27, 2026

We’re excited to announce a series of web crawler enhancements that improve crawl coverage, control, and maintenance for customers using external content in help center search and AI Agents and Agent Copilot.

This announcement includes the following sections:

What is changing?

Why is Zendesk making this change?

What do I need to do?

What is changing?

We are expanding the capabilities of our web crawler to help you better scope and target the content you want to sync.

With this release, you can now:

Scale and consolidate crawling: Configure up to 500 start URLs per crawler and crawl across multiple domains and subdomains.

Customize refresh schedules: Run crawls daily, weekly, monthly, or disable recurring runs by selecting never. Weekly is now the default schedule, and manual crawls remain available at any time.

Target relevant content: Define up to 500 include and 500 exclude filter rules per crawler. If a URL matches both an include and exclude rule, the exclude rule takes precedence

Why is Zendesk making this change?

Customers now have greater flexibility when configuring web crawlers. They can narrow crawl scope to specific sections of a website and crawl multiple domains simultaneously. These enhancements improve crawler effectiveness, resulting in better search results and AI responses.

What do I need to do?

These features are automatically available on all plans with no additional prerequisites or pricing changes.

To learn more, see:

Creating URL filters to scope website syncs

Using a web crawler to index external content

If you have feedback or questions related to this announcement, visit our community forum where we collect and manage customer product feedback. For general assistance with your Zendesk products, contact Zendesk Customer Support.