Announcing automatic link discovery for the web crawler

Announced on | Rollout starts | Rollout ends
May 15, 2026 | May 14, 2026 | May 28, 2026

We're excited to announce that web crawlers can now automatically discover and index your content by following links throughout your website. This improvement eliminates the need for a sitemap or for manually specifying every page you want to crawl. You can simply provide a starting URL and the crawler handles the rest.

The crawler also now supports complex sitemap structures, automatically following nested sitemap files to discover all available content without any additional configuration.

This announcement includes the following sections:

What is changing?

Why is Zendesk making this change?

What do I need to do?

What is changing?

The web crawler now features automatic content discovery that significantly simplifies setup and improves coverage:

Automatic link discovery: When configuring your web crawler, simply provide your website's homepage or any starting point. The crawler automatically discovers content by either finding the sitemap or following links throughout your website.

Nested sitemap support: For websites that use complex sitemap structures (for example, sitemap index files that reference multiple sitemap files), the crawler automatically follows the entire chain to discover all available content.

Why is Zendesk making this change?

Setting up web crawlers previously required a sitemap.xml file, which the crawler used to identify pages to crawl. Some customers experienced issues with missing sitemaps or unsupported sitemap indexes, which reduced crawl coverage and created gaps in their knowledge base.

This improvement makes crawler setup much more reliable by removing the dependency on sitemaps. If a sitemap isn’t available, the crawler can now explore your website directly, extract relevant content, and help ensure that your AI agents, Help Center search, Quick Answers, and Knowledge in Agent Workspace have access to complete and accurate information.

What do I need to do?

Automatic link discovery is enabled for all web crawlers. For existing crawlers, no action is required as they will continue to work as configured.

For more information about the web crawler and how it indexes external content, see Using a Web Crawler to Index External Content.

If you have feedback or questions related to this announcement, visit our community forum where we collect and manage customer product feedback. For general assistance with your Zendesk products, contact Zendesk Customer Support.