Troubleshooting issues with web crawler imports for AI agents (AI agents - Advanced only)

<p style="display: none;" id="docs-hc-snippet">You can use a web crawler to import content into your AI agent. This gives your AI agent the ability to create AI-generated answers to customer questions based on information in external websites.</p><div class="article-banners">
<div id="docs-wmp-link"><a href="https://support.zendesk.com/hc/en-us/articles/5411234991258-plan" target="_blank" rel="noopener noreferrer">What's my plan?</a></div>
<div class="article-banner" id="addon_ai-agents-adv"><figure class="wysiwyg-table"><table><tbody><tr>
<td><strong>Add-on</strong></td>
<td>AI agents - Advanced</td>
</tr></tbody></table></figure></div>
</div><div class="body">
<div class="p"><div class="note attention" id="adv_web_crawler_troubleshooting__note_q5r_bqv_p3c">This article applies only if your Zendesk account was created before March 10, 2026. Otherwise, see <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8357749301658" target="_blank" rel="noopener noreferrer">Connecting knowledge sources to power generative replies in AI agents</a>.</div></div>
<p class="p">You can <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/10791195957530" target="_blank" rel="noopener noreferrer">use a web crawler to import content</a> into your AI agent. This gives your AI agent the ability to create AI-generated answers to customer questions based on information in external websites.</p>
<p class="p">This article helps you troubleshoot issues you encounter when using a web crawler to import content for an AI agent, and contains the following topics:</p>
<div style="margin-bottom: 20px; text-align: right;">
<button id="master-toggle" class="btn fw-semibold text-dark" data-action="expand" style="background-color: #DEF991; border: none; outline: none; border-radius: 7px; padding: 10px 20px; font-size: 15px; transition: background-color 0.2s ease;" onmouseover="this.style.backgroundColor='#cbef6d'" onmouseout="this.style.backgroundColor='#DEF991'">
Expand all tabs
</button>
</div>
<script>
document.addEventListener('DOMContentLoaded', function () {
const toggleBtn = document.getElementById('master-toggle');

toggleBtn.addEventListener('click', function () {
const action = this.getAttribute('data-action');
const headers = document.querySelectorAll('.accordion');

headers.forEach(header => {
const isActive = header.classList.contains('active');
const panel = header.nextElementSibling; // Grabs the corresponding .panel container

if (action === 'expand' && !isActive) {
// Open it by simulating a standard click
header.click();
} else if (action === 'collapse' && isActive) {
// Close it by simulating a standard click
header.click();
}
});

// Flip button text and states
if (action === 'expand') {
this.setAttribute('data-action', 'collapse');
this.textContent = 'Collapse all tabs';
} else {
this.setAttribute('data-action', 'expand');
this.textContent = 'Expand all tabs';
}
});
});
</script><div class="topic nested1" id="topic_mhp_2bv_3hc" aria-labelledby="ariaid-title2">
<div class="accordion"><h3 class="title topictitle2" id="ariaid-title2">The crawl failed because it timed out</h3></div>
<div class="panel"><div class="body">
<p class="p">If a web crawl fails after several hours, a timeout usually causes it. Crawls are limited to five hours by default. If the failure happened five hours after the import started, chances are it is a timeout.</p>
<p class="p">If this happens, try these resolution steps:</p>
<ul class="ul" id="topic_mhp_2bv_3hc__ul_ksl_4bv_3hc">
<li class="li" data-list-item-id="e5e58848a12c3fd0b8d1586adc6756eac">If the website doesn't rely on JavaScript, set the <strong class="ph b">Crawler type</strong> to Raw HTTP client (Cheerio), a much faster crawler, and try again</li>
<li class="li" data-list-item-id="e22fa88b6f6041842e1ce8cc2a70cc270">If you know the site includes content you don't actually need, see <a class="xref" href="#topic_hpm_fbv_3hc">The crawl includes unnecessary pages</a>
</li>
<li class="li" data-list-item-id="e8af810c5e6eee8b2ae6e24d94071c1fb">Split the crawl into two or more separate crawls (using the <strong class="ph b">Include URLs</strong> or <strong class="ph b">Exclude URLs</strong> settings) that each scrape only parts of the website</li>
</ul>
</div></div>
</div>
<div class="topic nested1" id="topic_wgc_fbv_3hc" aria-labelledby="ariaid-title3">
<div class="accordion"><h3 class="title topictitle2" id="ariaid-title3">The crawl is missing pages</h3></div>
<div class="panel"><div class="body">
<p>If whole URLs or articles are missing from the crawl, review the crawler’s <strong>Start URLs</strong> and include rules. You can configure up to 500 Start URLs and up to 500 include rules for each crawler. Crawlers can also cover multiple domains and subdomains.</p>
<p>If you’re sure the settings are correct but pages are still missing, check the number of pages crawled in the import summary. Also verify that the relevant URLs aren’t excluded by an Exclude URLs rule. When a URL matches both an include rule and an exclude rule, the exclude rule takes precedence.</p>
<p>If the crawler still doesn’t capture the expected content, check the crawler’s maximum page or depth settings and review any errors in the <a href="https://support.zendesk.com/hc/en-us/articles/9962106058906#topic_jxx_rn5_3hc" target="_blank" rel="noopener noreferrer">import summary</a>.</p>
</div></div>
</div>
<div class="topic nested1" id="topic_hpm_fbv_3hc" aria-labelledby="ariaid-title4">
<div class="accordion"><h3 class="title topictitle2" id="ariaid-title4">The crawl includes unnecessary pages</h3></div>
<div class="panel"><div class="body">
<p>If the crawl includes more pages or articles than necessary, use <strong>Exclude URLs</strong> rules to prevent irrelevant content from being indexed. You can configure up to 500 exclude rules for each crawler.</p>
<p>You can also use Start URLs and Include URLs rules to limit the crawler to specific websites, domains, subdomains, or sections of your content. If a URL matches both an include rule and an exclude rule, the exclude rule takes precedence.</p>
<p>Be careful when excluding pages. Pages linked only from an excluded page might not be crawled unless they are also specified as Start URLs or included by another rule.</p>
<p class="p">See the following diagram as an example. Each circle represents a page, and each arrow represents a link from that page. If the crawl starts at the top page (as in, it's the only <strong class="ph b">Start URL</strong>) and the purple page is excluded, then the following is true:</p>
<ul class="ul" id="topic_hpm_fbv_3hc__ul_ajn_cmc_jhc">
<li class="li" data-list-item-id="e8ab68d7d8fe562f8baf971b9bb9ac4f5">The red page can't be crawled</li>
<li class="li" data-list-item-id="e977ca2e7309ece70771133a2767c3835">All yellow pages are crawled</li>
<li class="li" data-list-item-id="efe731502fef165ea25d156e1d16128c4">The green page is also crawled, even though it's linked from the excluded purple page, because it's also linked from one of the included yellow pages</li>
</ul>
<p class="p"><img class="wysiwyg-image" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/aiaa_web_crawler_troubleshooting_diagram.png" width="350" id="topic_hpm_fbv_3hc__image_apt_1cv_3hc"></p>
</div></div>
</div>
<div class="topic nested1" id="topic_k5b_gbv_3hc" aria-labelledby="ariaid-title5">
<div class="accordion"><h3 class="title topictitle2" id="ariaid-title5">The crawl includes the right pages but the wrong content</h3></div>
<div class="panel">
<div class="body">
<p class="p">If the crawl returns the correct pages, but returns the wrong content inside those pages, the <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8357749301658#topic_amk_g3w_cfc" target="_blank" rel="noopener noreferrer">advanced crawler settings</a> include tools to find and include or exclude such content. You need to find the correct CSS selector for the element you want to include or exclude and then insert that into the right setting. To do that, it's useful to understand what a CSS selector is and how to find it.</p>
<p class="p">This section contains the following topics:</p>
<ul class="ul" id="topic_k5b_gbv_3hc__ul_tbc_tcv_3hc">
<li class="li" data-list-item-id="e149d20f45bb9182d0bc35ff13b45906a"><a class="xref" href="#topic_mqg_dcv_3hc">Understanding and finding CSS selectors</a></li>
<li class="li" data-list-item-id="e90e49519a248edff2052b22bd2a2cca8"><a class="xref" href="#topic_jgj_mcv_3hc">The crawl is skipping page content</a></li>
<li class="li" data-list-item-id="e60b2a465a82cb5f1d3150c775235683f"><a class="xref" href="#topic_t5g_4cv_3hc">The crawl is returning too much or messy page content</a></li>
</ul>
</div>
<div class="topic nested2" id="topic_mqg_dcv_3hc" aria-labelledby="ariaid-title6">
<h3 class="title topictitle3" id="ariaid-title6">Understanding and finding CSS selectors</h3>
<div class="body">
<p class="p">This section introduces you to CSS selectors and walks you through finding the correct one. If you're already familiar with this information, skip to the troubleshooting sections below instead.</p>
<p class="p">This section contains the following topics:</p>
<ul class="ul" id="topic_mqg_dcv_3hc__ul_qqs_rcv_3hc">
<li class="li" data-list-item-id="e51e34d9a5e1761c2f6999de82f549a74"><a class="xref" href="#topic_ppc_3cv_3hc">About CSS selectors</a></li>
<li class="li" data-list-item-id="ee63b1c2f035150dca573e1c2d0e7bab0"><a class="xref" href="#topic_i1k_jcv_3hc">Finding a CSS selector</a></li>
<li class="li" data-list-item-id="e7731ea32b622690eb666aa4019f9d9ce"><a class="xref" href="#topic_qqk_lcv_3hc">Verifying a CSS selector</a></li>
</ul>
</div>
<div class="topic nested3" id="topic_ppc_3cv_3hc" aria-labelledby="ariaid-title7">
<h4 class="title topictitle4" id="ariaid-title7">About CSS selectors</h4>
<div class="body">
<p class="p">CSS selectors are patterns used to select and target specific HTML elements on a web page. They make it easier to find and extract exactly the data you need from complex web pages.</p>
<p class="p">In web crawling and scraping, CSS selectors help extract data by pinpointing the exact parts of a page's structure, such as <code class="ph codeph">&lt;div&gt;</code>, <code class="ph codeph">&lt;span&gt;</code>, or elements with certain classes and IDs. For example, the selector <code class="ph codeph">.product-title</code> targets all elements with the class <code class="ph codeph">"product-title"</code>. The pound sign (#) is used to select elements by their unique ID. For instance, <code class="ph codeph">#main-header</code> selects the element with <code class="ph codeph">id="main-header"</code>.</p>
</div>
</div>
<div class="topic nested3" id="topic_i1k_jcv_3hc" aria-labelledby="ariaid-title8">
<h4 class="title topictitle4" id="ariaid-title8">Finding a CSS selector</h4>
<div class="body">
<p class="p">First, you need to find the CSS selector you want to use. The following instructions assume you're using the Chrome web browser. However, the steps are likely similar for other browsers.</p>
<p class="p">To find a CSS selector:</p>
<div class="p"><ol class="ol" id="topic_i1k_jcv_3hc__ol_x2s_jdv_3hc">
<li class="li" data-list-item-id="eed5d5218dada0c0113f4e2d9b25d1ab0">Locate the text or clickable item on the webpage you want to target</li>
<li class="li" data-list-item-id="eb4b3f803384efca5eeefac18d8ecd051">
<p>Right-click directly on that element and select <strong class="ph b">Inspect</strong></p>
<p class="p">The Chrome DevTools panel opens, and the matching piece of code is highlighted.</p>
</li>
<li class="li" data-list-item-id="ef4ded61a544c47d45d5935af044b89e6">
<p>In the DevTools panel, right-click the highlighted code and select <strong class="ph b">Copy</strong> &gt; <strong class="ph b">Copy selector</strong></p>
<p class="p">Now you have the CSS selector copied to your clipboard.</p>
</li>
</ol></div>
</div>
</div>
<div class="topic nested3" id="topic_qqk_lcv_3hc" aria-labelledby="ariaid-title9">
<h4 class="title topictitle4" id="ariaid-title9">Verifying a CSS selector</h4>
<div class="body">
<p class="p">After you find the CSS selector, it's a good idea to verify it.</p>
<p class="p">To verify the CSS selector:</p>
<div class="p"><ol class="ol" id="topic_qqk_lcv_3hc__ol_tkt_ndv_3hc">
<li class="li" data-list-item-id="ef84e57824b8879194666f4cb6a64ea2b">
<p>With DevTools still open, press Ctrl+F (on Windows or Linux) or Cmd+F (on Mac)</p>
<p class="p">This activates a search bar inside the Elements tab of the DevTools panel.</p>
</li>
<li class="li" data-list-item-id="e05043c91e8245c77ca5f4771bfdc9ca7">Paste the CSS selector you just copied into this search box</li>
<li class="li" data-list-item-id="ed3d3b83da83ef9bc654ba974ba089008">Verify that the highlighted elements in the HTML and on the page itself (often with a colored outline) match what you expect</li>
</ol></div>
<p class="p">If only the element you wanted is highlighted, your selector is accurate. If too many or the wrong elements are highlighted, try a parent element or adjust your selection.</p>
<p class="p">Optionally, you can test different selectors. Sometimes, shorter or more specific selectors work better. You can click on parent or child elements in the HTML to see their CSS classes or IDs and try copying those selectors as well.</p>
<p class="p">The next two sections walk you through how to use these selectors to target content you do or don't want to crawl.</p>
</div>
</div>
</div>
<div class="topic nested2" id="topic_jgj_mcv_3hc" aria-labelledby="ariaid-title10">
<h3 class="title topictitle3" id="ariaid-title10">The crawl is skipping page content</h3>
<div class="body">
<p class="p">If your crawl has the right pages, but is missing content from those pages, the following <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8357749301658#topic_amk_g3w_cfc" target="_blank" rel="noopener noreferrer">advanced crawler settings</a> can help:</p>
<ul class="ul" id="topic_jgj_mcv_3hc__ul_v3c_qdv_3hc">
<li class="li" data-list-item-id="e9b47d7897d0fa7f3e47d7fc5081d14ae">
<strong class="ph b">HTML transformer</strong>: The crawler initially scrapes all HTML from a page and then applies an HTML transformer to remove extraneous content. Sometimes the transformer can go too far and remove content you actually want to keep. So the first thing to try when content is missing is to turn this setting to None so that no content is removed and then <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9962106058906#topic_jxx_rn5_3hc" target="_blank" rel="noopener noreferrer">check the import summary</a>
</li>
<li class="li" data-list-item-id="e0b1c4f3d8bc6599b0ded19f63f404c41">
<strong class="ph b">Keep HTML elements</strong>: Keep only specific HTML elements by providing one or more CSS selectors. All other content is ignored, helping you focus on relevant information</li>
<li class="li" data-list-item-id="e098193d276d4257241f18f04dd1f2041">
<strong class="ph b">Expand clickable elements</strong>: Use this option to capture the content behind accordions and dropdowns. The default setting is intended to cover webpages that follow standard web development practice and define dropdowns as <code class="ph codeph">aria=false</code>. That means, if the crawler encounters such an element, it clicks it open. Enter a CSS selector for any element that should be clicked, such as buttons or links that expand hidden content. This helps the crawler capture all the text. Make sure the selector is valid</li>
<li class="li" data-list-item-id="ecedeb02100126f986ee07db0f14e460c">
<strong class="ph b">Make containers sticky</strong>: If the expandable content closes when a different element is clicked, you can use this setting to ensure that such elements stay open after they're clicked. So again, enter a CSS selector for any element that should be clicked and then stay open even after other elements have been clicked, such as buttons or links that expand hidden content</li>
<li class="li" data-list-item-id="ef4097b52abedb8170717eb0d528c35bb">
<strong class="ph b">Wait for selector</strong> and <strong class="ph b">Soft wait for selector</strong>: If the page has dynamic content that appears only after a certain amount of time, the crawler may miss it unless it is directed to wait. There are two ways to tell the crawler to wait with a CSS selector.<ul class="ul" id="topic_jgj_mcv_3hc__ul_lgm_wdv_3hc">
<li class="li" data-list-item-id="e8d1a689442a4d3c2e61645ce94a60183">The <strong class="ph b">Wait for dynamic content</strong> setting determines the amount of time the crawler waits. If the selector isn't found before the time limit, it's considered a failed request and will retry a couple times</li>
<li class="li" data-list-item-id="e175e6d332cbda10491f7703c67cfe760">The <strong class="ph b">Soft wait for selector</strong> determines how long the crawler waits, but also ensures the crawler continues to crawl the page if the selector isn't found, which prevents failures</li>
<li class="li" data-list-item-id="eb0c273f3a0e5bc32819e3ff55cfb1d51">These settings don't work with the Raw HTTP client (Cheerio) crawler type because it doesn't get any JavaScript content</li>
</ul>
</li>
<li class="li" data-list-item-id="e78f235d2d32691a9831d08036904141f">
<strong class="ph b">Maximum scroll height</strong>: Some pages are so long that the crawler gives up before the end. If you're missing content below a certain point, you can use this setting to force the crawler to scroll a specified number of pixels</li>
</ul>
</div>
</div>
<div class="topic nested2" id="topic_t5g_4cv_3hc" aria-labelledby="ariaid-title11">
<h3 class="title topictitle3" id="ariaid-title11">The crawl is returning too much or messy page content</h3>
<div class="body">
<p class="p">If your crawl has the right pages, but has extra or unnecessary content on those pages (for example, marketing text, navigation, headers or footers, or even cookies) that you suspect is interfering with the AI agent's answers, use the following <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8357749301658#topic_amk_g3w_cfc" target="_blank" rel="noopener noreferrer">advanced crawler settings</a> to exclude that content:</p>
<ul class="ul" id="topic_t5g_4cv_3hc__ul_ur1_zdv_3hc">
<li class="li" data-list-item-id="eeb4bb1403b4f990c2b440b1feaf24e6c">
<strong class="ph b">Keep HTML elements</strong>: Keep only specific HTML elements by providing one or more CSS selectors. All other content is ignored, helping you focus on relevant information. For many help centers, this is the simplest approach to make sure main article content is targeted while avoiding navigation, related articles, and unnecessary banners and headers</li>
<li class="li" data-list-item-id="ece82ef01523cd537b7c6418d5c53f825">
<strong class="ph b">Remove HTML elements</strong>: Use CSS selectors to specify which HTML elements to remove from the crawl. This is the most precise and powerful way to exclude specified, known content</li>
</ul>
</div>
</div>
</div>
</div>
<script id="docs-schema-markup" type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "TechArticle",
"@id": "https://support.zendesk.com/hc/en-us/articles/9961828111258",
"url": "https://support.zendesk.com/hc/en-us/articles/9961828111258",
"headline": "Troubleshooting issues with web crawler imports for AI agents (AI agents - Advanced only)",
"description": "You can troubleshoot web crawler import issues for AI agents by adjusting settings like crawl scope, timeout limits, and CSS selectors to ensure accurate and relevant content extraction from websites.",
"articleSection": "How to use AI agents - Advanced",
"datePublished": "2025-11-18T15:24:55+00:00",
"dateModified": "2026-06-08T16:07:21.667369+00:00",
"inLanguage": "en-US",
"mainEntityOfPage": {
"@type": "WebPage",
"@id": "https://support.zendesk.com/hc/en-us/articles/9961828111258"
},
"author": {
"@type": "Organization",
"name": "Zendesk",
"@id": "https://www.zendesk.com"
},
"publisher": {
"@type": "Organization",
"name": "Zendesk",
"@id": "https://www.zendesk.com"
},
"isPartOf": {
"@type": "WebSite",
"name": "Zendesk Help",
"@id": "https://support.zendesk.com/hc/"
},
"about": {
"@type": "SoftwareApplication",
"name": "Zendesk",
"@id": "https://www.zendesk.com",
"softwareAddOn": {
"@type": "SoftwareApplication",
"name": "AI agents",
"@id": "https://support.zendesk.com/hc/en-us/sections/8988994355994"
}
}
}
</script>
</div><p class="p">Related articles:</p><ul class="ul" id="adv_web_crawler_troubleshooting__ul_r1p_jbv_3hc">
<li class="li" data-list-item-id="ea5691fb4cf50dd0084bc08dccc8ba684"><a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9961827483162" target="_blank" rel="noopener noreferrer">Best practices for using a web crawler to import content for AI agents</a></li>
<li class="li" data-list-item-id="efbbef96ce8ffafb893a15846a9bd20f5"><a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9962106058906" target="_blank" rel="noopener noreferrer">Managing imported knowledge sources for AI agents</a></li>
</ul>