Best practices for using a web crawler to import content for AI agents (AI agents - Advanced only)

<body>
<p id="docs-hc-snippet" style="display: none;">You canuse a web crawler to import contentinto your AI agent. This gives your AI agent the ability to create AI-generated answers to customer questions based on information in external websites.</p>
<div class="article-banners">
<div id="docs-wmp-link">
<a href="https://support.zendesk.com/hc/en-us/articles/5411234991258-plan" rel="noopener" target="_blank">What's my plan?</a>
</div>

<div class="article-banner" id="addon_ai-agents-adv">
<table>
<tbody>
<tr>
<td>
<strong>Add-on</strong>
</td>
<td>AI agents - Advanced</td>
</tr>
</tbody>
</table>
</div>

</div>

<div class="body">

<div class="p">
<div class="note attention" id="adv_web_crawler_best_practices__note_u5r_zpv_p3c">
<span class="attentiontitle"></span> This article applies only if your Zendesk
account was created before March 10, 2026. Otherwise, see <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8357749301658" target="_blank">Connecting knowledge sources to power generative replies in AI
agents</a>.</div>
</div>
<p class="p">You can <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/10791195957530" target="_blank">use a web crawler to import content</a> into your
AI agent. This gives your AI agent the ability to create AI-generated answers to
customer questions based on information in external websites. </p>
<p class="p">This article gives you some best practices for using a web crawler to import content for
an AI agent. </p>
<p class="p">This article contains the following topics:</p>
<ul class="ul" id="adv_web_crawler_best_practices__ul_drd_xz5_3hc">
<li class="li"><a class="xref" href="#topic_npd_jz5_3hc">Use the web crawler on the right type of sites</a></li>
<li class="li"><a class="xref" href="#topic_oz1_kz5_3hc">Limit reimports to a reasonable frequency</a></li>
<li class="li"><a class="xref" href="#topic_y2t_lz5_3hc">Keep the overall number of knowledge sources low</a></li>
<li class="li"><a class="xref" href="#topic_rzd_mz5_3hc">Check the import summary</a></li>
<li class="li"><a class="xref" href="#topic_ovw_mz5_3hc">Start small and test</a></li>
</ul>
<p class="p">Related articles: </p>
<ul class="ul" id="adv_web_crawler_best_practices__ul_dd2_vz5_3hc">
<li class="li"><a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9961828111258" target="_blank">Troubleshooting issues with web crawler imports
for AI agents</a></li>
<li class="li"><a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9962106058906" target="_blank">Managing imported knowledge sources for AI
agents</a></li>
</ul>
</div>
<div aria-labelledby="ariaid-title2" class="topic nested1" id="topic_npd_jz5_3hc">
<h2 class="title topictitle2" id="ariaid-title2">Use the web crawler on the right type of sites</h2>
<div class="body">
<p class="p">The web crawler is best suited for websites that function as help centers or product
description pages. For e-commerce pages, we recommend <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8357756844442" target="_blank">building an integration</a> capable of retrieving relevant
product information and adding that information in a <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8357749494810" target="_blank">dialogue</a> or <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8979864563610" target="_blank">procedure</a>. </p>
<p class="p">It’s recommended to use a Zendesk help center as your primary knowledge source.
Websites can have any format, including dynamic elements and JavaScript, which means
they’re much harder to predictably ingest. While the web crawler has powerful <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8357749301658#topic_amk_g3w_cfc" target="_blank">configuration options</a>, these require
enablement and practice. Zendesk help centers are, by nature, simpler and more
predictable in format, leading to better results. Imports should also generally be
faster when using a Zendesk help center.</p>
<p class="p">Only publicly accessible websites can be crawled. If a website requires
authentication, the web crawler can't access it.</p>
</div>
</div>
<div aria-labelledby="ariaid-title3" class="topic nested1" id="topic_oz1_kz5_3hc">
<h2 class="title topictitle2" id="ariaid-title3">Limit reimports to a reasonable frequency</h2>
<div class="body">
<p class="p">Imports aren't a real-time web search. The AI agent doesn't search live data in a
help center, file, or website. Rather, the information is imported into the AI agent
on a one-time or recurring basis. The AI agent uses this imported information when
generating its replies. </p>
<p class="p">Daily imports aren’t recommended unless the knowledge source is updated very
frequently. For most organizations, a weekly or monthly cadence is fine. Remember
that you can always <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9962106058906#topic_mxn_4n5_3hc" target="_blank">manually reimport</a> if new changes need to
be reflected outside the scheduled reimport.</p>
</div>
</div>
<div aria-labelledby="ariaid-title4" class="topic nested1" id="topic_y2t_lz5_3hc">
<h2 class="title topictitle2" id="ariaid-title4">Keep the overall number of knowledge sources low</h2>
<div class="body">
<p class="p">You can add multiple knowledge sources to a single AI agent, including multiple web
crawls. Nevertheless, it’s recommended to keep the overall number of knowledge
sources within a reasonable limit. In some cases, having lots of sources can lead to
reduced accuracy and increased latency.</p>
</div>
</div>
<div aria-labelledby="ariaid-title5" class="topic nested1" id="topic_rzd_mz5_3hc">
<h2 class="title topictitle2" id="ariaid-title5">Check the import summary</h2>
<div class="body">
<p class="p">If you have a successful crawl but encounter other issues (for example, the AI
agent’s answers are incomplete or poor), you can <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9962106058906#topic_jxx_rn5_3hc" target="_blank">review the import summary</a> to check whether
all expected URLs and content were imported. This is the first and best way to
understand what’s been imported and what to troubleshoot after import.</p>
</div>
</div>
<div aria-labelledby="ariaid-title6" class="topic nested1" id="topic_ovw_mz5_3hc">
<h2 class="title topictitle2" id="ariaid-title6">Start small and test</h2>
<div class="body">
<p class="p">If you want to check whether content was crawled correctly and you have pages that
follow a specific pattern, the fastest thing to do is restrict your crawl just to
one or two examples of those pages. You can use a <strong class="ph b">Start URL</strong> of one target
page and a <strong class="ph b">Max crawling depth</strong> of zero. Alternatively, you can set the <strong class="ph b">Max
pages to crawl</strong> to some low number that can be quickly processed.</p>
</div>
</div>
<script id="docs-schema-markup" type="application/ld+json">{
"@context": "https://schema.org",
"@type": "TechArticle",
"@id": "https://support.zendesk.com/hc/en-us/articles/9961827483162",
"url": "https://support.zendesk.com/hc/en-us/articles/9961827483162",
"headline": "Best practices for using a web crawler to import content for AI agents (AI agents - Advanced only)",
"description": "You can use a web crawler to import content from publicly accessible websites into your AI agent, following best practices to optimize import frequency, source selection, and troubleshooting.",
"articleSection": "AI agent best practices",
"datePublished": "2025-11-18T15:23:55+00:00",
"dateModified": "2026-06-08T16:01:07.520857+00:00",
"inLanguage": "en-US",
"mainEntityOfPage": {
"@type": "WebPage",
"@id": "https://support.zendesk.com/hc/en-us/articles/9961827483162"
},
"author": {
"@type": "Organization",
"name": "Zendesk",
"@id": "https://www.zendesk.com"
},
"publisher": {
"@type": "Organization",
"name": "Zendesk",
"@id": "https://www.zendesk.com"
},
"isPartOf": {
"@type": "WebSite",
"name": "Zendesk Help",
"@id": "https://support.zendesk.com/hc/"
},
"about": {
"@type": "SoftwareApplication",
"name": "Zendesk",
"@id": "https://www.zendesk.com",
"softwareAddOn": {
"@type": "SoftwareApplication",
"name": "AI agents",
"@id": "https://support.zendesk.com/hc/en-us/sections/8988994355994"
}
}
}</script>
</body>