Importing knowledge sources to power generative replies in AI agents (AI agents - Advanced only)

<body>
<p id="docs-hc-snippet" style="display: none;">Knowledge sources are the information your AI agent uses to create AI-generated answers to questions from your customers. Adding knowledge sources to your AI agent lets it generate answers to help customers without requiring you to script every response.</p>
<div class="article-banners">
<div id="docs-wmp-link">
<a href="https://support.zendesk.com/hc/en-us/articles/5411234991258-plan" rel="noopener" target="_blank">What's my plan?</a>
</div>

<div class="article-banner" id="addon_ai-agents-adv">
<table>
<tbody>
<tr>
<td>
<strong>Add-on</strong>
</td>
<td>AI agents - Advanced</td>
</tr>
</tbody>
</table>
</div>

</div>

<div class="body">

<div class="p">
<div class="note attention" id="knowledge_import__note_o41_f3s_hjc">
<span class="attentiontitle"></span> <span class="ph">This article applies only to customers who bought AI agents
- Advanced before March 10, 2026. Otherwise, see</span>
<a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8357749301658" target="_blank">Connecting knowledge sources to power generative
replies in AI agents</a>.</div>
</div>
<p class="p">Knowledge sources are the information your AI agent uses to create AI-generated answers
to questions from your customers. Adding knowledge sources to your AI agent lets it
generate answers to help customers without requiring you to script every response. </p>
<p class="p">This article contains the following topics:</p>
<ul class="ul" id="knowledge_import__ul_ekh_v3w_cfc">
<li class="li"><a class="xref" href="#topic_qlm_13w_cfc">About knowledge sources</a></li>
<li class="li"><a class="xref" href="#topic_ehq_b3w_cfc">Importing a Zendesk help center</a></li>
<li class="li"><a class="xref" href="#topic_sxl_c3w_cfc">Importing a Salesforce help center</a></li>
<li class="li"><a class="xref" href="#topic_d31_d3w_cfc">Importing a Freshdesk help center</a></li>
<li class="li"><a class="xref" href="#topic_mjn_2mx_chc">Importing a Confluence site or space</a></li>
<li class="li"><a class="xref" href="#topic_hvw_f3w_cfc">Importing a CSV file</a></li>
<li class="li"><a class="xref" href="#topic_amk_g3w_cfc">Importing content with a web crawler</a></li>
</ul>
<p class="p">Related article:</p>
<ul class="ul" id="knowledge_import__ul_qm1_v3w_cfc">
<li class="li"><a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9185497386394" target="_blank">Configuring search rules for knowledge sources
for AI agents</a></li>
<li class="li"><a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9962106058906" target="_blank">Managing imported knowledge sources for AI agents
(AI agents - Advanced only)</a></li>
</ul>
</div>
<div aria-labelledby="ariaid-title2" class="topic nested1" id="topic_qlm_13w_cfc">
<h2 class="title topictitle2" id="ariaid-title2">About knowledge sources</h2>
<div class="body">
<p class="p">You can import the following different types of knowledge source into an advanced AI
agent:</p>
<ul class="ul" id="topic_qlm_13w_cfc__ul_b3b_q3s_hjc">
<li class="li">
<strong class="ph b">Help centers</strong>: Web-based help centers powered by Zendesk, Salesforce, or
Freshdesk.</li>
<li class="li">
<strong class="ph b">Confluence</strong>: Confluence sites or spaces that your organization uses to
store content.</li>
<li class="li">
<strong class="ph b">CSV files</strong>: CSV files formatted with article information.<p class="p">Importing CSV
files is a good solution for importing knowledge bases that aren't natively
supported or that are protected by single sign-on (SSO).</p>
</li>
<li class="li">
<strong class="ph b">Content imported with a web crawler</strong>: Information from a webpage or set of
webpages.<p class="p">This option is best suited for importing information from a
knowledge base, FAQ, or product description page. It is less suited to
e-commerce web shops. For e-commerce pages, Zendesk recommends <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8357756844442" target="_blank">building an integration</a> capable of
retrieving relevant product information and adding that information in a
<a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8357749494810" target="_blank">dialogue</a> or <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8979864563610" target="_blank">generative procedure</a>.</p>
</li>
</ul>
<p class="p">You can add multiple knowledge sources to a single AI agent. For example, you can
import articles from multiple Zendesk help centers, multiple CSV files, or a
combination of both. Nevertheless, we recommend keeping the overall number of
knowledge sources within a reasonable limit. Having lots of sources can in some
cases lead to reduced accuracy and increased latency.</p>
<p class="p">It’s important to understand that your AI agent doesn't search live data in a help
center, file, or website. Instead, the information is imported into the AI agent on
a one-time or recurring basis. The AI agent uses this imported information when
generating its replies.</p>
</div>
</div>
<div aria-labelledby="ariaid-title3" class="topic nested1" id="topic_ehq_b3w_cfc">
<h2 class="title topictitle2" id="ariaid-title3">Importing a Zendesk help center</h2>
<div class="body">
<p class="p">Client admins can import a Zendesk help center.</p>
<p class="p"><strong class="ph b">To import a Zendesk help center</strong></p>
<ol class="ol" id="topic_ehq_b3w_cfc__ol_fbf_lqw_cfc">
<li class="li"><span class="ph">In the AI agents workspace, <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8357756913178#topic_t25_qvj_tdc" target="_blank">select the AI agent</a> you want to work
with.</span></li>
<li class="li"><span class="ph">Click <img class="image" id="topic_ehq_b3w_cfc__d9e164" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/ai_agents_content_icon.png">
<strong class="ph b">Content</strong> in the sidebar, then select <strong class="ph b">Knowledge sources</strong>.</span></li>
<li class="li">Click <strong class="ph b">Add source</strong>.<p class="p">The Add source pane opens.</p>
</li>
<li class="li">In <strong class="ph b">Type</strong>, select <strong class="ph b">Zendesk</strong>. </li>
<li class="li">In <strong class="ph b">Help center URL</strong>, enter your subdomain, including the help center
locale (for example, <em class="ph i">https://yoursubdomain.zendesk.com/hc/en-us</em>).<p class="p">If
you don't provide a locale, the help center's default locale is
uploaded.</p>
</li>
<li class="li">In <strong class="ph b">Source name</strong>, enter a name for your source. <p class="p">This name is used in
reporting within the AI agents reporting dashboard.</p>
</li>
<li class="li">In <strong class="ph b">Import frequency</strong>, select how often the help center content should be
reimported: <ul class="ul" id="topic_ehq_b3w_cfc__ol_oyg_54r_hgc">
<li class="li">
<strong class="ph b">Daily</strong>: Content is reimported every day excluding Sundays and the
15th of the month. Not recommended unless your knowledge source is
updated very frequently.</li>
<li class="li">
<strong class="ph b">Weekly</strong>: Content is reimported each week on Sunday.</li>
<li class="li">
<strong class="ph b">Monthly</strong>: Content is reimported on the 15th of each month.</li>
<li class="li">
<strong class="ph b">Never</strong>: Content is imported once and is never reimported. <p class="p">The
exact timing of a reimport isn't guaranteed. The reimport is
processed during the day it's scheduled for, but might not always be
ready at a consistent time.</p>
<p class="p">Regular reimporting keeps the AI
agent up to date. For most organizations, a weekly or monthly
cadence is fine. Remember that you can always manually reimport if
new changes need to be reflected outside the scheduled reimport.</p>
<img class="image" id="topic_ehq_b3w_cfc__image_lwy_frw_cfc" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/aiaa_knowledge_importing_zendesk.png" width="450">
</li>
</ul>
</li>
<li class="li">If your help center requires users to sign in, or if you otherwise want to
import restricted articles: <ol class="ol" id="topic_ehq_b3w_cfc__ol_xnl_hrw_cfc" type="a">
<li class="li">Toggle <strong class="ph b">Import private articles</strong> on. </li>
<li class="li">In <strong class="ph b">Email</strong>, enter the email address of a user authorized to access
the restricted content.<p class="p">This is typically the email address of a
Knowledge admin.</p>
</li>
<li class="li">In <strong class="ph b">API access token</strong>, enter an <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4408889192858#topic_bsw_lfg_mmb" target="_blank">API token that you generate</a>
for this purpose. </li>
</ol>
</li>
<li class="li">Click <strong class="ph b">Import</strong>.</li>
</ol>
<div class="p">
<div class="note" id="topic_ehq_b3w_cfc__note_s1c_4qw_cfc">
<span class="notetitle">Note:</span> When importing restricted articles, if the email or API
token is incorrect, no warnings or errors are presented. Instead, only public
articles are imported, not restricted articles. It's highly recommended to
double-check the credentials before importing and test that the AI agent can
answer questions using restricted articles after the import has finished.</div>
</div>
</div>
</div>
<div aria-labelledby="ariaid-title4" class="topic nested1" id="topic_sxl_c3w_cfc">
<h2 class="title topictitle2" id="ariaid-title4">Importing a Salesforce help center</h2>
<div class="body">
<p class="p">Client admins can import a Salesforce help center.</p>
<p class="p"><strong class="ph b">To import a Salesforce help center</strong></p>
<ol class="ol" id="topic_sxl_c3w_cfc__ol_f4j_krw_cfc">
<li class="li"><span class="ph">In the AI agents workspace, <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8357756913178#topic_t25_qvj_tdc" target="_blank">select the AI agent</a> you want to work
with.</span></li>
<li class="li"><span class="ph">Click <img class="image" id="topic_sxl_c3w_cfc__d9e164" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/ai_agents_content_icon.png">
<strong class="ph b">Content</strong> in the sidebar, then select <strong class="ph b">Knowledge sources</strong>.</span></li>
<li class="li">Click <strong class="ph b">Add source</strong>.<p class="p">The Add source pane opens.</p>
</li>
<li class="li">In <strong class="ph b">Type</strong>, select <strong class="ph b">Salesforce</strong>. </li>
<li class="li">Click <strong class="ph b">Sign in to Salesforce</strong>. </li>
<li class="li">Log in to your Salesforce environment. </li>
<li class="li">In <strong class="ph b">Help center URL</strong>, enter the full URL of your Salesforce help
center.</li>
<li class="li">In <strong class="ph b">Source name</strong>, enter a name for your source.<p class="p">This name is used in
reporting within the AI agents reporting dashboard. </p>
</li>
<li class="li">In <strong class="ph b">Import frequency</strong>, select how often the help center content should be
reimported: <ul class="ul" id="topic_sxl_c3w_cfc__ol_r3r_zpr_hgc">
<li class="li">
<strong class="ph b">Daily</strong>: Content is reimported every day excluding Sundays and the
15th of the month. Not recommended unless your knowledge source is
updated very frequently.</li>
<li class="li">
<strong class="ph b">Weekly</strong>: Content is reimported each week on Sunday.</li>
<li class="li">
<strong class="ph b">Monthly</strong>: Content is reimported on the 15th of each month.</li>
<li class="li">
<strong class="ph b">Never</strong>: Content is imported once and is never reimported. <p class="p">The
exact timing of a reimport isn't guaranteed. The reimport is
processed during the day it's scheduled for, but might not always be
ready at a consistent time.</p>
<p class="p">Regular reimporting keeps the AI
agent up to date. For most organizations, a weekly or monthly
cadence is fine. Remember that you can always manually reimport if
new changes need to be reflected outside the scheduled
reimport.</p>
</li>
</ul>
</li>
<li class="li">Click <strong class="ph b">Import</strong>.</li>
</ol>
</div>
</div>
<div aria-labelledby="ariaid-title5" class="topic nested1" id="topic_d31_d3w_cfc">
<h2 class="title topictitle2" id="ariaid-title5">Importing a Freshdesk help center</h2>
<div class="body">
<p class="p">Client admins can import a Freshdesk help center.</p>
<p class="p"><strong class="ph b">To import a Freshdesk help center</strong></p>
<ol class="ol" id="topic_d31_d3w_cfc__ol_drw_mrw_cfc">
<li class="li"><span class="ph">In the AI agents workspace, <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8357756913178#topic_t25_qvj_tdc" target="_blank">select the AI agent</a> you want to work
with.</span></li>
<li class="li"><span class="ph">Click <img class="image" id="topic_d31_d3w_cfc__d9e164" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/ai_agents_content_icon.png">
<strong class="ph b">Content</strong> in the sidebar, then select <strong class="ph b">Knowledge sources</strong>.</span></li>
<li class="li">Click <strong class="ph b">Add source</strong>.<p class="p">The Add source pane opens.</p>
</li>
<li class="li">In <strong class="ph b">Type</strong>, select <strong class="ph b">Freshdesk</strong>. </li>
<li class="li">In <strong class="ph b">Help center URL</strong>, enter the full URL of your Freshdesk help
center.<p class="p">You can add your whole help center, or only a certain section of
your help center. </p>
</li>
<li class="li">In <strong class="ph b">Source name</strong>, enter a name for your source.<p class="p">This name is used in
reporting within the AI agents reporting dashboard. </p>
</li>
<li class="li">In <strong class="ph b">Import frequency</strong>, select how often the help center content should be
reimported:<ul class="ul" id="topic_d31_d3w_cfc__ol_tsw_1qr_hgc">
<li class="li">
<strong class="ph b">Daily</strong>: Content is reimported every day excluding Sundays and the
15th of the month. Not recommended unless your knowledge source is
updated very frequently.</li>
<li class="li">
<strong class="ph b">Weekly</strong>: Content is reimported each week on Sunday.</li>
<li class="li">
<strong class="ph b">Monthly</strong>: Content is reimported on the 15th of each month.</li>
<li class="li">
<strong class="ph b">Never</strong>: Content is imported once and is never reimported. <p class="p">The
exact timing of a reimport isn't guaranteed. The reimport is
processed during the day it's scheduled for, but might not always be
ready at a consistent time.</p>
<p class="p">Regular reimporting keeps the AI
agent up to date. For most organizations, a weekly or monthly
cadence is fine. Remember that you can always manually reimport if
new changes need to be reflected outside the scheduled
reimport.</p>
</li>
</ul>
</li>
<li class="li">In <strong class="ph b">API access token</strong>, enter an API token that you generate in Freshdesk
for this purpose. </li>
<li class="li">Click <strong class="ph b">Import</strong>.</li>
</ol>
</div>
</div>
<div aria-labelledby="ariaid-title6" class="topic nested1" id="topic_mjn_2mx_chc">
<h2 class="title topictitle2" id="ariaid-title6">Importing a Confluence site or space</h2>
<div class="body">
<p class="p">Client admins can import a Confluence site or space.</p>
<p class="p">Confluence connections are created and managed in Knowledge. You must <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9796599390874" target="_blank">create a Confluence connection in Knowledge</a> before you
can connect a Confluence site or space to your AI agent.</p>
<p class="p">Unlike other knowledge sources for AI agents, you can’t specify a reimport frequency
for Confluence content. Confluence connections automatically sync every 24 hours,
but you can <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9796584600218#topic_x2c_xlb_xgc" target="_blank">manually resync the content</a> if needed.</p>
<p class="p"><strong class="ph b">To import a Confluence site or space</strong></p>
<div class="p">
<ol class="ol" id="topic_mjn_2mx_chc__ol_g1l_mmx_chc">
<li class="li"><span class="ph">In the AI agents workspace, <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8357756913178#topic_t25_qvj_tdc" target="_blank">select the AI agent</a> you want to work
with.</span></li>
<li class="li"><span class="ph">Click <img class="image" id="topic_mjn_2mx_chc__d9e164" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/ai_agents_content_icon.png">
<strong class="ph b">Content</strong> in the sidebar, then select <strong class="ph b">Knowledge sources</strong>.</span></li>
<li class="li">Click <strong class="ph b">Add source</strong>.<p class="p">The Add source pane opens.</p>
</li>
<li class="li">In <strong class="ph b">Type</strong>, select <strong class="ph b">Confluence</strong>.<p class="p"><img class="image" id="topic_mjn_2mx_chc__image_zvg_3nx_chc" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/aiaa_import_confluence_source_authorize.png" width="300"></p>
</li>
<li class="li">In <strong class="ph b">Domain</strong>, enter the URL of your Zendesk subdomain (for example,
<code class="ph codeph">https://yoursubdomain.zendesk.com</code>).</li>
<li class="li">In <strong class="ph b">Email</strong>, enter the email address of a Zendesk admin. </li>
<li class="li">In API access token, enter an <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4408889192858#topic_bsw_lfg_mmb" target="_blank">API token that you generate</a> for
this purpose. </li>
<li class="li">Click <strong class="ph b">Authorize</strong>.</li>
<li class="li">Select a Confluence site or space that’s already been connected to your
Zendesk account, or <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9796599390874#topic_ftg_xjt_wgc" target="_blank">create a new Confluence
connection</a>.<p class="p">You can select more than one.</p>
<p class="p"><img class="image" id="topic_mjn_2mx_chc__image_fry_gnx_chc" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/aiaa_import_confluence_source_select.png" width="300"></p>
</li>
<li class="li">Click <strong class="ph b">Save</strong>.</li>
</ol>
</div>
<p class="p">The Confluence site or space you selected is added to your list of knowledge sources.
</p>
</div>
</div>
<div aria-labelledby="ariaid-title7" class="topic nested1" id="topic_hvw_f3w_cfc">
<h2 class="title topictitle2" id="ariaid-title7">Importing a CSV file</h2>
<div class="body">
<p class="p">Client admins can import a CSV file as a knowledge source.</p>
<p class="p"><strong class="ph b">To import a CSV file</strong></p>
<div class="p">
<ol class="ol" id="topic_hvw_f3w_cfc__ol_bwx_hsw_cfc">
<li class="li"><span class="ph">In the AI agents workspace, <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8357756913178#topic_t25_qvj_tdc" target="_blank">select the AI agent</a> you want to work
with.</span></li>
<li class="li"><span class="ph">Click <img class="image" id="topic_hvw_f3w_cfc__d9e164" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/ai_agents_content_icon.png">
<strong class="ph b">Content</strong> in the sidebar, then select <strong class="ph b">Knowledge sources</strong>.</span></li>
<li class="li">Click <strong class="ph b">Add source</strong>.<p class="p">The Add source pane opens.</p>
</li>
<li class="li">In <strong class="ph b">Type</strong>, select <strong class="ph b">File (CSV)</strong>. </li>
<li class="li">Click <strong class="ph b">Select knowledge source CSV file</strong>. </li>
<li class="li">Select the CSV file you want to import. <p class="p">See <a class="xref" href="#topic_nv3_n3w_cfc">Required formatting for the CSV file</a> to ensure your file is
properly formatted. </p>
</li>
<li class="li">In <strong class="ph b">Source name</strong>, enter a name for your source.<p class="p">This name is used in
reporting within the AI agents reporting dashboard.</p>
<p class="p"><img class="image" id="topic_hvw_f3w_cfc__image_kmk_psw_cfc" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/aiaa_knowledge_importing_csv.png" width="450"></p>
</li>
<li class="li">Click <strong class="ph b">Import</strong>.</li>
</ol>
</div>
<div class="p">
<div class="note" id="topic_hvw_f3w_cfc__note_tpc_ksw_cfc">
<span class="notetitle">Note:</span> CSV files cannot be reimported automatically. If your
content changes, you need to update the CSV file or create a new one and import
it again.</div>
</div>
</div>
<div aria-labelledby="ariaid-title8" class="topic nested2" id="topic_nv3_n3w_cfc">
<h3 class="title topictitle3" id="ariaid-title8">Required formatting for the CSV file</h3>
<div class="body">
<p class="p">The CSV file you upload as a knowledge source must have one row for each article
you want to import. The file must include the following columns: </p>
<ul class="ul" id="topic_nv3_n3w_cfc__ul_in1_rsw_cfc">
<li class="li">
<strong class="ph b">title</strong>: The title of the article. </li>
<li class="li">
<strong class="ph b">content</strong>: The full content of the article. <ul class="ul" id="topic_nv3_n3w_cfc__ul_stv_rsw_cfc">
<li class="li">The content can contain HTML tags, so there's no need to strip those
out. In fact, tags can help as they give structure to the articles,
and that structure helps the AI agent understand the article
sections. </li>
<li class="li">The content can also contain Markdown, but the Markdown must be
valid or else the content of that cell will not be imported.
Additionally, if the Markdown has been written in such a way that
the cell is a single line of more than 2,000 characters, importing
the cell fails without showing any warnings. </li>
</ul>
</li>
</ul>
<p class="p">You can also include the following optional columns: </p>
<ul class="ul" id="topic_nv3_n3w_cfc__ul_gnt_ssw_cfc">
<li class="li">
<strong class="ph b">labels</strong>: A list of label names separated by a space. The values can be
anything you want to categorize content by. </li>
<li class="li">
<strong class="ph b">locale</strong>: Used to organize articles by language or market. While the
values can technically be anything, it's recommended to follow standard
locale notation (for example, en-US or fi-FI). </li>
<li class="li">
<strong class="ph b">article_url</strong>: The external web address where the article can be found.
This is used in source attribution in the widget and reporting within the AI
agents reporting dashboard. </li>
</ul>
<p class="p">The file format must also use:</p>
<ul class="ul" id="topic_nv3_n3w_cfc__ul_uqg_p1s_hgc">
<li class="li">Commas (,) as a column separator and double quotes (") as the string quote
character. </li>
<li class="li">The first row for column headers.</li>
<li class="li">ASCII characters only. CSV files fail to import if they contain any
non-ASCII characters.</li>
</ul>
<p class="p">You can download a template at the bottom of this article.</p>
</div>
</div>
</div>
<div aria-labelledby="ariaid-title9" class="topic nested1" id="topic_amk_g3w_cfc">
<h2 class="title topictitle2" id="ariaid-title9">Importing content with a web crawler</h2>
<div class="body">
<p class="p">Client admins can import website content using a web crawler.</p>
<p class="p">For more information about web crawler imports, see <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9961827483162" target="_blank">Best practices for using a web crawler to import content for AI
agents</a> and <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9961828111258" target="_blank">Troubleshooting issues with web crawler imports for AI
agents</a>.</p>
<div class="p">
<div class="note" id="topic_amk_g3w_cfc__note_prl_5sw_cfc">
<span class="notetitle">Note:</span> Currently, you can't use the web crawler on a site
protected by single sign-on (SSO). Instead, you can <a class="xref" href="#topic_hvw_f3w_cfc">import a CSV file</a>.</div>
</div>
<p class="p"><strong class="ph b">To import web-crawled content</strong></p>
<div class="p">
<ol class="ol" id="topic_amk_g3w_cfc__ol_yvd_5sw_cfc">
<li class="li"><span class="ph">In the AI agents workspace, <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/8357756913178#topic_t25_qvj_tdc" target="_blank">select the AI agent</a> you want to work
with.</span></li>
<li class="li"><span class="ph">Click <img class="image" id="topic_amk_g3w_cfc__d9e164" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/ai_agents_content_icon.png">
<strong class="ph b">Content</strong> in the sidebar, then select <strong class="ph b">Knowledge sources</strong>.</span></li>
<li class="li">Click <strong class="ph b">Add source</strong>.<p class="p">The Add source pane opens.</p>
</li>
<li class="li">In <strong class="ph b">Type</strong>, select <strong class="ph b">Web crawler</strong>. </li>
<li class="li">In <strong class="ph b">Source name</strong>, enter a name for your source.<p class="p">This name is used in
reporting within the AI agents reporting dashboard.</p>
</li>
<li class="li">Select <strong class="ph b">Crawl exact URL</strong> if you want the web crawler to import
information only from webpages listed in the Start URLs field, not including
any sub-pages. <p class="p">When this option is not selected, the web crawler applies
a maximum crawling depth of 15 sub-pages for any URLs listed in Start
URLs.</p>
</li>
<li class="li">In <strong class="ph b">Start URLs</strong>, enter the URLs you want the web crawler to go
through.<p class="p">List each URL on a separate line. You can use an asterisk (*)
to denote wildcard values. If you do, note that the pattern
<em class="ph i">https://example.com/path/**</em> will:</p>
<ul class="ul" id="topic_amk_g3w_cfc__ul_mkg_fms_hjc">
<li class="li">Match the following URLs: <ul class="ul" id="topic_amk_g3w_cfc__ul_q41_lms_hjc">
<li class="li">
<em class="ph i">https://example.com/path/</em> (trailing slash)</li>
<li class="li"><em class="ph i">https://example.com/path/subpage</em></li>
</ul>
</li>
<li class="li">Not match the following URL:<ul class="ul" id="topic_amk_g3w_cfc__ul_hw5_rms_hjc">
<li class="li">
<em class="ph i">https://example.com/path</em> (no trailing slash) </li>
</ul>
</li>
</ul>
<p class="p">As a workaround, you can use both the exact URL
(<em class="ph i">https://example.com/path</em>) and the wildcard
(<em class="ph i">https://example.com/path/**</em>).</p>
</li>
<li class="li">In <strong class="ph b">Import frequency</strong>, select how often the crawled content should be
reimported: <ul class="ul" id="topic_amk_g3w_cfc__ol_f4k_cqr_hgc">
<li class="li">
<strong class="ph b">Daily</strong>: Content is reimported every day excluding Sundays and
the 15th of the month. Not recommended unless your knowledge source
is updated very frequently.</li>
<li class="li">
<strong class="ph b">Weekly</strong>: Content is reimported each week on Sunday.</li>
<li class="li">
<strong class="ph b">Monthly</strong>: Content is reimported on the 15th of each
month.</li>
<li class="li">
<strong class="ph b">Never</strong>: Content is imported once and is never reimported.
<p class="p">The exact timing of a reimport isn't guaranteed. The reimport
is processed during the day it's scheduled for, but might not
always be ready at a consistent time.</p>
<p class="p">Regular reimporting
keeps the AI agent up to date. For most organizations, a weekly
or monthly cadence is fine. Remember that you can always
manually reimport if new changes need to be reflected outside
the scheduled reimport.</p>
</li>
<li class="li"><img class="image" id="topic_amk_g3w_cfc__image_vrn_3tw_cfc" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/aiaa_knowledge_importing_web_crawler.png" width="450"></li>
</ul>
</li>
<li class="li">(Optional) Expand <strong class="ph b">Crawler settings</strong> to configure advanced crawler
settings. <p class="p">See <a class="xref" href="#topic_zsr_p3w_cfc">Configuring advanced crawler settings</a> for
details.</p>
<div class="note" id="topic_amk_g3w_cfc__note_njp_jtw_cfc">
<span class="notetitle">Note:</span> These settings are recommended
only for organizations with complex technical requirements. Many
organizations do not need these settings. </div>
</li>
<li class="li">(Optional) Expand <strong class="ph b">HTML processing</strong> to configure advanced HTML
settings. <p class="p">See <a class="xref" href="#topic_sll_s3w_cfc">Configuring advanced HTML settings</a> for
details.</p>
<div class="note" id="topic_amk_g3w_cfc__note_gcb_ntw_cfc">
<span class="notetitle">Note:</span> These settings are recommended
only for organizations with complex technical requirements. Many
organizations do not need these settings. </div>
</li>
<li class="li">Click <strong class="ph b">Import</strong>.</li>
</ol>
</div>
</div>
<div aria-labelledby="ariaid-title10" class="topic nested2" id="topic_zsr_p3w_cfc">
<h3 class="title topictitle3" id="ariaid-title10">Configuring advanced crawler settings</h3>
<div class="body">
<div class="p">
<ol class="ol" id="topic_zsr_p3w_cfc__ol_jnz_4tw_cfc">
<li class="li">
<a class="xref" href="#topic_amk_g3w_cfc">Under the Crawler settings
heading</a>, in <strong class="ph b">Crawler type</strong>, select one of the following
options: <ul class="ul" id="topic_zsr_p3w_cfc__ol_xgq_stw_cfc">
<li class="li">
<strong class="ph b">Adaptive switching between browser and raw HTTP
(Default)</strong>: Fast and renders JavaScript content if present. </li>
<li class="li">
<strong class="ph b">Headless browser (Firefox + Playwright)</strong>: Reliable,
renders JavaScript content, best in avoiding blocking, but might
be slow. </li>
<li class="li">
<strong class="ph b">Raw HTTP client (Cheerio)</strong>: Fastest, but doesn’t render
JavaScript content. </li>
<li class="li">
<strong class="ph b">Raw with JavaScript</strong>: Crawl the page as if using
JavaScript. </li>
</ul>
</li>
<li class="li">Select <strong class="ph b">Include URLs</strong> or <strong class="ph b">Exclude URLs</strong> to customize the
crawling scope set in the Start URLs field above. <p class="p">In the field below
each setting, enter the URLs you want to include or exclude. Enter
each URL on its own line.</p>
These settings affect only the links
found while crawling sub-pages. If you want to crawl a page, make sure
to specify its URL in the Start URLs field. <p class="p">For example, if the URL
structure is inconsistent as in the example below: </p>
<ul class="ul" id="topic_zsr_p3w_cfc__ul_x4q_25w_cfc">
<li class="li">Start URL:
<code class="ph codeph">https://support.example.com/en/support/home</code>
</li>
<li class="li">Article URL:
<code class="ph codeph">https://support.example.com/en/support/solutions/articles/…</code>
</li>
</ul>
You can add the following URL in the <strong class="ph b">Include URLs</strong> field: <ul class="ul" id="topic_zsr_p3w_cfc__ul_ofl_h5w_cfc">
<li class="li">
<code class="ph codeph">https://support.example.com/en/support/**</code>
</li>
</ul>
<p class="p">This way, the web crawler will include all of the articles, even
though their path is different from the starting URL. </p>
As another
example, the following page is very broad and includes irrelevant pages
(for example, the careers page): <ul class="ul" id="topic_zsr_p3w_cfc__ul_azf_k5w_cfc">
<li class="li">Start URL: <code class="ph codeph">https://www.example.com/en</code>
</li>
</ul>
To exclude these irrelevant pages, you can add the following URL in
the <strong class="ph b">Exclude URLs</strong> field: <ul class="ul" id="topic_zsr_p3w_cfc__ul_vx3_l5w_cfc">
<li class="li"><code class="ph codeph">https://www.example.com/en/careers/**</code></li>
</ul>
This way, the web crawler will exclude all of the content from the
careers page and its sub-pages.<div class="note tip" id="topic_zsr_p3w_cfc__note_fgf_ytw_cfc">
<span class="tiptitle">Tip:</span> More powerful than plain text, <em class="ph i">globs</em> are patterns that let
you use special characters to create dynamic URLs for the web
crawler to search. Here are a few examples: <ul class="ul" id="topic_zsr_p3w_cfc__ul_evw_ztw_cfc">
<li class="li">
<code class="ph codeph">https://support.example.com/**</code> lets the
crawler access all URLs starting with
https://support.example.com/. </li>
<li class="li">
<code class="ph codeph">https://{store,docs}.example.com/**</code> lets
the crawler access all URLs starting with
https://store.example.com/ or https://docs.example.com/. </li>
<li class="li">
<code class="ph codeph">https://example.com/**/*\?*foo=*</code> allows the
crawler to access all URLs that contain foo query parameters
with any value. </li>
</ul>
Learn more about globs and test them on the <a class="xref" href="https://www.digitalocean.com/community/tools/glob?comments=true&amp;glob=https%3A%2F%2Fexample.com%2Fscrape_this%2F%2A%2A&amp;matches=false&amp;tests=https%3A%2F%2Fexample.com%2Ftools%2F&amp;tests=https%3A%2F%2Fexample.com%2Fscrape_this%2F&amp;tests=https%3A%2F%2Fexample.com%2Fscrape_this%2F123%3Ftest%3Dabc&amp;tests=https%3A%2F%2Fexample.com%2Fdont_scrape_this" target="_blank">DigitalOcean
website</a>.</div>
</li>
<li class="li">In <strong class="ph b">Max pages to crawl</strong>, enter the maximum number of pages the web
crawler will go through, including the starting URL.<p class="p">This includes the
starting URL, pagination pages, pages with no content, and more. The
web crawler will automatically stop after reaching this
limit.</p>
</li>
<li class="li">In <strong class="ph b">Max crawling depth</strong>, enter the maximum number of links the web
crawler will follow from the starting URL.<p class="p">The starting URL has a
depth of 0. Pages linked directly from the starting URL have a depth
of 1, and so on. Use this setting to prevent accidental web crawler
runaway. </p>
</li>
<li class="li">In <strong class="ph b">Proxy configuration</strong>, select one of the following options: <ul class="ul" id="topic_zsr_p3w_cfc__ol_chh_55w_cfc">
<li class="li">
<strong class="ph b">Datacenter (Default)</strong>: Fastest method to scrape data. </li>
<li class="li">
<strong class="ph b">Residential</strong>: Reduced performance, but less likely to be
blocked. Ideal for when the default proxy is blocked or when you
need to crawl from a specific country.<p class="p"><img class="image" id="topic_zsr_p3w_cfc__image_ozb_r5w_cfc" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/aiaa_knowledge_importing_crawler_settings.png" width="450"></p>
</li>
</ul>
</li>
</ol>
</div>
</div>
</div>
<div aria-labelledby="ariaid-title11" class="topic nested2" id="topic_sll_s3w_cfc">
<h3 class="title topictitle3" id="ariaid-title11">Configuring advanced HTML settings</h3>
<div class="body">
<div class="p">
<ol class="ol" id="topic_sll_s3w_cfc__ol_gg2_ptw_cfc">
<li class="li">
<a class="xref" href="#topic_amk_g3w_cfc">Under the HTML processing
heading</a>, in <strong class="ph b">Keep HTML elements</strong>, enter a CSS selector
to keep only specified HTML elements.<p class="p">All other content will be
removed, helping you focus only on relevant information. </p>
</li>
<li class="li">In <strong class="ph b">Remove HTML elements</strong>, choose which HTML elements to remove
before converting to text, Markdown, or saving as HTML.<p class="p">This helps
exclude unwanted content. </p>
</li>
<li class="li">In <strong class="ph b">Expand clickable elements</strong>, enter a valid CSS selector matching
DOM elements that will be clicked.<p class="p">This is useful for expanding
collapsed sections in order to capture their text content. </p>
</li>
<li class="li">In <strong class="ph b">HTML transformer</strong>, select one of the following values to define
how to clean up the HTML to keep only important content and remove
extraneous content (for example, navigation or pop-ups): <ul class="ul" id="topic_sll_s3w_cfc__ol_sh3_bvw_cfc">
<li class="li">
<strong class="ph b">Extractus</strong>: (Not recommended) Uses the Extractus library. </li>
<li class="li">
<strong class="ph b">None</strong>: Removes only the HTML elements specified in the
Remove HTML elements option above. </li>
<li class="li">
<strong class="ph b">Readable text</strong>: Uses Mozilla's Readability library to
extract the main article content, removing navigation, headers,
footers, and other non-essential elements. Works best for
article-rich websites and blogs. </li>
<li class="li">
<strong class="ph b">Readable text if possible</strong>: Uses Mozilla's Readability
library to extract the main content, but falls back to the
original HTML if the page doesn't appear to be an article. This
is useful for websites with mixed content types, such as
articles or product pages, as it preserves more content on
non-article pages.<p class="p"><img class="image" id="topic_sll_s3w_cfc__image_ssw_dvw_cfc" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/aiaa_knowledge_importing_html_processing.png" width="450"></p>
</li>
</ul>
</li>
<li class="li">In <strong class="ph b">Wait for dynamic content</strong>, enter the number of seconds the
crawler should wait for dynamic content to load. By default, it waits
five seconds or until the page finishes loading, whichever comes
first.</li>
<li class="li">In <strong class="ph b">Soft wait for selector</strong>, enter CSS selectors for HTML elements
that the crawler should wait to load before extracting content.<p class="p">If the
selected element isn't present, the crawler still crawls that
page.</p>
<p class="p">List each CSS selector on a separate line.</p>
</li>
<li class="li">In <strong class="ph b">Wait for selector</strong>, enter CSS selectors for HTML elements that
the crawler must wait to load before extracting content.<p class="p">If the
selected element isn't present, the crawler doesn't crawl that
page.</p>
<p class="p">List each CSS selector on a separate line.</p>
</li>
<li class="li">In <strong class="ph b">Max scroll height</strong>, enter the maximum number of pixels the
crawler should scroll. <p class="p">The crawler scrolls the page to load more
content until the network is idle or this scroll height is reached.
Set this to 0 to disable scrolling completely.</p>
<p class="p">This setting
doesn't apply when using the raw HTTP client, as it doesn't run
JavaScript or load dynamic content.</p>
</li>
<li class="li">In <strong class="ph b">Make containers sticky</strong>, enter CSS selectors for HTML elements
where child content should be kept, even if it's hidden.<p class="p">List each CSS
selector on a separate line.</p>
<p class="p">This is helpful when using the
Expand clickable elements option on pages that remove hidden content
from the page entirely.</p>
<p class="p"><img class="image" id="topic_sll_s3w_cfc__image_px4_3ms_dfc" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/aiaa_knowledge_importing_html_processing_2.png" width="450"></p>
</li>
</ol>
</div>
</div>
</div>
</div>
<script id="docs-schema-markup" type="application/ld+json">{
"@context": "https://schema.org",
"@type": "TechArticle",
"@id": "https://support.zendesk.com/hc/en-us/articles/10791195957530",
"url": "https://support.zendesk.com/hc/en-us/articles/10791195957530",
"headline": "Importing knowledge sources to power generative replies in AI agents (AI agents - Advanced only)",
"description": "You can import various types of knowledge sources such as help centers, Confluence sites, CSV files, or web-crawled content into an advanced AI agent to enable it to generate AI-powered replies based on imported information rather than live data.",
"articleSection": "Configuring generative replies in AI agents",
"datePublished": "2026-05-19T17:39:53+00:00",
"dateModified": "2026-05-19T18:17:19.338488+00:00",
"inLanguage": "en-US",
"mainEntityOfPage": {
"@type": "WebPage",
"@id": "https://support.zendesk.com/hc/en-us/articles/10791195957530"
},
"author": {
"@type": "Organization",
"name": "Zendesk",
"@id": "https://www.zendesk.com"
},
"publisher": {
"@type": "Organization",
"name": "Zendesk",
"@id": "https://www.zendesk.com"
},
"isPartOf": {
"@type": "WebSite",
"name": "Zendesk Help",
"@id": "https://support.zendesk.com/hc/"
},
"about": {
"@type": "SoftwareApplication",
"name": "Zendesk",
"@id": "https://www.zendesk.com",
"softwareAddOn": {
"@type": "SoftwareApplication",
"name": "AI agents",
"@id": "https://support.zendesk.com/hc/en-us/sections/8988994355994"
}
}
}</script>
</body>