Using a web crawler to index external content

<body>
<p id="docs-hc-snippet" style="display: none;">The web crawler lets you crawl and index external content for usewherever you use external content in your Zendesk accountwithout developer resources. You can set up multiple crawlers to crawl and index different content in the same or different websites. You can also crawl a list of specific URLs without having to crawl an entire site.</p>
<div class="article-banners">
<div id="docs-wmp-link">
<a href="https://support.zendesk.com/hc/en-us/articles/5411234991258-plan" rel="noopener" target="_blank">What's my plan?</a>
</div>

<div class="article-banner" id="suite_all">
<table>
<tbody>
<tr>
<td>
<strong>All Suites</strong>
</td>
<td>Team, Growth, Professional, Enterprise, or Enterprise Plus</td>
</tr>
</tbody>
</table>
</div>

<div class="article-banner" id="support_with_guide_all">
<table>
<tbody>
<tr>
<td>
<strong>Support with</strong>
</td>
<td>Knowledge Professional or Enterprise</td>
</tr>
</tbody>
</table>
</div>

</div>
<div id="docs-ai-summary">
<p id="docs-ai-summary-toggle">Summary: <span id="up-triangle">◀</span><span id="down-triangle">▼</span></p>
<p id="docs-ai-summary-content">Use the web crawler to index public external content for help center search and agent knowledge without developer help. You can crawl an entire site or specific URLs, apply URL filters, run multiple crawlers, and keep content searchable by source. The crawler supports adaptive crawling for JavaScript content, follows internal links only, and can keep previous results if a sync fails.</p>
</div>

<div class="body">

<p class="p">The web crawler lets you crawl and index external content for use <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9822956298010" target="_blank">wherever you use external content in your Zendesk account</a>
without developer resources. You can set up multiple crawlers to crawl and index
different content in the same or different websites. You can also crawl a list of
specific URLs without having to crawl an entire site. </p>
<p class="p">When users perform a search in your help center, relevant external content discovered by
the crawler is ranked and presented on the search results page, where users can filter
the results and click the links to view the external content link in another browser
tab.</p>
<div class="p">This article covers the following topics:<ul class="ul" id="guide_setting_up_search_crawler__ul_q4g_3s2_dhc">
<li class="li"><a class="xref" href="#topic_vs5_js2_dhc">About the web crawler</a></li>
<li class="li"><a class="xref" href="#topic_a4k_kfl_h2c">Setting up a web crawler</a></li>
</ul>
</div>
</div>
<div aria-labelledby="ariaid-title2" class="topic nested1" id="topic_vs5_js2_dhc">
<h2 class="title topictitle2" id="ariaid-title2">About the web crawler</h2>
<div class="body">
<p class="p">You can create <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4408831783962#topic_ndw_vxj_ytb" target="_blank">up to 50 web crawlers</a> to crawl and index
external content in the same or different websites. Content from crawled websites
and pages within websites are made available <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9822956298010" target="_blank">wherever you use external content in your Zendesk
account</a>. External sites that you want to crawl must be public
(non-authenticated). </p>
<div class="p">For each web crawler that you create, you can choose to either crawl an entire site
or limit your crawl to individual pages. If you choose to:<ul class="ul" id="topic_vs5_js2_dhc__ul_r5r_td1_zhc">
<li class="li">
<strong class="ph b">Crawl the entire site</strong>, the crawler automatically discovers links from
the configured start URLs, then uses them to crawl all pages within that
site. </li>
<li class="li">
<strong class="ph b">Limit your crawl to individual pages</strong>, you can specify up to 500 URLs
that you want to crawl. The <strong class="ph b">Sync only specified URLs </strong> option is
automatically selected if you enter more than one URL. However, if you enter
just one starting URL, you can still manually select this option to restrict
the crawl to a single page. If you enter a sitemap, this option is
deselected and disabled, as the crawler must crawl all pages within the
sitemap. </li>
</ul>
</div>
<p class="p">When you create a new crawler, the name that you assign to the crawler will be used
to create the Source value. <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4890968422298" target="_blank">Source values</a> are used as filters in your help center
search. If you want to change the name later, you can always edit or assign a
different source name. See <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4708551571098" target="_blank">Managing web crawlers</a>.</p>
<p class="p">You can use a simple crawler setup or you can choose to configure advanced settings
that let you create URL filters that fine-tune your crawler setup. You an create URL
filters that will either include or exclude specific URLs based on specific wildcard
patterns. See <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/10993342489370" target="_blank">Creating URL filters to scope webstite
syncs</a>.</p>
<p class="p">After the crawler is configured it runs periodically. If a sitemap is accepted as an
input to the start URL, the crawler follows the instructions on the sitemap.
Otherwise, the crawler discovers links automatically by following links on the top
level page to visit the secondary pages and ingest their content. The crawler
continues in this manner, visiting each linked page, and then following the links to
the next page in the site hierarchy. This continues until the crawler has either
crawled all linked content four levels deep or the entire site, whichever comes
first. The crawler only follows links that lead to pages within the site domain; no
external pages are visited.</p>
<p class="p">Web crawlers index content that is in the page source on the initial page load, even
if it's hidden by a UI element, such as an accordion. Web crawlers can also use
adaptive crawling to capture content that requires JavaScript to render. Before
processing each section of a website, the crawler automatically samples a small
number of pages and compares results between a standard fetch and a full browser
render. If the browser render captures significantly more content, the crawler
switches to browser mode for that section of the site. This means different parts of
the same website can be crawled in different modes. For example, a static blog
section might use standard fetching, while a dynamic app section uses browser
rendering. This process happens automatically with no configuration required.</p>
<p class="p">If the crawler fails to collect information from a website during a regularly
scheduled crawl (for example, if the website is down or if there are network
issues), the help center will retain the results from the previous crawl, which will
continue to be searchable in the help center.</p>
</div>
</div>
<div aria-labelledby="ariaid-title3" class="topic nested1" id="topic_a4k_kfl_h2c">
<h2 class="title topictitle2" id="ariaid-title3">Setting up a web crawler</h2>
<div class="body">
<div class="p">You can set up multiple crawlers in your help center to crawl and index different
content in the same or different websites. When setting up a web crawler, consider
the following:<ul class="ul" id="topic_a4k_kfl_h2c__ul_n1z_wcy_x4b">
<li class="li">The web crawler does not work with websites that use <a class="xref" href="https://www.gnu.org/software/gzip/" target="_blank">gzip</a> file compression encoding. You will not see search results
from these sites. </li>
<li class="li">A crawl-delay will not be respected by the web crawler when set on external
site robots.txt records. </li>
<li class="li">The changefreq tag doesn't affect the web crawler in any way.</li>
</ul>
</div>
<div class="p">
<div class="note" id="topic_a4k_kfl_h2c__note_ldx_cvv_ytb">
<span class="notetitle">Note:</span> You are responsible for using the help center web
crawler in compliance with all applicable laws and the terms and conditions of
the relevant websites. You should only add sitemaps where you own the domain
associated with such sitemaps. By using the help center web crawler, you confirm
that you own the domains for all sitemaps added to the crawler and that you have
the right to crawl such websites.</div>
</div>
<p class="p"><strong class="ph b">To set up the web crawler</strong></p>
<div class="p">
<ol class="ol" id="topic_a4k_kfl_h2c__ol_bkw_rpg_ytb">
<li class="li" dir="ltr">
<span class="ph">In <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9041943683354#topic_b5q_nwn_s2c" target="_blank">Knowledge admin</a>, click <strong class="ph b">Settings</strong> (<img class="image" id="topic_a4k_kfl_h2c__d8e85" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/guide_icon_settings.png">) in the sidebar.</span>
</li>
<li class="li" dir="ltr">Click <strong class="ph b">Search settings</strong>.</li>
<li class="li">Under <strong class="ph b">Crawlers</strong>, click <strong class="ph b">Manage</strong>.<p class="p"><img class="image img-screenshot" id="topic_a4k_kfl_h2c__image_ums_n3l_x5b" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/guide-search-settings-2026.png" width="600"></p>
</li>
<li class="li">Click <strong class="ph b">Add Crawler</strong>. <p class="p"><img class="image img-screenshot" id="topic_a4k_kfl_h2c__image_fbn_31v_ytb" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/guide-ew-add-crawler-no-crawler.png" width="600"></p>
</li>
<li class="li">Click <strong class="ph b">Continue</strong>.<p class="p"><img class="image" id="topic_a4k_kfl_h2c__image_ab4_33t_yhc" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/Crawler-connect-to-website.png" width="300"></p>
</li>
<li class="li">In the <strong class="ph b">Source and URLs</strong> tab, configure the following:<ul class="ul" id="topic_a4k_kfl_h2c__ul_myk_2rb_ckc">
<li class="li">
<strong class="ph b">Connection name</strong>: Identifies your web crawler on the crawler
management list, and is used to create the search source value used
as the filter in your help center search.</li>
<li class="li">
<strong class="ph b">Syncing frequency</strong>: Interval at which the web crawler runs and
syncs data to your Zendesk account. You can either accept the Weekly
default setting, or select Never (for a one-time sync), Daily, or
Monthly. </li>
<li class="li">Configure the following to specify the pages you want to crawl:<ul class="ul" id="topic_a4k_kfl_h2c__ul_glj_vsz_yjc">
<li class="li">
<strong class="ph b">URLs</strong>: Enter the URLs for the domain/pages that you
want to crawl (for example, test.com/faq.htm). You can enter
up to 500 URLs in this field. </li>
<li class="li">
<strong class="ph b">Sync only specified URLs</strong>: Specify whether you want to
crawl only the URLs you specified or all URLs inside the
specified pages. <ul class="ul" id="topic_a4k_kfl_h2c__ul_tds_gtz_yjc">
<li class="li">
<strong class="ph b">Select the check box</strong> if you want to crawl
only the URLs that you entered. For example, if you
entered example.com in the URL field and selected
this checkbox, the crawler will only crawl and index
the example.com page.</li>
<li class="li">
<strong class="ph b">Deselect the check box</strong> if you want to crawl
every page linked to the URLs you entered. For
example, if you entered example.com and deselected
this checkbox, the crawler will crawl and index the
example.com page and all pages linked within that
page.</li>
</ul>
</li>
</ul>
<img class="image img-screenshot" id="topic_a4k_kfl_h2c__image_dpv_xz5_ytb" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/knw-crawler-sourceandurl-new-080326.png" width="700">
</li>
</ul>
</li>
<li class="li">(Optional) To add URL filters that will include or exclude specific URLs
based on specific wildcard patterns:<ol class="ol" id="topic_a4k_kfl_h2c__ol_cfk_lly_ckc" type="a">
<li class="li">Click the <strong class="ph b">Advanced settings</strong> tab and toggle on <strong class="ph b">Look for
more URLs while syncing</strong>.<div class="p">
<img class="image" id="topic_a4k_kfl_h2c__image_sls_4sb_ckc" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/knw-newcrawl-toggle-adv.png" width="250"><div class="note" id="topic_a4k_kfl_h2c__note_u1g_4jy_ckc">
<span class="notetitle">Note:</span> For more information about URL filters
and how to create them, see <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/10993342489370" target="_blank">Creating URL filters to
scope website syncs</a>.</div>
</div>
</li>
<li class="li">
<p class="p">If you want to only crawl pages that</p>
<ul class="ul" id="topic_a4k_kfl_h2c__ul_ik3_2sb_ckc">
<li class="li">
<strong class="ph b">Match </strong>the wildcard patterns you create, select
<strong class="ph b">Limit sync to specific patterns</strong> and add the
patterns you want to sync, then enter the <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/10993342489370" target="_blank">URLs and wildcards</a>
you want to include in the <strong class="ph b">URLs </strong>field. </li>
<li class="li">
<strong class="ph b">Do not match</strong> the wildcard patterns you create, select
<strong class="ph b">Exclude specific patterns from syncing</strong> and enter
the <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/10993342489370" target="_blank">URLs and wildcards</a>
you want to exclude in the URLs field.</li>
</ul>
</li>
</ol>
</li>
<li class="li">Review the <strong class="ph b">Summary</strong> banner to verify your settings and make adustments
if necessary.<p class="p"><img class="image img-screenshot" id="topic_a4k_kfl_h2c__image_m4q_zly_ckc" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/knw-crawler-syncsummary.png" width="600"></p>
</li>
<li class="li">Click <strong class="ph b">Sync</strong>. <div class="p">The web crawler is added to the Crawlers page and the
crawler begins to crawl the pages and index its content. The crawler
will run at intervals defined in the syncing frequency field (see Step 6
above). If the crawler fails, the crawler owner will receive an email
notification with troubleshooting tips to help resolve the issue. The
crawler will try again periodically.<div class="note" id="topic_a4k_kfl_h2c__note_gt3_4rc_ztb">
<span class="notetitle">Note:</span> Zendesk/External-Content is the user agent for the web crawler. To
prevent the crawler from failing due to a firewall blocking
requests, whitelist (or allow-list)
<strong class="ph b">Zendesk/External-Content</strong>.</div>
</div>
<p class="p"><img class="image" id="topic_a4k_kfl_h2c__image_zzy_zct_13c" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/guide-oneclick-crawlerspage.png" width="700"></p>
</li>
</ol>
</div>
<div class="p">If you're setting up a web crawler to pull in external content for:<ul class="ul" id="topic_a4k_kfl_h2c__ul_o1y_lt2_dhc">
<li class="li">Help center search, then you need to select the content that you want to
include and exclude in help center search results. See <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4593607942298" target="_blank">Including external content in your help
center search results.</a>
</li>
<li class="li">Knowledge section of the context panel for agents, see <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/7263163614874" target="_blank">Configuring Knowledge</a> in the
context panel.</li>
</ul>
</div>
</div>
</div>
<script id="docs-schema-markup" type="application/ld+json">{
"@context": "https://schema.org",
"@type": "TechArticle",
"@id": "https://support.zendesk.com/hc/en-us/articles/4593564000410",
"url": "https://support.zendesk.com/hc/en-us/articles/4593564000410",
"headline": "Using a web crawler to index external content",
"description": "You can set up a web crawler to index public external content, choose whole sites or specific URLs, and configure sync settings and URL filters for search results.",
"articleSection": "Using help center search",
"datePublished": "2022-05-11T12:11:46+00:00",
"dateModified": "2026-08-20T11:57:32.101908+00:00",
"inLanguage": "en",
"mainEntityOfPage": {
"@type": "WebPage",
"@id": "https://support.zendesk.com/hc/en-us/articles/4593564000410"
},
"author": {
"@type": "Organization",
"name": "Zendesk",
"@id": "https://www.zendesk.com"
},
"publisher": {
"@type": "Organization",
"name": "Zendesk",
"@id": "https://www.zendesk.com"
},
"isPartOf": {
"@type": "WebSite",
"name": "Zendesk Help",
"@id": "https://support.zendesk.com/hc/"
},
"about": {
"@type": "SoftwareApplication",
"name": "Zendesk",
"@id": "https://www.zendesk.com",
"softwareAddOn": {
"@type": "SoftwareApplication",
"name": "Knowledge",
"@id": "https://support.zendesk.com/hc/en-us/sections/4405298860954"
}
}
}</script><script id="ai-summary-script">
//<![CDATA[
const ai_summary_style = document.createElement('style');
ai_summary_style.innerHTML = `#styled-article-body{padding:0 0 0}#article-body #docs-ai-summary{background-color:#f3f3f3;border:1px solid #d3d3d3;padding:10px 10px 6px;margin-bottom:16px}#article-body #docs-ai-summary #docs-ai-summary-toggle{font-size:.9em;font-weight:700;margin-bottom:4px;cursor:pointer;display:flex;justify-content:space-between;align-items:center}#article-body #docs-ai-summary #docs-ai-summary-toggle #up-triangle{margin-left:auto;padding-right:10px;display:none}#article-body #docs-ai-summary #docs-ai-summary-toggle #down-triangle{margin-left:auto;padding-right:10px}#article-body #docs-ai-summary #docs-ai-summary-content{font-size:.9em;padding-left:25px;padding-right:25px;margin-bottom:12px}`;
document.head.appendChild(ai_summary_style);

const toggleAISummary = () => {
const summary = document.getElementById("docs-ai-summary-content");
const upTriangle = document.getElementById("up-triangle");
const downTriangle = document.getElementById("down-triangle");
if (summary.style.display === "none") {
summary.style.display = "block";
upTriangle.style.display = "none";
downTriangle.style.display = "inline";
} else {
summary.style.display = "none";
upTriangle.style.display = "inline";
downTriangle.style.display = "none";
}
};

document.getElementById("docs-ai-summary-toggle").addEventListener("click", toggleAISummary);
//]]>
</script>
</body>