Troubleshooting the web crawler

<body>
<p id="docs-hc-snippet" style="display: none;">The web crawler lets you crawl and index external content for usewherever you use external content in your Zendesk accountwithout developer resources (seeSetting up the web crawler). You can use this article to troubleshoot crawler setup and page errors that you may encounter while setting up the web crawler in your application.</p>
<div class="article-banners">
<div id="docs-wmp-link">
<a href="https://support.zendesk.com/hc/en-us/articles/5411234991258-plan" rel="noopener" target="_blank">What's my plan?</a>
</div>

<div class="article-banner" id="suite_all">
<table>
<tbody>
<tr>
<td>
<strong>All Suites</strong>
</td>
<td>Team, Growth, Professional, Enterprise, or Enterprise Plus</td>
</tr>
</tbody>
</table>
</div>

<div class="article-banner" id="support_with_guide_all">
<table>
<tbody>
<tr>
<td>
<strong>Support with</strong>
</td>
<td>Knowledge Professional or Enterprise</td>
</tr>
</tbody>
</table>
</div>

</div>
<div id="docs-ai-summary">
<p id="docs-ai-summary-toggle">Summary: <span id="up-triangle">◀</span><span id="down-triangle">▼</span></p>
<p id="docs-ai-summary-content">Use this guide to troubleshoot web crawler setup and indexing problems, including domain verification, sitemap errors, locale and title detection, HTTP status codes, invalid URL domains, robots.txt blocks, and record limits. It helps you identify why pages are not being crawled or indexed and what to check so your external content can be available in search and other places you use it.</p>
</div>

<div class="body">

<p class="p">The web crawler lets you crawl and index external content for use <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/9822956298010" target="_blank">wherever you use external content in your Zendesk account</a>
without developer resources (see <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4593564000410" target="_blank">Setting up the web crawler</a>). You can use this article to
troubleshoot crawler setup and page errors that you may encounter while setting up the
web crawler in your application. </p>
<p class="p">This article contains the following topics:</p>
<ul class="ul" id="troubleshooting_crawler__ul_nlc_4qp_ytb">
<li class="li"><a class="xref" href="#topic_exy_t5p_ytb">Crawler setup errors</a></li>
<li class="li"><a class="xref" href="#topic_j4h_3vc_x4b">Record errors</a></li>
<li class="li"><a class="xref" href="#topic_nch_5wh_zgc">Robot.txt errors</a></li>
</ul>
</div>
<div aria-labelledby="ariaid-title2" class="topic nested1" id="topic_exy_t5p_ytb">
<h2 class="title topictitle2" id="ariaid-title2">Crawler setup errors</h2>
<div class="body">
<p class="p">Crawler setup errors are generated when the web crawler cannot run successfully.
Crawler setup errors generate an email notification that is sent to the crawler
owner configured during web crawler setup.</p>
</div>
<div aria-labelledby="ariaid-title3" class="topic nested2" id="topic_klp_vvp_ytb">
<h3 class="title topictitle3" id="ariaid-title3">Domain ownership could not be verified</h3>
<div class="body">
<p class="p">The web crawler attempts to verify domain ownership each time it runs, which can
take up to 24 hours. If the domain verification fails, the crawler owner is
notified by email and the <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4708551571098" target="_blank">Crawlers page</a> will show a Crawl status
of "Domain verification failed."</p>
<p class="p">To troubleshoot domain verification errors, verify the homepage of your website
(otherwise known as the index or root page) is up and publicly available. The
page should not have any user login, password, IP restrictions, or other
authentication requirements.</p>
</div>
</div>
<div aria-labelledby="ariaid-title4" class="topic nested2" id="topic_n3p_qwp_ytb">
<h3 class="title topictitle3" id="ariaid-title4">Sitemap could not be processed</h3>
<div class="body">
<p class="p" dir="ltr">The web crawler uses the sitemap defined at <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4593564000410" target="_blank">crawler setup</a> each time it runs. If
the sitemap cannot be processed, the crawler owner receives an email
notification and the crawler will not run. If this happens, verify the
following:</p>
<div class="p">
<ul class="ul" id="topic_n3p_qwp_ytb__ul_mqm_dcc_ztb">
<li class="li" dir="ltr">The web crawler is pointing to the correct sitemap URL and can
locate it successfully. You can edit the crawler to view the current
sitemap URL. See<a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4708551571098" target="_blank"> Managing web crawlers</a>. </li>
<li class="li" dir="ltr">The sitemap is served and publicly available. The page should
not be restricted by any user login, password, IP restrictions, or other
authentication.</li>
<li class="li">The sitemap is an XML URL sitemap that follows the <a class="xref" href="https://www.sitemaps.org/protocol.html#validating" target="_blank"><u class="ph u">Sitemaps XML
protocol.</u></a>.</li>
</ul>
</div>
</div>
</div>
</div>
<div aria-labelledby="ariaid-title5" class="topic nested1" id="topic_j4h_3vc_x4b">
<h2 class="title topictitle2" id="ariaid-title5">Record errors</h2>
<div class="body">
<p class="p">Record errors occur when there are no setup errors, but the web crawler cannot
successfully scrape and index the pages defined in the crawler sitemap (<a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4593564000410" target="_blank">see Setting up the web crawler</a>). When a record error
occurs, an email notification is sent to the crawler owner with a link to a CSV file
that lists the affected pages and their associated errors. </p>
</div>
<div aria-labelledby="ariaid-title6" class="topic nested2" id="topic_q41_twp_ytb">
<h3 class="title topictitle3" id="ariaid-title6">Locale not detected</h3>
<div class="body">
<p class="p" dir="ltr">The error "Locale not detected" indicates that the web crawler could
not detect any locale or the detected locale does not match any current help
center locales.</p>
<p class="p" dir="ltr">To determine the locale of a record, the crawler tries the following
approaches. The first successful strategy determines the locale of the
records.</p>
<div class="p" dir="ltr">
<ol class="ol" id="topic_q41_twp_ytb__ol_pg1_rhc_ztb">
<li class="li" dir="ltr">Extract the locale from the lang attribute in the &lt;html&gt;
tag.</li>
<li class="li" dir="ltr">Extract the locale from the Content-Language header.</li>
<li class="li" dir="ltr">Extract the locale from the &lt;meta&gt; tag.</li>
<li class="li" dir="ltr">Perform textual analysis of the content (CLD - Compact
Language Detection).</li>
</ol>
</div>
<p class="p">The "Locale not detected" error results from of one of the following issues:</p>
<ul class="ul" id="topic_q41_twp_ytb__ul_jbf_c3c_ztb">
<li class="li">The locale or language identified does not match a locale or language
configured in any help center in your account. To see which languages are
configured in each help center in your account, see <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4408827609882" target="_blank">Configuring your help center to support
multiple languages</a>. Find the locale codes for your configured
languages in <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4408821324826-Zendesk-language-support-by-product#h_01EYXD488X3XK23TG9VPG0W6KS" target="_blank">Zendesk language support by
product</a>.</li>
<li class="li">The web crawler could not determine a locale or language.</li>
</ul>
<p class="p">To resolve this issue, verify the following:</p>
<ul class="ul" id="topic_q41_twp_ytb__ul_y4l_f3c_ztb">
<li class="li">The lang attribute in the html tag matches a locale from the help
center.</li>
<li class="li">The HTTP Content-Language header matches a locale from the Help center.</li>
<li class="li">The meta element with the Content-Language set in the http-equiv attribute
matches a locale from the help center.</li>
</ul>
<p class="p">See <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4707428787354" target="_blank">Understanding web crawler locales</a>.</p>
</div>
</div>
<div aria-labelledby="ariaid-title7" class="topic nested2" id="topic_skr_wwp_ytb">
<h3 class="title topictitle3" id="ariaid-title7">Title not detected</h3>
<div class="body">
<p class="p" dir="ltr">The error "Title not detected" indicates that the web crawler could not
detect the title of a record. The web crawler uses the following approaches to
determine the title of a record:</p>
<ol class="ol" id="topic_skr_wwp_ytb__ol_d5x_bjc_ztb">
<li class="li" dir="ltr">Extract the content of the &lt;title&gt; tag.</li>
<li class="li" dir="ltr">Extract the content of the &lt;h1&gt; tag.</li>
<li class="li" dir="ltr">Extract the textual content from the &lt;body&gt; tag.</li>
</ol>
<p class="p" dir="ltr">The first successful strategy determines the locale of the records. The
crawler only indexes the first 255 characters of the extracted content. The
record is not indexed if the above strategies didn’t determine any content.</p>
<p class="p">To resolve this issue, make sure the affected page has one of the tags listed
above.</p>
</div>
</div>
<div aria-labelledby="ariaid-title8" class="topic nested2" id="topic_lqg_xwp_ytb">
<h3 class="title topictitle3" id="ariaid-title8">Body not found</h3>
<div class="body">
<p class="p" dir="ltr">The error "Body not found" indicates that the web crawler could not
detect the body of a page. To resolve this error, make sure the affected page is
properly marked with the &lt;body&gt; tag.</p>
</div>
</div>
<div aria-labelledby="ariaid-title9" class="topic nested2" id="topic_ysv_nkq_xgc">
<h3 class="title topictitle3" id="ariaid-title9">HTTP [status code]</h3>
<div class="body">
<p class="p" dir="ltr">If the error code field in the CSV for a record contains HTTP and a
status code, it means that the page could not be indexed because the page could
not be accessed. If the page could be successfully indexed (HTTP 2xx) you will
not receive an HTTP status code error.</p>
<p class="p">The most common error codes are:</p>
<ul class="ul" id="topic_ysv_nkq_xgc__ul_k5b_pkq_xgc">
<li class="li">
<strong class="ph b">404 - Page not found</strong> - The page either does not exist or was moved to
another URL. To resolve this issue, make sure the sitemap that the crawler
is using is current and that all URLs in the <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4593564000410" target="_blank">sitemap point to existing
pages</a>.</li>
<li class="li">
<strong class="ph b">403 - Forbidden</strong> - The crawler is restricted from accessing the page
due to some access control mechanism, such as it being behind a log in or IP
address restriction. To resolve this issue, verify the following:<ul class="ul" id="topic_ysv_nkq_xgc__ul_l5b_pkq_xgc">
<li class="li">You have added Zendesk/External-Content, the web crawler user agent,
to your allowlist.</li>
<li class="li">The pages you want to index are publicly accessible, as the crawler
cannot crawl pages with restricted access. If the pages you want to
crawl and index cannot be made publicly accessible then you should
explore indexing them using the <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4408830243482" target="_blank">Federated Search (External
Content) API</a>.</li>
</ul>
</li>
<li class="li">
<strong class="ph b">5xx - Server error </strong>- The page could not be crawled due to a <a class="xref" href="https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#5xx_server_errors" target="_blank">server error</a>. The site may be
temporarily unavailable. To resolve this issue, visit the one or more of the
pages with this error to make sure the site is up. If the site is down,
contact the site administrator. When the error is fixed, wait for the
crawler to run again within its regular cadence (every 12-24 hours).</li>
</ul>
</div>
</div>
<div aria-labelledby="ariaid-title10" class="topic nested2" id="topic_mqx_xwp_ytb">
<h3 class="title topictitle3" id="ariaid-title10">Invalid URL domain</h3>
<div class="body">
<p class="p" dir="ltr">The error "Invalid URL domain" indicates the URL of the page listed in
the sitemap is not on the domain you configured during <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4593564000410" target="_blank">crawler setup</a>. </p>
<p class="p" dir="ltr">To resolve this issue, verify that the domain of the page that
triggered the error is on the same domain as is defined for your web crawler. If
the page linked in your sitemap is pointing to a page that is hosted on a
different domain from the one configured during crawler setup, you can do one of
the following:</p>
<ul class="ul" id="topic_mqx_xwp_ytb__ul_otq_rkq_xgc">
<li class="li">Set up a new web crawler for the affected page.</li>
<li class="li">Move the page from the external domain to the domain configured for the web
crawler.</li>
</ul>
</div>
</div>
<div aria-labelledby="ariaid-title11" class="topic nested2" id="topic_shq_ywp_ytb">
<h3 class="title topictitle3" id="ariaid-title11">Undetermined</h3>
<div class="body">
<p class="p" dir="ltr">The error "Undetermined" may be caused by one or more of the
following:</p>
<ul class="ul" id="topic_shq_ywp_ytb__ul_xqs_jlc_ztb">
<li class="li" dir="ltr">
<strong class="ph b">You have exceeded the external records limit for your instance
-</strong> If you've exceeded the <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4408831783962#topic_ndw_vxj_ytb" target="_blank">external records limit</a>, the latest
external records in excess of the limit will not be indexed or updated. To
resolve this issue you can do one or more of the following:<ul class="ul" id="topic_shq_ywp_ytb__ul_rhx_klc_ztb">
<li class="li">Delete some of your crawlers, whereby the external record of those
pages is deleted in your instance and then the pages previously not
indexed due to hitting the limit can be indexed. See <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4708551571098" target="_blank">Managing web
crawlers.</a>
</li>
<li class="li">Delete individual records via the <a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4699774781338" target="_blank">Federated Search API</a>.
However, if the crawler indexing this page is still active or if a
custom API integration that adds this page is active, the page will
reappear next time the crawler runs or the integration syncs.</li>
<li class="li">Remove pages that one or more crawlers are using from the sitemap.
The next time the crawler runs it will re-index the remaining pages
and delete the ones removed from the sitemap.</li>
<li class="li">Point one or more crawlers to a sitemap with fewer pages. The next
time the crawler runs it will re-index the remaining pages and
delete the ones removed from the sitemap.</li>
</ul>
</li>
<li class="li">
<strong class="ph b">The page is using JavaScript location redirects</strong> - The web crawler
does not observe <a class="xref" href="https://developer.mozilla.org/en-US/docs/Web/API/Window/location#example_1_navigate_to_a_new_page" target="_blank"><u class="ph u">JavaScript location
redirects</u></a>. If the page uses JavaScript location redirects,
the crawler cannot reach the content of the page. <p class="p">To resolve this issue,
do one of the following:</p>
<ul class="ul" id="topic_shq_ywp_ytb__ul_ctw_wmc_ztb">
<li class="li">Make sure the sitemap points directly to the page you want to
index.</li>
<li class="li">Implement <a class="xref" href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Redirections" target="_blank">HTTP redirects</a>.</li>
</ul>
</li>
</ul>
</div>
</div>
</div>
<div aria-labelledby="ariaid-title12" class="topic nested1" id="topic_nch_5wh_zgc">
<h2 class="title topictitle2" id="ariaid-title12">Robot.txt errors</h2>
<div class="body">
<p class="p" dir="ltr">A robots.txt file tells the crawler which parts of a website it's allowed
to access. Its primary purpose is to prevent overloading a website with excessive
crawl requests.</p>
<p class="p" dir="ltr">Rather than being a configuration step, robots.txt acts as a set of
guidelines that informs the crawler whether it can crawl the entire site or specific
sections. The only time customers need to engage with robots.txt is when the crawler
is blocked or the robots.txt file is invalid. In these cases, the system will
generate one of the following errors that must be addressed before the site can be
successfully crawled and synced.</p>
</div>
<div aria-labelledby="ariaid-title13" class="topic nested2" id="topic_kkg_xwh_zgc">
<h3 class="title topictitle3" id="ariaid-title13"><strong class="ph b">Crawl blocked by website</strong></h3>
<div class="body">
<p class="p" dir="ltr">This error occurs when the robots.txt file is configured to prevent all
user agents, including the crawler, from accessing the site.</p>
<p class="p">To ensure the Zendesk crawler has permission to access the site while optionally
blocking other crawlers, you can add an override rule to the robots.txt file to
allow the Zendesk crawler.</p>
<div class="p">Example 1: Allow only Zendesk/External-Content
<pre class="pre codeblock"><code>User-agent: Zendesk/External-Content
Allow: /</code></pre>
</div>
<div class="p">Example 2: Block Googlebot
<pre class="pre codeblock"><code>User-agent: Googlebot
Disallow: /</code></pre>
</div>
</div>
</div>
<div aria-labelledby="ariaid-title14" class="topic nested2" id="topic_yy1_2b3_zgc">
<h3 class="title topictitle3" id="ariaid-title14">Invalid robots.txt file</h3>
<div class="body">
<p class="p" dir="ltr">This error occurs when the robots.txt file exists but contains syntax
errors or invalid rules, making it unreadable by crawlers and causing them to
ignore or cancel the crawl.</p>
<p class="p" dir="ltr">To resolve this issue, review and correct your robots.txt file to
ensure it adheres to the proper syntax and accurately specifies crawler
permissions. Use online tools, such as <a class="xref" href="https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt" target="_blank">Google’s Robots Testing Tool</a>,
to validate your robots.txt file.</p>
</div>
</div>
</div>
<script id="docs-schema-markup" type="application/ld+json">{
"@context": "https://schema.org",
"@type": "TechArticle",
"@id": "https://support.zendesk.com/hc/en-us/articles/4593607724186",
"url": "https://support.zendesk.com/hc/en-us/articles/4593607724186",
"headline": "Troubleshooting the web crawler",
"description": "You troubleshoot web crawler setup, record, and robots.txt errors by checking domain, sitemap, locale, page structure, access, and redirect issues.",
"articleSection": "Using help center search",
"datePublished": "2022-05-11T12:12:24+00:00",
"dateModified": "2026-08-14T14:56:12.647280+00:00",
"inLanguage": "en",
"mainEntityOfPage": {
"@type": "WebPage",
"@id": "https://support.zendesk.com/hc/en-us/articles/4593607724186"
},
"author": {
"@type": "Organization",
"name": "Zendesk",
"@id": "https://www.zendesk.com"
},
"publisher": {
"@type": "Organization",
"name": "Zendesk",
"@id": "https://www.zendesk.com"
},
"isPartOf": {
"@type": "WebSite",
"name": "Zendesk Help",
"@id": "https://support.zendesk.com/hc/"
},
"about": {
"@type": "SoftwareApplication",
"name": "Zendesk",
"@id": "https://www.zendesk.com",
"softwareAddOn": {
"@type": "SoftwareApplication",
"name": "Knowledge",
"@id": "https://support.zendesk.com/hc/en-us/sections/4405298860954"
}
}
}</script><script id="ai-summary-script">
//<![CDATA[
const ai_summary_style = document.createElement('style');
ai_summary_style.innerHTML = `#styled-article-body{padding:0 0 0}#article-body #docs-ai-summary{background-color:#f3f3f3;border:1px solid #d3d3d3;padding:10px 10px 6px;margin-bottom:16px}#article-body #docs-ai-summary #docs-ai-summary-toggle{font-size:.9em;font-weight:700;margin-bottom:4px;cursor:pointer;display:flex;justify-content:space-between;align-items:center}#article-body #docs-ai-summary #docs-ai-summary-toggle #up-triangle{margin-left:auto;padding-right:10px;display:none}#article-body #docs-ai-summary #docs-ai-summary-toggle #down-triangle{margin-left:auto;padding-right:10px}#article-body #docs-ai-summary #docs-ai-summary-content{font-size:.9em;padding-left:25px;padding-right:25px;margin-bottom:12px}`;
document.head.appendChild(ai_summary_style);

const toggleAISummary = () => {
const summary = document.getElementById("docs-ai-summary-content");
const upTriangle = document.getElementById("up-triangle");
const downTriangle = document.getElementById("down-triangle");
if (summary.style.display === "none") {
summary.style.display = "block";
upTriangle.style.display = "none";
downTriangle.style.display = "inline";
} else {
summary.style.display = "none";
upTriangle.style.display = "inline";
downTriangle.style.display = "none";
}
};

document.getElementById("docs-ai-summary-toggle").addEventListener("click", toggleAISummary);
//]]>
</script>
</body>