Creating URL filters to scope website syncs

<body>
<p id="docs-hc-snippet" style="display: none;">When you set up a web crawler, you can choose to configure advanced settings that let you fine-tune your sync by adding URL filters. URL filters let you use wildcard patterns to define specific paths that narrow the scope of a website sync.</p>
<div class="article-banners">
<div id="docs-wmp-link">
<a href="https://support.zendesk.com/hc/en-us/articles/5411234991258-plan" rel="noopener" target="_blank">What's my plan?</a>
</div>

<div class="article-banner" id="suite_all">
<table>
<tbody>
<tr>
<td>
<strong>All Suites</strong>
</td>
<td>Team, Growth, Professional, Enterprise, or Enterprise Plus</td>
</tr>
</tbody>
</table>
</div>

<div class="article-banner" id="support_with_guide_all">
<table>
<tbody>
<tr>
<td>
<strong>Support with</strong>
</td>
<td>Knowledge Professional or Enterprise</td>
</tr>
</tbody>
</table>
</div>

</div>
<div id="docs-ai-summary">
<p id="docs-ai-summary-toggle">Summary: <span id="up-triangle">◀</span><span id="down-triangle">▼</span></p>
<p id="docs-ai-summary-content">URL filters let you narrow a web crawler sync by including or excluding specific pages with exact URLs, prefix matches, and wildcard patterns. You can avoid syncing admin or employee-only content, control which paths are crawled, and reduce manual setup. The article explains how include and exclude rules work, plus common pattern examples like single-level, multi-level, and subdomain matches.</p>
</div>

<div class="body">

<p class="p">When you set up a web crawler, you can choose to configure advanced settings that let you
fine-tune your sync by adding URL filters. URL filters let you use wildcard patterns to
define specific paths that narrow the scope of a website sync.</p>
<p class="p" dir="ltr">Instead of syncing everything under a broad starting point, you can define
which URLs should be included and which should be excluded. You can use exact URLs,
prefix matches, and wildcard patterns to narrow the sync scope, which helps reduce
manual setup and avoid syncing irrelevant pages such as admin or employee-only sections. </p>
<div class="p" dir="ltr">The web crawler uses URL filters to determine which URLs to include or exclude
in the crawl:<ul class="ul" id="topic_q3g_cvx_vjc__ul_xl5_qqf_dkc">
<li class="li">Exclude filters are evaluated first. If a URL matches an exclude filter, it will
be rejected, even if it matches an include filter.</li>
<li class="li">Include filters act as an allowlist when specified</li>
<li class="li">If no filters are configured, all URLs are allowed.</li>
</ul>
</div>
<div class="p">This article covers the following topics:<ul class="ul" id="topic_q3g_cvx_vjc__ul_q4g_3s2_dhc">
<li class="li"><a class="xref" href="#topic_orv_zxx_vjc">Understanding URL filter patterns</a></li>
<li class="li"><a class="xref" href="#topic_ask_kzx_vjc">Common URL filter patterns</a></li>
</ul>
</div>
<div class="p">Related articles:<ul class="ul" id="topic_q3g_cvx_vjc__ul_fq3_lxx_vjc">
<li class="li"><a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4593564000410" target="_blank">Using a web crawler to index external
content</a></li>
<li class="li"><a class="xref" href="https://support.zendesk.com/hc/en-us/articles/4708551571098" target="_blank">Managing web crawlers</a></li>
</ul>
</div>
</div>
<div aria-labelledby="ariaid-title2" class="topic nested1" id="topic_orv_zxx_vjc">
<h2 class="title topictitle2" id="ariaid-title2">Understanding URL filter patterns</h2>
<div class="body">
<p class="p" dir="ltr">When you set up a web crawler, you can use the optional URL discovery tab
in the set up flow to define URL filters that use wildcards to specify the exact
pages you want to sync. When using URL discovery, you can choose to either include
or exclude pages that match the paths and patterns you specify. </p>
<p class="p" dir="ltr">URL filters support exact matches, prefix matches, and wildcard patterns.
The pattern you enter determines whether the sync includes or excludes a single URL,
one level of a path, or multiple levels of a site structure. The following image
shows the URL discovery page in the web crawler setup flow where you can configure
URL filters that will include or exclude specific web pages. </p>
<p class="p"><img class="image img-screenshot" id="topic_orv_zxx_vjc__image_q3k_pby_ckc" src="https://zen-marketing-documentation.s3.amazonaws.com/docs/en/knw-webcrwl-urldiscover.png" width="700"></p>
<p class="p" dir="ltr">The same pattern can behave differently depending on whether you use it as
an inclusion or exclusion filter. For example, if you use
<code class="ph codeph">https://example.com/solutions/*</code> as an inclusion filter, pages
one level below <code class="ph codeph">/solutions/</code> are included, but deeper paths are not.
If you use the same pattern as an exclusion filter, those one-level pages are
excluded, but deeper paths are not excluded.</p>
<p class="p" dir="ltr">If you exclude an exact URL or path, all URLs under that excluded URL are
also excluded from the sync.</p>
<p class="p" dir="ltr">Use the following guidelines when creating URL filters:</p>
<ul class="ul" id="topic_orv_zxx_vjc__ul_fwy_byx_vjc">
<li class="li" dir="ltr">An exact URL with no wildcard and no trailing slash matches only that
specific URL.</li>
<li class="li" dir="ltr">A trailing slash with no wildcard acts as a prefix match for content
under that directory.</li>
<li class="li" dir="ltr">
<code class="ph codeph">*</code> matches characters within a single path or domain
segment and does not cross <code class="ph codeph">/</code>.</li>
<li class="li" dir="ltr">
<code class="ph codeph">**</code> matches across path segments, including
<code class="ph codeph">/</code>.</li>
<li class="li" dir="ltr">Wildcard patterns can be used in both inclusion and exclusion
filters.</li>
<li class="li" dir="ltr">In patterns such as <code class="ph codeph">/path/**/docs</code>,
<code class="ph codeph">**</code> requires at least one intermediate segment, so
<code class="ph codeph">/path/docs</code> does not match.</li>
</ul>
<div class="p">Before configuring URL filters, familiarize yourself with a few technical notes:<ul class="ul" id="topic_orv_zxx_vjc__ul_wdl_nrf_dkc">
<li class="li">The code strips whitespace from URLs before processing</li>
<li class="li">Non-percent-encoded unicode URLs are marked as malformed</li>
<li class="li">URL filters are applied before robots.txt rules</li>
<li class="li">Off-domain URLs are silently dropped before filter evaluation</li>
<li class="li">Pattern (glob) matching is case-sensitive</li>
</ul>
</div>
</div>
</div>
<div aria-labelledby="ariaid-title3" class="topic nested1" id="topic_ask_kzx_vjc">
<h2 class="title topictitle2" id="ariaid-title3">Common URL filter patterns</h2>
<div class="body">
<p class="p">The following table illustrates common URL filter patterns and their matches. </p>
<div class="p">
<div class="tablenoborder">
<table border="1" cellpadding="4" cellspacing="0" class="table" frame="border" id="topic_ask_kzx_vjc__table_jrq_mdz_vjc" rules="all" summary="">
<colgroup>
<col style="width:25%">
<col style="width:25%">
<col style="width:25%">
<col style="width:25%">
</colgroup>
<thead class="thead" style="text-align:left;">
<tr class="row">
<th class="entry nocellnorowborder" id="d87e201" style="vertical-align:top;">Pattern</th>
<th class="entry nocellnorowborder" id="d87e204" style="vertical-align:top;">Meaning</th>
<th class="entry nocellnorowborder" id="d87e207" style="vertical-align:top;">Matches</th>
<th class="entry cell-norowborder" id="d87e210" style="vertical-align:top;">Does not match</th>
</tr>
</thead>
<tbody class="tbody">
<tr class="row">
<td class="entry nocellnorowborder" headers="d87e201" style="vertical-align:top;">https://example.com/solutions</td>
<td class="entry nocellnorowborder" headers="d87e204" style="vertical-align:top;">Exact URL match only</td>
<td class="entry nocellnorowborder" headers="d87e207" style="vertical-align:top;">https://example.com/solutions</td>
<td class="entry cell-norowborder" headers="d87e210" style="vertical-align:top;">https://example.com/solutions/page1</td>
</tr>
<tr class="row">
<td class="entry nocellnorowborder" headers="d87e201" style="vertical-align:top;"><a class="xref" href="https://example.com/solutions/" target="_blank">https://example.com/solutions/</a></td>
<td class="entry nocellnorowborder" headers="d87e204" style="vertical-align:top;">Prefix match for anything under that directory</td>
<td class="entry nocellnorowborder" headers="d87e207" style="vertical-align:top;">https://example.com/solutions/,
https://example.com/solutions/page1</td>
<td class="entry cell-norowborder" headers="d87e210" style="vertical-align:top;"><a class="xref" href="https://example.com/solutions" target="_blank">https://example.com/solutions</a></td>
</tr>
<tr class="row">
<td class="entry nocellnorowborder" headers="d87e201" style="vertical-align:top;"><a class="xref" href="https://example.com/solutions/*" target="_blank">https://example.com/solutions/*</a></td>
<td class="entry nocellnorowborder" headers="d87e204" style="vertical-align:top;">Matches one level below /solutions/</td>
<td class="entry nocellnorowborder" headers="d87e207" style="vertical-align:top;">https://example.com/solutions/page1,
https://example.com/solutions/education</td>
<td class="entry cell-norowborder" headers="d87e210" style="vertical-align:top;">https://example.com/solutions/education/podcasting</td>
</tr>
<tr class="row">
<td class="entry nocellnorowborder" headers="d87e201" style="vertical-align:top;"><a class="xref" href="https://example.com/solutions/**" target="_blank">https://example.com/solutions/**</a></td>
<td class="entry nocellnorowborder" headers="d87e204" style="vertical-align:top;">Matches all levels below /solutions/</td>
<td class="entry nocellnorowborder" headers="d87e207" style="vertical-align:top;">https://example.com/solutions/page1,
https://example.com/solutions/education/podcasting</td>
<td class="entry cell-norowborder" headers="d87e210" style="vertical-align:top;"><a class="xref" href="https://example.com/solutions" target="_blank">https://example.com/solutions</a></td>
</tr>
<tr class="row">
<td class="entry nocellnorowborder" headers="d87e201" style="vertical-align:top;"> </td>
<td class="entry nocellnorowborder" headers="d87e204" style="vertical-align:top;"> </td>
<td class="entry nocellnorowborder" headers="d87e207" style="vertical-align:top;"> </td>
<td class="entry cell-norowborder" headers="d87e210" style="vertical-align:top;"> </td>
</tr>
<tr class="row">
<td class="entry nocellnorowborder" headers="d87e201" style="vertical-align:top;"><a class="xref" href="https://example.com/*.html" target="_blank">https://example.com/*.html</a></td>
<td class="entry nocellnorowborder" headers="d87e204" style="vertical-align:top;">Matches .html files in the root path only</td>
<td class="entry nocellnorowborder" headers="d87e207" style="vertical-align:top;">https://example.com/page.html,
https://example.com/index.html</td>
<td class="entry cell-norowborder" headers="d87e210" style="vertical-align:top;"><a class="xref" href="https://example.com/docs/page.html" target="_blank">https://example.com/docs/page.html</a></td>
</tr>
<tr class="row">
<td class="entry nocellnorowborder" headers="d87e201" style="vertical-align:top;"><a class="xref" href="https://example.com/**/*.html" target="_blank">https://example.com/**/*.html</a></td>
<td class="entry nocellnorowborder" headers="d87e204" style="vertical-align:top;">Matches .html files below the root at any depth</td>
<td class="entry nocellnorowborder" headers="d87e207" style="vertical-align:top;">https://example.com/docs/page.html,
https://example.com/a/b/c/index.html</td>
<td class="entry cell-norowborder" headers="d87e210" style="vertical-align:top;"><a class="xref" href="https://example.com/page.html" target="_blank">https://example.com/page.html</a></td>
</tr>
<tr class="row">
<td class="entry nocellnorowborder" headers="d87e201" style="vertical-align:top;">ttps://example.com/path/**/docs</td>
<td class="entry nocellnorowborder" headers="d87e204" style="vertical-align:top;">Matches /docs at any depth under /path/</td>
<td class="entry nocellnorowborder" headers="d87e207" style="vertical-align:top;">https://example.com/path/foo/docs,
https://example.com/path/foo/bar/docs</td>
<td class="entry cell-norowborder" headers="d87e210" style="vertical-align:top;"><a class="xref" href="https://example.com/path/docs" target="_blank">https://example.com/path/docs</a></td>
</tr>
<tr class="row">
<td class="entry nocellnorowborder" headers="d87e201" style="vertical-align:top;"><a class="xref" href="https://*.example.com/**" target="_blank">https://*.example.com/**</a></td>
<td class="entry nocellnorowborder" headers="d87e204" style="vertical-align:top;">Matches any subdomain of example.com</td>
<td class="entry nocellnorowborder" headers="d87e207" style="vertical-align:top;">https://blog.example.com/page,
https://shop.example.com/products/item</td>
<td class="entry cell-norowborder" headers="d87e210" style="vertical-align:top;"><a class="xref" href="https://example.com/page" target="_blank">https://example.com/page</a></td>
</tr>
<tr class="row">
<td class="entry row-nocellborder" headers="d87e201" style="vertical-align:top;"><a class="xref" href="https://*.com/**" target="_blank">https://*.com/**</a></td>
<td class="entry row-nocellborder" headers="d87e204" style="vertical-align:top;">Matches any .com domain</td>
<td class="entry row-nocellborder" headers="d87e207" style="vertical-align:top;">https://example.com/page, https://other.com/docs/guide,
https://sub.example.com/page</td>
<td class="entry cellrowborder" headers="d87e210" style="vertical-align:top;"><a class="xref" href="https://example.org/page" target="_blank">https://example.org/page</a></td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
<script id="docs-schema-markup" type="application/ld+json">{
"@context": "https://schema.org",
"@type": "TechArticle",
"@id": "https://support.zendesk.com/hc/en-us/articles/10993342489370",
"url": "https://support.zendesk.com/hc/en-us/articles/10993342489370",
"headline": "Creating URL filters to scope website syncs",
"description": "You learn how to create URL filters that include or exclude website paths with exact matches, prefixes, and wildcards to scope a crawler\u2019s sync.",
"articleSection": "Product help",
"datePublished": "2026-07-09T13:16:04+00:00",
"dateModified": "2026-08-20T11:53:19.274458+00:00",
"inLanguage": "en",
"mainEntityOfPage": {
"@type": "WebPage",
"@id": "https://support.zendesk.com/hc/en-us/articles/10993342489370"
},
"author": {
"@type": "Organization",
"name": "Zendesk",
"@id": "https://www.zendesk.com"
},
"publisher": {
"@type": "Organization",
"name": "Zendesk",
"@id": "https://www.zendesk.com"
},
"isPartOf": {
"@type": "WebSite",
"name": "Zendesk Help",
"@id": "https://support.zendesk.com/hc/"
},
"about": {
"@type": "SoftwareApplication",
"name": "Zendesk",
"@id": "https://www.zendesk.com"
}
}</script><script id="ai-summary-script">
//<![CDATA[
const ai_summary_style = document.createElement('style');
ai_summary_style.innerHTML = `#styled-article-body{padding:0 0 0}#article-body #docs-ai-summary{background-color:#f3f3f3;border:1px solid #d3d3d3;padding:10px 10px 6px;margin-bottom:16px}#article-body #docs-ai-summary #docs-ai-summary-toggle{font-size:.9em;font-weight:700;margin-bottom:4px;cursor:pointer;display:flex;justify-content:space-between;align-items:center}#article-body #docs-ai-summary #docs-ai-summary-toggle #up-triangle{margin-left:auto;padding-right:10px;display:none}#article-body #docs-ai-summary #docs-ai-summary-toggle #down-triangle{margin-left:auto;padding-right:10px}#article-body #docs-ai-summary #docs-ai-summary-content{font-size:.9em;padding-left:25px;padding-right:25px;margin-bottom:12px}`;
document.head.appendChild(ai_summary_style);

const toggleAISummary = () => {
const summary = document.getElementById("docs-ai-summary-content");
const upTriangle = document.getElementById("up-triangle");
const downTriangle = document.getElementById("down-triangle");
if (summary.style.display === "none") {
summary.style.display = "block";
upTriangle.style.display = "none";
downTriangle.style.display = "inline";
} else {
summary.style.display = "none";
upTriangle.style.display = "inline";
downTriangle.style.display = "none";
}
};

document.getElementById("docs-ai-summary-toggle").addEventListener("click", toggleAISummary);
//]]>
</script>
</body>