Creating a web crawler as an employee service AI agent knowledge source (EAP)

For employee service (ES) AI agents, you can create a web crawler as a knowledge
source to find answers by scraping pages on third-party websites. You must
be a Knowledge admin to create and manage knowledge connections.



What's my plan?






Employee Service Suite

Growth, Professional, Enterprise, or Enterprise Plus







Summary: ◀▼


You can create a web crawler as a knowledge source for employee service
AI agents to find answers by scraping third-party websites with sitemaps.
Choose public or session cookie authentication for access. The crawler
syncs initially and updates every six hours, allowing you to add this
source to AI agents for improved employee support knowledge.




Location:
Admin Center > AI > AI agents > Sources


For employee service (ES) AI agents, you can create a web crawler as
a knowledge source to
find answers by scraping pages on third-party websites. You must be a
Knowledge admin to
create and manage knowledge connections.


A web crawler connection relies on the target website having a sitemap
that the connection
can locate. Websites without sitemaps cannot be indexed. You can connect
using either the
website URL or the sitemap URL.



To create a web crawler connection


In
Admin Center,
click
AI agents
in the sidebar, then select Sources.


Click Add source, then for Web
crawler, click Connect.


Enter a unique connection name, then enter a website URL or a
sitemap URL.


Select an authentication method:


No authentication for public
websites that don't require a login.


Session cookie authentication
for websites that require a login.

If you
select this session cookie authentication, enter
a cookie key and
value.





Click Save.

The initial connection sync begins. It might take some time
to sync
all content, depending on the volume. After the initial sync,
the scraping cycle runs
every six hours to capture updates. The If-Modified-Since
header is used where supported
to determine whether pages have changed.


At this point, you can add the knowledge
source to one or more AI agents by clicking
Add to AI agent. Or, you can
wait to
create and deploy an ES AI agent with this knowledge source
at a later time (see
Creating, deploying and editing employee service AI agents
(EAP)).






{
"@context": "https://schema.org",
"@type": "TechArticle",
"@id": "https://support.zendesk.com/hc/en-us/articles/10861021658778",
"url": "https://support.zendesk.com/hc/en-us/articles/10861021658778",
"headline": "Creating a web crawler as an employee service AI agent knowledge source (EAP)",
"description": "You can create a web crawler as a knowledge source for employee service AI agents to scrape and index content from third-party websites with sitemaps, using either public access or session cookie authentication.",
"articleSection": "Connecting sources",
"datePublished": "2026-06-03T17:53:40+00:00",
"dateModified": "2026-07-31T22:33:29.893493+00:00",
"inLanguage": "en-US",
"mainEntityOfPage": {
"@type": "WebPage",
"@id": "https://support.zendesk.com/hc/en-us/articles/10861021658778"