Error: Crawl blocked by the website

Issue symptoms

In my AI agent, I tried to add the link to my website under external content sources or external connections. However, the status of my website shows as Blocked.

Resolution steps

To solve this issue:

Open the details of the connection that failed

Click Learn how to update your robotx.txt file to read what went wrong

Fix the crawl accordingly

Alternatively, follow the hyperlink of this guide: Troubleshooting the web crawler