Manage a robots.txt file

[CXL]

2 Playbooks

Audit your robots.txt file

Ensure that there’s no mistake in your robots.txt file restricting search engines from areas you want them to crawl.

Create a robots.txt file

Let search engines know which pages on the site to crawl and how they should interact with your site when crawling it.

Last edited by @hesh_fekry 2023-11-14T12:30:49Z