Let search engines know which pages on the site to crawl and how they should interact with your site when crawling it.
Some things that you might not want search engine bots to crawl are a staging version of your website or resources like a PDF. This will help you ensure non-relevant pages aren’t taking up your crawl budget leaving the relevant pages not crawled.
Check whether you already have a robots.txt file by typing domain.com/robots.txt into your browser address bar.
- Replace domain.com with your domain name.
- Some website builders, CMS platforms, and ecommerce platforms, like WordPress, BigCommerce, and Shopify, will automatically create a robots.txt file.
- You should see a 404 error page if you don’t have a robots.txt file.
- If you’re using WordPress and found a robots.txt file, log into your web hosting server and check for a robots.txt file in the root. If there’s none, it is a virtual file.
- Copy the content you saw in domain.com/robots.txt page into the file you just created.
For example, if you want to allow Googlebot, Google’s search engine bot, to crawl a specific page you would use:
```If you want to disallow Googlebot from crawling a page, you would use:
User-agent: Googlebot Disallow: /page-url/
- Replace `/page-url/` with the URL of the page you want the rules to target. ## Test your robots.txt by copying and pasting the file contents to a tester like [Google’s ](https://www.google.com/webmasters/tools/robots-testing-tool?utm_source=support.google.com/webmasters/&utm_medium=referral&utm_campaign=%206062598)or [TechnicalSEO’s](https://technicalseo.com/tools/robots-txt/) to check for any errors or warnings. If there are, check the line of code against what is written in robots.txt file and correct it. ## Save your robots.txt file and upload it to your domain’s root folder.