Discover and fix indexing mistakes in your XML sitemap.
Type the URL to your sitemap file into your browser address bar, usually domain.com/sitemap.xml or domain.com/sitemap_index.xml to check whether you already have an XML sitemap set up.
Create a sitemap if you can’t find one. See the additional resources for more information on how to do this.
When looking at your sitemap, hold the CTRL + F (Windows) or CMD + F (Mac) keys and type your domain name into the search box.
A Sitemap should include a maximum of 50,000 URLs to avoid URLs not being indexed.
If the number in the search tab is more than 50,000, break down your sitemap into more than one file.
If yours is more, share that sitemap into more than one file with each file having less than 50,000 URLs. Then follow the instruction in the create a sitemap playbook to set up your sitemap index file.
Install the Screaming Frog SEO Spider on your computer to discover errors, if any, in your sitemap.
In Screaming Frog, Click on Mode and select List. Click on Upload > Download XML Sitemap and type the URL to your sitemap. If you have multiple sitemaps, type the URL to your sitemap index sitemap and click on ok.
- Wait for the tool to read and crawl the sitemap(s).
- On the right bar, scroll to the Response Codes dropdown and select all but “All” and “Success (2xx)” one after the other. For each option you choose, if there are results, click on the export button to save it as a .CSV file to your computer.
Note down those URLs for fixing. Screaming Frog would specify the error.
A 404 error means the URL was not found though it’s on your sitemaps. Look at those URLs in your website to check if they exist. Check for instances where you changed the URL or if the link in your sitemap is the incorrect one:
- If it’s incorrect, edit your sitemap to add the correct URL.
- If you deleted the page or post, then take them out of your sitemap.
5xx errors have to do with your server, for this, reach out to your web hosting provider to fix it. That usually comes in instructions from the company or the support staff resolving the issue for you. Some companies have instructions on fixing this in their documentation.
Use webmaster tools like the Google Search Console and Bing Webmaster Tools to check for any errors or warnings for your sitemaps.
- Access the Google Search Console Account for your domain, and navigate to your sitemap just below ‘Coverage’ on the ‘Coverage’ page. For Bing, access Bing Webmaster Tools and click on ‘Sitemaps’.
- If you don’t see any sitemaps, it means that you haven’t submitted your sitemaps to search engines. For Google, log in to Google Search Console, click on ‘Sitemaps’ in the left menu, and enter your sitemap’s URL to submit it. For Bing, log into Bing Webmaster Tools, click on ‘Sitemaps’, and submit the URL to your sitemap or sitemap index.
Click on “Learn More” to see Google’s recommendations on fixing particular errors.
- When done with the above, click on Sitemaps on the left menu bar in your Bing Webmasters to confirm if you have submitted your sitemap.
- If it isn’t there, type the URL and submit.
- After submitting on Bing, if it displays ‘error’, confirm that you typed the accurate sitemap URL.
Type domain.com/robots.txt and check for a line that reads similar to this:
If you can’t find a line like this, add it to your robots.txt. To do that, leave a line between the items in your robots.txt and paste the above code.
Replace domain.com with your domain name, and replace sitemapurl.xml with your actual sitemap name, for example, sitemap.xml. If you have a sitemap index file, use that instead.
For instruction on how to edit a robots.txt file, check the additional resources. This can help all search engine crawlers, including those that you didn’t submit your sitemaps to, discover your sitemaps.