Keep a search engine from indexing a page

Step-by-step Playbooks
, , , ,
1

Contributors

@brandon-leuangpaseuth @andreea-macoveiciuc-content-expert

Business Benefits

Prevent search engines from indexing a page on your website.

Check whether a page is indexed by opening it up in your web browser.

  1. Press CTRL+U (Windows) or CMD+U (MacOS) to view the page source, CTRL+F or CMD+F to open the search box, and typing noindex.
  2. If you find a match in the source, then that page is currently not indexed. However, if the page still shows up in search results, skip to the last step.

Use search engine URL inspection tools to check whether search engines can currently index the page.

  • Navigate to the URL Inspection tool in your Google Search Console, type the page URL, and scroll to Coverage to check whether Google can index the page.
  • For Bing, go to URL Inspection in your Bing Webmaster dashboard, type the URL, and click on Inspect.

Decide whether you will use Meta robots tag or an X-Robots-Tag header.

If you don’t have access to a web developer, you want to handle this yourself, or you only want to keep search engines from indexing a few pages at a time, the meta robots tag is fine.

If you want search engines to not index pages with specific parameters or bulk pages, an X-Robots tag is easier. Reach out to your web developer to implement it.

Add a robots meta tag to the section of the HTML source code for the page you don’t want indexed.

Open the page in an HTML editor, add between the and `` tags, and save the file.

If you’re using WordPress, using a plugin like Yoast SEO makes this simpler. Open the page in the WordPress Page Editor on your WordPress dashboard, navigate to Yoast settings, and click on the Advanced tab. Select No under Allow search engines to show this Page in search results?

If you’re using any other page builders or ecommerce platforms, check your documentation for more information on how to edit a page’s source HTML.

Add page URLs to block lists in search engine webmaster tools to exclude them from search engine results.

  • This is pending when search engines crawl your site and recognize the robots meta tag.
  • In Google Search Console, navigate to Index > Removals. Click New Request and type the URL to be blocked in both the Temporarily Remove URL and Clear Cached URL tabs.
  • In Bing Webmasters, navigate to Configurations > Block URLs. Click the Add new URL to block button, type in the URL, and under Block type, select URL & Cache.

Last edited by @hesh_fekry 2023-11-14T16:25:32Z