Learn SEO: The Robots Meta Tag

Learn how you can hack your way to the top with proven, data-backed tactics directly to your email every week.

The Robots meta tag provides directions to crawlers on how to present your websites to engines. Those directions are in the form of an HTML string value within the robots meta tag. 

One of the most common uses of the robots meta tag values are the noindex, nofollow, or noarchive tag.

<meta name="robots" content="noindex">

Warning: One crucial thing to keep in mind is that for engines such as Google, Bing, or Yahoo to follow the directions you set within the robots meta tag, they need to reach the page to retrieve that information. That means that if you add a noindex tag within your meta robots tag engines on a specific page, but you also block engines from accessing that page from your robots.txt, then they might still index your page if they find links pointing to it.  

Also, the robots tag usually applies to search engine crawlers, and you need to specify a specific non-crawler bot name otherwise. For example, 

<meta name="AdsBot-Google" content="noindex">

Ways to implement a robots meta tag

There are two ways to implement a robots meta tag, and they both have the same effect on crawlers. 

<meta> tag placed in <head> section

One of the most common ways to utilize the robots meta tag is by placing it as a meta tag in the head section of your page like this. 

meta robots tag html

By placing the value “robots” in the attribute “name” we specify that all crawlers should follow the value “noindex” in the attribute “content.” In many cases, you might want to refer to specific crawlers by using their user-agent name in the “name” attribute. In this case, you will have to replace the value “robots” with the user-agent you want to follow a specific directive. For example, 

<meta name="googlebot" content="noindex" />

One meta tag can contain multiple directives, and one page can contain various meta robot tags if you want to target different user-agents or directives. 

<meta name="robots" content="noindex, nofollow">
<meta name="googlebot" content="nosnippet">

X-Robots-Tag in HTTP Header

Similarly to using the robots meta tag, the robots directives can be applied as an X-Robots-Tag in the HTTP header of any given URL. 

Any directive used in the meta tag can also be applied to the HTTP header tag. For example 

HTTP/1.1 200 OK
Date: Sun, 29 June 2020 00:28 GMT
X-Robots-Tag: noindex

Or in case you want to reference a specific user-agent within the X-Robots-Tag here is an example

HTTP/1.1 200 OK
Date: Sun, 29 June 2020 00:28 GMT
X-Robots-Tag: googlebot: noindex

In case of the conflicting robots directives engines will select the most restrictive ones. For example,

<meta name="robots" content="max-snippet:20">
<meta name="robots" content="nosnippet">

Search engines will pick the “nosnippet” because it’s the most restrictive.

List of robots directives

  • all
    • There are no restrictions to robots. This is the default directive if you don’t have a robots meta tag set up. Nothing will change if you add it.
  • noindex
    • Suggest that engines should not index the page that the noindex tag appears.
  • nofollow
    • Asks engines not to follow any of the links that appear on the page.
  • none
    • None is an equivalent to noindex, nofollow combination
  • noarchive
    • Do not show a cache link on the SERPs
  • nosnippet
    • Do not show a text or video snippet in the SEPRs for a given page. Keep in mind that a static image might appear if it helps improve the user experience.
  • max-snippet
    • Maximum number of character that engines should show in the search engine result pages meta description.
  • max-image-preview
    • Maximum size of an image preview that engines are going to use.
  • max-video-preview
    • Maximum amount of seconds for a video preview.
  • notranslate
    • Don’t offer translation for the page in SERPs.
  • noimageindex
    • Do not index images that are on the page
  • unavailable_after
    • Do not show the page in the SERPs after the specified date.

There are plenty of practical implementations of robots meta tag usage. Especially when you need to implement it site-wide, X-Robots-Tag can be extremely useful since you can use regex to target flexible paths. Here’s an example of targeting all PDF files

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex, nofollow";

Robots Meta Tags for WordPress

Implementing robots meta tags for WordPress can be done quickly using SEO plugins such as Yoast, Rank Math, or All in One SEO.

Leave a Reply

Your email address will not be published. Required fields are marked *

Stay informed

Join hundreds of entrepreneurs, marketers and SEO specialists receiving a weekly data-backed, proven SEO tactic straight to their email.