Make Sure You Are Using Meta Robots Tags Correctly
Chad Faith
Director of Content
What are meta robots tags? They contain values that instruct search engines what to follow and what not to follow. They are actually a piece of code that is often placed in the <head> section of your web pages. Don’t look down on them! These simple-looking codes can give you the power to decide about what pages you want to hide from web crawlers and which pages to index. Are you new to meta robots tag usage? Don’t panic. Read on to learn more about the various robots meta tag values and the effects they have on search engines.
Understanding Meta Robots Tag Values
- noydir: Prevents Yahoo! from using your page’s description as the snippet for your page in its search results (Yahoo! has closed its directory and this tag is deprecated. However, you might still come across it once in a while.)
- noyaca: Prevents snippets of SERPs from using your page description from the Yandex Directory.
- Unavailable_after: Informs search engines a date/time after which they should stop showing it in search results. Has to be in RFC850 format, e.g. Tuesday, 16-Apr-18 16:51:05 UTC.
- notranslate: Search engines stop showing translations of the page in their search results.
- nosnippet: Block search engines from showing a video or text snippet of your page in the search results. It also stops them from permitting access to a cached copy of your page in their results listings.
- noarchive: Stops a search engine from displaying a cached copy of your page in their search results listings.
- nocache: Same as the above, but only used by MSN or Live.
- noimageindex: Prevent images on your page(s) from being indexed.
- all: Shortcut for follow, index.
- none: Shortcut for nofollow, noindex.
- nofollow: Instructs search engines robots to stop following links on the page.
- follow: Tells search engines that they are allowed to follow links on the page, for the purpose of discovering other pages (default on all pages).
- noindex: Forbid search engines from adding this page to their index.
- index: Permit search engines to add the page to their index, so that it can be discovered by people who’s performing a search.
Pro tip: The above values or parameters can be inserted on their own, or together in the content attribute of tag. They should be separated by a comma as well.
Robots.txt Files, Conflicting Parameters, and Search Engine Rules
Do keep in mind that meta robots tags have different reactions to instructions in your robots.txt file. In addition, conflicting rules may cause unprecedented behaviors. For example, if the page is blocked via robots.txt, search engines will not be able to see your meta tags. Next, you should avoid setting conflicting values in your meta robots tag. For example, an individual can make the mistake of using both noindex and index parameters. In many cases of conflict, the most restrictive option is usually chosen. Search engines that have to choose between showing and not showing, will often go for the latter.
What should you do if you wish to send specific instructions to a specific search engine and not to others? In this case, you can alter the value of the content attribute to your target search engine, e.g. MSNBOT or GOOGLEBOT. You can have peace of mind knowing that some search engines will totally ignore instructions that they do not understand or support.
Not sure where to go from here? Working with an experienced SEO consultant can help you see which parameters work and which don’t work. When it comes to SEO, be it local or national, you do not need to go it alone!
