Google Has Cancelled Support For Robots.txt Noindex

Google has recently announced that GoogleBot will no longer use the Robots.txt which has been used by this search engine for page indexing purpose.

Google has given time till September 1, 2019 to publishers to remove the Robot.txt nonindex directive and start using alternative suggested by Google.

If you are a New Player in SEO, this jargon may sound like a foreign term but, it’s worth noting it. The GoogleBot or Google Crawler is an automation tool that goes through pages and index it in Google’s database. Based on this indexation process, Google ranks websites accordingly.

Now, it’s time to know why Google is cancelling it-

Google doesn’t consider no-index Robot.txt as an official directive. It’s true that Google used to be in support of it but, it didn’t work efficiently in 8% of the cases. This directive was not a foolproof. Google has already instructed the websites that they have remove this directive till September 1, 2019.

This was the exact Tweet published by Google below

So, now the inevitable question is – What Are the Alternatives Suggested by Google?

Google definitely didn’t want to leave websites and businesses in a clueless state of mind, so they have provided a comprehensive list of things that one could do-

1. Block Search Indexing With No-Index Meta Tag:

To stop Robot.txt to crawl from the indexing page, you need to use Noindex meta tag and, add it in the section of your page.

< meta name=”robots” content=”noindex” >
You can use HTTP response headers with X-Robots-Tag preventing crawler not to index your page.
HTTP/1.1 200 OK
(…)
X-Robots-Tag: noindex

2. Using 404 and 410 HTTP Status Codes:

410 and 404 both are the similar status codes which is shown when the target resource is not available any more at the origin server.

3. Use Password Protection:

You can hide a page with the login option as Google never indexes a page which is password protected.

4. Disallow Robots Using Robot.txt

You can use the disallow directive in the robot.txt file to instruct search engine not to index your site.

5. Use The Search Console Remove URL Tool:

You can use this tool to remove URL temporary from the search engine.

If you want to learn more about content removing procedure of Google’s search engine, head to the Google Head Center.