{"id":871,"date":"2024-05-24T06:52:47","date_gmt":"2024-05-24T06:52:47","guid":{"rendered":"https:\/\/webbullindia.com\/blogsss\/?p=871"},"modified":"2025-01-30T14:43:38","modified_gmt":"2025-01-30T14:43:38","slug":"google-has-cancelled-support-for-robots-txt-noindex","status":"publish","type":"post","link":"https:\/\/webbullindia.com\/blog\/google-has-cancelled-support-for-robots-txt-noindex\/","title":{"rendered":"Google Has Cancelled Support For Robots.txt Noindex"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Google has recently announced that GoogleBot will no longer use the Robots.txt which has been used by this search engine for page indexing purpose.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Google has given time till September 1, 2019 to publishers to remove the Robot.txt nonindex directive and start using alternative suggested by Google.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you are a New Player in SEO, this jargon may sound like a foreign term but, it\u2019s worth noting it. The GoogleBot or Google Crawler is an automation tool that goes through pages and index it in Google\u2019s database. Based on this indexation process, Google ranks websites accordingly.<\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_68 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title \" >Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/webbullindia.com\/blog\/google-has-cancelled-support-for-robots-txt-noindex\/#Now_its_time_to_know_why_Google_is_cancelling_it\" title=\"Now, it\u2019s time to know why Google is cancelling it-\">Now, it\u2019s time to know why Google is cancelling it-<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/webbullindia.com\/blog\/google-has-cancelled-support-for-robots-txt-noindex\/#1_Block_Search_Indexing_With_No-Index_Meta_Tag\" title=\"1. Block Search Indexing With No-Index Meta Tag:\">1. Block Search Indexing With No-Index Meta Tag:<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/webbullindia.com\/blog\/google-has-cancelled-support-for-robots-txt-noindex\/#2_Using_404_and_410_HTTP_Status_Codes\" title=\"2. Using 404 and 410 HTTP Status Codes:\">2. Using 404 and 410 HTTP Status Codes:<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/webbullindia.com\/blog\/google-has-cancelled-support-for-robots-txt-noindex\/#3_Use_Password_Protection\" title=\"3. Use Password Protection:\">3. Use Password Protection:<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/webbullindia.com\/blog\/google-has-cancelled-support-for-robots-txt-noindex\/#4_Disallow_Robots_Using_Robottxt\" title=\"4. Disallow Robots Using Robot.txt\">4. Disallow Robots Using Robot.txt<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/webbullindia.com\/blog\/google-has-cancelled-support-for-robots-txt-noindex\/#5_Use_The_Search_Console_Remove_URL_Tool\" title=\"5. Use The Search Console Remove URL Tool:\">5. Use The Search Console Remove URL Tool:<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading has-medium-font-size\"><span class=\"ez-toc-section\" id=\"Now_its_time_to_know_why_Google_is_cancelling_it\"><\/span>Now, it\u2019s time to know why Google is cancelling it-<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Google doesn\u2019t consider no-index Robot.txt as an official directive. It\u2019s true that Google used to be in support of it but, it didn\u2019t work efficiently in 8% of the cases. This directive was not a foolproof. Google has already instructed the websites that they have remove this directive till September 1, 2019.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This was the exact Tweet published by Google below<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-large\"><img fetchpriority=\"high\" decoding=\"async\" width=\"500\" height=\"393\" src=\"http:\/\/69.62.72.169\/webbullindia\/wordpresssss\/wp-content\/uploads\/2024\/05\/google-update.webp\" alt=\"\" class=\"wp-image-872\" srcset=\"https:\/\/webbullindia.com\/blog\/wp-content\/uploads\/2024\/05\/google-update.webp 500w, https:\/\/webbullindia.com\/blog\/wp-content\/uploads\/2024\/05\/google-update-300x236.webp 300w\" sizes=\"(max-width: 500px) 100vw, 500px\" \/><\/figure>\n<\/div>\n\n\n<p class=\"wp-block-paragraph\"><strong>So, now the inevitable question is \u2013 What Are the Alternatives Suggested by Google?<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Google definitely didn\u2019t want to leave websites and businesses in a clueless state of mind, so they have provided a comprehensive list of things that one could do-<\/p>\n\n\n\n<h2 class=\"wp-block-heading has-medium-font-size\"><span class=\"ez-toc-section\" id=\"1_Block_Search_Indexing_With_No-Index_Meta_Tag\"><\/span>1. Block Search Indexing With No-Index Meta Tag:<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">To stop Robot.txt to crawl from the indexing page, you need to use Noindex meta tag and, add it in the section of your page.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">< meta name=\u201drobots\u201d content=\u201dnoindex\u201d ><br>You can use HTTP response headers with X-Robots-Tag preventing crawler not to index your page.<br>HTTP\/1.1 200 OK<br>(\u2026)<br>X-Robots-Tag: noindex<\/p>\n\n\n\n<h2 class=\"wp-block-heading has-medium-font-size\"><span class=\"ez-toc-section\" id=\"2_Using_404_and_410_HTTP_Status_Codes\"><\/span>2. Using 404 and 410 HTTP Status Codes:<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">410 and 404 both are the similar status codes which is shown when the target resource is not available any more at the origin server.<\/p>\n\n\n\n<h2 class=\"wp-block-heading has-medium-font-size\"><span class=\"ez-toc-section\" id=\"3_Use_Password_Protection\"><\/span>3. Use Password Protection:<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">You can hide a page with the login option as Google never indexes a page which is password protected.<\/p>\n\n\n\n<h2 class=\"wp-block-heading has-medium-font-size\"><span class=\"ez-toc-section\" id=\"4_Disallow_Robots_Using_Robottxt\"><\/span>4. Disallow Robots Using Robot.txt<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">You can use the disallow directive in the robot.txt file to instruct search engine not to index your site.<\/p>\n\n\n\n<h2 class=\"wp-block-heading has-medium-font-size\"><span class=\"ez-toc-section\" id=\"5_Use_The_Search_Console_Remove_URL_Tool\"><\/span>5. Use The Search Console Remove URL Tool:<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">You can use this tool to remove URL temporary from the search engine.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you want to learn more about content removing procedure of Google\u2019s search engine, head to the Google Head Center.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Google has recently announced that GoogleBot will no longer use the Robots.txt which has been used by this search engine for page indexing&#8230;<\/p>\n","protected":false},"author":4,"featured_media":873,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_kadence_starter_templates_imported_post":false,"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"footnotes":""},"categories":[207],"tags":[200,205,206,203,202,86,204,201],"class_list":["post-871","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-google","tag-best-seo-company-in-noida","tag-google-news","tag-indexing-changes","tag-noindex-directive","tag-robots-txt","tag-seo-company-in-noida","tag-seo-updates","tag-top-seo-company-in-noida"],"_links":{"self":[{"href":"https:\/\/webbullindia.com\/blog\/wp-json\/wp\/v2\/posts\/871","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/webbullindia.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/webbullindia.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/webbullindia.com\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/webbullindia.com\/blog\/wp-json\/wp\/v2\/comments?post=871"}],"version-history":[{"count":0,"href":"https:\/\/webbullindia.com\/blog\/wp-json\/wp\/v2\/posts\/871\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/webbullindia.com\/blog\/wp-json\/wp\/v2\/media\/873"}],"wp:attachment":[{"href":"https:\/\/webbullindia.com\/blog\/wp-json\/wp\/v2\/media?parent=871"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/webbullindia.com\/blog\/wp-json\/wp\/v2\/categories?post=871"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/webbullindia.com\/blog\/wp-json\/wp\/v2\/tags?post=871"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}