.Google's John Mueller answered a question about why Google indexes web pages that are actually disallowed coming from creeping by robots.txt and also why the it's secure to neglect the associated Search Console files concerning those creeps.Bot Traffic To Question Specification URLs.The person inquiring the question documented that bots were actually creating hyperlinks to non-existent query guideline Links (? q= xyz) to pages along with noindex meta tags that are actually likewise obstructed in robots.txt. What urged the question is actually that Google.com is actually crawling the hyperlinks to those web pages, receiving blocked by robots.txt (without watching a noindex robotics meta tag) after that receiving shown up in Google Look Console as "Indexed, though shut out through robots.txt.".The person inquired the adhering to question:." Yet below's the big question: why will Google index webpages when they can't also observe the content? What's the advantage during that?".Google's John Mueller verified that if they can not creep the page they can't observe the noindex meta tag. He likewise produces an intriguing acknowledgment of the internet site: hunt driver, advising to dismiss the outcomes since the "common" users will not view those results.He created:." Yes, you are actually right: if our team can't crawl the page, our team can't see the noindex. That claimed, if we can't crawl the web pages, after that there's not a great deal for us to index. Thus while you might view a number of those webpages along with a targeted web site:- question, the typical consumer will not observe them, so I definitely would not bother it. Noindex is additionally fine (without robots.txt disallow), it only implies the Links will end up being actually crawled (as well as end up in the Browse Console record for crawled/not catalogued-- neither of these conditions lead to concerns to the remainder of the web site). The important part is actually that you don't create all of them crawlable + indexable.".Takeaways:.1. Mueller's solution verifies the limits in using the Internet site: search advanced search operator for diagnostic reasons. One of those explanations is actually because it's certainly not connected to the routine hunt index, it's a distinct point altogether.Google's John Mueller discussed the site hunt driver in 2021:." The short answer is that a site: inquiry is certainly not suggested to be comprehensive, nor made use of for diagnostics objectives.A site query is a specific sort of search that limits the end results to a specific web site. It's basically just the word website, a bowel, and afterwards the site's domain name.This concern restricts the end results to a certain site. It's not indicated to be a thorough selection of all the pages coming from that internet site.".2. Noindex tag without using a robots.txt is actually alright for these sort of conditions where a robot is linking to non-existent web pages that are obtaining discovered through Googlebot.3. Links along with the noindex tag will certainly create a "crawled/not listed" entry in Search Console and also those will not have an adverse impact on the remainder of the site.Review the inquiry and also respond to on LinkedIn:.Why would Google index webpages when they can not even observe the information?Included Graphic by Shutterstock/Krakenimages. com.