.Google.com's John Mueller responded to a question concerning why Google.com marks web pages that are forbidden from crawling through robots.txt as well as why the it's safe to disregard the related Explore Console records about those crawls.Robot Traffic To Question Specification URLs.The individual asking the inquiry chronicled that bots were actually generating hyperlinks to non-existent concern specification URLs (? q= xyz) to webpages with noindex meta tags that are additionally obstructed in robots.txt. What triggered the question is that Google is creeping the web links to those webpages, receiving obstructed through robots.txt (without envisioning a noindex robots meta tag) after that getting shown up in Google Search Console as "Indexed, though blocked out by robots.txt.".The person asked the complying with inquiry:." Yet listed here's the significant inquiry: why will Google index web pages when they can not also view the web content? What's the perk because?".Google's John Mueller validated that if they can not crawl the webpage they can't view the noindex meta tag. He also makes an interesting reference of the web site: search operator, urging to dismiss the results given that the "common" customers won't observe those end results.He created:." Yes, you are actually correct: if we can't creep the web page, our company can not view the noindex. That said, if our experts can't crawl the webpages, at that point there is actually certainly not a whole lot for our team to index. So while you might observe a number of those web pages along with a targeted web site:- question, the average consumer will not view them, so I would not bother it. Noindex is actually additionally alright (without robots.txt disallow), it simply implies the Links will definitely wind up being actually crawled (and wind up in the Browse Console document for crawled/not indexed-- neither of these statuses create concerns to the rest of the website). The important part is that you do not produce all of them crawlable + indexable.".Takeaways:.1. Mueller's response confirms the restrictions in using the Site: hunt accelerated search driver for analysis main reasons. Some of those factors is actually because it's certainly not linked to the regular search mark, it's a separate factor altogether.Google's John Mueller commented on the web site hunt operator in 2021:." The quick response is actually that a website: question is actually not suggested to be total, nor utilized for diagnostics functions.A web site query is a details sort of hunt that confines the results to a specific web site. It's essentially just words website, a bowel, and afterwards the site's domain name.This inquiry limits the outcomes to a details internet site. It is actually certainly not implied to become a detailed selection of all the web pages coming from that website.".2. Noindex tag without utilizing a robots.txt is fine for these sort of conditions where a bot is actually connecting to non-existent webpages that are acquiring discovered through Googlebot.3. URLs along with the noindex tag will definitely create a "crawled/not recorded" item in Browse Console and that those will not possess a bad effect on the remainder of the internet site.Check out the concern as well as respond to on LinkedIn:.Why will Google.com index pages when they can't also observe the web content?Included Graphic by Shutterstock/Krakenimages. com.