First reported by Greg Boser on his SEO Rockstars radio show, it seems that Google’s Media-Partner bot is now being used for feeding the index.
Word out of the presently-running WebmasterWorld conference is that Matt Cutts of Google confirms that this is the case. Shoemoney reports that “Matt said it is a bandwidth saving feature to have GoogleBot and MediaBot both contributing to big daddy.”
Seems to me a simple test would be to print out the visitor’s user-agent in the web page and see what gets indexed. As a test, I’ll do that on this site (see the bottom of the sidebar for your user-agent). But would that qualify as “cloaking” and get me in trouble for serving different content to the Googlebot and human visitors?
According to Shoemoney, Matt also stated that having Adsense gives zero advantage to getting listed (of course) but that you could get in trouble by serving different content to the Mediabot and Googlebot. It would be ridiculous for me to get in “trouble” for posting the users’ user agent.
What gets me here is that Google’s potentially bypassing webmaster’s robots.txt blocking of the traditional googlebot and getting by with a different user agent. There are plenty of reasons that a webmaster may want a page to not get indexed, but be available for the Adsense bot. Maybe they’re following the robots.txt for the regular googlebot and won’t use content gained from the media bot if the regular googlebot is blocked.
