robots.txtgooglebot

How to block fake Googlebots?


I guess a fake Googlebot visited my site. Here is the entry log:

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
66.249.73.72

I think like that because it crawled some addresses that do not exist! actually, they had been created by me at all

the fake bot has some stracture, it adds a spicefic word to first of my URLs for instance this page is exist

https://stackoverflow.com/user

but the bot crawled :

https://stackoverflow.com/some-word-user
https://stackoverflow.com/some-word-jobs

and here my robots.txt.

User-agent: *
Disallow: /search?q=*
Disallow: *?replytocom
Disallow: /*add-to-cart=*
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: -----

Solution

    1. First you should know, googlebot crawls not existing addresses too, i.e. on trying to discover new content.
    2. Second, i personally would better live with fake googlebots, as to risk to exclude googlebot per its IP. Google adds new IPs to googlebot. Again: don't risk it.