robots.txt

Disallow all for all user agents except one user agent?

How to disallow all for all user agents except one user agent? For example disallow all for all user agent, but allow for Googlebot only?

Solution 
  • User-agent: *
Disallow: /

User-agent: google
Allow: /

    This sample robots.txt tells crawlers that if they are not with google. then it is preferred they don't crawl your site. While google has been given the greenpass to crawl anything on the site.

    This file should be stored at www.example.com/robots.txt.

    Please read up on robots.txt