web-crawlerrobots.txtsearch-engine-bots

Allow and disallow in robots.txt file

I want to disallow all files and folders on my site from SE bots, except a special folder and files in it. Can I use these lines at robots.txt file?

User-agent: *
Disallow: /
Allow: /thatfolder

Is it right?

Solution 
  • User-agent:* 
Allow: /thatfolder
Disallow: /

    This should do. (just changed the order)