I have a Magento multi-site installation. I want to disallow just one of my stores, but not the others. I am trying to take advantage of the "Default Robots" option on the Magento config, so I left the default robots to "Index,Follow" for all stores except the one I want to disallow, which I changed to "Noindex,Nofollow". But when I go to my-URL/robots.txt, I get a 404 error.
Any idea why this might 404?
Can someone point my in the direction of how I might make a robots.txt that only disallows that one site?
This setting will only add a meta robots tag to your pages, e.g.
<meta name="robots" content="INDEX,FOLLOW" />
See: http://www.robotstxt.org/meta.html
The Magento application does not interface with your robots.txt