search-enginerobots.txtgooglebot

Google indexed my domain anyway?


I have a robots.txt like below but Google has still indexed my domain. Basically they've indexed mydomain.com but not mydomain.com/any_page

UserAgent: *
Disallow: /

I mean how can I go back further than / which I thought was the root of domain?

Note this domain is a work in progess, hence I don't want Google or any other search engines seeing it for a minute.


Solution

  • If you don't have one already, get a Google Webmaster Tools account. It includes a URL removal tool that may work for you.

    This doesn't address the problem of search engines possibly ignoring or misinterpreting your robots.txt file, of course.

    If you REALLY want your site to be off the air until it's launched, your best bet is to actually take it off the air. Make the site inaccessible except by password. If you put HTTP Basic authentication on your documentroot, then no search engine will be able to index anything, but you'll have full access with a password.