Is noindex an optional directive in a robots.txt file, or are user-agent, disallow, allow and crawl-delay the only options?
For example, is this valid for the contents of a robots.txt file?
user-agent: *
disallow: /
noindex: /
noindex is not a valid directive for a robots.txt file. It is a valid directive for a META robots tag, though.
The only standard directives for robots.txt are "User-agent" and "Disallow". Some browsers support an extended set of directives including "Crawl-delay", "Allow" and "Sitemap". http://rield.com/cheat-sheets/robots-exclusion-standard-protocol seems to have a thorough explanation of the standard and extended directives.