seobigcommercerobots.txt

Can a robots.txt disallow use an asterisk for product id wildcard?


Is the following valid in my robots.txt file?

Disallow: /*?action=addwishlist&product_id=*

rather than writing individually for every product like below:

Disallow: /*?action=addwishlist&product_id=1212 

Or do I need to define below line for each individual products?

Disallow: /*?action=addwishlist&product_id=1212
Disallow: /*?action=addwishlist&product_id=1213
Disallow: /*?action=addwishlist&product_id=1214

or * sign will be taken as any product id. product_id=*


Solution

  • Googlebot recognizes the asterisk in place of the product ID as tested with their robots.txt test tool:

    You could also leave the asterisk off, because robots.txt rules are all "starts with" rules. An asterisk at the end of the rule is equivalent to the same rule without the asterisk.

    enter image description here

    Keep in mind that while the major search engines support wildcards in Disallow rules, other bots won't. Wildcards in disallow were not part of the original robots.txt specification from robotstxt.org. Less sophisticated crawlers that don't understand these wildcards will still crawl these URLs.