phpurlrobots.txt

How to block this kind of URLs in robot.txt file?

here are two set of conditions and I want to block all url with /search/jobdescription? and search/jobdescription/

  1. http://<--sitename-->/search/jobdescription?id=0154613&css=a&act=a

  2. http://<--sitename-->/search/jobdescription/id/0154613/css/a/act/a

How to block both occurances in robot.txt file.

Solution

  • As simple as:

    User-agent: *
Disallow: /search/jobdescription