urlrobots.txt

Disallow common sub-page of several directories while allowing the directories themselves

I have the following structure

http://example.com/universities/UNIVERSITY_ID/review

I am trying to disallow the review page, but keep the page of UNIVERSITY_ID which is dynamic.

how can I accomplish this in the robots.txt?

would this work?

Disallow: /universities/*/review
Solution

  • Using wildcard is better solution to tackle this issue.

    User-agent: *
disallow: /*review$

    reference: https://webmasters.stackexchange.com/questions/72722/can-we-use-regex-in-robots-txt-file-to-block-urls