robots.txt

  • Resolved djsteveb

    (@djsteveb)


    2 years, 11 months ago

    I am wondering if this plugin just blocks the bad bots from all access – and if so, would it be possible to add an option to allow the bots to GET robots.txt and block all other accesses?

    I know that many bad bots do not follows the directive, and saw that you point that out on the FAQ on the site – but I am hoping to further limit some of the main bots attempts, and free up server resources – so allowing them to get the robots.txt file may help with some.

    Which in my case, if I could just get a couple of the bots to read the robots file and follow it – then it would reduce thousands of requests per day –

    otherwise I may need to export the ips and do a lookup to get their host cidr and block them via the iptaples or whatever to reduce the cognitive load on php / sql / etc.

    Really appreciate this info, and the work put into this plugin!

    Steve

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Author Bill Minozzi

    (@sminozzi)

    Hi Steve

    You are welcome.

    Our plugin let all bots read robots.txt.
    As you said, bad bots do not follows the robots.txt’s directives.

    Please, leave us a 5 star review.

    Cheers,
    Bill

    Plugin Author Bill Minozzi

    (@sminozzi)

    Hi,

    I hope this is sorted, gonna go ahead and mark as resolved to help keep things organized. Feel free to post again with any further questions or feedback, etc. Thank you.

    Cheers,
    Bill

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘robots.txt’ is closed to new replies.