Will the plugin block bots trying to access every link from robots.txt?

  • Resolved yrzhn

    (@yrzhn)


    8 months, 3 weeks ago

    Hello,

    I just want to clarify a few things before installing and activating the plugin.

    There are some .PDF reports on our sites which Google indexed and we added a few Disallow lines with them in robots.txt. Visitors can still get to that file by hitting a download button on one of the pages. I want to make sure that no one gets banned for trying to download the reports, or is this not possible?

    Will the plugin overwrite my current robots.txt? Does it add the link in the footer and the rule in robots.txt automatically?

    I guess these things are considered and only the invisible black hole link will move bots to the black list but I want to make sure that there are no other consequences. ??

    Thank you ??

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Author Jeff Starr

    (@specialk)

    Hi @yrzhn,

    Glad to help:

    1. “I want to make sure that no one gets banned for trying to download the reports, or is this not possible?”

    The plugin only bans bots that disobey your robots.txt rules AND follow the hidden blackhole link. Bots that do that are disobeying the rules and so get banned. The plugin doesn’t even look at any other links or URLs, it monitors only the hidden link, which again is prohibited via robots.txt and other techniques.

    2. “Will the plugin overwrite my current robots.txt? Does it add the link in the footer and the rule in robots.txt automatically?”

    No it appends a couple of rules automatically IF the site is using WordPress’ auto-generated robots.txt file. Otherwise the rules need added manually. Nothing ever is overwritten. This all is explained in the plugin docs, settings page, help tab, and elsewhere just fyi.

    I hope this helps, let me know if I can provide any further information.

    Thread Starter yrzhn

    (@yrzhn)

    Thank you for the surprisingly quick and very clear response, Jeff. ??

    Stay well!

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Will the plugin block bots trying to access every link from robots.txt?’ is closed to new replies.