Specific Page Protection and Facebot… it ignores robots.txt

  • Resolved fightsmarttrav

    (@fightsmarttrav)


    4 years, 11 months ago

    Hey guys,

    Super cool idea!

    I believe that Facebot / Twitter bot ignores robots.txt rules. I might be wrong about that, but if we were to completely ‘ban’ this bot permanently, it would stop facebook from indexing open graph data on our entire site.

    So:

    1. Does blackhole ban the facebook twitter bot? I would like to selectively block it from accessing certain pages… but not ban the IP entirely. Not sure how your banning works, but page specific banning would be pretty cool.

    2. Can you enable the black hole on specific pages, that we know have a ‘do not index’ specified. This way, only bots that break the rules will be punished?

    I think that’s it for now!

    Thanks for making such a cool thing!

    The page I need help with: [log in to see the link]

Viewing 3 replies - 1 through 3 (of 3 total)
  • Plugin Author Jeff Starr

    (@specialk)

    Thanks for the feedback!

    1) No, the plugin whitelists all the big player user agents, Google, Twitter, FB, et al. You can find a list in the plugin docs and in the contextual Help tab right on the plugin settings page.

    I like the idea of page-specific banning, and will look at adding in a future update of probably the pro version.

    2) Also not a current feature, but will try to incorporate the idea along with anything added for page-specific Blackhole.

    Also for this:

    “I believe that Facebot / Twitter bot ignores robots.txt rules.”

    In addition to the robots.txt rule, Blackhole adds nofollow to the trigger link. So in order to be blocked, FB or whoever would need to disobey BOTH robots.txt and local nofollow attribute.

    Thanks again for the feedback, feel free to post again with any further ideas, questions, etc.

    • This reply was modified 4 years, 11 months ago by Jeff Starr. Reason: adds info
    Thread Starter fightsmarttrav

    (@fightsmarttrav)

    Hey thanks for the response and the great support!

    So I’m just curious, how does this thing work? Is it adding IP’s as banned to .htaccess?

    If so, how long is the ban period?

    If it’s permanent, I guess I might just be concerned that spammers behind VPN’s might end up getting ‘legitimate’ users behind VPNs banned as well… but hey… what kind of shade-ball uses a VPN anyway… am I right? ??

    Plugin Author Jeff Starr

    (@specialk)

    Glad to help:

    1) “how does this thing work? Is it adding IP’s as banned to .htaccess?”

    None of my plugins make use of .htaccess. It’s all WordPress core/API and vanilla PHP. You can find a more complete description on the plugin homepage/readme.txt.

    2) “how long is the ban period?”

    Until the bot is removed/deleted from the Bad Bot Log.

    Check out the Help tab on the Blackhole settings page, it provides a LOT more infos about how it works, what’s up, etc.

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Specific Page Protection and Facebot… it ignores robots.txt’ is closed to new replies.