robot txt

Glad to help:

1. “How do I know the robt.txt file is active”

When you visit https://example.com/robots.txt, what happens? (Replace example.com with your actual domain)

2. “adding it to the list doesn’t help”

Where exactly are you adding the bot? To the robots.txt file, or to the plugin somehow?

3. “I thought this plugin stopped all bots”

As explained in the documentation and plugin settings page (via the Help tab), this plugin blocks any bot that follows the hidden link and falls into the trap. Nowhere does the plugin claim to block “all bots”.

4. “But I cant find the file on the server. Maybe you can indicate where I should look for it or create aa text file to upload.”

Again it depends, when you visit https://example.com/robots.txt, what happens? (Replace example.com with your actual domain)

Hi Jeff,

I checked the robot file , it is not there, I created one with the given lines in the settings, I upload that to the main directory of my website?

There is a bot list in the plugin, I add an IP address there but it still is able to put items in the cart.

I understand that it doesn’t block all bots. Apparently not, as I have one bugging me and just put item in the cart and go on to the next item making a mess.

Henri

Hi Henri,

Not sure what all you’ve got going on, but here are some points that may be useful for you:

WordPress generates a “virtual” robots.txt file when a physical robots.txt file does not exist in the root directory of your site. This is why I recommended checking the URL given in my previous reply.

Also, it sounds like you are dealing with a bot that is targeting specific aspects of your site. So waiting around for it to fall into the blackhole trap may not be the best strategy. I would recommend looking at captcha-type plugins that can stop pesky bots from submitting forms and so forth.

Those things in mind, to respond to your comments/questions:

One: “I checked the robot file , it is not there, I created one with the given lines in the settings, I upload that to the main directory of my website?”

Did you check for the file on the server itself, or by visiting the URL described in my previous reply? As explained, if there is no physical file, WP will generate a dynamic/virtual robots.txt. If the file exists, then “yes” you need to add the prescribed robots rules. OR if the file does not exist on the server, then the plugin will automatically add the correct robots rules for you. Note that all of this is explained in the plugin documentation, Help tab on settings page, and elsewhere.

Two: “There is a bot list in the plugin, I add an IP address there but it still is able to put items in the cart.”

How exactly are you accomplishing this? I ask because with the free version there is no option for manually adding bots. So I am very curious as to how you’re doing this.

Three: “Apparently not, as I have one bugging me and just put item in the cart and go on to the next item making a mess.”

Yeah as explained this is a honeypot-style plugin that blocks bad bots that disobey your site’s robots rules and fall into the hidden trap. What you are dealing with specific bot attacks or whatever would be better handled by a more targeted/direct approach.

I hope this helps, let me know if I can provide any further information.

HI,

I do not use the free version but the paid for version, I am pleased with the plugin.

I am not an internet wizard I get by, I do not understand anything about the robot text file you are explaining to me, but I checked the url you gave me with of course my domain. No file there. But you say it is automatically added then I take your word for that.

I am able to add IP addresses, but it doesn’t seem to make a difference.

Since your answer the pestering has stopped and several bots fell in the black hole, so the plugin functions well.

Thank you for your help.

Henri

Glad to hear the plugin is working well for you.

For this:

“I checked the url you gave me with of course my domain. No file there.”

There definitely should be a robots.txt file there, unless maybe WP is installed in a subdirectory..? In which case the URL to check would include the subdirectory. Or, another reason could be that some other plugin or function is disabling robots.txt..?

Bottom line is that there needs to be a robots.txt file that includes the Blackhole rules, in order to prevent good bots from falling into the trap. I highly recommend sorting out the robots.txt file before using this plugin. It is a vital part of how it functions.

Hi,

You are right the file is in the wp directory, I was looking a directory up. There is nothing visible in it, do I add the lines?

Henri

thanks, I added the lines to the robot.txt file.

Henri

The robots.txt file needs located in the site’s root public directory, like in the example URL that I included in my previous reply. Like https://example.com/robots.txt Otherwise bots will not find it. You can ignore the robots file located in the subdirectory, or redirect or whatever your strategy is fine; bots will neither find nor use it.

For now though, make absolute sure that there is a robots.txt file in the root directory, and then follow the installation steps for Blackhole, including adding the recommended lines to your robots file. The lines can be found in the plugin settings, right near the top of the page.

all good, file is there and I added the lines.

thank you for your help.

Henri

You are welcome. As long as the site is not using any sort of page-caching plugin, you should be good to go. Cheers, Henri.