Forbidden 3rd party links

Hi @wmbweb,

The status codes shown by Broken Link Checker and its documentation could be found on;
https://wpmudev.com/docs/wpmu-dev-plugins/broken-link-checker/#broken-link-status-codes
https://wpmudev.com/docs/wpmu-dev-plugins/broken-link-checker/

Regarding the “Forbidden” message on your links to ‘Bookshop.org’, it seems like ‘Bookshop.org’ is blocking requests from your site. This may happen if they detect too many requests coming from a single IP address and deduce it as a potential threat.

I suggest to reach out to the Bookshop support team and ask them if they can whitelist your site’s IP and BLCs, allowing the tool to function properly.

Please make sure to allow BLC User Agent and IPs on your site as well.
https://wpmudev.com/docs/wpmu-dev-plugins/broken-link-checker/#broken-link-checker-user-agent
https://wpmudev.com/docs/wpmu-dev-plugins/broken-link-checker/#broken-link-checker-ip

Kind regards,
Zafer

Hi again @wmbweb,

We haven’t heard from you in a while, we will go ahead and mark this thread as resolved. If you have any additional questions or require further help, please let us know!

Kind regards,
Zafer

Hi. Not resolved.

Bookshop.org reached out to its dev and affiliate team to see if they can fix the Forbidden issue. They said they were not able to reproduce the issue, for example from this page on my site:
https://www.witchesmassbay.com/2023/05/12/book-short-takes/
They have been able to click the links and go to the Bookshop.org pages via different browsers. I have too.
See link below for screenshots from the link checker showing some of the Forbidden links.

https://www.witchesmassbay.com/delete/

Note from Bookshop.org: I talked with our dev team about this, to clarify, we tested the links from your website and to Bookshop.org and they are working – what is being blocked by Cloudflare is the bot that Broken Link Checker uses to check for broken links. We can change our cloudflare settings to allow the bot, and are devs are looking into it. If there is any info Broken Link Checker can provide us to help allow their traffic, it will probably speed up the process. Sorry this is so drawn out, but spam and malicious bots can be a real problem for us.

@wpmudevsupport15 Please see the two posts above. My issue hasn’t been resolved. I did get more feedback from Bookshop.org that may help. Thanks.

Hi @wmbweb,

Could you please ask the Bookshop.org support if it’s possible for them to whitelist our User Agent as mentioned in the documentation we have provided?

WPMU DEV Broken Link Checker Spider

https://wpmudev.com/docs/wpmu-dev-plugins/broken-link-checker/#broken-link-checker-user-agent

I hope that helps.

Kind Regards,
Nebu John

So Bookshop.org whitelisted your User Agent but I still have 80 bad links (plus, some links are on multiple pages).

Hi @wmbweb

I pinged our BLC Team to review this closer. We will post an update here as soon as more information is available.

Kind Regards,
Kris

• This reply was modified 9 months, 1 week ago by Kris - WPMU DEV Support.

Hi again @wmbweb

Based on our logs we still see 403 forbidden. Can you ask Bookshop.org support to check their logs? They should be able to see it are they still blocking us.

Kind Regards,
Kris

Kris & All,

On Feb. 12, Bookshop.org said: “Our dev team notified me that we have changed our rules to allow the “WPMU DEV Broken Link Checker Spider” to work on our site. So the link checker should now function and no longer get a “forbidden” alert. If you are still having issues let me know, fingers crossed the issue is fixed!”

Hi @wmbweb

When we cURL your Bookshop.org URL with “WPMU DEV Broken Link Checker Spider” rule we get HTTP/2 200. But we still see 403 forbidden errors. Can you ask them to confirm that our IPs are whitelisted:
https://wpmudev.com/docs/wpmu-dev-plugins/broken-link-checker/#broken-link-checker-ip

Kind Regards,
Kris