Broken link checker not picking up updated links

Apologies. I had forgotten that I had replaced the action
link_library_reciprocal_check
with a modified version to prevent the link checking always timing out.
So, it is my code that is accessing the old database table of links, not yours.

When I remove my customised version of this code, I’m back to the link checking always timing out, so I’ll need to examine your new code to see if I can re-instate my timeout workaround.

But this query can be closed as a red herring.

Ah, that would definitely explain it! I was really scratching my head on that one when I saw your initial post. Since data is stored as custom post types instead of links in the database, getting data is different.

If you have implemented a good approach to avoid timeouts, would you be willing to share it with me? I am sure that others could benefit from a solution for this.

Hi! Sorry for delay in response. I’ve been away from the keyboard for a couple of days.

I have indeed implemented a workaround to avoid timeouts but it involves running a program outwith WordPress so is not a ‘proper’ solution.

I have replaced the action/function ‘link_library_reciprocal_link_checker’ with an action/function of my own which is largely the same as yours except when the check_type is ‘broken’ instead of calling the function CheckReciprocalLink, I call a function of my own.

Instead of actually checking the url, this function opens a scratch file and writes details of the link (name and URL) to this file.

Then, at the end of the link_library_reciprocal_link_checker function, after all the link details have been written to the scratch file, I output a hyperlink that links to en external program that reads the links from the scratch file and checks them.

I tried to get this program to work as a function within WordPress but, although I managed to get it to work on my test site (which has just 65 links) I failed to get it to work on my live site, which has 138 links. I tried checking the links in batches but that didn’t work either. I managed to get two batches to run, checking links 1-20 and then links 21-40 but when I tried to add in a third batch, checking links 41-60, the processed failed again (I get a ‘Server Error’).

I think that if you could add a parameter to your broken link checking process, so that the user specified how many links to check at one go, then that might work. First click would check and report on the first batch of links and then display a ‘Continue’ button. Clicking on ‘Continue’ would then check the next batch of links and so on.

If you decide to try developing something like that, I’d be happy to help test it out for you.

I just re-wrote the link checker using ajax. This way, after you request for it to check reciprocal links or broken links, it goes to a page where it will fetch results via AJAX one link at a time, then update the page. This way, there can never be too many links to process at once.

Please try this out and let me know if it works for you. The action function you registered will not work as is, since the name of the action filter changed.

Just tried it out on my test site and the link checking goes ok.

BUT, despite my having not checked the ‘delete links that return a 403 error’, these links were deleted. And they didn’t even go into Trash – so I was unable to recover them for correction.

Can’t risk this on my live site yet.

Sorry about that. Will investigate and correct this evening.

This has been corrected in latest update. It will now properly consider whether or not the checkbox is checked.

Great! That works for me. Successfully checks all the links in my live site and does not delete those that return error code 403. Thanks.

I note that the ‘delete 403 links’ option only applies to reciprocal links. I don’t have any reciprocal inks so am not able to test whether clicking the option works.

As far as I’m concerned, there’s no need to add this option to the broken links checking. I want the deletion of links to remain firmly under my control.

Having reviewed the output from running the broken link checker, I find that it is reporting error 403 for some valid links. When I put them through my own link checker, I get a timeout error for all but one of these*. Are you able to detect a timeout error and report that instead of 403?

(I use curl to do the checking, with a timeout of 4 secs and get the following error message
Curl error: Operation timed out after 4001 milliseconds with 0 bytes received)

* The one link that doesn’t timeout for me is kmorlich.weebly.com
When put through your checker, the return is error 403 but my checker reports it as a valid link. Have you any clue as to why this is happening?

I will investigate. I am using the WordPress wp_remote_get function to check whether or not links can be accessed. I think that curl is not supported on all PHP installations by default, so I wanted to avoid usign a function that people would have trouble with.

Now, the function has a 5 second timeout by default.

First thing I can try doing is to increase that timeout to 10 seconds. With the way the checker updates via AJAX, that would work well.

For the HTTPS link that is coming back with a 403, I will test. Is that your only https link in your library?

Yannick

That might work for the links that are timing out. I tried increasing the CURLOPT_TIMEOUT to 10 secs in my (CURL based) link checker and that did the trick for most of my timeouts.

Re https – there are lots of https links in my library. My link checker gives a warning when http is used for an https link, so I’ve been updating them to the correct format.

I have updated the timeout time to 10 seconds in latest version, 6.0.42, available later this evening. That update corrected the timeout with www.aagm.co.uk

Now, that FreakCircus link actually comes out as a valid link for me. So not sure why it would timeout on your side…

I am away from home now till Tuesday next week. Will check this out and report back on my return.

Have now updated to v 6.1.1 and rerun the broken link checker. You seem to have fixed the timeout problem as all links that were previously failing due to timeouts are now OK.

But there is still one valid link which your checker reports with error 403 but passes my checks OK. This is kmorlich.weebly.com

That https://kmorlich.weebly.com/ link does not give me any issues on my web site. The only thing I can think of is that this site is secure (https), is yours also https? I have seen some instances where one https site does not like talking to a http site. I think you might just have to accept that it comes up as 403.