Worwordfence generating huge amount of 503 errors

WF support will likely chime in, but in the meantime — have you tried putting the Firewall in Learning mode for a week so that it learns about the “normal” traffic pattern for your site?

You can also locate the google access attempts and whitelist them.

With 403 (Permission denied) errors, it could be WF blocking access, and hence Learning mode or white-listing would a good idea.

However, with 503 (Server overloaded, don’t wanna respond to the requests) we are probably unfortunately into an area that need more close view on server/site configurations and some debugging. Probably a quick thing to fix on a system one can see and touch, but without seeing it, I am merely rattling off some of the many potential problems.

Some (shared) hosting setups limit the number of Apache or PHP worker processes you can use at a time. To protect the shared resources on that server. In some cases down to 3 worker-processes only, which means that there is a severe limit on how many requests can be handled in parallel.

Generally, you hit the 503 error either on real overloads, or they can happen if for example a bug in a plugin cause a never-ending loop. In such a case it never relinquishes it’s PHP worker process for the next web-call to be serviced. You eventually loose them all, as they might be killed off only after hitting the CPU limits set in your Apache or PHP configs.

BUT.. In this case, with a hosting account holding 2100 + 9000 pages/links that Google and all the other bots want to pull on and check out, all it takes to see 503 errors is a general slowness in handling requests (like when you re-enable WordFence with a large number of accumulated blocked IPs and accesses to check against).

The fact that it works with WordFence disabled could either mean that WordFence has a real “never-ending-loop” type bug, which I at this point consider quite unlikely.
Or it could mean that you simply need to reconfigure your hosting setup to actually match the load of a larger/older site-need. Or, see further down below.

You don’t mention your hosting setup. But if on a VPS or dedicated server, you would go check how many processes are actually being used, and reconfigure your setup.
With PHP-FPM/FastCGI you would go up the number of PHP worker processes allowed. Same for Apache.. If your site STILL gets exhausted and issue 503s, then maybe you have a looping, never-ending plugin. In which case, upping the # of allowed worker-processes would help little or nothing. It would just eat up the additional workers as well.

On a shared hosting setup, you usually cannot just go add more workers, as they cannot allow you such access.

Another distinct problem could be, if you have accumulated a large number of IP blocks over time. Especially if you block many IPs one at a time, rather than blocking off whole ranges. Making WordFence slow down over time. Having to work harder and harder to check incoming accesses (in slow PHP and WordPress).
In that case, you would need to reduce that load. Either just by consolidating all those IP blocks removing overlaps, or by “moving some blocking Left in the path”. Like by moving the majority of your current IP blocking from PHP (WordPress/WordFence) into Apache (block with “Deny” in htaccess), or even further left in the access path into a real firewall. That will remove a lot of load. Spare PHP and WordPress from even getting started on most IP blocks, by allowing Apache to quickly block them early on.
(There is a way to easily transform your accumulated WordFence IP Blocks into Apache or firewall type blocks, should you wanna try that route).

Another potential problem that could significantly slow your system response, and hence cause 503 overload errors, is if your dead/expired transients are getting overblown. Without Object Caching, transients (temporary data) are sitting in and clogging up your options table. That can make it grow exponentially. Install a plugin like WP-Optimize. It will show you how many expired transients your site is carrying around, and allow you to clean them out.

I could continue the list of potentials that need to be checked out to get your setup down under it’s overload situation. Better caching configuration come to mind as well.
But all of this is a kind worthless exercise for a system I cannot see or touch. When one is unable to actually check out how it is behaving in the dance that is serving web-requests. Something is obviously stepping on your system’s toes and hitting its bad bunions.

I really thank you both for the answers. My hosting- Godaddy- did troubleshoot the sites and did analyze everything with the servers one day ago. More than 2 hours were used to do this until a senior team told that the issue wasn’t caused on their end. I can’t use many cache plugins because Godaddy already has a cache of its own and it is not compatible with several other plugins. My plan is managed wordpress so, there are a lot of things that I can’t do myself. After deactivating Wordfence I managed to resend my sitemaps to Google ( that were pendant, some of them since May) and they were all reached without problem. About the blocking process, from time to time I delete blocked ranges or individual IPs to make the plugin lighter. I also have installed plugins to minify processes and compress, browser leverage, etc. I thought that the problem would be with one of them but it wasn’t. so, for now, I keep Wordfence deactivated until I find some way of making it work properly. I have wordfence for years, and it always did work well with my system configuration and hosting. So, the changes on the plugin in last months are probably the answer to this issue I don’t know. Thanks

When Godaddy says “Not our problem”, it simply means that their system is doing exactly what they expect it to do. That they do not see anything that they want to fix.

503 error is how they limit your resources on ALL their hosting accounts.
Google GoDaddy 503, and you will see endless hits on people seeing that issue. That is not an unusual way to limit usage for any shared hosting.

They also state it directly in their help-pages. See such as

https://www.godaddy.com/help/website-errors-503-service-temporarily-unavailable-5089

Personally, a number of years back I for years had a Dedicated Server fronted by an also private Cisco Firewall with them. This worked fine, because on a dedicated server I control the limitations, they impose none. It is “my” hardware.

My only shared account with GoDaddy could never be used to run any real kind of site at all. It only held a 1-2 page service site.

On sitemaps.

Submitting sitemaps to Google is merely something one has to do to tell them about pages that are harder or slower to find.. Gets them started. But GoogleBot WILL over time automatically find any and all other pages that are naturally linked up on your site. Most sites do not even need a sitemap. Google just follow all the links it finds across your pages, and WILL eventually find all your pages.

Your pages do not disappear from Google search because Google cannot use your sitemap. That is completely irrelevant to what pages go into search.
Your pages disappear from Google because THE INDIVIDUAL POSTS and PAGES are returning 503 errors. Google will not direct their search customers to failing page content. But they wouldn’t dream of sending a searcher to your sitemap file.

That is also why your Google Search Dashboard shows all the individual pages as failing with 503s. Why you say thousands of 503s.
That loading your sitemap from your server fails with the same “I am overloaded” error is just an irrelevant fact, but caused by the same reason. That GoDaddy is heavily limiting your resource usage on ALL shared accounts.

For all but Dedicated Servers (where you are in control of the entire piece of hardware) GoDaddy has a long history of severely overloading servers with domains and sites (more accounts per server, more profit) and hence having to limit each account’s resource usage to an extreme degree. It is what they do..

Check on the Internet how many domains are assigned to your IP address (meaning that the same server/cluster is servicing both your domain and those others).. I once did for fun.. My shared IP with Godaddy had more than 4400 other websites of varying sizes on it. And each server can have multiple/many IP addresses assigned to it, so multiply up from there. ?? All those sites then in addition on the back-end share database servers, which means that they only see a minuscule portion of the database server’s query caches, if any at all. Something that can only be helped with front-end Object Caching.

That is how shared hosting is done. Nothing new there. It depends on the hosting provider how many sites they decide to load on each server.
But independent of provider, this is how they make a profit on selling $3-$6 hosting accounts. By putting many thousands of domains/web-sites on the same server-structure, and not allowing any individual site to steal too much resources from the other thousands of sites sharing the fun. ??

And if you really have a combined 11,000 pages/links, your account is definitely too large for small shared hosting.
Think about it. Just the amount of hits you get from the Google/Yandex/Baidu/Naver/… search engines (and all the unwelcome bots) wanting to check and recheck all your pages repeatedly can be a serious killer.
If you then add WordFence checking each new connection against “bad stuff” (and hence slowing down) all those hits so they hang on longer fractions of a second, the problem multiplies up heavily.

But unless the WordFence folks know of a specific problem, where WordFence makes connections hang around for much too long, your problem seems to be a generic overload one.

Hi
I am very thankful for all the insights. I am not a tech person but am a thinker so, your answer really helps to get a whole picture. Meanwhile i ?did sleep? on the issue, as they say, and accidentally may have found a probable cause to my issues. Yesterday, when I deactivated temporarily Wordfence, I picked some Godaddy Ips that since months (precisely about since May) have been hitting my sites everyday with thousands of hits and were blocked with wordfence, to keep them blocked using another security plugin. and guess what, 2 of them- one in every site -are now white listed and can’t be blocked. They were not when I blocked them with wordfence. when they began hitting the sites I sent to the hosting malware team the info and the now white-listed IPs are the ones that have been hitting the most both sites. So, probably, the fact that they are now white listed, can be a good reason for generating all the 503 errors because they are set on my hosting as well.
So, today, I deleted wordfence and all data and reinstalled a new fresh one to see what happens. I am on Managed wordpress but not on a basic plan. And besides all you say about your experience with Godaddy, and the fact that you have your good reasons ( I myself also ask why the malware team did not reach to me to say that, of the IPs I did report that were attacking my sites 2 of them were monitoring or something like that, and that could be the reason of all these issues) I also am grateful to Godaddy because before this hosting I could not have my first blog and after that the new one anywhere else without being put down on the web. They have provided security and also help, when i need. And i am sure that the issues that happen at Godaddy also happen in other hosting companies. And also there is the fact that several of them do not allow wordfence. And i need my freedom to chose the way how I secure my sites. so, i will give feedback if the issue returns. thanks

Hi Fatimajesus, I’m a successful pro blogger and can share a couple of observations. I spent about a year struggling with shared hosting, hundreds of hours, never really “getting” that by simply paying money for a VPS I could solve nearly all my problems with site errors, shutdowns, etc. As Caleb alludes to, if you’ve got a site with a volume of content, first step is to be totally sure your server can handle peak loads. Beyond that, again as Caleb says, moving things “left” is of huge benefit. At the least, be sure you are fluid and comfortable with SFTP or other ways to modify your site configuration files, and get to know the .htaccess like cooking your favorite meal. And if you’ve got a VPS, get ready to have fun moving even farther “left.”

Ditto on the site maps, generally a waste of time for reasonably designed and maintained websites.

Watch out for caching. Tends to over-promise and under-deliver, you can end up with quite a few caching systems going at once, thus confusing issues and introducing redundancy that at the least does nothing for you, at worst causes errors.

Lastly, once you move to Wordfence Premium, be sure to try out the country blocking options. Using it wisely can significantly reduce your server load.

MTN

Thanks I appreciate all the insights.

Hi
this is a final feedback to close this issue. After 2 days of having reinstalled wordfence but without blocking the IPs now white-listed the 503 errors did began to reduce what shows, undoubtedly that Wordfence itself was not the cause of this issue. But also noticed another thing: there are cases when other security plugins (like Ithemes) do not block some IPs because they are white-listed and Wordfence keeps blocking them. I mean, I made a test and tried to block a white listed IP on Ithemes and it did not block it because of that. After that, did the same with Wordfence and it blocked it although some other times I’ve noticed that many white-listed IPs are not blocked by Wordfence. This perhaps could be more attuned to prevent issues like my own in last days, I don’t know, but leave the suggestion to the authors’ team. Thanks

Hi @fatimajesus
Could you please provide more details about how did you block these IPs in first place (via Advanced Blocking)? and where exactly you found them whitelisted after that?
For sure trying to block a whitelisted IP in Wordfence will show up this message:

The IP address [IP] is whitelisted and can’t be blocked. You can remove this IP from the whitelist on the Wordfence options page.

Thanks.

Hi
Last May I found on my logs that certain IP addresses were attacking my sites (different IPs to each one of the sites) with thousands of hits (6000 or more). It happened that they were all set on my hosting service as well. So, I went to Wordfence and blocked them one by one except for about ten that were of the same range so, these last ones were blocked using advanced blocking. Right away i contacted the hosting malware team, sent them screenshots with the activity of the Ips in question and that’s it.
Some days ago, when I found that disabling Wordfence stopped the issues on the sites, I used the plugin Ithemes to manually block those Ips because every single day they continued to come to the sites sending thousands of hits. and I founds, astonished, that the ones that were more active were also now white-listed. It was the plugin Ithemes that gave me the message and said that they couldn’t be blocked.
but there’s more. I sent an E-mail to the malware team complaining about the fact that they white-listed those 2 IPs after I did block them with wordfence and they didn’t tell me anything and that did cause many issues on the sites. Some hours later, those IPs were no longer white listed (tested this with Wordfence and Ithemes) and also, another one of the reported Ips that was not white-listed, became white-listed some hours after that and I know it because Ithemes told me that it couldn’t block it because it is white listed (but did block it one day before). Wordfence kept blocking it, though. It is true that Wordfence gives that message and does not block certain IPs,when they are white listed , I found it many times, but apparently this is not applied to all cases of white listed IPs.

Thanks