3000 High CPU Brute Force Attack Single Blocked IP 503

follow up to my previous post…
I got nailed with another single ip bot attack which didn’t stop at 4000+ attempts at my wp-login.php.

My server load went sky high.
I ended up using an IP whitelist in my htaccess file which gives a super quick error 403 to the bot.

My server load dropped back down dramatically.
I am very concerned that although Wordfence will stop the HACK from being successful, the server load was too much.

In my massive researching and testing, it seemsServer Overload is not talked about much compared to password protection and preventing a successful brute force attack.

This has become my utmost security concern with WordPress the past year, so much so I am considering moving to another CMS even though my business has been built around WP development for over 10 years.

I am rarely concerned with someone actually hacking my sites as WordFence does do a really great job but I can’t help but feel like the “bigger” problem is being ignored or simply swept under the rug.

The hack attempts never stop and pretty much kill anyone on a shared hosting plan and even worse cause my AWS services to exponentially grow in cost due to this excessive unwanted and illegitimate traffic. Is it a conspiracy, hmmm.

It really seems we as an internet community (I’m speaking to hosts, developers, security advisors, etc) are not leveraging the knowledge of the community on this one. If we’re all being hacked by the same IP’s upwards of several hundreds of thousands of compromised systems, why are we not already automatically blocking those IPs before they hit us? At the server level?

Why are those IP’s not on a hosting providers first line of defense very much like spamhouse is to email?

If my system was compromised it should be my responsibility to clean my sh*t and jump through many pain in the a** hoops to get removed from those lists.

Why is hacking so risk free. It seems like we still haven’t devised the right technologies to make attempting to hack painful and harsh with consequences.

Can’t we come up with a way to ruin their day?

Heck I’d even set up a bunch of dummy WP sites specifically for dishing it out if anyone has any ideas.

How can we fight back and take our internet back?

@pingram3541: I was at the Suits & Spooks conference in Washington DC earlier this year and this subject came up. Specifically, the legality of “hacking back”. I know you’re not directly suggesting that, but just wanted to make you aware that it’s discussed in the infosec community, which is indicative of the level of frustration out there.

Your pain is definitely shared.

Can you give us an indication of how much of your resources (bandwidth, anything else you’re billed for) are used by attacks. It might be difficult to differentiate between attacks and real traffic, but if you can, share what you have available.

Regards,

Mark Maunder – Wordfence Founder/CEO.

@themadproducer: Sorry I replied to the other poster first. Should have replied to you since this is your thread.

Thanks for bringing this to our attention. We have a feature in the queue which will be in one of the next three releases that will solve this issue – the problem of high load during intense brute-force attacks.

Having said that I should point out that our current code does a pretty good job of minimizing load when an IP has been blocked. If you can share any data here showing load etc, it would be much appreciated.

Regards,

Mark.

@WordFence Thanks for the feedback Mark. As you already guessed yes its a bit difficult at the moment to calculate the actual cost. I might look into a way to get tangible numbers and I’ll definitely keep you informed if I figure out a way to map it concisely. Luckily for myself I’ve only seen my own costs jump by a few dollars a month. I can say the majority (I would guess around 90%) of the traffic to my own websites are unscrupulous visits which is concerning. I’m just lucky my hosting plans only barely go over my limits for now.

I can also say I have a lot of clients with shared hosting that used to perform moderately based on their very low internet profiles which now perform terribly and the hosts report back that the traffic is root cause and of course use this is an opportunity to try to up-sell. Some have been throttled and even suspended due to exceeding their quotas! Similar to my sites, when I look at the traffic logs I am hard pressed to find visits that are for legitimate reasons. For me, it is more costly in my time in that it takes that much longer for me to get simple tasks and development chores completed due to such slow page loads, let alone the constant interruptions in my work flow having to stop everything and deal with a security problem. In some cases I could see where it would be faster for me to clone a site, work off my local environment and then push it back up.

The 2 main areas I see growing perpetually out of control on the free web is email and website security. Attackers, spammers and phishers are simply unrestricted to continually keep banging on the door while the flip side is the current security measures in place are mainly a nuisance for legitimate folks. In the real world, someone opens the door and gives you a black eye…the internet is too passive in my opinion.

@pingram3541
@mark

Thanks for adding input to this thread. It feels like an AA meeting for website security lol.

Here are some trends and info I’ve noticed in the past few weeks of testing:
– every single day, my one server with 4 WP sites, gets a dozen different BFA’s (brute force attacks) and only once so far has been a Botnet Attack of a low 150hits.
– since using Wordfence to track attempts and stop them…almost every attack quit within 2-50 attempts with a single IP attack, but, sometimes it ran as high as 150-200 attempts before the bot gave up. (stupid bots)
– 2 days ago I endured 2 BFA,s with 3,500 and 15,000 stubborn hits each. (especially stubborn relentless stupid bots)
– during these attacks, Wordfence and WP Statistics were tracking every hit as well as my raw access logs of course. My shared server package states that 2% cpu is what I am allowed. Sure, I will get spikes and occasional sustained high cpu, especially when I am doing a ton of site maintenance in the temporary cpu range of 3-6%, but during that last high volume attack, Cpu was hovering between 18-20% for a long period of time.
– In an attempt to relieve the pressure and weed out all possibilities, I disabled WP Heartbeat, disabled WP Stats and eventually, whitelisted my IP for wp-login access via htaccess. AS soon as I did the htaccess part, CPU load averages slowly started to fall back down. At that point, WF stopped tracking the hits because of course, they were stopped at the server level, so I knew htaccess was working.
– unfortunately, I have no scientific way of accurately measuring real time CPU usage on the server except the cpu% average that shows up in cPanel and gets updated every 5min.

I am interested to test out any new WF feature that might allow me to use WF as the first line of defense rather than the htaccess whitelist. I really appreciate the WF stats report and it saves so much time when reviewing and analyzing activity.

@pingram3541
Yes, how can a single IP be allowed to knock on my door 15,000 times in 13 hours without any resistance or repercussion. And to permanently blacklist IPs is not a well rounded solution since the IP may be dynamic, or part of an infected innocent user’s PC or server.

@mark…just thinking of ideas here…
I know WF was not directly responsible for the super high cpu% during the BFA’s, but if it contributed due to the processing of hack attempts (which I haven’t proved but it seemed this way) …then the feature I would like to see is maybe a combination of WF and htaccess. Perhaps, if a certain amount of hits to sensitive URL’s is tracked from a single IP origin, then WF triggers the IP to be blacklisted in htaccess for 30days etc. This would be the best of both worlds assuming.

So for example…3 failed login attempts triggers the banned URL thus the offending IP gets served the usual 503 for x-amount of days. But then, if another consecutive 50 attacks is tracked from this same banned IP, then implement an IP block via htaccess. Now the processing overhead for this IP has been reduced to a minimum. So it’s a 2 stage defense depending on the severity or longevity of the attack.

What about a Botnet attack of say 500 consecutive unique IPs in a short time frame? Perhaps this triggers a temporary lockdown…BLOCK ALL IPs from wp-login with a instant notification to the webmaster who can then monitor the attack and make a decision as to how to proceed.

Just throwing you ideas Mark…and I am definitely not a security expert but a am good at troubleshooting and idea invention.

I’m loving it. This is exactly the spark I was hoping to see. @themadproducer great idea, this would definitely minimize the amount of manual work experienced users currently have to do but bring it to the table for everybody. I can however see how the folks at wordfence might have avoided this because we also have to account for the different server set ups which could increase the odds of accidental lockouts and even crippling a site entirely. Also my sites are nginx so I’d hope we’d accommodate the nginx.conf counterparts and a way to make sure users that are nginx are aware a reboot of the service may be required.

Another light bulb idea!!!
Continuing with the Botnet…
After WF has determined that a botnet is occurring…
– all ip’s are blocked from accessing wp-login.php via htaccess
– email gets sent to the administrator
– in the email, there is a LINK which allows the admin to enter an IP address of his current location so that WF will immediately white list it in htaccess thereby allowing him to login to WP immediately no matter what IP or location he is at.
Obviously, as a fail safe, one could always log into FTP and edit the htaccess file manually.

Absolutely a great idea. Take it a step further, the email provides a private one time generated link to the page which captures the “referrer source ip” via HTTP upon visit which writes the necessary rule(s) to allow that ip automatically. Additionally the page also should provide a simple form to supply an override ip or additional ip’s you’d like to allow access (I.e. a button to add additional ip input fields one at a time as needed) along with a convenient link to wp-login.php to jump right in.

Thanks for helping design Wordfence guys!

That was a lot. Many good ideas here. I’m going to try and distill them down into the very best of the best:

Goal: We need to not just block the bad guys. We need to reduce their ability to consume future server resources down to something that is completely negligible.

Secondary goal: If providing this kind of load-free blocking impacts admin access to site, ensure that this problem is solved elegantly.

I’m going to leave it there – I know you guys threw out a few other ideas, but lets go with that for now because that feels like the core of it.

What I’m going to do is enter a bug in our bug/feature tracker and get the product team around this later this coming week (with a link to this chat). We actually already have something in our system which is scheduled for 3 releases from now. But it treats the symptom, not the root issue, and it also might be implemented more elegantly.

Thanks. What a great way to spend a Saturday afternoon – having your customers design your product.

Regards,

Mark.

Incidentally, the issue I updated on our side (internally) is numbered 567 in case you’d like to reference that in further discussions with Tim, Colette or Matt R who all contribute on these forums and are part of our team.

Regards,

Mark.

Thanks Mark. Can’t wait to see how this might materialize in the near future. Additionally I’m more than willing to help in any way I can . Cheers!

Yes thank you from all of us Matt.
Your response was a welcomed one and it pretty much made my wknd.
Also thanks for your input pingram3541.

I believe that a strategic improvement, even if noticed or discussed by only a few of us here, could eventually develop into a real improvement in WF and benefit the many. Also, WF deserves recognition, and perhaps if some of these ideas or similar are developed, then WF will unquestionably rise to the top of the heap for WP security solutions and perhaps set new standards.
Cheers 567 ??

“You should change the notification frequency of WF.”

And exactly where is this setting?

How do you stop the constant brute force attacks or is that not possible?