Server IP Blocked

I really do appreciate your willingness to help on this guys. We are trying this now and I will respond shortly. Again, thank you very much for being willing to reach out and respond to me on this.

Here is the response on this:

Step 1
root@host:[~]: curl https://ifconfig.io/
69.16.221.200

Step 2
root@host:[~]: dig api.www.remarpro.com +short
198.143.164.251

Step 3 – This process appears to hang when I attempt to run it. However if I take a look at the process I am seeing that it is stalling when attempting to make a connection to the IP (198.143.164.251).

Taking a look at the previous notes on the ticket and testing some things what I was able to determine is that these calls seems to be failing when they are originating from the server Main IP address, however if I modify some of the commands to use one of the servers other IP addresses this appears to be working. Below are some of the examples I found.

root@host:[~]: ping api.www.remarpro.com
PING api.www.remarpro.com (198.143.164.251) 56(84) bytes of data.
— api.www.remarpro.com ping statistics —
4 packets transmitted, 0 received, 100% packet loss, time 3000ms

root@host:[~]: ping -I 69.16.221.201 api.www.remarpro.com
PING api.www.remarpro.com (198.143.164.251) from 69.16.221.201 : 56(84) bytes of data.
64 bytes from api.www.remarpro.com (198.143.164.251): icmp_seq=1 ttl=56 time=32.9 ms
64 bytes from api.www.remarpro.com (198.143.164.251): icmp_seq=2 ttl=56 time=30.1 ms
64 bytes from api.www.remarpro.com (198.143.164.251): icmp_seq=3 ttl=56 time=30.1 ms
64 bytes from api.www.remarpro.com (198.143.164.251): icmp_seq=4 ttl=56 time=30.2 ms
— api.www.remarpro.com ping statistics —
4 packets transmitted, 4 received, 0% packet loss, time 3001ms
rtt min/avg/max/mdev = 30.169/30.884/32.992/1.223 ms

root@host:[~]: curl -v https://api.www.remarpro.com
* About to connect() to api.www.remarpro.com port 443 (#0)
* Trying 198.143.164.251…
* Connection timed out
* Failed connect to api.www.remarpro.com:443; Connection timed out
* Closing connection 0
curl: (7) Failed connect to api.www.remarpro.com:443; Connection timed out

root@host:[~]: curl –interface 69.16.221.201 -i -v https://api.www.remarpro.com
About to connect() to api.www.remarpro.com port 443 (#0)
Trying 198.143.164.251…
Name ‘69.16.221.201’ family 2 resolved to ‘69.16.221.201’ family 2
Local port: 0
Connected to api.www.remarpro.com (198.143.164.251) port 443 (#0)
Initializing NSS with certpath: sql:/etc/pki/nssdb
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
Server certificate:
—–Some content removed to save space—–

With this information in mind I performed a trace from the server to api.www.remarpro.com, and it would appear that these connections are being Dropped once they hit WordPress. I also tested this using the 69.16.221.201 IP on the server and looks to have taken the same route and there was no issue with the connection. Below is the two command I performed with the IPs specified in the commands.

root@host:[~]: mtr –no-dns –address 69.16.221.200 -r -c 20 api.www.remarpro.com –port 443
Start: Mon Jul 13 03:57:49 2020
HOST: host.antishrill.com Loss% Snt Last Avg Best Wrst StDev
1.|– 69.16.220.3 0.0% 20 0.9 0.7 0.6 1.0 0.0
2.|– 209.59.157.210 0.0% 20 1.4 1.3 1.1 1.7 0.0
3.|– 209.59.157.62 0.0% 20 1.5 1.5 1.2 2.0 0.0
4.|– 209.59.157.80 0.0% 20 24.2 24.4 24.2 25.1 0.0
5.|– 206.126.236.86 0.0% 20 23.9 24.3 23.9 32.3 1.8
6.|– 64.125.29.122 85.0% 20 30.0 30.6 30.0 30.9 0.0
7.|– 64.125.28.190 0.0% 20 30.0 30.2 30.0 31.0 0.0
8.|– 64.125.29.213 85.0% 20 31.3 30.9 30.3 31.3 0.0
9.|– 64.125.31.83 0.0% 20 30.0 30.7 29.9 36.9 1.8
10.|– 128.177.108.98 0.0% 20 31.1 31.1 30.7 32.3 0.3
11.|– 108.178.47.247 0.0% 20 30.3 30.4 30.3 30.8 0.0
12.|– ??? 100.0 20 0.0 0.0 0.0 0.0 0.0

root@host:[~]: mtr –no-dns –address 69.16.221.201 -r -c 20 api.www.remarpro.com –port 443
Start: Mon Jul 13 03:57:06 2020
HOST: host.antishrill.com Loss% Snt Last Avg Best Wrst StDev
1.|– 69.16.220.2 0.0% 20 1.1 0.8 0.6 1.1 0.0
2.|– 209.59.157.208 0.0% 20 1.2 1.4 1.2 1.8 0.0
3.|– 209.59.157.62 0.0% 20 1.4 1.4 1.2 1.9 0.0
4.|– 209.59.157.80 0.0% 20 24.4 24.4 24.1 24.7 0.0
5.|– 206.126.236.86 0.0% 20 24.0 25.0 23.8 40.3 3.6
6.|– 64.125.29.122 75.0% 20 37.9 34.2 31.3 38.2 3.5
7.|– 64.125.28.190 0.0% 20 30.4 30.8 30.0 33.6 0.8
8.|– 64.125.29.213 70.0% 20 32.2 31.4 30.5 33.0 0.8
9.|– 64.125.31.83 0.0% 20 30.3 30.1 29.9 30.4 0.0
10.|– 128.177.108.98 0.0% 20 32.3 31.1 30.7 32.3 0.2
11.|– 108.178.47.245 0.0% 20 30.8 30.5 30.3 30.8 0.0
12.|– 198.143.164.251 0.0% 20 30.2 30.2 30.2 30.3 0.0

Because the package loss appears to only be happening when using the main IP of the server, it would appear that something on WordPress’s end is either rate limiting the IP or blocking it. While performing some investigation and checking within WordPress’s documentation and Support forum I found the below link where another customer has a large number of domains on one IP and they are having the same issue with calls to api.www.remarpro.com being blocked. I see that there are currently around 600 Domains on this server and they all appear to be using the Main server IP at this time.

Hello,

After reading all of this and rethinking the problem…

I’m suspicious of some ‘Border’ router between your host and the backbone possibly having a hand in this problem where it may be blocking one of your IP addresses for that webserver at LiquidWeb.

You could hire someone to track that down and possibly get it fixed or you could just knuckle down and ask your host for two new IP Addresses… Results, if any, may vary greatly with hiring specialists. I’m a big fan of DIY solutions.

Make a list of all the sites experiencing the issue then move them to the new IP addresses. I’d put half on one and the other half on the other.

The first few will be telling… If they work properly after the move then you’ll know you are on the right track…

I don’t suggest you use any of your other 3 IP addresses that do work because that same border router may see the increased traffic from them as a threat or a problem and decide to block one or all of them. They may decide to block by CIDR Range this time and hurt you and your neighbors at LiquidWeb or their server farm(s). The new IPs seem to be the right idea to me.

I know it’s a lot of work but I see that work as positive steps once you’ve seen the first few work instead of the frustration of ‘no solution’ you see right now.

I almost completely reject the problem being at WordPress itself. Except there’s going to be a border router there, a firewall, or something ‘right outside the door’ at WordPress dot org.

I also suggest you might watch the sites on the problem IP to see if they start working after a time as you reduce the load by splitting off these problem sites on the problem IP to the new IPs.

I’ll also suggest you might start thinking about a multisite WordPress install where one WordPress then requests a single update for each plugin or theme that it then shares across the system. But think hard about that as multisite isn’t for everyone and/or every situation.

Here’s to hoping this will get you going in the right direction.

One more thought on this for the ‘next time’…

You might want to set up some staggered manual cron tasks for each of those sites you have. Something that will execute cron once a day (but late at night when traffic is lower). Your goal might be to run those once a day on each website just to cause a cron task to fire on that WordPress instance on that one IP and then the next and so on and so on! A minute separation might be all that’s needed for each instance.

I suggest this as that cron almost has to run if triggered manually instead of depending on the built-in pseudo-cron and a minute is a long time when you’re talking ethernet traffic. That minute should keep multiple requests from upsetting any firewalls and bot traps.

Some people suggest disabling cron via the wp_config file when setting up manual cron tasks but I’m more concerned about cron running at least once a day. I don’t see the absolute need to control it otherwise.

I appreciate your willingness to respond.