Site Crashing – Apache possible culprit

  • My site https://www.fallfishtenkara.com has crashed 6 times in the last 30 hours or so. I enter the comand ping www.fallfishtenkara.com

    PING https://www.fallfishtenkara.com (104.18.58.163) 56(84) bytes of data.
    64 bytes from 104.18.58.163: icmp_seq=1 ttl=61 time=1.60 ms
    64 bytes from 104.18.58.163: icmp_seq=2 ttl=61 time=1.58 ms
    64 bytes from 104.18.58.163: icmp_seq=3 ttl=61 time=1.50 ms
    64 bytes from 104.18.58.163: icmp_seq=4 ttl=61 time=1.50 ms
    ^C
    — https://www.fallfishtenkara.com ping statistics —
    4 packets transmitted, 4 received, 0% packet loss, time 3005ms
    rtt min/avg/max/mdev = 1.503/1.549/1.606/0.065 ms

    Not to mention I am able to SSH into the server, so it would appear that the server is running fine

    Next I run the command top and the results come back:

    PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
   28 root      20   0       0      0      0 R 34.4  0.0   3:10.47 kswapd0
12337 isaact8+  20   0   24940   1584   1064 R  8.6  0.2   0:00.04 top
 1403 root       0 -20       0      0      0 S  4.3  0.0   0:01.26 kworker/u3:1
11789 www-data  20   0  394080   7268   2656 R  4.3  0.7   0:03.03 apache2
11795 www-data  20   0  394080   7572   2672 D  4.3  0.7   0:02.89 apache2
11797 www-data  20   0  394336   7172   2748 R  4.3  0.7   0:03.02 apache2
11822 www-data  20   0  393828   9144   5652 R  4.3  0.9   0:02.84 apache2
11830 www-data  20   0  392036   7184   3600 R  4.3  0.7   0:02.82 apache2
11893 www-data  20   0  393060   7732   4412 R  4.3  0.8   0:02.61 apache2
11977 www-data  20   0  392040   9740   5664 R  4.3  1.0   0:02.43 apache2
12095 www-data  20   0  392036   7632   4996 R  4.3  0.7   0:02.34 apache2
    1 root      20   0   33472    120    120 S  0.0  0.0   0:01.90 init
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kthreadd
    3 root      20   0       0      0      0 S  0.0  0.0   0:00.20 ksoftirqd/0
    5 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker/0:0H
    6 root      20   0       0      0      0 S  0.0  0.0   0:00.12 kworker/u2:0
    7 root      20   0       0      0      0 S  0.0  0.0   0:17.88 rcu_sched

    A quick check on google tells me that kswapd0 is process that manages virtual memory… So I’m thinking that I’m having memory allocation issues so I run the command df -h and the results come back 

    Filesystem                 Size  Used Avail Use% Mounted on
udev                       487M  8.0K  487M   1% /dev
tmpfs                      100M  336K  100M   1% /run
/dev/disk/by-label/DOROOT   30G  8.7G   20G  32% /
none                       4.0K     0  4.0K   0% /sys/fs/cgroup
none                       5.0M     0  5.0M   0% /run/lock
none                       498M     0  498M   0% /run/shm
none                       100M     0  100M   0% /run/user

    and the command free -m 

    total       used       free     shared    buffers     cached
Mem:           994        930         63         16          0         22
-/+ buffers/cache:        907         86
Swap:         2047       1524        523

    I am concerned about the amount free memory in the buffer/cache row so I run the command ps aux | more and the results come back 

    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  33472    32 ?        Ss   Jun07   0:01 /sbin/init
root         2  0.0  0.0      0     0 ?        S    Jun07   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    Jun07   0:00 [ksoftirqd/0]
root         5  0.0  0.0      0     0 ?        S<   Jun07   0:00 [kworker/0:0H]
root         7  0.1  0.0      0     0 ?        S    Jun07   0:21 [rcu_sched]
root         8  0.0  0.0      0     0 ?        S    Jun07   0:16 [rcuos/0]
root         9  0.0  0.0      0     0 ?        S    Jun07   0:00 [rcu_bh]
root        10  0.0  0.0      0     0 ?        S    Jun07   0:00 [rcuob/0]
root        11  0.0  0.0      0     0 ?        S    Jun07   0:00 [migration/0]
root        12  0.0  0.0      0     0 ?        S    Jun07   0:00 [watchdog/0]
root        13  0.0  0.0      0     0 ?        S<   Jun07   0:00 [khelper]
root        14  0.0  0.0      0     0 ?        S    Jun07   0:00 [kdevtmpfs]
root        15  0.0  0.0      0     0 ?        S<   Jun07   0:00 [netns]
root        16  0.0  0.0      0     0 ?        S<   Jun07   0:00 [writeback]
root        17  0.0  0.0      0     0 ?        S<   Jun07   0:00 [kintegrityd]
root        18  0.0  0.0      0     0 ?        S<   Jun07   0:00 [bioset]
root        19  0.0  0.0      0     0 ?        S<   Jun07   0:00 [kworker/u3:0]
root        20  0.0  0.0      0     0 ?        S<   Jun07   0:00 [kblockd]
root        21  0.0  0.0      0     0 ?        S<   Jun07   0:00 [ata_sff]
root        22  0.0  0.0      0     0 ?        S    Jun07   0:00 [khubd]
root        23  0.0  0.0      0     0 ?        S<   Jun07   0:00 [md]
root        24  0.0  0.0      0     0 ?        S<   Jun07   0:00 [devfreq_wq]

    I’ve been following this article mostly https://www.smashingmagazine.com/2010/12/what-to-do-when-your-website-goes-down/ it is a little old but I feel that it is still applicable to my situation. At this point after I’ve run the command ps aux | more and I see that Apache is not running I’m pretty sure my issue is that Apache has crashed.

    When I run the command /etc/init.d/apache2 restart my website comes back online, sometimes for a few hours sometime less than that…

    Looking into the error.log file I find this:

    [Tue Jun 07 21:37:50.727260 2016] [mpm_prefork:notice] [pid 1709] AH00169: caught SIGTERM, shutting down
[Tue Jun 07 21:37:51.685688 2016] [mpm_prefork:notice] [pid 13608] AH00163: Apache/2.4.7 (Ubuntu) PHP/5.5.9-1ubuntu4.17 configured -- resuming normal operations
[Tue Jun 07 21:37:51.685791 2016] [core:notice] [pid 13608] AH00094: Command line: '/usr/sbin/apache2'
[Tue Jun 07 21:39:45.617022 2016] [core:alert] [pid 13687] [client 108.162.237.250:50121] /var/www/wp-content/updraft/.htaccess: deny not allowed here, referer: https://vaultpress.com
[Tue Jun 07 21:39:45.881250 2016] [core:alert] [pid 13655] [client 173.245.52.140:54743] /var/www/wp-content/updraft/.htaccess: deny not allowed here, referer: https://vaultpress.com
[Tue Jun 07 21:39:46.198784 2016] [core:alert] [pid 13697] [client 173.245.52.140:17020] /var/www/wp-content/updraft/.htaccess: deny not allowed here, referer: https://vaultpress.com
[Tue Jun 07 21:39:46.738494 2016] [core:alert] [pid 13687] [client 108.162.237.250:35083] /var/www/wp-content/updraft/.htaccess: deny not allowed here, referer: https://vaultpress.com
[Tue Jun 07 21:39:47.268006 2016] [core:alert] [pid 13655] [client 108.162.237.250:43514] /var/www/wp-content/updraft/.htaccess: deny not allowed here, referer: https://vaultpress.com
[Tue Jun 07 21:44:54.466348 2016] [core:alert] [pid 13838] [client 162.158.57.65:21824] /var/www/wp-content/plugins/akismet/.htaccess: Require not allowed here, referer: https://www.fallfishten$
[Tue Jun 07 21:44:55.475237 2016] [core:alert] [pid 13839] [client 162.158.57.29:49788] /var/www/wp-content/plugins/akismet/.htaccess: Require not allowed here, referer: https://www.fallfishten$
[Tue Jun 07 21:45:16.580154 2016] [core:alert] [pid 13859] [client 162.158.57.65:15744] /var/www/wp-content/plugins/akismet/.htaccess: Require not allowed here, referer: https://www.fallfishten$
[Tue Jun 07 21:45:16.672686 2016] [core:alert] [pid 13816] [client 162.158.57.29:34160] /var/www/wp-content/plugins/akismet/.htaccess: Require not allowed here, referer: https://www.fallfishten$
[Tue Jun 07 22:03:14.437769 2016] [core:notice] [pid 13608] AH00052: child pid 14133 exit signal Segmentation fault (11)
[Tue Jun 07 22:03:14.437882 2016] [core:notice] [pid 13608] AH00052: child pid 14275 exit signal Segmentation fault (11)
[Tue Jun 07 22:03:14.437900 2016] [core:notice] [pid 13608] AH00052: child pid 14325 exit signal Segmentation fault (11)
[Tue Jun 07 22:03:14.437914 2016] [core:notice] [pid 13608] AH00052: child pid 14329 exit signal Segmentation fault (11)
[Tue Jun 07 22:03:14.437950 2016] [core:notice] [pid 13608] AH00052: child pid 14182 exit signal Segmentation fault (11)
[Tue Jun 07 22:03:14.437998 2016] [core:notice] [pid 13608] AH00052: child pid 14005 exit signal Segmentation fault (11)
[Tue Jun 07 22:03:14.438077 2016] [mpm_prefork:notice] [pid 13608] AH00169: caught SIGTERM, shutting down
[Tue Jun 07 22:03:27.262255 2016] [mpm_prefork:notice] [pid 1116] AH00163: Apache/2.4.7 (Ubuntu) PHP/5.5.9-1ubuntu4.17 configured -- resuming normal operations
[Tue Jun 07 22:03:27.264088 2016] [core:notice] [pid 1116] AH00094: Command line: '/usr/sbin/apache2'
[Wed Jun 08 03:26:17.729067 2016] [core:notice] [pid 1116] AH00051: child pid 12012 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Wed Jun 08 03:26:17.843558 2016] [core:notice] [pid 1116] AH00051: child pid 12033 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Wed Jun 08 03:26:17.843974 2016] [core:notice] [pid 1116] AH00051: child pid 12076 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Wed Jun 08 03:26:18.850228 2016] [mpm_prefork:notice] [pid 1116] AH00169: caught SIGTERM, shutting down
[Wed Jun 08 03:26:19.538198 2016] [mpm_prefork:notice] [pid 12749] AH00163: Apache/2.4.7 (Ubuntu) PHP/5.5.9-1ubuntu4.17 configured -- resuming normal operations
[Wed Jun 08 03:26:19.538497 2016] [core:notice] [pid 12749] AH00094: Command line: '/usr/sbin/apache2'

    The issue started within a day or so after I updated my server. I ran the command sudo apt-get update && time sudo apt-get dist-upgrade which I have done about a dozen times in the past with no issues.

    At this point I am totally lost. Support from my host said that “Your apache instance is core dumping. You may need to use advanced debug tools like strace and gdb to debug the issues you are experiencing with your apache server….core dumping is a sign that either a plugin or module is misbehaving, or it can be a sign that a process is hitting a OOM event (out of memory) and is being forcefully closed.” Problem is I have no idea how to debug with strace or gdb…

    I am self hosting on a Digital Ocean droplet, so any help or pointers would be appreciated.

Viewing 1 replies (of 1 total)
  • Thread Starter isaacmtait

    (@isaacmtait)

    Update: After running command dmesg | grep -i memory to check my droplet’s dmesg log for OOM events I get

    [    0.000000] Scanning 1 areas for low memory corruption
[    0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000] init_memory_mapping: [mem 0x3ea00000-0x3ebfffff]
[    0.000000] init_memory_mapping: [mem 0x3c000000-0x3e9fffff]
[    0.000000] init_memory_mapping: [mem 0x00100000-0x3bffffff]
[    0.000000] init_memory_mapping: [mem 0x3ec00000-0x3fffdfff]
[    0.000000] Early memory node ranges
[    0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff]
[    0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff]
[    0.000000] Memory: 996256K/1048176K available (7354K kernel code, 1142K rwdata, 3396K rodata, 1332K init, 1440K bss, 51920K reserved)
[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[    0.014728] Initializing cgroup subsys memory
[    0.051337] Freeing SMP alternatives memory: 32K (ffffffff81e6c000 - ffffffff81e74000)
[    1.012431] Freeing initrd memory: 18780K (ffff88003ed99000 - ffff88003fff0000)
[    1.014168] Scanning for low memory corruption every 60 seconds
[    1.371029] Freeing unused kernel memory: 1332K (ffffffff81d1f000 - ffffffff81e6c000)
[    1.377943] Freeing unused kernel memory: 828K (ffff880001731000 - ffff880001800000)
[    1.396530] Freeing unused kernel memory: 700K (ffff880001b51000 - ffff880001c00000)
[    4.157479] [TTM] Zone  kernel: Available graphics memory: 508964 kiB

    Is there anything there I should be concerned about?

Viewing 1 replies (of 1 total)
  • The topic ‘Site Crashing – Apache possible culprit’ is closed to new replies.