• blackvault

    (@blackvault)


    Ok here is one I am dying for help on.

    I have a fairly high traffic site that was built on Boonex Dolphin. I have split up the site, and built 3 different WordPress installations (using the same DB). My CPU load has skyrocketed (usually under 1 according to top, it hovers all over the place, going 6, 10, and even higher. Sometimes, 60+ with tons of php processes. I am also seeing a ton of php defunct processes.

    I have tried hard to tweak settings, but nothing seems to work. Here is the my.cnf:

    [mysqld]
    max_allowed_packet=268435456
    query_cache_size = 512M
    query_cache_limit = 4M
    tmp_table_size = 256M
    max_heap_table_size = 256M
    thread_cache_size = 16k
    table_open_cache = 2000
    table_definition_cache = 4K
    open_files_limit = 3K
    key_buffer = 2048M
    innodb_buffer_pool_size = 640M
    long_query_time = 5
    join_buffer_size = 2M
    max_connections = 250
    wait_timeout = 28000
    interactive_timeout = 28000

    I was struggling with MYSQL crashes, and nothing stuck out in the error logs that clued me in. The changes I made (so far) have stopped the crashes, however, I am still seeing the high load.

    I don’t have many plugins, and most are the ones that are VERY popular, so nothing off brand. Here are my server specs:

    Intel Core 2 Core2Duo E8400 3.0 GHz
    8 GB DDR2-800 SDRAM
    1 Gbps Uplink Port Speed
    CentOS 6.0 – 64 Bit

    Here is my phpinfo: https://www.theblackvault.com/phpinfo.php

    Here is a screen shot of my ps -awfux.

    ANY help would be greatly appreciated.

    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    root         2  0.0  0.0      0     0 ?        S    Mar08   0:00 [kthreadd]
    root         3  0.0  0.0      0     0 ?        S    Mar08   0:04  \_ [migration/0]
    root         4  0.0  0.0      0     0 ?        S    Mar08   0:30  \_ [ksoftirqd/0]
    root         5  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [stopper/0]
    root         6  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [watchdog/0]
    root         7  0.0  0.0      0     0 ?        S    Mar08   0:03  \_ [migration/1]
    root         8  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [stopper/1]
    root         9  0.0  0.0      0     0 ?        S    Mar08   0:26  \_ [ksoftirqd/1]
    root        10  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [watchdog/1]
    root        11  0.0  0.0      0     0 ?        S    Mar08   0:14  \_ [events/0]
    root        12  0.0  0.0      0     0 ?        S    Mar08   0:18  \_ [events/1]
    root        13  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [cgroup]
    root        14  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [khelper]
    root        15  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [netns]
    root        16  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [async/mgr]
    root        17  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [pm]
    root        18  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [sync_supers]
    root        19  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [bdi-default]
    root        20  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [kintegrityd/0]
    root        21  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [kintegrityd/1]
    root        22  0.0  0.0      0     0 ?        S    Mar08   0:48  \_ [kblockd/0]
    root        23  0.0  0.0      0     0 ?        S    Mar08   0:45  \_ [kblockd/1]
    root        24  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [kacpid]
    root        25  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [kacpi_notify]
    root        26  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [kacpi_hotplug]
    root        27  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [ata_aux]
    root        28  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [ata_sff/0]
    root        29  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [ata_sff/1]
    root        30  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [ksuspend_usbd]
    root        31  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [khubd]
    root        32  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [kseriod]
    root        33  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [md/0]
    root        34  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [md/1]
    root        35  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [md_misc/0]
    root        36  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [md_misc/1]
    root        37  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [linkwatch]
    root        38  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [khungtaskd]
    root        39  0.0  0.0      0     0 ?        S    Mar08   0:40  \_ [kswapd0]
    root        40  0.0  0.0      0     0 ?        SN   Mar08   0:00  \_ [ksmd]
    root        41  0.0  0.0      0     0 ?        SN   Mar08   0:08  \_ [khugepaged]
    root        42  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [aio/0]
    root        43  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [aio/1]
    root        44  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [crypto/0]
    root        45  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [crypto/1]
    root        53  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [kthrotld/0]
    root        54  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [kthrotld/1]
    root        56  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [kpsmoused]
    root        57  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [usbhid_resumer]
    root        58  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [deferwq]
    root        89  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [kdmremove]
    root        90  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [kstriped]
    root       122  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [i915]
    root       123  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [i915-dp]
    root       248  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [scsi_eh_0]
    root       249  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [scsi_eh_1]
    root       250  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [scsi_eh_2]
    root       251  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [scsi_eh_3]
    root       252  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [scsi_eh_4]
    root       253  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [scsi_eh_5]
    root       309  0.0  0.0      0     0 ?        S    Mar08   2:38  \_ [jbd2/sda3-8]
    root       310  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [ext4-dio-unwrit]
    root       318  0.0  0.0      0     0 ?        S    Mar08   0:24  \_ [flush-8:0]
    root       722  0.0  0.0      0     0 ?        S    Mar08   0:00  \_ [kjournald]
    root       819  0.0  0.0      0     0 ?        S    Mar08   0:04  \_ [kauditd]
    root         1  0.0  0.0  19356  1000 ?        Ss   Mar08   0:03 /sbin/init
    root       399  0.0  0.0  10840   260 ?        S<s  Mar08   0:00 /sbin/udevd -d
    root      7323  0.0  0.0  10836   204 ?        S<   Mar08   0:00  \_ /sbin/udevd -d
    root      7324  0.0  0.0  10836   204 ?        S<   Mar08   0:00  \_ /sbin/udevd -d
    root      1084  0.0  0.0  93156   756 ?        S<sl Mar08   0:22 auditd
    root      1214  0.0  0.0 249472  4988 ?        Sl   Mar08   0:33 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
    root      1244  0.0  0.0  18240   472 ?        Ss   Mar08   0:11 irqbalance --pid=/var/run/irqbalance.pid
    named     1260  0.0  0.1 245716 12764 ?        Ssl  Mar08   0:00 /usr/sbin/named -u named
    dbus      6259  0.0  0.0  21432   408 ?        Ss   Mar08   0:00 dbus-daemon --system
    root      6275  0.0  0.0   4080   468 ?        Ss   Mar08   0:00 /usr/sbin/acpid
    68        6285  0.0  0.0  38080  1388 ?        Ssl  Mar08   0:01 hald
    root      6286  0.0  0.0  20400   480 ?        S    Mar08   0:00  \_ hald-runner
    root      6315  0.0  0.0  22520   436 ?        S    Mar08   0:00      \_ hald-addon-input: Listening on /dev/input/event0 /dev/input/event1
    68        6329  0.0  0.0  18008   588 ?        S    Mar08   0:00      \_ hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
    nobody    6343  0.0  0.0  57912   608 ?        Ss   Mar08   0:13 proftpd: (accepting connections)
    theblack 22981  0.0  0.0  62120  4684 ?        SL   16:20   0:00  \_ proftpd: theblack - 76.95.131.233: IDLE
    root      6358  0.0  0.0  66216   548 ?        Ss   Mar08   0:16 /usr/sbin/sshd
    root     15937  0.0  0.0 104896  1848 ?        Ss   Mar10   0:08  \_ sshd: root@pts/0
    root     15943  0.0  0.0 108336  1332 pts/0    Ss   Mar10   0:00  |   \_ -bash
    root      7992  1.0  0.0 110336  1164 pts/0    R+   16:52   0:00  |       \_ ps -awfux
    root      7893  0.8  0.0 103460  4292 ?        Ss   16:52   0:00  \_ sshd: root [priv]
    sshd      7894  0.0  0.0  67560  1636 ?        S    16:52   0:00  |   \_ sshd: root [net]
    root      7902  1.0  0.0 103460  4292 ?        Ss   16:52   0:00  \_ sshd: root [priv]
    sshd      7903  0.0  0.0  67560  1632 ?        S    16:52   0:00  |   \_ sshd: root [net]
    root      7915  0.6  0.0 103460  4188 ?        Rs   16:52   0:00  \_ sshd: root [priv]
    sshd      7916  0.0  0.0  67560  1636 ?        S    16:52   0:00      \_ sshd: root [net]
    ntp       6367  0.0  0.0  30732  1316 ?        Ss   Mar08   0:04 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
    root      6700  0.0  0.0  19612   872 ?        Ss   Mar08   0:03 /usr/sbin/dovecot
    dovenull  6712  0.0  0.0  42400  1556 ?        S    Mar08   0:00  \_ dovecot/pop3-login
    dovenull  6713  0.0  0.0  42732  2284 ?        S    Mar08   0:03  \_ dovecot/imap-login
    dovecot   6714  0.0  0.0  13316   824 ?        S    Mar08   0:02  \_ dovecot/anvil
    root      6715  0.0  0.0  13568   968 ?        S    Mar08   0:02  \_ dovecot/log
    dovenull  6717  0.0  0.0  42400  1568 ?        S    Mar08   0:00  \_ dovecot/pop3-login
    dovenull  6718  0.0  0.0  42776  2424 ?        S    Mar08   0:07  \_ dovecot/imap-login
    root      6719  0.0  0.0  16432  1544 ?        S    Mar08   0:04  \_ dovecot/config
    greenewa 19906  0.0  0.0  25304  3276 ?        S    Mar09   0:08  \_ dovecot/imap
    dovecot   7423  0.0  0.0  31560  2020 ?        S    16:51   0:00  \_ dovecot/auth
    greenewa  7428  0.0  0.0  23088  1912 ?        S    16:51   0:00  \_ dovecot/imap
    govsec    7429  0.0  0.0  23088  1912 ?        S    16:51   0:00  \_ dovecot/imap
    govsec    7438  0.0  0.0  23424  2508 ?        S    16:51   0:00  \_ dovecot/imap
    greenewa  7441  0.0  0.0  23488  2660 ?        S    16:51   0:00  \_ dovecot/imap
    mailnull  6721  0.0  0.0  71136   800 ?        Ss   Mar08   0:16 /usr/sbin/exim -bd -q1h
    root      6749  0.0  0.2 167784 17600 ?        Ss   Mar08   0:33 /usr/local/cpanel/3rdparty/perl/514/bin/spamd -d --allowed-ips=127.0.0.1 --pidfile=/var/run/spamd.pid --max-children=5
    root       307  0.0  0.8 188816 64588 ?        S    Mar10   1:04  \_ spamd child
    root     23282  0.0  0.6 174252 49120 ?        S    12:04   0:01  \_ spamd child
    root      6780  0.0  0.0 114640   952 ?        Ss   Mar08   0:02 /usr/sbin/abrtd
    root      6788  0.0  0.0 112520   652 ?        Ss   Mar08   0:04 abrt-dump-oops -d /var/spool/abrt -rwx /var/log/messages
    root      6811  0.0  0.0 116856   708 ?        Ss   Mar08   0:03 crond
    root      6825  0.0  0.0  21104   368 ?        Ss   Mar08   0:00 /usr/sbin/atd
    root      7063  0.0  0.0  77476  2120 ?        S    Mar08   1:06 cPhulkd - processor
    root      7089  0.0  0.0 114256  3172 ?        S    Mar08   0:07 cpsrvd (SSL) - waiting for connections
    root      7112  0.0  0.0 116080   804 ?        S    Mar08   0:00 cpdavd - accepting connections on 2077 and 2078
    root      7155  0.0  0.0  38372  2016 ?        S    Mar08   0:07 queueprocd - wait to process a task
    root      7207  0.0  0.0  85868  7372 ?        S    Mar08   0:53 tailwatchd
    root      7218  0.0  0.0  30556  1272 ?        SN   Mar08   0:00 cpanellogd - sleeping for logs
    root      7304  0.0  0.0   4064   436 tty1     Ss+  Mar08   0:00 /sbin/mingetty /dev/tty1
    root      7307  0.0  0.0   4064   436 tty2     Ss+  Mar08   0:00 /sbin/mingetty /dev/tty2
    root      7310  0.0  0.0   4064   436 tty3     Ss+  Mar08   0:00 /sbin/mingetty /dev/tty3
    root      7312  0.0  0.0   4064   436 tty4     Ss+  Mar08   0:00 /sbin/mingetty /dev/tty4
    root      7314  0.0  0.0   4064   436 tty5     Ss+  Mar08   0:00 /sbin/mingetty /dev/tty5
    root      7316  0.0  0.0   4064   436 tty6     Ss+  Mar08   0:00 /sbin/mingetty /dev/tty6
    nobody    5760  0.0  0.2 322488 17204 ?        S    11:05   0:00 /usr/bin/php
    nobody    5761  0.0  0.8 361432 68640 ?        S    11:05   0:01 /usr/bin/php
    nobody    5762  0.0  0.2 319408 22408 ?        S    11:05   0:00 /usr/bin/php
    nobody    5763  0.0  0.2 322488 19872 ?        S    11:05   0:00 /usr/bin/php
    nobody    5764  0.0  0.2 322488 21012 ?        S    11:05   0:00 /usr/bin/php
    nobody    5765  0.0  0.1 316116 14812 ?        S    11:05   0:00 /usr/bin/php
    nobody    5766  0.0  0.2 322488 18604 ?        S    11:05   0:00 /usr/bin/php
    nobody    5767  0.0  0.6 349604 54448 ?        S    11:05   0:01 /usr/bin/php
    nobody    5768  0.0  0.2 321976 20284 ?        S    11:05   0:00 /usr/bin/php
    nobody    5769  0.0  0.7 369804 58472 ?        S    11:05   0:02 /usr/bin/php
    nobody    5807  0.0  0.3 322580 30712 ?        S    11:05   0:00 /usr/bin/php
    nobody    6316  0.0  0.9 369436 76544 ?        S    11:07   0:05 /usr/bin/php
    nobody    6319  0.0  0.9 359160 71788 ?        S    11:07   0:02 /usr/bin/php
    nobody    6320  0.0  1.0 368908 79664 ?        S    11:07   0:05 /usr/bin/php
    nobody    6321  0.0  1.0 368360 80380 ?        S    11:07   0:03 /usr/bin/php
    nobody    6322  0.0  0.6 340572 53428 ?        S    11:07   0:05 /usr/bin/php
    nobody    6323  0.0  0.9 359360 72236 ?        S    11:07   0:05 /usr/bin/php
    nobody    6324  0.0  0.8 359100 71484 ?        S    11:07   0:02 /usr/bin/php
    nobody    6325  0.0  0.5 333752 46432 ?        S    11:07   0:01 /usr/bin/php
    nobody    6326  0.0  0.7 349556 61812 ?        S    11:07   0:02 /usr/bin/php
    nobody    6327  0.0  0.9 364952 77196 ?        S    11:07   0:02 /usr/bin/php
    nobody    6328  0.0  0.8 356848 67960 ?        S    11:07   0:02 /usr/bin/php
    nobody    6340  0.0  0.6 340824 52460 ?        S    11:07   0:02 /usr/bin/php
    nobody    6344  0.0  0.7 344688 57876 ?        S    11:07   0:02 /usr/bin/php
    nobody    6346  0.0  0.9 365980 77444 ?        S    11:07   0:02 /usr/bin/php
    nobody    6348  0.0  0.5 333236 46652 ?        S    11:07   0:01 /usr/bin/php
    nobody    6356  0.0  0.5 333484 43008 ?        S    11:07   0:01 /usr/bin/php
    nobody    6359  0.0  0.7 354236 62308 ?        S    11:07   0:01 /usr/bin/php
    nobody    6368  0.0  0.8 361104 70408 ?        S    11:07   0:01 /usr/bin/php
    root      6831  0.0  0.3  96536 26100 ?        Ss   11:07   0:01 /usr/local/apache/bin/httpd -k start -DSSL
    nobody   17927  0.0  0.0  96536  2892 ?        S    16:15   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    root     17928  0.0  0.1  79264  9688 ?        S    16:15   0:00  \_ /usr/local/cpanel/3rdparty/bin/perl /usr/local/cpanel/bin/leechprotect
    nobody   17929  0.3  0.5 1356700 41152 ?       Sl   16:15   0:06  \_ /usr/local/apache/bin/httpd -k start -DSSL
    theblack  7960 10.0  0.5 234168 42772 ?        R    16:52   0:00  |   \_ /usr/bin/php /home/theblack/public_html/documentarchive/index.php
    nobody   17930  0.5  0.2 1356352 18656 ?       Sl   16:15   0:11  \_ /usr/local/apache/bin/httpd -k start -DSSL
    theblack  7970  6.5  0.3 220856 31200 ?        R    16:52   0:00  |   \_ /usr/bin/php /home/theblack/public_html/documentarchive/index.php
    nobody   17931  0.3  0.4 1356700 37348 ?       Sl   16:15   0:08  \_ /usr/local/apache/bin/httpd -k start -DSSL
    theblack  7986  2.0  0.1 216224 11672 ?        R    16:52   0:00  |   \_ /usr/bin/php /home/theblack/public_html/modules/index.php
    nobody   17932  0.5  0.5 1356400 40744 ?       Sl   16:15   0:11  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   18073  1.6  0.2 1356932 22440 ?       Sl   16:15   0:37  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody     531  2.3  0.6 1360820 47900 ?       Sl   16:42   0:13  \_ /usr/local/apache/bin/httpd -k start -DSSL
    theblack  7922  4.6  0.0      0     0 ?        Z    16:52   0:00      \_ [php] <defunct>
    theblack  7923  4.8  0.0      0     0 ?        Z    16:52   0:00      \_ [php] <defunct>
    theblack  7949  7.6  0.0      0     0 ?        Z    16:52   0:00      \_ [php] <defunct>
    theblack  7951  8.0  0.0      0     0 ?        Z    16:52   0:00      \_ [php] <defunct>
    theblack  7953 11.5  0.5 236880 45388 ?        R    16:52   0:00      \_ /usr/bin/php /home/theblack/public_html/documentarchive/index.php
    theblack  7955 11.5  0.0      0     0 ?        R    16:52   0:00      \_ [php]
    theblack  7959 10.0  0.5 234192 42744 ?        R    16:52   0:00      \_ /usr/bin/php /home/theblack/public_html/documentarchive/index.php
    theblack  7965  8.5  0.4 226136 36276 ?        R    16:52   0:00      \_ /usr/bin/php /home/theblack/public_html/documentarchive/index.php
    theblack  7968  6.5  0.3 318092 27484 ?        R    16:52   0:00      \_ /usr/bin/php /home/theblack/public_html/modules/index.php
    theblack  7969  6.0  0.3 219044 28896 ?        R    16:52   0:00      \_ /usr/bin/php /home/theblack/public_html/documentarchive/index.php
    theblack  7975  3.5  0.3 316824 24316 ?        R    16:52   0:00      \_ /usr/bin/php /home/theblack/public_html/modules/index.php
    theblack  7976  4.0  0.2 316228 20412 ?        R    16:52   0:00      \_ /usr/bin/php /home/theblack/public_html/modules/index.php
    theblack  7977  3.0  0.2 315716 18400 ?        R    16:52   0:00      \_ /usr/bin/php /home/theblack/public_html/modules/index.php
    theblack  7978  3.0  0.2 315448 17844 ?        R    16:52   0:00      \_ /usr/bin/php /home/theblack/public_html/modules/index.php
    theblack  7979  2.5  0.2 315192 17340 ?        R    16:52   0:00      \_ /usr/bin/php /home/theblack/public_html/modules/index.php
    theblack  7985  3.0  0.1 216408 12384 ?        R    16:52   0:00      \_ /usr/bin/php /home/theblack/public_html/modules/index.php
    theblack  7987  3.0  0.1 216224 11900 ?        R    16:52   0:00      \_ /usr/bin/php /home/theblack/public_html/modules/index.php
    root      1479  0.0  0.0  11344  1388 pts/0    S    16:43   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/god.theblackvault.com.pid
    mysql     1769  3.5  7.4 4531972 596736 pts/0  Sl   16:43   0:18  \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/lib/mysql/god
    root@god [/var/lib/mysql]#
  • The topic ‘Boonex Dolphin -> WordPress Migration – High CPU, PHP Defunct Process, and more’ is closed to new replies.