1

My first post, unfortunately due to an issue on my dedicated (Linux Centos 6) server running some personal and one commercial website.

Server version: Apache/2.4.18 (Unix)
Server built:   Mar  7 2016 20:22:35
Cpanel::Easy::Apache v3.32.10 rev9999

root@server213-171-196-40 [~]# free
             total       used       free     shared    buffers     cached
Mem:      16212880   15939912     272968    1939060      65928   13730952
-/+ buffers/cache:    2143032   14069848
Swap:      4194296    1477616    2716680

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
[b] 2900 nobody    20   0 90636  14m 2408 R 99.1  0.1   0:07.42 httpd[/b]
   72 root      20   0     0    0    0 R 17.3  0.0   8:08.35 kswapd0
  436 root      20   0     0    0    0 S  2.3  0.0   1:54.42 md1_raid1
    1 root      20   0 19356  668  452 S  0.0  0.0   0:00.72 init
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd
    3 root      RT   0     0    0    0 S  0.0  0.0   0:00.03 migration/0
    4 root      20   0     0    0    0 S  0.0  0.0   0:00.37 ksoftirqd/0
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/0
    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.05 watchdog/0
    7 root      RT   0     0    0    0 S  0.0  0.0   0:00.04 migration/1

Above is an example from top when the httpd process is 'thrashing'. It obviously pushes the load on the server right up, and when there's 4 httpd processes doing the same, the websites become very unresponsive as you can imagine. normally the httpd processes are running at 0.1% 0.5% range, ticking over nicely.

The spikes in CPU load only started early hours of Monday morning, and I had not re-built apache / changed any config on Sunday.

i run quite a few wordpress sites - I double checked and see any updates to plugins at that time.

My linux skills a bit limited, I picked a PID for one of the maxed httpd processes, and tried to trace what it was doing using strace -p command

showed this scrolling past at a fast rate.. looks to my untrained eye like DOS attack? flooding the server?

1457649966.654499 read(114, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024 <0.000005>
1457649966.654515 read(114, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024 <0.000005>
1457649966.654531 read(114, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024 <0.000005>
1457649966.654545 read(114, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024 <0.000006>
1457649966.654559 read(114, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024 <0.000005>
1457649966.654573 read(114, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024 <0.000005>
1457649966.654587 read(114, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1024) = 1024 <0.000005>

I could obviously be wrong - in 10 or so seconds of a spike, my logfile out output was over 2 million rows! so a fantstic amount of short running 'reads'.

Another trace on a runaway process:

root@server213-171-196-40 [/usr/local/iftop-0.17]# strace -c -p 23369
Process 23369 attached
^CProcess 23369 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.044961           0   2367035           read
------ ----------- ----------- --------- --------- ----------------
100.00    0.044961               2367035           total

I don't actually understand what is happening there - my guess is something is somehow pushing a huge amount of noise/empty data at my httpd processes, and doing so every 20 seconds/every couple of minutes.

Or it's something else completely and I'm well off the mark.

I'm trying to put some of my sites under cloudflare to a) speed up the graphics intensive sites using their CDN, but also to try and add an extra layer of defense. hasn't done anything I can see yet. I can't even work out which url / website is the cause, if any.

The server does have mod_security, with the standard set of rules - I did turn off an ip check as read somewhere it could slowdown serving (no effect though). The server has plenty of memory (16 gig), 4 cpus (an 8 core intel machine) and the apache processes which spike show a tiny memory usage, it's all cpu.

PHP version 5.6.18 (or .19, need to double check).

Netstat at times of 'spike'don't show any single ip having more than 20 connections - and the server isn't overloaded (I've turned off my busy website which was getting over 15,000 visitors a day to remove traffic as the cause).

root@server213-171-196-40 [~]# netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

      1 108.162.221.138
      1 108.162.246.252
      1 141.101.98.176
      1 173.245.50.109
      1 199.16.156.125
      1 213.128.67.90
      1 213.205.194.66
      1 222.186.34.163
      1 79.70.61.9
      1 86.128.207.3
      1 94.14.114.56
      1 95.151.139.42
      1 Address
      1 servers)
      2 108.162.222.88
      2 173.245.56.127
      4 86.4.247.37
      5 136.243.48.85
      6 78.147.41.131
      7 90.220.251.88

The server was working fine up to Monday morning - under heavier load than it is now.

From httpd.conf:

Timeout 300
TraceEnable Off
ServerSignature Off
ServerTokens ProductOnly
FileETag All
StartServers 5
<IfModule prefork.c>
MinSpareServers 5
MaxSpareServers 10
</IfModule>
ServerLimit 256
MaxRequestWorkers 150
MaxConnectionsPerChild 15000
KeepAlive On
KeepAliveTimeout 5
MaxKeepAliveRequests 100

Current using the prefork mpm, but I was running using the event mpm happily, I rebuilt apache under the 'standard' pre-fork set-up as per WHM/CPANAL easyapache3 to see if re-building and changing the config for apache would work, it didn't.

If any of you have seen similar output of a trace, or the same sort of http thrashing, I'd love to hear your advice.

  • I'm currently having the exact same problem. Did you ever find out a solution? – greatwitenorth Oct 6 '17 at 4:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.