0

I am the reluctant "sysadmin" of a medium-sized cooking blog that runs on a t2.small EC2 instance, and utilizes WordPress on the LAMP stack. Despite being a fairly embarrasingly bad "sysadmin" it actually runs really reliably and efficiently despite shipping a LOT of traffic. I've done everything I can but every now and then (about once a year) someone manages to hammer it with a DDOS type attack. What tends to happen is that the apache server dies and anyone request anything from the website gets a HTTP timeout. Restarting the httpd service immediately fixes the problem.

What I'm thinking of doing is scheduling a CRON job to call the website every 5ish minutes and if it finds that it gets a timeout more than twice, it restarts the httpd service and sends me an email. Now, this is NOT one of those "I can't set up my apache server properly so I'll restart it every hour" solutions. The apache is tweaked and runs beautifully when it isn't being hacked. And I've put in as much firewall anti-hacking best practice as I possibly can. My rationale is basically "hackers and crazies are always finding new ways to hammer me and when they do I'd rather have 10 minutes of outage and an email than potentially hours of downtime (if I'm asleep or away)". The question is, is this a good solution or is there a better, more sysadminny way to do this using some existing technology?

NOTE: When I say the apache server dies it actually looks fine when you run top -U apache. WHat it does is writes about 10

(12)Cannot allocate memory: AH00159: fork: Unable to fork new process

to the error_log and just doesn't do anything at all until restarted.

  • 4
    Don't reinvent the wheel. There are numerous things that can do this already, but you may as well just use systemd, as your Linux system should already be using it to manage services. – Michael Hampton Dec 23 '16 at 21:33
  • 3
    Why don't you solve the problem rather than trying to work around it ? – user9517 supports GoFundMonica Dec 23 '16 at 21:45
  • +1 For what @Hanginoninquietdesperation said. If it tries to fork and it can't, then something is configured wrong. You run out of memory, file descriptors or whatever. – Florin Asăvoaie Dec 23 '16 at 22:06
  • 1
    Monit will achieve your goal, but go with what Michael Hampton says first : mmonit.com/monit – Tim Dec 23 '16 at 22:15
  • I understand where you guys are coming from, but as I explained in the thread, this is not a regular occurrence based on a poorly-configured apache Server. My server never runs out of resources and is a well-oiled machine. But when you start running websites that do 1M+ page views a month you get people try to hack then and break them. Every time someone hacks my website I secure it so that the next time it won't work. How do I fix a denial of service attack that hasn't happened yet? There is no issue with my Server when it isn't being hacked. BTW Thanks Michael, I'll check it out! – Duncan Edwards Dec 31 '16 at 18:34
1

You can do this pretty easily by checking the return code of a curl command:

#!/bin/bash
# Request website (timeout five seconds)
/usr/bin/curl -m5 $1 > /dev/null
RC=$?
if [ $RC -ne 0 ]; then
    echo Website: $1 not running, restarting apache. Curl RC: $RC
    /usr/sbin/service httpd restart
fi

The idea is that if the return code does not equal 0, i.e., the request failed, then restart apache.

You would put it in cron something like this:

*/5 * * * * /path/to/check_site.sh website.url.com >> /path/to/log/file.log 2>&1

This would make sure that it is logging the results.

Note: This is a bandaid. It is a good idea to get on and make sure this isn't running repeatedly. Perhaps put in a send me an email line?

  • I didn't end up doing this because I realised that the error was a lack of memory and the best way of fixing this was to create a 1GB swap drive (which is not there on EC2 by default). Now, any spike in traffic just eats into my CPU credits a little and brings nothing down. – Duncan Edwards Mar 2 '17 at 15:01
0

For a while, Microsoft was ignoring robots.txt and hammering my server with their search engine crawler. When things got slow, I would see as many as 200 httpd instances, most of them Microsoft crawlers! So I implemented the following script, which checks to see how many httpd instances are running, and if over a certain number, kills them all.

This is pretty unsophisticated. You can change the number of httpd instances on line 5 from "30" to anything you like. You can change the time between monitoring by changing "sleep 30" to any number of seconds you prefer. I looked into only killing Microsoft crawler instances, but it would have taken parsing the apache logs, which I didn't feel like getting into.

#!/bin/bash
while [ 1 ]
    do
    set `psg httpd | wc`
    if test $1 -gt 30
    then
        echo killing
        killall httpd
    else
        echo -n $1 ' '
    fi
    sleep 30
done
  • You can restrict this using: MaxSpareServers nn in <IfModule prefork.c> of httpd.conf – Duncan Edwards Dec 24 '16 at 14:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.