I'm DOSing my own Apache server, and I can't stop immediately. What can I do to make things better in the short term?

Question

We've got a BUNCH of distributed clients that, among other things, upload log files to our Apache server.

We messed up log rotation on some logs, so we're now uploading relatively large files from these clients many times a day. Obviously, the fix is to get log rotation working, and put some more intelligence on the clients so they stop doing this. Getting ANY change pushed out to the clients will take days, possibly a week or two.

In the meantime, our 3 T1s are 100% saturated with these uploads, and many vital connections (much more important than the log uploads) are timing out.

The log uploading is handled by a Python script running under mod-wsgi, and we tried having it (via the Python script) immediately send a 200 success. This does not work -- curl (what we use to do the upload) will report 200 and a broken pipe after 30 seconds, but it's still uploading for those 30 seconds.

Any suggestions for something we can do about this? We don't really care that much if we lose log files, but we really want the vital traffic to go through.

Answer 1

Would it be possible to change your DNS to point to a server (or load-balanced server farm) which can handle the load, and/or a server with better net connectivity, so you're not bottlenecked at the 3 T-1's?

If the clients are connecting to an IP address, not a DNS name, have you talked to your upstream provider(s) about changing your routing, so that the IP address in question is routed to server space onsite at your provider or nearby at a colo facility?

Answer 2

Why don't you try another way without apache, like iptables. if it fits your need.
here is good start.
Cutter: http://www.lowth.com/cutter/

Apache or client will not stop exact second. Because connections have already been established. even if you restart apache. or block port 80 for upload locations. it will wait till end of timeout.

but with cutter, you can kill connection at certain second. which will give you enough bandwidth/resource during the time you need. shorten your conn. timeouts in apache configuration to save memory & dead/zombie apache childs. then use cutter to kill unnecessary uploads.

Answer 3

Sending a HTTP response is all well and good, but what you really need to do is to close the connection. I don't know mod_wsgi intimately, but in mod_php, for example, a simple:

exit(0);

at the top of the script works like a charm. You'll still (potentially) read an input buffer's-worth of bytes before the script gets called, but that's generally not a problem.

The other alternative, since you're using Apache, is to block things at a higher level; add a new GET string to your updated clients, and then add a config like:

RewriteEngine on
RewriteCond %{QUERY_STRING} !.*isNotBraindead=1.*
RewriteRule path/to/python/script.py - [F,L]

Answer 4

Along the lines of another suggestion, is there a specific DNS name that's being resolved for the data upload? You could try simply yanking the A record if it's not used for anything else.

Not sure what your network capabilities are, but you could also consider whether you can deploy ACLs upstream from you to block client traffic (assuming transmitting this log data is the only thing they're reaching you for). You could also use something like NBAR to discard traffic directed to the upload URL. These methods could preserve the limited bandwidth on your T lines, but unclear whether you have this level of network access.

I'm a bit surprised about your comment that the clients upload several times a day but you're still getting flooded after you immediately exit / return 200 due to the ~30 seconds of transmission.

Answer 5

In Apache configuration temporarily add:

<Location /some/url/to/upload/handler>
Deny from all
</Location>

Basically, forbid access to the URL used for the upload handler.

This presumes that the full URL path for upload handler isn't used for anything else.

Because an error, ie., non 200 status is returned, Apache will drop connection straight away.

Answer 6

I would look at slowing the traffic at the router and letting the TCP/IP protocol back off on the speed. For example you could use tc to implement a limit on the speed of the connections of your client servers to port 80 of the log server. If you limit the total speed available to all of those clients to, say, half of your available bandwidth then you would still be able to use the connections for the rest of your traffic.

If tc is not available or not suitable, then look at any other tool to provide Quality of Service (QoS) which should provide the ability to limit the bandwidth available to particular questions.

Answer 7

I'd agree with the others that throttling the bandwidth is the answer (NOT QoS routing - becuase the service is always HTTP - you just want some clients to go slower).

Potentially you could this with a reverse proxy (squid setup described here), using iptables or with one of the many throttling modules for Apache (search here for bandwidth and your webserver version).

You can probably do the iptables thing without any reboots / hardware / software changes, but the apache module will give you very fine grained control over which clinets / URLs to apply the throttling to - although I'm not sure how many of them actually work with uploads.

HTH

C.