We've got a BUNCH of distributed clients that, among other things, upload log files to our Apache server.
We messed up log rotation on some logs, so we're now uploading relatively large files from these clients many times a day. Obviously, the fix is to get log rotation working, and put some more intelligence on the clients so they stop doing this. Getting ANY change pushed out to the clients will take days, possibly a week or two.
In the meantime, our 3 T1s are 100% saturated with these uploads, and many vital connections (much more important than the log uploads) are timing out.
The log uploading is handled by a Python script running under mod-wsgi, and we tried having it (via the Python script) immediately send a 200 success. This does not work -- curl (what we use to do the upload) will report 200 and a broken pipe after 30 seconds, but it's still uploading for those 30 seconds.
Any suggestions for something we can do about this? We don't really care that much if we lose log files, but we really want the vital traffic to go through.