0

We have a cluster setup of 3 server, on all 3 servers weblogic is running to host website. On the top of weblogic using Apache as load balancer.

Last year I had upgraded Apache http server from 2.4.9 to 2.4.29. Before upgrade things were working fine but post upgrade after every 2-4 weeks Apache hangs. Website becomes inaccessible, on checking the server we don't see any error in fact the moment when apache hangs it stops logging too.

We monitored and found no abnormal memory usage, no suspicious type of request to website, no suspicious process running before the issue occurs.

After exploring over the internet, I found that problem could be related to MPM configuration. Apache was using event MPM but I found that event MPM configuration was not declared in apache configuration, I think during upgrade default configuration was replaced by old configuration.

So we have modified MPM configuration to use below event mpm configuration and this change worked. We did not face this issue for next 6 months.

StartServers 4 MinSpareThreads 256 MaxSpareThreads 512 ThreadsPerChild 64 MaxRequestWorkers 2048 MaxConnectionsPerChild 0

We thought it is fixed but now after 6 months Apache hangs again, only symptoms we can see increasing closed_wait connections from 20 to 550 in next 1 hours, no accessibility of website, apache stopped logging to it's ssl and error logs, we don't see any abnormal behaviour in different running process data, memory usage, also don't see any suspicious request before issue occurred.

0

It looks like a 'slow DoS attack' using tools like slowloris. In such type of attacks, the attacker tries to keep alive (open) legitimate connections to the server until the server hangs. We use SPA ( https://www.cyberxnetworks.com/index.php?cmd=website&aid=242 ) to stop such type of attack.

  • We have script to capture netstat every minute. I don't see too many connection before the issue happens. Number of connections used to be within normal range, and keeps rotating, don't see any idle connection for too long. – HarishS Jun 26 at 12:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.