0

We have a cluster setup of 3 server, on all 3 servers weblogic is running to host website. On the top of weblogic using Apache as load balancer.

Last year I had upgraded Apache http server from 2.4.9 to 2.4.29. Before upgrade things were working fine but post upgrade after every 2-4 weeks Apache hangs. Website becomes inaccessible, on checking the server we don't see any error in fact the moment when apache hangs it stops logging too.

We monitored and found no abnormal memory usage, no suspicious type of request to website, no suspicious process running before the issue occurs.

After exploring over the internet, I found that problem could be related to MPM configuration. Apache was using event MPM but I found that event MPM configuration was not declared in apache configuration, I think during upgrade default configuration was replaced by old configuration.

So we have modified MPM configuration to use below event mpm configuration and this change worked. We did not face this issue for next 6 months.

StartServers 4 MinSpareThreads 256 MaxSpareThreads 512 ThreadsPerChild 64 MaxRequestWorkers 2048 MaxConnectionsPerChild 0

We thought it is fixed but now after 6 months Apache hangs again, only symptoms we can see increasing closed_wait connections from 20 to 550 in next 1 hours, no accessibility of website, apache stopped logging to it's ssl and error logs, we don't see any abnormal behaviour in different running process data, memory usage, also don't see any suspicious request before issue occurred.

2 Answers 2

1

This is a common issue. CLOSE_WAIT connections are not being used but consuming apache resources.

I faced the Close wait connections problem some time ago. And after a lot of investigation here is the trick in case you need it.

You will need to add this to your Linux server

vi /etc/sysctl.conf

#Added to fix close_wait connections
#recommended 30. Default 60
net.ipv4.tcp_fin_timeout = 30
#in seconds, time in which the close_wait connections will be removed
net.ipv4.tcp_keepalive_time = 60
  
###duration of a request in seconds# probes * intvl = time to process if the connection is still alive
net.ipv4.tcp_keepalive_intvl = 2
#number of retries to test the connection
net.ipv4.tcp_keepalive_probes = 2

Also, after applying this you need to refresh the references to that file using sysctl -p command.

After that, you have to add the keepalive=On directive in your proxy pass.

ProxyPass /home ajp://localhost:8009/home keepalive=On retry=5

This will help to remove the close_wait connections after a minute of being in CLOSE_WAIT state. Apache uses threads in the worker mode, and those threads are handled by the OS, if the system is not able to close it, the system will do.

Remember to restart the whole apache process too.

PD: This helps if you are using ipv4. Not sure about ipv6

Hope it helps

0

It looks like a 'slow DoS attack' using tools like slowloris. In such type of attacks, the attacker tries to keep alive (open) legitimate connections to the server until the server hangs. We use SPA ( https://www.cyberxnetworks.com/index.php?cmd=website&aid=242 ) to stop such type of attack.

1
  • We have script to capture netstat every minute. I don't see too many connection before the issue happens. Number of connections used to be within normal range, and keeps rotating, don't see any idle connection for too long.
    – HarishS
    Jun 26, 2019 at 12:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.