Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm pushing a moderate amount of traffic (about 20 req/s) to a simple PHP API running on an Apache server. The Apache server slumbers (mod_status barely registers one active request), however, I'm seeing "Connection Refused" errors happening periodically despite this. I tried winding up uname -n and MinSpareServers, but to no effect.

I ran a diagnostic script like this while under load, and I'd see about 1 in 10 requests failing with "Connection refused"

while [ 1 ]; do echo "" | telnet <server> 80; sleep 1; done

However, when running a similar request on the local machine, it was fine.

while [ 1 ]; do echo "" | telnet localhost 80; sleep 1; done

Initially I thought it might be that all the requests are coming from the same client, and the server was being throttled. However, while under load, all other hosts that try and connect will see the same intermittent problem.

The fact that localhost behaves differently suggests to me that it's a networking, rather than apache, problem, but I'm not sure where to diagnose this. Is there logging that I can inspect/enable to see if the connection refusal is happening at a O/S (or possibly iptables) level.

Relevant software:

  • Ubuntu Lucid, 2.6.33 kernel
  • Apache 2.2.14

UPDATE:

In our iptables config I can see the following rule under "Chain TCPACCEPT":

655K 34M ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 limit: avg 12/sec burst 24

It seems like this rate limit could definitely be causing some these "Connection Refused" errors. I can disable the firewall for a short time to see if this resolves the issue, but I assume this rule is there to help cope with DOS attacks. The current limit seems too small for both avg and burst. What are more sensible limits (I can adjust these in our iptables config)?

share|improve this question

1 Answer 1

up vote 1 down vote accepted

If it's happening periodically, it's most likely not iptables but to view your iptables rules, run

sudo iptables -L -n -v

Your iptables logs will be under /var/log but it can vary depending on your configuration. Check under the /etc/rsyslog.d directory to see if there's a configuration file for your iptables log location.

You could also use netstat or lsof to see what's happening on the port also.

netstat -an | grep "80.*LIST"

lsof -i :80

Finally, you may want to take a network capture to see if there are TCP RSTs being sent back or there's simply no response coming back - use tcpdump for that.

Update to Question Update:

What happens if you disable that iptables rule? It does a couple of things but one of them appears to be restricting tcp segments that only have the syn packet set to 12/sec average with a burst limit. If you're sending an average of 20/sec, I believe that you're exceeding the limits set in the iptables rule.

Therefore, you could either turn off this rule on the firewall to test (I don't recommend turning off your firewall completely) or you could modify it to say 22/sec with a burst of 36.

I would strongly think about modifying configuration that you're not familiar with or are unsure as to why it's there, before modifying.

share|improve this answer
    
Thanks for the reply and sorry for the delay. I'll update my question based on what I can see in our iptables config. –  Ross May 24 '12 at 0:56
    
I replied and likewise, sorry for the delay! –  Mark Hillick Jun 26 '12 at 21:03
    
Thanks for your help Mark! I've doubled the limits in the mean time and we've not experienced the issues since. I'll keep an eye on it though. –  Ross Jul 24 '12 at 3:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.