Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm having a major problem with my server expending excessive amounts of outgoing bandwidth for nothing. The server OS is CentOS 6.4 x64 with the 2.6.32-431 kernel if that matters.

Here is a quick tcpdump log file:

20:10:17.448636 IP 76.200.77.222.broad.pt.fj.dynamic.163data.com.cn.57827 > svgmain.http: Flags [.], ack 463681, win 65520, options [nop,nop,sack 1 {460801:462241}], length 0
20:10:17.448698 IP svgmain.http > 76.200.77.222.broad.pt.fj.dynamic.163data.com.cn.57827: Flags [.], seq 468001:469441, ack 0, win 123, length 1440
20:10:17.454074 IP svgmain.http > 76.200.77.222.broad.pt.fj.dynamic.163data.com.cn.57827: Flags [.], seq 469441:470881, ack 0, win 123, length 1440
20:10:17.637167 IP 76.200.77.222.broad.pt.fj.dynamic.163data.com.cn.57827 > svgmain.http: Flags [.], ack 465121, win 65520, options [nop,nop,sack 1 {466561:468001}], length 0
20:10:17.637221 IP svgmain.http > 76.200.77.222.broad.pt.fj.dynamic.163data.com.cn.57827: Flags [.], seq 470881:472321, ack 0, win 123, length 1440
20:10:17.637230 IP svgmain.http > 76.200.77.222.broad.pt.fj.dynamic.163data.com.cn.57827: Flags [.], seq 472321:475201, ack 0, win 123, length 2880
20:10:17.638062 IP 76.200.77.222.broad.pt.fj.dynamic.163data.com.cn.57827 > svgmain.http: Flags [.], ack 468001, win 65520, length 0
20:10:17.638078 IP svgmain.http > 76.200.77.222.broad.pt.fj.dynamic.163data.com.cn.57827: Flags [.], seq 475201:478081, ack 0, win 123, length 2880
20:10:17.642977 IP 76.200.77.222.broad.pt.fj.dynamic.163data.com.cn.57827 > svgmain.http: Flags [.], ack 470881, win 65520, length 0
20:10:17.642988 IP svgmain.http > 76.200.77.222.broad.pt.fj.dynamic.163data.com.cn.57827: Flags [P.], seq 478081:480961, ack 0, win 123, length 2880

There are thousands more entries and it is eating up 100GB or more per day. I've tried a number of different firewall rules. My iptables rules are:

-A INPUT -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m state --state NEW -j DROP
-A INPUT -p tcp -m tcp --dport 80 -j fail2ban-HTTP
-A INPUT -p tcp -m tcp --dport 22 -j fail2ban-SSH
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -s 198.101.197.171/32 -p tcp -m tcp --dport 3306 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 443 -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m limit --limit 2/sec --limit-burst 2 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -o eth0 -p tcp -m tcp --sport 80 -m state --state ESTABLISHED -j ACCEPT
-A OUTPUT -o eth0 -p tcp -m tcp --sport 443 -m state --state ESTABLISHED -j ACCEPT
-A fail2ban-HTTP -j RETURN
-A fail2ban-SSH -j RETURN

I'm an amateur sysadmin so I'm not sure about all of the ins and outs of preventing these sorts of attacks.

iftop shows arbitrary IPs that change as the receivers of the data, nethogs shows /usr/bin/httpd as the process taking the bandwidth. iotop shows no activity though telling me that no actual files are being read.

Thoughts?

share|improve this question

closed as off-topic by MadHatter, Andrew, Ward, mdpc, John Dec 18 '13 at 15:32

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions must be relevant to professional system administration. Server Fault is dedicated to professional system and network administrators. End user and enthusiast questions are off-topic (contact your system administrator or hire a professional to help you out). Please see the Help Center for more information." – MadHatter, Ward, mdpc, John
If this question can be reworded to fit the rules in the help center, please edit the question.

1  
And what's in your web server log files? –  MadHatter Dec 18 '13 at 2:52
4  
What's the "attack"? You've got firewall rules permitting new inbound TCP connections on port 80 and you're running an httpd process. You're running a web server and other hosts are accessing it. If you don't mean to be running a web server stop the process and close the port. –  Evan Anderson Dec 18 '13 at 2:56
    
looks like that one is from China, you could look for patterns in the Class C's of the ip's hitting you and consider dropping all inbound requests from those netblocks. –  user16081 Dec 18 '13 at 3:12
    
I definitely do want httpd running. However, I can chmod 000 all of the folders in my webroot and the traffic will keep marching on. As well, there is nothing in my access_log file. It seems like whatever is happening is originating at port 80 but not in the sense that it is an HTTP request. –  eBrian Dec 18 '13 at 3:13
    
… assuming of course that traffic from China is abnormal. –  MikeyB Dec 18 '13 at 3:14

1 Answer 1

up vote 7 down vote accepted

OK, you have:

  • lots of traffic being sent from your HTTP port
  • nethogs shows that your httpd daemon is sending it
  • the destination IP is assorted computers on the internet
  • iotop isn't showing anything (all that means is it's cached in memory)

My diagnosis: You're running a web server.

Check your web server access logs to see what people are accessing. Is it normal traffic? Consider using a CDN to reduce your outgoing bandwidth.


Oh, nothing in the logs?

  • Are you sure httpd is logging where you think it's logging?
  • Try killing httpd, capturing traffic, then restarting httpd so you can catch the start of the request.
  • If they're downloading huge files, the logs may? not appear until the download is completed.
  • Using apache? Try enabling the server-status module so you can ask httpd what it's doing.
share|improve this answer
    
I do get regular web traffic to the access_log, that is why I'm convinced that it's not necessarily a file. Also because if I start Apache, I don't even have to have accessible vhosts for this to continue on. Let me try your suggestions. –  eBrian Dec 18 '13 at 3:19
    
Is your apache version vulnerable to the range header DoS attack? –  MikeyB Dec 18 '13 at 3:21
    
OK, so, it did end up being a file which server-status lead me to discover. A vulnerable comments page had about 120,000 spam comments taking about 40 seconds to load and 60MB later. I missed this the first time because access_log showed nothing out of the ordinary, which is to say it only had a request or two here and there. However, that was more than enough to saturate port 80. I learned a lot and my problem is resolved. Thanks MikeyB. –  eBrian Dec 18 '13 at 4:32

Not the answer you're looking for? Browse other questions tagged or ask your own question.