Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

One of my development servers (CentOS 6) has suddenly seen a huge spike in inbound network traffic

Router traffic graph (Blue is inbound)

It's slowing things down to a crawl, SSH is taking >10 seconds to log in and there is a delay when typing, websites are timing out, Nagios is upset because NRPE checks keep timing out (this is my Nagios host) so there seems to be a sudden huge storm of nework traffic, but I can't figure out where it's coming from. The server has a public IP so is directly accessible, it runs a very restrictive IPTables rule-set (Only allowing 80, 443 and a couple of other utility ports for things like Jenkins). I've tried using tools like iftop but they're not showing anything out of the ordinary. Not sure if this is because IPTables is blocking the connections so they're not showing up, but since I'm not sure whether these are external devices trying to connect to my server, or something else. It seems strange that it's making SSH slow and other services non-responsive but the network traffic started about the same time and I first started getting issues. Where do I look to figure out where this traffic is coming from and how to stop it? I don't have direct access to any of the routers but I can do whatever I want on the server. I looked in /var/log/messages and there are a lot of weird messages about DNS I've never seen before but they don't appear to be errors, just over-verbose logging (see below).

Standard useful stuff;

[sr@ns309372 ~]$ sudo uptime
 23:51:41 up  6:30,  3 users,  load average: 0.03, 0.12, 0.11
[sr@ns309372 ~]$ sudo free -m
             total       used       free     shared    buffers     cached
Mem:          3920       2197       1722          0        103       1060
-/+ buffers/cache:       1032       2887
Swap:         1019          0       1019
[sr@ns309372 ~]$ sudo tail -n 30 /var/log/messages
Apr 18 23:11:08 ns309372 named[2451]: success resolving 'ftp.halifax.rwth-aachen.de/A' (in 'rwth-aachen.de'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:11:10 ns309372 named[2451]: success resolving 'deneb.dfn.de/AAAA' (in 'dfn.de'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:11:10 ns309372 named[2451]: success resolving 'ns1.leaseweb.nl/AAAA' (in 'leaseweb.nl'?) after disabling EDNS
Apr 18 23:11:15 ns309372 named[2451]: success resolving 'ns4.leaseweb.net/AAAA' (in 'leaseweb.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:11:22 ns309372 named[2451]: success resolving 'pkg.jenkins-ci.org/A' (in 'jenkins-ci.org'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:11:30 ns309372 named[2451]: success resolving 'mirror.ovh.net/A' (in 'ovh.net'?) after disabling EDNS
Apr 18 23:11:30 ns309372 named[2451]: success resolving 'mirror.ovh.net/AAAA' (in 'ovh.net'?) after disabling EDNS
Apr 18 23:33:54 ns309372 named[2451]: success resolving 'vs1.nagios.org/A' (in 'nagios.org'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:34:36 ns309372 named[2451]: success resolving 'ns.ripe.net/A' (in 'ripe.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:34:36 ns309372 named[2451]: success resolving 'dns1.ntli.net/AAAA' (in 'ntli.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:34:37 ns309372 named[2451]: success resolving 'dns2.ntli.net/AAAA' (in 'ntli.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:34:37 ns309372 named[2451]: success resolving 'dns2.ntli.net/A' (in 'ntli.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:34:38 ns309372 named[2451]: success resolving 'sec1.apnic.net/AAAA' (in 'apnic.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:34:39 ns309372 named[2451]: success resolving 'sec3.apnic.net/AAAA' (in 'apnic.net'?) after disabling EDNS
Apr 18 23:34:40 ns309372 named[2451]: success resolving 'sec3.apnic.net/A' (in 'apnic.net'?) after disabling EDNS
Apr 18 23:34:40 ns309372 named[2451]: success resolving 'dns2.ntli.net/AAAA' (in 'ntli.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:35:02 ns309372 named[2451]: success resolving 'urlatron.com/AAAA' (in 'urlatron.com'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:35:03 ns309372 named[2451]: success resolving 'urlatron.com/A' (in 'urlatron.com'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:35:56 ns309372 named[2451]: success resolving 'bitbucket.org/A' (in 'bitbucket.org'?) after disabling EDNS
Apr 18 23:48:26 ns309372 named[2451]: success resolving '113.155.23.94.in-addr.arpa/PTR' (in '155.23.94.in-addr.arpa'?) after disabling EDNS
Apr 18 23:48:29 ns309372 named[2451]: success resolving '8.137.145.217.in-addr.arpa/PTR' (in '217.in-addr.arpa'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:48:29 ns309372 named[2451]: success resolving '10.169.216.196.in-addr.arpa/PTR' (in '169.216.196.in-addr.arpa'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:48:29 ns309372 named[2451]: success resolving 'ns2.lacnic.net/AAAA' (in 'lacnic.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:48:30 ns309372 named[2451]: success resolving 'ns2.dns.br/AAAA' (in 'br'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:48:30 ns309372 named[2451]: success resolving 'ns2.dns.br/A' (in 'br'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:48:34 ns309372 named[2451]: success resolving 'ns2.afrinic.net/A' (in 'afrinic.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:50:03 ns309372 named[2451]: success resolving 'urlatron.com/A' (in 'urlatron.com'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:50:04 ns309372 named[2451]: success resolving 'ns2.ecogeek.org/A' (in 'ecogeek.org'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:50:05 ns309372 named[2451]: success resolving 'ns1.ecogeek.org/AAAA' (in 'ecogeek.org'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:50:05 ns309372 named[2451]: success resolving 'urlatron.com/AAAA' (in 'urlatron.com'?) after reducing the advertised EDNS UDP packet size to 512 octets
[sr@ns309372 ~]$ sudo ifconfig
eth0      Link encap:Ethernet  HWaddr 00:27:0E:0B:86:51  
          inet addr:188.165.192.119  Bcast:188.165.192.255  Mask:255.255.255.0
          inet6 addr: fe80::227:eff:fe0b:8651/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:456082 errors:0 dropped:91 overruns:0 frame:0
          TX packets:821015 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:59793427 (57.0 MiB)  TX bytes:1008283171 (961.5 MiB)
          Interrupt:43 Base address:0xc000 

eth0:0    Link encap:Ethernet  HWaddr 00:27:0E:0B:86:51  
          inet addr:94.23.155.32  Bcast:94.23.155.32  Mask:255.255.255.255
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:43 Base address:0xc000 

eth0:1    Link encap:Ethernet  HWaddr 00:27:0E:0B:86:51  
          inet addr:94.23.155.113  Bcast:94.23.155.113  Mask:255.255.255.255
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:43 Base address:0xc000 

eth0:2    Link encap:Ethernet  HWaddr 00:27:0E:0B:86:51  
          inet addr:178.32.48.78  Bcast:178.32.48.78  Mask:255.255.255.255
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:43 Base address:0xc000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:169675 errors:0 dropped:0 overruns:0 frame:0
          TX packets:169675 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:172646550 (164.6 MiB)  TX bytes:172646550 (164.6 MiB)
[sr@ns309372 ~]$ sudo sar -n DEV 1 3
Linux 2.6.38.2-grsec-xxxx-grs-ipv6-64 (ns309372.ovh.net)    18/04/12    _x86_64_    (2 CPU)

23:57:35        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
23:57:36           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:36       dummy0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:36         eth0     13.00      8.00      1.11      5.08      0.00      0.00      0.00
23:57:36        tunl0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:36         sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:36      ip6tnl0      0.00      0.00      0.00      0.00      0.00      0.00      0.00

23:57:36        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
23:57:37           lo     10.00     10.00      2.92      2.92      0.00      0.00      0.00
23:57:37       dummy0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:37         eth0     11.00      8.00      0.91      3.47      0.00      0.00      0.00
23:57:37        tunl0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:37         sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:37      ip6tnl0      0.00      0.00      0.00      0.00      0.00      0.00      0.00

23:57:37        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
23:57:38           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:38       dummy0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:38         eth0      7.00      9.00      7.54      1.33      0.00      0.00      0.00
23:57:38        tunl0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:38         sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:38      ip6tnl0      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Average:        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
Average:           lo      3.33      3.33      0.97      0.97      0.00      0.00      0.00
Average:       dummy0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         eth0     10.33      8.33      3.19      3.30      0.00      0.00      0.00
Average:        tunl0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:      ip6tnl0      0.00      0.00      0.00      0.00      0.00      0.00      0.00

I have a number of 'virtual' interfaces to support multiple IPs, only one physical interface

share|improve this question

1 Answer 1

up vote 2 down vote accepted

I would do two things:

  1. Log traffic in iptables. You create a rule and use the LOG (uses kernel log system) and ULOG (directs logs to sockets instead of the kernel system) targets. So you would probably be interested in all -A INPUT packets being sent to -j LOG with perhaps --log-prefix "incoming packets " and --log-level 6 (it follows syslog levels)

  2. Monitor your processes to see if any are the recipient of the vast bandwidth. I suggest the use of nethogs. It may be that traffic is making it past iptables, but not being written to disk but instead being immediately discarded by some oddball process.

share|improve this answer
    
Left nethogs open for 5 mins, it doesn't seem to show anything out of the ordinary imgur.com/sFeSW IP tables is logging lots of dropped DNS packets (gist.github.com/2417610) The two IP addresses are owned by my ISP which at least gives me a point of contact. –  sam Apr 19 '12 at 0:50
    
@sam That was fast work - you are a thing of legend. –  Wesley Apr 19 '12 at 0:51
1  
It's 2AM, what else am I supposed to do? =) –  sam Apr 19 '12 at 0:53
    
So it was mostly my own fault, I used to use a secondary DNS service from my ISP, I rebuilt my server and started using Route53 for DNS so didn't install a DNS server but forgot to disable the secondary DNS service. I guess yesterday it suddenly decided to start sending huge numbers of queries to my server asking it for DNS records for the domains registered with the secondary DNS service. As soon as I turned SDNS off traffic dropped down to normal levels. –  sam Apr 19 '12 at 1:37

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.