21

To be honest, I'm poor at server administration, but my boss asked me for help. His server has a limited bandwidth of 2GB/day and today he received warning from company that he exceeded it and used... 24GB.

As it is impossible cause he was way, he asked me if I can track down the problem. I have no idea where to start or what to do.

Any information would be helpful how can I find out whats wrong.

Machine is running on Ubuntu 12.04. The most curious thing is that, according to diagrams received from hoster, only outgoing transfer was used.

EDIT

Thanks for suggestions, i'll run tcpdump and try to examine the result

4
  • Please post your comments as a comment, not in the body of your question.
    – EEAA
    Jul 15, 2012 at 22:53
  • Here's my question, can I run tcpdump in background, so it would collect data from lets say a day?
    – kamil
    Jul 15, 2012 at 22:55
  • You can, yes. You'd need to run it from within a screen session or something like that. Beware, though...you need to make sure that you have enough disk to store all of the captured data. You probably ought to read the tcpdump man page and see how to configure it to only capture the TCP/IP headers, discarding the rest of the packet data.
    – EEAA
    Jul 15, 2012 at 22:57
  • serverfault.com/a/160525/59925
    – quanta
    Aug 2, 2012 at 10:02

8 Answers 8

21

For immediate monitoring you can use iftop. This will show you the currently active connections and the bandwidth they are using. Once you've identified a high traffic connection, find the local port number and use netstat to find which process the connection belongs to.

sudo netstat -tpn | grep 12345

For longer term monitoring I would suggest something like darkstat. This can give you a breakdown per host and port which might allow you to figure out what the traffic is related to.

2
  • 1
    iotop? Or were you thinking of iftop?
    – EEAA
    Jul 15, 2012 at 23:03
  • @ErikA Uh, yes.. that's what I said!
    – mgorven
    Jul 15, 2012 at 23:04
9

I'd recommend installing ntop.

http://www.ntop.org/

Put that on a host gateway/router location and watch traffic for a day/week. Ntop provides a web UI where you can get a breakdown by IP/port/protocol.

6

The best tool for this is probably iftop, and easily apt-get'able via sudo apt-get install iftop. It will display output according to the culprit IP's/hostnames:

             191Mb      381Mb                 572Mb       763Mb             954Mb
└────────────┴──────────┴─────────────────────┴───────────┴──────────────────────
box4.local            => box-2.local                      91.0Mb  27.0Mb  15.1Mb
                      <=                                  1.59Mb   761kb   452kb
box4.local            => box.local                         560b   26.8kb  27.7kb
                      <=                                   880b   31.3kb  32.1kb
box4.local            => userify.com                         0b   11.4kb  8.01kb
                      <=                                  1.17kb  2.39kb  1.75kb
box4.local            => b.resolvers.Level3.net              0b     58b    168b
                      <=                                     0b     83b    288b
box4.local            => stackoverflow.com                   0b     42b     21b
                      <=                                     0b     42b     21b
box4.local            => 224.0.0.251                         0b      0b    179b
                      <=                                     0b      0b      0b
224.0.0.251           => box-2.local                         0b      0b      0b
                      <=                                     0b      0b     36b
224.0.0.251           => box.local                           0b      0b      0b
                      <=                                     0b      0b     35b


─────────────────────────────────────────────────────────────────────────────────
TX:           cum:   37.9MB   peak:   91.0Mb     rates:   91.0Mb  27.1Mb  15.2Mb
RX:                  1.19MB           1.89Mb              1.59Mb   795kb   486kb
TOTAL:               39.1MB           92.6Mb              92.6Mb  27.9Mb  15.6Mb

Don't forget about the classic and powerful sar and netstat utilities on older *nix!

Another great tool is nload, a great tool for monitoring bandwidth in real time and easily installed in Ubuntu or Debian with sudo apt-get install nload.

Device eth0 [10.10.10.5] (1/2):
=====================================================================================
Incoming:


                               .         ...|
                               #         ####|
                           .. |#|  ...   #####.         ..          Curr: 2.07 MBit/s
                          ###.###  #### #######|.     . ##      |   Avg: 1.41 MBit/s
                         ########|#########################.   ###  Min: 1.12 kBit/s
             ........    ###################################  .###  Max: 4.49 MBit/s
           .##########. |###################################|#####  Ttl: 1.94 GByte
Outgoing:
            ##########  ###########    ###########################
            ##########  ###########    ###########################
            ##########. ###########   .###########################
            ########### ###########  #############################
            ########### ###########..#############################
           ############ ##########################################
           ############ ##########################################
           ############ ##########################################  Curr: 63.88 MBit/s
           ############ ##########################################  Avg: 32.04 MBit/s
           ############ ##########################################  Min: 0.00 Bit/s
           ############ ##########################################  Max: 93.23 MBit/s
         ############## ##########################################  Ttl: 2.49 GByte
3

Well, a packet capture is typically the first place to start in situations like these. Ensure that tcpdump is installed ($ sudo apt-get install tcpdump), and then run the following:

$ sudo tcpdump -w packet.log

This will write a log of all packets to packet.log. Let that run for a few minutes, then download that file and inspect using Wireshark. If the mystery traffic is still happening, it should be quite obvious with a cursory glance through the packet capture data.

1
  • Using wireshark use Statistics menu - Endpoints. You can then select IP or TCP or other lists and sort by amount received/transmitted by each endpoint. Where endpoint will be IP address or machine. But some endpoints might be gateway or switch so would have to do another tcpdump and wireshark on that gateway to track down bandwidth user.
    – gaoithe
    Aug 12, 2015 at 16:02
3

Have a look at tcpdump. It can dump all network traffic (not only tcp as the name would suggest), which you can then read with an application like Wireshark. In Wireshark it's very easy to filter certain types of data and even plot graphs of the network I/O.

Another useful tool might be netstat which displays a list of ongoing network connections. Perhaps there are connections that shouldn't be there. Tcpdump is much more useful (capture a few minutes, then check if you can already see the source), but netstat might give you a quick overview.

Upon reading this by the way, my first thoughts are that you have malware on your server or that it is being used for amplification attacks. But to examine this you'll need to run tcpdump first.

Edit: Note that tcpdump probably needs to be run as root, perhaps you need to use sudo tcpdump.

Another edit: Since I can't really find a good webpage to link on what amplification attacks are in general, here's a short version:

Protocols like DNS run on UDP. UDP traffic is connectionless, and thus you can very easily spoof the IP address of someone else. Because a DNS answer is usually larger than the query, this can be used for a DoS attack. The attacker sends a query requesting all records the DNS server has on a given name, and tells the DNS server that the request originated from X. This X is the target that the attacker wants to DoS. The DNS server then kindly replies, sending the (big, say 4kB) reply to X.

This is amplification because the attacker sends less data than X actually receives. DNS is not the only protocol with which this is possible.

1
  • 1
    Yes. tcpdump -i any -w /tmp/traffic.pcap. Using wireshark use Statistics menu - Endpoints. Sort by amount received/transmitted.
    – gaoithe
    Aug 12, 2015 at 16:03
2

After searching for quite a while whats the problem (over 60GB bandwidth in few days) I discovered that my server was a DDOS attack source.

First of all, I tried to install Oracle DB on it, therefore I created oracle user. Hackers somehow managed to break pass for that user (I guess I should make it harder :( ), they made a hidden dir under Oracle home, with a crontab there, which manually runned some deamons which flooded target server.

Moreover, hackers created 2 new users on my server: avahi and colord. What should I do about them? I googled and seems that software with same name is not dangerous, but I deleted those users (and oracle too).

Futheremore I deleted whole oracle home, with everything in it.

I guess I need to secure my server more, as it may be attacked again, thanks everyone for help!

3
  • 5
    i'd wipe your server and start again, i never trust a server after a compromise. nuke the site from orbit, it's the only way to be sure. Jul 17, 2012 at 8:27
  • 1
    Yupp I'm going to do that, firstly backup of some data (svn repository, usefull scripts)
    – kamil
    Jul 17, 2012 at 8:32
  • 1
    make sure that data still has integrity, it's been know for intruders to modify source code and scripts to place back doors. I'd certainly compare the latest svn source with a recently checked out version before the intrusion got in. Jul 17, 2012 at 8:52
0

Capturing all the packets sent in a day when you're exceeding your bandwidth quota is probably not the most sensible approach - how are you going to get the data of the system for analysis?

What access do you have on the box? What ports are open? Have you checked the logs for the services which are running? Something like awstats will summarize FTP, HTTP and SMTP logs (assuming these servers are configured to record the data in logs). OTOH mrtg will record and track network usage by endpoint / port.

0

if you think you bandwidth hog is apache, I've had success with this tool in the past

apache top

http://www.howtogeek.com/?post_type=post&p=324

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .