How find out what or who is using my bandwidth?

Question

To be honest, I'm poor at server administration, but my boss asked me for help. His server has a limited bandwidth of 2GB/day and today he received warning from company that he exceeded it and used... 24GB.

As it is impossible cause he was way, he asked me if I can track down the problem. I have no idea where to start or what to do.

Any information would be helpful how can I find out whats wrong.

Machine is running on Ubuntu 12.04. The most curious thing is that, according to diagrams received from hoster, only outgoing transfer was used.

EDIT

Thanks for suggestions, i'll run tcpdump and try to examine the result

Answer 1

For immediate monitoring you can use iftop. This will show you the currently active connections and the bandwidth they are using. Once you've identified a high traffic connection, find the local port number and use netstat to find which process the connection belongs to.

sudo netstat -tpn | grep 12345

For longer term monitoring I would suggest something like darkstat. This can give you a breakdown per host and port which might allow you to figure out what the traffic is related to.

Answer 2

Have a look at tcpdump. It can dump all network traffic (not only tcp as the name would suggest), which you can then read with an application like Wireshark. In Wireshark it's very easy to filter certain types of data and even plot graphs of the network I/O.

Another useful tool might be netstat which displays a list of ongoing network connections. Perhaps there are connections that shouldn't be there. Tcpdump is much more useful (capture a few minutes, then check if you can already see the source), but netstat might give you a quick overview.

Upon reading this by the way, my first thoughts are that you have malware on your server or that it is being used for amplification attacks. But to examine this you'll need to run tcpdump first.

Edit: Note that tcpdump probably needs to be run as root, perhaps you need to use sudo tcpdump.

Another edit: Since I can't really find a good webpage to link on what amplification attacks are in general, here's a short version:

Protocols like DNS run on UDP. UDP traffic is connectionless, and thus you can very easily spoof the IP address of someone else. Because a DNS answer is usually larger than the query, this can be used for a DoS attack. The attacker sends a query requesting all records the DNS server has on a given name, and tells the DNS server that the request originated from X. This X is the target that the attacker wants to DoS. The DNS server then kindly replies, sending the (big, say 4kB) reply to X.

This is amplification because the attacker sends less data than X actually receives. DNS is not the only protocol with which this is possible.

Answer 3

I'd recommend installing ntop.

http://www.ntop.org/

Put that on a host gateway/router location and watch traffic for a day/week. Ntop provides a web UI where you can get a breakdown by IP/port/protocol.

Answer 4

Well, a packet capture is typically the first place to start in situations like these. Ensure that tcpdump is installed ($ sudo apt-get install tcpdump), and then run the following:

$ sudo tcpdump -w packet.log

This will write a log of all packets to packet.log. Let that run for a few minutes, then download that file and inspect using Wireshark. If the mystery traffic is still happening, it should be quite obvious with a cursory glance through the packet capture data.

Answer 5

Capturing all the packets sent in a day when you're exceeding your bandwidth quota is probably not the most sensible approach - how are you going to get the data of the system for analysis?

What access do you have on the box? What ports are open? Have you checked the logs for the services which are running? Something like awstats will summarize FTP, HTTP and SMTP logs (assuming these servers are configured to record the data in logs). OTOH mrtg will record and track network usage by endpoint / port.

Answer 6

if you think you bandwidth hog is apache, I've had success with this tool in the past

apache top

http://www.howtogeek.com/?post_type=post&p=324

Answer 7

After searching for quite a while whats the problem (over 60GB bandwidth in few days) I discovered that my server was a DDOS attack source.

First of all, I tried to install Oracle DB on it, therefore I created oracle user. Hackers somehow managed to break pass for that user (I guess I should make it harder :( ), they made a hidden dir under Oracle home, with a crontab there, which manually runned some deamons which flooded target server.

Moreover, hackers created 2 new users on my server: avahi and colord. What should I do about them? I googled and seems that software with same name is not dangerous, but I deleted those users (and oracle too).

Futheremore I deleted whole oracle home, with everything in it.

I guess I need to secure my server more, as it may be attacked again, thanks everyone for help!