Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

To be honest, I'm poor at server administration, but my boss asked me for help. His server has a limited bandwidth of 2GB/day and today he received warning from company that he exceeded it and used... 24GB.

As it is impossible cause he was way, he asked me if I can track down the problem. I have no idea where to start or what to do.

Any information would be helpful how can I find out whats wrong.

Machine is running on Ubuntu 12.04. The most curious thing is that, according to diagrams received from hoster, only outgoing transfer was used.

EDIT

Thanks for suggestions, i'll run tcpdump and try to examine the result

share|improve this question
    
Please post your comments as a comment, not in the body of your question. –  EEAA Jul 15 '12 at 22:53
    
Here's my question, can I run tcpdump in background, so it would collect data from lets say a day? –  kamil Jul 15 '12 at 22:55
    
You can, yes. You'd need to run it from within a screen session or something like that. Beware, though...you need to make sure that you have enough disk to store all of the captured data. You probably ought to read the tcpdump man page and see how to configure it to only capture the TCP/IP headers, discarding the rest of the packet data. –  EEAA Jul 15 '12 at 22:57
    
serverfault.com/a/160525/59925 –  quanta Aug 2 '12 at 10:02
add comment

8 Answers

For immediate monitoring you can use iftop. This will show you the currently active connections and the bandwidth they are using. Once you've identified a high traffic connection, find the local port number and use netstat to find which process the connection belongs to.

sudo netstat -tpn | grep 12345

For longer term monitoring I would suggest something like darkstat. This can give you a breakdown per host and port which might allow you to figure out what the traffic is related to.

share|improve this answer
    
iotop? Or were you thinking of iftop? –  EEAA Jul 15 '12 at 23:03
    
@ErikA Uh, yes.. that's what I said! –  mgorven Jul 15 '12 at 23:04
add comment

I'd recommend installing ntop.

http://www.ntop.org/

Put that on a host gateway/router location and watch traffic for a day/week. Ntop provides a web UI where you can get a breakdown by IP/port/protocol.

share|improve this answer
add comment

Have a look at tcpdump. It can dump all network traffic (not only tcp as the name would suggest), which you can then read with an application like Wireshark. In Wireshark it's very easy to filter certain types of data and even plot graphs of the network I/O.

Another useful tool might be netstat which displays a list of ongoing network connections. Perhaps there are connections that shouldn't be there. Tcpdump is much more useful (capture a few minutes, then check if you can already see the source), but netstat might give you a quick overview.

Upon reading this by the way, my first thoughts are that you have malware on your server or that it is being used for amplification attacks. But to examine this you'll need to run tcpdump first.

Edit: Note that tcpdump probably needs to be run as root, perhaps you need to use sudo tcpdump.

Another edit: Since I can't really find a good webpage to link on what amplification attacks are in general, here's a short version:

Protocols like DNS run on UDP. UDP traffic is connectionless, and thus you can very easily spoof the IP address of someone else. Because a DNS answer is usually larger than the query, this can be used for a DoS attack. The attacker sends a query requesting all records the DNS server has on a given name, and tells the DNS server that the request originated from X. This X is the target that the attacker wants to DoS. The DNS server then kindly replies, sending the (big, say 4kB) reply to X.

This is amplification because the attacker sends less data than X actually receives. DNS is not the only protocol with which this is possible.

share|improve this answer
add comment

Well, a packet capture is typically the first place to start in situations like these. Ensure that tcpdump is installed ($ sudo apt-get install tcpdump), and then run the following:

$ sudo tcpdump -w packet.log

This will write a log of all packets to packet.log. Let that run for a few minutes, then download that file and inspect using Wireshark. If the mystery traffic is still happening, it should be quite obvious with a cursory glance through the packet capture data.

share|improve this answer
add comment

Capturing all the packets sent in a day when you're exceeding your bandwidth quota is probably not the most sensible approach - how are you going to get the data of the system for analysis?

What access do you have on the box? What ports are open? Have you checked the logs for the services which are running? Something like awstats will summarize FTP, HTTP and SMTP logs (assuming these servers are configured to record the data in logs). OTOH mrtg will record and track network usage by endpoint / port.

share|improve this answer
add comment

if you think you bandwidth hog is apache, I've had success with this tool in the past

apache top

http://www.howtogeek.com/?post_type=post&p=324

share|improve this answer
add comment
up vote 0 down vote accepted

After searching for quite a while whats the problem (over 60GB bandwidth in few days) I discovered that my server was a DDOS attack source.

First of all, I tried to install Oracle DB on it, therefore I created oracle user. Hackers somehow managed to break pass for that user (I guess I should make it harder :( ), they made a hidden dir under Oracle home, with a crontab there, which manually runned some deamons which flooded target server.

Moreover, hackers created 2 new users on my server: avahi and colord. What should I do about them? I googled and seems that software with same name is not dangerous, but I deleted those users (and oracle too).

Futheremore I deleted whole oracle home, with everything in it.

I guess I need to secure my server more, as it may be attacked again, thanks everyone for help!

share|improve this answer
1  
i'd wipe your server and start again, i never trust a server after a compromise. nuke the site from orbit, it's the only way to be sure. –  The Unix Janitor Jul 17 '12 at 8:27
    
Yupp I'm going to do that, firstly backup of some data (svn repository, usefull scripts) –  kamil Jul 17 '12 at 8:32
    
make sure that data still has integrity, it's been know for intruders to modify source code and scripts to place back doors. I'd certainly compare the latest svn source with a recently checked out version before the intrusion got in. –  The Unix Janitor Jul 17 '12 at 8:52
add comment

The best tool for this is probably iftop, and easily apt-get'able via sudo apt-get install iftop. It will display output according to the culprit IP's/hostnames:

             191Mb      381Mb                 572Mb       763Mb             954Mb
└────────────┴──────────┴─────────────────────┴───────────┴──────────────────────
box4.local            => box-2.local                      91.0Mb  27.0Mb  15.1Mb
                      <=                                  1.59Mb   761kb   452kb
box4.local            => box.local                         560b   26.8kb  27.7kb
                      <=                                   880b   31.3kb  32.1kb
box4.local            => userify.com                         0b   11.4kb  8.01kb
                      <=                                  1.17kb  2.39kb  1.75kb
box4.local            => b.resolvers.Level3.net              0b     58b    168b
                      <=                                     0b     83b    288b
box4.local            => stackoverflow.com                   0b     42b     21b
                      <=                                     0b     42b     21b
box4.local            => 224.0.0.251                         0b      0b    179b
                      <=                                     0b      0b      0b
224.0.0.251           => box-2.local                         0b      0b      0b
                      <=                                     0b      0b     36b
224.0.0.251           => box.local                           0b      0b      0b
                      <=                                     0b      0b     35b


─────────────────────────────────────────────────────────────────────────────────
TX:           cum:   37.9MB   peak:   91.0Mb     rates:   91.0Mb  27.1Mb  15.2Mb
RX:                  1.19MB           1.89Mb              1.59Mb   795kb   486kb
TOTAL:               39.1MB           92.6Mb              92.6Mb  27.9Mb  15.6Mb

Don't forget about the classic and powerful sar and netstat utilities on older *nix!

Another great tool is nload, a great tool for monitoring bandwidth in real time and easily installed in Ubuntu or Debian with sudo apt-get install nload.

Device eth0 [10.10.10.5] (1/2):
=====================================================================================
Incoming:


                               .         ...|
                               #         ####|
                           .. |#|  ...   #####.         ..          Curr: 2.07 MBit/s
                          ###.###  #### #######|.     . ##      |   Avg: 1.41 MBit/s
                         ########|#########################.   ###  Min: 1.12 kBit/s
             ........    ###################################  .###  Max: 4.49 MBit/s
           .##########. |###################################|#####  Ttl: 1.94 GByte
Outgoing:
            ##########  ###########    ###########################
            ##########  ###########    ###########################
            ##########. ###########   .###########################
            ########### ###########  #############################
            ########### ###########..#############################
           ############ ##########################################
           ############ ##########################################
           ############ ##########################################  Curr: 63.88 MBit/s
           ############ ##########################################  Avg: 32.04 MBit/s
           ############ ##########################################  Min: 0.00 Bit/s
           ############ ##########################################  Max: 93.23 MBit/s
         ############## ##########################################  Ttl: 2.49 GByte
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.