2

First I should say that my Linux knowledge is minimal; just enough to set up some servers (Apache, Tomcat, Couch, etc). I built a MiniITX server to host some simple sites, act as an SSH tunnel while I'm away, and act as a torrent server. It was not properly secured for a long time (iptables was empty, all ports open, no firewall) though my router did not have much port forwarding set up beyond HTTP, FTP, and SSH.

A week or two ago my bandwidth at home dropped from around 27Mbps to 2Mbps and my upload went from 7Mbps to 0.06Mbps. When I unplug the server from the LAN, by bandwidth shoots back up.

I threw up a restrictive iptables, removed most of the port forwarding, and checked my router logs to see if there were any open connections from the server (malware?) but there were none.

What would you do? What are the first things you'd check?

I can of course reinstall everything from scratch, but I'd like to find the root cause.

EDIT

Connected to LAN:

sudo route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.0.0     *               255.255.255.0   U     0      0        0 eth0
default         192.168.0.1     0.0.0.0         UG    100    0        0 eth0

iptraf > IP traffic monitor > eth0 alt text

which shows me x103 (the problem server) and x130 (the Mac from which I'm SSHing in) with Packets and Bytes just flying up at a constant rate, never stopping. I'm guessing that this is an infinite feedback loop, where any iptraf update needs to be sent over the wire, resulting in another update being logged, etc. Anyway, it's showing a TCP flow rate of 26 kbits/s which simply cannot account for the multi-Mbps drop in both upload and download.

iptraf > Detailed interface statistics > eth0

alt text

iptraf > Statistical breakdowns > By TCP/UDP port > eth0

alt text

Improve this question

6 Answers 6

Reset to default
3

TCPDUMP can show you the traffic on the wire. This can get messy, so you may want to redirect it to a file and browse it later:
tcpdump -s 0 -Ai eth0

-s 0 sets the length of the packet capture, 0 means as much as possible so feel free to adjust as needed.
-A prints the traffic in ASCII so you can read some of it. i eth0 sets the interface to watch. You should make sure this is your WAN interface.

With any luck you'll see what's generating your bandwidth problems.

Improve this answer
1
  • Well, it sounds like the box is NAT'ed behind a router, so he wants the LAN interface, not WAN interface (although it's likely eth0 that he wants anyways).
    – gravyface
    Dec 30, 2010 at 21:29
3

I would unplug it immediateley. Maybe you are acting as a spam relay or something worse. Then check locally.

It would also be better to boot from a live linux (like knoppix or ubuntu live or your preferred brand), do not boot it from the hard disk any more.

It is not easy to diagnose without further knowledge, but most probably someone has taken over this machine and is using it for something. Could also be a botnet or be used for dos attacks.

I would suggest to isolate this machine from everything else. And maybe other machines could also be infected.

Make a backup of your data. Use security software, like virus checkers. Learn how to build a protected environment (machines, lan, firewall, router) and start fresh. Be careful when using your data from these machines, could be infected.

There is a potential risk in such setups that you will be held responsible for forbidden or criminal activity that has been done by someone else using your machine and IP, also your internet service provider does not like what happened (if this really happened).

Do not feel guilty as this happens even to many experienced admins but try to act responsible and also learn from it :) Good luck!

Improve this answer
4
  • I agree and it's been unplugged since I first discovered it. When I plug it back into my LAN I'll unplug the WAN and use my isolated 3G connection to assist in the troubleshooting or reinstalling.
    – rcampbell
    Dec 30, 2010 at 20:47
  • 1
    Nuke it from orbit. It's the only way to be sure.
    – Tom O'Connor
    Dec 31, 2010 at 9:59
  • Do you have any recommendation for a good nix FOSS virus checker?
    – rcampbell
    Dec 31, 2010 at 10:02
  • I don't know about good, but there aren't that many available. ClamAV for actual viruses and chkrootkit and rkhunter for checking for rootkits.
    – ptman
    Dec 31, 2010 at 10:05
1

tcpdump as others have suggested, although if you suspect that the box may be rooted, I'd want to capture traffic from another trusted machine that's running as a bridge between the box in question and the LAN switch (built-in to the router or otherwise) or if you have an old hub, you can use that too.

You mention that it's a torrent server: are you by chance seeding a pile of torrents? That'll quickly kill an Internet connection if there's too many peers and/or too much bandwidth consumed.

Improve this answer
1
  • There hasn't been any torrent activity for a couple of weeks, with the last seeding/downloading taking place before this problem began. I'm running TorrentFlux.
    – rcampbell
    Dec 31, 2010 at 9:39
1

If you're willing to risk running it live again (see mit's answers), you can get an immediate sense for what's going on by running iptraf.

tcpdump will give you a more definite answer (easier to understand in wireshark)

Improve this answer
1
  • I ran iptraf on both lo and eth0 and it's showing a TCP flow rate of only 26 kbits/s (see attached screen shot). Maybe I'm reading this wrong, but it's showing almost no traffic, yet my bandwidth dropped as expected to the same levels I've seen before. Could it somehow be the server interfering with the router? I do see some UDP packets being sent from the router (x1) to the server (x130)...
    – rcampbell
    Dec 31, 2010 at 10:01
1

After running the tests given in the various answers, it became clear that my Linux box wasn't actually transmitting much of anything. This left me with the router - a D-Link DIR-655 A3 - as the problem. It's off the shelf (not customized) and it's worked fine for years. I spend some time playing with my Windows Media Center PC which is also connected to the router. I noticed the bandwidth drop happen again - with the server unplugged - once I started a torrent client. I know it isn't my ISP throttling the bandwidth when it notices torrents, because when I connect directly to the modem the problem disappears.

I think the problem is my router's QoS Engine or WISH (Wireless Intelligent Stream Handling) capabilities going crazy. Both of these services shape traffic, both LAN and WAN traffic. I disabled both services and so far the problem seems to have gone away.

I want to thank everyone who posted a suggestion; these tips helped me discover that the problem wasn't actually with my server.

Improve this answer
1

What is the output of sudo route before and after the LAN is plugged?

EDIT: Eth0 is you're LAN, where is the interface for the WAN - can you post the output of ifconfig -a and route (again) when both networks are operational?

Don't worry about viruses, backdoors, or spam forwarding - so far I don't see a good reason that you have it. Lets just diagnose this as a networking problem.

The iptraf you are showing adds up only to about 60K bits/sec - that's not a lot of traffic. You were talking about a bandwidth dropping from 27M to 2M bits/sec. Please reproduce the problem (plug-in the public interface, etc.) and then run iptraf, ifconfig -a, and route.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .