2

I am using rsync over ssh to backup data from windows to linux. This creates 1 rsync process on the server for each connection as this is over ssh. Each rsync connections runs at 1 mbps. Do you have an idea on how many of those connections I can have on a 2Ghz - 4G RAM machine before the CPU hits 100%. I couldn't find any good info online on that.

Laurent

1
  • 2
    I'll leave the empirical testing as an exercise to the reader.
    – dlamblin
    Aug 23, 2009 at 6:44

5 Answers 5

2

Rsync does not scale linearly, especially when they aren't started at the same time. The most intensive part is the initial bit where it reads the directories and sync's the file lists. The newest version of rsync side steps this so then you are only left with checksums. If CPU usage becomes an issue, you can disable this part too.

The SSH side of things generally won't use too much cpu even when you start pushing serious bandwidth. Its highly optimised and won't do very much. For record, I've push 45meg a second through a ssh encrypted tunnel without any of the cores on the box blinking. Load jumped up a little from the disk access of pushing the data on to the raid array.

1

I'd start doing some scale testing. Start one process, check its CPU and memory loads in taskmanager. Run four in parallel, and do the same for each. Presuming this scales linearly, you should get a fair idea as to when you'd start hitting CPU/RAM limits. If it doesn't scale linearly, you'll need to get more data points and do math.

1

Quite a lot if you can set up the clients to not rsync at the same time.

But I would not be nervous about the CPU/RAM, since I think that the bottleneck will be the bandwidth to the harddrives.

Anyhow follow the advice from sysadmin1138, and keep a eye open on the resources with a easy tool like gkrellm (that can show you cpu, ram, harddrives and network at the same time).

1

The main bottle neck will most likely be your disks, not your CPUs. rsync is a tool for copying files. The fact that it does encryption aswell is a plus, but most of it's time will be spent reading disks.

The most important figures is not your CPU speed, but your disk speed. If you want more speed, you should use some sort of RAID.

1
  • RAID will only speed up your disk access if you use RAID0 (stripes). All other forms of RAID generally don't result in a speed gain. If you want faster disk access, you need faster disks, and make sure they are spread nicely across your disk adapters.
    – wolfgangsz
    Aug 23, 2009 at 14:14
0

As suggested by others, the best way to work this out is just to test it.

However, I don't believe you'll hit the limits of your CPU or RAM before the system becomes unusable because of the IO load you're placing on it.

I also don't believe that it will scale linearly, I would expect to reach a point where running an actually extra rsync made it slower.

If you really want to test something, I'd test how many you can run concurrently before it becomes slower than running them one by one. On normal sata disks I'd expect this number to be less than a dozen.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .