Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I am using rsync over ssh to backup data from windows to linux. This creates 1 rsync process on the server for each connection as this is over ssh. Each rsync connections runs at 1 mbps. Do you have an idea on how many of those connections I can have on a 2Ghz - 4G RAM machine before the CPU hits 100%. I couldn't find any good info online on that.

Laurent

share|improve this question

migrated from stackoverflow.com Aug 23 '09 at 6:44

This question came from our site for professional and enthusiast programmers.

2  
I'll leave the empirical testing as an exercise to the reader. –  dlamblin Aug 23 '09 at 6:44

5 Answers 5

Quite a lot if you can set up the clients to not rsync at the same time.

But I would not be nervous about the CPU/RAM, since I think that the bottleneck will be the bandwidth to the harddrives.

Anyhow follow the advice from sysadmin1138, and keep a eye open on the resources with a easy tool like gkrellm (that can show you cpu, ram, harddrives and network at the same time).

share|improve this answer

Rsync does not scale linearly, especially when they aren't started at the same time. The most intensive part is the initial bit where it reads the directories and sync's the file lists. The newest version of rsync side steps this so then you are only left with checksums. If CPU usage becomes an issue, you can disable this part too.

The SSH side of things generally won't use too much cpu even when you start pushing serious bandwidth. Its highly optimised and won't do very much. For record, I've push 45meg a second through a ssh encrypted tunnel without any of the cores on the box blinking. Load jumped up a little from the disk access of pushing the data on to the raid array.

share|improve this answer

I'd start doing some scale testing. Start one process, check its CPU and memory loads in taskmanager. Run four in parallel, and do the same for each. Presuming this scales linearly, you should get a fair idea as to when you'd start hitting CPU/RAM limits. If it doesn't scale linearly, you'll need to get more data points and do math.

share|improve this answer

The main bottle neck will most likely be your disks, not your CPUs. rsync is a tool for copying files. The fact that it does encryption aswell is a plus, but most of it's time will be spent reading disks.

The most important figures is not your CPU speed, but your disk speed. If you want more speed, you should use some sort of RAID.

share|improve this answer
    
RAID will only speed up your disk access if you use RAID0 (stripes). All other forms of RAID generally don't result in a speed gain. If you want faster disk access, you need faster disks, and make sure they are spread nicely across your disk adapters. –  wolfgangsz Aug 23 '09 at 14:14

As suggested by others, the best way to work this out is just to test it.

However, I don't believe you'll hit the limits of your CPU or RAM before the system becomes unusable because of the IO load you're placing on it.

I also don't believe that it will scale linearly, I would expect to reach a point where running an actually extra rsync made it slower.

If you really want to test something, I'd test how many you can run concurrently before it becomes slower than running them one by one. On normal sata disks I'd expect this number to be less than a dozen.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.