Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Background:

I use a cluster with my coworkers and if anyone writes too many big files at the same time I can't even move the cursor in vim (the horror!). I'm not sure if it is more related to the number of files open or the "bandwith" of hard drive being used but it definitely only happens when a big write operation is taking place (including downloading big files). I know you can limit hard drive space by quotas per user, but I've never heard of being able to limit data transfer speeds.

Question

Is it possible to limit the amount of data written per second by a user in linux?

share|improve this question

migrated from stackoverflow.com Jan 19 '12 at 3:31

This question came from our site for professional and enthusiast programmers.

5  
gotta run, but look into ionice(1) –  sarnold Jan 19 '12 at 0:36
    
perhaps something like PAM, ulimit? linuxtopia.org/online_books/linux_administrators_security_guide/… –  sequenceGeek Jan 19 '12 at 0:47
    
For ionice info: friedcpu.wordpress.com/2007/07/17/… –  sequenceGeek Jan 19 '12 at 0:49
    
@sequenceGeek: you might want to look at the hard drive's SMART data and see if it has been having trouble. –  wallyk Jan 19 '12 at 0:58
    
@wallyk: thanks for the tip - I didn't know about SMART up until now. I think it solely has to do with the write speed, however. Networked cluster with NFS = problems :( –  sequenceGeek Jan 19 '12 at 2:29
show 2 more comments

3 Answers 3

up vote 3 down vote accepted

Use cgroups together with CFQ I/O scheduler (the default for many distributions). CFQ is aware of cgroups and can give any user, group or process only x% of disk I/O time.

So, if you have one cgroup called sequenceGeek, having 90% of maximum resources, you can then have another cgroup called coworkers, having 10% of resources. Or something similarly fair.

man cgrules.conf, man cgset and man cgconfig.conf should get you started. This at least in Fedora 16, did not have time to check other distributions right now if they have similar config file names. Probably they do.

EDIT: Oh, just noticed from your comments you are using NFS. Then your network might be saturated and you need to use QoS with tc and iptables. Google for Advances Linux Routing How-To, it has ready-made examples of throttling the traffic and creating priority classes. Or if possible, do the throttling at your firewall/router/load balancer.

Could also be an old kernel version having issues with NFS or bad NFS mount options.

share|improve this answer
add comment

I'm not entirely sure, but after reading a bit from man limits.conf, and assuming you're root, I'd suggest to add hard limits for other people's accounts on data (maximum data size), priority and nice. Something like setting their limits lower than yours, and then setting whichever app you're running's priority to higher than they can. Of course, you should research a bit more and test before making any changes, as it might seriously interfere with their user experience.

Good luck.

share|improve this answer
add comment

If you cannot move your cursor, my guess is that your CPU is waiting on IO for a very high amount of time. You can check this by running top, and looking at the amount listed at %wa near the top.

The blkio controller of the aforementioned cgroups is a method of controlling the amount of IO someone or some group can do. I'm in a bit of a hurry right now, but if you read the kernel docs (/usr/share/doc/kernel-<version>/Documentation/cgroups), I'm sure you'll manage. You will need a fairly recent kernel though, but I think anything above 2.6.32 (the EL6 kernel) will probably do.

You need to configure the blkio.throttle.write_bps_device file in your cgroups filesystem. With that, together with the /etc/cgconfig.conf and /etc/cgrules.conf, you can limit any user or group in the amount of bandwidth on your storage.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.