Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have a production server running different process which take advantage of existing linux buffers to improve their performance, such as a mysql server. This server has exhaustive IO load.

On this server we plan to run a specific process which deal with huge archives (>10GB). Whenever we run this process, overall performance is affected. I suppose linux buffers are emptied and refilled with this process, so mysql server and other processes cannot use caches pages as usual.

No matter if we reduce io load by ionice. The performance is badly affected

This server has a smart array controller with 4 raid 1/0 groups with 4 disks each. The disk group used by this process is different of disks groups used by other processes, so I don't think there is some kind of IO bw shortage.

I guess if I could limit how linux use IO buffers of an existing running process instead of all OS policy we could avoid this situation.

Of course I could execute the process on a separate server, but I wonder if my guess is possible or even logical.

share|improve this question

migrated from stackoverflow.com Aug 31 '12 at 6:28

This question came from our site for professional and enthusiast programmers.

    
What exactly is the problem? If a process has huge IO, then it is normal to consume huge buffers. The way IO works, everything you write goes into buffers and is lazily written to disk. You can tune the parameters in /proc/sys/vm so writeback occurs sooner (that way less is in buffers), but I would assume this probably rather decreases performance. --- either way, if you really must limit one process, LXC is the easiest and most lightweight thing to use. –  Damon Aug 30 '12 at 20:43
    
Surely the problem isn't the buffer usage per se, it's that all that I/O activity is throwing out the existing cached data the other applications rely on for performance. LXC won't help with that, as the I/O will still be buffered by the underlying kernel. –  Andy Ross Aug 30 '12 at 22:20
    
If you say buffers you talk about cache? –  Nils Jul 27 '13 at 21:38

1 Answer 1

Why not just run it in a VM? In qemu/kvm, for example, simply use a -drive option pointing to the relevant block device with a cache=none option set. That forces the VM to do direct I/O to the device without polluting the host cache. Then you can simply modulate the amount of memory the VM sees to control the total amount of disk caching it is allowed to do.

share|improve this answer
    
I agree there's probably no way to change how linux throws out existing cached pages belonging to an specific process. Obviously linux can be tuned in a general way. However the idea of run that process on a vm is good and enough. Thanks. –  Xavi Sep 18 '12 at 11:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.