Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Whenever I run du -sh or rsync I get my disk utilisation at 100%. This causes mysql slow queries to appear, which usually causes my site going down.

Any tips or tricks running these to commands somehow, so the priority for disk access stays with Mysql and other core services ?

share|improve this question
add comment

4 Answers

Yes, with ionice:

ionice -c 3 <pid>

where <pid> is the process ID of the process you want to slow down. This will set the I/O class of this program to idle, which means it will get IO time when nothing else is asking for it.

share|improve this answer
1  
i think you can also do command & ionice -c 3 $! to do it in one go. need to check that –  Sirex Mar 9 '12 at 11:56
add comment

You can use ionice -c3 as a prefix for your commands, that way those commands will run within idle I/O class, meaning they will only get I/O time when no other process needs it. This requires your system to use CFQ I/O scheduler, though. It's the default I/O scheduler in most Linux distributions, nowadays.

However, that kind of behaviour is usually symptom about something else. Is your disk nearly full? That causes slowdowns for systems having lots of writes, since everything tends to get fragmented. Or do you have tens of millions of small files around?

share|improve this answer
    
Was going to say just this. It will probably be worth your while to have a look at your storage systems. If you can't manage the storage as it's being used it is probably indicative of it being maxed out, and so highly likely to be a bottleneck in your platform. –  Coops Mar 8 '12 at 13:42
    
I will try doing so. I think I ll have to change my rsync command a little. Originally I was running rsync from backup machine, and it used to lock my production machine. I assume it is possible to rewrite rsync so i run it from production machine with "ionice -c3" to the backup machine. –  Katafalkas Mar 9 '12 at 7:45
add comment

Do you have a kernel that has PREEMPT turned on? There are various kernel PREEMPT settings that can be turned on that will avoid lockups, such as during heavy disk load (perhaps at some cost of performance of server apps). It depends on your distribution how much is turned on by default.

In Debian backports there now is for example linux-image-rt-amd64 which includes the PREEMPT_RT realtime patch set. This patch set may help even more, however it may have its own problems...

Run something like this to see what is set:

# grep PREEMPT /boot/config-2.6.32-5-amd64    <-- in case of a Debian stable amd64 kernel

Result:

# CONFIG_TREE_PREEMPT_RCU is not set
CONFIG_PREEMPT_NOTIFIERS=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set

You may want to have it set to something more aggressive. Look around in the kernel documentation that comes with the source of your kernel to see what would work for you. And check what kernels your distribution provides that may have more aggressive PREEMPT settings. Those could be meant for desktop usage and/or real time usage. But that doesn't mean it won't work on a server. Provided they're part of the standard distro those should get the usual security fixes.

share|improve this answer
    
Again, PREEMPT has to do with the CPU, not IO, so won't help here. These settings are to prevent the kernel from using too much cpu on behalf of a process all at once, thus preventing other processes from getting any for some period of time ( a few milliseconds ). They are only needed if you have a program that is interacting with hardware that has specialized low latency requirements where a few millisecond delay would cause problems. –  psusi Mar 9 '12 at 15:04
add comment

In addition to ionice(1) rsync might be pushing blocks needed by mysql out from disk cache. You can patch rsync with fadvise support to avoid putting the stuff read by rsync into cache.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.