Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a command that runs a disk snapshot (on EC2, freezing an XFS disk and running an EBS snapshot command), which is set to run on a regular schedule as a cron job. Ideally I would like to be able to have the command delayed for a period of time if the disk is being used heavily at the moment the task is scheduled to run.

I'm afraid that using nice/ionice might not have the proper effect, as I would like the script to run with high priority while it is running (i.e. wait for a good time, then finish fast).

Thanks.

UPDATE:

This is what I ended up going with. It checks /proc/diskstats and runs my job when the current IO activity hits 0, or we timeout. I'll probably have to tweak this when I look at what kind of IO activity our servers actually get in production:

#!/bin/bash

DEVICE=sdf

# we want to make a snapshot when IO in progresses reaches this:
LOW_THRESHOLD=0

TIMER=0
MAX_SEC_DELAY=120

# Get the number of IO operations in progress:
ioInProgress(){
    grep $DEVICE /proc/diskstats | awk '{print $12}'
}

# Wait for a good time to run snapshot, else timeout:

while [[ $TIMER -lt $MAX_SEC_DELAY && $(ioInProgress) -gt $LOW_THRESHOLD  ]]; do
    TIMER=`expr $TIMER + 1`
    sleep 0.5
done

# Recording delay required:
echo $TIMER
echo "Executing snapshot"
run-the-snapshot
share|improve this question
1  
an alternative maybe use a different io scheduler redhat.com/magazine/008jun05/features/schedulers this may give all process running fairer access to the disk subsystem and stop io starvation. –  The Unix Janitor Apr 15 '10 at 13:40

3 Answers 3

up vote 2 down vote accepted

i found a perl script here http://www.skolnick.org/cgi-bin/list.pl?file=serverload.pl

This should do what you need.

You could probably implement this with a simple shell script that parse uptime, and only being execution once the load average has lowered to a certain value.

Beware that if you server is constantly busy, with a run away process, then your cronjobs will never execute!.

Perhaps a better idea if you have memory is to run you cron jobs at a the lowest OS priority, thus they will only consume spare resources.

share|improve this answer

What about writing a little poll/daemon script that, beginning at your scheduled time, checks iostat (vmstat) for low disk activity and continues to check every 5 minutes until the disk activity is lower/below a preset threshold or a period of time has elapsed, whatever comes first?

share|improve this answer

You can take a look at the batch command Maybe it will fit your needs.

man batch for further details (it is a part of the at subsystem)

share|improve this answer
    
Thanks, batch looks great. I decided to go with a homebrew solution since I realized I am more interested in disk IO. –  jberryman Apr 17 '10 at 15:39

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.