7

My tar command

tar --remove-files -cvif 2011-08-02_14-05-09.tar 2011-08-02_14-05-09

Writes 80MB/s. That's beyond the capacity we can spare because MySQL starts queuing up writes and eventually we get monitoring alerts cause systems won't wait forever on MySQL and time out.

Q: Can I throttle tar to only do 20MB/s?

Didn't find anything in the manpage and we already tried with the lowest ionice class: /usr/bin/ionice -c3 tar <...> but that doesn't impact the MBs written/s and MySQL still freezes up.

Maybe ionice is ignored because we're on a /dev/md0 software RAID device?

2
  • I guess as a hack we could immediately let tar bzip the archive cause we do have CPU cycles to spare. But I'm really interested to see if I can cap IO in a solid manner.
    – kvz
    Aug 2, 2011 at 14:39
  • ionice not working for you may have been because you were running with the incorrect scheduler. It apparently only works with the CFQ scheduler. Oct 31, 2011 at 12:28

2 Answers 2

5

You can use ratepipe.

I downloaded and tried it out and it seems to DTRT. It should work for you:

tar --remove-files -cvif - 2011-08-02_14-05-09 | ratepipe -r 20 > 2011-08-02_14-05-09.tar
3
  • 2
    Great idea! I found an alternative called pv (linux.die.net/man/1/pv) that can be found in apt. Syntax would become tar --remove-files -cvif - 2011-08-02_14-05-09 | pv -L 20m > 2011-08-02_14-05-09.tar Can also e.g. show a progress bar
    – kvz
    Aug 2, 2011 at 15:32
  • 1
    That's it! I knew there was a sexier program around, but couldn't remember pv.
    – MikeyB
    Aug 2, 2011 at 16:09
  • This is a gem of a find. Thanks. ratepipe rocks. Oct 31, 2011 at 10:37
1

You could probably write the tar to stdout (using -f -) and pipe it into something that throttles. I'll see if I can find something quickly.

Since tar will not buffer much, it should wait on the reads.

Edit: MikeB beat me to it: ratepipe will do.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .