14

I have a cron that basically do a simple "tar zcf" in the night.

The server has:

  • 8 Cores - Intel(R) Xeon(R) CPU E5606 @ 2.13GHz
  • 25GB RAM
  • Ubuntu 12.04.2 LTS
  • Hardware RAID 1 (LSI Logic / Symbios Logic MegaRAID SAS SMC2108) with two 2.728TB harddrives

As you can see on the monitoring screenhost:

http://clip2net.com/s/57YRKP

During almost all the time of the tar, disc I/O goes to >90% and make all other app (mysql, apache) to slow down a lot.

2 questions:

  • Is it normal to have so high disc I/O during the backup?
  • Is there a way to limit the disc I/O so other app can continue to work correctly?

Thank you!

Improve this question

4 Answers 4

Reset to default
11

Beside the rather general approach with ionice there is a nice device mapper target (ioband) which allows precise control over the bandwidth to a (DM) block device. Unfortunately it is not part of the standard kernel.

Furthermore you can probably speed up tar by

  1. Reading the file names into the disk cache: find /source/path -printf ""
  2. Reading the inodes into the disk cache: find /source/path -perm 777 -printf ""
  3. Making tar read and write larger blocks from and to the disk by e.g. using a pipe with mbuffer or buffer (with at least 100 MiB of RAM): tar ... | mbuffer -m 256M -P 100 -p 1 ...
Improve this answer
9
  • Why does reading the file names/inodes into the cache reduce disk IO while tar'ing? I would expect it to increase the average IO while reducing the total time only slightly.
    – scai
    May 27, 2013 at 11:13
  • 3
    @scai This does not help with SSDs; my recommendation refers to spinning harddisks only. What kills performance with those is head movement. The file names are stored in continuous blocks, the inodes are stored in continuous blocks and the file content is stored in continuous blocks. If you do it the tar way then you read the file (and subdirectory) names of one directory, access the inode for one file, then the file itself, then the inode for the next file, then the next file itself... That causes more head movement than reading all names and inodes after each other.
    – Hauke Laging
    May 27, 2013 at 11:15
  • @scai The performance impact depends on what you do. It is rather small for full backups (probably depends on the file sizes) but I noticed a big difference for differential backups (not for tar, though, as I don't use that but this should be a general effect).
    – Hauke Laging
    May 27, 2013 at 11:18
  • Just to be sure I understood correctly. For 1. and 2., we just have to call the find command and Linux will automatically cache it?
    – acemtp
    May 27, 2013 at 12:38
  • @acemtp That is correct. find without (e.g.) -perm will not access the file inode, though. But that allows for the optimization to use two find calls. If you make the same find call twice (with little time in between), the second one is usually going to finish within seconds (or less). Depending on the amount of free memory and the amount of data cached at a certain point the data is thrown out of the cache. Reading too much may thus just slow down the operation. If you can feed the backup program with file names via stdin then you can prevent this by reading blocks of e.g. 100 files.
    – Hauke Laging
    May 27, 2013 at 13:05
13

It's expected to see high I/O during backups because they're generally made over large file trees with large files. You can use ionice to prioritize I/O jobs in Linux with classes and levels. IIRC, class 2, level 7 is the lowest, non starving level which will make it practically invisible to other I/O loads and users. See man ionice for usage and details.

Improve this answer
1

I would recommend ditching tar and going with rsync (as mentioned by Dogsbody). I use BackupPC to backup files on my Windows and Linux systems and it supports using tar as well as rsync and automatically takes care of the hard linking for you as well as provides a nice web interface.

http://backuppc.sourceforge.net/

Improve this answer
0

As others have replied, yes this is normal, and ionice is a good generic way of not letting it affect your system.

A number of times I've seen people tar things up when they don't need to though. If any percentage of the data you are copying hasn't changed since the last copy then I'd suggest giving rsync a try.

This will reduce IO by only copying the files that have changed since last copy. you won't be able to reduce the IO by more than half as all the data would still need to be read but you will significantly reduce the amount of data written (which depending on your hardware can be a slower operation as well).

If you want separate copies/backups each time it's run then the most powerful option is –link-dest which allows you to hard link unchanged files to a previous backup. This saves HUGE amounts of space on the backup server. e.g. I backup a machine (Fred), Fred has a 20GB HD and I back up/copy the entire drive excluding /proc and /dev. I now have a 20GB directory on my backup server. The next day I backup Fred again and –link-dest to yesterdays backup. Rsync compares the remote files with the local copy and if exactly the same will not bother transferring them but will hard link the new file to yesterdays file. Any files that have changed are copied down a fresh (or partially copied using yesterdays backup if possible). If only 100MB of files of changed since yesterday I now have two directories both with 20GB of files but only taking up 20.1Gb of space on the backup server!

I hope that helps and still answers your question.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.