Percona's xtrabackup utility has a --throttle option to reduce the IO load of the backup job.
The docs say that the value passed is the number of read/write pairs per second. Is 1000/sec an appropriate value on modern hardware? How about 5? I cannot find any meaningful frame of reference for these values.
For reference - I am reading from and writing to the same drive array (10k SAS). A 55gb backup job with --throttle=20 ran in roughly an hour with no apparent strain on the system during off-peak hours. But I honestly don't know if this is a high or low value for the throttle.