2

Note that while this question is a little Redis-specific, the main problem is generic: a process takes so much HDD IO write bandwidth that other processes can't write anything.

We've got an Ubuntu VM inside Ubuntu-based Xen XCP host (installed on two HDDs in software RAID1). That VM is running Redis server under about 2K commands/s load.

Problem: when said Redis server does BGREWRITEAOF, it blocks its clients for about 10 seconds.

Details:

Only AOF persistence is used, no RDB. Redis is configured to fsync AOF file once per second.

On BGREWRITEAOF Redis forks and does all disk-intensive work in the child process. Meanwhile, main process keeps appending data to its AOF file.

BGREWRITEAOF takes about 10 seconds (1.5GBs of data, 150 MB/s disk write speed). The child process doing rewrite consumes all HDD IO write throughput.

Parent process attempts to fsync, it takes more than two seconds, data protection kicks in, and blocking write gets called, blocking the parent process until BGREWRITEAOF is finished with the disk.

Here is a detailed info and discussion that lead me to above interpretation of events.

Question: It looks fishy to me that a process is allowed to take so much disk IO that everything else is blocked. Is there something that I can do on system level to fix that? I'm OK if BGREWRITEAOF will take a little more time, as long as parent process is allowed to save its data while rewrite is active.

Please note that I'm aware of workarounds, like moving AOF persistence to slave, using no-appendfsync-on-rewrite Redis config option etc.; this question is specifically about resolving the problem, not working around it.

2 Answers 2

1

AFAICS you can try to change IO scheduler. Try to use this command:

echo cfq > /sys/block/$DEVICE/queue/scheduler

Where $DEVICE is your RAID1 disk. This command install 'Completely Fair Queuing' scheduler for your device.

2
  • 1
    I'd probably stay away from the CFQ I/O scheduler. It can be more of a problem on server systems and virtual guests. It's also probably the default for this Ubuntu system.
    – ewwhite
    Oct 30, 2012 at 9:21
  • It says noop [deadline] cfq on VM. Oct 30, 2012 at 9:34
0

I would suggest changing your I/O scheduler and applying some light tuning techniques. While I don't have a comprehensive tuning guide, some of the answers and suggestions detailed in this question may help you as well.

Consider changing the I/O elevator to deadline or the noop algorithm, and retest. You can make this change on the fly using the technique detailed in another answer. Add an entry to the GRUB kernel command like to make this persistent across reboots (add: elevator=deadline)

Perhaps some detail about the underlying hardware or host system setup would help. Is there any battery-backed or flash-backed write cache on the storage subsystem? That can make a difference.

Finally, you can try some light benchmarking/monitoring tools to see what's going on. If you have access to iostat, for instance, you can run it in another terminal window as you test your application.

E.g. iostat -x 1 will run with 1-second samples and provide some indication on read/write speed and I/O service time and wait time. I also like collectl for this purpose.

6
  • See dstat and iostat dumps here: groups.google.com/forum/#!msg/redis-db/vSAvnYVtX9w/NRYlBwOrCTsJ Oct 30, 2012 at 9:35
  • It seems that I'm using deadline scheduler. Will try noop and (for completeness) cfq. Oct 30, 2012 at 9:40
  • Okay. I see... You have no write cache for your storage, so you're at a disadvantage there. You're literally waiting for rotating disks to acknowledge I/O. You also have multiple layers of I/O elevators; one in the host, one in the guest. This may be a situation where you want deadline in the host and noop in the guest.
    – ewwhite
    Oct 30, 2012 at 9:40
  • No battery-backed or flash-backed cache, sadly. This is a plain and humble Hetzner box. Oct 30, 2012 at 9:40
  • I'll add that using CFQ will allow you to starve processes with a little more granularity. It also opens up the possibility of using ionice on a per-process basis. (ionice is not compatible with the other schedulers)
    – ewwhite
    Oct 30, 2012 at 9:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .