1

I run a server with md-raid (software) level 1 (2 HDDs mirroring) and several different applications run on it.

The problem arises when file-intensive jobs (e.g. extracting large archives) run while some users are also using realtime applications (e.g. game servers). The file intensive job will fill the IO queue forcing md-raid to block further IO until the two drives are synced, which will starve realtime applications and forces them to pause and wait, which in turn causes major annoyance with users.

The IO intensive job is running on lowest process- and io-priorities (nice -n +19 ionice -c 3 ...) with CFQ scheduling but other applications are still being starved.

Is there a way to make md-raid to be more lenient on keeping the disks in sync?

Are there other possible solutions to not starve important applications?

0

The problem is likely caused by pagecache flushing, rather than I/O queue fill.

To check it, try to untar a big archive and, in another terminal, do a cat /proc/meminfo. You will see some big values under Dirty memory; when that "dirty" memory is flushed, big I/O happens and your realtime application suffers.

It behavior can be configured, however, it is not trivial as by default Linux I/O stack favours throughput rather than latency. Anyway, as a good start, try to use the deadline scheduler and be sure NCQ is enabled on your disks.

3
  • Thank you, but NCQ is enabled and switching to deadline seems to have made the dropouts much more frequent, possibly even when not much IO is happening. – Cobra_Fast Apr 14 '16 at 21:31
  • Ok, so lets try the reverse path: can you use the NOOP scheduer? It changes anything? – shodanshok Apr 15 '16 at 6:15
  • noop doesn't change much. In the meantime I tried using the blkio cgroup controller for io throttling, but it can't throttle buffered io. – Cobra_Fast Apr 15 '16 at 17:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.