3

I am running a server with linux software raid 10. It is a dual CPU system with 64GB Ram. 2x16GB dimms related to each of the CPUs. I want to use dd to backup kvm virtual machines and run into a serious io problem. First I thought is related to the raid but it’s a problem of the linux memory management. Here is an example:

  1. Memory is fine: https://i.stack.imgur.com/NbL60.jpg
  2. I start dd: https://i.stack.imgur.com/kEPN2.jpg
  3. You see also nmon shows the disk access: https://i.stack.imgur.com/Njcf5.jpg
  4. After a while the "buffers" are big and the copy progress stops https://i.stack.imgur.com/HCefI.jpg
  5. Here is meminfo: https://i.stack.imgur.com/KR0CE.jpg
  6. Here the dd output: https://i.stack.imgur.com/BHjnR.jpg
  7. I can manually resolve temporary the problem and force dropping the cache: "sync; echo 3 > /proc/sys/vm/drop_caches"
  8. The call needs few seconds and instantly after that the dd speed reach normal level. Sure I can a cronjob every min or such things but thats not a real solution. https://i.stack.imgur.com/zIDRz.jpg https://i.stack.imgur.com/fO8NV.jpg

Does anybody have a solution or a configuration hint? Here is also my sysctl but all values are centos defaults: https://i.stack.imgur.com/ZQBNG.jpg

Edit1

I make a other test and make a dd to disk instead /dev/null. This time also in one command without pv. So its only one process. dd if=/dev/vg_main_vms/AppServer_System of=AppServer_System bs=4M

  1. It start with reading without writing (target is not on the same disks) https://i.stack.imgur.com/jJg5x.jpg
  2. After a while the writing starts and the reading slow down https://i.stack.imgur.com/lcgW6.jpg
  3. After that a writing only time comes: https://i.stack.imgur.com/5FhG4.jpg
  4. Now starts the main problem. The copy process slow down to below 1mbs und nothing happend: https://i.stack.imgur.com/YfCXc.jpg
  5. The dd process now needs 100% cpu time (1 core) https://i.stack.imgur.com/IZn1N.jpg
  6. And again I can manually resolve temporary the problem and force dropping the cache: sync; echo 3 > /proc/sys/vm/drop_caches. After that the same game starts again...

Edit2

For the local dd I can workaround with the parameter iflag=direct and oflag=direct. But this is no universal solution because there is also other file access like copy files to the local samba shares from a vm and there I cant use such parameters. There must be a tweak of the system file cache rules, because it cant be normal that you cant copy large files without such problems.

6
  • I've deleted my answer as it clearly does not address your main problem. Additional interesting information would be your distribution of choice and kernel version. May 12, 2014 at 17:28
  • I am running CentOS 6.5 with all updates and the current CentOS Kernel 2.6.32-431.17.1.el6.x86_64.
    – user219392
    May 13, 2014 at 7:05
  • I've retested this now on a spare server I've had temporary access to with 32GB RAM and Scientific Linux 6.5 (another RHEL clone with 2.6.32-431.11.something kernel) without being able to reproduce that slowdown. So either there's something strange going on above 32GB or the very latest update of the kernel has some bug. For giggles, you could invest $2 on AWS for an hour of r3.4xlarge instance with 122GB RAM to see if you can reproduce it there. May 14, 2014 at 13:51
  • Thx for your reply. I have nearly identical server with 32GB ram that did not have the problem. I make some new tests and post it in a new question serverfault.com/questions/596565/… . It has someting to do with dual cpu and memory sharing / access or simple with 64GB mem. But I think the first one.
    – user219392
    May 16, 2014 at 17:03
  • Just for reference: I've observed the same behaviour on my desktop system, and I'd be interested in a solution, too. Disk write speed is down to 1 Mb/s, and increases to 100 Mb/s after dropping the caches, though not always and not always instantly.
    – dirkt
    Nov 13, 2017 at 14:11

1 Answer 1

0

Just a wild guess. Your problem may be big dirty page flushing. Try to set up /etc/sysctl.conf like:

# vm.dirty_background_ratio contains 10, which is a percentage of total system memory, the 
# number of pages at which the pdflush background writeback daemon will start writing out 
# dirty data. However, for fast RAID based disk system this may cause large flushes of dirty
# memory pages. If you increase this value from 10 to 20 (a large value) will result into 
# less frequent flushes:
vm.dirty_background_ratio = 1

# The value 40 is a percentage of total system memory, the number of pages at which a process
# which is generating disk writes will itself start writing out dirty data. This is nothing
# but the ratio at which dirty pages created by application disk writes will be flushed out
# to disk. A value of 40 mean that data will be written into system memory until the file 
# system cache has a size of 40% of the server's RAM. So if you've 12GB ram, data will be
# written into system memory until the file system cache has a size of 4.8G. You change the
# dirty ratio as follows:
vm.dirty_ratio = 1

Then do sysctl -p to reload, drop the caches again (echo 3 > /proc/sys/vm/drop_caches).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .