I am running a server with linux software raid 10. It is a dual CPU system with 64GB Ram. 2x16GB dimms related to each of the CPUs. I want to use dd to backup kvm virtual machines and run into a serious io problem. First I thought is related to the raid but it’s a problem of the linux memory management. Here is an example:
- Memory is fine: https://i.stack.imgur.com/NbL60.jpg
- I start dd: https://i.stack.imgur.com/kEPN2.jpg
- You see also nmon shows the disk access: https://i.stack.imgur.com/Njcf5.jpg
- After a while the "buffers" are big and the copy progress stops https://i.stack.imgur.com/HCefI.jpg
- Here is meminfo: https://i.stack.imgur.com/KR0CE.jpg
- Here the dd output: https://i.stack.imgur.com/BHjnR.jpg
- I can manually resolve temporary the problem and force dropping the cache: "sync; echo 3 > /proc/sys/vm/drop_caches"
- The call needs few seconds and instantly after that the dd speed reach normal level. Sure I can a cronjob every min or such things but thats not a real solution. https://i.stack.imgur.com/zIDRz.jpg https://i.stack.imgur.com/fO8NV.jpg
Does anybody have a solution or a configuration hint? Here is also my sysctl but all values are centos defaults: https://i.stack.imgur.com/ZQBNG.jpg
Edit1
I make a other test and make a dd to disk instead /dev/null. This time also in one command without pv. So its only one process. dd if=/dev/vg_main_vms/AppServer_System of=AppServer_System bs=4M
- It start with reading without writing (target is not on the same disks) https://i.stack.imgur.com/jJg5x.jpg
- After a while the writing starts and the reading slow down https://i.stack.imgur.com/lcgW6.jpg
- After that a writing only time comes: https://i.stack.imgur.com/5FhG4.jpg
- Now starts the main problem. The copy process slow down to below 1mbs und nothing happend: https://i.stack.imgur.com/YfCXc.jpg
- The dd process now needs 100% cpu time (1 core) https://i.stack.imgur.com/IZn1N.jpg
- And again I can manually resolve temporary the problem and force dropping the cache:
sync; echo 3 > /proc/sys/vm/drop_caches
. After that the same game starts again...
Edit2
For the local dd I can workaround with the parameter iflag=direct and oflag=direct. But this is no universal solution because there is also other file access like copy files to the local samba shares from a vm and there I cant use such parameters. There must be a tweak of the system file cache rules, because it cant be normal that you cant copy large files without such problems.