I have a massive problem with the linux page cache which slows down the IO. For instance if I copy a lvm partition with dd, linux cache the data in the buffers or caches (free –m). That is not the problem, but after the buffer reaches a special value, the copy process stop and slow down to a few mbs or even kbs. I have done many tests with writing to disk or /dev/null the problem has nothing to do with the source drive or the destination.
- There are two nearly identical servers. Both runing CentOS 6.5 with the same kernel. They have the same disks, the same setup, the same other hardware, the same in all ways. The only difference is that one sever has 2 CPUs and 64GB ram and the other one 1 CPU and 32 GB ram.
- Here is also an image of the following copy process: http://i.stack.imgur.com/tYlym.jpg
- Here a new version also with meminfo. The meminfo is from a different run, so the values are not identival, but it ist the complte same behavior: http://i.stack.imgur.com/4SIJG.jpg
- Start copy with dd or other filesystem copy programm.
- Buffer or cache start to fill. All is fine.
- Buffer or cache reaches a maximum number (on 64GB ram server a value like 32GB or 17GB; on 32GB ram server all free memory)
- On 64GB ram server the copy process now stops or is limited to few mbs. On the 32GB ram server all is fine.
- On 64GB ram server I can solve the problem for a short moment by forcing the cache with "sync; echo 3 > /proc/sys/vm/drop_caches". But of course the buffer starts to grow again instantly and the problem occur again.
The problem has either something to do with the second cpu or with the total amount of memory. I have the “feeling” that the problem coud be, that every cpu has his own 32GB ram and the copy process running only on the cpu. So finaly the copy process inceased the buffer / cache to nearly 32GB or to the not used memory of the other cpu and then linux thinks hey there is still memory so lets us increase the buffer further but the hardware below cant access the memory, or something like that.
Has anybody an idea or a solution? Sure I can use dd with direct flag, but that dont solve the problem, because there is also extern access via samba and so on.
- VM settings: http://pastebin.com/U9E9KkFS
- /proc/sys/vm/zone_reclaim_mode was on the 32 GB ram server 0 and on the 64 GB ram server 1. I never touch this values. The installer set these. I temp changed it to 0 and retry the test. Now all memory is uses for buffer and cache. So it looks great and like the other server. But then it instantly start swaping with full speed... I set swapiness to 0. That helps, but it still swaping few mb per secound. And it increase the buffers every secound. So it dont swap the buffer, it swap the memory of the vms to get more memory to increase the buffers... crazy. But maybe this is normal!?
/proc/buddyinfo and numactl --hardware: http://pastebin.com/0PmXxxin
- /proc/sys/vm/zone_reclaim_mode is for sure the technical rigth way, but the maschine did not work very well after it. For instance: if I copy a disk linux use now 100% of the free mem to buffer (not as before only XGB and then stop). But at the secound at that the last free mem was used to buffer, linux start swaping vm memory and increase the total amount of buffer and caches. The swap is normaly not needed in my system so the swap memory is on the same disk as some vms. In the result if a make a backup of these vms linux write the swap at the same time as I read from the disk for the backup. So it is bad to swap the vms but it is even worser that linux destroy my backup read speed... So the setting of /proc/sys/vm/zone_reclaim_mode to 0 dont solve the full problem... currently I run in a screen a script that sync and fushes cache every 10 sec... not nice but work much better for me. I have no webserver or normal file server on the system. I only run vms, make backups and store backups via samba. I dont like the solution.