Till recent time we have been making mongo backups archiving LVM snapshot on hidden SECONDARY server with RAID10 on SAS disks.
Currently operation on making snapshot from volume with 500 Gb of data on it takes lo-o-o-ong time.
Final archive is about 100Gb, and after creation of first 35 Gb during 30 minutes disk speed goes down incredibly. Till the moment when archive reached 52 Gb it was working for 6+ hours. I know that doing LVM snapshot during active I/O on main volume could slow disk operations, but couldn't image that it could slow it in tens times!
Now I'm thinking how to improve speed of backup.
As a potential solution I see the way where we stop mongodb on backup server and run LVM snapshot from "offline" server/volume. Then enable it again and re-sync with main replicaSet.
But that way could cause an issue when php-mongo-driver doesn't react properly on server list changes in replicaSet. And we need to restart php-fpm on all connected clients in order to fix that and make php seeing new changes quickly.
Would you please recommend us the best ways of how to make a mongo backup on data 500Gb+ under active I/O load.
Thank you in advance!