LVM+MongoDB. Dropping in speed of backup creation

Question

Till recent time we have been making mongo backups archiving LVM snapshot on hidden SECONDARY server with RAID10 on SAS disks.

Currently operation on making snapshot from volume with 500 Gb of data on it takes lo-o-o-ong time.

Final archive is about 100Gb, and after creation of first 35 Gb during 30 minutes disk speed goes down incredibly. Till the moment when archive reached 52 Gb it was working for 6+ hours. I know that doing LVM snapshot during active I/O on main volume could slow disk operations, but couldn't image that it could slow it in tens times!

Now I'm thinking how to improve speed of backup.

As a potential solution I see the way where we stop mongodb on backup server and run LVM snapshot from "offline" server/volume. Then enable it again and re-sync with main replicaSet.

But that way could cause an issue when php-mongo-driver doesn't react properly on server list changes in replicaSet. And we need to restart php-fpm on all connected clients in order to fix that and make php seeing new changes quickly.

Would you please recommend us the best ways of how to make a mongo backup on data 500Gb+ under active I/O load.

Thank you in advance!

Answer 1

Regarding the LVM snapshot slow down, if you are doing a lot of randomly distributed writes over a large number of files, which you would be if you were inserting/udpating documents on a large MongoDB dataset, then LVM may indeed struggle to keep up. A repair/compact on the database (or a resync of the node from scratch) may help in terms of the number of files changed (compact will "defrag" the data in the files, repair/resync will rewrite the data from scratch), but that may only be an incremental improvement.

In terms of your proposed solution, it sounds like a good one, but I would advise you to make the node you are backing up from a hidden node, then you can eliminate the concern about the driver and set changes. The drivers use the isMaster command to determine the layout of a set, and when a node is hidden it is not returned by that command.

You can then shut down the node, do your snapshot, and restart once finished to let it resync.