3

I have an Ubuntu server hosted on Digital Ocean that has outgrown it's existing backup solution.

The relevant parts of the stack I use are Node.js, MongoDB, and Elasticsearch.

So far backups have been done by dumping the entire MongoDB database, saving the ES configuration, and copying all other files (logs, the code itself, etc.) in the application directory. If it's the first of the month all user files are also copied, otherwise only files changed since the first of the month are added. All of these are then zipped up in one file and uploaded to Amazon S3.

The data size has reached the point where this process takes too much disk space and the file can't be uploaded to S3 in one shot.

What is the next level for an application of this size (8 GB of user files, 125,000 users, 3,000 other docs, all searchable in ES)?

I understand opinion-based questions are not OK on Server Fault. I'm not asking for opinions, just what the normal, cost-effective, solution is for an application of this size.

UPDATE: These are the relevant parts of the script and configuration I'm attempting to use Duplicity with. I'm using Node to manage the backup since it fits with my existing logging solution, is already scheduled to align with everything else in a low-activity time, and is portable between OSes.

Node script, logging of course needs improvement:

// Walks a directory recursively and returns a flat list of files
function walkDir() {};

// Node based rm -rf
function rmrf() {};

exec("mongodump --out dump", { cwd: process.cwd() }, function(err, dta) {
    if (err) return log("Error backing up: couldn't dump MongoDB!");

    exec("sudo duply ats backup", function(err) {
        if (err) log("Error running Duplicity");
        else rmrf("dump");

        log("Exiting.");

        process.exit();
    });
});

Duplicity config:

GPG_PW='GPG password'

TARGET='s3://s3-us-east-1.amazonaws.com/bucket'

TARGET_USER='Known working AWS credentials'
TARGET_PASS='AWS secret key'

SOURCE='/var/appdir'

MAX_AGE=6M

DUPL_PARAMS="$DUPL_PARAMS --exclude "/var/appdir/elasticsearch/data/**" "

I've tried --s3-use-new-style, using s3+http://, and setting S3_USE_SIGV4 but have had no luck.

This is the log I'm getting from Duplicity:

Start duply v1.5.10, time is 2015-07-05 09:30:13.
Using profile '/root/.duply/ats'.
Using installed duplicity version 0.6.23, python 2.7.6, gpg 1.4.16 (Home: ~/.gnu                                                                     pg), awk 'GNU Awk 4.0.1', bash '4.3.11(1)-release (x86_64-pc-linux-gnu)'.
Signing disabled. Not GPG_KEY entries in config.
Test - Encryption with passphrase (OK)
Test - Decryption with passphrase (OK)
Test - Compare (OK)
Cleanup - Delete '/tmp/duply.25562.1436103014_*'(OK)

--- Start running command PRE at 09:30:14.155 ---
Skipping n/a script '/root/.duply/ats/pre'.
--- Finished state OK at 09:30:14.183 - Runtime 00:00:00.027 ---

--- Start running command BKP at 09:30:14.208 ---
Reading globbing filelist /root/.duply/ats/exclude
BackendException: No connection to backend
09:31:27.427 Task 'BKP' failed with exit code '23'.
--- Finished state FAILED 'code 23' at 09:31:27.427 - Runtime 00:01:13.218 ---

--- Start running command POST at 09:31:27.465 ---
Skipping n/a script '/root/.duply/ats/post'.
--- Finished state OK at 09:31:27.491 - Runtime 00:00:00.026 ---
1
  • It's unclear from this what your data size actually is. Is this an 8GB backup? Jul 3, 2015 at 17:22

2 Answers 2

3

I have good experience backing up using duplicity. If you are able to do a snapshot and mount it read only, then it's a very good option to have consistent incremental backup.

Usually the problem is with backing up databases (MongoDB, ElasticSearch, MySQL, you name it) is consistency. The same things apply to backing common files, but with databases, the risks of data corruption are probably the highest.

You have few options (hopefully others will add more)

  1. Dump the database and backup the dump. That is the simplest, safest and most straightforward.

  2. Stop the database (or use other method to make on-disk data consistent) and do the backup. (this way it causes a long downtime, not always possible)

  3. Stop the database (as in #2), do a snapshot (volume or fs, make sure the fs is consistent at that point), start the database, mount the snapshot readonly and back that up. But not all setups are suitable for this.

  4. Stop the database (as in #2), do a snapshot (this time it works only for volumes, make sure the fs is consistent at that point), start the database, back up the snapshot as a block device. This might increase the size of backup and again it might not be possible on all configurations.

  5. Back up the live database files, and hope it will work when you restore. (You're playing with fire here.) If at all possible, stay away from this.

  6. If your technology has a special way of backing up, use that. (Like direct snapshot backup from ELB to S3.)

Any way you choose, please keep in mind you definitely should test that you are able to restore from the backup several times, from several different backups.

#!/bin/bash
BACKUP_BASE="/data/backups/"
DIRNAME="mongo"
BUCKET="mybackups"
ARCHIVE_DIR="/data/backups_duplicity_archives/${DIRNAME}"
VERBOSE="-v 4"
S3_PARAMS="--s3-use-new-style" # --s3-use-multiprocessing" # --s3-use-rrs"
export PASSPHRASE="something"
export AWS_ACCESS_KEY_ID="AN_ID"
export AWS_SECRET_ACCESS_KEY="A_KEY"

cd ${BACKUP_BASE}
rm -rf ${BACKUP_BASE}/${DIRNAME}
/usr/bin/mongodump -h 10.0.0.1 -o ${BACKUP_BASE}/${DIRNAME}/databasename --oplog

/usr/bin/duplicity $S3_PARAMS --asynchronous-upload ${VERBOSE} --archive-dir=${ARCHIVE_DIR} incr --full-if-older-than 14D ${BACKUP_BASE}/${DIRNAME} "s3+http://${BUCKET}/${DIRNAME}"
if [ ! $! ]; then
        /usr/bin/duplicity $S3_PARAMS ${VERBOSE} --archive-dir=${ARCHIVE_DIR} remove-all-but-n-full 12 --force "s3+http://${BUCKET}/${DIRNAME}"
        /usr/bin/duplicity $S3_PARAMS ${VERBOSE} --archive-dir=${ARCHIVE_DIR} remove-all-inc-of-but-n-full 4 --force "s3+http://${BUCKET}/${DIRNAME}"
fi
7
  • Thank you for the detailed response! Consistency is not much of an issue since the application is low on data-changing requests. I'm working Duplicity into the backup script now. Restoring is a bit messy, but the app isn't too complex data-wise, it writes within its folder and calls its search engine and DB, as long as all three are present it's good to go and is pretty resilient. All I need to do is install it from its bootstrap zip, shut it down, restore the files in, start it up, and reindex everything in ES.
    – A.M.K
    Jul 5, 2015 at 1:54
  • I'm getting BackendException: No connection to backend Task 'BKP' failed with exit code '23'. from Duplicity. It appears to be failing to connect to S3. I've double checked my credentials and created a new bucket for it, but it isn't working. Do you have any suggestions?
    – A.M.K
    Jul 5, 2015 at 2:27
  • @A.M.K Can you please update your question with the command you're running? Sans the credentials of course ... I'll update the answer with the script I'm using.
    – Fox
    Jul 5, 2015 at 6:42
  • Oh and about the corruption - please keep in mind that corrupted database could mean, that you won't be able to read nothing out of it. Not that you can lose a record or two ... but knowing this, it's all your call.
    – Fox
    Jul 5, 2015 at 11:16
  • I've updated my question. I've been running a Mongo dump for existing backups like you have, so AFAIK corruption is no more an issue than it is for any other type of file. The system isn't dependent on relational docs at all, so one being corrupted trashes just that one and not any others.
    – A.M.K
    Jul 5, 2015 at 13:32
1

The data size has reached the point where this process takes too much disk space and the file can't be uploaded to S3 in one shot.

Upload them each as their own separate files. It's probably not the case that you're doing some kind of fancy de-duplication, and if you are changing that to be reference-based would be helpful.

So far backups have been done by dumping the entire MongoDB database

There are incremental backup tools (https://github.com/EqualExperts/Tayra) for MongoDB. I'd look into using those if your update load is relatively low.


Since you're on Digital Ocean, a local backup isn't an option. It is something to think about strategically, though. If you were hosted directly on Amazon, filesystem snapshots to S3 would likely be helpful for you.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .