AWS EC2 instance (Windows Server 2008): daily EBS snapshot without downtime

Question

TL;TR

I need to backup an AWS EBS-volume without downtime.

In details

I have a "Windows Server 2008" instance "EC2" on Amazon AWS with two EBS-volumes as C: and D: system drivers.

On C: driver works the OS, on D: driver works a web service, i need to backup only D:.

The web service is a Delphi SOAP application used for download some files according to request, eg.:

http://www.example.com/dir/service.dll?filename=foo.zip

(in this case the web service responds with the foo.zip file, located in D:\dir\files\).

On each request, this application writes some files (session files and log files) in D:\dir\temp\. This is a problem, because take a snapshot of in use volume it's not safe, from AWS docs:

You can take a snapshot of an attached volume that is in use. However, snapshots only capture data that has been written to your Amazon EBS volume at the time the snapshot command is issued. If you can pause any file writes to the volume long enough to take a snapshot, your snapshot should be complete.

My idea is implement in my web service a read only mode, based on the existence of a file in a directory.

procedure onRequest(Sender: TObject; Request: TWebRequest; Response: TWebResponse; var Handled: Boolean);
begin
    FReadOnlyMode := FileExists('D:\dir\flag\read_only_mode.txt');
    // ... other code

On each request the service check if the file exists and if it exists, the server going in the readonly mode and don't write anything (no log file, no session file.).

The backup strategy is based on a nightly in Windows cron task, that execute a batch file, eg. backup.bat:

:: 1. stop all web service writes operation
echo.>D:\dir\flag\read_only_mode.txt

:: 2. Use sync.exe for flush all data to disk D
sync.exe d

:: 3. Use Amazon CLI for snapshot the volume:
aws ec2 create-snapshot --volume-id vol-XXXXXXXXXX --description "D snapshot."

:: 4. Remove `read_only_mode` flag:
del /F /Q D:\dir\flag\read_only_mode.txt

(more info on Windows Sync and Amazon CLI)

The question is: my EBS snapshots will be consistent?

I'm new on AWS any other suggestions are appreciated!

Sorry for my bad english

Answer 1

In theory, for your EBS snapshots to be consistent, you would need to be running no stateful applications at the time the snapshot starts being created. Sadly, Windows itself can be considered stateful which can make a bit of a mess of things, especially if you are using write caching.

There are a couple of interesting but confusing parts to this:

• The 'snapshot' part of the snapshotting process is done immediately, you don't need to disable writes for the entire time a snapshot is pending. A large volume might take hours to complete the snapshot, but this is just the backup or copy portion of the snapshot. As long as Amazon doesn't suffer a power cut during this process the final snapshot will be the data on disk at exactly the time of the snapshot.
• Snapshots are taken asynchronously. So the command line tool should not block while the snapshot is being created, and it's return has no relevance to the snapshot status.

All things considered however, this generally doesn't cause a major issue. The issue of consistency is generally a small one. The main thing that catches people out is the issue of caching. If you have disk write caching enabled in your OS or application (it's on in windows by default), just because you think you've written a file to disk, doesn't mean the file has finished being written.

The other major issue that catches people out is transactions. If for example if you making a website that allows files to be uploaded your issues would be related to the snapshot taking place part of the way through the process. A common example of this would be a database record existing saying file '1234' was uploaded to 'd:/files/1234', but the file not existing as the file had not finished being copied to that location when the snapshot happened, but the database record being already committed.

One thing that people get confused about is the word consistency in this context, it doesn't refer to the ability to restore the snapshot. You will always be able to create a volume from the snapshot, the issue is if your application will be able to understand what state it was in afterwards.

If you were to example take a EBS snapshot of a windows system drive during a windows update procedure, you will end up with some files updated and some files not. Luckily Windows understands the update process and should notice the update failed and handle that on start up.

If you can write your application to not worry about files partially being written to disk (do you care if a log entry is missing?) you should not need to use a read only mode. One way to approach this with file uploads is to have the upload folder different to the final location (on the same disk). Once the file upload is completed and successful written, move the file to it's resting location. The move should be performed as a file 'rename' rather than byte by byte copy, and happen instantly. If the file never makes it to it's final location, you know the upload failed.