0

I have found similar questions here but the answer are either outdated or are from the command line.

The case is this. I have an EC2 instance using instance store (this was the only AMI available for Debian 6 in Ireland). Now through the AWS GUI I can do a snapshot of the instance volume and/or even create a volume. But an image made from the snapshot doesn't boot.

What is the best solution to either clone an EC2 instance that uses instance store OR from the created snapshot of the instance store to launch a new EBS instance (identical clone) FROM the gui aws management console and not command line ? Before turning this down consider that there is not similar question on how to do it via the aws management console.

hint can't be done is not an appropriate answer. As you can create a snapshot of the instance store backed instance and/or a volume and create an AMI from that snapshot.

7
  • 1
    a) What is wrong with using the command line - you definitely can do this using the API tools, I don't think converting instance-store to EBS can be done with the GUI b) The 'snapshots' created from an instance-store volume are completely different from those created from an EBS volume, and cannot be directly used to create an EBS volume. Instance-store 'snapshots' (rarely referred to as such) essentially amount to a signed bundle of all the files (an AMI), while EBS snapshots are point-in-time, block level, differential images. c) Try to create an AMI and launch a new instance using that.
    – cyberx86
    Apr 6, 2012 at 5:36
  • thanks for your comment but instance-store snapshots are defined like that always by amazon inside the aws console and not "rarely". Good to know that it is not bootable but the question is specifically for running this on was management console and not a debate about the utility of command lines too that is not in question. thanks for your effort.
    – devnull
    Apr 6, 2012 at 5:40
  • Fair enough, I can appreciate that sometimes people prefer a GUI approach - the point though, is that if a valid method exists, not using it, without a good reason, is merely an academic exercise. That said, there do appear to be a number of EBS-root Debian 6 AMIs available in EU-West-1 - see the Debian Wiki; perhaps one of them will work for you. I'd suggest posting the console output, which might have some clues as to why your AMI doesn't boot.
    – cyberx86
    Apr 6, 2012 at 5:49
  • well for the sake of precision (and no debate) it is not an academic exercise. It is meant to help the hundreds (or thousands) of AWS GUI (management console) users. For example all these command line tutorials and answers to create an ami from an EBS-root debian are the academic exercise when with 1 click you can do it from the aws management console online. At any rate I am thankful for your advice. So how would I move all the data to a new EBS debian 6 instance from the old, instance-store one ? snapshot + volume and add to new ebs instance? thanks again for your valuable help.
    – devnull
    Apr 6, 2012 at 6:01
  • a) Creating an AMI in the AWS console and launching a new instance from that should give you a cloned instance (still instance-store) - if it doesn't you need to figure out why (e.g. attached devices with different names - use the console output to do so.). b) If you use a new AMI (not one created from your existing instance) you will have to reinstall any software you added. c) You can make a EBS-root AMI from an instance-store AMI (I only know the command-line way) either using rsync or the API tools. See this question for details.
    – cyberx86
    Apr 6, 2012 at 6:12

1 Answer 1

6

You cannot do anything with the data on an instance-store volume from the AWS console. There is a good reason for that, which I will get into later.

First let me clarify a few points from your question:

The case is this. I have an EC2 instance using instance store (this was the only AMI available for Debian 6 in Ireland).

There are several EBS-root Debian 6 AMIs available in EU-West-1 (both 32-bit and 64-bit). Some of these can be found on the Debian Wiki.

Now through the AWS GUI I can do a snapshot of the instance volume and/or even create a volume. But an image made from the snapshot doesn't boot.

What AWS refers to as 'snapshots' can only be made from EBS volumes (see image). By their very nature, instance store volumes are ephemeral - temporary. The only way to capture the state of an instance-store volume is to bundle the files on it (almost like you would by making an archive) - these are not block-level. The snapshots that are mentioned in the AWS console are for EBS volumes - these are block level. In all likelihood, the snapshot you created was of an EBS volume that was attached to the instance, and, since this was not the root volume, trying to run an instance from it continually failed.

AWS Console

Additionally, you cannot add instance-store volumes to a running instance. Instance-storage is a characteristic of the instance type - it comes in fixed volume sizes (e.g. you cannot add a 15GB instance-store volume) and can only be added at the time the instance is launched (either as part of the ec2-run-instances command or if it is built into the AMI.

In general, instance-store instances should be avoided unless you have a very good reason to use them (typically high-io, transient tasks)

Now, to try and answer some of the questions you have asked:

What is the best solution to either clone an EC2 instance that uses instance store OR from the created snapshot of the instance store to launch a new EBS instance (identical clone) FROM the gui aws management console and not command line ? Before turning this down consider that there is not similar question on how to do it via the aws management console.

Use the command line. Before you think I am dismissing what you have asked, allow me to explain. AWS does not access your files - none of the actions that you can perform remotely require any knowledge of what is actually on your instance. So, creating an EBS snapshot occurs at the block level - it doesn't need an understanding of the file-system or permissions to read the files contained on the volume (imagine if AWS needed to be able to read every conceivable file-system on a linux operating system, it would be an impossible task). On the other hand, the process of creating a 'bundle' from an instance-store volume requires read access to the files on the system. You will note that the ec2-bundle-vol must be run from the instance in question (unlike the ec2-create-snapshot command, which can be run without access to the instance).

The command line and API exist for several reasons. Firstly, they are essential to automation - you can't script the 'AWS Console'. Just as important though, they offer a far wider range of options. While the AWS Console continues to add new features all the time, for one reason or another, there are still many things that can only be done from the command line. Choose what works best for you, but do not dismiss either the command line or the GUI interface as both have their uses.

So, since the console has been ruled out, you must use the command line to proceed.

Option 1:

Create a new instance-store AMI from your existing instance. You will need to run:

  • ec2-bundle-vol to create the AMI
    ec2-bundle-vol -d /mnt -k $EC2_PRIVATE_KEY -c $EC2_CERT -u USERID -s SIZE
  • ec2-upload-bundle to save it to S3
    ec2-upload-bundle -b BUCKETNAME -m MANIFESTFILE -a ACCESSKEY -s SECRETKEY
  • ec2-register to register the AMI (this last one can be done from the AWS console). Register AMI

This is definitely the easiest approach, but personally, I wouldn't go with it. If you are going to put in the effort, you might as well go all out and convert to EBS.

Option 2:

Copy the files you have to an EBS volume and create a new image based on that volume.

You need to first create a new EBS volume (in the same availability zone as your existing instance), and attach that volume to the instance.

Note: It is a good idea to remove any references to ephemeral storage from your /etc/fstab if you plan to use the AMI on different instance types, as the ephemeral storage available on each instance type varies (typically mounted on /mnt).

There are two approaches to this, depending on what you currently have.

  1. If you have a bundled volume (e.g. from 'Option 1') you can unbundle the volume, and then use dd to copy it onto your EBS volume (you may need to download the bundle first with ec2-download-bundle):

    ec2-unbundle -m /local/path/to/manifest.xml -s SOURCE_DIRECTORY -d DESTINATION_DIRECTORY dd if=/path/to/image of=/dev/NAME

  2. Otherwise, you can use rsync to copy the files on your root volume to the new EBS volume.

    • Format your EBS volume (e.g. mkfs.ext3 /dev/sdf)
    • Mount the EBS volume (e.g. mkdir /mnt/ebs && mount /dev/sdf /mnt/ebs)
    • Stop as many services running on the instance as you can (to minimize the risk of inconsistencies)
    • Flush the caches: (e.g. sync)
    • Copy the files over: (e.g. rsync -aHAXxSP --exclude /mnt/ebs / /mnt/ebs). You could instead use dd, but for a mounted source file system, rsync is better.
    • Flush the caches: (e.g. sync)
    • Unmount the EBS volume: (e.g. umount /mnt/ebs)
    • Check the file system: (e.g. fsck /dev/sdf)

    From here, the rest you can do in the AWS console: make a snapshot of the EBS volume and create an AMI from that snapshot. Launch your new AMI to ensure that everything works.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .