3

I'm planning to run a process that produces and saves data on an amazon ec2 micro spot instance. I read on their website that if the bidding price exceeds my maximum price my instance will be automatically terminated. How can I save the data before the instance terminates?

Improve this question

2 Answers 2

Reset to default
5

Regularly checkpoint your data to a data store that is external to the machine. An Amazon S3 bucket may be a strong candidate for that. Currently, there is no charge for data in, and the charge to post a new object every minute is very low ($.01 per 1,000 put requests).

Improve this answer
2
  • Is there an automated solution? How would I be able to implement it?
    – alexyorke
    Aug 17, 2011 at 2:18
  • Amazon only provides an API through the SDK. You could create your own tools, write it into whatever application you're running, or script something around it using existing tools. One place to look is s3tools.org/s3tools
    – Jeff Ferland
    Aug 17, 2011 at 3:15
5

Use a persistent EBS disk as your storage medium - it will function just like a regular volume, but will continue to exist beyond instance termination (note that the EBS root volumes are NOT persistent by default).

Depending on your specific setup, you may either keep all the data on the EBS volume, and each time you launch the instance, attach the EBS volume and continue processing data or you can use the EBS volume as temporary storage, saving data that has been fully processed to an external location (e.g. S3). Either way, EBS should be part of your setup to provide seamless continuation from where the last instance left off, and to provide the best data throughput.

Improve this answer
4
  • +1 as that's another good answer. There are possible issues with half-written files, etc. that need to be thought of if you go that route. Treat it like you would any machine with an HDD that you would expect to get the plug pulled on all the time. This may also be a bit more complicated if you have several instances that may spawn at once -- one EBS can only be attached to one instance at a time.
    – Jeff Ferland
    Aug 17, 2011 at 3:20
  • 1
    One of the nice things about EBS is that writes are guaranteed if the process returns success, even if the actual data has not been written to the EBS. (i.e. if you write data, the operating system says its written, and your instance terminates - EBS will write that data even if it is after instance termination). As for the limitation of an EBS only being attached to a single instance, you could use a clustered file system (e.g. gluster) to distribute or replicate the data and make it accessible to multiple instances - the increase in complexity is not always justified though.
    – cyberx86
    Aug 17, 2011 at 3:54
  • Actually as alexy13 plans to use micro instance, they will use EBS storage anyway, so data will persist instance termination.
    – oker
    Aug 19, 2011 at 8:54
  • 1
    @oker: I don't believe that to be correct - root EBS volumes do not persist termination unless explicitly set to do so. If you launch a t1.micro AMI (say Amazon's Linux) - it will create an EBS volume, but as soon as you terminate it the EBS volume is deleted. Only in AMIs that preset the EBS root to persist or if you manually change the persistence will the root EBS volume remain. (On the other hand, attached EBS volumes always persist).
    – cyberx86
    Aug 19, 2011 at 12:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .