1

I'm using EC2 p2.xlarge instances to train an image processing model. My target classes for the prediction task are quite a few (20) and to get the best accuracy I'm going to train one model per target.

So I did a snapshot of my main volume and created 4 duplicate volumes (general purpose SSD) so that I can run 4 more p2 instances concurrently. The problem is, except for the main volume, I'm experiencing an exteremely low IO performance. To give you an example, one training epoch which takes 200s on main instance is 'estimated' to take 10,000s on a duplicate instance.

What am i missing?

Improve this question

1 Answer 1

Reset to default
3

EBS snapshots are stored in S3. When you create a volume from a snapshot, blocks are first pulled incrementally from S3 to EBS in the FIRST read operation, and from that point all subsequent reads for that specific block are performed against the low latency infrastructure of EBS.

To grant optimal EBS performance after creating volumes from snapshots, you should "pre-warm" your volumes by running a full read of all the volume blocks, in order to ensure no further IO operation requires a slow request to S3.

New, fresh EBS volumes don't suffer from this behavior because they are "born" in EBS and do not require data pulls from S3.

Improve this answer
4
  • Pre-warming instructions.
    – Tim
    May 23, 2017 at 19:28
  • lmgtfy.com/?q=Prewarm+ebs+volume
    – ma.tome
    May 23, 2017 at 19:33
  • 3
    generally lmgtfy links would be considered mildly offensive or rude. Suggest you remove it. Yes the OP could've Googled it themselves, but anyone asking this level of question typically needs a bit more help. Including the actual command from that page would've made your answer better.
    – Tim
    May 23, 2017 at 19:40
  • The question was "what am I missing?", derived from an unexpected low performance of a block storage cloud solution. The answer provides a direct answer to this issue (which in fact is not a real issue as this behavior is explained in the service documentation) and provides a hint to workaround it. The exact way to execute this workaround is not, by any mean, the point of the question or, consequently , the answer.
    – ma.tome
    May 23, 2017 at 19:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.