Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've built an application which forks process pools to fetch web content.

I have 3 ec2 instances mounted onto one s3 bucket via fuse.

I'm experiencing intermittent errors writing files.

From what I gather s3 doesn't seem to like lots of concurrent writes, I might need to write to EBS then sync/copy.

Is that the right solution or is there a better approach?

share|improve this question

migrated from stackoverflow.com Oct 8 '12 at 14:33

This question came from our site for professional and enthusiast programmers.

add comment

1 Answer

It turns out the problem wasn't concurrent writes.

The ec2 Micro instances steal cpu when attempting long bursts of cpu usage, this was in turn causing problems for lots of io activity.

This lovely video explains the issue clearly: -

http://www.youtube.com/watch?v=EQOmqi_n_ZY

The solution was to change from micro to small instances.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.