Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I tapped in to Amazons AWS offerings and please explain this in high level - if I am thinking right.

So I have few Python scraping scripts on my local machine. I want to use AWS for super fast internet connectivity and cheaper price - win / win!

  • I understand that I can deploy a centOS/Ubuntu instance on EC2. Install the necessary Python libraries. Start and stop instances using boto (Python) to save costs. Am I thinking right so far? (Is it feasible?)

  • I will CRON some scripts that will start fetching (scraping) HTML files for parsing later on. So these HTML files are copied over to S3 for storage (or shall I dump them to my local machine since that is how I will be parsing and storing in MySQL?).

Please advise if I make any sense with my assumptions and the little knowledge I have of AWS with my few hours of reading/Googling about the service.

share|improve this question
up vote 5 down vote accepted

The basic premise of your setup seems fine, however, there are a few items that you may want to factor in.

Firstly, EC2 network (and I/O) bandwidth is dependant on instance type. If you are hoping to use t1.micro instances do not expect 'super fast internet connectivity' - even with an m1.small, you may not see the performance you are looking for. Also, keep in mind that you pay for bandwidth used on EC2 (and not just for instance time).

With regard to your first point, there should be no real difficulty in setting up Python on an EC2 instance. However, the potential difficulty arises from coordinating your instances. For example, if you have 2 instances running, how will you split the task between them? How will each instances 'know' what the other has done (presuming you aren't going to manually partition a list of URLs). Moreover, if you are launching an instance, will one of the EC2 instances be responsible for handling that or will your local machine deal with it (if it is one of the EC2 instances, how do you determine which instance will be responsible for the task (i.e. to prevent the 'launch' task being executed by every instance) and how do you redistribute the tasks to include the new instance? How do you determine which instance(s) to terminate automatically?

Undoubtedly, all of the above are possible (corosync/heartbeat, pacemaker, auto-scaling, etc.) but easy to overlook initially. Regardless, if you are looking for the 'best price' you will probably want to go with spot instances (as opposed to on-demand), however, for that to work, you do need a fairly robust architecture. (It is worth noting that the spot prices fluctuates significantly - at times exceeding the on-demand price; depending on the time-scale over which you are working, you will either want to set a low upper spot price, or determine the best approach (spot/on-demand) on a regular (hourly) basis to minimize your costs.) Although, I can't confirm it at the moment, the simplest (and cheapest) option may be AWS' auto-scaling. You need to setup Cloudwatch alarms (but Cloudwatch does provide 10 free alarms) and auto-scaling itself does not have a cost associated with it (other than the cost of new instances and the Cloudwatch costs).

Given that I really have no idea of the scope of your undertaking, I might ask why not simply use EC2 for the parsing and processing. Especially if the parsing is complex, the pages can be fetched faster than they can be processed, and you have a large number of pages (presumable, otherwise you wouldn't be going to through the effort of setting up AWS), it might be more efficient to simply process the pages on EC2, and when everything is done, download a dump of the database. Arguably, this might simplify things a bit - have one instance running MySQL (with the data stored on an EBS volume), each instance queries the MySQL instance for the next set of records (and perhaps marks those as reserved), fetches and processes, and saves the data to MySQL.

If you are not going to run MySQL on EC2, you can either store your HTML files on S3, as you have mentioned, or can go with saving them on an EBS volume. The advantage of S3 is that you don't need to pre-allocate storage (especially useful if you don't know the size of the data you are dealing with) - you pay for PUTs/GETs and storage; the downside is speed - S3 is not meant to be used as a filesystem, and (even though you can mount it as a filesystem) it would be fairly inefficient to be saving each individual file to S3 (as in you will want to accumulate a few pages and them upload them to S3). Additionally, if you have a large volume of files (tens of thousands) the processing of fetching all the filenames, etc. can be slow. EBS volumes are meant to be used as storage attached to an instance - the advantage is in speed - both transfer rates and the fact it has a 'filesystem' (so reading a list of files, etc. is quick) - EBS volumes persist beyond instance termination (except for EBS root volumes, which do not by default (but can be made to)). The downsides of EBS volumes are that you have to pre-allocate a quantity of storage (which cannot be modified on the fly) - and you pay for that amount of storage (regardless of whether all of it is in use); you also pay for I/O operations (also, the performance of EBS volumes is dependent on network speed - so larger instances get better EBS performance). The other advantage of EBS is that, being a filesystem, you can perform a task like gzipping the files very easily (and I imagine that if you are downloading a lot of html pages you will not want to be fetching individual files of S3 later on).

I am not really going to speculate on the possibilities (keeping in mind that at a very large scale, something like map-reduce/hadoop would be used to manage this kind of task), but as long as you have an approach for partitioning the task (e.g. MySQL instance) and managing the scaling of instances (e.g. auto-scaling), the idea you have should work fine.

share|improve this answer
Can you point me to the documentation about limiting bandwidth according to instance type? This is the first time I see it, and I'd like to confirm it. – Hristo Deshev Sep 29 '11 at 9:02
Just to clarify, by 'bandwidth' I mean transfer-rate/throughput. It is mentioned in a few places, two were easy to find. "Because Amazon EBS volumes require network access, you will see faster and more consistent throughput performance with larger instances." AWS EBS Also, the 'I/O performance' on the instance types page refers to network I/O performance (see, for example, the cluster compute instance types which note '10GbE' against I/O performance) – cyberx86 Sep 29 '11 at 11:45
Thanks! The "instance types" page clears it for me. Something else crossed my mind - if you are running small instances you face the chance of being run on a hardware host with other small instances. And maybe those can saturate the host network interfaces and hurt EBS I/O performance. I remember reading somewhere that by using large instances you are least likely to run on a host machine that's under load from other users. – Hristo Deshev Sep 29 '11 at 13:00
Thank you so much for such an elaborate response. It definitely makes me think/explore different areas/possibilities. I was initially thinking of downloading a text file of URLs on EC2 since MySQL can be expensive and I guess can be avoided. The HTML files can range from few 1000s to 10000s. Each instances are independent because the text file with URLs take care of it. EBS volume makes sense, gZipping a nightly download of files and downloading to my local machine for further processing since parsing it is pretty elaborate. I didn't know hadoop can be applied here? Never used hadoop.... – ThinkCode Sep 29 '11 at 13:31
Didn't get a straightforward answer on the web about this - can IP addresses be rotated on the fly since it is scraping and the IPs can be throttled I suppose? (I am a lil' confused with the FREE stuff on AWS. 720 hours, some bandwidth,... Spot instances make sense too since this isn't mission critical tasks and more emphasis on costs. $1 an hour is still a lot of money for me! Thorough testing will answer the pricing Qs I guess)! – ThinkCode Sep 29 '11 at 13:36

You can interact with different instance via SQS. Its a queueing service. You can queue the input url to SQS. Each instance will take url from SQS sequentially. But SQS wont give same input to multiple instance. Thats the main advantage here..

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.