I want to build a webapp with Python back-end. The front-end of the webapp will just have a simple input box where the user puts an address. Then, on the back-end I want to scrape content from the webpage in that address and then process that data. This data processing might become quite heavy.

I'm quite experienced with Python, although with webapps, hosting, and so on, I have zero experience. I bought a domain name and hosting on namecheap, even though I just found out that maybe I won't be needing to use their hosting, as someone in /r/webhosting recommended me to use AWS instead. This was the methodology I was recommended to follow:

  • switch to AWS (or Azure)
  • create python lambda/function to do whatever you want to do with this request...
  • put lambda behind api-gateway
  • put Cloudfront in front of api-gateway (optional but hey..)

I have seen tutorials to point the namecheap domain into an EC2 instance. But how do I go about doing this with Lambda? Is it the same principle? Can you point me to any tutorials that might teach me how to handle this? Is Lambda actually the optimal solution for this?



When you create you API Gateway, you’ll be provided by an endpoint URL. So any HTTP request hitting that endpoint would trigger the lambda in question. The endpoint looks something like this:


Then you can pass the parameters (URL to scrape) either via path or query parameters.

To make things easier, encapsulate the API gateway via cloudfront so that api URL looks something like:



When it comes to your question on whether Lambda is the optimal solution, i would say it depends. Some clear benefits of Lambda are being serverless, it scales "infinitely" if your application is design to do so (https://docs.aws.amazon.com/lambda/latest/dg/scaling.html) having the service taking care of it, and it costs you nothing if you are not using it.

Also i recommend that you look at the limitations (https://docs.aws.amazon.com/lambda/latest/dg/limits.html), of which i would specifically look into the function timeout (15 minutes), important to know if you are planning to do processing that takes longer than that.

If you decide to indeed got for the API Gateway/Lambda combination here are some useful links:

How to setup Lambda with API Gateway: https://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started-with-lambda-integration.html)

Setup API Gateway with a custom DNS name: https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-custom-domains.html

Delegate subdomain to Route 53 (AWS' DNS service) without migrating parent domain: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/CreatingNewSubdomain.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.