The service I am building the architecture for has a video processing part in it. Basically, some people can upload videos (stored in S3) that will be split into frames with high resolution. From these high resolution frames, derived images (thumbnails, sprites) are built, and all of this is to be stored on S3.
My question is: where and how should I run my video processing script? I would like to use Lambda, but I'm not sure its limitations (300s timeout and 512 MB disk storage) allow me to do so. Note that the videos can weight up to several GB.
The solution I'm currently thinking about is:
- read the video from S3 using a stream (so it doesn't have to be fully copied)
- convert the video into frames, and directly upload them to S3 (I also have to make sure not to fill up the storage with the frames)
- read individually or by chunks the frames from S3, and use them to generate thumbnails and sprites, that are uploaded to S3. I'm a little bit worried here about the time that this could take, even if my Lambda is in the same region as my S3 bucket.
Do you think this is the best solution? Can you think of a simpler one? Or do you think it's a better idea to run the processing on a standard EC2 instance (maybe with some queuing system like SQS in the middle)?