0

My site receives numerous uploads and I can't afford to waste cycles and time processing them on the spot (e.g., trimming, optimizing, resizing, etc.). What technologies exist to delay and queue these non-essential commands for execution at a later time? Upon execution I would also like to use cpulimit to allocate a certain % and also limit memory usage.

Running Ubuntu 10.04 LTS

1
  • I edited your question to make it less like a shopping question and less subjective, both of which are off-topic.
    – EEAA
    Sep 27, 2012 at 20:19

1 Answer 1

0

Well, I'm not sure what the "best" is. You'll need to decide that for yourself.

You need some sort of message queue. There are many of these around: MSMQ, Gearman, RabbitMQ, Amazon SQS, etc.

Myself? I use RabbitMQ for communication between disparate parts of our application, and to queue up actions that should not be performed synchronously. It takes a bit of time to wrap your head around the difference between exchanges and queues, and to figure out how best to arrange things for your environment, but after that, it's a piece of cake to use.

3
  • Thanks. Reading about RabbitMQ now. I take it you run it in tandem with cron, which check to see if there are any messages queued and then executed them if there are?
    – MarkL
    Sep 27, 2012 at 22:00
  • For delayed processing, yes you could do it that way, yes. Or, you have an RMQ consumer that is always bound and "listening" to the queue for messages. When a message is placed in the queue, it can act immediately.
    – EEAA
    Sep 27, 2012 at 22:50
  • OK. You put me on the right path. I zeroed in on Redis + Resque, especially since I already use Redis as a memcache respacement. Thanks.
    – MarkL
    Sep 27, 2012 at 23:31

Not the answer you're looking for? Browse other questions tagged .