There are many open source Message queues for Linux, and I need some help deciding what I should go for.
My problem is simple - I get sent a list of files that needs to be processed. Each job can't be split up, but they are self contained and can be spread to multiple computers.
I'm thinking of solving this using a message queue. Multiple clients send a message to a central queue. Each queue has a number of subscribers that will take jobs from that queue when they have finished processing the current job.
Ideally it should have the following qualities
- Message queue must be able to store unprocessed messages in case of a shutdown/reboot
- A job can only be processed by a single subscriber (don't want duplicate jobs)
- The subscribers should be able to send jobs of their own, that will be processed by a different set of subscribers.
Can anyone suggest a simple to use message queue?