I'm looking for a lightweight task queue manager for linux. The way I see it, different processes should be submitting task to the queue, the queue manager would execute them one by one in order of submission.
Is there any software that does that. I've looked at celery and geaman, but they are too heavy for the task I'm at.
Except for task spooler, I have literally not found anything else that serves this need. Tsp is great, but I am not always in a position to compile or install on every server I have to work on.
I've been struggling with this for a while, and after 3 iterations (starting from more complex to less, actually), I've come up with a pure bash implementation that appears to work fine so far. You can find it at https://github.com/sitaramc/bq.
Bq uses the fact that on Unix, "mv" (within the same file system) is an atomic operation, for all its locking needs.
It's just one bash script so installation is trivial.
The script is liberally commented and you should be able to review it in a few minutes if you wish to.
Previously I used the Berkely lpd for this (before cups, it was the default printing system for Linux). Although the queue management is implemented as a compiled daemon, the printer interface is a simple shell script (usually a wrapper around ghostview).
It's very stable and provides tools for managing the queue(s).
batch executes commands when system load levels permit.
Maybe "at" can solve your requirement already?
for i in $(seq 1 1 3); do echo "echo $i \$(date)>> /tmp/atqueue ; sleep 120 ; echo $i 2nd \$(date)>> /tmp/atqueue "|batch; done
I currently use GNU Parallel to run batches like this, which has the advantage that I can use all the machines I have access to at once. However, I need to be able to share execution resources between different users and better reporting.
