Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

My scripts are using wget to retrieve data from Internet. When many users are using this script I got very high load (about 20.00), because of disk I/O. Wget is automatically started each hour by cron. I would like to limit one wget to one customer at the same time. How to do this ?

I'm using CentOS 5.7 64-bit.

share|improve this question
Do the solution have to be secured against malicious local users? Do you expect that they would try to circumvent this limit? – kupson Feb 13 '12 at 15:01
No. I just see a lot of wgets running and I have made a script which kills them if load is higher than 19, but that's not good for me. – Spacedust Feb 13 '12 at 19:30
up vote 2 down vote accepted

You could add a file lock to your script. For example with flock command (package util-linux on Debian):

    flock --nb 200 || echo "waiting for previous script completion"
    flock 200

    # script commands here
    wget ....

) 200>$HOME/.wgetlock
share|improve this answer
It's a php script, so this might be hard to implement. – Spacedust Feb 13 '12 at 20:46
PHP script executed by cron? Then create small wrap-around script as shown above and call PHP script from it (replace wget ....), or use flock directly in PHP. – kupson Feb 13 '12 at 21:34

Check out the options available to you in /etc/security/limits.conf. You can set per-user process and memory limits there.

share|improve this answer
It's for all processes of user, not the one wget script. – kupson Feb 13 '12 at 15:00
Something that will limit one cronjob running at the same time per user will be fine too. – Spacedust Feb 13 '12 at 16:55

If it is the same request and content over and over, just cache the result for a few seconds and serve up the same content for different users.

You could use memcache to implement this easily enough.

If the content differs per user be aware that queueing like you ask will cause long delays.

share|improve this answer
I think that this script is executed by cron, not in a web server. – kupson Feb 13 '12 at 21:37
Yes. By cron. We use memcached already + varnish, but this is only a partial solution. – Spacedust Feb 14 '12 at 22:42

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.