Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I would like to control users programs execution based on server load. Each user uses bash. Are there way to make a some kind of queue where execution for the commands from command line will be pended until resources will be available?

thanks Arman.

share|improve this question
add comment

3 Answers

up vote 4 down vote accepted

Arman, the paradigm you are suggesting was the rigid time sharing model of mainframes. In the mainframe, one process would dominate the machine until it either completed or was suspended. As time progressed, people got tired of runaway jobs and kernels started butting in more frequently, putting processes on hold so that other processes could have a turn. This feature was called a scheduler. Work with me for a minute.

Lots of work goes into the scheduler so that everyone who uses a system feels like the system is there to do their bidding, even if only one process gets a turn at bat at a time. Now with multiple CPUs and later cores, the computer really can do more than one thing at a time. On multi-core systems, the concept of the computer being "busy" is a little fuzzier. In Linux, processes are assigned a CPU and have an affinity for it when they are paused and resumed, although the scheduler reserves the right to move the process to another core.

So, your real requirement could be expressed in this scenario. Three people have been granted access to your new system: Michelle, Nick, and Orville. All three people would love to see how fast the computer can render a complicated frame in a famous raytracer, POV-Ray. Michelle is your lead developer and the boss said she can use the machine any way she wants. Nick is one of your junior developers and the boss said he can work as long as he doesn't interfere with Michelle's occasional frame rendering. Nick is more excited about the machine and has all kinds of fun queued up. Orville works in the graphics department and the boss only wants him to see the pictures and scene files, but he doesn't want Orville to interfere with any rendering. To keep it neat and orderly, we will have three user groups: staff, newbie, and outsider. I don't recommend using those names unless you want to be added to the last group. An excerpt from /etc/group might read:

  staff:1001::michelle,theboss
  newbie:1002::nick,alode,ian
  outsider:1003::orville
  raytracers:1004::staff,newbie

We will use a two-prong approach to solve this problem. The first prong will be limits and the second will be simple file permissions (chmod and chgrp).

First we'll mess with limits. The pluggable authentication module (PAM) enforces these through its pam_limits module. We'll make sure that it is configured correctly below. Look at the man page for limits.conf for details on the kind of restrictions you can enforce. Two pieces of terminology are important here: soft limits and hard limits. The soft limit should always be smaller than the hard limit. The hard limit is an upper boundary that cannot be changed once it is set. Soft limits are used if the user would like a warning that they are about to get squished and also serve as the default setting if the user is permitted to change it with the ulimit command. The details on ulimit are probably in the man page for your shell (e.g. man bash). We are interested in the hard limit because we will use it for the The Enforcer who will do the dirty work. Experiment with ulimit for a while until you understand the effects and see that the numbers are something you understand.

Now comes the tricky part. Tricky because if you get it wrong, either Michelle or Nick will come complaining to you or to your boss. Michelle might complain that Nick is interfering and Nick might complain that he can't get any time on the new system. Notice that this is not a technical problem, but rather your best guess at interpreting the boss's desire and translating it into system configuration. Communication on what limits are in place might limit your pain.

So let us set these limits on Nick and Orville in a way that will not upset Nancy. We'll detail just the newbie group settings.

  1. Maximum scheduling priority set to 15. 20 is the default and is the lowest priority.
  2. Maximum amount of memory consumed is restricted to 1 GB. Given that this is raytracing, this limits the complexity of the scene.

An important note is that we intend to set the default priority of all staff jobs to 15 with a maximum of 10. Yes, in this case, 15 is less than 10 because 20 is the lowest priority and 0 is the highest for a non-root user. It's easier to grasp if you stand on your head.

I need to mention that I am using and Ubuntu box and your mileage may vary with how distributions have PAM set up, specifically, pam_limits. I have this line on my system, session required pam_limits.so in these files:

  • /etc/pam.d/login
  • /etc/pam.d/sshd
  • /etc/pam.d/cron
  • /etc/pam.d/at
  • /etc/pam.d/su
  • /etc/pam.d/sudo
  • /etc/pam.d/gdm
  • /etc/pam.d/gdm-autologin

Then, let us put our restrictions into either /etc/limits.conf or into individual files in /etc/limits.d/. If this is a once-in-a-lifetime deal, then put it into /etc/limits.conf, but the wise thing would be to create a file like /etc/limit.d/raytracers and put these lines into it (we'll restrict Orville while we're at it). See the man page for limits.conf for details and examples are in /etc/limits.conf.

 @newbie hard data 1024000
 @newbie soft priority 20
 @newbie hard priority 15
 @newbie hard maxlogins 1
 @outsider hard data 102400
 @outsider soft 20
 @outsider hard 20
 @outsider hard maxlogins 1
 @staff hard priority 10
 @staff soft priority 15

The users need to log out and back in again for the changes to take effect. You are going to be smart though and test these limitations out before handing them over to the users. Use the ulimit command to check that the settings are getting applied. The maxlogins is needed because the limits are applied per-login shell and a user could get around them by logging in multiple times.

Okay, we finally have the users locked down so that the staff always gets a turn at the machine. Now comes the observer. This is tough because somehow you have to figure out what it means to be an observer. You've got a few hundred standard commands you don't want to lock out. So either you want to protect against selected applications being run or you want to force them to run a single, main application. Adding X-Windows (GNOME or KDE) really, really complicates this and will take a substantial amount of effort on your part. For this reason, you may want to just restrict how much CPU time they can take (through limits, of course) and leave it at that.

But, maybe you paid big money for access to POV-Ray and that's what the boss wants to keep the observers from running. Just do a chgrp -R raytracers /directory/with/application/and/settings. Then, ensure that the top tree has permission of 750 so that an observer can't even go into the directory and run the program.

If you only want the observer to run a particular application, then you have two choices. If it is strictly a text environment, then make their shell the name of the program and make sure the program is in /etc/shells. You might have to make some exceptions with PAM modules to keep the user from doing an scp or ftp. If it's a graphical environment, you will have to tinker with your windowing environment. Graphical environments are fairly complex to lock down completely and will take some experimentation on your part.

Here's a test question: How did we force staff jobs to always dominate jobs from newbies and observers? (Hint: look at our entries in /etc/limit.d/raytracers)

share|improve this answer
    
@zerolagtime: Thank you for nice explanation. I will experiment starting with terminal. Unfortunately current model allows users to login with vnc(that was decided by boss:) ). May be I can provide vnc only for advanced users. Concerning to you test :) the boss can always nice to 14 his jobs :) –  Arman Oct 3 '10 at 7:10
    
VNC will use the gdm method in PAM, thereby enforcing the limits. Unless all the users are connecting to the host through vnc and connecting to the same user account. If all of the users are sharing a single account, then you have no control. They have to use their login/password in order for my scheme to be effective. –  zerolagtime Oct 3 '10 at 13:57
    
Arman, Thanks for asking this question. I learned more about how users and daemon processes can be constrained. I had no reason to learn this before you asked. I learned something too. My initial plan was to modify the log in script, but that was a hack. I like what I found in the pam_limits module quite a bit. PAM is full of interesting corners. –  zerolagtime Oct 3 '10 at 17:54
    
Are there way to control IO also via pam? I did not find it, but I found that kernel supports cgroup so one can play with that as well. –  Arman Oct 4 '10 at 7:10
    
(Just for information if you like the question you can up-vote it.) –  Arman Oct 4 '10 at 7:11
show 3 more comments

Use the at / batch command.

(examples)

share|improve this answer
    
thanks for mentioning batch. Is it possible to make a transparent the usage of batch for users? Is it possible to catch any command line from bash and sent it to batch? –  Arman Oct 2 '10 at 10:09
    
You could create your own interactive shell that just sent each option to a batch command. It would wait for that job to exit the queue (atq) and then spit out the captured stdout/stderr. Rather complicated if you ask me. –  zerolagtime Oct 3 '10 at 2:06
    
Another option is a queuing system specifically geared towards a user on a lightweight workstation submitting jobs to either a compute cluster or a system that is just larger. Condor is just one cluster software that can accommodate job distribution and will take system load into account. The man page to batch also mentions NQS. Either way, interactive delays that wait until the system load goes down will not sit will with your users. Human Factors essentials. You'd be better off giving them a farm of systems to use. –  zerolagtime Oct 3 '10 at 2:06
add comment

Arman,

I've thought about your question for a couple minutes and I'd like to suggest that it sounds like you might be on wrong track entirely. This just doesn't fit the mold of how 'nix was designed to run.

A shell typically spawns TONS of little processes; things pipe input and output from one to the next and any delay in this system will make it almost unusable. If your system resources are such that users cannot run grep without their process getting queued up, they are just going to go crazy.

I would suggest looking into other roads. One would be experimenting with "nice" levels on a per process or per user basis. If your users are running jobs that take huge amounts of resources, perhaps limited the ways they can launch those particular jobs to be queued would be in order, but the way the jobs themselves function should be updated for this use case.

Another thing to look into would be isolating your users in virtual environments using vserver or some other paravirtual guest thereby allowing them a limited slice of the over all server resources.

Hope this helps you a little bit. If you shared more information about what kind of tasks people are running that consume so much resources maybe people could offer more precise solutions. As it stands, I just wanted to say that it really sounds like you are trying to solve a problem with the wrong tools, sort of like preventing flats on your car by taking the wheels off and dragging it along like a sled. Catch my drift?

Caleb

share|improve this answer
    
@caleb: Thank you for the nice explanation. Probably I should rephrase my question as following: Assuming we have a porsche from private collection and we would like to provide it for the driving school. The users should be divided to the experts, beginners and guests. The expert can use as it is with full power,beginners only start the motor, lights and stop it, the guests can only get in an get out: eg only observe. SO now the question can we organise similar model for the computing pool? It is important to provide full functionality for experts, and keep the machine on-line. –  Arman Oct 2 '10 at 12:58
    
Maybe one can try to filter all user defined processes and manage them? But how? –  Arman Oct 2 '10 at 13:02
    
@arman: I think your analogy still breaks down. You're trying to dump guest and beginner users in an F1 car that is already going 150mph around the track DURING a race. It's just not a good idea. You don't give bumbling newbies shell accounts on production boxes. –  Caleb Oct 2 '10 at 13:21
    
Things you can think about however would be creating chroot environments so that users in the newbie group would only get a small subset of commands and a sort of sand box world to play in instead of being on the main track. Also perhaps just monitoring the process list of some users for long running or resource heavy jobs using some basic ps comands so you can nice or kill of unwanted jobs. I still assert that if you are more specific with what kind of jobs your users need to run we could offer better suggestions. –  Caleb Oct 2 '10 at 13:23
    
>You don't give bumbling newbies shell accounts on production boxes. Actually that what we do. The users are experts in the data but they are not an experts in usage of computing node. We are providing all resources/possibilities to learn and to become as an expert. –  Arman Oct 4 '10 at 7:52
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.