Queue based shell in linux?

Question

I would like to control users programs execution based on server load. Each user uses bash. Are there way to make a some kind of queue where execution for the commands from command line will be pended until resources will be available?

thanks Arman.

Answer 1

Arman, the paradigm you are suggesting was the rigid time sharing model of mainframes. In the mainframe, one process would dominate the machine until it either completed or was suspended. As time progressed, people got tired of runaway jobs and kernels started butting in more frequently, putting processes on hold so that other processes could have a turn. This feature was called a scheduler. Work with me for a minute.

Lots of work goes into the scheduler so that everyone who uses a system feels like the system is there to do their bidding, even if only one process gets a turn at bat at a time. Now with multiple CPUs and later cores, the computer really can do more than one thing at a time. On multi-core systems, the concept of the computer being "busy" is a little fuzzier. In Linux, processes are assigned a CPU and have an affinity for it when they are paused and resumed, although the scheduler reserves the right to move the process to another core.

So, your real requirement could be expressed in this scenario. Three people have been granted access to your new system: Michelle, Nick, and Orville. All three people would love to see how fast the computer can render a complicated frame in a famous raytracer, POV-Ray. Michelle is your lead developer and the boss said she can use the machine any way she wants. Nick is one of your junior developers and the boss said he can work as long as he doesn't interfere with Michelle's occasional frame rendering. Nick is more excited about the machine and has all kinds of fun queued up. Orville works in the graphics department and the boss only wants him to see the pictures and scene files, but he doesn't want Orville to interfere with any rendering. To keep it neat and orderly, we will have three user groups: staff, newbie, and outsider. I don't recommend using those names unless you want to be added to the last group. An excerpt from /etc/group might read:

  staff:1001::michelle,theboss
  newbie:1002::nick,alode,ian
  outsider:1003::orville
  raytracers:1004::staff,newbie

We will use a two-prong approach to solve this problem. The first prong will be limits and the second will be simple file permissions (chmod and chgrp).

First we'll mess with limits. The pluggable authentication module (PAM) enforces these through its pam_limits module. We'll make sure that it is configured correctly below. Look at the man page for limits.conf for details on the kind of restrictions you can enforce. Two pieces of terminology are important here: soft limits and hard limits. The soft limit should always be smaller than the hard limit. The hard limit is an upper boundary that cannot be changed once it is set. Soft limits are used if the user would like a warning that they are about to get squished and also serve as the default setting if the user is permitted to change it with the ulimit command. The details on ulimit are probably in the man page for your shell (e.g. man bash). We are interested in the hard limit because we will use it for the The Enforcer who will do the dirty work. Experiment with ulimit for a while until you understand the effects and see that the numbers are something you understand.

Now comes the tricky part. Tricky because if you get it wrong, either Michelle or Nick will come complaining to you or to your boss. Michelle might complain that Nick is interfering and Nick might complain that he can't get any time on the new system. Notice that this is not a technical problem, but rather your best guess at interpreting the boss's desire and translating it into system configuration. Communication on what limits are in place might limit your pain.

So let us set these limits on Nick and Orville in a way that will not upset Nancy. We'll detail just the newbie group settings.

1. Maximum scheduling priority set to 15. 20 is the default and is the lowest priority.
2. Maximum amount of memory consumed is restricted to 1 GB. Given that this is raytracing, this limits the complexity of the scene.

An important note is that we intend to set the default priority of all staff jobs to 15 with a maximum of 10. Yes, in this case, 15 is less than 10 because 20 is the lowest priority and 0 is the highest for a non-root user. It's easier to grasp if you stand on your head.

I need to mention that I am using and Ubuntu box and your mileage may vary with how distributions have PAM set up, specifically, pam_limits. I have this line on my system, session required pam_limits.so in these files:

• /etc/pam.d/login
• /etc/pam.d/sshd
• /etc/pam.d/cron
• /etc/pam.d/at
• /etc/pam.d/su
• /etc/pam.d/sudo
• /etc/pam.d/gdm
• /etc/pam.d/gdm-autologin

Then, let us put our restrictions into either /etc/limits.conf or into individual files in /etc/limits.d/. If this is a once-in-a-lifetime deal, then put it into /etc/limits.conf, but the wise thing would be to create a file like /etc/limit.d/raytracers and put these lines into it (we'll restrict Orville while we're at it). See the man page for limits.conf for details and examples are in /etc/limits.conf.

 @newbie hard data 1024000
 @newbie soft priority 20
 @newbie hard priority 15
 @newbie hard maxlogins 1
 @outsider hard data 102400
 @outsider soft 20
 @outsider hard 20
 @outsider hard maxlogins 1
 @staff hard priority 10
 @staff soft priority 15

The users need to log out and back in again for the changes to take effect. You are going to be smart though and test these limitations out before handing them over to the users. Use the ulimit command to check that the settings are getting applied. The maxlogins is needed because the limits are applied per-login shell and a user could get around them by logging in multiple times.

Okay, we finally have the users locked down so that the staff always gets a turn at the machine. Now comes the observer. This is tough because somehow you have to figure out what it means to be an observer. You've got a few hundred standard commands you don't want to lock out. So either you want to protect against selected applications being run or you want to force them to run a single, main application. Adding X-Windows (GNOME or KDE) really, really complicates this and will take a substantial amount of effort on your part. For this reason, you may want to just restrict how much CPU time they can take (through limits, of course) and leave it at that.

But, maybe you paid big money for access to POV-Ray and that's what the boss wants to keep the observers from running. Just do a chgrp -R raytracers /directory/with/application/and/settings. Then, ensure that the top tree has permission of 750 so that an observer can't even go into the directory and run the program.

If you only want the observer to run a particular application, then you have two choices. If it is strictly a text environment, then make their shell the name of the program and make sure the program is in /etc/shells. You might have to make some exceptions with PAM modules to keep the user from doing an scp or ftp. If it's a graphical environment, you will have to tinker with your windowing environment. Graphical environments are fairly complex to lock down completely and will take some experimentation on your part.

Here's a test question: How did we force staff jobs to always dominate jobs from newbies and observers? (Hint: look at our entries in /etc/limit.d/raytracers)

Answer 2

Use the at / batch command.

(examples)

Answer 3

Arman,

I've thought about your question for a couple minutes and I'd like to suggest that it sounds like you might be on wrong track entirely. This just doesn't fit the mold of how 'nix was designed to run.

A shell typically spawns TONS of little processes; things pipe input and output from one to the next and any delay in this system will make it almost unusable. If your system resources are such that users cannot run grep without their process getting queued up, they are just going to go crazy.

I would suggest looking into other roads. One would be experimenting with "nice" levels on a per process or per user basis. If your users are running jobs that take huge amounts of resources, perhaps limited the ways they can launch those particular jobs to be queued would be in order, but the way the jobs themselves function should be updated for this use case.

Another thing to look into would be isolating your users in virtual environments using vserver or some other paravirtual guest thereby allowing them a limited slice of the over all server resources.

Hope this helps you a little bit. If you shared more information about what kind of tasks people are running that consume so much resources maybe people could offer more precise solutions. As it stands, I just wanted to say that it really sounds like you are trying to solve a problem with the wrong tools, sort of like preventing flats on your car by taking the wheels off and dragging it along like a sled. Catch my drift?

Caleb