Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

TL;DR: CPU-hungry processes are sharing cores on a multi-core server while other cores sit idle, whereas I expect the Linux scheduler to distribute jobs evenly across cores. What could cause this behavior?

My workgroup has a new 4-core server running SuSE that is experiencing some strange CPU-scheduling behavior that our admin (the guy who set up the box) didn't know about and can't seem to fix. I'm hoping you all can help me diagnose what might be causing this strange behavior. Here are the symptoms:

1) Every SSH session seems to be limited to using a single core. I have tested this a variety of ways, but the simplest was creating a simple infinite-looping C program and running multiple copies. They always share one core if started from a single SSH session, and I can't even control which core; it seems to be set at login time. Furthermore, even using multiple simultaneous SSH sessions, the only cores that I can ever utilize this way are cores 0 and 3; 1 and 2 never get touched, no matter how many sessions or processes I start.

2) If, from an SSH session like the ones mentioned above, I start my program with "nohup" to divorce it from the current session, it will use a different core than the rest of the programs started from that session. However, all nohup-started programs from the same SSH session will again share a core with each other. Interestingly, these nohup-started programs are always assigned to cores 1 and 2.

The expected behavior, of course, is what I've always seen on other Linux systems (I'm mostly familiar with RHEL, Fedora, and Ubuntu): I should be able to utilize all 4 cores from a single session with or without using nohup; furthermore, jobs should occasionally jump cores to balance the load time evenly between them.

Here's an example of 2 processes running on one core: http://i.imgur.com/K9rH3.png (Sorry, can't post images directly yet, even though I've got enough cred on other StackExchange sites). Note in the above that each "burn" process takes up 100% of one core if run in isolation, but here they are sharing a core for some reason while three cores sit idle. Also note that these two processes shared the "Cpu3" core in excess of 20 minutes without changing to another core to balance the load (this was after the image was taken; after 20 minutes I stopped watching).

My first thought when encountering this problem was that "ulimit" was in effect, but it doesn't look like it to my (admittedly non-expert) eye:

dmonner@bass:~> ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 128465
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) 13988192
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 128465
virtual memory          (kbytes, -v) 40007280
file locks                      (-x) unlimited

Additionally, /etc/security/limits.conf is empty (or at least has no lines that aren't comments), so I don't think it's PAM enforcing limits.

I have sudo access on the machine, and even if I 'sudo su' and then run processes, I am core-limited in the same way.

So: Does anyone have ideas as to what can cause this kind of behavior, and how to get rid of it?

share|improve this question

1 Answer 1

The Linux scheduler will (if the application doesn't do it on it's own) never bind anything to any specific core. Quite the contrary actually, most of the time the scheduler will bounce the load across all cores at a seemingly random fashion. It is the only part of the kernel which has a kind of holistic view of the system and it tries to "optimize" system response/throughput according some heuristics which in term manifests itself as the kernel bouncing the processes across the cores.

You have to explicitly pin processes to certain cores via something like taskset(1) to accomplish that. Can you provide some evidence for what you are experiencing?

share|improve this answer
    
I checked some of my running processes via 'taskset -p <pid>' and they all came back with "f", which I assume is a 4-bit hex mask that says all 4 cores are okay for that process. When I watch individual core utilization in 'top' (after pressing 1 to show all cores), I see that the process never bounces around; it always stays on the same core. I'm unsure how my application could peg itself to a core; my simple while(1) loop in C doesn't do this explicitly. This behavior applies to lots of programs I've tried to run, in many languages. –  dmonner Dec 6 '11 at 0:30
    
Linked an image in the question to show you what 'top' is showing me; let me know if you need more or better evidence! –  dmonner Dec 6 '11 at 0:54
    
Is the linux kernel so stupid? Normally it makes sense these days to limit activity to few cores to allow the others to turn off and save energy... and enable... BOOST functionalty in the processor. As such, current schedulers should NOT have threads hop at random but use few selected cores preferably. –  TomTom Dec 6 '11 at 14:51
    
Well it is best judgement on the kernel side - coming from HPC we have automatic (NUMA) topology aware socket binding and pinning for all running processes. Gives as a few % better run time across a day. –  pfo Dec 6 '11 at 15:04
    
@OP: did you check your PAM settings for SSH and what /etc/security/* is telling you? –  pfo Dec 6 '11 at 15:06

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.