TL;DR: CPU-hungry processes are sharing cores on a multi-core server while other cores sit idle, whereas I expect the Linux scheduler to distribute jobs evenly across cores. What could cause this behavior?
My workgroup has a new 4-core server running SuSE that is experiencing some strange CPU-scheduling behavior that our admin (the guy who set up the box) didn't know about and can't seem to fix. I'm hoping you all can help me diagnose what might be causing this strange behavior. Here are the symptoms:
1) Every SSH session seems to be limited to using a single core. I have tested this a variety of ways, but the simplest was creating a simple infinite-looping C program and running multiple copies. They always share one core if started from a single SSH session, and I can't even control which core; it seems to be set at login time. Furthermore, even using multiple simultaneous SSH sessions, the only cores that I can ever utilize this way are cores 0 and 3; 1 and 2 never get touched, no matter how many sessions or processes I start.
2) If, from an SSH session like the ones mentioned above, I start my program with "nohup" to divorce it from the current session, it will use a different core than the rest of the programs started from that session. However, all nohup-started programs from the same SSH session will again share a core with each other. Interestingly, these nohup-started programs are always assigned to cores 1 and 2.
The expected behavior, of course, is what I've always seen on other Linux systems (I'm mostly familiar with RHEL, Fedora, and Ubuntu): I should be able to utilize all 4 cores from a single session with or without using nohup; furthermore, jobs should occasionally jump cores to balance the load time evenly between them.
Here's an example of 2 processes running on one core: http://i.imgur.com/K9rH3.png (Sorry, can't post images directly yet, even though I've got enough cred on other StackExchange sites). Note in the above that each "burn" process takes up 100% of one core if run in isolation, but here they are sharing a core for some reason while three cores sit idle. Also note that these two processes shared the "Cpu3" core in excess of 20 minutes without changing to another core to balance the load (this was after the image was taken; after 20 minutes I stopped watching).
My first thought when encountering this problem was that "ulimit" was in effect, but it doesn't look like it to my (admittedly non-expert) eye:
dmonner@bass:~> ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 128465 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) 13988192 open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 128465 virtual memory (kbytes, -v) 40007280 file locks (-x) unlimited
Additionally, /etc/security/limits.conf is empty (or at least has no lines that aren't comments), so I don't think it's PAM enforcing limits.
I have sudo access on the machine, and even if I 'sudo su' and then run processes, I am core-limited in the same way.
So: Does anyone have ideas as to what can cause this kind of behavior, and how to get rid of it?