I'm running scientific software (WRF-NMM, atmosphere simulation, weather prediction). On all computers I used it, job running times of run cycles were consistent. All those machines were one socket, desktop machines. However, on dual socket computer I have VERY uneven running times. For example, my job running times equals anything between 2hr 50min to more than 5 hours. Or, we can say that standard deviation of running times is very high.
This means not very efficient use of computer resources, because I have to take into account longest running times to have system free for next cycle job, but in most time this mean having wasted resources because most of runs finish hour or two sooner.
This does not happen on other computers I have used for this jobs, and running times are consistent to few minutes differences at maximum (standard deviation of running times very low and this is good).
I can't determine cause of this high variability on dual socket computer. Do you have any idea what to look?
Additional information that can help to reduce guessing:
Computer is dedicated to this job, no other users on system, so nothing except necessary OS services runs on it. RAM is utilized less than 50% during software run (3500M of 8000M according to htop, no swapping). Computer load during software run is 8.00 most of the time, and if it is not then it vary to ~ 8.30 at short intervals at most (I don't know what brings it to ~ .30 over 8.00, but that little overshoot can't explain such large variability of job run times).
WRF software uses mpich2 (gforker in my case) to spread it's process to all 8 cores.
Computer hardware cosists of dual Xeon E5530, 8GB RAM in single channel mode (2x4GB modules) in rack case (Fujitsu Primergy RX200 R5). There is no cpuspeed or similar software that could slow down CPU frequency. Computer is located in airconditioned server room, so there is no slowing down due to high temperatures. OS is Centos 5, latest updates. No hadware defects that I can detect; dmesg clean, mcelog empty.
Finally, I'm 100% sure that variablity isn't caused by WRF sowtware itself (it's configuration).
Any idea is welcome. Many thanks!
