Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm running scientific software (WRF-NMM, atmosphere simulation, weather prediction). On all computers I used it, job running times of run cycles were consistent. All those machines were one socket, desktop machines. However, on dual socket computer I have VERY uneven running times. For example, my job running times equals anything between 2hr 50min to more than 5 hours. Or, we can say that standard deviation of running times is very high.

This means not very efficient use of computer resources, because I have to take into account longest running times to have system free for next cycle job, but in most time this mean having wasted resources because most of runs finish hour or two sooner.

This does not happen on other computers I have used for this jobs, and running times are consistent to few minutes differences at maximum (standard deviation of running times very low and this is good).

I can't determine cause of this high variability on dual socket computer. Do you have any idea what to look?

Additional information that can help to reduce guessing:

Computer is dedicated to this job, no other users on system, so nothing except necessary OS services runs on it. RAM is utilized less than 50% during software run (3500M of 8000M according to htop, no swapping). Computer load during software run is 8.00 most of the time, and if it is not then it vary to ~ 8.30 at short intervals at most (I don't know what brings it to ~ .30 over 8.00, but that little overshoot can't explain such large variability of job run times).

WRF software uses mpich2 (gforker in my case) to spread it's process to all 8 cores.

Computer hardware cosists of dual Xeon E5530, 8GB RAM in single channel mode (2x4GB modules) in rack case (Fujitsu Primergy RX200 R5). There is no cpuspeed or similar software that could slow down CPU frequency. Computer is located in airconditioned server room, so there is no slowing down due to high temperatures. OS is Centos 5, latest updates. No hadware defects that I can detect; dmesg clean, mcelog empty.

Finally, I'm 100% sure that variablity isn't caused by WRF sowtware itself (it's configuration).

Any idea is welcome. Many thanks!

share|improve this question
    
Profile a few runs and try to catch a good one and a bad one, then compare the profiles. –  David Schwartz Dec 25 '12 at 22:58
    
David, many thanks for your input, can you explain a bit what you mean by "profile a run"? Thanks. –  WeatherMan Dec 26 '12 at 18:53
    
I mean to gather information on what the computer is actually doing during the run to figure out specifically which operations are taking longer. If you care about performance, you should already be doing this on a regular basis. You'd be amazed how often performance is severely reduced by some tiny, easy to fix thing that you just have to look for. –  David Schwartz Dec 26 '12 at 18:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.