Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Hi I have a small test network that I'm using to do some number crunching for my dissertation. My network consists of three machines with intel gigabit network cards (all em drivers with two hosts setup as laggs) with default MTU settings etc... running Freebsd 9.1 and mpich2.

when I run mpdringtest I get latencies of:

time for 1   loops = 0.00105500221252 seconds
time for 2   loops = 0.101227998734 seconds
time for 3   loops = 0.200784921646 seconds
time for 4   loops = 0.300806045532 seconds
time for 10  loops = 0.942656040192 seconds
time for 100 loops = 11.5054900646 seconds

the network latency is:

Max Latency:
from \ to     city17    glados     blackmesa
city17                  0.335      0.739
glados        0.319                0.689 
blackmesa     0.992     0.746 

Mean Latency:
from \ to     city17    glados     blackmesa
city17                  0.205      0.525 
glados        0.195                0.518 
blackmesa     0.599   0.503

for hosts
city17:8
glados:4
blackmesa:4

Based on my assumptions each loop should take approximately about the sum of the upper or lower triangular portion of the mean latency matrix (attained via ping). This seems to match the one ring results. However for more than one ring the latency seems to increase considerably. I read the manpage on this topic and it didn't really explain much besides saying it passed a message around in a ring. Are these results fairly normal w.r.t. latency and mdpringresults?

The switch is an HP procurve 1810g-8 if it matters (no other traffic is running through the switch).

share|improve this question

1 Answer 1

Rough numbers from my experience: Gigabit gives you about 50µs (0.50ms) on the physical layer and with "correct" interrupt handling and on current machines/OS you should see less than 100µs ping between peers.

The max latency might indicate that you're having a problem due to interrupt coalescing (e.g. the NIC letting the packet sit in a buffer until a timeout appears) which I noticed to be a problem with Broadcom cards more than Intel.

If I understand the matrix correctly then you're seeing latency between 200ms and 500ms in the mean which sounds way too much for me.

Try starting out with pinging between two hosts, to check for possible interrupt coalescing I recommend to look at zig-zag times (e.g. 1 2 3 1 2 3 1 2 3) in the output.

Also, try a flood ping between machines to see a good average that cancels out the coalescing.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.