2

I have two servers both with gigabit network cards that were experiencing severe latency issues communicating with one another. The culprit eventually turned out to be that one of the servers was patched into a 100mb switch port.

pinging the servers always returned <1ms

Is there a tool that could show the actual latency / rate of transfer between the servers compared to the maximum that should be possible?

3

Perform state monitoring, collect data, visualise.

  1. Your OS has tools to report the current state of your network interfaces. Use them and compare them to the expected state. Automate this.
  2. Use snmp or native counters to collect samples. Use 64bit counters for fast interfaces, or a really small sample interval.
  3. When you collect stats, graph them. Graphite is quite the thing these days.

Then, realise with monitoring, it's never real-time. You're always looking at the past.

And watch Jason Dixon at devopsdays Rome: The State of Open Source Monitoring: The good, the bad, the terrible, and a glimpse into our future.

3

You can use a tool like iperf to test the actual latency, jitter and transfer speeds possible between two servers. That's a testing tool, though.

In your specific case, it would be quite evident from the OS to see what the link speeds of the various interfaces were.

  • Checking the link speed is part of configuration management. That's not even monitoring. Good point. – Henk Langeveld Jan 19 '13 at 17:19
0

Testing latency is not that hard. You can test your latency with any two servers that are connected via a network.

Perform a series of 'ping-pong' communications between the servers for a set amount of time (eg 10 seconds). Divide the completed 'ping-pongs' between the servers by the time. You now have your transactions per second.

Let's run thru the numbers.

Test duration: 10 seconds Transactions complete 5,000

Therefore you did 1,000 transaction per second. If your ping-pong transaction was 128 bytes, you did 128,000 bytes per second. That equates to 1,024,000 bits per second. Invert that number (1/1,024,000) and you can say the latency of your transaction was 0.000000978. That's 978 nanoseconds per round trip.

The 978 nanoseconds is your seconds per bit. That's how long it took to send a bit from one machine to the other and back again. That is the definition of latency.

A good tool to use to perform this test is netperf (http://www.netperf.org/netperf/). It will run a ping-pong test over a set period of time and give you the stats I listed above.

On one server, run "netserver"

On the other server, run netperf -t UDP_RR -H -l 10 -- -r 32

You'll get all the info I listed above. You can run the maths with those results.

A shout out to Solarflare's Onload User's Guide. They outline this test methodology in great detail.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.