The ping response time doesn't reflect the real network response time

Question

I encountered a weird problem that the response time returned by ping is almost fixed at 98ms.

Either I ping the gateway, or I ping a local host or a internet host. The response time is always around 98ms although the actual delay is obvious.

However, the reverse ping (from a local machine to this host) works properly.

The following is my route table and the result:

route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    100    0        0 eth1
60.194.136.0    0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 eth1
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1

# ping the gateway
ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_req=1 ttl=64 time=98.7 ms
64 bytes from 192.168.1.1: icmp_req=2 ttl=64 time=97.0 ms
64 bytes from 192.168.1.1: icmp_req=3 ttl=64 time=96.0 ms
64 bytes from 192.168.1.1: icmp_req=4 ttl=64 time=94.9 ms
64 bytes from 192.168.1.1: icmp_req=5 ttl=64 time=94.0 ms
^C
--- 192.168.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 94.030/96.149/98.744/1.673 ms 

#ping a local machine
ping 192.168.1.88
PING 192.168.1.88 (192.168.1.88) 56(84) bytes of data.
64 bytes from 192.168.1.88: icmp_req=1 ttl=64 time=98.7 ms
64 bytes from 192.168.1.88: icmp_req=2 ttl=64 time=96.9 ms
64 bytes from 192.168.1.88: icmp_req=3 ttl=64 time=96.0 ms
64 bytes from 192.168.1.88: icmp_req=4 ttl=64 time=95.0 ms
^C
--- 192.168.1.88 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 95.003/96.696/98.786/1.428 ms

#ping a internet host
 ping google.com
PING google.com (74.125.128.139) 56(84) bytes of data.
64 bytes from hg-in-f139.1e100.net (74.125.128.139): icmp_req=1 ttl=42 time=99.8 ms
64 bytes from hg-in-f139.1e100.net (74.125.128.139): icmp_req=2 ttl=42 time=99.9 ms
64 bytes from hg-in-f139.1e100.net (74.125.128.139): icmp_req=3 ttl=42 time=99.9 ms
64 bytes from hg-in-f139.1e100.net (74.125.128.139): icmp_req=4 ttl=42 time=99.9 ms
^C64 bytes from hg-in-f139.1e100.net (74.125.128.139): icmp_req=5 ttl=42 time=99.9 ms

--- google.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 32799ms
rtt min/avg/max/mdev = 99.862/99.925/99.944/0.284 ms

I am running iperf to test the bandwidth, the rate is quite low for a LAN connection.

iperf -c 192.168.1.87 -t 50 -i 10 -f M
------------------------------------------------------------
Client connecting to 192.168.1.87, TCP port 5001
TCP window size: 0.06 MByte (default)
------------------------------------------------------------
[  4] local 192.168.1.139 port 54697 connected with 192.168.1.87 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  6.12 MBytes  0.61 MBytes/sec
[  4] 10.0-20.0 sec  6.38 MBytes  0.64 MBytes/sec
[  4] 20.0-30.0 sec  6.38 MBytes  0.64 MBytes/sec
[  4] 30.0-40.0 sec  6.25 MBytes  0.62 MBytes/sec
[  4] 40.0-50.0 sec  6.38 MBytes  0.64 MBytes/sec
[  4]  0.0-50.1 sec  31.6 MBytes  0.63 MBytes/sec

Answer 1

ping will try to do a DNS PTR query for the IP addresses, which may be the delay you're seeing. Run ping -n to disable.

You could also be seeing different behaviour for non-ICMP traffic. Try using hping to send UDP and TCP packets on various ports to check this.

Answer 2

iperf udp mode is more representative of the actual frame transition latency and failure rate.

some of the session-oriented tcp tests gloss over loss.

there are also some cool things you can do with iperf like jitter measurements.

http://openmaniak.com/iperf.php