0

I am having difficulty identifying what a specific problem with one of your vlaned networks.

They have a voip phone which regularly cuts out during long phone calls.

I have done several ping tests from both the phone and directly into the ethernet socket back to our hardware and am getting similar results which indicates there is some kind of issue with our switches.

I am seeing a pattern when running ping tests to various IPs. either internal or external, I get a similar response.

128 bytes from 81.187.30.81: icmp_seq=4 ttl=56 time=4.628 ms
128 bytes from 81.187.30.81: icmp_seq=5 ttl=56 time=7.929 ms
128 bytes from 81.187.30.81: icmp_seq=6 ttl=56 time=6.676 ms
128 bytes from 81.187.30.81: icmp_seq=7 ttl=56 time=4.753 ms
128 bytes from 81.187.30.81: icmp_seq=8 ttl=56 time=7.558 ms
128 bytes from 81.187.30.81: icmp_seq=9 ttl=56 time=5.083 ms
128 bytes from 81.187.30.81: icmp_seq=10 ttl=56 time=5.491 ms
128 bytes from 81.187.30.81: icmp_seq=11 ttl=56 time=4.603 ms
128 bytes from 81.187.30.81: icmp_seq=12 ttl=56 time=40.480 ms
128 bytes from 81.187.30.81: icmp_seq=13 ttl=56 time=34.199 ms
Request timeout for icmp_seq 14
Request timeout for icmp_seq 15
128 bytes from 81.187.30.81: icmp_seq=14 ttl=56 time=131.104 ms
128 bytes from 81.187.30.81: icmp_seq=15 ttl=56 time=78.239 ms
128 bytes from 81.187.30.81: icmp_seq=16 ttl=56 time=34.453 ms
128 bytes from 81.187.30.81: icmp_seq=17 ttl=56 time=4.789 ms
128 bytes from 81.187.30.81: icmp_seq=18 ttl=56 time=5.350 ms
128 bytes from 81.187.30.81: icmp_seq=19 ttl=56 time=4.637 ms
128 bytes from 81.187.30.81: icmp_seq=20 ttl=56 time=6.120 ms
128 bytes from 81.187.30.81: icmp_seq=21 ttl=56 time=4.843 ms

notice how the response time spikes up and falls back down? this happens constantly, probably every 20-30 packets or so.

I ran 4 ping tests. Firstly, I pinged the VOIP server IP, next the router thirdly I pinged a NAS drive on the network, lastly, I pinged the switch I am connected to

here are the results:

--- 81.187.30.81 ping statistics ---
1200 packets transmitted, 1178 packets received, 1.8% packet loss
round-trip min/avg/max/stddev = 4.258/11.437/162.421/18.855 ms

--- 10.9.0.1 ping statistics ---
1200 packets transmitted, 1180 packets received, 1.7% packet loss
round-trip min/avg/max/stddev = 0.324/6.035/155.700/19.131 ms

--- 10.9.0.10 ping statistics ---
1200 packets transmitted, 1179 packets received, 1.8% packet loss
round-trip min/avg/max/stddev = 0.374/6.637/142.619/19.748 m

--- 10.10.47.1 ping statistics ---
1200 packets transmitted, 1200 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.990/6.858/96.510/15.386 ms

pinging the switch (10.10.47.1) gives no packet loss but anything past the switch does. what could be causing this issue at the switch?

This has been plaguing our business for a long time and I cannot figure it out!

any help would be much appreciated

Netstat - s results

tcp:
407733 packets sent
114594 data packets (51164281 bytes)
43 data packets (15304 bytes) retransmitted
0 resends initiated by MTU discovery
262041 ack-only packets (40 delayed)
0 URG only packets
0 window probe packets
27157 window update packets
3910 control packets
0 data packets sent after flow control
398462 checksummed in software
398462 segments (63078229 bytes) over IPv4
0 segments (0 bytes) over IPv6
952633 packets received
78233 acks (for 51157575 bytes)
971 duplicate acks
0 acks for unsent data
892703 packets (1141511278 bytes) received in-sequence
480 completely duplicate packets (141573 bytes)
3 old duplicate packets
0 received packets dropped due to low memory
0 packets with some dup. data (0 bytes duped)
3701 out-of-order packets (5191954 bytes)
0 packets (0 bytes) of data after window
0 window probes
22 window update packets
169 packets received after close
0 bad resets
0 discarded for bad checksums
942473 checksummed in software
942473 segments (1191620724 bytes) over IPv4
0 segments (0 bytes) over IPv6
0 discarded for bad header offset fields
0 discarded because packet too short
2423 connection requests
0 connection accepts
0 bad connection attempts
0 listen queue overflows
1641 connections established (including accepts)
2407 connections closed (including 209 drops)
61 connections updated cached RTT on close
61 connections updated cached RTT variance on close
31 connections updated cached ssthresh on close
701 embryonic connections dropped
712195 segments updated rtt (of 66367 attempts)
265 retransmit timeouts
16 connections dropped by rexmit timeout
0 connections dropped after retransmitting FIN
0 persist timeouts
0 connections dropped by persist timeout
0 keepalive timeouts
0 keepalive probes sent
0 connections dropped by keepalive
21402 correct ACK header predictions
858064 correct data packet header predictions
151 SACK recovery episodes
0 segment rexmits in SACK recovery episodes
0 byte rexmits in SACK recovery episodes
131 SACK options (SACK blocks) received
3709 SACK options (SACK blocks) sent
0 SACK scoreboard overflow
0 LRO coalesced packets
0 times LRO flow table was full
0 collisions in LRO flow table
0 times LRO coalesced 2 packets
0 times LRO coalesced 3 or 4 packets
0 times LRO coalesced 5 or more packets
0 limited transmits done
158 early retransmits done
0 times cumulative ack advanced along with SACK
0 probe timeouts
0 times retransmit timeout triggered after probe
0 times fast recovery after tail loss
0 times recovered last packet
0 connections negotiated ECN
0 times congestion notification was sent using ECE
0 times CWR was sent in response to ECE
0 times packet reordering was detected on a connection
0 times transmitted packets were reordered
0 times fast recovery was delayed to handle reordering
0 times retransmission was avoided by delaying recovery
0 retransmissions not needed
udp:
12534 datagrams received
0 with incomplete header
0 with bad data length field
0 with bad checksum
14 with no checksum
8967 checksummed in software
7746 datagrams (826559 bytes) over IPv4
1221 datagrams (234651 bytes) over IPv6
23 dropped due to no socket
4342 broadcast/multicast datagrams undelivered
0 times multicast source filter matched
0 dropped due to full socket buffers
0 not for hashed pcb
8169 delivered
4204 datagrams output
3438 checksummed in software
1734 datagrams (105647 bytes) over IPv4
1704 datagrams (348698 bytes) over IPv6
ip:
981761 total packets received
0 bad header checksums
955088 headers (19102536 bytes) checksummed in software
0 with size smaller than minimum
0 with data size < data length
42669 with data size > data length
0 packets forced to software checksum
0 with ip length > max ip packet size
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
0 fragments received
0 dropped (dup or out of space)
0 dropped after timeout
0 reassembled ok
976858 packets for this host
114 packets for unknown/unsupported protocol
0 packets forwarded (0 packets fast forwarded)
22 packets not forwardable
4767 packets received for unknown multicast group
0 redirects sent
427240 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
0 output datagrams fragmented
0 fragments created
0 datagrams that can't be fragmented
0 tunneling packets that can't find gif
0 datagrams with bad address in header
0 packets dropped due to no bufs for control data
401072 headers (8021604 bytes) checksummed in software
icmp:
23 calls to icmp_error
0 errors not generated 'cuz old message was icmp
Output histogram:
destination unreachable: 23
0 messages with bad code fields
0 messages < minimum length
0 bad checksums
0 messages with bad length
0 multicast echo requests ignored
0 multicast timestamp requests ignored
Input histogram:
echo reply: 14296
destination unreachable: 61
router advertisement: 41
0 message responses generated
ICMP address mask responses are disabled
igmp:
12 messages received
0 messages received with too few bytes
0 messages received with wrong TTL
0 messages received with bad checksum
0 V1/V2 membership queries received
0 V3 membership queries received
0 membership queries received with invalid field(s)
0 general queries received
0 group queries received
0 group-source queries received
0 group-source queries dropped
12 membership reports received
0 membership reports received with invalid field(s)
12 membership reports received for groups to which we belong
0 V3 reports received without Router Alert
16 membership reports sent
ipsec:
0 inbound packets processed successfully
0 inbound packets violated process security policy
0 inbound packets with no SA available
0 invalid inbound packets
0 inbound packets failed due to insufficient memory
0 inbound packets failed getting SPI
0 inbound packets failed on AH replay check
0 inbound packets failed on ESP replay check
0 inbound packets considered authentic
0 inbound packets failed on authentication
0 outbound packets processed successfully
0 outbound packets violated process security policy
0 outbound packets with no SA available
0 invalid outbound packets
0 outbound packets failed due to insufficient memory
0 outbound packets with no route
arp:
35 ARP requests sent
1015 ARP replies sent
0 ARP announcements sent
417805 ARP requests received
20 ARP replies received
418374 total ARP packets received
0 ARP conflict probes sent
0 invalid ARP resolve requests
0 total packets dropped due to lack of memory
33 total packets dropped due to no ARP entry
4 total packets dropped during ARP entry removal
23 ARP entries timed out
0 Duplicate IPs seen
mptcp:
0 data packets sent
0 data bytes sent
0 data packets received
0 data bytes received
0 packets with an invalid MPCAP option
0 packets with an invalid MPJOIN option
0 times primary subflow fell back to TCP
0 times secondary subflow fell back to TCP
0 DSS option drops
0 other invalid MPTCP options
0 times the MPTCP subflow window was reduced
0 bad DSS checksums
0 times received out of order data
0 subflow switches

7
  • Could you do a netstat -s and attach the results to your question? One possible culprit could be retransmission storms due to poor cables, bad switches and whatnot. Isolating offending equipment may take some time though, but netstat is a place to start.
    – MrMajestyk
    Mar 9, 2015 at 11:31
  • please note that average latency on many ethernet networks is ~ 0.2 ms, you are getting higher latency even in the good case. Mar 9, 2015 at 11:38
  • @ GiovanniToraldo the pings to my switch are fine...for the most part but I still get the blocks of ping spikes similar to the ping results in my original post but no packet loss. hence why I am getting the higher than average ping times.
    – Baz
    Mar 9, 2015 at 11:49
  • @MrMajestyk I have attached the netstat -s results for you
    – Baz
    Mar 9, 2015 at 12:26
  • Looking at the netstat output, I only see 27157 window update packets as really off. Those are sent when stalling is happening (maybe other cases too - not sure about that). Basically, this may mean that some part is too slow when it comes to receiving traffic and that causes TCP window size updates. Now - finding the culprit is harder. You might try netstat on other stations to isolate specific swithes or other components like that.
    – MrMajestyk
    Mar 9, 2015 at 12:52

0

You must log in to answer this question.

Browse other questions tagged .