1

We sometimes have 90%+ more packet loss on our server, but it does not always appends. Right now it works perfectly, but just half an hour ago, it had just that problem.

Our service provider is telling us to go in a recovery system to test if this is really a hardware problem and not software on our side. However, I don't see anything that can cause packet loss on our side, especially if it is not consistent.

Is there anything we could check before doing an other test on the recovery system?

We have a dedicated server at Hetzner.de. It is connected to 100MBit ethernet. We did not try to change anything on the hardware side, because our server provider want that we check our software before to continue to check the hardware.

Here is the mtr reports I have made. During that the report, we had 3 burst of packet loss and the rest of the time the server was reachable :

Client to server

HOST: mbp                         Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.0.1.1                   0.0%  1000    0.4   0.2   0.2   3.4   0.2
  2.|-- 10.0.1.1                   0.3%  1000   27.5  29.7   5.9 237.3  34.6
  3.|-- 10.170.172.121             0.4%  1000   17.2  41.9   7.2 334.1  44.2
  4.|-- 216.113.123.158            1.4%  1000   44.4  58.6  10.6 299.6  49.2
  5.|-- 216.113.123.194            1.1%  1000   36.6  72.9  19.4 330.7  48.1
  6.|-- paix-nyc.init7.net         0.7%  1000   57.1  75.8  18.4 313.8  49.1
  7.|-- r1lon1.core.init7.net      1.4%  1000  199.8 150.9  87.1 373.7  56.4
  8.|-- r1fra1.core.init7.net      0.6%  1000  244.2 150.1  98.6 438.6  53.6
  9.|-- gw-hetzner.init7.net       1.4%  1000  175.3 140.6 100.5 397.2  49.7
 10.|-- hos-bb2.juniper2.rz16.het 39.0%  1000  120.0 136.7 103.5 362.6  44.3
 11.|-- hos-tr4.ex3k13.rz16.hetzn  0.8%  1000  145.4 132.2 106.8 393.3  36.9
 12.|-- static.98.43.9.5.clients. 39.8%  1000  116.0 131.5 106.1 371.8  34.4

Server to client

HOST: thetransitapp               Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. static.97.43.9.5.clients.you 29.0%  1000    7.2   7.4   0.9  24.9   1.9
  2. hos-tr1.juniper1.rz16.hetzne 38.7%  1000    6.1   9.6   0.2  78.8   7.6
  3. hos-bb2.juniper4.ffm.hetzner 36.2%  1000   11.8  11.4   5.8  29.0   1.5
  4. r1fra1.core.init7.net        38.1%  1000   12.4  13.9   5.5  22.9   3.9
  5. r1lon1.core.init7.net        36.3%  1000   23.5  26.5  17.6  37.6   4.4
  6. r1nyc1.core.init7.net        35.5%  1000   92.3  93.8  86.1 103.0   3.7
  7. paix-ny.ia-unyc-bb05.vtl.net 35.5%  1000   95.5  96.4  87.6 134.7   5.3
  8. 216.113.123.169              36.3%  1000  101.5 102.0  94.4 124.9   3.6
  9. 216.113.124.42               34.7%  1000  113.1 107.7  96.7 117.6   3.6
 10. 216.113.123.157              37.5%   999  106.5 107.4 101.5 115.0   1.5
 11. ???                          100.0   999    0.0   0.0   0.0   0.0   0.0
 12. modemcable004.103-176-173.mc 36.7%   999  111.2 147.9 107.2 342.0  48.3

Here is the ethernet configuration

Settings for eth0:
    Supported ports: [ TP MII ]
    Supported link modes:   10baseT/Half 10baseT/Full 
                            100baseT/Half 100baseT/Full 
                            1000baseT/Half 1000baseT/Full 
    Supports auto-negotiation: Yes
    Advertised link modes:  10baseT/Half 10baseT/Full 
                            100baseT/Half 100baseT/Full 
                            1000baseT/Half 1000baseT/Full 
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Link partner advertised link modes:  10baseT/Half 10baseT/Full 
                                         100baseT/Half 100baseT/Full 
                                         1000baseT/Full 
    Link partner advertised pause frame use: No
    Link partner advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: MII
    PHYAD: 0
    Transceiver: internal
    Auto-negotiation: on
    Supports Wake-on: pumbg
    Wake-on: g
    Current message level: 0x00000033 (51)
    Link detected: yes

ifconfig of eth0:

eth0      Link encap:Ethernet  HWaddr c8:60:00:bd:2f:9d  
          inet addr:5.9.43.98  Bcast:5.9.43.127  Mask:255.255.255.224
          inet6 addr: fe80::ca60:ff:febd:2f9d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3521 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2117 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2882770 (2.7 MiB)  TX bytes:910907 (889.5 KiB)
          Interrupt:30 Base address:0x8000 
7
  • You need to provide more information, is it a physical server or virtual server, network set up, which ethernet adapter does it have, what's the proof you're having packet loss, relevant log entries, whatever else may be relevant. Did you switch cables and ethernet ports and/or adapters to rule out a physical problem in those.
    – aseq
    Aug 21, 2012 at 21:21
  • Absent any other changes to your environment (which you should undo or thoroughly investigate), do what your provider asks in order to get them to troubleshoot the problem.
    – voretaq7
    Aug 21, 2012 at 21:26
  • I've edited the question and I will continue to do so, Thanks!
    – gcamp
    Aug 21, 2012 at 21:34
  • Are you sure it's connected to 100Mb? MII reported that the link speed is 1000Mb. A mismatch here indicates a negotiation failure.
    – bahamat
    Aug 21, 2012 at 21:51
  • "1 Gbit OnBoard connected at 100 Mbit" so I guess it should be connected at 100MBit? I will look into that.
    – gcamp
    Aug 21, 2012 at 22:00

2 Answers 2

2

In my opinion it's hetzner fault. I've been arguing with them for a very long time about similar case.

We had those problems and were reporting it to the hosting company. The answer was always the same - "Please attach mtr in both directions" - they would answer like that even during the fault. So we did write a daemon that will launch mtr each time we have any packet loss between servers :

if [ -z $1 ] ; then
                echo "Give target host"
else
                host=$1
                while true ; do
                                loss=`ping -c 10 $host | grep packet | awk {'print $6'} | sed s/%//g`
                                if [ $loss -ge 1 ]; then
                                                echo `date` >> /root/scripts/loss_measure_mtr.log
                                                mtr -s 1500 -r -c 1000 -i 0.1 $host >> /root/scripts/loss_measure_mtr.log
                                fi

                done


fi

Then with this information they answered :

At this time there was an incoming attack in the subnet. In this case it is
possible that packet-loss occurs at servers in the same subnet.

Best Regards

Michael Straetz

Hetzner Online AG
Support
90431 Nürnberg / Germany
Tel: +49 (911) 234 226 54
Fax: +49 (911) 234 226 8 977
http://www.hetzner.de

What is exactly happening ? I dont' know but it looks almost the same :

Sun Aug 12 01:13:20 CEST 2012
HOST: app                         Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.                      94.1%  1000    0.2   0.2   0.1   0.4   0.1
  2. static.1.24.24.46.clients.you  0.0%  1000    3.0   1.9   0.7  19.4   1.5
  3. hos-tr4.juniper2.rz13.hetzne  9.4%  1000    0.6   1.9   0.4 133.2   8.0
  4. hos-bb2.juniper1.rz1.hetzner  5.4%  1000   38.6   7.1   3.0 112.9  11.5
  5. hos-tr1.ex3k3.rz1.hetzner.de 10.9%  1000    4.4   5.1   3.6  23.6   1.8**
  6. static.88-128-24-108.clients 15.5%  1000    3.6   3.5   3.4   4.6   0.1
HOST: app                         Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.                  94.5%  1000    0.2   0.2   0.1   0.6   0.1
  2. static.1.24.24.46.clients.you  0.0%  1000    1.2   1.9   0.7  19.3   1.6
  3. hos-tr4.juniper2.rz13.hetzne  9.3%  1000    0.6   1.8   0.4 136.8   7.9
  4. hos-bb2.juniper1.rz1.hetzner  2.7%  1000    3.3   7.0   3.0 113.1  11.5
  5. hos-tr1.ex3k3.rz1.hetzner.de  8.5%  1000    7.0   5.1   3.6  26.8   2.0
  6. static.88-128-24-108.clients 12.8%  1000    3.6   3.5   3.3   4.5   0.1

I have tens of mtr's like this.

In my opinion it's their infrastructure problems. Notice that loss is occuring on the nodes : hos-tr1.ex3k3.rz1.hetzner.de, hos-tr4.juniper2.rz13.hetzner.de and so on.

If they don't fix that I'll probably migrate to linode or amazon.

3
  • 1
    Out of curiosity, are you in DC16? If that's the case, I really hope they fix that problem. They have great prices and they were reliable before that. We are thinking about switching, too, but it's just sad to compare prices anywhere else.
    – gcamp
    Aug 22, 2012 at 0:03
  • I'm in DCs : 1,10,12,13 and have some vhosts in DCs that i don't know. The problem does not depend on DC I'm in. I'm in hetzner for two reasons - they're cheap and have acceptable initial fees (in contrast to ovh which has low fees but is more expensive when it comes to 1year rent) and second - they're German :)
    – wojciechz
    Aug 22, 2012 at 0:17
  • The problem suddenly stopped, because I don't see any problems here anymore. Maybe they fixed it?
    – gcamp
    Aug 24, 2012 at 19:24
1

This isn't an answer but it's too long for a comment, hence my posting it as an answer.

I don't completely agree with the assesments made in the existing answser and some of the comments to this question.

The problem with using any tool that uses ICMP via ping and traceroute (like mtr if I'm understanding how it works) is that said tool is testing how each hop in the path responds to the ICMP traffic, meaning the test is sent TO each hop and then measures that hops response. This is not a true test of the quality of the path THROUGH each hop in the path, meaning it's not testing the transmission of "real" traffic THROUGH the path. Each hop may choose to give your ICMP based test low priority or it may drop it altogether, hence the variation in results from one hop to the next. If you had a true problem at hop 10 (in your first screen grab) then that problem would carry through (and be cumulative) at each successive hop. As you can see in your sceen grab, hop 10 is showing 39% packet loss but hop 11 is showing almost no packet loss. If hop 10 were really dropping "real" traffic then the problem would manifest at hop 11 as well. In fact, hop 11 would probably show more packet loss (cumulative of the loss at hop 10, the loss on the link between hops 10 and 11, and the loss at hop 11).

What you should be doing is testing with a tool that sends real traffic from one end to the other, like iperf.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .