0

I have two GCE instances in europe-west1-b region. One of these instances is a CI server that is getting stuck after making an SSH connection to a remote server it is deploying to.

My usual diagnostics always include an mtr trace to check network health but today these two instances are showing packet loss down the entire path.

                                       My traceroute  [v0.86]
gitlab-2 (0.0.0.0)                                                  Wed Jul  4 10:40:12 2018
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                    Packets               Pings
 Host                                             Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 209.85.248.116                                15.8%   101   38.6  39.0  37.4  47.1   1.7
 2. 108.170.245.14                                 9.9%   101   38.4  38.7  37.5  54.1   1.6
 3. prs-b2-link.telia.net                         16.0%   101   38.9  39.2  37.4  53.1   2.0
 4. prs-bb3-link.telia.net                        10.9%   101  143.2 143.6 142.5 156.0   1.3
 5. nyk-bb3-link.telia.net                        11.0%   101  124.5 125.2 123.8 133.4   1.6
 6. chi-b21-link.telia.net                        13.9%   101  145.0 144.1 142.9 147.2   0.6
 7. serverhub-ic-324864-chi-b21.c.telia.net       12.0%   101  145.5 144.8 143.5 160.9   2.0
 8. agg1.c13.r05.s101.chi03.singlehop.net         15.0%   100  158.7 214.1 146.2 689.3 137.5
 9. aswg1.c02.r05.s101.chi03.singlehop.net        11.0%   100  144.4 144.2 141.2 145.5   0.3
10. devdb01.com-2.mobi                            11.0%   100  143.4 143.7 142.7 145.0   0.1

This looks like a problem to me but straight ping to the target shows consistent latency and no loss.

Does any one know if this is a quirk of GCE networking set up or a genuine issue?

UPDATE: So the answer is no a trace from a GCE instance should not be showing dropped packets at all hops. Now the issue is resolved, I'm taking a completely different route with a different transit provider:

                                      My traceroute  [v0.86]
gitlab-2 (0.0.0.0)                                                        Wed Jul  4 14:24:28 2018
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                          Packets               Pings
 Host                                                   Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 72.14.238.62                                         0.0%   102    4.7   6.2   4.6  26.1   3.9
 2. 108.170.244.167                                      0.0%   101    5.6   5.5   4.6  35.3   4.0
 3. ???
 4. ae27.cs1.cdg11.fr.eth.zayo.com                       0.0%   101  136.4 137.3 125.9 146.6   4.4
 5. ae0.cs1.cdg12.fr.eth.zayo.com                        0.0%   101  141.4 137.4 122.7 151.2   4.7
 6. ae2.cs1.lhr11.uk.eth.zayo.com                        0.0%   101   97.5  98.0  97.4 103.6   1.0
 7. ae5.cs1.lga5.us.eth.zayo.com                        98.0%   101  136.9 137.0 136.9 137.2   0.0
 8. ae0.cs2.lga5.us.eth.zayo.com                         0.0%   101  137.8 138.9 125.9 171.7   6.5
 9. ae3.cs2.ord2.us.eth.zayo.com                         0.0%   101  149.3 143.8 125.3 168.3   7.6
10. ae27.cr2.ord2.us.zip.zayo.com                        0.0%   101  141.3 138.3 125.6 150.9   4.2
11. ae17.er2.ord7.us.zip.zayo.com                        0.0%   101  142.1 138.6 126.5 163.2   4.9
12. 128.177.108.98.IPYX-142927-900-ZYO.zip.zayo.com      0.0%   101  140.7 138.5 128.1 145.4   4.1
13. agg1.c13.r05.s101.chi03.singlehop.net                0.0%   101  163.1 215.6 140.4 762.2 144.8
14. aswg1.c02.r05.s101.chi03.singlehop.net               0.0%   101  141.7 138.8 125.7 148.9   5.0
15. devdb01.com-2.mobi                                   0.0%   101  141.7 137.3 123.9 142.8   4.3
1
  • In addition to the mtr packet loss an hping to SSH port of the target is showing sporadic duplicate packets being received. – Guy Thouret Jul 4 '18 at 10:56
0

UPDATED: https://status.cloud.google.com/incident/compute/18007

Google have acknowledged a problem with europe-west1-b Google Cloud Status

1
  • 1
    That must have just gone up, first place I checked. Thanks. – Guy Thouret Jul 4 '18 at 12:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.