0

I have a compute engine VM running in Google Cloud Platform (n1-standard-2 in australia-southeast1-b).

The instance is running a web server, has both external and internal static IP, uses default net and subnet, and has a domain assigned to it.

After some random period of time, the machine becomes unreachable. I was to find the cause so far but when I try tcptraceroute, this is the result:

Healthy

$ tcptraceroute my.domain.nz 443
Selected device wlp2s0, address 192.168.100.204, port 38543 for outgoing packets
Tracing the path to my.domain.nz (35.189.255.255) on TCP port 443 (https), 30 hops max
 1  192.168.100.1  2.982 ms  1.719 ms  1.884 ms
 2  192.168.1.254  2.540 ms  1.991 ms  2.091 ms
 3  * * *
 4  * * *
 5  ae8-10.akbr6.global-gateway.net.nz (122.56.116.5)  4.685 ms  3.459 ms  3.439 ms
 6  ae7-2.akbr7.global-gateway.net.nz (122.56.119.53)  3.908 ms  2.855 ms  2.693 ms
 7  xe5-0-4.sgbr3.global-gateway.net.nz (122.56.127.186)  29.059 ms  30.652 ms  28.932 ms
 8  ae2-10.sgbr4.global-gateway.net.nz (202.50.232.246)  27.784 ms  27.875 ms  35.276 ms
 9  google-gsw.sgbr4.global-gateway.net.nz (202.50.237.198)  30.394 ms  28.655 ms  28.791 ms
10  108.170.247.33  29.646 ms  29.712 ms  29.606 ms
11  209.85.242.142  29.048 ms  28.481 ms  28.239 ms
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  my.domain.nz (35.189.255.255) [open]  31.312 ms  29.198 ms  29.871 ms

Unhealthy

$ tcptraceroute my.domain.nz 443
Selected device wlp2s0, address 192.168.100.204, port 41597 for outgoing packets
Tracing the path to my.domain.nz (35.189.255.255) on TCP port 443 (https), 30 hops max
 1  192.168.100.1  4.428 ms  1.306 ms  1.813 ms
 2  192.168.1.254  2.654 ms  1.908 ms  2.545 ms
 3  * * *
 4  * * *
 5  ae8-10.akbr6.global-gateway.net.nz (122.56.116.5)  102.289 ms  3.407 ms  3.176 ms
 6  ae7-2.akbr7.global-gateway.net.nz (122.56.119.53)  4.630 ms  3.817 ms  3.299 ms
 7  xe5-0-2.sgbr3.global-gateway.net.nz (122.56.127.178)  119.213 ms  27.778 ms  144.073 ms
 8  ae2-10.sgbr4.global-gateway.net.nz (202.50.232.246)  28.512 ms  29.801 ms  144.985 ms
 9  google-gsw.sgbr4.global-gateway.net.nz (202.50.237.198)  29.008 ms  132.171 ms  29.510 ms
10  108.170.247.65  28.283 ms  28.180 ms  27.605 ms
11  209.85.242.190  27.815 ms  27.783 ms  27.681 ms
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
Destination not reached

I was suspecting there is something going on with our local network, but if I reboot the VM from GCP console we can access it again.

If I try accessing the website from my mobile network it works. Ssh and ping/curl from any remote server works too.

Not really sure how to debug it any further nor what could be wrong.

Any ideas?

  • As per the output you have posted you seem to have a connectivity issue but that doesn’t mean there is a problem. As traceroute uses icmp it could be that the route taken by packets in your second Unhealthy try has gone through nodes that are configured to drop icmp packets for security reasons. A simple firewall rule could be the cause. When you say the machines become unreachable, which connection method were you using initially? Not the ones from traceroute. Can you provide the errors you get when connecting? – JMD May 31 '18 at 15:08
  • Hasn't happened for quite a while now, if it comes back I'll put the error logs here. – Jan Richter May 31 '18 at 22:57
  • I'm voting to close this question as off-topic because the issue is specific to the user's implementation. – Magellan Jul 26 '18 at 4:46
  • Cannot find why was sshguard installed on the machine. The only thing that was installed manually was gitlab omnibus package which doesn't mention anything. /var/log/dpkg entries have no sshguard mentions. – Jan Richter Jul 26 '18 at 19:10
0

Blame the sshguard!

After further investigation, we found out the problem occurs if a colleague in the office tries to push to the server (VM is a git server) with invalid SSH key.

Further debugging and help by briansneddon from the Google Cloud Slack community I found out the problem using this command:

sudo /sbin/iptables -L -v -n

Chain INPUT (policy ACCEPT 33560 packets, 35M bytes)
 pkts bytes target     prot opt in     out     source               destination         
72689   68M sshguard   all  --  *      *       0.0.0.0/0            0.0.0.0/0           
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
Chain OUTPUT (policy ACCEPT 32966 packets, 36M bytes)
 pkts bytes target     prot opt in     out     source               destination         
Chain sshguard (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 3259  263K DROP       all  --  *      *       123.123.123.123      0.0.0.0/0  

You can see the IP 123.123.123.123 is being dropped by sshguard.

Solution

  1. Remove the active block by running sudo /sbin/iptables -F sshguard.
  2. Add the IP to /etc/sshguard/whitelist
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.