3

I restarted my Google Cloud VM and now cannot access it via ssh. I am getting below messages from the logs -

Sep 26 21:51:02  NetworkManager[1300]: <info>  [1569523862.8802] dhcp4 (eth0): canceled DHCP transaction
Sep 26 21:51:34  NetworkManager[1300]: <info>  [1569523894.8143] device (eth0): state change: ip-config -> failed (reason 'ip-config-unavailable', sys-iface-state: 'managed')
Sep 26 21:51:34  NetworkManager[1300]: <info>  [1569523894.8148] manager: NetworkManager state is now DISCONNECTED
Sep 26 21:51:34  NetworkManager[1300]: <warn>  [1569523894.8151] device (eth0): Activation: failed for connection 'System eth0'
Sep 26 21:51:34  NetworkManager[1300]: <info>  [1569523894.8153] device (eth0): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed')```

3
  • Have you changed anything about the VM? The network interface is failing. You might try changing the instance size or create an image to launch a new VM. Sep 26, 2019 at 19:45
  • I haven't changed anything on VM but had changed mysql settings in it. It was complaining of disk space which was 92% used. I rebooted it successfully but to my surprise , I could not open ssh and my external IP had also been changed. I stopped the instance and restarted it again, it loaded with my previous external IP but still inaccessible with above error. Sep 26, 2019 at 20:40
  • I have a VM with exactly the same issue, but it is not caused by a full disk. I'm still troubleshooting, so I don't know the exact answer yet, but I'm guessing that it is related to an automatic update from CentOS 7.6 → 7.7, which happened on the same day that the NetworkManager errors began. Sep 27, 2019 at 4:50

4 Answers 4

2

Your error messages indicate that dhclient was unable to run successfully. Did you also see in your logs something like, dhcp4 (eth0): client pid XXXX exited with status 127? Or if not, are there other log entries related to dhcp4?

Assuming you got the error exited with status 127, I'll proceed with suggesting a solution. (If you have a different error, let me know and I'll edit the answer.)

Since you have no network connectivity, you'll need to log in to the VM through the serial console, authenticating with a password to a user account that has sudo access to root.

Exit code 127 means the program was unable to run. Why? Well, try it: run the dhclient command and see what happens:

#> sudo dhclient --help
dhclient: error while loading shared libraries: libdns-export.so.1102: cannot open shared object file: No such file or directory

Ok, in this case, dhclient is unable to load a required library. If you search Google you might find this old redhat bug which suggests the issue is fixed by relinking the libraries with ldconfig:

#> sudo /sbin/ldconfig

After that, dhclient runs successfully:

#> sudo dhclient --help
Internet Systems Consortium DHCP Client 4.2.5
Copyright 2004-2013 Internet Systems Consortium.
[…]

Then, restarting NetworkManager succeeds:

#> sudo systemctl restart NetworkManager.service

And the network is up:

#> ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=52 time=1.98 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=52 time=0.258 ms
3
  • I will do above later. I had above error Sep 27, 2019 at 6:41
  • @MartinKarari, did you find the root problem of your NetworkManager error? Oct 6, 2019 at 1:57
  • If you haven't configured an account with a password, you can stop the unreachable VM, detach the boot disk, attach it to another VM, and manually edit the shadow file to set a password. Then, detach the disk, re-attach to the original VM, and log in via serial console. Nov 2, 2019 at 19:50
0

The reason that the network stack is failing on instance startup is that your VM's booting disk has become 100% full.

The solution is to resize the disk larger.

I wrote this article which details the steps required:

Resize Root Filesystem

7
  • Okay, I guess so and I wanted to confirm but I am also not able to ssh to the same via gcloud console Sep 26, 2019 at 22:00
  • If networking is broken how do you expect SSH to work? If you read my article it shows you how to use the serial port to see the log files generated during boot. Look for a message about disk full. Sep 26, 2019 at 22:03
  • I now get it, from serial port log, I haven't traced anything on disk only below complain Sep 27 01:09:06 my host NetworkManager[1362]: <info> [1569535746.8264] dhcp4 (eth0): canceled DHCP transaction Sep 27 01:09:38 my host NetworkManager[1362]: <info> [1569535778.7677] device (eth0): state change: ip-config -> failed (reason 'ip-config-unavailable', sys-iface-state: 'managed') Sep 27 01:09:38 my host NetworkManager[1362]: <info> [1569535778.7685] manager: NetworkManager state is now DISCONNECTED Sep 27 01:09:38 my host NetworkManager[1362]: <warn> [1569535778.7691] device Sep 26, 2019 at 22:18
  • Reboot the instance, then look at the serial port output. Sep 26, 2019 at 23:35
  • Gives me the same issue when booting... I have logged-in through the serial console and did manual network configuration and now I can connect but these settings revert on reboot. Again, my local ip is inaccesible by other VMs but is accessible via its public IP Sep 27, 2019 at 0:36
0

Below temporarily fixes my issue;

(i) Login to your VM via serial console. You may have to click on the return button for login to appear.

(ii) Confirm that your VM memory and disk-space is fine. You may use such tools as `df -h to check your disk space, if not enough, you may follow John Hanley tutorial above and increase it. When all is okay and the only network is failing, go to step iii

(iii) Disable NetworkManager and configure network manually, on centos, can be done as follows:

systemctl stop NetworkManager
systemctl disable NetworkManager
ifconfig eth0 <internal_ip> netmask 255.255.255.0
route add default gw <your-cloud-gateway>

After above, you will have full access to your Google VM, however, you VM will not be visible to other VM, you will have to configure this using route command as below

route add <other-VM-internal-IP> netmask 0.0.0.0 gw <you-cloud-gateway>

You will have to do above for every VM you want to be visible by your Virtual server

This fix is temporal in that you will possibly lose you setting on reboot, any way you can place these command in /etc/rc.local for them to be performed on start-up

Disclaimer: I am not a much-experienced but above was my way to fix a critical server that just had a mayday and needed my clients to get service restored fast as I look for a permanent solution.

0

Since your NetworkManager state is now DISCONNECTED, the only way to connect to you instance is interacting with the serial console

There is high chance that you did not assigned a local password for a user on a virtual machine instance in advance and stuck here but no worries! you can add follow this instruction to overcome this issue also:

Step one, create startup script in cloud shell using the command:

$ nano startup.sh and copy the following into it.

#! /bin/bash echo "password" | passwd --stdin username

(where password is the desired password, must leave quotation marks. And username is the desired username) Press ctrl+x and then y and then enter to save and exit

Step two, use the following command to insert startup script into your instance.

gcloud compute instances add-metadata example-instance
--metadata-from-file startup-script=path/to/file

(where example instance is your instance name and path to file is where the startup script file is located, in the case that you have not changed directories after creating your file the path would be ./startup.sh

Step three go back to vm instances in your console and restart your instance. Once it has spun up, click on the [Connect to serial console] button.

Once the startup process has finished, you can enter your credentials. You may need to push enter to get the login prompt.

Once logged in we can verify your hard drive space to confirm the log entries using the command:

df -h

Also you can use following command in your script to delete temporary file in order to cleanup your hard disk:

#! /bin/bash rm -rf /tmp/* rm -rf /var/tmp/* rm -rf /usr/tmp/* rm -rf /var/log/*

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .