1

I have Angular, NodeJS and nginx running on 2 core 8GB RAM ubuntu 18.04 instance on google compute engine. The server was operating normally (HDD 49%, CPU below 10% and RAM below 40%) but suddenly web frontend was not working (like server was stopped). At that time, I checked google cloud console and the system was working normally. Then I tried to ssh using WinSCP but couldn't connect to the host. Then I tried with web ssh on cloud console website, it neither. I restarted the server then everything was normal .

I checked all logs for my database and node server, server has been running entire time with no issue. I concluded that the OS was working the entire time.

I used monitoring on google console, the cpu was running but no network activity for the above period(incoming/outgoing)

  1. Any explanation for above scenario? Should I troubleshoot it with server or issue from google?

  2. What steps i can follow to identify problems in above scenarios? Which logs should I check?

  3. In addition, any logs from google console that I can check for troubleshooting?

3 Answers 3

0

For the description of your issue, I can only think that the google agent restarted the network services and, for some reason, it failed to start them properly. Normally this can only be confirmed inside the OS by checking that the agent is running or checking the agent logs, however normally there is a generic message in Stackdriver logging that says “OS agent configuration started”. When the instance was restarted the agent started the configuration again.

2
  • No. Couldn't find any clue. First log after long time is shutting down it by myself
    – Salitha
    Mar 19, 2020 at 18:19
  • In that case there is a possibility that the Network Manager service has overwritten some configuration files, that depends of your OS but you could check that if you go to your network configuration files, normally network manager adds a comment like "Managed by Network Manager", however if it is working, maybe that message change for "Managed by google agent"
    – AdolfoOG
    Mar 24, 2020 at 14:59
0

As @AdolfoOG mentioned - you have to check the logs in order to troubleshoot this.

But when it happened you couldn't SSH into your machine. Even if you can't you can still check the serial console output via console gui or gcloud compute instances get-serial-port-output instance_name --zone=my_zone command in Cloud Shell.

To be able to log in [enable interaction with serial console][4] gcloud compute instances add-metadata instance-name --metadata serial-port-enable=TRUE and connect to it with gcloud compute connect-to-serial-port instance-name.

Now when you're logged in you can check all the logs or services when the issue persists.

0

At first, you should collect logs:

  1. Go to Compute Engine -> VM instances -> click on NAME_OF_YOUR_VM -> at the VM instance details find section Logs and click on Serial port 1 (console)
  2. Reboot your VM instance again.
  3. Check full boot log for any errors or/and warnings.

If you found errors/warning related to disk space you can try to resize it accordingly to the documentation Resizing a zonal persistent disk, also accordingly to the article Recovering an inaccessible instance or a full boot disk:

If an instance is completely out of disk space or if it is not running a Linux guest environment, then automatically resizing your root filesystem isn't possible, even after you've increased the size of the persistent disk that backs it. If you can't connect to your instance, or your boot disk is full and you can't resize it, you must create a new instance and recreate the boot disk from a snapshot to resize it.

Otherwise try to troubleshoot your VM instance via serial console:

  1. Enable serial console connection with gcloud command:

     gcloud compute instances add-metadata NAME_OF_YOUR_VM_INSTANCE \
     --metadata serial-port-enable=TRUE
    

or go to Compute Engine -> VM instances -> click on NAME_OF_YOUR_VM_INSTANCE -> click on EDIT -> go to section Remote access and check Enable connecting to serial ports

  1. Create temporary user and password to login: shutdown your VM and set a startup script by adding at the section Custom metadata key startup-script and value:

     #!/bin/bash
     useradd --groups google_sudoers tempuser
     echo "tempuser:password" | chpasswd
    

and then start your VM.

  1. Connect to your VM via serial port with gcloud command:

     gcloud compute connect-to-serial-port NAME_OF_YOUR_VM_INSTANCE
    

    or go to Compute Engine -> VM instances -> click on NAME_OF_YOUR_VM_INSTANCE -> and click on Connect to serial console

  2. Check what went wrong.

  3. Disable access via serial port with gcloud command:

     gcloud compute instances add-metadata NAME_OF_YOUR_VM_INSTANCE \
     --metadata serial-port-enable=FALSE
    

or go to Compute Engine -> VM instances -> click on NAME_OF_YOUR_VM_INSTANCE -> click on EDIT -> go to section Remote access and uncheck Enable connecting to serial ports. Keep in mind that accordingly to the documentation Interacting with the serial console:

Caution: The interactive serial console does not support IP-based access restrictions such as IP whitelists. If you enable the interactive serial console on an instance, clients can attempt to connect to that instance from any IP address. Anybody can connect to that instance if they know the correct SSH key, username, project ID, zone, and instance name. Use firewall rules to control access to your network and specific ports.

If you weren't able to connect via serial console, try follow the documentation Troubleshooting SSH section Inspect the VM instance without shutting it down and inspect the disk of your VM on another VM. Same way you can transfer your data to another working VM instance.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .