Debian Server on GCP suddenly unable to access via SSH since hard disk was over consumed

Question

I am struggling with this since more than a month now and have tried many other solutions posted on the forum.

Compute Engine VM instance disk was over consumed due to a failed backup script and since then SSH authentication is not working.

I have already tried following: A. Increased the size of the disk from GCP console.

B. Have taken snapshot of the disk and created new instance from that

C. Have create new instance with new disk and attached the snapshot as the additional disk (when i remove the attached snapshot this instance works, but if i attach the disk with problem it is not working)

I have tried various startup scripts that i came across

Tried this solution sudo resize2fs /dev/sda1

Tried the solution listed here https://github.com/mbrukman/stackexchange-answers/blob/master/stackoverflow/24021214/fdisk.sh

When i look into the errors of the serial port log, i get the following failed commands

[[0;1;31mFAILED[0m] Failed to mount Huge Pages File System.
See 'systemctl status dev-hugepages.mount' for details.
[[0;1;31mFAILED[0m] Failed to mount POSIX Message Queue File System.
See 'systemctl status dev-mqueue.mount' for details.
[[0;32m  OK  [0m] Started Load Kernel Modules.
[[0;1;31mFAILED[0m] Failed to mount Debug File System.
See 'systemctl status sys-kernel-debug.mount' for details.
[[0;1;31mFAILED[0m] Failed to start Create list of requ…vice nodes for the current kernel.
See 'systemctl status kmod-static-nodes.service' for details.
[[0;1;31mFAILED[0m] Failed to start Remount Root and Kernel File Systems.
See 'systemctl status systemd-remount-fs.service' for details.
[[0;32m  OK  [0m] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
         Starting udev Coldplug all Devices...
         Starting Load/Save Random Seed...
         Starting Create Static Device Nodes in /dev...
         Starting Apply Kernel Variables...
[[0;1;31mFAILED[0m] Failed to start Flush Journal to Persistent Storage.
See 'systemctl status systemd-journal-flush.service' for details.
[[0;1;31mFAILED[0m] Failed to start udev Coldplug all Devices.
See 'systemctl status systemd-udev-trigger.service' for details.
[[0;32m  OK  [0m] Started Load/Save Random Seed.
[[0;1;31mFAILED[0m] Failed to start Create Static Device Nodes in /dev.
See 'systemctl status systemd-tmpfiles-setup-dev.service' for details.
[[0;32m  OK  [0m] Started Apply Kernel Variables.
         Starting udev Kernel Device Manager...
[[0;32m  OK  [0m] Reached target Local File Systems (Pre).
[[0;32m  OK  [0m] Reached target Local File Systems.
         Starting Raise network interfaces...
         Starting Create Volatile Files and Directories...
[[0;1;31mFAILED[0m] Failed to start Create Volatile Files and Directories.
See 'systemctl status systemd-tmpfiles-setup.service' for details.
[[0;1;31mFAILED[0m] Failed to start Entropy daemon using the HAVEGE algorithm.
See 'systemctl status haveged.service' for details.
[[0;32m  OK  [0m] Reached target System Time Synchronized.
         Starting Update UTMP about System Boot/Shutdown...
[[0;1;31mFAILED[0m] Failed to start Raise network interfaces.
See 'systemctl status networking.service' for details.
[[0;32m  OK  [0m] Started udev Kernel Device Manager.
[[0;32m  OK  [0m] Reached target Network.
[[0;32m  OK  [0m] Reached target Network is Online.
[[0;32m  OK  [0m] Started Update UTMP about System Boot/Shutdown.
[[0;32m  OK  [0m] Reached target System Initialization.
[[0;32m  OK  [0m] Listening on UUID daemon activation socket.
[[0;32m  OK  [0m] Started Daily apt download activities.
[[0;32m  OK  [0m] Started Daily apt upgrade and clean activities.
[[0;32m  OK  [0m] Listening on ACPID Listen Socket.
[[0;32m  OK  [0m] Reached target Sockets.
[[0;32m  OK  [0m] Started ACPI Events Check.
[[0;32m  OK  [0m] Reached target Paths.
[[0;32m  OK  [0m] Reached target Basic System.
         Starting getty on tty2-tty6 if dbus and logind are not available...
[[0;32m  OK  [0m] Started Regular background program processing daemon.
[[0;32m  OK  [0m] Started ACPI event daemon.
[[0;32m  OK  [0m] Started Unattended Upgrades Shutdown.
         Starting LSB: bitnami init script...
         Starting System Logging Service...
[[0;32m  OK  [0m] Started Deferred execution scheduler.
         Starting LSB: Start NTP daemon...
         Starting Expand the root partition and filesystem on boot...
         Starting Permit User Sessions...
[[0;32m  OK  [0m] Started Daily Cleanup of Temporary Directories.
[[0;32m  OK  [0m] Reached target Timers.
         Starting LSB: start and stop Stackdriver Agent...
[[0;1;31mFAILED[0m] Failed to start getty on tty2-tty6 …dbus and logind are not available.
See 'systemctl status getty-static.service' for details.
[[0;1;31mFAILED[0m] Failed to start LSB: bitnami init script.
See 'systemctl status bitnami.service' for details.
[[0;1;31mFAILED[0m] Failed to start LSB: Start NTP daemon.
See 'systemctl status ntp.service' for details.
[[0;1;31mFAILED[0m] Failed to start Expand the root partition and filesystem on boot.
See 'systemctl status expand-root.service' for details.
[[0;32m  OK  [0m] Started Permit User Sessions.
[[0;1;31mFAILED[0m] Failed to start LSB: start and stop Stackdriver Agent.
See 'systemctl status stackdriver-agent.service' for details.

Answer 1

Since i couldn't find any solution, i created another instance of compute engine from the last backup and whatever few days data that was lost was re-entered. luckily it was a website server with very little activity, else the only other solution was to take expensive support service of Google!

From the log i figured a massive glitch from google itself which corrupted the disk so much that none of the services were starting.

And i noticed the trigger point was when they introduced some new monitoring service and at the same time my disks went out of space due to a failure of backup script. All of this happened together. This is of-course guess work, as i couldn't identify the problem accurately without any access to the system.

Answer 2

I would recommend that:

1- Take a snapshot of the disk (the instance that you can't SSH). 2- Create a new disk from that snapshot and make sure the new disk is larger than the original. 3- Create a new instance with a new boot disk and attach the snapshot disk as an additional disk.

If this continues to be an issue, you can try to connect via the serial port. If this still doesn't work, it may be worth running a startup script that unmounts the attached disk to see if you can connect to the instance and then remount it to access the data.