0

I've have spent a full day trying to fix it. An unexpected 500 error appeared on the site, then I restarted and never be able to connect the SSH anymore. I resized the disk to a significant size (i.e. 500GB), while I know the actual file size only around 5GB. Can anyone help resolve the issue? Any help is greatly appreciated!

Below is the error message I captured from the "serial console". There is this error.


Feb 18 06:41:14 primary-server google-accounts: ERROR Exception calling the response handler. [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/'].#012Traceback (most recent call last):#012  File "/usr/lib/python3/dist-packages/google_compute_engine/metadata_watcher.py", line 200, in WatchMetadata#012    handler(response)#012  File "/usr/lib/python3/dist-packages/google_compute_engine/accounts/accounts_daemon.py", line 285, in HandleAccounts#012    self.utils.SetConfiguredUsers(desired_users.keys())#012  File "/usr/lib/python3/dist-packages/google_compute_engine/accounts/accounts_utils.py", line 318, in SetConfiguredUsers#012    mode='w', prefix=prefix, delete=True) as updated_users:#012  File "/usr/lib/python3.6/tempfile.py", line 681, in NamedTemporaryFile#012    prefix, suffix, dir, output_type = _sanitize_params(prefix, suffix, dir)#012  File "/usr/lib/python3.6/tempfile.py", line 269, in _sanitize_params#012    dir = gettempdir()#012  File "/usr/lib/python3.6/tempfile.py", line 437, in gettempdir#012    tempdir = _get_default_tempdir()#012  File "/usr/lib/python3.6/tempfile.py", line 372, in _get_default_tempdir#012    dirlist)#012FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/']
Ubuntu 18.04.2 LTS primary-server ttyS0

primary-server login: Feb 18 06:41:22 primary-server snapd[1512]: stateengine.go:102: state ensure error: Get https://api.snapcraft.io/api/v1/snaps/sections: dial tcp: lookup api.snapcraft.io on [::1]:53: read udp [::1]:58853->[::1]:53: read: connection refused

I resize the disk to 500 GB.


Feb 18 06:41:34 primary-server snapd[1512]: daemon.go:611: gracefully waiting for running hooks
Feb 18 06:41:34 primary-server snapd[1512]: daemon.go:613: done waiting for running hooks
Feb 18 06:41:34 primary-server snapd[1512]: daemon stop requested to wait for socket activation

Mysql service was stopped and started over and over again, then


Feb 18 06:42:33 primary-server sshd[3744]: /etc/ssh/sshd_config line 31: Deprecated option RSAAuthentication
Feb 18 06:42:33 primary-server sshd[3744]: /etc/ssh/sshd_config line 38: Deprecated option RhostsRSAAuthentication
Feb 18 06:42:33 primary-server sshd[3755]: /etc/ssh/sshd_config line 16: Deprecated option UsePrivilegeSeparation
Feb 18 06:42:33 primary-server sshd[3755]: /etc/ssh/sshd_config line 19: Deprecated option KeyRegenerationInterval
Feb 18 06:42:33 primary-server sshd[3755]: /etc/ssh/sshd_config line 20: Deprecated option ServerKeyBits
Feb 18 06:42:33 primary-server sshd[3755]: /etc/ssh/sshd_config line 31: Deprecated option RSAAuthentication
Feb 18 06:42:33 primary-server sshd[3755]: /etc/ssh/sshd_config line 38: Deprecated option RhostsRSAAuthentication

The error indicated no enough space. I resize the disk to 500GB and memory too, but still the same error.

Feb 18 06:42:33 primary-server systemd[1]: Started OpenBSD Secure Shell server.
Feb 18 06:42:33 primary-server google-accounts: INFO Removing user jameswalker11.
Feb 18 06:42:34 primary-server google_accounts_daemon[2207]: userdel: /etc/passwd.3759: No space left on device
Feb 18 06:42:34 primary-server google_accounts_daemon[2207]: userdel: cannot lock /etc/passwd; try again later.
Feb 18 06:42:34 primary-server google-accounts: WARNING Could not remove user jameswalker11. Command '['userdel', '-r', 'jameswalker11']' returned non-zero exit status 1..
Feb 18 06:42:34 primary-server google-accounts: INFO Removing user jameswalker11 from the Google sudoers group.
Feb 18 06:42:34 primary-server google_accounts_daemon[2207]: Removing user jameswalker11 from group google-sudoers
Feb 18 06:42:34 primary-server google_accounts_daemon[2207]: gpasswd: /etc/group.3760: No space left on device
Feb 18 06:42:34 primary-server google_accounts_daemon[2207]: gpasswd: cannot lock /etc/group; try again later.
Feb 18 06:42:34 primary-server google-accounts: WARNING Could not update user jameswalker11. Command '['gpasswd', '-d', 'jameswalker11', 'google-sudoers']' returned non-zero exit status 1..
Feb 18 06:42:34 primary-server google-accounts: INFO Removing user logic.
Feb 18 06:42:34 primary-server google_accounts_daemon[2207]: userdel: /etc/passwd.3761: No space left on device
Feb 18 06:42:34 primary-server google_accounts_daemon[2207]: userdel: cannot lock /etc/passwd; try again later.
Feb 18 06:42:34 primary-server google-accounts: WARNING Could not remove user logic. Command '['userdel', '-r', 'logic']' returned non-zero exit status 1..
Feb 18 06:42:34 primary-server google-accounts: INFO Removing user logic from the Google sudoers group.
Feb 18 06:42:34 primary-server google_accounts_daemon[2207]: Removing user logic from group google-sudoers
Feb 18 06:42:34 primary-server google_accounts_daemon[2207]: gpasswd: /etc/group.3764: No space left on device
Feb 18 06:42:34 primary-server google_accounts_daemon[2207]: gpasswd: cannot lock /etc/group; try again later.
Feb 18 06:42:34 primary-server google-accounts: WARNING Could not update user logic. Command '['gpasswd', '-d', 'logic', 'google-sudoers']' returned non-zero exit status 1..
Feb 18 06:42:34 primary-server google-accounts: ERROR Exception calling the response handler. [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/'].#012Traceback (most recent call last):#012  File "/usr/lib/python3/dist-packages/google_compute_engine/metadata_watcher.py", line 200, in WatchMetadata#012    handler(response)#012  File "/usr/lib/python3/dist-packages/google_compute_engine/accounts/accounts_daemon.py", line 285, in HandleAccounts#012    self.utils.SetConfiguredUsers(desired_users.keys())#012  File "/usr/lib/python3/dist-packages/google_compute_engine/accounts/accounts_utils.py", line 318, in SetConfiguredUsers#012    mode='w', prefix=prefix, delete=True) as updated_users:#012  File "/usr/lib/python3.6/tempfile.py", line 681, in NamedTemporaryFile#012    prefix, suffix, dir, output_type = _sanitize_params(prefix, suffix, dir)#012  File "/usr/lib/python3.6/tempfile.py", line 269, in _sanitize_params#012    dir = gettempdir()#012  File "/usr/lib/python3.6/tempfile.py", line 437, in gettempdir#012    tempdir = _get_default_tempdir()#012  File "/usr/lib/python3.6/tempfile.py", line 372, in _get_default_tempdir#012    dirlist)#012FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/']

0

You may have a corrupted filesystem that is being mounted read-only. Try running a filesystem check fsck as described in "Verify that your disk has a valid file system" steps described here https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-instances#troubleshooting_general_issues_with_instances

If fsck finds errors it will ask you if it should fix them - say yes. Then rerun fsck until it completes without finding any problems.

4
  • Great, let me give it a try, really appreciate that!! – Cheng Feb 18 '20 at 14:16
  • I run the fsck steps, and get the result ``` fsck from util-linux 2.29.2 e2fsck 1.43.4 (31-Jan-2017) cloudimg-rootfs: clean, 1325931/5120000 files, 10481403/10485499 blocks ``` then, mount the disk and check for master booting record, ``` $ sudo parted /dev/sdb print Model: Google PersistentDisk (scsi) Disk /dev/sdb: 129GB Sector size (logical/physical): 512B/4096B Partition Table: msdos Disk Flags: Number Start End Size Type File system Flags 1 1049kB 42.9GB 42.9GB primary ext4 boot ``` Maybe a stupid question, but what after? – Cheng Feb 18 '20 at 15:00
  • It looks like the filesystem does not have errors. You still don't know why it can't be written to. Maybe mount the filesystem locally and check the permissions of '/tmp', '/var/tmp' - directories mentioned in the error message as being unwritable. Run df to see how much space there is on the partition. You may also need to grow your partition with resize2fs to fit the new larger disk size - Google may increase the disk space but not resize the existing partition to use it. – yolabingo Feb 18 '20 at 18:23
  • Thanks again!!! 1. tmp and var/tmp permissions seems fine. all 777 2. df show 40G for the attached/mounted disk, but I actually resize the original instance to 1000G. 3. How to resize2fs? as I can't access the instance via ssh, only GCloud command now. Please advise. Thanks. – Cheng Feb 18 '20 at 19:35
0

You could try take a snapshot and create a new instance using the snapshot with the new disk size, here are the steps.

1
  • Thanks. I tried that it is not working. The source file itself seems to have the issue, so the new snapshot carries the same issue. Hence, the new instance created from the snapshot has the exact same problem even I have new disk size... I set the size to 1000GB actually.. – Cheng Feb 19 '20 at 2:41
0

Thanks yolabingo and Jorge's help. I was able to fix the issue by following the steps below.

  1. Create a snapshot from the VM instance "primary-server".

2.1 Create a new instance based on a Linux distribution.

2.2 Once it's been created, stop the instance.

2.3 Follow this guide to add an additional disk[2].

NOTE: Basically you have to edit the instance "primary-server" and add an additional disk, don't forget to select the snapshot option from the "Source type" list and click on the snapshot you just created.

2.4 Start the instance.

2.5 Mount the disk by following this guide[3].

NOTE: Please skip step 4. The additional disk it's actually a boot disk so it's been already formatted. So, don't apply the format to it; just mount it, .

2.6 Check the permissions of the file "/etc/fstab".

NOTE: The permissions should be "-rw-r--r--" and the owner "root"

2.6.1 Delete files to reduce the disk size.

2.7 Unmount the disk at OS level.

2.8 Stop the instance.

2.9 Detach the additional disk from the new instance in GCP.

NOTE: Please follow this guide[4] and instead of clicking on X next to the boot disk, please click on X next to the additional disk.

2.10 Create a new instance and instead of using an image in the "boot disk" section, please use the disk you just restored.

NOTE: For this please go to the "Boot disk" section and click on the "Change" button, then go to the "Existing" TAB and select the disk you just restored.

Please let me know if you have further questions. I'd be glad to assist you.

I'll be waiting for your reply.


REFERENCES:

[1] https://cloud.google.com/compute/docs/disks/add-persistent-disk#inaccessible_instance [2] https://cloud.google.com/compute/docs/disks/add-persistent-disk#create_disk [3] https://cloud.google.com/compute/docs/disks/add-persistent-disk#formatting [4] https://cloud.google.com/compute/docs/disks/detach-reattach-boot-disk#detach_disk

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.