Brief explanation about the issue-

I am experiencing VM restart every now and then. So whatever disk we attached get automatically unmounted. So as per the recommendations what we did is added mounting of disk with uuids in fstab. We tested the changes by restarting the VMs and every time mounting of disks worked fine. We made this changes in all VMs which we use on Azure. Yesterday we saw issue with one of our VM (xxxxxxxxxxxxxxxx.cloudapp.net), when we ssh into it we could not see mounted disks with "df -h" also the OS disk was full, with "fdisk -l" I could see the unmounted disks. I then tried few things

  • I tried restarting VM thinking that restarting VM with make fstab run and auto mount the disks. which didn't happened but till that time we were able to ssh successfully.
  • we then tried reset password of VM which look approximately and hour after that we see portal showing failed message. After that when we do "ssh azureuser@xxxxxxxxxxxxxxxx.cloudapp.net" we get this "ssh: connect to host xxxxxxxxxxxxxxxx.cloudapp.net port 22: Connection refused" looking at the Audit log we discovered this error

    OPERATION NAME Microsoft.Resources/deployments/write
    STATUS Failed
    LEVEL Error

    OPERATION NAME Microsoft.ClassicCompute/virtualMachines/extensions/write
    STATUS Failed
    LEVEL Error
    PROPERTIES statusCode:RequestTimeout statusMessage:{"error":{"code":"ResourceDeploymentFailure","message":" The resource provision operation did not complete within the allowed timeout period. Please seehttp://aka.ms/arm-deploy for usage details. "}}

I enabled Boot diagnostics option in VM and then tried restarting and now I could see error log in Boot diagnostics. With this error I see because the disk is full OS is not booting.

Traceback (most recent call last):
  File "/usr/bin/cloud-init", line 618, in <module>
  File "/usr/bin/cloud-init", line 614, in main
    get_uptime=True, func=functor, args=(name, args))
  File "/usr/lib/python2.7/dist-packages/cloudinit/util.py", line 1875, in log_time
    ret = func(*args, **kwargs)
  File "/usr/bin/cloud-init", lne 489, in stats_wrapper
    atomic_write_json(status_path, status)
  File "/usr/bin/cloud-init", line 434, in atomic_write_json
    raise e
IOError: [Errno28] No space left on device 
  • we also tried creating a VM with an image of VM which we created after this issue. Creation of VM worked fine with image but we were unable to ssh to that too.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.