0

Already asked this over on ask.openstack.org but I figured it would get more views here too.

I am running OpenStack Mitaka on a 5 node cluster, 1 controller/network node and 4 compute nodes. I am able to create VMs and use them normally throughout my day but when I come back in the morning some instances are powered off.

I've checked to make sure there is enough disk space (OK). I've checked if the problem is on any compute node (YES). I've also checked the RAM and CPU usage (OK).

I've tried to change the handle_virt_lifecycle_events variable in nova.conf to true and false with no success as some research has led me to.

It seems like the power state from the DB to the hypervisor (KVM) being mismatched is causing this.

Any ideas?

Here's the only relevant information that I could fine from /var/log/nova/nova-compute.log:

2016-06-08 21:20:18.208 1738 INFO nova.compute.manager [-] [instance: 493ad84c-1e46-410c-8fab-078c77ddea10] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor.
2016-06-08 21:20:18.228 1738 INFO nova.compute.manager [-] [instance: 2dcb6116-55ab-4987-a4f5-ce3e0f1edda4] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor.
2016-06-08 21:20:18.232 1738 INFO nova.compute.manager [-] [instance: d1d0172d-20aa-4333-b9e9-aabc064287bc] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor.
2016-06-08 21:20:18.298 1738 WARNING nova.compute.manager [-] [instance: 493ad84c-1e46-410c-8fab-078c77ddea10] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4
2016-06-08 21:20:18.314 1738 WARNING nova.compute.manager [-] [instance: 2dcb6116-55ab-4987-a4f5-ce3e0f1edda4] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4
2016-06-08 21:20:18.334 1738 WARNING nova.compute.manager [-] [instance: d1d0172d-20aa-4333-b9e9-aabc064287bc] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4
0

In /etc/nova/nova.conf, set the following option to disable the synchronization

sync_power_state_interval=-1

This will prevent enforcing VM state from DB.

Quoting from here:

The synchronization task will see that the DB is out of sync and will try to 'correct' the inconsistency by powering off the VM.

  • This won't stop the instance from shutting itself down, it will only stop nova from reporting its status correctly. – Michael Hampton Nov 8 '16 at 13:20
  • It made the difference at my end (Mitaka). as stated here: link – adiflinux Nov 8 '16 at 13:58
  • If nova shut down the VM, then yes it would help, but if the instance shut down from within the instance, as it seems happened here, then it will not. – Michael Hampton Nov 8 '16 at 13:59
  • @Michael Hampton - I think the error message is missleading. You'll mostly find these two lines simultaneously: ...During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor. ...Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4 indicating that nova periodic task were responsible for that. Worth trying.. – adiflinux Nov 8 '16 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.