OpenStack Queens: Horizon Fails after rebooting the controller node

Question

The controller node was rebooted and some VMs were running. After that, when logging to Horizon, I keep receiving the following error: Error: Unable to retrieve instances. And I can no longer see the VMs that was running. The following is the output of /var/log/apache2/error.log. I am beginner and I am running out of ideas to solve the issue, any help is much appreciated.

[Mon Jul 11 13:26:04.516711 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458] WARNING horizon.exceptions Recoverable error: The server is
currently unavailable. Please try again at a later time.<br /><br />
[Mon Jul 11 13:26:04.516739 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]
[Mon Jul 11 13:26:04.516745 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]
[Mon Jul 11 13:26:04.516750 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]  (HTTP 503) (Request-ID: req-88698cf2-ea4a-4bdf-99f9-32f6e49
7338f)
[Mon Jul 11 13:26:04.648861 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458] ERROR openstack_dashboard.dashboards.project.trunks.panel Ca
ll to list enabled services failed. This is likely due to a problem communicating with the Neutron endpoint. Trunks panel will not be displayed.
[Mon Jul 11 13:26:04.691047 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458] ERROR openstack_dashboard.dashboards.admin.aggregates.panel Call to list supported extensions failed. This is likely due to a problem communicating with the Nova endpoint. Host Aggregates panel will not be displayed.
[Mon Jul 11 13:26:04.730674 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458] ERROR openstack_dashboard.dashboards.admin.trunks.panel Call to list enabled services failed. This is likely due to a problem communicating with the Neutron endpoint. Trunks admin panel will not be displayed.
[Mon Jul 11 13:26:04.785539 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458] ERROR openstack_dashboard.dashboards.project.instances.tables Failed to retrieve quota information
[Mon Jul 11 13:26:04.785552 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458] Traceback (most recent call last):
[Mon Jul 11 13:26:04.785556 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]   File "/usr/share/openstack-dashboard/openstack_dashboard/dashboards/project/instances/tables.py", line 396, in allowed
[Mon Jul 11 13:26:04.785559 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]     limits = api.nova.tenant_absolute_limits(request, reserved=True)
[Mon Jul 11 13:26:04.785562 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]   File "/usr/share/openstack-dashboard/openstack_dashboard/api/nova.py", line 907, in tenant_absolute_limits
[Mon Jul 11 13:26:04.785566 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]     tenant_id=tenant_id).absolute
[Mon Jul 11 13:26:04.785569 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]   File "/usr/lib/python2.7/dist-packages/novaclient/v2/limits.py", line 100, in get
[Mon Jul 11 13:26:04.785572 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]     return self._get("/limits%s" % query_string, "limits")
[Mon Jul 11 13:26:04.785575 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]   File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 351, in _get
[Mon Jul 11 13:26:04.785578 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]     resp, body = self.api.client.get(url)
[Mon Jul 11 13:26:04.785581 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]   File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 304, in get
[Mon Jul 11 13:26:04.785584 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]     return self.request(url, 'GET', **kwargs)
[Mon Jul 11 13:26:04.785592 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 83, in request
[Mon Jul 11 13:26:04.785596 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]     raise exceptions.from_response(resp, body, url, method)
[Mon Jul 11 13:26:04.785598 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458] ClientException: The server is currently unavailable. Please try again at a later time.<br /><br />
[Mon Jul 11 13:26:04.785601 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]
[Mon Jul 11 13:26:04.785604 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]
[Mon Jul 11 13:26:04.785606 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]  (HTTP 503) (Request-ID: req-9b69b1fa-6832-4d2f-ac33-de6fbb9814aa)
[Mon Jul 11 13:26:04.823817 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458] ERROR openstack_dashboard.dashboards.project.instances.tables Failed to retrieve quota information
[Mon Jul 11 13:26:04.823830 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458] Traceback (most recent call last):
[Mon Jul 11 13:26:04.823834 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]   File "/usr/share/openstack-dashboard/openstack_dashboard/dashboards/project/instances/tables.py", line 396, in allowed
[Mon Jul 11 13:26:04.823837 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]     limits = api.nova.tenant_absolute_limits(request, reserved=True)
[Mon Jul 11 13:26:04.823840 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]   File "/usr/share/openstack-dashboard/openstack_dashboard/api/nova.py", line 907, in tenant_absolute_limits
[Mon Jul 11 13:26:04.823844 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]     tenant_id=tenant_id).absolute
[Mon Jul 11 13:26:04.823847 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]   File "/usr/lib/python2.7/dist-packages/novaclient/v2/limits.py", line 100, in get
[Mon Jul 11 13:26:04.823861 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]     return self.request(url, 'GET', **kwargs)
[Mon Jul 11 13:26:04.823864 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]   File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 83, in request
[Mon Jul 11 13:26:04.823867 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]     raise exceptions.from_response(resp, body, url, method)
[Mon Jul 11 13:26:04.823870 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458] ClientException: The server is currently unavailable. Please try again at a later time.<br /><br />
[Mon Jul 11 13:26:04.823873 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]
[Mon Jul 11 13:26:04.823876 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]
[Mon Jul 11 13:26:04.823878 2022] [wsgi:error] [pid 1509:tid 140620901611264] [remote 192.168.100.66:50458]  (HTTP 503) (Request-ID: req-3eab18f5-4d6f-4f32-b505-2d718ce

Answer 1

You need to check if all other openstack services are running, like nova, neutron etc. systemctl status openstack-nova-api.service should show you if nova-api is running (the service responsible for managing VMs). Check all other services, too. Usually, the services should be enabled so a reboot would start them. If they are not enabled, enable them, e.g. systemctl enable --now openstack-nova-api.service. The --now flag also starts the service in case it is stopped. How did you deploy this environment? The commands may differ if it was deployed by kolla or other deployment methods.