1

I have a fresh OpenStack Kilo installation.

When I try to create a snapshot from the dashboard or from the nova cli, the snapshot isn't being created.

The local snapshot ends with success, but nova get's a "HTTP 400: Invalid disk format 'None' for image" error when trying to upload the image to glance.

The full log follows at the end.

The question is: Why is this error occurring? And how can I solve it?

Update: After some tests, I discovered that Nova uses the Glance API v1 to upload the image. The request done by nova is right and both container_format and bare_format is sent. The request is also complaint with the Glance v1 API specification. However, something is going wrong, as we can see at the following curl command to the Glance API:

curl -i -X PUT -H 'X-Auth-Token: MY_TOKEN_HERE' \
    -H 'x-image-meta-property-image_state: available' \
    -H 'x-image-meta-name: rarytest2-snapshot' -H 'x-image-meta-container_format: bare' \
    -H 'User-Agent: python-glanceclient' -H 'X-User-Id: e45cf65e7ae94cd5aa86341c87308657' \
    -H 'x-image-meta-property-image_location: snapshot' -H 'Accept-Encoding: gzip, deflate' \
    -H 'x-glance-registry-purge-props: false' -H 'x-image-meta-property-ramdisk_id: ' \
    -H 'Connection: keep-alive' -H 'x-image-meta-property-kernel_id: ' \
    -H 'X-Service-Catalog: []' \
    -H 'x-image-meta-property-owner_id: 2dd6aa6ee0554c0ea5afce27dc63ff3a' \
    -H 'Accept: */*' -H 'X-Roles: _member_,admin' -H 'x-image-meta-size: 1549991936' \
    -H 'X-Tenant-Id: 2dd6aa6ee0554c0ea5afce27dc63ff3a' -H 'x-image-meta-is_public: False' \
    -H 'X-Identity-Status: Confirmed' -H 'Content-Type: application/octet-stream' \
    -H 'x-image-meta-disk_format: qcow2' \
    http://MY_IP_HERE:9292/v1/images/3b3cdac1-d76c-48f2-8150-0977d8720684

The expected behavior is update disk_format to qcow2 and container_format to bare. However, it doesn't occur.

For some strange reason, if a change the x-image-meta-disk_format and x-image-meta-container_format headers for x-image-meta-disk-format and x-image-meta-container-format (it is, changing underscore by hyphens) it works!

Full log:

2015-05-24 17:50:46.229 2400 INFO nova.compute.manager [req-b6489814-f538-4773-a15d-c1224b5583b7 e45cf65e7ae94cd5aa86341c87308657 2dd6aa6ee0554c0ea5afce27dc63ff3a - - -] [instance: 00652f00-5a32-44ad-8ad6-ac33610236ac] instance snapshotting
[...]
2015-05-24 17:50:58.670 2400 INFO nova.virt.libvirt.driver [req-b6489814-f538-4773-a15d-c1224b5583b7 e45cf65e7ae94cd5aa86341c87308657 2dd6aa6ee0554c0ea5afce27dc63ff3a - - -] [instance: 00652f00-5a32-44ad-8ad6-ac33610236ac] Snapshot extracted, beginning image upload
[...]
2015-05-24 17:51:01.251 2400 ERROR oslo_messaging.rpc.dispatcher [req-b6489814-f538-4773-a15d-c1224b5583b7 e45cf65e7ae94cd5aa86341c87308657 2dd6aa6ee0554c0ea5afce27dc63ff3a - - -] Exception during message handling: <html>
 <head>
  <title>400 Bad Request</title>
 </head>
 <body>
  <h1>400 Bad Request</h1>
  Invalid disk format 'None' for image.<br /><br />

 </body>
</html> (HTTP 400)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6791, in snapshot_instance
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     return self.manager.snapshot_instance(ctxt, image_id, instance)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     payload)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 327, in decorated_function
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance_uuid=instance_uuid)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 298, in decorated_function
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 355, in decorated_function
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 343, in decorated_function
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 403, in decorated_function
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     instance=instance)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 393, in decorated_function
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     *args, **kwargs)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3227, in snapshot_instance
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     task_states.IMAGE_SNAPSHOT)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3257, in _snapshot_instance
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     update_task_state)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1410, in snapshot
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     image_file)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/image/api.py", line 130, in update
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     purge_props=purge_props)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 398, in update
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     _reraise_translated_image_exception(image_id)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 396, in update
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     image_id, **image_meta)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 219, in call
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     return getattr(client.images, method)(*args, **kwargs)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 329, in update
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     resp, body = self.client.put(url, headers=hdrs, data=image_data)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 265, in put
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     return self._request('PUT', url, **kwargs)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 227, in _request
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher     raise exc.from_response(resp, resp.content)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher Invalid: <html>
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher  <head>
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   <title>400 Bad Request</title>
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher  </head>
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher  <body>
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   <h1>400 Bad Request</h1>
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher   Invalid disk format 'None' for image.<br /><br />
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher 
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher  </body>
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher </html> (HTTP 400)
2015-05-24 17:51:01.251 2400 TRACE oslo_messaging.rpc.dispatcher 
1

The problem occurs due to use of both Nginx and Glance API v1.

In short, the explanation is found at this thread in Stack Overflow.

Nginx, by default, drops headers with underscores. This is done to:

prevent ambiguities when mapping headers to CGI variables

Reference: Nginx - Pitfalls - Missing (disappearing) HTTP headers.

However, headers with underscores are specified in the Glance v1 API.

Reference: Glance v1 API - Retrieve image metadata.

I also discovered that headers with underscores are completely valid and complaint with the RFC 7230.

The solution is to add the following line to the nginx config file:

underscores_in_headers on;

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.