I'm running cloud:bionic-rocky. I've run into a wall trying to diagnose an issue with glance which is blocking the creation of new instances.
I can replicate the problem from the command line;
routergod@juju:~/openstack-base$ glance image-download --file cirros.image 2302424f-51b5-4075-8f2a-84bc2d0071bf
Error finding address for http://192.168.200.201:9292/v2/images/2302424f-51b5-4075-8f2a-84bc2d0071bf/file: Unable to establish connection to http://192.168.200.201:9292/v2/images/2302424f-51b5-4075-8f2a-84bc2d0071bf/file: ('Connection aborted.', BadStatusLine("''",))
The operation unexpectedly only produces a single line in the normal glance-api.log;
2019-06-14 10:04:55.651 27832 INFO eventlet.wsgi.server [-] 192.168.200.201 - - [14/Jun/2019 10:04:55] "GET /versions HTTP/1.1" 200 1025 0.002114
The debug logs are pretty brief too (see below).
If I log in to the glance server, I can download the image from rbd/ceph;
root@juju-533538-0-lxd-2:~# rbd --id glance -p glance export 2302424f-51b5-4075-8f2a-84bc2d0071bf cirros.test.img
Exporting image: 100% complete...done.
root@juju-533538-0-lxd-2:~# ls -l cirros.test.img
-rw-r--r-- 1 root root 12716032 Jun 14 10:11 cirros.test.img
All very confusing, I'm struggling to find an actual error or exception anywhere. Any clues please? Many thanks in advance!
/.J
Debug log:
2019-06-14 10:04:55.648 27832 DEBUG eventlet.wsgi.server [-] (27832) accepted ('192.168.200.201', 38290) server /usr/lib/python3/dist-packages/eventlet/wsgi.py:883
2019-06-14 10:04:55.650 27832 DEBUG glance.api.middleware.version_negotiation [-] Determining version of request: GET /versions Accept: */* process_request /usr/lib/python3/dist-packages/glance/api/middleware/version_negotiation.py:45
2019-06-14 10:04:55.651 27832 INFO eventlet.wsgi.server [-] 192.168.200.201 - - [14/Jun/2019 10:04:55] "GET /versions HTTP/1.1" 200 1025 0.002114
2019-06-14 10:04:56.229 27832 DEBUG eventlet.wsgi.server [-] (27832) accepted ('192.168.200.201', 38302) server /usr/lib/python3/dist-packages/eventlet/wsgi.py:883
2019-06-14 10:04:56.231 27832 DEBUG glance.api.middleware.version_negotiation [-] Determining version of request: GET /v2/images/2302424f-51b5-4075-8f2a-84bc2d0071bf/file Accept: */* process_request /usr/lib/python3/dist-packages/glance/api/middleware/version_negotiation.py:45
2019-06-14 10:04:56.231 27832 DEBUG glance.api.middleware.version_negotiation [-] Using url versioning process_request /usr/lib/python3/dist-packages/glance/api/middleware/version_negotiation.py:57
2019-06-14 10:04:56.231 27832 DEBUG glance.api.middleware.version_negotiation [-] Matched version: v2 process_request /usr/lib/python3/dist-packages/glance/api/middleware/version_negotiation.py:69
2019-06-14 10:04:56.232 27832 DEBUG glance.api.middleware.version_negotiation [-] new path /v2/images/2302424f-51b5-4075-8f2a-84bc2d0071bf/file process_request /usr/lib/python3/dist-packages/glance/api/middleware/version_negotiation.py:70
2019-06-14 10:04:56.530 27832 DEBUG oslo_policy._cache_handler [req-d6a7ad6c-8f7a-40e8-bc4e-8b08c8ce5998 41f5b059d3bf4bebad08ffcabd0899bf d25b3d096e5c47a3b1701b3fff2c2823 - f7f19dbcac14459c99e4dc1aad7e0df0 f7f19dbcac14459c99e4dc1aad7e0df0] Reloading cached file /etc/glance/policy.json read_cached_file /usr/lib/python3/dist-packages/oslo_policy/_cache_handler.py:40
2019-06-14 10:04:56.531 27832 DEBUG oslo_policy.policy [req-d6a7ad6c-8f7a-40e8-bc4e-8b08c8ce5998 41f5b059d3bf4bebad08ffcabd0899bf d25b3d096e5c47a3b1701b3fff2c2823 - f7f19dbcac14459c99e4dc1aad7e0df0 f7f19dbcac14459c99e4dc1aad7e0df0] Reloaded policy file: /etc/glance/policy.json _load_policy_file /usr/lib/python3/dist-packages/oslo_policy/policy.py:770
2019-06-14 10:04:56.536 27832 DEBUG glance_store.backend [req-d6a7ad6c-8f7a-40e8-bc4e-8b08c8ce5998 41f5b059d3bf4bebad08ffcabd0899bf d25b3d096e5c47a3b1701b3fff2c2823 - f7f19dbcac14459c99e4dc1aad7e0df0 f7f19dbcac14459c99e4dc1aad7e0df0] Attempting to import store glance.store.rbd.Store _load_store /usr/lib/python3/dist-packages/glance_store/backend.py:267
2019-06-14 10:04:56.538 27832 DEBUG glance_store.capabilities [req-d6a7ad6c-8f7a-40e8-bc4e-8b08c8ce5998 41f5b059d3bf4bebad08ffcabd0899bf d25b3d096e5c47a3b1701b3fff2c2823 - f7f19dbcac14459c99e4dc1aad7e0df0 f7f19dbcac14459c99e4dc1aad7e0df0] Store glance_store._drivers.rbd.Store doesn't support updating dynamic storage capabilities. Please overwrite 'update_capabilities' method of the store to implement updating logics if needed. update_capabilities /usr/lib/python3/dist-packages/glance_store/capabilities.py:97
2019-06-14 10:04:56.539 27832 DEBUG glance_store.driver [req-d6a7ad6c-8f7a-40e8-bc4e-8b08c8ce5998 41f5b059d3bf4bebad08ffcabd0899bf d25b3d096e5c47a3b1701b3fff2c2823 - f7f19dbcac14459c99e4dc1aad7e0df0 f7f19dbcac14459c99e4dc1aad7e0df0] Late loading location class glance_store._drivers.rbd.StoreLocation get_store_location_class /usr/lib/python3/dist-packages/glance_store/driver.py:112
2019-06-14 10:04:56.539 27832 DEBUG glance_store.location [req-d6a7ad6c-8f7a-40e8-bc4e-8b08c8ce5998 41f5b059d3bf4bebad08ffcabd0899bf d25b3d096e5c47a3b1701b3fff2c2823 - f7f19dbcac14459c99e4dc1aad7e0df0 f7f19dbcac14459c99e4dc1aad7e0df0] Registering scheme rbd with {'store': <glance_store._drivers.rbd.Store object at 0x7fe699a4bbe0>, 'location_class': <class 'glance_store._drivers.rbd.StoreLocation'>, 'store_entry': 'glance.store.rbd.Store'} register_scheme_map /usr/lib/python3/dist-packages/glance_store/location.py:142
