| opendevreview | Pranali Deore proposed openstack/glance master: Make location URL compatible with cinder backend https://review.opendev.org/c/openstack/glance/+/906359 | 07:41 |
|---|---|---|
| opendevreview | Pranali Deore proposed openstack/glance master: Add Location Import task flow https://review.opendev.org/c/openstack/glance/+/886749 | 07:41 |
| opendevreview | Pranali Deore proposed openstack/glance master: Add new add location api https://review.opendev.org/c/openstack/glance/+/881940 | 07:41 |
| opendevreview | Pranali Deore proposed openstack/glance master: Add functional tests for new add-location API https://review.opendev.org/c/openstack/glance/+/886947 | 07:41 |
| opendevreview | Pranali Deore proposed openstack/glance master: Add new get location api https://review.opendev.org/c/openstack/glance/+/882498 | 07:41 |
| opendevreview | Pranali Deore proposed openstack/glance master: Add api-ref and documentation for new location apis https://review.opendev.org/c/openstack/glance/+/890508 | 07:41 |
| opendevreview | Pranali Deore proposed openstack/glance master: Bump Images API version to 2.17 https://review.opendev.org/c/openstack/glance/+/890509 | 07:41 |
| *** mklejn_ is now known as mklejn | 08:34 | |
| opendevreview | Pranali Deore proposed openstack/glance master: Make location URL compatible with cinder backend https://review.opendev.org/c/openstack/glance/+/906359 | 10:58 |
| opendevreview | Pranali Deore proposed openstack/glance master: Add Location Import task flow https://review.opendev.org/c/openstack/glance/+/886749 | 10:59 |
| opendevreview | Pranali Deore proposed openstack/glance master: Add new add location api https://review.opendev.org/c/openstack/glance/+/881940 | 10:59 |
| opendevreview | Pranali Deore proposed openstack/glance master: Add functional tests for new add-location API https://review.opendev.org/c/openstack/glance/+/886947 | 10:59 |
| opendevreview | Pranali Deore proposed openstack/glance master: Add new get location api https://review.opendev.org/c/openstack/glance/+/882498 | 11:00 |
| opendevreview | Pranali Deore proposed openstack/glance master: Add api-ref and documentation for new location apis https://review.opendev.org/c/openstack/glance/+/890508 | 11:00 |
| opendevreview | Pranali Deore proposed openstack/glance master: Bump Images API version to 2.17 https://review.opendev.org/c/openstack/glance/+/890509 | 11:00 |
| pranali | abhishekk, rosmaita, dansmith, croelandt, mrjoshi glance weekly meeting in 10 minutes at #openstack-meeting | 13:50 |
| rosmaita | pranali: i have a conflict, so will be paying even less attention than usual ... apologies in advance | 13:50 |
| pranali | rosmaita, ack np !! | 13:51 |
| abhishekk | dansmith, is something changed in nova, these tests suddenly started failing for nova-ceph-multistore job | 14:22 |
| abhishekk | https://64d1a84195be846e8a85-cb762a113d4d5e3960c90c07c5780a8b.ssl.cf2.rackcdn.com/899870/14/check/nova-ceph-multistore/b3faf28/testr_results.html | 14:22 |
| dansmith | abhishekk: not that I know of | 14:27 |
| abhishekk | https://zuul.opendev.org/t/openstack/builds?job_name=nova-ceph-multistore&skip=0 | 14:27 |
| dansmith | abhishekk: that's a 400 from glance right? | 14:28 |
| abhishekk | yes, but it related to http location | 14:28 |
| dansmith | I don't think it would be a nova thing then, maybe devstack? but no, I idk | 14:29 |
| abhishekk | ack, will check | 14:33 |
| dansmith | we've been seeing failures related to other things, so I'd have to expect that most of those failures are unrelated to http images, but if not I'd be glad to know | 14:34 |
| abhishekk | surprisingly test failing in nova-ceph-multistore 'test_image_web_download_import' is passing in tempest-integrated-storage job | 14:39 |
| abhishekk | https://review.opendev.org/c/openstack/glance/+/899870 | 14:39 |
| dansmith | okay, well, I'm sure you can figure out why glance is returning 400, which is probably a clue :) | 14:40 |
| abhishekk | yeah | 14:40 |
| abhishekk | I need to configure http store and use this url to add the location http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-uec.tar.gz | 14:46 |
| abhishekk | which is working locally :/ | 14:50 |
| croelandt | pranali: apparently https://review.opendev.org/c/openstack/glance/+/886947/ causes a failure with functional tests | 14:57 |
| croelandt | well, functional-py38-fips fails but functional-py{39,310} works | 14:57 |
| croelandt | not sure if that is an issue with your patch | 14:57 |
| abhishekk | yes, that is issue with patch | 14:58 |
| abhishekk | the reason is image state transition form queued-importing-queued happens before test actually reach that point and it waits and timeouts for state transition from queued > importing | 14:59 |
| croelandt | but why is it only failing with 38-fips ? :) | 14:59 |
| croelandt | is it just a race condition that may happen with any other interpreter? | 15:00 |
| abhishekk | it fails randomly for either of functional jon | 15:02 |
| abhishekk | check other patches | 15:02 |
| croelandt | oh ok | 15:04 |
| croelandt | oh yeah from the functional patch, we get failures | 15:05 |
| abhishekk | yes | 15:07 |
| abhishekk | I will be back in couple of hours, travelling back to home | 15:07 |
| abhishekk | if possible please check nova-ceph-multistore failure till then | 15:07 |
| croelandt | abhishekk: https://review.opendev.org/c/openstack/glance/+/906359/ already shows failures because of BadStoreURI | 16:54 |
| croelandt | unfortunately, we don't log said uri | 16:55 |
| croelandt | but I think there might be a problem with the way we rewrite the location URL | 16:55 |
| abhishekk | you are talking in context of nova-ceph-multistore failure? | 16:56 |
| croelandt | yes | 17:00 |
| croelandt | I did not realize this was from the first patch | 17:01 |
| abhishekk | Feb 22 12:31:29.073537 np0036837391 glance-api[105921]: INFO glance_store._drivers.http [None req-bfa9796f-fb8c-4549-b484-5a06f762d9dd tempest-ImageLocationsAdminTest-1031512512 tempest-ImageLocationsAdminTest-1031512512-project-member] The HTTP URL is invalid: HTTPConnectionPool(host='download.cirros-cloud.net', port=80): Max retries exceeded with url: /0.3.1/cirros-0.3.1-x86_64-uec.tar.gz (Caused by NewConnectionError('<urllib3.connection.H | 17:05 |
| abhishekk | TTPConnection object at 0x7f18a218d660>: Failed to establish a new connection: [Errno -3] Lookup timed out')) | 17:05 |
| abhishekk | this error is further raised as BadStore from here | 17:06 |
| abhishekk | https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/http.py#L260 | 17:06 |
| abhishekk | glance location-add 2819a9e5-3a77-4abd-b31e-70e953295a4a --url https://download.cirros-cloud.net:80/0.3.1/cirros-0.3.1-x86_64-uec.tar.gz | 17:09 |
| abhishekk | HTTP 400 Bad Request: The Store URI was malformed. | 17:09 |
| abhishekk | if I pass --url as above I get same error message in glance logs | 17:09 |
| abhishekk | if I just pass it without port 80 then it succeeds | 17:10 |
| croelandt | hm | 17:10 |
| croelandt | why is this raised as BadStore? That seems buggy | 17:10 |
| croelandt | I guess we have no way of deciding whether the URI is bad or the server is unreachable | 17:11 |
| abhishekk | Don't know why it is implemented like that | 17:12 |
| croelandt | https://download.cirros-cloud.net:80/ is indeed unreachable | 17:12 |
| abhishekk | yes, but what happened suddenly from today | 17:14 |
| croelandt | I'm not sure, probably some kind of issue with cirros-cloud.net? Do we know an admin there? :) | 17:15 |
| abhishekk | :d | 17:17 |
| croelandt | do we even specify cirros-cloud.net:80 anywhere in the tests? | 17:18 |
| abhishekk | I don't know | 17:20 |
| croelandt | https://github.com/openstack/tempest/blob/b641452a46792b5c7e3f64a1165c466bb08f7370/tempest/config.py#L641 this seems to be the default value... but with an added port somehow | 17:22 |
| abhishekk | yes | 17:24 |
| croelandt | oh ok so http://download.cirros-cloud.net/:80 is ok | 17:25 |
| croelandt | but https + port80 is not, obviously | 17:25 |
| croelandt | so maybe they made httpS default or something | 17:25 |
| croelandt | no, does not seem like it | 17:27 |
| * croelandt scratches head | 17:27 | |
| abhishekk | 7a7d010b-fcbd-413a-af34-430264aa9747 | 17:29 |
| abhishekk | https://download.cirros-cloud.net:443/0.3.1/cirros-0.3.1-x86_64-uec.tar.gz | 17:29 |
| abhishekk | works with this | 17:29 |
| croelandt | yeah because https +443 | 17:29 |
| croelandt | but Tempest seems to be using http, not https | 17:30 |
| croelandt | so port 80 should be fine | 17:30 |
| abhishekk_ | got disconnected | 17:33 |
| abhishekk_ | so can we change the default url to http+port443 ? | 17:33 |
| croelandt | http+443 will fail as well | 17:35 |
| croelandt | only http/80 and https/443 can work | 17:35 |
| croelandt | I don't understand why http/80 does not work in this case by the way | 17:36 |
| abhishekk_ | https://64d1a84195be846e8a85-cb762a113d4d5e3960c90c07c5780a8b.ssl.cf2.rackcdn.com/899870/14/check/nova-ceph-multistore/b3faf28/controller/logs/etc/glance/glance-api_conf.txt | 17:37 |
| abhishekk_ | is it because tls is enabled? | 17:40 |
| abhishekk_ | locally it is working with http+port80 | 17:46 |
| abhishekk_ | locations | [{"url": "http://download.cirros- | | 17:46 |
| abhishekk_ | | | cloud.net:80/0.3.1/cirros-0.3.1-x86_64-uec.tar.gz", "metadata": {"store": | | 17:46 |
| abhishekk_ | | | "abcd"}}] | 17:46 |
| abhishekk_ | I guess this is the failure | 18:00 |
| abhishekk_ | https://github.com/openstack/tempest/blob/master/tempest/api/image/base.py#L190 | 18:00 |
| abhishekk_ | guess not :/ | 18:01 |
| abhishekk_ | looks like related to eventlet | 18:16 |
| abhishekk_ | the difference is dnspython-2.6.0, that last time job passed the version was dnspython-2.5.0 | 18:22 |
| abhishekk_ | yes dnspython-2.6.0 is issue | 18:39 |
| abhishekk_ | croelandt, dansmith I had dnspython 2.4.2 in envrionment and location-add was working | 18:40 |
| abhishekk_ | I updated it to use 2.6.1 and it is failing now with same error | 18:40 |
| abhishekk_ | So how can we blacklist this new version | 18:40 |
| abhishekk_ | https://paste.opendev.org/show/b0akqNwUmQ1O9JQHi0Qw/ | 18:41 |
| dansmith | abhishekk_: probably need to move to -qa | 18:42 |
| dansmith | but we need to put something in u-c I guess | 18:42 |
| dansmith | https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L574 | 18:42 |
| dansmith | I assume propose that change with justification and raise it | 18:43 |
| croelandt | abhishekk_: is 2.6.1 the first version causing issues? | 18:45 |
| abhishekk_ | no upstream it is 2.6.0 | 18:45 |
| abhishekk_ | I just used pip3 install --upgrade dnspython and it has updated it to 2.6.1 | 18:46 |
| croelandt | https://github.com/rthalley/dnspython/issues/1050 this might be related :) | 18:48 |
| abhishekk_ | yeah, the same issue we are facing | 18:49 |
| croelandt | so I guess weneed to push a patch to openstack/requirements that sets u-c to 2.5.0 and link this issue as a justification | 18:50 |
| abhishekk_ | yes | 18:50 |
| abhishekk_ | can you please do it, its almost midnight at my end | 18:50 |
| abhishekk_ | I will quickly report a bug | 18:50 |
| croelandt | yeah sure | 18:52 |
| abhishekk_ | croelandt, https://bugs.launchpad.net/glance/+bug/2054732 | 18:54 |
| croelandt | Done https://review.opendev.org/c/openstack/requirements/+/909923 | 18:55 |
| abhishekk_ | that was quick | 18:56 |
| dansmith | croelandt: ++ | 18:56 |
| croelandt | abhishekk_: easier than figuring out the issue was with dnspython :p | 18:58 |
| abhishekk_ | :D | 18:58 |
| abhishekk_ | hopefully gate situation will change over weekend | 18:59 |
| abhishekk_ | happy long weekend | 18:59 |
| croelandt | yeah see you on Monday | 19:01 |
| abhishekk_ | o/ | 19:02 |
| dansmith | abhishekk_: I'm pushing in -tc now, should be merging I think | 19:07 |
| abhishekk_ | ++. thank you | 19:07 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!