| opendevreview | Mridula Joshi proposed openstack/glance-specs master: Expose store specific information https://review.opendev.org/c/openstack/glance-specs/+/817391 | 05:50 |
|---|---|---|
| crohmann | Hello everyone. I have an OpenStack Ussuri with Glance image import via web-download method "not working". The image itself (so the shell object with just the metadata yet) is created and also the corresponding task. The task state is switching to "processing", but then nothing happens. No obvious errors / warnings in the log file. | 08:36 |
| crohmann | Where should I look next for why the download to the staging space does not start? | 08:41 |
| opendevreview | Rajat Dhasmana proposed openstack/glance-specs master: Expose store specific information https://review.opendev.org/c/openstack/glance-specs/+/817391 | 10:21 |
| simondodsley | My devstack has recently started to fail when uploading the cirros image... What would be causing this error: https://www.irccloud.com/pastebin/rW1KmIsS/ | 13:40 |
| abhishekk | rosmaita, jokke_, dansmith, croelandt, pdeore glance weekly meeting in 15 minutes at #openstack-meeting | 13:47 |
| abhishekk | simondodsley, which devstack version you are using, is it latest ? | 13:52 |
| simondodsley | master | 13:53 |
| abhishekk | looking at error, it looks like it is not setting default limits in keystone for glance | 13:53 |
| abhishekk | can you try by setting this 'GLANCE_ENABLE_QUOTAS' False in local.conf ? | 13:54 |
| simondodsley | Yes - I can try that | 13:55 |
| abhishekk | cool, let me know what happens | 13:56 |
| *** whoami-rajat__ is now known as whoami-rajat | 14:00 | |
| crohmann | rosmaita: simondodsley: May bug you with a glance question real quick? I am trying to use the web-download import method - so actually the api_image_import task via the web-download. I am able to create tasks just fine, but they only reach the processing state - with literally no indication on what happens. I just need a pointer on where to dig deeper? How can I debug the task execution further? | 14:29 |
| rosmaita | crohmann: maybe join the glance meeting, happening right now in #openstack-meeting | 14:30 |
| crohmann | There is no video, just IRC, right? rosmaita ? | 14:36 |
| rosmaita | yes, irc only | 14:37 |
| crohmann | can I just dump a question in there? Don't want to mess up the conversation flow. | 14:37 |
| rosmaita | crohmann: you have excellent timing, open discussion just starting! | 14:40 |
| simondodsley | abhishekk: sadly that didn't resolve it. Same error. | 14:47 |
| abhishekk | strange | 14:47 |
| akekane__ | last 8 minutes | 14:53 |
| akekane__ | simondodsley, I need to try it locally | 14:53 |
| simondodsley | I want to try it again... let me get back to you | 14:54 |
| akekane__ | ack | 14:54 |
| frickler | simondodsley: I saw that issue while testing https://review.opendev.org/c/openstack/devstack/+/817074 , are you sure you are using latest master? | 14:55 |
| simondodsley | I may have added the paramer for disabling the glance quotas in the wrong zuul file... checking | 14:56 |
| akekane__ | simondodsley, I think this patch mentioned by frickler will also solve your issue | 14:57 |
| *** akekane__ is now known as abhishekk | 14:58 | |
| simondodsley | abhishekk: yes - running in a CI and it always pulls master | 14:59 |
| abhishekk | ack | 14:59 |
| * akekane_ going for dinner | 15:12 | |
| simondodsley | abhishekk: frickler: setting `GLANCE_ENABLE_QUOTAS = False` fixes this - thanks for the pointers. | 15:42 |
| crohmann | dansmith: I did check on our installation again and much say we are running the dedicated eventlet app port 9292, eventlet.wsgi.server. And then with apache being the reverse proxy in front. So I am wondering if this the the same issue or if web-download should work with this deployment? | 16:07 |
| dansmith | crohmann: if you're running glance-api as an eventlet standalone, it should work fine, even if apache is in front | 16:07 |
| dansmith | are you saying you didn't realize that you were already doing that, or are you saying you're going to try that arrangement? | 16:08 |
| crohmann | I did not realize that Glance was actually NOT mod_wsgi, but other components are. | 16:08 |
| crohmann | That's what go me confused. | 16:08 |
| crohmann | so I am now wondering if there was anything else you can think of, that we should look at that could cause our web-download to not start. | 16:10 |
| crohmann | I mean the task is picked up by the tasks executor, it's just not starting a download / creating a file or throwing any errors. | 16:10 |
| dansmith | ah okay, so.. first off, you definitely do not want to just be creating the task, you want to be doing import | 16:10 |
| dansmith | can you show me what import call you're making? | 16:11 |
| crohmann | I did use that shorthand as well. | 16:11 |
| dansmith | and then logs showing the relevant timeframe | 16:11 |
| crohmann | "glance image-create-via-import --name crohmann-via-import --import-method web-download --uri https://cloud.debian.org/images/cloud/OpenStack/current-10/debian-10-openstack-amd64.raw --container-format bare --disk-format raw" | 16:12 |
| dansmith | okay, I'm not as familiar with using the create-via-import instead of create and then import, but yeah let me see logs | 16:12 |
| dansmith | I assume that ^ should work just like create..import | 16:13 |
| crohmann | ah, what would be your approach / command then? | 16:13 |
| crohmann | That image-create-via-import apparently is EXPERIMENTAL anyways. | 16:13 |
| dansmith | create and then import, but again, that's just because it's how I do things to test stepwise | 16:13 |
| dansmith | can you pastebin the logs? | 16:14 |
| crohmann | I was trying to emulate the steps tools like https://github.com/osism/openstack-image-manager do. | 16:14 |
| crohmann | Yeah - I am just now doing a clean run to have some readable logs. | 16:16 |
| crohmann | I would use your commands though, just to be save. Where do you set the URI in your approach? Is that an option for the create or import command? | 16:16 |
| *** tosky_ is now known as tosky | 16:17 | |
| dansmith | for import | 16:17 |
| dansmith | the create is just creating a record, with no data, the import is the thing that "fills" it, so it takes the uri | 16:17 |
| dansmith | but really, I wouldn't expect this is a problem with the client if you see the task getting picked up, so .. logs would be the best way to make progress here :) | 16:17 |
| *** priteau is now known as Guest7388 | 16:38 | |
| *** priteau_ is now known as priteau | 16:38 | |
| crohmann | sorry dansmith ... there you go: https://pastebin.com/yDnn3bdz | 16:43 |
| dansmith | crohmann: okay that's showing a POST of the task, | 16:46 |
| dansmith | which is not how it should be working | 16:46 |
| dansmith | so perhaps the client *is* doing something different for that create-and-import thing | 16:47 |
| dansmith | crohmann: can you try create and then import to see if it behaves differently? | 16:47 |
| crohmann | I just did, but then I am hitting an error with SQLAlchemy due to us using PostgreSQL with the handling of boolean vs. 0/1 :-( | 16:52 |
| crohmann | "oslo_db.exception.DBError: (psycopg2.errors.UndefinedFunction) operator does not exist: boolean <> integer" | 16:52 |
| crohmann | PostgreSQL does not like this: "UPDATE image_properties SET updated_at=%(updated_at)s, deleted=%(deleted)s, value=%(value)s WHERE image_properties.name = %(name_1)s AND image_properties.image_id = %(image_id_1)s AND image_properties.deleted != %(deleted_1)s]" | 16:53 |
| dansmith | well, | 16:54 |
| dansmith | that probably means it is indeed doing the right thing now, but yeah... that's a problem | 16:55 |
| dansmith | I'm surprised the SA dialect stuff doesn't handle that for us | 16:56 |
| dansmith | but as you probably no, there's very little if any testing on pg these days | 16:56 |
| dansmith | no? know. | 16:56 |
| crohmann | Yeah. We ran into PostgreSQL issues before and I know we have to switch. | 16:56 |
| crohmann | too bad there is no more sunsetting on PostgreSQL support than this: https://governance.openstack.org/tc/resolutions/20170613-postgresql-status.html | 16:58 |
| dansmith | well, there was some pushback, so it's one of those things where if we say "no" then there's pushback, but if we let it atrophy there isn't | 16:59 |
| dansmith | which sucks, but.. you know :/ | 16:59 |
| crohmann | I shall be setting up another DB on this DEV machine to at least be able to reproduce or dig further into the web-download issue. May I contact you again tomorrow or Monday when I got around to it? | 16:59 |
| dansmith | crohmann: can you show me the full log before you hit that failure just to see if it looks like it's getting further? | 17:00 |
| crohmann | sure - 1 minute | 17:00 |
| dansmith | crohmann: you are welcome to ping me tomorrow or next week. however, you should know I'll be on PTO for the rest of the year after friday :P | 17:00 |
| dansmith | crohmann: abhishekk is a more than ample substitute for me, and he will be around I think | 17:01 |
| dansmith | crohmann: he even rides a cool motorcycle | 17:01 |
| crohmann | dansmith: reload the pastebin, updated the log | 17:02 |
| dansmith | Error, this is a private paste or is pending moderation. If this paste belongs to you, please login to Pastebin to view it. | 17:03 |
| dansmith | not sure you can update a pastebin, it's a new url right? | 17:03 |
| crohmann | argh "Pastebin’s SMART filters have detected potentially offensive or questionable content in your paste. | 17:05 |
| crohmann | The content you are trying to publish has been deemed potentially offensive or questionable by our filters, because of this you’re receiving this warning. | 17:05 |
| dansmith | hah | 17:05 |
| dansmith | maybe they're punishing you for running postgres? :P | 17:06 |
| dansmith | paste.openstack.org | 17:06 |
| dansmith | slower and less good, but probably also less smart | 17:06 |
| crohmann | oh man ... that throws a proxy error 500 :-( | 17:12 |
| dansmith | lol | 17:13 |
| dansmith | https://paste.debian.net/ | 17:13 |
| crohmann | https://paste.debian.net/hidden/c72d74d7/ | 17:16 |
| crohmann | dansmith: Thanks for your patience ;-) | 17:16 |
| dansmith | crohmann: okay it didn't even get to the internal task bit | 17:17 |
| dansmith | but that's just because of where we hit the db issue, | 17:17 |
| dansmith | this one is clearly in the import path, which is what you want | 17:17 |
| dansmith | so you can file a bug if you want with the PG problem, that log tells us where it is | 17:18 |
| dansmith | and I could maybe hack up the patch quickly if you wanted to try to apply it locally | 17:18 |
| crohmann | I just switched to Sqlite for this one box. Let me see if that work. As for the PostgreSQL issue - I gladly raise a bug, if you are interested in fixing this. | 17:19 |
| dansmith | https://termbin.com/1j1f | 17:20 |
| crohmann | dansmith: You won't believe it ... with SQlite it appears to just work. Nothing changed but the database. | 17:24 |
| dansmith | I would believe it :) | 17:24 |
| dansmith | so "works" meaning the import and the web-download, etc? | 17:25 |
| dansmith | if you will file a bug for the pg thing and include that full trace in the bug, I will push the patch against it | 17:25 |
| crohmann | I sure will do so - thanks again. | 17:26 |
| dansmith | and also, if you're willing to file a bug against create-and-import not working (assuming it still doesn't, which I assume is true) that would also be great | 17:26 |
| dansmith | crohmann: shoot me the bug numbers/links here when you do, if you don't mind | 17:26 |
| crohmann | Well I was trying to use https://github.com/osism/openstack-image-manager initally. | 17:26 |
| crohmann | So I believe their approach at https://github.com/osism/openstack-image-manager/blob/1b952c783acb648f28299e6ccc0b7b9af77085b1/src/manage.py#L54 is not working? | 17:27 |
| dansmith | I expect not, and even if it does, it skips some other machinery we have around making it threadsafe, etc | 17:27 |
| dansmith | I have to jump on a call, bbl | 17:27 |
| crohmann | allright, it's late here anyways. I shall drop your the bug id | 17:28 |
| crohmann | Thanks a 1000 | 17:28 |
| dansmith | cool, thanks | 17:28 |
| crohmann | dansmith: Here you go with the bug id: https://bugs.launchpad.net/glance/+bug/1953063 | 18:09 |
| opendevreview | Dan Smith proposed openstack/glance master: Fix set_property_atomic() boolean type casting https://review.opendev.org/c/openstack/glance/+/820247 | 18:57 |
| dansmith | crohmann: thanks ^ | 18:57 |
| rosmaita | dansmith: that's a blast from the past: https://review.opendev.org/c/openstack/glance/+/544105 | 19:29 |
| dansmith | rosmaita: hah | 19:35 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!