| *** openstack has joined #openstack-glance | 07:29 | |
| *** ChanServ sets mode: +o openstack | 07:29 | |
| *** ralonsoh has joined #openstack-glance | 07:30 | |
| *** rcernin_ has quit IRC | 07:34 | |
| *** rcernin_ has joined #openstack-glance | 08:16 | |
| *** priteau has joined #openstack-glance | 08:24 | |
| *** rcernin_ has quit IRC | 08:26 | |
| openstackgerrit | Rico Lin proposed openstack/glance master: [TEST][DNM] Validate ARM64 Unittests https://review.opendev.org/742095 | 08:27 |
|---|---|---|
| nikparasyr | goodmorning. i'm trying to use the Interoperable Image Import to use image conversion and metadata injection. We are on stein. My configs look like: https://pastebin.com/GKYkgAX3 . after reconfiguring glance and restarting i try to import an image using: glance image-create-via-import --disk-format qcow2 --container-format bare --name conversion_test --import-method web-download --uri https://download.cirros-cloud.net/0.3.5/cir | 08:34 |
| nikparasyr | ros-0.3.5-x86_64-disk.img but the image is not converted and properties are not added. I can check the task via: glance task-list and glance task-show and its finished in success. The logs also do not show any error. I can see in logs the taskflow_executor transitions but none of these transitions is related to image conversion or metadata injection | 08:34 |
| nikparasyr | any idea of what i might be missing? 2 notes: we have ceph backend and also glance runs on a 3-node setup, on the documentation it states that for multinode glance the fs should be shared. this is not the case here. I would still expect to see some errors on the logs etc if this was the reason that the operation doesnt work | 08:35 |
| *** rcernin_ has joined #openstack-glance | 08:46 | |
| *** baojg has quit IRC | 08:56 | |
| *** baojg has joined #openstack-glance | 08:57 | |
| *** rcernin_ has quit IRC | 09:07 | |
| *** k_mouza has joined #openstack-glance | 09:21 | |
| *** rcernin_ has joined #openstack-glance | 09:41 | |
| openstackgerrit | Abhishek Kekane proposed openstack/glance master: Fix broken glance-cache-manage utility https://review.opendev.org/742115 | 09:42 |
| openstackgerrit | Abhishek Kekane proposed openstack/glance master: Fix broken glance-cache-manage utility https://review.opendev.org/742115 | 09:44 |
| *** baojg has quit IRC | 09:52 | |
| *** baojg has joined #openstack-glance | 09:53 | |
| *** k_mouza has quit IRC | 09:53 | |
| *** rcernin_ has quit IRC | 09:55 | |
| *** k_mouza has joined #openstack-glance | 10:01 | |
| *** ratailor_ has quit IRC | 10:10 | |
| *** ratailor_ has joined #openstack-glance | 10:10 | |
| *** Liang__ has quit IRC | 10:26 | |
| *** rcernin_ has joined #openstack-glance | 10:28 | |
| *** udesale_ has joined #openstack-glance | 10:56 | |
| *** udesale has quit IRC | 10:58 | |
| *** baojg has quit IRC | 11:13 | |
| *** baojg has joined #openstack-glance | 11:14 | |
| *** ratailor_ has quit IRC | 11:41 | |
| *** baojg has quit IRC | 11:54 | |
| *** baojg has joined #openstack-glance | 11:54 | |
| *** tkajinam has quit IRC | 11:56 | |
| *** amoralej is now known as amoralej|lunch | 12:09 | |
| nikparasyr | Hello. any idea about this? | 12:12 |
| nikparasyr | i'm trying to use the Interoperable Image Import to use image conversion and metadata injection. We are on stein. My configs look like: https://pastebin.com/GKYkgAX3 . after reconfiguring glance and restarting i try to import an image using: glance image-create-via-import --disk-format qcow2 --container-format bare --name conversion_test --import-method web-download --uri https://download.cirros-cloud.net/0.3.5/cirros-0.3.5-x86 | 12:12 |
| nikparasyr | _64-disk.img but the image is not converted and properties are not added. I can check the task via: glance task-list and glance task-show and its finished in success. The logs also do not show any error. I can see in logs the taskflow_executor transitions but none of these transitions is related to image conversion or metadata injection | 12:12 |
| nikparasyr | any idea of what i might be missing? 2 notes: we have ceph backend and also glance runs on a 3-node setup, on the documentation it states that for multinode glance the fs should be shared. this is not the case here. I would still expect to see some errors on the logs etc if this was the reason that the operation doesnt work | 12:12 |
| *** k_mouza has quit IRC | 12:18 | |
| *** baojg has quit IRC | 12:44 | |
| *** baojg has joined #openstack-glance | 12:45 | |
| *** rcernin_ has quit IRC | 12:54 | |
| *** amoralej|lunch is now known as amoralej | 13:21 | |
| openstackgerrit | Dan Smith proposed openstack/glance master: WIP: Make glance-api able to do async tasks in WSGI mode https://review.opendev.org/742065 | 13:34 |
| dansmith | abhishekk: ^ | 13:34 |
| dansmith | abhishekk: it seems like we've all been going on this assumption that it was impossible to port glance's async stuff to something that will work under uwsgi so we haven't even been challenging it, | 13:35 |
| dansmith | but unless I'm missing something it's pretty trivial | 13:35 |
| dansmith | abhishekk: that patch is just what I was able to hack up in front of the TV last night, so needs tests and more testing, but it seems to work fine for me | 13:35 |
| dansmith | abhishekk: I assume someone on the glance team ruled this out years ago as a valid option, so if that person is still around maybe we could get them to review to make sure I'm not missing something? | 13:38 |
| *** baojg has quit IRC | 13:46 | |
| *** baojg has joined #openstack-glance | 13:47 | |
| abhishekk | dansmith, apart from rosmaita, jokke and me no one is around from glance for last couple of years | 13:50 |
| dansmith | abhishekk: alright, well, if you could give that a little skim at least, I can start writing tests for it | 13:51 |
| abhishekk | dansmith, I will have a look in some time, in meeting atm | 13:52 |
| dansmith | abhishekk: I think that unless there's some reason not to, we should land this and flip the default back to wsgi mode in devstack | 13:52 |
| dansmith | abhishekk: ack, thanks | 13:52 |
| abhishekk | dansmith, just to FYI apart from this issue glance reloading config files is also not working for uwsgi | 13:53 |
| abhishekk | but I am not sure anyone is using that feature as me also not tested the same for last 3-4 cycles | 13:53 |
| dansmith | abhishekk: I'm sure that's not a feature someone would choose standalone eventlet over wsgi for :) | 13:53 |
| abhishekk | yeah | 13:54 |
| jokke | dansmith: we looked it at Pike when the wsgi_app.py was introduced and there was hard requirement to eventlet. Like said yesterday, there has been no interest or resources poured into that since | 13:54 |
| dansmith | jokke: what was the hard requirement for eventlet though? where did it come from? | 13:54 |
| *** m75abrams has quit IRC | 13:56 | |
| jokke | IIRC it was eventlet green pool for the workers | 13:56 |
| dansmith | jokke: for the tasks? | 13:56 |
| jokke | yeah | 13:57 |
| dansmith | jokke: so last night my first stab at this was literally a two-line change to make that work with native threads | 13:57 |
| openstackgerrit | Merged openstack/glance stable/train: Fix admin docs deploying under HTTPD https://review.opendev.org/742028 | 13:57 |
| dansmith | and it seemed to work fine to me | 13:57 |
| *** m75abrams has joined #openstack-glance | 13:57 | |
| jokke | dansmith: ok, sounds promising ... definitely wasn't that easy swap at the time. There was another worker engine available but there was something horribly wrong with that approach. Now mind you in Pike it was all py27 which handled threading etc. very differently | 14:00 |
| dansmith | jokke: okay, I don | 14:00 |
| dansmith | jokke: I don't think that's true as this is all pretty basic stuff, so I'm concerned that I'm missing something | 14:01 |
| dansmith | AFAICT the taskflow stuff doesn't dictate anything about threads, it just needs a threadpool model, which futurist provides for py2 | 14:01 |
| abhishekk | As far as I remember matt (who added uwsgi support in glance) has tried to give it a try couple of times and then just left it hanging | 14:02 |
| dansmith | jokke: just to be clear, were you the one the most up on this or was there someone else who is now gone? I remember mtreinish asking me about this in dublin but I never looked at any code and I thought he was just trying to evaluate from the outside | 14:02 |
| jokke | Is that threading approach you're looking at actually able to spawn the threads out of that uwsgi container or are all of those still stuck in that sync processing line of uwsgi? | 14:03 |
| dansmith | jokke: I dunno what "threads out of the container" means, but yes, I'm spawning native threads, which are in the uwsgi python process | 14:03 |
| jokke | It was something related to how the uwsgi container handled threading with GIL that pushed us out of that approach | 14:03 |
| abhishekk | and as we found that most of the stuff like reloading, graceful shutdown, chunked upload is not working so we didn't put efforts in that direction as we were busy in stabilizing import workflow and designing multiple stores support | 14:03 |
| dansmith | jokke: all you have to do is tell uwsgi to initialize the GIL (i.e. enable_threading=True) which devstack does by default | 14:04 |
| dansmith | abhishekk: yeah I just think everyone was told that it was massively difficult or impossible for some reason, and it seems to be very much not hard :) | 14:04 |
| abhishekk | after all it looks like what you are saying :D | 14:05 |
| jokke | dansmith: what we might have missed at the time could have been thread pooling from futures | 14:05 |
| dansmith | jokke: maybe, but that's a pretty trivial thing to implement too | 14:06 |
| dansmith | I mean, once you understand where the gaps are of course | 14:06 |
| jokke | yeah ... we had few people looking options at the time and all of us came to same conclusion that the threaded workers would not have worked within the uwsgi container and with all the other issues popping up we gave up at that point | 14:08 |
| jokke | but like said it's been couple of years and no-one been invested enough to work it out | 14:08 |
| dansmith | I think that there's clearly a breakdown in communication here, as this has left a bunch of other things in the community hanging, like devstack cleaning up a bunch of stuff, and of course, your lack of testability of all the import stuff. I think everyone believed that you guys had ruled this out as physically not possible, | 14:09 |
| dansmith | so it's a bit unfortunate | 14:09 |
| dansmith | I know our own product even deploys glance differently than, say, nova because of the eventlet requirement | 14:09 |
| jokke | Well we've been ever shrinking team and we have working deployment model so it was clear prioritization for us | 14:10 |
| dansmith | so, one thing we probably want to figure out is how to specify these thread pool sizes, | 14:10 |
| dansmith | in eventlet you can have a pool of a billion threads and it doesn't really change much because it's all one "thread" of execution | 14:11 |
| dansmith | but you don't want people to, at least by default, spin up a billion native threads | 14:11 |
| dansmith | I defaulted the size differently for those two models, | 14:11 |
| jokke | yeah, I think we have 1000 greenthreads in the eventlet pool | 14:11 |
| dansmith | but we probably want to make that a config option | 14:11 |
| dansmith | 1024 yeah | 14:11 |
| jokke | and that's for each worker | 14:11 |
| dansmith | right and so is this | 14:11 |
| dansmith | also, | 14:11 |
| dansmith | we hand taskflow a threadpool for each task, | 14:12 |
| dansmith | which has a separate config | 14:12 |
| dansmith | (which should be separate) | 14:12 |
| jokke | well honestly we don't even need to do that as we use only linear flows | 14:13 |
| dansmith | anyway, if we don't make it configurable, then someone with 128 cores will get 128*16 threads right now, which may be too much | 14:13 |
| dansmith | jokke: yeah, but it doesn't matter, it's fine | 14:13 |
| dansmith | it's the other pool that matters | 14:13 |
| jokke | hmm-m we could 'though probably increase the multi-store import performance quite a bit if we handed those tasks to a threadpool to run parallel | 14:13 |
| jokke | well we don't default workers to core count anymore | 14:14 |
| dansmith | maybe, or maybe you'd just kill the outbound network such that they all take a long time | 14:14 |
| dansmith | jokke: I know, but my default is 16 and that'll be 16 times the number of wsgi workers they have | 14:14 |
| jokke | Unless you specifically set it zero, we default to 4 or 8 or something reasonable like that | 14:14 |
| jokke | yeah | 14:14 |
| dansmith | wsgi parallelism is where you get your ability to handle lots of small requests in parallel, so that needs to be reasonably wide, | 14:15 |
| dansmith | you just don't want to inflate each of those to a bunch of threads waiting to do stuff over time | 14:15 |
| dansmith | anyway, we don't have to solve it now, but I think it should be tunable | 14:15 |
| jokke | And just personally after the initial poke I haven't had too much interest to dig into it either. Can't understand why anyone would want to have Apache overhead on their glance processes even without tasks | 14:18 |
| dansmith | as far as I can tell, most everyone wants that :) | 14:19 |
| jokke | dansmith: yeah, but no offence to anyone most of people who wants that don't know why but the reason is "because someone said APIs would be better ran under 'proper' web server" | 14:21 |
| jokke | it's not like there would be actual benefit of it | 14:21 |
| dansmith | I think there's plenty of actual benefit, but I understand your point | 14:22 |
| jokke | but it's not like Apache would be the worst offender in bloatware world either | 14:22 |
| dansmith | nor apache being the only one to run the wsgi app inside | 14:23 |
| jokke | that | 14:23 |
| abhishekk | dansmith looks like some problem in devstack if I set GLANCE_STANDALONE=False | 14:23 |
| jokke | but yet, if you ask what people actually run them under it's either Apache or Nginx | 14:23 |
| dansmith | abhishekk: yeah, you have to enable tls proxy I think | 14:23 |
| abhishekk | it adds something like below in glance-api.conf | 14:24 |
| abhishekk | [enabled_import_methods] | 14:24 |
| abhishekk | []= | 14:24 |
| dansmith | abhishekk: right | 14:24 |
| abhishekk | it should be added under default section like | 14:24 |
| dansmith | abhishekk: because in standalone mode right now, you can't have any import methods enabled or it will expose things that the tempest test will try to use and fail | 14:25 |
| abhishekk | enabled_import_methods = [] | 14:25 |
| dansmith | abhishekk: you need this: https://review.opendev.org/#/c/742047/ | 14:25 |
| abhishekk | ahh | 14:26 |
| abhishekk | ack | 14:26 |
| dansmith | abhishekk: because I suck.. you should fire me :) | 14:26 |
| abhishekk | no worries :D | 14:26 |
| dansmith | but I think you still have to enable tls-proxy or glance starts on the wrong port | 14:26 |
| abhishekk | ack | 14:27 |
| dansmith | I still don't have all of that right | 14:27 |
| dansmith | it would be much easier if everything was just wsgi :) | 14:27 |
| jokke | dansmith: also what lots of people don't think of is that apart from tight cpeh integration between Glance, Cinder and Nova majority of Glance processing time goes streaming data, not for the small quick API calls | 14:27 |
| dansmith | jokke: yep, which is probably why a full worker delegation model makes more sense for offloading those things | 14:32 |
| dansmith | the thing I really want, is for glance to be deployable in a sane way in CI so that tests can be written, which isn't the case right now | 14:33 |
| dansmith | which is scary because with this threading thing, I don't have a big batch of harsh import tests I can run against it to make sure it holds up | 14:33 |
| dansmith | so that's why I did the devstack work to make standalone mode more doable, and why I think ultimately making wsgi mode support import is the right thing if most people are going to deploy that way anyway | 14:33 |
| dansmith | so we can run tests against the deployment mechanism that people are using | 14:34 |
| dansmith | I'll leave the job of convincing the world to use the eventlet mode to you :) | 14:34 |
| *** Luzi has quit IRC | 14:34 | |
| dansmith | I've done my fair share of defending eventlet in nova over the "bah, eventlet isn't shiny and new enough, let's rip it out!" arguments, so I feel your pain | 14:35 |
| jokke | yeah, so like mentioned it hasn't been supported as the service doesn't work properly (the fact that you can start glance-api without it enforcing at least one of the import methods was concious decision that we left undocumented as defcore demand was at least one needs to be discoverable for user) but we wanted to keep the flexibility to not have them for example internal, non-public facing nodes | 14:40 |
| jokke | used purely by Cinder and Nova for exampe | 14:40 |
| jokke | if the service works properly I don't care how people runs it. It's like this container madness that is in power atm. Makes no sense what so ever, but if people who don't know better insists, it's up to them ;P | 14:41 |
| jokke | As long as it's something we know it should work, I'm fine with that | 14:42 |
| jokke | That said I'm also more than happy to leave any apache/<inser your webserver here> or docker issues being the responsibility of that deployer as long as the issue is not in our code :P | 14:43 |
| *** k_mouza has joined #openstack-glance | 14:45 | |
| *** Luzi has joined #openstack-glance | 14:48 | |
| *** Luzi has quit IRC | 14:50 | |
| *** rcernin_ has joined #openstack-glance | 14:52 | |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make our tempest job use import, standalone, and inject_metadata https://review.opendev.org/741276 | 14:56 |
| dansmith | abhishekk: ^ | 14:56 |
| *** m75abrams has quit IRC | 14:57 | |
| *** rcernin_ has quit IRC | 14:57 | |
| abhishekk | dansmith, thank you, something went wrong | 14:59 |
| dansmith | ah crap | 14:59 |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make our tempest job use import, standalone, and inject_metadata https://review.opendev.org/741276 | 14:59 |
| dansmith | there | 14:59 |
| abhishekk | cool | 15:00 |
| dansmith | er | 15:00 |
| dansmith | jesus | 15:01 |
| dansmith | gate too | 15:01 |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make our tempest job use import, standalone, and inject_metadata https://review.opendev.org/741276 | 15:01 |
| dansmith | cripes | 15:01 |
| abhishekk | :D | 15:01 |
| dansmith | _actually_ running now :) | 15:01 |
| abhishekk | yeah :D | 15:02 |
| dansmith | so I added a job to the wsgi patch to make it run in wsgi mode too, and hopefully enable import methods | 15:03 |
| dansmith | but I think we need to end up with a wsgi and non-wsgi job, both using import | 15:03 |
| dansmith | so maybe whatever is not the default in devstack can run in this other job defined here, with metadata injection enabled | 15:04 |
| abhishekk | yes | 15:04 |
| abhishekk | also I applied your async uwsgi patch in my fresh env and its working, need to perform more tests though | 15:07 |
| abhishekk | dansmith, ^^ | 15:07 |
| dansmith | abhishekk: excellent | 15:08 |
| abhishekk | will have a look again after dinner | 15:09 |
| dansmith | abhishekk: what would be great is if we could build up some tests in _tempest_ to validate all the things you're going to try by hand | 15:09 |
| dansmith | so we can repeatably run those against both solutions | 15:10 |
| abhishekk | dansmith, ack | 15:10 |
| abhishekk | I am not sure how we can test web-download in tempest though | 15:10 |
| knikolla | jokke: are you ok with reverting the wsgi doc change for now while we talk and investigate this more? | 15:11 |
| * abhishekk dinner break, back in 45 mins | 15:11 | |
| dansmith | abhishekk: add a conf knob to tempest with a URL, and if it's set, use that | 15:12 |
| dansmith | abhishekk: I'm sure gmann can help us figure out the best way | 15:12 |
| abhishekk | dansmith, ack, will look into it once back | 15:13 |
| *** baojg has quit IRC | 15:20 | |
| *** baojg has joined #openstack-glance | 15:21 | |
| gmann | abhishekk: dansmith happy to help, web-download can wotrk with config as you mentioned and discover-able from info/import API. | 15:25 |
| dansmith | gmann: sweet thanks | 15:26 |
| jokke | knikolla: we can do it if we get there | 15:40 |
| *** gyee has joined #openstack-glance | 15:41 | |
| knikolla | jokke: but i don't see it as productive to advertise that we're not going to support it if we're working to support it. | 15:43 |
| dansmith | it also concerns me that it seems to retroactively deprecate existing deployments | 15:44 |
| knikolla | docs are targeted to people who otherwise may not be aware of the internal development within a team. | 15:44 |
| jokke | dansmith: no, what happened like I said we never did, but the wording of the warning and the dock was conflicting and confusing giving impression that after queens we would. Thus after actually reading the body it was pretty obvious that it needed to be corrected | 15:46 |
| jokke | so if we get there within this cycle, I'm more than happy to revert it from Victoria, if we get there in W we change it there. But I don't want to give the impression that we do indeed support it before we are in a state we can | 15:47 |
| dansmith | I totally disagree that an operator who read that original doc as a blessing for wsgi mode will see your change as anything other than a retroactive deprecation of what they were told :) | 15:47 |
| jokke | dansmith: you just proved my point ;) | 15:48 |
| knikolla | jokke: not supporting something entirely is different from discouraging it with caveats. | 15:48 |
| jokke | the operator who wanted to read it as blessing of supported configuration, got wrong message which reinforces the reason to fix it | 15:49 |
| dansmith | jokke: so by that logic, would you approve backporting the deletion of the file? Because having it in tree conveys support, but it's not supported, so we should remove it and if they want to keep running that way, they can just find it in git and use it? | 15:53 |
| dansmith | I'm not suggesting that of course, I'm just saying I think the doc change is equally as nuclear, IMHO | 15:53 |
| knikolla | jokke, abhishekk: we're planning on organizing a meeting with the tc and the glance team. would the tc office hours this thursday work? http://eavesdrop.openstack.org/#Technical_Committee_Office_hours | 15:53 |
| jokke | that's right after the glance weekly, fine by me | 15:54 |
| jokke | dansmith: the file being there has no relation to being supported or not. It was experimental at the time we got stuck with due to devstack. There is no reason to do it and knowing people using it, no I would not allow backporting it's removal for many reasons. The fact that we just corrected the docs that clearly gave confusing message of experiment that never worked on supportable level needed | 15:58 |
| jokke | to be fixed | 15:58 |
| *** udesale_ has quit IRC | 15:59 | |
| dansmith | jokke: the file doesn't log an experimental warning that I can see, nor do the docs, so I think everyone would pretty much assume it's usable | 16:00 |
| dansmith | so it seems the same to me | 16:00 |
| dansmith | especially when things like OSA deploy in wsgi mode, AFAIK | 16:01 |
| dansmith | anyway, it's your call, not mine, I'm just opining that I don't think that's a reasonable thing to backport.. that's all :) | 16:02 |
| jokke | dansmith: That was their very own decision. When they were going that way, we told them very clearly that it does not have community support and the response was "we don't care, we're doing it anyways" | 16:02 |
| jokke | so in this discussion I'm not taking OSA as contributor for reasoning either way ;) | 16:03 |
| dansmith | that's cool, I'm just saying I think a lot of operators won't see it that way. I understand that it being against advice of the glance team makes it murky. | 16:04 |
| *** k_mouza has quit IRC | 16:04 | |
| *** k_mouza has joined #openstack-glance | 16:05 | |
| jokke | dansmith: this has been fight with no winners from the very beginning. And I do agree, we as community can blame who ever we want and waste as much time as we want but the real damage is to the ops who unfortunately took the ill advice | 16:07 |
| dansmith | just curious, how many people complain about things not working under wsgi? I think that it's entirely likely that a lot of them don't see the downside, if they're not leveraging import for example | 16:07 |
| knikolla | we're all winners if we improve the process thanks to what we learn from this with regard to tc-teams communication | 16:08 |
| dansmith | knikolla: if you say "we're all winners" one more time I'm going to throw up in my mouth :P | 16:08 |
| knikolla | haha, noted | 16:09 |
| knikolla | however that is the outlook i usually have with regards to seeing what can be improved | 16:09 |
| dansmith | :) | 16:09 |
| knikolla | i'm like: oooo, we found a flaw, yay | 16:09 |
| dansmith | abhishekk: I also finally got devstack redeployed with glance in eventlet mode and my changes don't seem to have broken anything in that model either | 16:12 |
| abhishekk | sounds promising | 16:12 |
| dansmith | abhishekk: I'm kinda surprised that none of those changes caused any unit tests to fail, but I'm going to start working on some for this | 16:14 |
| abhishekk | dansmith, looks like most of the part related to task_executor is mocked | 16:15 |
| jokke | dansmith: to us not that many. TBH majority of the communication people having issues have been osc anyways and fixed by telling to use glanceclient instead. On the same note all the deployments I've dealt with are running glance with eventlet. I know mnaser runs it in uwsgi and just tells his customers that most of the new features past 2-3 years are not available so that filters lots of | 16:16 |
| dansmith | abhishekk: mocked but not validated, since I've changed a lot of what gets called and returned | 16:16 |
| jokke | potential out. Their benefit is also that osc has not had feature parity ever so if you push your users to use osc, you won't know that the cloud is broken either | 16:16 |
| dansmith | jokke: well, those features aren't tested, so I'd be hesitant myself, but... yep, fair enough :) | 16:16 |
| *** k_mouza has quit IRC | 16:20 | |
| jokke | dansmith: in tempest | 16:20 |
| dansmith | jokke: are they tested upstream anywhere else? | 16:20 |
| jokke | we have some unit and functional test coverage (our functional tests actually starts the api for the calls so they execute against actual current service) | 16:21 |
| dansmith | given the number of things I found in the first hour of trying to use it, along with the number of things I can change under the hood without breaking tests, that doesn't give me a lot of confidence | 16:22 |
| dansmith | but that said, | 16:22 |
| dansmith | my focus really is improving from here and that's all | 16:22 |
| jokke | that's what broke our gate over the weekend ... the service didn't start in our functional pipes | 16:22 |
| jokke | dansmith: we're very aware | 16:22 |
| dansmith | I definitely don't want to keep arguing about this and damage any realationship that's going to prevent me from helping move it forward, so let's just roll okay? | 16:23 |
| jokke | dansmith: why do you think we've been trying to fight to get the devstack deploy the service correctly for past almost 3 years | 16:23 |
| jokke | ;) | 16:23 |
| dansmith | not taking that bait :) | 16:23 |
| jokke | :P | 16:26 |
| *** priteau has quit IRC | 16:28 | |
| *** k_mouza has joined #openstack-glance | 16:28 | |
| openstackgerrit | Merged openstack/glance master: Remove deprecated glance-replicator options https://review.opendev.org/741321 | 16:29 |
| *** ralonsoh_ has joined #openstack-glance | 16:41 | |
| *** ralonsoh has quit IRC | 16:43 | |
| *** k_mouza has quit IRC | 16:45 | |
| *** priteau has joined #openstack-glance | 17:00 | |
| *** amoralej is now known as amoralej|off | 17:05 | |
| abhishekk | knikolla, hey | 17:05 |
| abhishekk | sorry, just saw your message about Thursday | 17:06 |
| abhishekk | I am available between 1500 to 1530 hours on Thursday | 17:06 |
| knikolla | abhishekk: cool, i think that's when i proposed the meeting for | 17:07 |
| abhishekk | if it is for 30 minutes then great, I will be there | 17:08 |
| *** nikparasyr has quit IRC | 17:09 | |
| abhishekk | dansmith, gmann I am not familiar with conf knob, but looks like setting url in tempest.conf and use it in test? | 17:12 |
| gmann | abhishekk: yeah, I will write test on top of https://review.opendev.org/#/c/741425/ | 17:12 |
| abhishekk | cool | 17:13 |
| *** ajitha has joined #openstack-glance | 17:16 | |
| abhishekk | dansmith, conversion plugin is failing with uwsgi | 17:16 |
| ajitha | Hi all, We are running a third party CI. We are using py3.6 and Ubuntu18.04.. From yesterday, devstack is getting failed. Please find the logs. https://pastebin.com/zXZQm9D1 | 17:16 |
| abhishekk | dansmith, http://paste.openstack.org/show/796182/ | 17:17 |
| ajitha | Error is g-api did not start | 17:17 |
| dansmith | abhishekk: I dunno what that is specifically, but obviously this is the problem: unable to load configuration from oslo_concurrency.prlimit | 17:18 |
| jokke | ajitha: gmann and dansmith are on it iiuc. I'm sure they have it all fixed in no time | 17:18 |
| dansmith | I'm guessing because it's overriding subprocess and re-spawning with uwsgi, but that python module isn't loadable? | 17:18 |
| abhishekk | dansmith, need to look into it | 17:19 |
| dansmith | ajitha: pastebin your devstack local.conf or localrc | 17:19 |
| dansmith | abhishekk: can you import that module in a python shell? | 17:19 |
| abhishekk | let me check | 17:20 |
| *** priteau has quit IRC | 17:21 | |
| abhishekk | yes i can | 17:21 |
| dansmith | abhishekk: okay I think that's what uwsgi is trying to do with the -m ... | 17:23 |
| abhishekk | likely | 17:25 |
| dansmith | oh actually maybe not | 17:25 |
| dansmith | gmann: do you know anything about that? | 17:25 |
| dansmith | I'm guessing nothing else needs to spawn subprocesses from their API, so maybe this hasn't been seen before | 17:26 |
| ajitha | dansmith: sure. https://pastebin.com/Ga8j3ye1 | 17:26 |
| ajitha | for some reason we are using both localrc and local.conf.. | 17:26 |
| dansmith | ajitha: try setting GLANCE_STANDALONE=False first to see if that gets you going | 17:27 |
| ajitha | sure | 17:28 |
| dansmith | ajitha: I think that it's because you're not running tls-proxy and I still haven't fixed the hole where tls-proxy is not enabled, but glance is standalone | 17:28 |
| ajitha | it was running till Friday | 17:28 |
| gmann | dansmith: ajitha for GLANCE_STANDALONE=False - this is still note merged https://review.opendev.org/#/c/742047/1 | 17:28 |
| gmann | not merged | 17:28 |
| dansmith | oh yeah that too | 17:29 |
| dansmith | ajitha: yeah I've been breaking things :/ | 17:29 |
| gmann | with default i think it is working fine in upstream, trying to see in ajitha logs whats wrong | 17:29 |
| dansmith | abhishekk: so I actually think that might be a bug in oslo.processutils | 17:30 |
| ajitha | dansmith: ok | 17:30 |
| ajitha | gmann: Thanks | 17:30 |
| dansmith | abhishekk: I bet it's trying to exec the python interpreter, basically sys.argv[0], which is uwsgi for us | 17:30 |
| ajitha | dansmith: GLANCE_STANDALONE=False. will this work if its not merged | 17:31 |
| dansmith | ajitha: no, you need the patch gmann mentioned also | 17:31 |
| dansmith | ajitha: alternatively, you can "enable_service tls-proxy" | 17:31 |
| ajitha | ok will try that | 17:32 |
| abhishekk | dansmith, I ran into another problem in another env (not related to glance, trying to analyze that as it is important for me) | 17:33 |
| *** ralonsoh_ has quit IRC | 17:33 | |
| dansmith | abhishekk: ack, poking in -oslo right now | 17:35 |
| gmann | ajitha: dansmith rechecked on no-tls proxy job also- https://review.opendev.org/#/c/741807/2 | 17:35 |
| abhishekk | dansmith, ack | 17:35 |
| *** alistarle has joined #openstack-glance | 17:39 | |
| dansmith | abhishekk: in image_conversion.py, try nuking the prlimit= param to exec | 17:39 |
| *** ralonsoh has joined #openstack-glance | 17:41 | |
| *** alistarle has quit IRC | 17:44 | |
| abhishekk | dansmith, ack | 17:46 |
| *** alistarle has joined #openstack-glance | 17:46 | |
| dansmith | abhishekk: if that works, we'll have to either not use prlimit, or pass the interpreter name to that exec call when we're in uwsgi, | 17:46 |
| dansmith | but that shouldn't be hard | 17:46 |
| abhishekk | dansmith, taking some time | 17:54 |
| abhishekk | still working on other issue | 17:54 |
| abhishekk | current prlimit is something like this | 17:55 |
| abhishekk | prlimit=utils.QEMU_IMG_PROC_LIMITS | 17:56 |
| abhishekk | QEMU_IMG_PROC_LIMITS = putils.ProcessLimits(cpu_time=2, | 17:56 |
| abhishekk | address_space=1 * units.Gi) | 17:56 |
| abhishekk | this is current prlimit | 17:56 |
| dansmith | abhishekk: yeah, just remove prlimit= entirely as a test | 17:57 |
| abhishekk | conversion worked but some thing went wrong at the end | 18:00 |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make glance-api able to do async tasks in WSGI mode https://review.opendev.org/742065 | 18:01 |
| dansmith | abhishekk: with tests, fixes for functional ^ | 18:01 |
| dansmith | abhishekk: presumably this is no more broken today where nothing even starts, so we can iterate on things like this processutils thing, right? | 18:01 |
| abhishekk | dansmith, ack | 18:02 |
| dansmith | abhishekk: I'm more than happy to refuse to start wsgi_app.py with image_conversion enabled until we do, of course :) | 18:02 |
| abhishekk | dansmith, I am using swift store and got this error while uploading file around 1gb | 18:02 |
| abhishekk | https://etherpad.opendev.org/p/glance-error | 18:05 |
| abhishekk | need to check without conversion | 18:05 |
| abhishekk | working well without conversion | 18:07 |
| abhishekk | means either swift has problem while uploading raw image or something is wrong with uwsgi | 18:07 |
| *** ralonsoh has quit IRC | 18:08 | |
| dansmith | abhishekk: gosh that sure looks like a swift thing, but I don't know miuch | 18:10 |
| dansmith | abhishekk: does glance_store have eventlet code in it? | 18:10 |
| dansmith | I don't really even know what glance_store is | 18:10 |
| dansmith | oh you're exec'ing something | 18:11 |
| dansmith | so maybe that's just failing and the actual problem is hidden by this re-raise? | 18:11 |
| timburke | abhishekk, got swift proxy logs? it's not clear to me what the error is that led to it wanting to retry (which is probably a problem i ought to fix in swiftclient...) | 18:11 |
| abhishekk | timburke, not at the moment | 18:12 |
| abhishekk | I am trying to many things at a time | 18:12 |
| timburke | oh! it's probably the `Jul 21 17:58:55 konan-swift devstack@g-api.service[17934]: INFO swiftclient [-] RESP STATUS: 503 Service Unavailable` | 18:13 |
| timburke | if you've still got access to those logs, i'd go grepping for tx3f4007f67a7d469c9a466-005f172cde (when you get a chance, of course) | 18:14 |
| *** alistarle has quit IRC | 18:14 | |
| abhishekk | timburke, http://paste.openstack.org/show/796188/ | 18:18 |
| abhishekk | this is what I can see in s-proxy | 18:18 |
| dansmith | token timeout? | 18:18 |
| abhishekk | i guess so | 18:18 |
| dansmith | or maybe the broken pipe | 18:19 |
| dansmith | not sure if that's from the glance side or the swift side | 18:19 |
| * abhishekk kind off lost track what exactly I was doing initially :P | 18:20 | |
| timburke | no, not a token timeout -- proxy <-> backend connections are unauthenticated. object-server died? | 18:20 |
| dansmith | abhishekk: this is why you should be writing a tempest test so that we can re-try it :D | 18:21 |
| abhishekk | :D | 18:21 |
| timburke | whatcha got in s-object? | 18:22 |
| abhishekk | let me check | 18:24 |
| dansmith | timburke: in case the context isn't clear, we're trying to get glance to run under uwsgi with native threads instead of eventlet, so if there's anything about this that smells like uwsgi killing glance in the middle of a long transfer.... just keep that in mind | 18:26 |
| abhishekk | Jul 21 18:26:51 konan-swift object-server[9862]: ERROR __call__ error with PUT /sdb1/356/AUTH_457f85b795c14d5ebc851758667b1544/glance/9982064d-8cd0-4156-a201-6d91fe3145e5-00020 : [Errno 28] No space left on device (txn: txbfafdb133f9349288e88a-005f17336a) | 18:27 |
| abhishekk | Jul 21 18:26:51 konan-swift swif | 18:27 |
| dansmith | lol | 18:27 |
| dansmith | nevermind :D | 18:27 |
| abhishekk | dansmith, but I am able to upload qcow2 image | 18:29 |
| dansmith | abhishekk: right but with conversion it probably becomes much larger right? | 18:29 |
| dansmith | qemu-img info foo.img | 18:29 |
| dansmith | check "virtual size" | 18:30 |
| dansmith | that's how big it will be when converted to raw | 18:30 |
| abhishekk | ack | 18:30 |
| dansmith | so for example, the cirros image is 12M on disk, but expands to 44M as raw | 18:31 |
| timburke | object-server still should've *responded* :-( time for me to redo my dev env with some tiny disks... | 18:31 |
| abhishekk | 8 GB | 18:31 |
| abhishekk | :P | 18:31 |
| dansmith | abhishekk: from a 1GB qcow right? | 18:32 |
| abhishekk | yes | 18:32 |
| dansmith | what is the size of the backing store for swift | 18:32 |
| dansmith | ? | 18:32 |
| abhishekk | 6 GB | 18:32 |
| dansmith | 8GB won't fit into 6GB :P | 18:32 |
| abhishekk | makes sense | 18:32 |
| dansmith | note that glance also needs space for the original plus the new one, so 9GB minimum | 18:33 |
| abhishekk | I have total 20 gb of free | 18:33 |
| dansmith | okay | 18:33 |
| abhishekk | will continue tomorrow | 18:34 |
| dansmith | abhishekk: okay so aside from that and the prlimit thing we need to figure out, sounds promising right? | 18:34 |
| abhishekk | yes, I still need to test copy-image and multiple stores | 18:34 |
| abhishekk | but likely it looks more promising | 18:34 |
| dansmith | I've tested copy-image with multiple stores, but yeah, by all means | 18:34 |
| dansmith | and if we get the tempest test and job pointed at this change we'll get that in CI too | 18:34 |
| abhishekk | I need to test some combinations :D | 18:35 |
| dansmith | s/test/write some tests for/ | 18:35 |
| abhishekk | yeah | 18:35 |
| dansmith | anyway, catch you tomorrow :) | 18:35 |
| abhishekk | As you have tested copy means I assume you are using rbd? | 18:36 |
| dansmith | yeah | 18:36 |
| abhishekk | cool | 18:36 |
| dansmith | I tried to get copy working between two file repos briefly, but it wasn't actually writing the image | 18:36 |
| abhishekk | which means I need to run some tests against cinder store as well | 18:36 |
| dansmith | yeah | 18:36 |
| dansmith | s/run/write/ :) | 18:36 |
| *** ajitha has quit IRC | 18:37 | |
| abhishekk | :D | 18:37 |
| dansmith | it sounds like we need to come up with several new jobs, so that these things are being tested against those services in CI right? | 18:37 |
| dansmith | let's make that list so we can work on that once this all settles | 18:37 |
| abhishekk | +1 | 18:37 |
| * abhishekk signing out for the day | 18:41 | |
| dansmith | o/ | 18:43 |
| abhishekk | o/~ | 18:43 |
| *** ajitha has joined #openstack-glance | 18:54 | |
| *** ajitha has quit IRC | 19:26 | |
| *** ajitha has joined #openstack-glance | 19:46 | |
| ajitha | dansmith: http://paste.openstack.org/show/796189/ | 19:52 |
| ajitha | and in glance-api.conf [enabled_import_methods] | 19:59 |
| ajitha | gmann | 19:59 |
| gmann | ajitha: we need to wait for https://review.opendev.org/#/c/742047/ | 20:00 |
| ajitha | ok. Above log is when i enable GLANCE_STANDALONE=False. dansmith gmann | 20:02 |
| gmann | yes | 20:02 |
| *** ajitha has quit IRC | 20:07 | |
| dansmith | gmann: what a mess I made :) | 20:20 |
| dansmith | zuul restart didn't help | 20:20 |
| gmann | it is taking much more time now. | 20:21 |
| gmann | dansmith: 1 comment on import job i think you want to run single job in gate right? - https://review.opendev.org/#/c/741276/9/.zuul.yaml@260 | 22:04 |
| dansmith | gmann: I know it doesn't replace it.. I think what we will do is make one of those jobs be default and the other will be the opposite (either wsgi or standalone) with the injection and post-check | 22:05 |
| dansmith | gmann: they also need jobs to test things like cinder store and swift store, etc | 22:06 |
| dansmith | so I figure we'll sprinkle some of those things between them | 22:06 |
| gmann | ok,that make sense but you removed the irrelevant-files things for original job | 22:10 |
| dansmith | gmann: I was just doing that to see if you would notice | 22:10 |
| gmann | :) | 22:13 |
| dansmith | gmann: you want me to remove the depends-on to the job for the tempest patch why? so we can land the tempest test sooner than the job might? | 22:14 |
| gmann | dansmith: yes, as test run fin with current devstack defaults | 22:15 |
| dansmith | okay | 22:15 |
| dansmith | gmann: okay tempest patch updates | 22:16 |
| dansmith | *updated | 22:16 |
| gmann | for copy-image testing, is there way to know image is in what all store ? or store things is not user knowing things via API ? | 22:16 |
| gmann | thans | 22:16 |
| gmann | thanks | 22:16 |
| dansmith | gmann: yeah, you've seen my nova job right? | 22:16 |
| gmann | yeah | 22:16 |
| dansmith | there's a stores param on the image when multi-store is enabled, | 22:16 |
| dansmith | which has a list of store names | 22:16 |
| dansmith | I think we might be able to poke the api to see a list of stores and use that to decide if we can run a copy-image test | 22:17 |
| gmann | i am thinking how to add tempest tests for that which need to verify the images in store via API | 22:17 |
| dansmith | yeah, should be possible | 22:17 |
| gmann | run we can do /info/strore API will return the available store | 22:17 |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make our tempest job use import, standalone, and inject_metadata https://review.opendev.org/741276 | 22:18 |
| dansmith | gmann: ^ | 22:19 |
| dansmith | if that looks better maybe smcginnis will re-+W it for us | 22:19 |
| gmann | thanks | 22:19 |
| gmann | i did not find if store info in return with images dict in any API | 22:20 |
| dansmith | gmann: https://pastebin.com/M8tAdyt5 | 22:21 |
| dansmith | gmann: L15$, stores=cheap | 22:21 |
| dansmith | btw, I _hate_ how osc crams all the image properties into one line :( | 22:21 |
| *** rcernin_ has joined #openstack-glance | 22:21 | |
| *** rcernin_ has quit IRC | 22:22 | |
| gmann | ohk in properties, i was referring the api-ref and did not find, thanks | 22:22 |
| dansmith | cool | 22:23 |
| dansmith | smcginnis: thanks :) | 22:27 |
| smcginnis | Anytime. | 22:27 |
| *** rcernin has joined #openstack-glance | 22:34 | |
| dansmith | gmann: is it possible to disable a job from the template and replace it with a modification? i.e. what you thought I was trying to do? | 22:54 |
| dansmith | I guess I assumed we shouldn't do that so we're not losing coverage due to some configuration we're changing | 22:55 |
| *** tkajinam has joined #openstack-glance | 22:55 | |
| gmann | dansmith: no afaik. we need to remove template itself to replace the original integration jobs with modified one | 22:57 |
| dansmith | ack, I figured | 22:57 |
| dansmith | probably not good to do that in case we change something that makes us land things that would break other people running the template | 22:58 |
| gmann | yeah | 22:58 |
| dansmith | just thinking about how to avoid adding a bunch of jobs, | 22:58 |
| dansmith | given the gate is so strained | 22:58 |
| gmann | as you mentioned early, we can make new job as multi-store, import-worfklow and all those need to run with glance standalone. | 22:59 |
| dansmith | yeah, maybe we should just do that with my nova ceph multistore job | 22:59 |
| dansmith | turn on EVERYTHING ;) | 22:59 |
| gmann | yeah | 22:59 |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make our import-workflow job also convert images to raw https://review.opendev.org/742309 | 23:06 |
| dansmith | I will squash a bunch of this together once things get landed, but this ^ turns on image conversion also, which we know will not work on wsgi yet because of the subprocess stuff | 23:06 |
| dansmith | but that's the first thing we know of that will actually differ | 23:06 |
| gmann | +1 | 23:13 |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make image conversion use a proper python interpreter for prlimit https://review.opendev.org/742314 | 23:30 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!