| *** zzzeek has quit IRC | 00:01 | |
| *** zzzeek has joined #openstack-glance | 00:02 | |
| *** Underknowledge has quit IRC | 00:23 | |
| *** Underknowledge has joined #openstack-glance | 00:24 | |
| openstackgerrit | Dan Smith proposed openstack/glance master: DNM: Test volume quota increase https://review.opendev.org/c/openstack/glance/+/774174 | 00:26 |
|---|---|---|
| openstackgerrit | Dan Smith proposed openstack/glance master: DNM: Test volume quota increase https://review.opendev.org/c/openstack/glance/+/774174 | 00:27 |
| openstackgerrit | Dan Smith proposed openstack/glance master: DNM: Test volume quota increase https://review.opendev.org/c/openstack/glance/+/774174 | 00:31 |
| *** zzzeek has quit IRC | 01:33 | |
| *** zzzeek has joined #openstack-glance | 01:34 | |
| *** baojg has joined #openstack-glance | 01:42 | |
| *** zzzeek has quit IRC | 01:45 | |
| *** zzzeek has joined #openstack-glance | 01:48 | |
| *** zzzeek has quit IRC | 01:57 | |
| *** zzzeek has joined #openstack-glance | 01:59 | |
| *** dasp has quit IRC | 02:04 | |
| *** dasp has joined #openstack-glance | 02:32 | |
| *** lbragstad has quit IRC | 02:35 | |
| *** lbragstad has joined #openstack-glance | 03:15 | |
| *** rcernin has quit IRC | 03:24 | |
| *** rcernin has joined #openstack-glance | 03:27 | |
| *** Underknowledge has quit IRC | 03:33 | |
| *** Underknowledge2 has joined #openstack-glance | 03:33 | |
| *** Underknowledge2 is now known as Underknowledge | 03:34 | |
| *** baojg has quit IRC | 03:34 | |
| *** baojg has joined #openstack-glance | 03:36 | |
| *** udesale has joined #openstack-glance | 03:46 | |
| *** baojg has quit IRC | 04:07 | |
| *** zzzeek has quit IRC | 04:08 | |
| *** zzzeek has joined #openstack-glance | 04:09 | |
| *** ratailor has joined #openstack-glance | 04:57 | |
| *** baojg has joined #openstack-glance | 05:24 | |
| *** udesale has quit IRC | 06:27 | |
| *** udesale has joined #openstack-glance | 06:56 | |
| *** udesale has quit IRC | 07:08 | |
| *** udesale has joined #openstack-glance | 07:09 | |
| *** lpetrut has joined #openstack-glance | 07:17 | |
| *** rcernin has quit IRC | 07:57 | |
| *** vesper11 has joined #openstack-glance | 07:58 | |
| *** ralonsoh has joined #openstack-glance | 08:02 | |
| *** rcernin has joined #openstack-glance | 08:14 | |
| *** rcernin has quit IRC | 08:19 | |
| *** rcernin has joined #openstack-glance | 08:26 | |
| *** vesper11 has quit IRC | 08:30 | |
| *** udesale has quit IRC | 08:30 | |
| *** rcernin has quit IRC | 08:32 | |
| *** rcernin has joined #openstack-glance | 08:44 | |
| *** udesale has joined #openstack-glance | 08:49 | |
| *** rcernin has quit IRC | 08:50 | |
| *** rcernin has joined #openstack-glance | 09:01 | |
| *** rcernin has quit IRC | 09:06 | |
| *** rcernin has joined #openstack-glance | 09:27 | |
| *** rcernin has quit IRC | 09:43 | |
| *** nikparasyr has joined #openstack-glance | 09:44 | |
| *** rcernin has joined #openstack-glance | 10:19 | |
| *** rcernin has quit IRC | 10:21 | |
| *** udesale_ has joined #openstack-glance | 11:10 | |
| *** udesale has quit IRC | 11:13 | |
| *** k_mouza has joined #openstack-glance | 11:23 | |
| *** baojg has quit IRC | 11:25 | |
| *** baojg has joined #openstack-glance | 11:27 | |
| *** baojg has quit IRC | 11:52 | |
| *** baojg has joined #openstack-glance | 11:53 | |
| *** udesale_ has quit IRC | 12:07 | |
| *** baojg has quit IRC | 12:08 | |
| *** baojg has joined #openstack-glance | 12:09 | |
| *** baojg has quit IRC | 12:13 | |
| *** baojg has joined #openstack-glance | 12:28 | |
| *** lbragstad_ has joined #openstack-glance | 12:56 | |
| *** lbragstad has quit IRC | 12:56 | |
| *** ratailor has quit IRC | 13:06 | |
| openstackgerrit | Rajat Dhasmana proposed openstack/glance master: Add functional tests for cinder multiple store https://review.opendev.org/c/openstack/glance/+/750144 | 13:11 |
| *** tkajinam has quit IRC | 13:50 | |
| *** whoami-rajat__ has joined #openstack-glance | 14:01 | |
| whoami-rajat__ | dansmith: hey, I've updated the patch with your proposed changes https://review.opendev.org/c/openstack/glance/+/750144 | 14:01 |
| dansmith | whoami-rajat__: thanks, did you see the devstack patch I put up to address the quota bug with that cinder job? | 14:20 |
| whoami-rajat__ | dansmith: ah, there are failures on the cinder job? looking... | 14:23 |
| dansmith | whoami-rajat__: occasionally, and at least some of the time we run out of cinder quota while trying to do the glance upload | 14:23 |
| dansmith | probably because we're also running volume jobs for something else I would guess | 14:24 |
| dansmith | or maybe just a bunch of glance tests at the same time | 14:24 |
| dansmith | (I meant volume *tests* above not jobs) | 14:24 |
| dansmith | whoami-rajat__: https://bugs.launchpad.net/glance/+bug/1914665 | 14:24 |
| openstack | Launchpad bug 1914665 in Glance "Cinder Multistore job hits Cinder Quota error" [Undecided,New] | 14:24 |
| whoami-rajat__ | dansmith: ack, your patch makes sense +1, thanks for fixing this | 14:28 |
| whoami-rajat__ | (and is also working) | 14:28 |
| dansmith | whoami-rajat__: cool, thanks, just +2d your tests, maybe smcginnis would like to ack them for us :P | 14:29 |
| whoami-rajat__ | :D thanks \o/ | 14:30 |
| dansmith | whoami-rajat__: how come the cinder job is non-voting? if we want to run it each time we might as well make it voting, or move it to the experimental queue | 14:30 |
| *** udesale has joined #openstack-glance | 14:33 | |
| whoami-rajat__ | dansmith: that job is treated as a backend third party CI run on the cinder gate, it's non-voting (as it may be unstable and doesn't affect all patches) but it's good to make sure any glance change doesn't break the glance cinder setup | 14:35 |
| *** k_mouza has quit IRC | 14:44 | |
| *** k_mouza has joined #openstack-glance | 14:44 | |
| dansmith | whoami-rajat__: but it can be voting in our tree, even if not in cinder | 14:49 |
| smcginnis | multistore test patch looks good. Were there other patches that I should look at dansmith? | 14:51 |
| dansmith | smcginnis: yeah, this is actually what I meant for you: https://review.opendev.org/c/openstack/glance/+/750144 | 14:51 |
| smcginnis | Looking... | 14:51 |
| dansmith | oh that'ss the one you got | 14:51 |
| smcginnis | Hah, yep. | 14:52 |
| smcginnis | I was like, wait, this looks just like the other one. :D | 14:52 |
| dansmith | sorry | 14:52 |
| smcginnis | No worries! | 14:52 |
| dansmith | I thought maybe you meant you looked at the devstack patch | 14:52 |
| dansmith | you're not core over there TOO are you? | 14:52 |
| smcginnis | Haha, nope. | 14:52 |
| dansmith | https://review.opendev.org/c/openstack/devstack/+/774172 | 14:52 |
| dansmith | heh okay | 14:52 |
| smcginnis | I can certainly take a look though. | 14:52 |
| dansmith | oh yeah, it has cinder in the name, so your +1 is like gold | 14:53 |
| smcginnis | hah | 14:53 |
| dansmith | whoami-rajat__: there is something else about that job I want to look at.. the 30 second sleep that it does before it starts glance -- we really really shouldn't do that | 14:53 |
| dansmith | whoami-rajat__: what are we waiting for there, just cinder api being reachable? | 14:53 |
| whoami-rajat__ | dansmith: if glance team doesn't have issues i will make it voting | 14:54 |
| dansmith | whoami-rajat__: if we're going to run it on every patch and expect it to be stable, we should make it voting.. if we're not expecting it to be stable, it really should be in experimental.. nonvoting should be for things we're trying to stabilize, so yes, please propose to make it voting | 14:55 |
| whoami-rajat__ | dansmith: when cinder volume gets active, it collects the capabilities from backend and publishes it to the scheduler, i tested it manually and on gate as well so the range i saw was from 10sec to 22sec so kept a safe of 30 sec. i didn't have any alternative solution to it ... | 14:56 |
| dansmith | whoami-rajat__: okay, so this is some internal cinder thing we're waiting to happen.. you really think there's no way to tell from the outside that it has happened? | 14:57 |
| dansmith | maybe smcginnis knows? | 14:57 |
| dansmith | or rosmaita. between them I think they have all knowledge of humanity | 14:58 |
| whoami-rajat__ | dansmith: it's logged in the scheduler and that's how i get to know it, not sure of a way to tell devstack the same | 14:59 |
| whoami-rajat__ | i mean the scheduler logs | 14:59 |
| dansmith | whoami-rajat__: it also seems like if this *has* to be in place before glance starts that's very fragile | 14:59 |
| dansmith | whoami-rajat__: can we make glance lazy-load that the first time so we don't need it until we actually need it? | 14:59 |
| dansmith | whoami-rajat__: we should be able to do more work in devstack *while* that is happening, instead of having to block all work until it finishes | 15:00 |
| dansmith | like, would restarting the c-sch service cause it to happen immediately? | 15:01 |
| dansmith | or SIGHUP? | 15:01 |
| whoami-rajat__ | c-sch has no role in it but it should have received the capabilities from the backend when a create volume request comes to the scheduler | 15:01 |
| dansmith | is it create volume or volume type? | 15:02 |
| whoami-rajat__ | create volume | 15:02 |
| dansmith | okay, what volume is created at this point? doesn't glance create a volume for each image? or is it creating one volume to hold all the images? | 15:03 |
| dansmith | surely seems like the former because of the quota bug | 15:03 |
| *** lbragstad_ is now known as lbragstad | 15:03 | |
| whoami-rajat__ | glance creates a different volume for each image, when devstack triggers the cirros image creation then the first image-volume is created which fails if we don't wait for c-sch and c-vol to be properly setup | 15:04 |
| whoami-rajat__ | hence the wait time | 15:04 |
| dansmith | okay hold on let me go look at the code again, I thought you did this before you start g-api | 15:04 |
| dansmith | yeah, this happens before we even start g-api | 15:05 |
| dansmith | okay so maybe there are two things here | 15:05 |
| dansmith | lemme get some links | 15:06 |
| dansmith | this is the sleep: https://github.com/openstack/devstack/blob/edee6dc341e40939360b36ce9fd09052dea1ee4d/stack.sh#L1293-L1297 | 15:06 |
| dansmith | and you're saying that needs to happen before we try to upload an image the first time, yeah? | 15:06 |
| dansmith | then there is this: https://github.com/openstack/devstack/blob/edee6dc341e40939360b36ce9fd09052dea1ee4d/stack.sh#L1300 | 15:07 |
| dansmith | which says we need to wait on starting g-api until the *volume types* are created, somewhat separate issue, is that right? | 15:07 |
| whoami-rajat__ | the second one I'm not aware of but the first one is my finding | 15:09 |
| whoami-rajat__ | it seems to be a separate issue | 15:09 |
| dansmith | okay, I was squashing the two together because they're close and both related to cinder and glance | 15:09 |
| dansmith | whoami-rajat__: can you show me what c-sch logs that indicates we're ready? | 15:10 |
| whoami-rajat__ | let me find it | 15:11 |
| whoami-rajat__ | dansmith: sorry got involved in some other work, this is one comment i found in an old patch http://paste.openstack.org/show/802394/ | 15:23 |
| whoami-rajat__ | dansmith: this one i found from a recent run http://paste.openstack.org/show/802396/ | 15:29 |
| whoami-rajat__ | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_269/773566/1/check/glance-multistore-cinder-import/269d075/controller/logs/screen-c-sch.txt | 15:30 |
| whoami-rajat__ | scheduler logs if you want to look for more info ^ | 15:30 |
| dansmith | whoami-rajat__: okay, so wouldn't restarting c-vol make that happen immediately instead of waiting for a periodic? | 15:31 |
| whoami-rajat__ | hmm, not really sure that would work but I haven't tested that out myself | 15:32 |
| dansmith | I'm just going on how this kind of thing works in nova, | 15:32 |
| dansmith | but usually we do this sort of thing immediately at startup and then periodically after startup, which makes me think a quick bounce of c-vol would make it happen quicker | 15:32 |
| openstackgerrit | Lance Bragstad proposed openstack/glance master: Pass oslo.context RequestContext objects directly to policy enforcement https://review.opendev.org/c/openstack/glance/+/767425 | 15:33 |
| whoami-rajat__ | we can surely try it, will save a lot of time if it works | 15:33 |
| dansmith | whoami-rajat__: of those log lines, | 15:38 |
| dansmith | what are we looking for specifically? I see that's where capabilities gets updated, but what needs to be in there specifically? | 15:38 |
| whoami-rajat__ | dansmith: all the capabilities, it makes sure the scheduler detects a backend exists then it checks if it passes all configured filters from the capabilities (like has appropriate storage space for operation to complete) | 15:41 |
| dansmith | oh, so not some specific capability, just that the volume node has reported in at some point? | 15:41 |
| whoami-rajat__ | yep | 15:42 |
| dansmith | hrm, | 15:42 |
| dansmith | so this would be a thing for nova too surely | 15:42 |
| dansmith | just maybe we're always winning the race because we don't do the first nova op until much later? | 15:42 |
| dansmith | whoami-rajat__: would this not be enough to poll? $ openstack volume service list -f value -c Binary | grep volume | 15:45 |
| dansmith | that shows that the volume has registered | 15:45 |
| openstackgerrit | Rajat Dhasmana proposed openstack/glance master: DNM: testing glance cinder job with restarting c-vol https://review.opendev.org/c/openstack/glance/+/774255 | 15:45 |
| dansmith | I dunno when that happens compared to the communication to scheduler, but I'd kinda expect it to be the same.. do you know it's not? | 15:45 |
| whoami-rajat__ | c-vol service is registered and becomes active instantly still the reporting takes time | 15:46 |
| dansmith | okay | 15:46 |
| whoami-rajat__ | but i may be missing things since it's a long time i looked at that | 15:46 |
| whoami-rajat__ | i proposed glance + devstack patches to test the behavior with restart ^^ | 15:46 |
| dansmith | whoami-rajat__: cool, thanks | 15:47 |
| dansmith | whoami-rajat__: my async devstack patches make it so we can do other things while we wait for this, | 15:48 |
| dansmith | but I'd also like to wait for the actual thing to happen instead of just arbitrarily 30s all the time, and even better if we can poke c-vol to do its thing :) | 15:48 |
| dansmith | anyway, I'll look for some guidance from cinder people on what our options are here | 15:51 |
| whoami-rajat__ | dansmith: I'm also inclined to do it the right way but debugging it once took a lot of time and currently I'm involved in some other tasks | 15:54 |
| whoami-rajat__ | but i will surely look into it again | 15:54 |
| whoami-rajat__ | thanks for bringing it to attention :) | 15:54 |
| dansmith | whoami-rajat__: ack, yeah, I'm not trying to make it your problem, I just wanted to understand | 15:54 |
| dansmith | sounds like it's sufficiently obscure to justify the sleep on the first pass, but I'd like to make it better if we can | 15:54 |
| whoami-rajat__ | yeah, agree | 15:56 |
| *** lpetrut has quit IRC | 15:57 | |
| openstackgerrit | Lance Bragstad proposed openstack/glance master: nit: Clarify why check() is different from enforce() https://review.opendev.org/c/openstack/glance/+/774282 | 16:15 |
| openstackgerrit | Merged openstack/glance master: Add functional tests for cinder multiple store https://review.opendev.org/c/openstack/glance/+/750144 | 16:20 |
| dansmith | woot | 16:21 |
| *** nikparasyr has left #openstack-glance | 16:34 | |
| openstackgerrit | Merged openstack/glance master: Add devstack plugin script https://review.opendev.org/c/openstack/glance/+/773566 | 16:34 |
| *** k_mouza_ has joined #openstack-glance | 16:36 | |
| *** k_mouza has quit IRC | 16:37 | |
| *** ralonsoh has quit IRC | 17:14 | |
| *** udesale has quit IRC | 17:33 | |
| *** k_mouza_ has quit IRC | 18:11 | |
| *** hoonetorg has quit IRC | 18:19 | |
| *** hoonetorg has joined #openstack-glance | 18:22 | |
| *** k_mouza has joined #openstack-glance | 18:46 | |
| *** k_mouza has quit IRC | 18:50 | |
| *** hoonetorg has quit IRC | 19:06 | |
| *** hoonetorg has joined #openstack-glance | 19:10 | |
| *** hoonetorg has quit IRC | 19:22 | |
| *** hoonetorg has joined #openstack-glance | 19:28 | |
| *** hoonetorg has quit IRC | 20:18 | |
| *** hoonetorg has joined #openstack-glance | 20:22 | |
| openstackgerrit | Lance Bragstad proposed openstack/glance master: Bump requirements to perpare for secure RBAC https://review.opendev.org/c/openstack/glance/+/764236 | 21:15 |
| openstackgerrit | Lance Bragstad proposed openstack/glance master: Add basic/common personas to base policies https://review.opendev.org/c/openstack/glance/+/764241 | 21:15 |
| openstackgerrit | Lance Bragstad proposed openstack/glance master: Pass oslo.context RequestContext objects directly to policy enforcement https://review.opendev.org/c/openstack/glance/+/767425 | 21:15 |
| openstackgerrit | Lance Bragstad proposed openstack/glance master: Implement secure RBAC for image actions https://review.opendev.org/c/openstack/glance/+/764754 | 21:15 |
| openstackgerrit | Lance Bragstad proposed openstack/glance master: WIP: add logic to properly enforce secure RBAC when handling images https://review.opendev.org/c/openstack/glance/+/765210 | 21:15 |
| openstackgerrit | Lance Bragstad proposed openstack/glance master: Explicitly unset tempest_roles in devstack plugin https://review.opendev.org/c/openstack/glance/+/774308 | 21:15 |
| openstackgerrit | Lance Bragstad proposed openstack/glance master: Properly handle InvalidScope exceptions https://review.opendev.org/c/openstack/glance/+/774309 | 21:15 |
| *** Underknowledge3 has joined #openstack-glance | 21:39 | |
| *** Underknowledge has quit IRC | 21:41 | |
| *** Underknowledge3 is now known as Underknowledge | 21:41 | |
| *** k_mouza has joined #openstack-glance | 22:12 | |
| *** k_mouza has quit IRC | 22:17 | |
| *** dasp_ has joined #openstack-glance | 22:18 | |
| *** dasp has quit IRC | 22:20 | |
| openstackgerrit | Dan Smith proposed openstack/glance master: Pass ImageActionWrapper to internal plugins https://review.opendev.org/c/openstack/glance/+/774313 | 22:24 |
| openstackgerrit | Dan Smith proposed openstack/glance master: Make web-download revert all stores on fail https://review.opendev.org/c/openstack/glance/+/774314 | 22:24 |
| openstackgerrit | Dan Smith proposed openstack/glance master: DNM: Test web-download tempest negative test https://review.opendev.org/c/openstack/glance/+/774315 | 22:24 |
| *** rchurch has quit IRC | 22:29 | |
| *** rchurch has joined #openstack-glance | 22:32 | |
| openstackgerrit | Dan Smith proposed openstack/glance master: DNM: Test web-download tempest negative test https://review.opendev.org/c/openstack/glance/+/774315 | 22:36 |
| *** whoami-rajat__ has quit IRC | 22:39 | |
| openstackgerrit | Dan Smith proposed openstack/glance master: DNM Reproduce failure for bug 1914826 https://review.opendev.org/c/openstack/glance/+/774317 | 22:40 |
| openstack | bug 1914826 in Glance "web-download with invalid url does not report error" [Undecided,In progress] https://launchpad.net/bugs/1914826 - Assigned to Dan Smith (danms) | 22:40 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!