Monday, 2020-07-20

*** irclogbot_3 has quit IRC00:16
*** irclogbot_0 has joined #openstack-glance00:19
*** Liang__ has joined #openstack-glance01:35
openstackgerritErno Kuvaja proposed openstack/glance master: Fix admin docs deplying under HTTPD  https://review.opendev.org/74185901:45
*** Liang__ has quit IRC02:54
*** Liang__ has joined #openstack-glance02:55
*** rcernin has quit IRC03:02
*** Liang__ has quit IRC03:06
*** rosmaita has left #openstack-glance03:18
*** Liang__ has joined #openstack-glance03:18
*** rcernin has joined #openstack-glance03:20
*** rcernin has quit IRC03:25
*** rcernin has joined #openstack-glance03:26
*** m75abrams has joined #openstack-glance04:16
*** ratailor has joined #openstack-glance04:39
*** udesale has joined #openstack-glance05:48
*** nikparasyr has joined #openstack-glance06:45
*** Luzi has joined #openstack-glance07:02
*** nikparasyr has quit IRC07:08
*** amoralej|off is now known as amoralej07:16
*** rcernin has quit IRC07:24
*** m75abrams is now known as mabrams07:36
*** mabrams is now known as m75abrams07:36
*** rcernin has joined #openstack-glance07:38
*** rcernin has quit IRC07:44
*** rcernin has joined #openstack-glance07:48
*** rcernin has quit IRC07:52
*** ralonsoh has joined #openstack-glance07:55
*** ralonsoh_ has joined #openstack-glance08:53
*** ralonsoh has quit IRC08:55
*** k_mouza has joined #openstack-glance09:01
*** rcernin has joined #openstack-glance09:10
*** rcernin has quit IRC09:15
*** baojg has quit IRC09:30
*** ratailor has quit IRC09:58
*** ratailor_ has joined #openstack-glance09:58
*** k_mouza has quit IRC10:08
*** Luzi has quit IRC10:08
*** irclogbot_0 has quit IRC10:08
*** vesper11 has quit IRC10:08
*** jdillaman has quit IRC10:09
*** k_mouza has joined #openstack-glance10:10
*** Luzi has joined #openstack-glance10:10
*** irclogbot_0 has joined #openstack-glance10:10
*** vesper11 has joined #openstack-glance10:10
*** jdillaman has joined #openstack-glance10:10
*** m75abrams has quit IRC10:24
*** tkajinam has quit IRC10:29
*** trident has quit IRC10:35
*** trident has joined #openstack-glance10:36
*** Liang__ has quit IRC10:40
openstackgerritErno Kuvaja proposed openstack/glance master: Fix admin docs deplying under HTTPD  https://review.opendev.org/74185911:15
*** baojg has joined #openstack-glance11:56
*** baojg has quit IRC12:24
*** rosmaita has joined #openstack-glance12:26
*** amoralej is now known as amoralej|lunch12:40
*** baojg has joined #openstack-glance12:47
*** baojg has quit IRC12:48
*** nikparasyr has joined #openstack-glance12:50
*** udesale_ has joined #openstack-glance13:03
*** udesale has quit IRC13:06
*** rcernin has joined #openstack-glance13:07
*** mvkr has joined #openstack-glance13:09
*** k_mouza has quit IRC13:09
*** rosmaita has quit IRC13:11
*** rcernin has quit IRC13:12
*** rosmaita has joined #openstack-glance13:15
*** ratailor_ has quit IRC13:19
*** k_mouza has joined #openstack-glance13:29
*** k_mouza has quit IRC13:34
*** amoralej|lunch is now known as amoralej13:35
*** k_mouza has joined #openstack-glance13:37
*** k_mouza has quit IRC13:41
*** k_mouza has joined #openstack-glance13:47
*** jv_ has joined #openstack-glance13:50
*** k_mouza has quit IRC13:51
*** k_mouza has joined #openstack-glance14:03
*** k_mouza has quit IRC14:06
*** k_mouza has joined #openstack-glance14:27
dansmithabhishekk: looks like the image disappeared in the middle of the task? https://zuul.opendev.org/t/openstack/build/9d3381bac3544cfea4c1bb1bd5bbed2c/log/controller/logs/screen-g-api.txt#840814:37
dansmiththat's weird because the client continues to GET it, and waits until timeout because the task didn't report error I guess14:38
*** baojg has joined #openstack-glance14:43
*** baojg has quit IRC14:44
*** baojg has joined #openstack-glance14:45
abhishekkdansmith, seems weird, because it is behaving different for train-py3 only14:48
*** baojg has quit IRC14:52
*** baojg has joined #openstack-glance14:53
*** baojg has quit IRC14:54
*** baojg has joined #openstack-glance14:55
*** rcernin has joined #openstack-glance14:55
*** baojg has quit IRC14:57
*** baojg has joined #openstack-glance14:58
dansmithabhishekk: well, I wonder if it's racy14:59
*** rcernin has quit IRC15:00
abhishekkdansmith, sounds like a race to me15:02
abhishekkhttps://zuul.opendev.org/t/openstack/build/9d3381bac3544cfea4c1bb1bd5bbed2c/log/controller/logs/screen-g-api.txt#818615:02
abhishekkthis warning confirms that image from staging data is already deleted15:02
dansmithabhishekk: that's from a different test, but if that shouldn't be happening, then ... yeah, uncool15:03
*** baojg has quit IRC15:03
abhishekkdansmith, nope, that warning is not coming from taskflow15:06
*** Luzi has quit IRC15:09
*** ralonsoh_ is now known as ralonsoh15:10
*** nikparasyr has left #openstack-glance15:15
*** jgwentworth is now known as melwitt15:19
* abhishekk dinner break, back in 45 mins15:19
*** gyee has joined #openstack-glance15:51
*** amoralej is now known as amoralej|off16:11
*** yoctozepto has quit IRC16:12
*** yoctozepto has joined #openstack-glance16:13
*** k_mouza has quit IRC16:15
*** udesale_ has quit IRC16:28
jokkedansmith: looks like I might owe you beer or 7 when we manage to get around same table next time ;)16:36
*** rcernin has joined #openstack-glance16:56
*** rcernin has quit IRC17:01
openstackgerritMerged openstack/glance master: Fix admin docs deplying under HTTPD  https://review.opendev.org/74185917:04
openstackgerritErno Kuvaja proposed openstack/glance stable/ussuri: Fix admin docs deploying under HTTPD  https://review.opendev.org/74199217:08
dansmithjokke: looks like because why?17:15
dansmithbut also, I'm a very light drinker so it'll probably take me 7 years to collect on a debt of 7 beers :)17:16
jokkedansmith: a) your patience with us and b) seems like you managed to achieve something with devstack we failed by our calculations about 3 years :D17:16
jokkeIIUC you gave me a reason to archive my "Glance - It Doesn't Work in Devstack" t-shirt17:17
dansmithwell, it really wasn't a major struggle, just required some doing and explaining, but.. :)17:19
*** baojg has joined #openstack-glance17:20
*** ralonsoh has quit IRC17:48
rosmaitadansmith: jokke: left a suggestion on https://review.opendev.org/#/c/737596/1318:40
rosmaitajokke: rofl with your docs update18:43
jokkerosmaita: just reflecting reality on something I have overlooked ofr long time due to that Warning we had there18:47
jokkeI think I read the whole body first time last week and got kind of shoked18:47
dansmithjokke: so with that doc, I assume that either (a) you're okay with my patch to make it fail, or (b) okay if I make that patch a delete of wsgi_app.py right?18:50
dansmithif it's not supported and they shouldn't do it, then an extra exception to make it crashy is no hurting any valid use-case so no real objection? :)18:50
jokkedansmith: I think currently either approach will still break CI (as at least grenade is depending on it)18:51
jokkeWe might need to wait until V gets released before we can actually yank it out18:51
dansmithjokke: grenade can be fixed to make it deploy the thing in the new way18:51
dansmithwhich is how major changes like this garner the attention of the tc and qa people, which sounds appropriate here18:52
dansmithand the (a) case is fully supportable in CI right now, so I'd still rather do it18:52
dansmithalready have that grenade patch written18:52
dansmithhttps://review.opendev.org/#/c/741675/18:53
dansmithready to merge if you drop your -218:53
jokkedansmith: that would be the best way forward. The thing is we haven't been supporting running glance without any import methods since it was permanently turned on. This whole image import was defcore requirement for the image creation feature being discoverable and available in all clouds (the old PUT method might be behind policy wall due to all kind of security concerns rosmaita can shed more18:54
jokkelight of)18:54
dansmithjokke: just to be clear, "that" means "delete wsgi_app.py and make grenade fix it up" right?18:55
jokkeyeah18:55
jokkeLike we've had now 3ish years to find someone who cares about that deployment model enough to fix it which has not happened so I'd be more than happy to get just rid of it18:57
mnasercan we not remove it18:57
*** rcernin has joined #openstack-glance18:57
mnasersome of us dont care about interop image import and have users who do straight POST of image uploads18:57
mnaserif you had reached out to the community of deployers18:57
jokkemnaser: I tohught you wanted us to prevent starting the service in configuration it doesn't work properly on, this would do exactly that18:58
mnaseryou'd know that there are many people running glance under uwsgi18:58
mnaseryes, except i can turn off that option18:58
mnaserand then my cloud keeps running18:58
mnaseri dont have to end up running eventlet/glance-api18:58
mnaseropenstacksdk literally got interop image upload a few days ago and that was the efforts of mordred18:58
mnaserif you think it's time for us to yank support on that and everyone else running using a sorts of wsgi server, it's way too early18:59
abhishekkmnaser, I guess then you also never used reload config feature of glance?18:59
mnaserno.  i've never used it for any other service18:59
mnaserand fwiw, if neutron managed to de-couple successfully18:59
mnaseri cant imagine it being so much harder for glance18:59
mnaser(and honestly, to add it ot, interop image upload is a huge headache for ceph-backed deployers)19:00
mnaserbuffering all this stuff on disk and then uploading after is painful19:00
*** rcernin has quit IRC19:01
jokkemnaser: funny ... all our customers are screaming for it to get the conversion working so they don't need to deal with users trying to upload qcows etc.19:01
jokkefor exactly ceph use19:01
dansmithjust in case anyone cares, I think that we *should* retain pure-wsgi support in glance, not force everyone to use (or enable) import and make the wsgi server not agree to run configurations that can't possibly work19:01
abhishekkright19:01
mnaser1) we have a setting to disallow specific image formats19:02
mnaser2) we (vexxhost) even pushed a patch to horizon to let the dashboard dynamically determine the upload-able formats19:02
mnaser3) if you're doing bfv, you can use the cinder volume cache to workaround that specific issue19:03
jokkemnaser: python-glanceclient has supported it since the initial release. But lets not get agitated by going back to the whole client discussion, that horrse has been beated dead so many times already ;)19:05
mnaserjokke: this feels like a pattern where the glance team doesn't follow the rest of openstack19:06
mnaseruwsgi, glance client19:06
mnaserif you want to worry about user experience, kill python-glanceclient.  that's real life feedback from users19:07
jokkedansmith: I'm ok keeping the wsgi_app.py around and call it supported if someone is willing to make the work to fix all the issues it's bringing. I just have my concerns as it has not happened for the past years19:07
dansmithjokke: does that mean only if someone is going to make tasks run?19:08
mnaserwhat issues does wsgi_app.py have?19:08
dansmiththis API is already feature-flagged, so saying this doesn't work in that mode is already something we can properly advertise19:08
dansmithand nova has similar things that don't work if you make certain deployment decisions19:08
dansmithjokke: if tasks running has to be the end goal of keeping it, I totally agree with you that it's likely not to happen19:09
jokkemnaser: what lead to this very conversation is that we tried both. Neither worked out19:09
dansmithjokke: if it's just making it as consistent as possible (i.e. not advertising the things it can't do) then ... I think that's what my patch is doing :)19:09
mnaserno, uwsgi+glance works just fine19:10
mnaseri assure you that it does19:10
mnaserexcept for the import19:10
jokkemnaser: we just recently dropped the osc promotion from our docs and went back to support only python-glanceclient19:10
mnaserwhich i dont care about, so dansmith patch that you -2'd would stop me from starting a non-functional system19:10
mnaserjokke: that is infurirating19:10
mnaserand a huge fu to the python osc team19:10
mnaseryou're not doing things in the best interest of the project19:11
*** knikolla has joined #openstack-glance19:12
jokkedansmith: so the whole effort of making the interoperable image import was because that was defcore demand for user facing image creation. Like I mentioned earlier there is plan to drop the PUT method to work with service tokens alone just so we don't need to rewrite cinder and nova which are doing the right thing already.19:12
dansmithjokke: meaning drop the regular image create method we have now?19:13
jokkeSo I don't see how wsgi_app without image import would be operating service with the roadmap we have19:13
dansmithif so I definitely did not realize that was a desire on the table19:13
jokkedansmith: from regular users, yes19:13
mnaserwait19:13
mnaserare we seriously dropping19:13
mnaserPOST /images/foo/data ?19:14
jokkethat was the defcore requirement19:14
mnaserdefcore was like19:14
mnaser2015?19:14
mnaserinterop wg is what holds it19:14
mnaserand i dont think the interop wg is doing much more than like19:14
mnasercopyign over teh same api calls and maintaining that parity19:14
dansmithjokke: okay so this provides a lot more clarity to me on the push-back.. I'm definitely concerned about that roadmap decision, but ... at least I understand19:16
jokkedansmith: none of us doing the work were happy about it, specially as it took like 3 years of bikeshedding to get to an agreement in the spec how this gets done19:17
jokkeand that's why we tied it to the taskflow to make sure we don't need to go through that ever again19:18
jokkeand can have the extendability there19:18
jokkeit also opened obviously lots of doors to do different stuff as well, which is great19:21
jokkemnaser: I assume none of your clouds is (/plan to be) using multi-store either19:29
mnasernot anytime soon, but even if they are multistore, the fact that i have to get some share that spans across all my controllers to get import to work is a non starter19:30
jokkemnaser: we're working on that ... just turned out to be another non-trvial thing to solve19:30
mnaserjokke: if api and '19:31
mnaser'workers' were split off, that might be a lot easier to accomplish19:31
jokkeactually splitting the workers off becomes much easier once we do have some way for the nodes in the cluster communicate with eachother ... We just don't have any level of RPC between the Glance nodes19:33
abhishekkmnaser, so you might be aware that we have deprecated single store option19:33
mnaserthere's oslo.messaging :)19:34
mnaserand i really hope we don't have to actually request users to save things into a store19:34
jokkeOnce we get that, then effectively the 'worker' and the request receiving glance-api are separate and we can look into having just workers and just api's19:34
abhishekklike, stores and default_store and planning to remove those in this cycle19:35
knikollai just want to make sure i'm caught up to speed. there are two methods to upload images, the old one through PUT /v2/images/{image_id}/file and the new one (interop) which is multi-stage and you first upload the image through PUT /v2/images/{image_id}/stage and then POST to /v2/images/{image_id}/import. The latter one is the preferred one by the glance team because it allows processing in between or upload to19:37
knikollamultiple stores to happen. however the async nature of the latter makes it so it can't work unless it has eventles or a worker model19:37
jokkemnaser: you will still have default store, does not need to be user defined where the image is going. Just simplifying the glance_store lib nd getting rid of the old API that caused us so much gray hair19:37
jokkeknikolla: yeah. API level you're correct. User doesn't need to care as obviously client can handle that in one command. The async model also introduced hardened "copy_from" from the v1 times back and makes it possible for user to just call it with URI and let glance-api to take care of the whole heavy lifting19:39
* knikolla has nightmare flashbacks from v119:40
knikollathanks jokke, i have a better understanding now19:40
jokkeknikolla: this time we have actual filtering for it so you can define protocols, ports, hosts ... it's not your portscan as a service anymore :P19:41
jokkeand for you I mean admin in this case ^^19:41
jokkeSo admin can specify that only https listening port 443 is ok19:42
jokkeor if ports are only thing you're concerned you can filter FE 80, 8080, 44319:43
jokkeand thus make sure that it doesn't become portscan as a service like the v1 copy-from used to be ;D19:43
knikollabut even from an glance v1 API user perspective it was a nightmare to work with. Completely unlike any other openstack API. there were so many conditionals in my code for if glance v1.19:44
jokkeknikolla: it was never supposed to be user facing API ... it was always designed to be called from Nova only19:44
jokkethen there was tons of gafftape and hubbabubba to suddenly make it userfacing when the decision was made that services proxying calls to other services was not great idea19:45
jokkeNow later thinking what would have been great idea, is if we had split the v2 api to user facing, service facing and perhaps admin endpoints19:47
jokkebut it is what it is, at least we last remains of v1 api are gone as of last week :P19:48
jokkes/we/the/19:48
knikollai don't generally think splitting admin/public endpoints makes a ton of sense.19:49
dansmithknikolla: agreed19:49
dansmithjokke: when import becomes "the way" I assume that the old PUT interface will just proxy internally to the new code right?19:49
dansmithI worry about making nova have to use this multistep process for snapshots for no real reason19:50
jokkeyeah, that's why the perhaps ... sometimes I think it would have been great to have and lots of times I'm happy it doesn't exist19:50
dansmithpresumably import also takes approximately twice as long as the regular method because of the staging step?19:51
knikolladansmith: maybe more if conversion is introduced.19:51
jokkedansmith: the PUT interface will be direct to store bypassing the taskflow. Like there is very little reason to have nova uploading snapshot and driving it through introspection/convert/property injection. The right this are in place19:51
dansmithknikolla: certainly more if conversion is introduced :)19:51
dansmithjokke: oh you said "for users" before, so you're not going to _remove_ it you just want users to upload their images via import, is that right?19:52
jokke*the right things*19:52
knikollajokke: do you think there exists a path for gradually splitting a worker out of the API for the interop image import flow?19:52
jokkeknikolla: like I mentioned earlier, it's very possible once we get some form of fanout RPC happening. But we need mechanism to transfer the request which we do not have at the moment19:54
knikollai think that is not something exclusive to glance, and it's something that if we can solve in a consistent way across openstack can open the door for other services to consume.19:55
jokkedansmith: yeah, so latest idea/plan is just to lock it behind the service token ... so when it's coming from Nova/Cinder etc. it gets authenticated as service doing the request for valid user. And normal human users are supposed to go via the new endpoint which allows more processing in the cloud end19:55
dansmithisn't the solution to that to return a 3xx to the POST and redirect to one of the workers by name?19:55
dansmithjokke: and presumably we can have the fight about breaking every user ever at _that_ point :)19:56
jokkedansmith: sure we can, but if the client already does the right thing without 98% of users even knowing it and we just deprecate the old api with proper period, I let people have their fight over it19:57
dansmithosc doesn't19:58
dansmithand not everyone uses the python clients anyway :)19:58
jokkeif your automation needs to keep using it, get service token for it, for normal users there shouldn't be anything they need to be concerned about19:58
dansmithyeah, well, we'll see I guess :)19:58
knikollain our cloud, all our documentation points to osc for everything, including glance operations.19:59
jokkeexactly ... with the phase any of these happens I assume we might be fully there around 203819:59
dansmithjokke: but still, it's 2x the time, and 2x the network bandwidth regardless, right?19:59
jokkedansmith: nope, the network bandwidth is the same, now you just stream it through the api19:59
dansmithjokke: for the user, it's not doubled, for the api worker it is right?20:00
dansmithsorry, I mean assuming the staging area is on shared storage,20:00
jokkedansmith: and the client is held until the upload to glance is done. You have quick backend and little plugins doing processing, the service side goes quick20:00
dansmithwhich is the only way this works in a multi-controller thing right now20:00
jokkedansmith: yeah, in that case indeed20:01
dansmithbut the time is doubled, yeah? so the client waits for $transfer_time, $process_time, then $internal_transfer_time before finishing yeh?20:01
jokkewhich is one of the reasons we want to solve the fanout rpc so we can internally just shoot the import request to the nodes "Who ever has the data for it, please process"20:01
knikollajokke: i think dansmith's point is that you can't stream from the user directly to the backend20:02
jokkethe web-download doesn't care as the data is retrieved in the taskflow process, with glance-direct that would be indeed the case20:02
dansmithknikolla: right20:03
dansmithjokke: but users will be less likely to use web-direct20:03
jokkeinterestingly enough that was on of the biggest griefs we got for killing v1 API20:05
jokkelots of people want to just point to the public uri of image vendor X and not needing ot download it to their laptop/workstation just to be able to upload it back to the cloud they use20:06
dansmithsure, definitely a thing that needs to work, no doubt,20:06
dansmithand very cloudy users may be super good at that20:06
dansmithbut the average horizon user? no way.20:06
dansmithabhishekk: are we good to land this? https://review.opendev.org/#/c/741276/20:15
jokkeI'm actually not sure if Horizon supports it again, you used to be able to just copypaste the URI and off you go :P20:15
dansmithjokke: if you haven't seen it ^20:15
dansmiththis job validates that the cirros image was uploaded via import, metadata was injected, etc20:16
dansmithI need to drop the WIP but ... if there's something else we need there I can do it at the same time20:16
abhishekkdansmith, I am good with it20:18
dansmithcool20:18
* abhishekk signing out for the day (almost midnight :P)20:19
dansmithabhishekk: o/20:19
abhishekko/~20:20
jokkeI need to run to the store before they close (in about 36min) ... biab20:24
smcginnisjokke: That dnspython fix is already in flight.20:24
jokkesmcginnis: oh, ok cool :)20:25
smcginnisJust need all the zuul jobs to finally stop being flaky. ;)20:25
jokkeikr20:25
jokkethanks smcginnis I'll just abandon my patch20:25
smcginnisSorry, I already did. :)20:26
jokkenp20:26
smcginnisGet to the store!20:26
jokkeless clicks for me ... exactly!20:26
openstackgerritDan Smith proposed openstack/glance master: Make our tempest job use import, standalone, and inject_metadata  https://review.opendev.org/74127620:36
*** rcernin has joined #openstack-glance20:57
jokkeb20:59
*** rcernin has quit IRC21:02
openstackgerritMerged openstack/glance stable/ussuri: Fix admin docs deploying under HTTPD  https://review.opendev.org/74199221:12
openstackgerritErno Kuvaja proposed openstack/glance stable/train: Fix admin docs deploying under HTTPD  https://review.opendev.org/74202821:16
*** jdillaman has quit IRC21:35
dansmithsmcginnis: is the python thing you said is close to being fixed why the py3 jobs are failing in glance?21:37
smcginnisdansmith: It hasn't merged yet. It had to be rechecked.21:40
dansmithguh21:40
dansmithsmcginnis: but that's the reason I guess?21:40
smcginnisBah! And it was just finishing up tempest and had to restart.21:41
smcginnisdansmith: The reason for the failing glance tests?21:41
dansmithoye21:41
smcginnisI didn't look at the failures directly, but it sounded like they came to the same conclusion that it is the issue between eventlet and dnspython that was causing problems with the novnc tests in nova.21:42
dansmithhrm21:43
dansmithnot sure that is this21:43
dansmithit's tasky stuff so maybe21:43
smcginnisNot sure if it would speed things up, but you could throw a Depends-on one of the glance patches to have it pick up the requirements change.21:44
smcginnisCould be a good way to verify that's it while we wait for it to actually merge.21:44
dansmithoh it's not a huge rush or anything I just want to make sure it's not something else I need to chase21:44
smcginnisAlways something to chase. Hopefully this is it for this particular failure. ;)21:45
dansmithyeah I'd rather take the "my code is compiling" excuse and go outside or something...21:45
dansmithyeah21:45
smcginnisYeah, go outside. That's a much better use of time.21:45
dansmithwell, you don't have all the data21:45
dansmithif I go outside, I have to fix a fence21:45
dansmithin the sun21:45
smcginnisHaha, fair enough. In that case, keep poking at failing tests until it's cooler out.21:46
dansmithhehe21:46
dansmithspousal merit awards for the fence though.. DECISIONS.21:46
smcginnisDILEMMA21:47
*** tkajinam has joined #openstack-glance22:53
*** rcernin has joined #openstack-glance22:58
*** rcernin has quit IRC22:59
*** rcernin has joined #openstack-glance22:59

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!