Thursday, 2020-02-27

*** tbachman has quit IRC00:01
*** brinzhang has joined #openstack-nova00:03
*** brinzhang_ has joined #openstack-nova00:33
*** brinzhang has quit IRC00:36
*** ociuhandu has joined #openstack-nova00:42
*** gyee has quit IRC00:44
*** brinzhang has joined #openstack-nova00:44
*** brinzhang_ has quit IRC00:47
*** ociuhandu has quit IRC00:48
openstackgerritBrin Zhang proposed openstack/nova master: Add test coverage of existing os-volumes-attachments policies  https://review.opendev.org/70992901:07
*** yedongcan has joined #openstack-nova01:09
*** zhanglong has joined #openstack-nova01:17
*** huaqiang has quit IRC01:23
openstackgerritBrin Zhang proposed openstack/nova master: Introduce scope_types in os-volumes-attachments policy  https://review.opendev.org/70938801:31
*** brinzhang_ has joined #openstack-nova01:54
*** brinzhang has quit IRC01:57
*** brinzhang has joined #openstack-nova01:59
*** brinzhang has quit IRC02:01
*** brinzhang has joined #openstack-nova02:02
*** brinzhang_ has quit IRC02:02
*** brinzhang_ has joined #openstack-nova02:16
*** brinzhang has quit IRC02:19
*** zhanglong has quit IRC02:26
rm_workgibi: reviewed the routed-network tempest test -- that test needs a bit of cleanup02:26
*** zhanglong has joined #openstack-nova02:28
*** ociuhandu has joined #openstack-nova02:43
*** ociuhandu has quit IRC02:48
*** brinzhang has joined #openstack-nova02:54
*** brinzhang_ has quit IRC02:57
*** jdillaman has quit IRC03:03
*** nicolasbock has quit IRC03:09
*** brinzhang_ has joined #openstack-nova03:18
*** tbachman has joined #openstack-nova03:18
*** brinzhang_ has quit IRC03:19
*** brinzhang_ has joined #openstack-nova03:20
*** brinzhang_ has quit IRC03:21
*** brinzhang has quit IRC03:22
*** brinzhang has joined #openstack-nova03:22
*** brinzhang_ has joined #openstack-nova03:24
*** brinzhang has quit IRC03:25
*** brinzhang_ has quit IRC03:26
*** brinzhang has joined #openstack-nova03:26
*** rchurch has quit IRC03:28
*** brinzhang_ has joined #openstack-nova03:30
*** brinzhang_ has quit IRC03:31
*** brinzhang has quit IRC03:34
*** mkrai has joined #openstack-nova03:53
*** udesale has joined #openstack-nova04:35
*** ociuhandu has joined #openstack-nova04:45
*** imacdonn has quit IRC04:47
*** imacdonn has joined #openstack-nova04:48
*** ociuhandu has quit IRC04:49
*** larainema has joined #openstack-nova05:06
*** rcernin has quit IRC05:33
*** rcernin has joined #openstack-nova05:33
*** evrardjp has quit IRC05:34
*** evrardjp has joined #openstack-nova05:35
*** links has joined #openstack-nova05:53
*** zhanglong has quit IRC05:55
*** zhanglong has joined #openstack-nova05:57
*** ratailor has joined #openstack-nova06:01
*** kozhukalov has joined #openstack-nova06:07
*** rcernin has quit IRC06:24
*** lbragstad has quit IRC06:26
*** maciejjozefczyk has joined #openstack-nova06:43
*** ociuhandu has joined #openstack-nova06:45
*** ccamacho has quit IRC06:49
*** ociuhandu has quit IRC06:50
openstackgerritBrin Zhang proposed openstack/nova master: Add new default roles in os-volumes-attachments policies  https://review.opendev.org/71019006:52
*** maciejjozefczyk has quit IRC06:55
*** ociuhandu has joined #openstack-nova07:02
*** ociuhandu has quit IRC07:07
*** mmethot has quit IRC07:11
*** mmethot has joined #openstack-nova07:22
*** ralonsoh has joined #openstack-nova07:28
*** jawad_axd has joined #openstack-nova07:47
*** tesseract has joined #openstack-nova07:53
*** slaweq has joined #openstack-nova07:53
*** brinzhang has joined #openstack-nova08:00
*** tkajinam has quit IRC08:02
*** brinzhang has quit IRC08:02
*** damien_r has joined #openstack-nova08:02
*** brinzhang has joined #openstack-nova08:02
*** damien_r has quit IRC08:03
*** damien_r has joined #openstack-nova08:03
*** ociuhandu has joined #openstack-nova08:06
aarentsHi there, gibi when you will have time for that, can you confirm that https://review.opendev.org/#/c/702368/ is ok for you, I fixed the nits you reported.08:08
*** maciejjozefczyk has joined #openstack-nova08:11
*** ociuhandu has quit IRC08:12
*** ccamacho has joined #openstack-nova08:18
gibirm_work: thank you for the tempest review. I've started working on the pre-filter patch from mriedem08:22
gibiaarents: opening the patch now...08:22
rm_work:thumbsup:08:22
*** ociuhandu has joined #openstack-nova08:28
gibistephenfin: a relatively easy bugfix needs a second code https://review.opendev.org/#/c/70236808:31
gibiaarents: thanks for the bufix, I'm +2 on it08:31
*** udesale has quit IRC08:31
gibistephenfin: s/code/core/ // I need more coffeee08:32
*** udesale has joined #openstack-nova08:32
aarentsgibi: thks08:32
*** ociuhandu has quit IRC08:38
*** amoralej|off is now known as amoralej08:43
lyarwoodkashyap / stephenfin ; https://review.opendev.org/#/c/708745/ - a simple partial fix for the libvirt v6.0.0 qcow2 backing file format issue if you have time to review today.08:48
kashyaplyarwood: Morning, will look08:54
kashyaplyarwood: The discussions upstream turned even more complex; but I think we should go with this fix for now08:56
kashyapLooks good to me08:56
*** rpittau|afk is now known as rpittau09:09
*** ociuhandu has joined #openstack-nova09:14
*** elod has quit IRC09:19
*** brinzhang_ has joined #openstack-nova09:25
*** brinzhang_ has quit IRC09:27
*** brinzhang_ has joined #openstack-nova09:27
*** brinzhang has quit IRC09:28
*** breizhkoala has joined #openstack-nova09:36
*** ociuhandu has quit IRC09:41
*** xek__ has joined #openstack-nova09:43
*** maysams has joined #openstack-nova09:48
*** jangutter has joined #openstack-nova10:08
*** ociuhandu has joined #openstack-nova10:12
*** jangutter has quit IRC10:28
*** elod has joined #openstack-nova10:29
openstackgerritJohannes Kulik proposed openstack/nova master: Validate id as integer for os-aggregates  https://review.opendev.org/71021410:30
*** mkrai has quit IRC10:30
jkulikregarding ^ afaics, osc-lib first tries to get() whatever is supplied to `openstack aggregate show`, which the os-aggregates endpoint doesn't handle well, if it's a name and not an id.10:34
jkulikresulted in too many 500 errors in our environment10:34
*** ociuhandu has quit IRC10:39
*** mkrai has joined #openstack-nova10:47
kashyaplyarwood: Reviewed; I also posted the upstream libvirt conclusion for selective format probing (and not letting it recurse into multiple levels of backing chains)10:51
kashyap(Read this one: https://www.redhat.com/archives/libvir-list/2020-February/msg01001.html)10:51
*** mkrai has quit IRC10:52
lyarwoodkashyap: ack thanks10:58
*** brinzhang has joined #openstack-nova11:10
*** rpittau is now known as rpittau|bbl11:12
*** brinzhang_ has quit IRC11:14
*** kozhukalov has quit IRC11:17
*** ociuhandu has joined #openstack-nova11:18
*** ociuhandu has quit IRC11:18
*** ociuhandu has joined #openstack-nova11:19
*** ociuhandu has quit IRC11:20
*** kozhukalov has joined #openstack-nova11:21
*** ociuhandu has joined #openstack-nova11:21
*** ociuhandu has quit IRC11:26
*** ociuhandu has joined #openstack-nova11:26
*** ociuhandu has quit IRC11:29
*** ociuhandu has joined #openstack-nova11:30
*** kozhukalov has quit IRC11:35
*** ociuhandu has quit IRC11:35
*** kozhukalov has joined #openstack-nova11:35
*** ociuhandu has joined #openstack-nova11:36
*** ociuhandu has quit IRC11:36
*** ociuhandu has joined #openstack-nova11:37
*** jangutter has joined #openstack-nova11:38
*** ociuhandu has quit IRC11:39
*** ociuhandu has joined #openstack-nova11:39
*** ociuhandu has quit IRC11:40
*** tbachman has quit IRC11:41
*** ociuhandu has joined #openstack-nova11:42
*** kozhukalov has quit IRC11:43
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Use 'import foo from bar'  https://review.opendev.org/71023511:45
*** ociuhandu has quit IRC11:48
*** ociuhandu has joined #openstack-nova11:48
*** ivve has joined #openstack-nova11:53
*** ociuhandu has quit IRC11:56
*** ociuhandu has joined #openstack-nova11:56
*** ociuhandu has quit IRC11:57
*** ociuhandu has joined #openstack-nova11:57
*** ociuhandu has quit IRC11:58
*** iurygregory has joined #openstack-nova11:59
*** brinzhang_ has joined #openstack-nova11:59
*** brinzhang_ has quit IRC12:01
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_MULTIATTACH  https://review.opendev.org/71023812:01
openstackgerritStephen Finucane proposed openstack/nova master: virt: Remove QEMU_VERSION_REQ_SHARED  https://review.opendev.org/71023912:01
*** brinzhang_ has joined #openstack-nova12:01
*** brinzhang has quit IRC12:02
*** brinzhang_ has quit IRC12:03
*** nicolasbock has joined #openstack-nova12:03
*** brinzhang_ has joined #openstack-nova12:03
*** brinzhang_ has quit IRC12:05
*** brinzhang_ has joined #openstack-nova12:05
*** brinzhang_ has quit IRC12:06
*** brinzhang_ has joined #openstack-nova12:07
*** amoralej is now known as amoralej|lunch12:08
*** udesale_ has joined #openstack-nova12:19
*** udesale_ has quit IRC12:21
*** udesale_ has joined #openstack-nova12:21
*** udesale has quit IRC12:22
*** iurygregory has quit IRC12:32
*** mgariepy has joined #openstack-nova12:32
*** cz3 is now known as papiez12:33
*** papiez is now known as cz312:33
*** iurygregory has joined #openstack-nova12:36
*** ociuhandu has joined #openstack-nova12:38
*** mgariepy has quit IRC12:38
*** kozhukalov has joined #openstack-nova12:39
*** ociuhandu has quit IRC12:40
*** eharney has quit IRC12:45
*** ociuhandu has joined #openstack-nova12:48
*** ratailor has quit IRC12:50
*** mgariepy has joined #openstack-nova12:52
*** jangutter has quit IRC12:56
*** jangutter has joined #openstack-nova12:56
*** tbachman has joined #openstack-nova12:57
*** brinzhang_ has quit IRC12:59
*** brinzhang_ has joined #openstack-nova12:59
*** sean-k-mooney has joined #openstack-nova13:00
*** brinzhang_ has quit IRC13:00
*** brinzhang_ has joined #openstack-nova13:01
*** jangutter has quit IRC13:02
*** brinzhang_ has quit IRC13:02
*** brinzhang_ has joined #openstack-nova13:03
*** nicolasbock has quit IRC13:03
*** nicolasbock has joined #openstack-nova13:04
*** udesale_ has quit IRC13:05
*** udesale_ has joined #openstack-nova13:05
*** tbachman has quit IRC13:07
*** ociuhandu has quit IRC13:17
*** ociuhandu has joined #openstack-nova13:18
*** amoralej|lunch is now known as amoralej13:18
*** nicolasbock has quit IRC13:21
*** ociuhandu has quit IRC13:23
*** rpittau|bbl is now known as rpittau13:24
*** nicolasbock has joined #openstack-nova13:26
*** ociuhandu has joined #openstack-nova13:26
*** jangutter has joined #openstack-nova13:31
*** lpetrut has joined #openstack-nova13:36
*** waverider has joined #openstack-nova13:44
*** iurygregory has quit IRC13:46
*** brinzhang has joined #openstack-nova13:53
*** brinzhang has quit IRC13:54
*** brinzhang has joined #openstack-nova13:54
*** nweinber has joined #openstack-nova13:55
*** eharney has joined #openstack-nova13:56
*** brinzhang has quit IRC13:56
*** brinzhang has joined #openstack-nova13:57
*** brinzhang has quit IRC13:58
*** brinzhang has joined #openstack-nova13:59
*** lbragstad has joined #openstack-nova13:59
*** brinzhang has quit IRC14:00
*** jawad_axd has quit IRC14:04
*** tbachman has joined #openstack-nova14:05
*** zhanglong has quit IRC14:05
openstackgerritStephen Finucane proposed openstack/nova stable/train: Recalculate 'RequestSpec.numa_topology' on resize  https://review.opendev.org/70834014:10
stephenfinlyarwood: I rebased that onto one of your patches to avoid a merge conflict. Any change you could re +2? ^14:11
* lyarwood looks14:11
lyarwoodstephenfin: done, elod ^ if you have a chance btw.14:14
*** mgariepy has quit IRC14:14
openstackgerritStephen Finucane proposed openstack/os-traits master: setup.cfg: Configure 'python-requires'  https://review.opendev.org/70453414:25
openstackgerritStephen Finucane proposed openstack/os-traits master: Remove use of six  https://review.opendev.org/71027414:25
openstackgerritStephen Finucane proposed openstack/os-traits master: Switch to hacking 2.x  https://review.opendev.org/71027514:25
openstackgerritStephen Finucane proposed openstack/os-resource-classes master: Switch to hacking 2.x  https://review.opendev.org/71027614:25
stephenfinbauzas, gibi, efried: Some dead simple python27 cleanup patches there, if you want the stats ^14:26
bauzasheh14:26
* bauzas clicks14:26
efriedsean-k-mooney: did you see my comments yesterday about ksa->sdk conversion?14:29
*** yedongcan has left #openstack-nova14:32
lyarwoodhttps://review.opendev.org/#/c/708745/ - For any non-RH cores, this is a simple partial fix for an issue introduced recently in Libvirt v6.0.0.14:33
sean-k-mooneyefried: no14:34
sean-k-mooneyi was logged off ill look at irc logs unless you want to summerise14:34
efriedsean-k-mooney: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2020-02-26.log.html#t2020-02-26T23:11:2314:34
efriednot too much to summarize14:35
sean-k-mooneyah i see14:35
efriedI don't like the idea of un-singleton-ing, which would mean we do the whole Adapter setup with every API call, versus putting in something that will keep the token alive for us. If that thing is converting to sdk, win/win.14:36
sean-k-mooneyso ya my theory is that by making it a singolton in the current patch we are not refrshing the token14:36
sean-k-mooneyso the fact we were always createing the adapters in other case was masking that14:36
efriedyup14:36
sean-k-mooneyso unless we do the swich i think the singolton is wrong or at least we would need to add an expiry/token check to it if we kept it14:37
sean-k-mooneyi have been using https://review.opendev.org/#/c/679914/14:38
sean-k-mooneyin my testing14:38
sean-k-mooneyoh which is now merged cool14:38
sean-k-mooneyso that seams to work fine14:38
sean-k-mooneyif that has all the feature we need then ya using the sdk i think make sense if it will handel the tokens for us14:39
sean-k-mooneyefried: do you know if an update to the series is in the works14:44
efriedI don't know, no.14:44
sean-k-mooneyok14:44
efriedIf there's now real sdk impl (rather than just REST primitive passthrough) that we could switch to, so much the better, though I would argue that should be done later, since it doesn't involve a problem we need to solve immediately.14:45
sean-k-mooneyin that case i might just put a testing patch on the top of the chain that swaps the function calls and runs my multi node job to see if it fixes it14:45
sean-k-mooneyya i was thinkig of just doint the s/get_ksa_adapter/get_sdk_adapter/14:46
efriedyeah14:46
efriedworth a try14:46
sean-k-mooneyand see if that works14:46
efriedI don't remember for sure whether that's all it takes...14:46
*** nweinber has quit IRC14:47
efriedbut easy enough to sniff out.14:47
sean-k-mooneyif we are basically doing raw request i think it is14:47
efriedIf you have a live env handy it might make sense to try it out to winnow out any easy problems.14:47
sean-k-mooneyif we wanted to use the new proxy function that would be more involed14:47
efriedyeah, it wouldn't be wrt the requests, it would be with the setup.14:47
*** nweinber has joined #openstack-nova14:48
sean-k-mooneydid you see i updated https://review.opendev.org/#/c/673341/ yesterday by the way14:49
sean-k-mooneyim going to try and do the rest of the cain later today14:49
efriedsean-k-mooney: yeah, I was going to wait until all updated to start reviewing, unless you think I should look sooner.14:51
sean-k-mooneyam whatever works for you. most of the comments were gibis so i think he is the one that is more pressing14:51
sean-k-mooneythat said waiting till they are all done is proably for the best14:52
sean-k-mooneythat way you will have less context switching14:52
efriedmordred: I think there's not too much context to be missed here. In proposed patch series, nova is talking to cyborg using a raw ksa adapter. I insisted we make that client a singleton, and we're seeing tokens expire after a while (I guess ksa doesn't take care of keeping them fresh). We want to cut over to using sdk anyway, so if sdk handles expiration under the covers, two birds.14:53
sean-k-mooneyso ya im just going to grab lunch. when i get back ill quickly do the test patch for the cyborg series and then ill start on the provider.yaml patches14:53
efried++14:53
*** mgariepy has joined #openstack-nova14:55
*** udesale_ has quit IRC14:57
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Always provide the size in bytes when calling virDomainBlockResize  https://review.opendev.org/70759014:57
openstackgerritStephen Finucane proposed openstack/nova master: images: Remove Libvirt specific configurable use from qemu_img_info  https://review.opendev.org/70759114:58
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_MULTIATTACH  https://review.opendev.org/71023815:03
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Remove QEMU_VERSION_REQ_SHARED  https://review.opendev.org/71023915:03
openstackgerritStephen Finucane proposed openstack/os-traits master: setup.cfg: Configure 'python-requires'  https://review.opendev.org/70453415:05
openstackgerritStephen Finucane proposed openstack/os-traits master: setup.cfg: Remove unused 'build_sphinx' configuration  https://review.opendev.org/70453515:05
openstackgerritStephen Finucane proposed openstack/os-traits master: Remove use of six  https://review.opendev.org/71027415:05
openstackgerritStephen Finucane proposed openstack/os-traits master: Switch to hacking 2.x  https://review.opendev.org/71027515:05
stephenfinlyarwood: I've a question left at https://review.opendev.org/#/c/706898/ btw15:08
*** ociuhandu has quit IRC15:08
stephenfinregarding how we're going to backport this fix if depends on a new oslo.utils feature15:08
stephenfinnot sure if you've thought through that yet or no15:09
*** ileixe has quit IRC15:10
*** ileixe has joined #openstack-nova15:11
mordredefried: ok. that's something I think we should still look in to ... ksa should be auto-renewing tokens ... but sdk should _certainly_ be doing it15:12
mordredefried: and I'd consider it a bug in sdk if it's not - even if it's something we should be getting ksa to do for us - so that sounds like a good plan15:13
efriedack, thx15:13
mordredefried: are these tokens created from credentials nova has? or tokens received from the user that nova is re-using?15:13
*** ociuhandu has joined #openstack-nova15:14
efriedmordred: from conf, via our soon-to-be-famous oslo.config loader15:14
efriedum, I think15:14
efriedNow I can't remember whether we decided to do cyborg-ness as user or admin...15:14
efried...user15:15
efriedmordred: so you can conf the session & adapter opts, but the auth should be the nova user's.15:15
efriedWhich means it makes even less sense that the token is expiring.15:15
*** ileixe has quit IRC15:16
lyarwoodstephenfin: sorry just back from a call15:16
lyarwoodstephenfin: yeah I was talking of backporting this more in a downstream context where we control everything within a given container15:17
lyarwoodstephenfin: and then only to stable/train15:17
lyarwoodstephenfin: for upstream, what do you mean in terms of vendoring?15:17
stephenfinI mean copying the newly updated contents of 'oslo_utils.imageutils' into the nova tree15:20
*** alex_xu has quit IRC15:20
*** gyee has joined #openstack-nova15:21
openstackgerritGEET JAIN proposed openstack/nova master: Fix bug where aggregate api works with invalid id  https://review.opendev.org/71029015:21
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Remove QEMU_VERSION_REQ_SHARED  https://review.opendev.org/71023915:22
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_MULTIATTACH  https://review.opendev.org/71023815:22
lyarwoodstephenfin: ah wait, so in the context of the bugfix I just need to bump requirements in https://review.opendev.org/#/c/706900/15:22
*** Sundar has joined #openstack-nova15:23
stephenfinFor the master patch, yup15:24
stephenfinAssuming a new oslo.utils patch has been cut since15:24
lyarwoodhttps://review.opendev.org/#/c/710078/ - cut but not released15:25
mordredefried: yeah - it should *really* be renewing15:29
openstackgerritLee Yarwood proposed openstack/nova master: workarounds: Add option to disable native LUKSv1 decryption by QEMU  https://review.opendev.org/70803015:30
openstackgerritLee Yarwood proposed openstack/nova master: workarounds: Connect RBD volumes to the compute host as block devices  https://review.opendev.org/70802915:30
sean-k-mooneyefried: we should be useing the token form the api request not form the config15:31
sean-k-mooneywe are not adding setting up the service user in the config and we are not adding a cyborg section to the nova.conf15:31
sean-k-mooneyso i dont think the issue is with renewing the token in the nova code15:32
sean-k-mooneyi think with the singelton we create the client onece capturing the context and token at that point15:32
sean-k-mooneyand then it never gets updated again15:32
sean-k-mooneymordred: ^ that is also relevent to you i guess15:34
mordredyeah. that - if we're using the api request token, then when we get the singleton sdk connection object, we need to call conn.connect_as on it to get a new connection with the credentials we've been given, otherwise we're making all of the requests with the creds of the original user :)15:37
sean-k-mooneyya we are not doing that15:38
sean-k-mooneyso that is likely the bug15:38
mordredconn.connect_as lets us keep discovery cache appropriately and all of that - but will construct a new underlying Auth construct and a new Adapter15:38
mordredsean-k-mooney: computers are hard15:38
sean-k-mooneymordred: we would do that here right https://review.opendev.org/#/c/631242/51/nova/accelerator/cyborg.py@32 just before the retrun?15:39
sean-k-mooneyalthough im not sure that would be safe15:39
sean-k-mooneywe could context switch at any point so we would have to take the lock15:39
sean-k-mooneyand do it before making any call15:39
mordredsean-k-mooney: sdk is threadsafe15:41
mordredyou shouldn't need a lock15:41
sean-k-mooneyis it eventlet safe15:42
mordred_should_ be15:42
mordredif it's not it's a bug15:42
sean-k-mooneyif the con.connect_as returns a new object that we then make new calls on it should work15:42
mordredI mean - but that said - with eventlet who the hell knows15:42
mordredsean-k-mooney: yes - connect_as returns a new object15:42
mordredit returns a whole new Connection with all the settings of the old one but with just the auth bits replaced15:43
sean-k-mooneyand that object basicaly poxies the requeqes on it to the adapter it was created form but with the update creds15:43
mordredit has to create a new adapter - because an adapter is a container of session+auth+endpoint-config - you can't use the old adapter if you have new auth, you need a new adapter15:44
mordredbut it does keep/share the things that are safe to keep/share15:44
sean-k-mooneyis so then we just need to change "return _CYBORG_CLIENT" to "return _CYBORG_CLIENT.connect_as(context)15:44
mordredyeah15:44
mordredwith a little hand wavey, because we'll want to make sure we're doing the right things with context ... but pseudo-code-wide yes15:45
sean-k-mooneyok so ill swap to the sdk adapter and add the connect_as15:45
sean-k-mooneymordred: ya, this isnt my code and i havent realy review it im just testing it but i can relyably repoduce that the current code is borked so tring to fine the minimal correct fix and test that in a follow up patch15:46
sean-k-mooneyif it fixes the issue then we can squash it in and move on to the next thing15:47
mordred++15:47
mordredlet me know if you have issues and want me to dive in further15:48
mordredI'm sure I could find some lovely new ways to curse15:48
*** jangutter has quit IRC15:53
*** jangutter has joined #openstack-nova15:53
*** maciejjozefczyk has quit IRC15:54
efriedIf we have to build a new adapter anyway, then I suppose we might as well just do as dansmith suggests and un-singleton for now. We'll want to do the SDK cutover later, but there's no way we'll forget to do that.15:56
dansmith++15:56
sean-k-mooneyim not sure if the connect_as is cheaper then building the adapter from scratch but ya i just want to see this working and then iterate on it later to not loose the momentem we have while people are around to ask questions15:57
sean-k-mooney*answer15:58
openstackgerritStephen Finucane proposed openstack/os-resource-classes master: Switch to hacking 2.x  https://review.opendev.org/71027615:58
*** priteau has joined #openstack-nova16:00
*** factor has quit IRC16:08
*** factor has joined #openstack-nova16:08
*** tbachman has quit IRC16:08
*** ccamacho has quit IRC16:08
*** factor has quit IRC16:11
*** factor has joined #openstack-nova16:13
*** links has quit IRC16:14
*** ociuhandu_ has joined #openstack-nova16:21
*** factor has quit IRC16:22
*** factor has joined #openstack-nova16:22
*** factor has quit IRC16:23
*** ociuhandu has quit IRC16:25
*** ociuhandu_ has quit IRC16:26
*** tesseract has quit IRC16:27
*** canori01 has quit IRC16:33
openstackgerritMerged openstack/os-traits master: setup.cfg: Configure 'python-requires'  https://review.opendev.org/70453416:34
openstackgerritMerged openstack/os-traits master: setup.cfg: Remove unused 'build_sphinx' configuration  https://review.opendev.org/70453516:34
*** damien_r has quit IRC16:39
*** waverider has quit IRC16:40
*** lpetrut has quit IRC16:40
gibisean-k-mooney: re: provider config. I have to take tomorrow off so I will look at your update earliest on Monday16:40
*** mriedem has joined #openstack-nova16:40
*** ileixe has joined #openstack-nova16:43
*** damien_r has joined #openstack-nova16:44
*** dosaboy has quit IRC16:45
Sundardansmith, gibi, sean-k-mooney, efried: On the rescheduling front, I don't have a multi-node env to test it. It is not going to be easy for me to get the lab resources, even without FPGAs, to set that up. Plus, I have other things I need to focus on. Any ideas on bringing this to a timely closure?16:48
dansmithSundar: functional tests?16:48
efriedSundar: Also, sean-k-mooney set up a multinode CI job.16:48
dansmithmultinode ci might be hard to trigger a reschedule though16:49
dansmithSundar: tangential to this part, did you see my comment about moving the event to the front of the queue, below the -2'd ksa patch?16:49
Sundardansmith: Yes, I saw that. I have moved it before the ksa patch in my local env. The next commit should have it.16:50
dansmithSundar: cool16:50
Sundardansmith: With functional tests, I can check if the logs show rescheduling messages, and that the tests pass. Is there anything more you are looking for?16:51
efriedYou can do better than that to prove a reschedule occurred: use the fixture that makes the initial host predictable and prove that it lands on a different one.16:51
gibiefried ++16:52
dansmithSundar: "tests passing" isn't a bar to meet, "tests validating something useful" is16:52
efriedthere should be some good examples16:52
dansmithyep, what efried said16:52
Sundarefried: Thanks, will look for that. If you happen to remember the(rough) name of the fixture, off the top of your head, I'll start with that.16:52
dansmithSundar: what specifically are you stuck on? there's lots of feedback to work on without much response from you yet that I can see.. is there something specific that you're blocked on?16:53
*** ccamacho has joined #openstack-nova16:53
Sundardansmith: Not sure that any of my changes will actually work in the real world, without a multinode env. I tried to get a 2nd server but it will take time, if at all.16:53
efriedSundar: HostNameWeigherFixture I think16:53
gibiSundar: HostNameWeigherFixture makes the host selection peridctable16:53
dansmithSundar: functional tests man :)16:54
Sundarefried, gibi: Thanks, got it16:54
*** jmlowe has quit IRC16:54
dansmithSundar: if you have reasonable functional tests that don't mock out insane levels of things and validate reasonable conditions, then any real world failure we find later is understandable16:54
dansmithSundar: test_servers.py has a bunch of reschedule-specific tests16:55
Sundardansmith: Ok, let's just focus on functional tests, then. BTW, https://review.opendev.org/gitweb?p=openstack/nova.git;f=nova/conductor/manager.py;hb=refs/changes/44/631244/63#l602 is specific to the resheduling path AFAICS. is there any other path, like rebuild/resize/whatever, that invokes this?17:00
dansmithSundar: only if one of those things needs a reschedule, AFAIK17:01
*** dosaboy has joined #openstack-nova17:01
SundarOk. BTW, I will be travelling over the next couple of days. I will respond with the comments I accepted by today, and will post the rest in a few days.17:04
dansmithokay17:04
*** tbachman has joined #openstack-nova17:05
Sundardansmith: In https://review.opendev.org/#/c/631244/63/nova/conductor/manager.py@848, is there any resource to be cleaned up in the exception path?17:05
dansmithSundar: seems like there's network and block devices to clean up, but I'm not sure why the error paths above aren't doing that17:07
dansmithSundar: should you be deleting arqs?17:07
dansmithor are you assuming those get deleted when the error'd instance gets deleted?17:07
dansmithSundar: there is a _cleanup_after_reschedule_fails() method in there...17:08
sean-k-mooneygibi: ack that is fine. i likely will not be ready until then anyway17:08
dansmither, cleanup_when_reschedule_fails17:09
dansmithwhich cleans up allocated network resources at least17:09
SundarWell, not in this patch for sure, because delete ARQ functionality comes later. However, we should do that in the delete patch because Create ARQs may succeed but bind may throw an exception potentially. Then, Cyborg should be informed to delete the created ARQs.17:09
dansmithSundar: okay your existing code won't even leave the instance in error state, which is bad.. but that cleanup method will do that for you17:09
Sundardansmith: Yes, I see it. Good. Thanks.17:10
*** ileixe has quit IRC17:11
Sundardansmith: I could tie the delete ARQs to _cleanup_when_reschedule_fails(). I'll check to see what other impact it has.17:11
dansmithSundar: if deleting the instance in un-scheduled state will clean those up, then it's probably okay to punt to that.. the only problem might be if the ARQs are holding some resource for the dead instance17:12
Sundardansmith: If the bind succeeds, then ARQs are holding real resources. So, if some other task fails during rescheduling, then Cyborg should be told to delete the ARQs. I think the changes we already have should take care of that.17:14
*** tbachman has quit IRC17:16
dansmithSundar: right but the bind is right before the call back to compute, so if that fails, no bind was done, and if it has failed before that, we're already unbound from the original host, so not holding resource... is that right?17:17
Sundardansmith: bind is async. The call here just kicks off the process, and the actual bind may succeed or fail after that. If the bind fails, either Cyborg sends a failure notification or Nova gets a timeout (in corner cases). That is already handled. if the bind succeeds, Cyborg has allocated and prepared resources. But something else in Nova may17:21
Sundarfail. It is this second path that I am not terribly sure about.17:21
dansmithI think you're missing my point17:21
dansmithor maybe you're talking about the reschedule racing with the *original* bind?17:21
*** rpittau is now known as rpittau|afk17:23
Sundardansmith: I am talking of the bind during the rescheduling in https://review.opendev.org/#/c/631244/63/nova/conductor/manager.py@842 . Yes, if there is a failure in _create_and_bind_arqs( ) or earlier, no resources are allocated.17:24
*** bbowen has quit IRC17:25
dansmithSundar: so you're concerned about what happens if the bind during reschedule fails and not knowing whether things are actually bound or not?17:25
dansmithif so that's... pretty scary17:25
SundarI was talking of the scenario where _create_and_bind_arqs() succeeds, Nova goes ahead with the reschedule, CYborg prepares devices in the new host, and then things go south.17:25
SundarThings go south in Nova17:26
SundarSo, every failure path in reschedule must call Cyborg to delete ARQs17:26
dansmithdepends on where it goes south, but there's not much we can do about it17:26
dansmithtry to delete the arqs when we can in error paths,17:26
*** larainema has quit IRC17:26
dansmithbut as long as the instance delete will clean things up gracefully, that's the safety net17:27
dansmithafter your bind in the reschedule path, the rest of it is identical to the regular build process17:27
SundarI am trying to see if I am missing any error paths. AFAICS, we haven't missed any in the conductor. In the compute manager, if the reschedule takes the same code paths as the priginal schedule, we should be good.17:27
dansmithright17:27
SundarOk, then we are good17:28
dansmithif your bind in reschedule in conductor succeeds, then it's a regular build after that17:28
*** jangutter has quit IRC17:29
*** jangutter_ has joined #openstack-nova17:29
*** maciejjozefczyk has joined #openstack-nova17:31
*** evrardjp has quit IRC17:35
*** LiangFang has quit IRC17:35
*** evrardjp has joined #openstack-nova17:35
*** maysams has quit IRC17:35
*** Liang__ has joined #openstack-nova17:35
*** maciejjozefczyk has quit IRC17:39
*** igordc has joined #openstack-nova17:39
*** ileixe has joined #openstack-nova17:39
*** maciejjozefczyk has joined #openstack-nova17:39
*** priteau has quit IRC17:40
openstackgerritMerged openstack/os-traits master: Remove use of six  https://review.opendev.org/71027417:45
melwittmgoddard: do you know if there's any workaround for this bug you're working on? https://bugs.launchpad.net/nova/+bug/185300917:56
openstackLaunchpad bug 1853009 in OpenStack Compute (nova) "Ironic node rebalance race can lead to missing compute nodes in DB" [Undecided,In progress] - Assigned to Mark Goddard (mgoddard)17:56
*** mriedem has left #openstack-nova17:57
mgoddardmelwitt: don't restart nova-compute? :/17:59
mgoddardwell that's more how not to hit it17:59
mgoddardlet me check my notes18:00
mgoddardmelwitt: also when you say I'm working on it, mostly I'm just waiting for reviews18:00
*** jmlowe has joined #openstack-nova18:01
mgoddardNothing of much help in my notes18:01
*** igordc has quit IRC18:01
mgoddardRestarting nova-compute one by one might help, but it might also create new problems18:02
mgoddardI think the approach we took was to apply those patches (backported to rocky)18:02
mgoddardrather than workaround18:02
melwittmgoddard: gotcha ... thanks. I was looking through the lp bug trying to figure whether there's a workaround and wondered if you might already know18:03
*** igordc has joined #openstack-nova18:04
mgoddardsometimes you can do some DB hacking to change the compute node IDs. I can't remember if it works in this case18:04
*** jmlowe has quit IRC18:05
*** jmlowe has joined #openstack-nova18:05
melwittmgoddard: thanks. definitely will review your patches, I was trying to work out whether there's any workaround possible to stop the bleeding in the immediate term while the patches get reviewed18:08
sean-k-mooneymgoddard: we have a customer that is seeing similar behavior when they do a ooo scale out.18:09
mgoddardif it's a problem there might be something that could be worked out. It should be possible to reproduce with a multinode control plane and a few ironic nodes (possibly fake)18:10
mgoddardkeep restarting nova computes, and eventually the DB and placement will go astray18:10
mgoddardanyways, need to head off now18:10
mgoddardgood luck!18:10
melwittthanks mgoddard18:11
*** macz_ has joined #openstack-nova18:12
*** damien_r has quit IRC18:14
*** macz_ has quit IRC18:14
*** amoralej is now known as amoralej|off18:14
*** macz_ has joined #openstack-nova18:15
*** jmlowe has quit IRC18:27
*** jangutter_ has quit IRC18:27
*** jmlowe has joined #openstack-nova18:30
*** jmlowe has quit IRC18:34
*** jmlowe has joined #openstack-nova18:35
*** jmlowe has quit IRC18:36
*** mriedem has joined #openstack-nova18:45
*** jangutter has joined #openstack-nova18:48
*** jangutter has quit IRC18:50
*** tbachman has joined #openstack-nova18:54
CeeMacWhat would cause a block device mapping failure when deploying a new instance? I see the volume creating fine in cinder and on the backend iscsi storage (zadara) but there seems to be something preventing the volume mounting to the vm. Just trti g to piece the interconnecting pieces together ready for another round of troubleshooting tomorrow18:56
*** jmlowe has joined #openstack-nova18:56
CeeMacThis is on a new compute/storage node I added today. I've been through the configs again and nothing obvious jumps out at me as being missed. Running rocky btw.18:57
*** jmlowe has quit IRC18:59
melwittCeeMac: there's likely error logging about it in nova-compute logs19:03
*** sean-k-mooney has quit IRC19:05
CeeMacmelwitt: yeah, I couldn't see anything specific beyond it timing out after x seconds etc19:08
*** maciejjozefczyk has quit IRC19:08
CeeMacI'll take another run at it and double check the logs in the morning19:08
*** ralonsoh has quit IRC19:08
melwittCeeMac: ok. I'd turn on debug logging (debug=True in nova.conf for nova-compute) and find which command times out. I'm guessing it will be something os-brick is trying to do. and then go from there19:09
CeeMacThe other weird thing I noticed was when I tried a live migration os-brick wasn't able to see the volume as there were no active iscsi sessions. Except I can see the server connecting to the storage and the volume getting created on the back end.19:10
CeeMacDoes os-brick log somewhere specific?19:10
melwittit logs in nova-compute but only at level DEBUG19:10
CeeMacRight19:11
CeeMacI'll check that first then, thanks19:11
melwittyeah, once you find what command is timing out or returning something unexpected, you can start trying to piece it together with things that might be happening with the backend19:11
-openstackstatus- NOTICE: Memory pressure on zuul.opendev.org is causing connection timeouts resulting in POST_FAILURE and RETRY_LIMIT results for some jobs since around 06:00 UTC today; we will be restarting the scheduler shortly to relieve the problem, and will follow up with another notice once running changes are reenqueued.19:11
CeeMacmelwitt: I'm hoping it'll be something obvious, it's a while since I last added a host and I've not made any changes to the config files that I can recall19:14
melwittI hope so too19:17
*** jmlowe has joined #openstack-nova19:31
*** eharney has quit IRC19:43
-openstackstatus- NOTICE: The scheduler for zuul.opendev.org has been restarted; any changes which were in queues at the time of the restart have been reenqueued automatically, but any changes whose jobs failed with a RETRY_LIMIT, POST_FAILURE or NODE_FAILURE build result in the past 14 hours should be manually rechecked for fresh results19:45
*** maciejjozefczyk has joined #openstack-nova19:45
*** ociuhandu has joined #openstack-nova19:48
*** gyee has quit IRC19:49
*** gyee has joined #openstack-nova19:49
openstackgerritMerged openstack/os-traits master: Switch to hacking 2.x  https://review.opendev.org/71027519:52
*** maciejjozefczyk has quit IRC19:57
*** maciejjozefczyk has joined #openstack-nova19:58
*** jmlowe has quit IRC20:07
*** nicolasbock has quit IRC20:07
*** kozhukalov has quit IRC20:27
*** martinkennelly has joined #openstack-nova20:28
*** kozhukalov has joined #openstack-nova20:28
*** jmlowe has joined #openstack-nova20:30
*** Sundar has quit IRC20:35
*** mlavalle has joined #openstack-nova20:40
efriedNova meeting in ~15 minutes in #openstack-meeting20:45
*** ociuhandu has quit IRC20:53
*** kozhukalov has quit IRC21:08
*** tbachman has quit IRC21:11
*** slaweq has quit IRC21:15
*** david_4 has joined #openstack-nova21:23
*** aarents has quit IRC21:23
david_4Hi, I am new to Nova, I wanted add a scheduler filter and want to know how to test filters in nova-scheduler. I have an openstack env running with 3 head nodes and 3 work node, I am not using devstack.21:25
david_4any help please !!21:34
*** kozhukalov has joined #openstack-nova21:40
*** dpawlik has quit IRC21:43
*** rcernin has joined #openstack-nova21:44
*** martinkennelly has quit IRC21:46
*** xek__ has quit IRC21:53
*** tbachman has joined #openstack-nova22:09
*** slaweq has joined #openstack-nova22:11
*** david_4 has quit IRC22:11
*** slaweq has quit IRC22:15
*** slaweq has joined #openstack-nova22:18
*** tbachman has quit IRC22:19
*** N3l1x has joined #openstack-nova22:20
*** maciejjozefczyk has quit IRC22:20
*** slaweq has quit IRC22:22
*** eharney has joined #openstack-nova22:29
*** jawad_axd has joined #openstack-nova22:38
openstackgerritGhanshyam Mann proposed openstack/nova master: Fix H702 pep8 error with latest hacking  https://review.opendev.org/71034422:40
*** jawad_axd has quit IRC22:43
efriedgmann: Is that the only thing you want in that patch? ^22:45
efriedif so, I'll fast approve22:45
*** tkajinam has joined #openstack-nova22:51
*** tkajinam has quit IRC22:51
*** tkajinam has joined #openstack-nova22:51
*** ivve has quit IRC22:57
*** owalsh has quit IRC22:58
mriedemif david_4 ever comes back, point him at something like https://github.com/openstack/nova/blob/master/nova/tests/functional/test_json_filter.py as an example of how to test scheduler filters in functional tests - easier than getting a real environment setup (assuming he can navigate how to run functional tests)23:01
*** alex_xu has joined #openstack-nova23:01
mriedemsmell ya later!23:02
*** mriedem has left #openstack-nova23:02
*** nweinber has quit IRC23:03
*** slaweq has joined #openstack-nova23:11
*** ociuhandu has joined #openstack-nova23:12
*** slaweq has quit IRC23:16
*** owalsh has joined #openstack-nova23:16
*** ociuhandu_ has joined #openstack-nova23:19
*** ociuhandu_ has quit IRC23:21
*** tbachman has joined #openstack-nova23:21
*** aarents has joined #openstack-nova23:21
*** ociuhandu has quit IRC23:22
gmannefried: yeah, that is only error for now.23:31
*** kozhukalov has quit IRC23:35
gmannefried: wait, few more seems. let me update.23:38
*** jawad_axd has joined #openstack-nova23:40
*** jawad_axd has quit IRC23:45
*** jamesdenton has quit IRC23:49
*** jamesdenton has joined #openstack-nova23:50

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!