Tuesday, 2023-01-31

opendevreviewMerged openstack/nova master: Make resource tracker use UUIDs instead of names  https://review.opendev.org/c/openstack/nova/+/86391900:02
*** dasm is now known as dasm|off01:44
sahido/07:54
sahidouch, merge conflicts on rpc version :/07:58
opendevreviewSahid Orentino Ferdjaoui proposed openstack/nova master: compute: enhance compute evacuate instance to support target state  https://review.opendev.org/c/openstack/nova/+/85838307:59
opendevreviewSahid Orentino Ferdjaoui proposed openstack/nova master: api: extend evacuate instance to support target state  https://review.opendev.org/c/openstack/nova/+/85838407:59
opendevreviewSahid Orentino Ferdjaoui proposed openstack/nova master: compute: enhance compute evacuate instance to support target state  https://review.opendev.org/c/openstack/nova/+/85838311:51
opendevreviewSahid Orentino Ferdjaoui proposed openstack/nova master: api: extend evacuate instance to support target state  https://review.opendev.org/c/openstack/nova/+/85838411:51
*** tkajinam is now known as Guest300312:02
opendevreviewRajesh Tailor proposed openstack/nova master: Fix concurrent migration of vms with multiattach volume failure  https://review.opendev.org/c/openstack/nova/+/87227812:40
opendevreviewRajesh Tailor proposed openstack/nova master: Fix concurrent migration of vms with multiattach volume failure  https://review.opendev.org/c/openstack/nova/+/87227813:17
sahidbauzas, sean-k-mooney can I request reviews on the evacuate api feature, i'ts all green. I had to rebase and fix conflicts this morning du to a rpc change14:09
sahidnow i think we are good, i hope :-)14:10
sean-k-mooneyya i was waiting for ci to report back before pining bauzas 14:26
sean-k-mooneyim +2 on the first and +1.5 on the second pending bauzas's review14:26
sean-k-mooneyif they are happy then ill escalate to +2w14:26
*** dasm|off is now known as dasm14:28
opendevreviewAlexey Stupnikov proposed openstack/nova stable/ussuri: [stable-only] Disable nova-grenade-multinode voting  https://review.opendev.org/c/openstack/nova/+/87228114:42
opendevreviewAlexey Stupnikov proposed openstack/nova stable/ussuri: reenable greendns in nova.  https://review.opendev.org/c/openstack/nova/+/83343714:43
bauzassean-k-mooney: sahid: I'll look again when I have time today14:53
sahidack ++ thank you bauzas 14:58
opendevreviewDan Smith proposed openstack/nova master: Protect against a deleted node id file  https://review.opendev.org/c/openstack/nova/+/87220415:23
elodillesbauzas: are you editing the Nova meeting page or may I edit it?15:37
bauzaselodilles: do it15:38
bauzaselodilles: I'm doing 4 things at the same time15:38
elodillesbauzas: :-o15:38
bauzasmulti-tasking 15:38
bauzasbut man, this is hard15:38
elodillesbauzas: i can imagine :S i don't like context switching :/15:39
elodillesbauzas: done. you can also edit now (if you want to do a 5th thing :S)15:45
bauzaselodilles: yup, will do indeed15:50
bauzas#startmeeting nova16:00
opendevmeetMeeting started Tue Jan 31 16:00:27 2023 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
opendevmeetThe meeting name has been set to 'nova'16:00
bauzashey ho16:00
gmanno/16:00
elodilleso/16:00
Ugglao/16:01
dansmitho/16:01
gibio716:01
bauzasokay let's start16:01
bauzas#topic Bugs (stuck/critical) 16:01
bauzas#info No Critical bug16:01
bauzas#link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 27 new untriaged bugs (+1 since the last meeting)16:02
bauzas#info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster16:02
bauzasbug baton can be passed over to artom if he wants16:02
artomSure16:02
artomI feel like I've skipped.... many weeks16:02
artom(unintentionally)16:02
bauzasthanks16:03
bauzas#info bug baton is being passed to artom16:03
bauzasartom: again, this is just a temptative16:03
bauzasdon't feel that much obliged16:03
artomWell don't say that16:04
bauzasok, so, before we move on, do people want to discuss about some , per say, CVE bugfix or everything is OK since we're on track ?16:04
bauzaswe know we need to progress with wallaby16:04
bauzasbut I wanted to know whether people wanted to discuss it by now ?16:05
bauzascrickets ? ok, so, let's move on16:06
bauzas#topic Gate status 16:06
bauzas#link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs 16:06
bauzas#link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status16:06
bauzasgate is still wedgy, but tbh, I had to focus on other priorities last week and this week :(16:07
gibisame here, I made no progress on the functional test instability16:07
bauzasfrom what I see, the gate is quite difficult but still mergeable16:07
dansmithit's better16:07
dansmithI've been unable to reproduce any of the problems locally, unfortunately16:07
bauzason the functest sage ?16:07
bauzassaga* 16:08
bauzasdansmith: that's the problem, I tried too16:08
bauzasdansmith: running the exact same way than the gate using the testr subunits16:08
bauzasbut got nothing16:08
bauzashence me having pivoted, and tbh, there were people afraid of some security bug last week that diverted my attention16:09
bauzasbut ok, if things are better, that's cool16:09
bauzaseither way, I'll try to go looking at it back when I'm done with my own prios (including some feature implementation I promised)16:10
bauzas#info Please look at the gate failures and file a bug report with the gate-failure tag.16:10
bauzas#info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures16:10
bauzasok, let's move on thenb16:11
bauzas#topic Release Planning 16:11
bauzas#link https://releases.openstack.org/antelope/schedule.html16:11
bauzas#info Antelope-3 is in 2 weeks16:11
bauzas /o\16:11
bauzas#link https://etherpad.opendev.org/p/nova-antelope-blueprint-status Blueprint status for 2023.116:12
bauzasso I created a tracking etherpad16:12
bauzasmost of the blueprints are in the 'unknown' territory16:12
bauzasI need to investigate the crime scene and see whether the owner created some changes16:12
bauzasso, use that etherpad as you want16:13
bauzasideally, blueprint owners are more than welcome to amend the etherpad with their comments, including patches to review 16:13
bauzasand cores are welcome to add comments on things they want to review or already did16:14
bauzasI tried to add a 'API Impact' bullet on the blueprints that were actually touching the API, not exclusively by a microversion16:14
bauzasok, if anything else, moving on ?16:15
bauzasif not* 16:15
sean-k-mooneysure16:16
sean-k-mooneylets proceed16:16
bauzas#topic Review priorities 16:17
bauzas#link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2)16:17
bauzas#info As a reminder, cores eager to review changes can +1 to indicate their interest, +2 for committing to the review16:17
bauzasnothing to mention here16:18
bauzasmoving on16:18
bauzas#topic Stable Branches 16:18
bauzaspassing the mic to elodilles16:18
elodilles#info stable releases from last week: nova 26.1.0 (zed); nova 25.1.0 (yoga) nova 24.2.0 (xena) -- with CVE fix16:18
elodilles#info stable branches seem to be unblocked / OK back till xena16:18
elodilles#info wallaby branch is blocked by tempest (thanks gmann for working on it - https://lists.openstack.org/pipermail/openstack-discuss/2023-January/031941.html )16:19
elodilles#info ussuri and train gates are broken broken ('Could not build wheels for bcrypt, cryptography' errors)16:19
gmannstable/wallaby fix is not merged yet #link https://review.opendev.org/c/openstack/devstack/+/87178216:19
elodillesussuri/train might work with newer pip version16:19
bauzasgmann: thanks for having worked hardly on that16:19
gmannsdk job started failing which I need to make non voting to get first fix merged16:19
bauzasgmann: what a pile of cards16:20
gmannyeah tempest pin on stable EM is always like this :)16:20
elodillesgmann: openstacksdk got released some hrs ago, isn't that fixing the gate?16:20
elodillesjust asking16:20
gmannelodilles: need to check, it is failing on <=stable/xena only16:21
elodillesack16:21
gmann#link https://zuul.openstack.org/builds?job_name=openstacksdk-functional-devstack&branch=stable%2Fxena&branch=stable%2Fwallaby&skip=016:21
elodillesanyway, last item from me:16:21
elodilles#info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci16:21
elodillesEOM16:22
bauzascool16:22
bauzaswell, not so cool, but moving on16:22
bauzas#topic Open discussion 16:22
bauzas(gmann) PyPi additional maintainers audit for Nova repo 16:22
bauzasgmann: I've seen your email16:22
gmannok16:22
bauzasand I looked at nova16:23
bauzasos-vif is impacted16:23
bauzasos-resource-classes too16:23
bauzasos-traits16:24
bauzasand that's it IIRC16:24
gmannosc-placement  also i think16:24
bauzasoh, actually placement too16:25
gmannah yes. its 'openstack-placement'16:26
bauzasgmann: can you summarize the issue and what's requested for each of the projects ?16:26
bauzasnot for me, but for others16:26
gmannsure16:26
gmannTC discussed about those additional maintainers in PyPi which can get released without OpenStack knowing about it or more additional maintainers can be added16:27
gmannwhich is nothing but security issue16:27
gmannTC decided to clean that  but wanted projects to audit their additional maintained first to check if they can be removed (if they  are inactive or agreed to) or the package maintenance can be handover to them (then retire it from openstack)16:28
gmannxstatic-font-awesome repo from horizon is one example of handing over maintenance to additional external maintainers and use it in horizon as one of the user16:28
bauzasgmann: so I amended the tracking etherpad https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup#L14116:29
gmannbut mostly packages are openstack initiated one and who setup the PyPi initially are in additional maintainers list which should be easy to clean up16:30
bauzasI'm personnally OK with removing all marked people on those16:30
bauzasthey are no longer active contributors on the projects16:30
bauzasdo people disagree with this ?16:30
bauzaspasting it 16:30
gmannhere in nova, we can check if any of those maintainers are active (in OpenStack or in PyPi maintenance) and we need to talk to them first16:30
bauzasnovaos-vif has berrange and jaypipes as maintainers.os-resource-classes has cdent as maintainerplacement has cdent as maintainerosc-placement has malor as maintaineros-traits has o-tony as maintainer16:31
dansmithbauzas: fine with me16:31
gibifine by me too16:31
bauzastbh I even don't know malor 16:31
bauzaswhich is a bit of a security risk16:31
elodilles+116:32
gmann+116:32
sean-k-mooneyi tought i got os-vif moved over to me at some point but sure16:32
bauzasgmann: OK, I'll just leave a comment in the etherpad saying that the nova community agrees on removing those names16:32
gmannbauzas: perfect. thanks 16:32
bauzassean-k-mooney: tbh, I'm OK with leaving only openstackci as the sole maitainer16:33
sean-k-mooneyya thats fine16:33
bauzasfrom a security pov, I understand the concerns16:33
bauzasand that reminds me the beam accelerator security issue 16:34
sean-k-mooneyoh it was more we need to fix somehting but it was launchpad i got them to amend16:34
bauzascan we then consider this done ?16:35
sean-k-mooneyi think so at least form our part16:35
gmannyes, that is needed from project side. adding audit result in etherpad16:36
sean-k-mooneythe maintainer need to actuly be updated16:36
bauzasI mean, are we done discussing this topic now ?16:36
bauzasif so, nothing left on the agenda16:36
bauzasexcept maybe documing the volume detach issues but since they are actually unrelated to our releases, I think I'll just drop this iteam16:37
bauzasitem16:37
bauzasso16:37
bauzasany other item to raise by now ?16:37
UgglaCan you quickly discuss about :https://review.opendev.org/c/openstack/nova/+/868089/816:38
UgglaI would like to know if we would like to backport it and to which version.16:39
* bauzas looks16:40
bauzasisn't it changing the logic ?16:41
gibibased on the impl it is backportable16:41
bauzaswait16:41
gibiit has a dependency on https://review.opendev.org/q/topic:bug%252F1628606 which is being backported16:42
bauzasthis is post_live_mig_at_dest()16:42
bauzaswhich is run on the dest16:42
bauzasmy question is, in a rolling upgrade scenario with operators moving workloads from old compute to new compute16:42
bauzaswould that break them ?16:43
gibibauzas: the bug is ther until the dest is upgraded16:43
bauzaswe say we gonna rely on https://review.opendev.org/c/openstack/nova/+/79113516:44
bauzasbut correct me if I'm wrong, this won't happen for a A to B livemig if A is old, nope ?16:44
bauzasthis chit-chat limbo dance between two hosts is confusing16:44
bauzasI never know which compute runs which codepath16:45
gibithis patch trying to fix a bug that happens on the dest after the live migration finished16:45
gibiat that point there is no return to the source host16:45
gibiAmit fixed that in this case we set the instance to point to the dest host so a hard reboot can recover the instance16:45
bauzasbut we backported Amit's patch, right?16:46
gibiyes16:46
bauzasok, so we're safe16:46
gibiUggla had a case where a very similar failure could leave the instance in Migarting state16:46
gibithis fix is putting the instance in error in this case too16:47
bauzasI see16:47
bauzasok, then to answer Uggla's question, I don't see any controversy to backport such change once we merge it16:47
gibiyepp16:47
Uggladown to ?16:48
gibithe same version as Amit's 16:48
bauzasyup16:48
gibiI think that is proposed back to train16:48
bauzaswhich is train iirc16:48
bauzasyup16:48
bauzasbut as we said, we're on hold on wallaby16:49
bauzasnothing prevents us to do the work tho16:49
bauzascan we end the meeting then ?16:50
Ugglaok 4 me.16:50
gibinothing else from me16:51
bauzasthen16:51
bauzasthanks all16:51
bauzas#endmeeting16:52
opendevmeetMeeting ended Tue Jan 31 16:52:04 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:52
opendevmeetMinutes:        https://meetings.opendev.org/meetings/nova/2023/nova.2023-01-31-16.00.html16:52
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-01-31-16.00.txt16:52
opendevmeetLog:            https://meetings.opendev.org/meetings/nova/2023/nova.2023-01-31-16.00.log.html16:52
elodillesthanks o/16:55
kashyapsean-k-mooney: Running the stable/xena locally gives me this yak to shave  - https://paste.opendev.org/show/bsRpOt3yk8k6LUw5D9P0/16:56
elodilleskashyap: maybe i'm wrong but this can be fixed by updating/installing explicitly setuptools?16:57
kashyapProbably; I'm on Fedora 3616:58
kashyapelodilles: I'm trying to see if this erroneous failure of stable/xena related to my backport or not (it doesn't seem so)16:58
kashyaphttps://zuul.opendev.org/t/openstack/build/796a6b05d72c4bbd87a3375028d43a1f16:58
kashyapThis one: nova.tests.unit.virt.libvirt.test_driver.LibvirtConnTestCase.test_check_can_live_migrate_dest_numa_lm [0.046959s] 16:58
kashyap(And that's the backport - https://review.opendev.org/c/openstack/nova/+/851205)16:58
sean-k-mooneyoh thats a known issue16:59
elodillesyes, a known one and fixed in upstream gate i think16:59
sean-k-mooneyuse_2to3 was remvoed in a setuptools verison16:59
kashyapelodilles: Ah, thx.  gibi also pointed that there's a rename, hence the fail: https://review.opendev.org/c/openstack/nova/+/87197517:00
kashyapThanks, gibi!17:00
sean-k-mooneyhttps://github.com/gibizer/openstack-tox-docker/blob/main/ussuri/Dockerfile17:00
gibiI thiunk the yoga container from here https://github.com/gibizer/openstack-tox-docker work on xena too17:00
* kashyap needs to head out shortly for a doc apptmt; be back later17:00
sean-k-mooneygibi: yes it does 17:00
sean-k-mooneykashyap: we swapped form the unmaintained suds_junko repo to a differnt one17:01
sean-k-mooneybut that is not your issue17:01
sean-k-mooneyyou will need to clamp your pip/virtualevn and tox version17:01
elodillesyepp. in upstream xena has newer versions (ubuntu) that's why we needed this up till ussuri ( https://review.opendev.org/c/openstack/nova/+/810461 )17:03
elodillesso i guess, in fedora this is needed in xena as well17:04
opendevreviewSahid Orentino Ferdjaoui proposed openstack/nova master: api: extend evacuate instance to support target state  https://review.opendev.org/c/openstack/nova/+/85838417:16
sahidaddressed minor comments from Rajesh Tailor, thank you !17:16
dansmithsean-k-mooney: gibi bauzas: Things like this https://review.opendev.org/c/openstack/nova/+/872204 are incredibly difficult to make "work" in the functional tests because of all the ways and patterns we start and restart multiple fake computes18:32
bauzasgdam shit18:33
dansmithhow terrible would it be if we just always mock out those host consistency checks in functionals and rely on unit testing to cover them, and the tempest jobs to really cover the regular happy path(s) ?18:33
dansmithbecause the levels of stupid mocking to get even some of them to pass are probably worse and more complex than just not ever running those in the functionals18:34
bauzasdansmith: I need to look at your failures but unfortunately I need to quit today (EOB)18:35
dansmithbauzas: okay well, I'm just talking about a general read on the principle of the thing, but ... okay18:36
bauzasdansmith: if you want, mock them indeed and just leave the unittests for checking them18:37
dansmithack18:38
dansmithprobably need a read from the other two before I go down that route18:38
sean-k-mooneyyou could use a class constant to enable/disbale it18:39
sean-k-mooneybut enabel it by default in the base test18:39
dansmithto enable/disable the mocking you mean?18:39
sean-k-mooneyyep18:39
sean-k-mooneykind of like the microversion class constant18:39
sean-k-mooneyor the db one18:40
dansmithokay, I'm not sure that will get me much, assuming you mean "so you can test it in one functional that does things in a specific way" because of the way the rest of the singleton virt node mocking works18:40
dansmithbut I can try18:40
sean-k-mooneyno i mean by default mock it out so it passes18:41
sean-k-mooneyand if you need to test it then you would set MOCK_STABLE_UUID=False18:41
sean-k-mooneywhere you are explictly testing something that cares18:41
sean-k-mooneyfollowing this pattern https://github.com/openstack/nova/blob/master/nova/test.py#L158-L17318:42
dansmithwhat I'm saying is, even with the mock disabled for one test, the virt node mock always returns None, so testing this in a full stack is kinda difficult without *also* making that parameterized18:43
sean-k-mooneyoh ok18:43
dansmithbut perhaps if I put them both behind that flag I'll get what I need, I'll have to see18:43
sean-k-mooneythis is just needed for the last patch correct?18:44
sean-k-mooneyas in you already worked though the other test isseus in the previous ones18:44
dansmithyeah18:45
sean-k-mooneyso do you want to go up one level and just mock out _ensure_existing_node_identity and the check function for host and hypervisor hostname18:46
dansmiththat's what I was going to do, after two hours of trying to mock only the second call to it,18:46
dansmithbut there are just too many permutations of start, restart, stop, start, etc18:47
dansmithbut let me try to the flag for both the node mock and that one and see if i think it results in something meaningful18:47
dansmithI feel like it will probably end up with 10,000 tests having that set, just so one can have it unset, but still need mocks to make it not very useful, vs. just unit testing it in isolation18:48
dansmithbut I'll see18:48
dansmithtbh I wasn't thinking about tying the node mock to the compute _ensure one, so will try that first18:49
dansmithactually, maybe this will be better anyway and then I can write some dedicated lifecycle tests to simulate the manual testing we've been doing with devstack19:05
opendevreviewMerged openstack/nova master: Fixup patch for stable-compute-uuid series  https://review.opendev.org/c/openstack/nova/+/87184819:16
opendevreviewDan Smith proposed openstack/nova master: Check our nodes for hypervisor_hostname changes  https://review.opendev.org/c/openstack/nova/+/87222019:31
opendevreviewDan Smith proposed openstack/nova master: Protect against a deleted node id file  https://review.opendev.org/c/openstack/nova/+/87220419:31
opendevreviewDan Smith proposed openstack/nova master: Move comment about _destroy_evacuated_instances()  https://review.opendev.org/c/openstack/nova/+/87234819:31
*** dasm is now known as dasm|off19:33
opendevreviewDan Smith proposed openstack/nova master: Protect against a deleted node id file  https://review.opendev.org/c/openstack/nova/+/87220419:55
opendevreviewDan Smith proposed openstack/nova master: Move comment about _destroy_evacuated_instances()  https://review.opendev.org/c/openstack/nova/+/87234819:55
opendevreviewDan Smith proposed openstack/nova master: Protect against a deleted node id file  https://review.opendev.org/c/openstack/nova/+/87220420:11
opendevreviewDan Smith proposed openstack/nova master: Move comment about _destroy_evacuated_instances()  https://review.opendev.org/c/openstack/nova/+/87234820:11
opendevreviewMerged openstack/nova master: Add further workaround features for qemu_monitor_announce_self  https://review.opendev.org/c/openstack/nova/+/86732423:07

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!