Tuesday, 2021-10-12

bauzasgood morning Nova07:45
pslestangHey all, I'm facing an issue on a fresh installed devstack when running tox in nova directory which seems related to a conflict between oslo-vmware and suds-jurko 07:57
pslestangERROR: Cannot install -r /opt/stack/nova/test-requirements.txt (line 28) because these package versions have conflicting dependencies.07:57
pslestangThe conflict is caused by:07:57
pslestang    oslo-vmware 3.9.1 depends on suds-jurko>=0.607:57
pslestang    The user requested (constraint) suds-jurko===0.607:57
pslestangIs there a recommended fix or a solution to handle that?07:58
bauzaspslestang: weirdo07:58
bauzaswhich tox target ?07:58
bauzasI usually don't run my unittest on devstack but rather on my local machine07:59
bauzasat least because sometimes my whole local nova repo can disappear if I clean up devstack :)08:00
bauzasI rather prefer to have a local working repo and a git remote on my devstack08:00
pslestangI did not specify any target08:00
bauzasso, just 'tox' ?08:01
pslestangyep08:01
bauzasok, so unittests08:01
bauzasyou should tox -r 08:01
bauzaswhich will recreate the tox venv08:01
pslestangok trying08:01
bauzasand you should also specify a target :)08:02
bauzas-e py38 per say :)08:02
bauzasalso, running the whole testsuite is nice but... needs a lot of coffee to get answers :)08:03
bauzasyou should rather point tox to check only a few tests you know 08:03
pslestangrunning 'tox -r' it does not change anything08:03
bauzaspslestang: paste the outputs please08:04
bauzasso I can try to reproduce the exact command 08:04
bauzasmmm, maybe we have a regression in our CI, checking zuul08:06
bauzasactually, no, all looks good https://zuul.openstack.org/builds?project=openstack/nova08:07
pslestangwhich paste tool are you using, paste.openstack.org tells me that I send spam08:09
pslestangbauzas: https://privatebin.net/?18300286850a2784#BxNwm4jkzNKC4gYbPiTS11uXJvLUuq6UVHxRAv7yYXyK08:13
bauzaspslestang: you can force paste.o.o to accept your paste AFAICR08:14
bauzashah08:14
bauzas    error in suds-jurko setup command: use_2to3 is invalid.08:14
bauzasthat rings a bell to me :)08:15
bauzastesting locally08:18
pslestangok thank you08:18
bauzaspslestang: one quick thought could be that your nova local repo is a bit old and not getting the latest releases08:19
bauzaspslestang: yeah, confirmed, I can run tox locally on py38 target without any problem08:19
pslestangThe repo is fresh from this morining08:20
bauzas?08:20
bauzasI just did a git pull before I recreated the venv :)08:20
bauzaspslestang: last commit on your nova git repo ?08:21
bauzasmine is :08:21
pslestangfdfdba265833d237e22676f9a223ab8ca0fe1e0308:21
bauzascommit fdfdba265833d237e22676f9a223ab8ca0fe1e03 (HEAD -> master, origin/master, origin/HEAD)08:21
bauzasMerge: a8d3ab2513 ad227d708508:21
bauzasAuthor: Zuul <zuul@review.opendev.org>08:21
bauzasDate:   Fri Oct 8 02:36:41 2021 +000008:21
bauzas    Merge "Update min supported service version for Yoga"08:21
pslestangsame here08:21
bauzaspslestang: can you try to run tox on a local repo which is *not* devstack ?08:22
bauzasgit clone nova and shoot the tox runs08:22
pslestanglet me give a try08:22
bauzasI'd suspect some pip cache 08:22
pslestangsame result, I'm gonna try by cleaning pip cache it sounds like a good point08:25
fricklerpslestang: bauzas: https://bugs.launchpad.net/devstack/+bug/1946340 . we should mark that as FAQ somewhere08:44
fricklerthe issue isn't seen in our CI because we use pre-built wheels08:44
fricklerif you do a local build without old pip cache, it will fail the same way08:45
pslestangfrickler: you're completely right08:46
bauzaspslestang: yeah, eventually spotted the depency issue with devstack by reinstalling my testbed08:47
bauzasdependency* even08:47
pslestangI will try by pining the setuptools to 58.0.0 as proposed https://bugs.launchpad.net/devstack/+bug/1946340/comments/508:51
*** bauzas_ is now known as bauzas08:57
bauzasergh, network glitch here08:57
bauzasfrickler: fwiw, cinder installation on devstack failed due to this09:06
bauzasI can't see any proper workaround09:07
sean-k-mooney[m]for now you just remove oslo.vmware from cinders requirements file09:07
bauzasfrickler: pslestang: about documenting it, we already discussed about the 2_to_3 issues we got and the pinned setuptools and we said "well, this isn't a nova issue, so why should we create some relnotes for this ?"09:07
sean-k-mooney[m]ill try and submit a fix to oslo.vmware to use suds-community shortly09:07
bauzassean-k-mooney: locally, you mean ? ok09:08
sean-k-mooney[m]this is not a devstack bug09:08
sean-k-mooney[m]yes locally09:08
sean-k-mooney[m]i just commented it out09:08
sean-k-mooney[m]but the issue is in oslo.vmware it depens on suds-jaroko or something like that09:09
sean-k-mooney[m]suds-community is python 3 compatible09:09
sean-k-mooney[m]so once we fix there requirements file you can skip the local workaround09:09
bauzasyes09:09
fricklerbauzas: yes, local workaround is to remove oslo.vmware from cinder requirements09:15
bauzaskk09:15
fricklersean-k-mooney[m]: there is already one, it needs to be added to reqs first, though: https://review.opendev.org/c/openstack/oslo.vmware/+/81337709:16
fricklersean-k-mooney[m]: also, the most recent version of suds-community still hasn't fixed the 2to3 issue. but they provide wheels on pypi, which masks that issue09:17
sean-k-mooney[m]ah ok good09:17
fricklerthey need a new tag with this one https://github.com/suds-community/suds/pull/5809:17
fricklerah, that'll be a major version bump, so the test release is suds-community-1.0.0b109:21
fricklerwhich I can install locally from source just fine, so that looks like progress09:21
mdboothdansmith: Liking DEVSTACK_PARALLEL: Speedup:  1.2226 :)09:27
sean-k-mooney[m]it works pretty well09:27
mdboothIt had a smaller benefit in a GCE vm, but this one on PSI seems to like it09:28
sean-k-mooney[m]are you using the ci flavors or normal ones09:28
mdboothnormal09:29
sean-k-mooney[m]the normal ones are all backed by ceph and have lower iops09:29
sean-k-mooney[m]so ya it helps more there09:29
sean-k-mooney[m]the ci ones have local nvme storage and it benifits less09:29
sean-k-mooney[m]the more io overhead there is the more parrallel helps09:30
opendevreviewBalazs Gibizer proposed openstack/nova master: Add a WA flag waiting for vif-plugged event during reboot  https://review.opendev.org/c/openstack/nova/+/81341910:15
sean-k-mooneyhum now that is interesting.10:29
sean-k-mooneydid people know that if you hit b in gerrit it loads git blam on the side10:30
sean-k-mooney*blame10:30
gibithat sounds usefull10:32
*** thelounge94 is now known as redrobot13:02
*** redrobot is now known as thelounge9413:04
*** thelounge94 is now known as redrobot13:04
dansmithmdbooth: sweet, I have a bunch of other parallel points to add, but I need to circle back and finish them13:29
opendevreviewnorman shen proposed openstack/nova master: Recreate mdev devices according to placement  https://review.opendev.org/c/openstack/nova/+/81022013:39
* bauzas disappears for kids taxi and then going to the garage 14:21
bauzasbut I'll back around 1515UTC (45 mins before the meeting)14:21
gibikashyap: I read the driver part of https://review.opendev.org/c/openstack/nova/+/762330 and left comments. (I still not read the test parts). I don't feel this commit is ready. If feel this is patched together in a rush. 14:23
opendevreviewAlexey Stupnikov proposed openstack/nova master: Rollback problematic port bindings on source host only  https://review.opendev.org/c/openstack/nova/+/81365914:25
pslestangsean-k-mooney: FYI it seems like there is already someone patching oslo.vmware to use suds-community instead of suds-jurko https://review.opendev.org/c/openstack/oslo.vmware/+/81337714:25
sean-k-mooneyyes14:25
sean-k-mooneypslestang: frickler  mentioned that above14:26
pslestangsean-k-mooney: ouup's I missed it14:27
kashyapgibi: Hi, I'll look and investigate.  The submitter told me they even tested it in a real deployment (I took their word)14:31
gibikashyap: I can accept that it works but for me it is really hard to follow. maybe other in the core team has more knowledge to figure out what happens.14:32
kashyapgibi: Thanks for the review time!  We definitely don't want this rushed in.  And needs careful integration testing.  As it impacts live migration14:32
kashyapgibi: No, if it's hard to follow for you, that's a reason enough to clean it up.  And also it needs code comments too14:33
gibiso I'm more concerned about understandabilty now than correctness14:33
gibiahh, ok14:33
kashyapYeah; I agree this needs more explnaations14:33
gibilyarwood: I looked into https://bugs.launchpad.net/nova/+bug/1946339 in short I don't see what happens and I cannot reproduce it locally while it is happening on the gate frequently. I'm a bit stuck14:40
sean-k-mooneygibi: lyarwood  is hopfule preparing for an operation later today and will be recovering for the next ~2 weeks 14:46
gibisean-k-mooney: ack, I know. I just wanted to get back to him. (I update the bug with my finding)14:47
sean-k-mooneyah test_unshelve_offloaded_server_with_qos_port_pci_update_fails14:47
sean-k-mooneygibi: presumable this is some interaction between artoms fix for updating the pci device slot in the port profile on unshleve and minium band with based scheduling14:50
opendevreviewAlexey Stupnikov proposed openstack/nova master: Rollback problematic port bindings on source host only  https://review.opendev.org/c/openstack/nova/+/81365914:50
sean-k-mooneygibi: have you tried ensuring that the pci device claimied as part of the unshelve is different14:50
sean-k-mooneye.g. boot a vm, shelve it, boot another vm ensuring it claims the same pci device as the first then try unslevleing the first vm14:51
sean-k-mooneyoh this is in the func tests14:52
sean-k-mooneynot an end user bug14:52
gibinope14:53
gibiand the first exception is part of the test 14:53
sean-k-mooneyso it looks like somehow the db get torwn down too early?14:53
gibisomething like that14:53
gibibut I don't get how can that be14:53
sean-k-mooneythat is presumable created by the fixture in setup 14:54
sean-k-mooneyso ya that is hard to understand14:54
sean-k-mooneygibi: this is not messing with any global state right https://github.com/openstack/nova/blob/a8d3ab2513c39aeac3393b2154988316dfa2db3a/nova/tests/functional/test_servers_resource_request.py#L2523-L252614:56
sean-k-mooneygibi: we have our own isntance of placement per test function14:56
sean-k-mooneythat has its own db14:56
gibiI assume we have placement per test case per test executor14:57
gibiotherwise everything would be unstable14:57
sean-k-mooneyya that is my assumtion too14:57
gibibtw the sriov placement tree is created in the test case setup so that is per test case for sure14:57
gibiand the uuids shoudl be unique14:57
sean-k-mooneycould this be failing due to the update_avaiable_resouce_provier periodic task?15:01
sean-k-mooneyif that ran it would raise the same error internally but it might now catch it15:01
gibithere is no proof in the logs that the period runs https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b15/713498/27/check/nova-tox-functional-py38/b1582a8/job-output.txt15:02
gibiwe have test cases where we trigger the periodic manually, so I think we even turn of the periodic for the func test15:03
gibiI cannot reproduce it even if I delay the compute side of the execution locally, the wait for server status timeouts first not the conductor15:10
gibiso I don't get how the conductor can time out in the gate15:10
gibiI will try pulling out the test execution order from a failed gate run and see if that reproduce it locally or not..15:12
sean-k-mooney your referign to "oslo_messaging.exceptions.MessagingTimeout: No reply on topic conductor\n"15:13
gibiyepp15:13
sean-k-mooneythis is also using the in memory driver so there is no networking issues that could be at play15:14
sean-k-mooneywith that said do you know what we set the time out too15:14
sean-k-mooneywe make spawn and other thing synconouse in the functional tests15:14
sean-k-mooneyi wonder if we only see this in a slow node15:15
sean-k-mooneylike is the time out 1 second of something tiny like that15:15
sean-k-mooneywhen using the fake driver15:15
gibiI don't know the timeout value in the funct test, but I put a sleep(10) in the compute side just before raising the expected exception, and it makes the test waiting for the server state time out instead of the fake driver 15:15
sean-k-mooneyso i think this is where that messaging timeout gets raised https://github.com/openstack/oslo.messaging/blob/ca939fc0e4683efce87b567a9a074063a9c75b4f/oslo_messaging/_drivers/impl_fake.py#L207-L21415:20
sean-k-mooneywhich happend due to a  _queue.Empty from  return self.greenlet.switch()15:21
sean-k-mooneythe resource provider error on the compute was boubled up all the way  to the rpc server15:22
sean-k-mooneythat first traceback is form here https://github.com/openstack/oslo.messaging/blob/ca939fc0e4683efce87b567a9a074063a9c75b4f/oslo_messaging/rpc/server.py#L178-L18015:25
gibibut that line is visible in case of a successful run too15:26
sean-k-mooneywell after that message is printed we sent the failure to the conductor/api here https://github.com/openstack/oslo.messaging/blob/ca939fc0e4683efce87b567a9a074063a9c75b4f/oslo_messaging/rpc/server.py#L18615:27
sean-k-mooneywould that be recied by https://github.com/openstack/nova/blob/master/nova/conductor/api.py#L140-L141 ?15:29
gibithis is the log from a successful run https://paste.opendev.org/show/809928/15:29
sean-k-mooneyno it would be in the conductor maager https://github.com/openstack/nova/blob/fdfdba265833d237e22676f9a223ab8ca0fe1e03/nova/conductor/manager.py#L94115:29
sean-k-mooneygibi: right so in that case we see more output in the python loggin capture15:30
sean-k-mooneywhere as ehre it stops and we get stderr15:31
gibisean-k-mooney: it is strange, the timeout is from self.compute_task_api.build_instances not from  unshelve_instance 15:31
sean-k-mooneybuild instance?15:32
sean-k-mooneyso here https://github.com/openstack/nova/blob/a8d3ab2513c39aeac3393b2154988316dfa2db3a/nova/tests/functional/test_servers_resource_request.py#L2533-L253415:32
sean-k-mooneyoh15:32
sean-k-mooneywe mess withthe name before we boot the vm15:32
sean-k-mooneyare we somethimes landing on host215:32
sean-k-mooneywe are not forcing it to boot on host115:33
gibiwe request to land on host1 first https://github.com/openstack/nova/blob/a8d3ab2513c39aeac3393b2154988316dfa2db3a/nova/tests/functional/test_servers_resource_request.py#L55415:33
gibiin the meantime I confirm that it is not about testcase ordering. the same order that failed on the gate passing for me locally15:35
gibiI don't get how can we fail on the build_instance so late15:37
gibithe VM went to active according to the test 15:37
gibishit is it is a reschedule 15:39
gibithat calls back to build_instance15:39
gibiFile "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 2263, in _do_build_and_run_instance15:39
sean-k-mooneyya i was just checkign that we dont use build an drun instance in unshelve which we dont  we use driver.spawn15:41
sean-k-mooneyhttps://github.com/openstack/nova/blob/fdfdba265833d237e22676f9a223ab8ca0fe1e03/nova/compute/manager.py#L667515:41
sean-k-mooneygibi: you think for some reason it did not boot on host1 and reschduled to host215:41
sean-k-mooneywe could tetst this by disabling host1 and see if that cause the error15:42
sean-k-mooneywe shoudl see the schduler select the host though right15:42
gibiI can test that15:43
gibiif I disable host1 at the start of the test case then the initial boot will fail and the VM is in ERROR state15:45
gibi* initial boot fails15:45
sean-k-mooney\noslo_messaging.exceptions.MessagingTimeout: No reply on topic conductor\n', 'host2')15:45
gibiso the test forces the VM to host115:45
sean-k-mooneythe message time out was for host 215:45
gibicould it be that         # make host1 unusable so the subsequent unshelve needs to select host215:45
gibi        self.admin_api.put_service(15:45
gibi            self.compute1_service_id, {"status": "disabled"})15:45
gibisorry15:45
gibicoudl it be that https://github.com/openstack/nova/blob/fdfdba265833d237e22676f9a223ab8ca0fe1e03/nova/tests/functional/test_servers_resource_request.py#L2548 does not disable the host quickly enough and ushelve selects host1?15:46
gibihm, but why host2 is the call, true15:47
* bauzas is back15:47
bauzasquick reminder15:47
bauzasnova meeting in 12-ish minutes in this channel15:48
sean-k-mooneyits posible although that is ment to be a blocking call15:49
sean-k-mooneygibi: have you tied inverting the logic an disbaling host2 instead of host one 15:50
sean-k-mooneyhere https://github.com/openstack/nova/blob/a8d3ab2513c39aeac3393b2154988316dfa2db3a/nova/tests/functional/test_servers_resource_request.py#L2548-L254915:50
gibiso change the test from boot:host1, shelve offload, unshelve:host2 -> to boot:host2, shelve offload, unshelve:host1 /15:50
sean-k-mooneyyou could be right that we are racing with that disabel and we neeed to add a get15:50
gibi?15:50
sean-k-mooneyno i ment use replace self.compute1_service_id with self.compute2_service_id15:51
gibiI cannot simply disable host2 as we need to unshelve on a compute where the placement RP tree is wrong15:51
sean-k-mooneyhere https://github.com/openstack/nova/blob/a8d3ab2513c39aeac3393b2154988316dfa2db3a/nova/tests/functional/test_servers_resource_request.py#L2548-L254915:51
sean-k-mooneyso it unshelve to the same hsot15:51
bauzasgibi: sean-k-mooney: discussing about https://bugs.launchpad.net/nova/+bug/1946339 ?15:51
gibibauzas: yes15:52
sean-k-mooneyright i know it would break the test logic15:52
bauzasack thanks15:52
sean-k-mooneybut im wondering if some how we are not landing on host 215:52
sean-k-mooneye.g. to confirm your assertion that perhaps the disabel is not working15:52
sean-k-mooneywe might need a wait before the unshleve15:52
gibiif I just disable host2 then the test times out at https://github.com/openstack/nova/blob/a8d3ab2513c39aeac3393b2154988316dfa2db3a/nova/tests/functional/test_servers_resource_request.py#L2558 as no expcetion happens host1 can unshelve the instance15:54
sean-k-mooneywe should be abel to assert the service is disabeld form the responce https://docs.openstack.org/api-ref/compute/?expanded=update-compute-service-detail#update-compute-service15:55
gibiyeah that would only help if I could reproduce the problem15:58
gibichecking the response15:58
gibiI still have to track down why could we re-schedule15:58
bauzas#startmeeting nova16:00
opendevmeetMeeting started Tue Oct 12 16:00:01 2021 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
opendevmeetThe meeting name has been set to 'nova'16:00
bauzashola folks 16:00
bauzas#link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting16:00
gibio/16:00
sean-k-mooneyo/16:00
gmanno/16:01
elodilleso/16:01
bauzasok, let's start16:01
bauzas #topic Bugs (stuck/critical) 16:02
bauzas No Critical bug16:02
bauzas #link 18 new untriaged bugs (+5 since the last meeting): #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New16:02
bauzassorry I didn't had to triage the bugs this week, will do that tomorrow16:02
bauzasany bug to discuss ?16:03
bauzasok, looks not16:04
bauzas #topic Gate status 16:04
bauzas Nova gate bugs #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure16:04
bauzasI saw gibi and sean-k-mooney discussing about one of them16:04
bauzasfolks, do you want to discuss about this now or off the meeting ?16:04
bauzascontext : https://bugs.launchpad.net/nova/+bug/194633916:04
sean-k-mooneyi think off meeting16:04
gibiI have nothing specific to say :)16:04
bauzascool, moving on then16:05
gibistill trying to reproduce / understand what happens16:05
sean-k-mooneyim still reviewing the logs and trying to repoduce16:05
bauzasacvk16:05
bauzas Placement periodic job status #link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly16:05
bauzasso you'll see we have a problem with one job16:05
bauzasplacement-nova-tox-functional-py3816:05
gibihttps://zuul.openstack.org/build/055ea1806bf14b1994f1ebdcee720134/log/job-output.txt#80816:07
gibistrange16:07
gibiubuntu-focal | /bin/sh: 1: Syntax error: "(" unexpected16:07
bauzasit looks to me the tox call is wrong16:07
bauzasI guess because of the regex16:07
bauzashave we modified this ?16:07
bauzas/home/zuul/.local/tox/bin/tox  --showconfig  -efunctional-py38  ^((?!(?:api|notification)_sample_tests|functional\.db\.).)*$ > /tmp/ansible.quewmudg16:07
gibiI'm not waware of any modification recently16:08
bauzasI need to take time to look at the regex16:08
bauzasto verify whether it works16:08
gibiwe touched tox.ini in febr16:08
gibithat was a looong time ago16:08
* bauzas uses https://regex101.com/16:09
bauzasthe regex looks valid to me16:09
gmanndoes not seems like any recent change in openstack-tox-functional-py38 base job too16:11
dansmiththat has to be something not in tox.init right?16:11
gmannyea16:12
dansmiththat "get tox envlist config" is some chunk of inlined bash from ansible right/16:12
bauzasyes16:12
bauzashttps://zuul.openstack.org/build/055ea1806bf14b1994f1ebdcee720134/log/job-output.txt#80716:12
dansmithright16:12
bauzasthat's when calling /home/zuul/.local/tox/bin/tox  --showconfig  -efunctional-py38  ^((?!(?:api|notification)_sample_tests|functional\.db\.).)*$ > /tmp/ansible.quewmudg16:12
bauzaswe get a stdout telling "/bin/sh: 1: Syntax error: "(" unexpected" 16:12
sean-k-mooneyhas anyone just tried runnign the func test locally16:13
gibiyes16:13
bauzasnot yet16:13
gibiit work for me16:13
bauzasthat's placement so we can't create a LP bug, but I'd appreciate if we could track in down16:13
bauzastrack it* down16:13
bauzasI'm not used to storyboard honestly16:14
bauzasbut I guess this is time for me to fly16:15
gibiit is not hard it is just slow :)16:15
gmannvery slow :)16:15
dansmithand hard :)16:15
bauzaswell, this is not that's hard, it's just a new foreign language for me to understand :)16:16
* bauzas wonders whether someone will provide a Duolingo feature for Storyboard language16:16
bauzasbut I can create a "bug" 16:17
sean-k-mooneyor you know we could just move it to launchpad16:17
gmann+116:17
sean-k-mooneysince it now part of the comptue project deliverables16:17
bauzassean-k-mooney: please don't make my dreams tru16:17
bauzastrue*16:17
bauzashonestly, who's looking at the storyboard stories we have for placement ?16:17
bauzasI don't :shame:16:18
opendevreviewBalazs Gibizer proposed openstack/placement master: DNM: check tox func job  https://review.opendev.org/c/openstack/placement/+/81367016:18
gibilet see if it still fails ^^ 16:18
bauzasgibi: thanks16:18
gmanngibi: I do not think it will start jobs unless you change any file16:18
bauzasI'll create the SB task and let's dissuss this later 16:19
gmannthat is what we got error from zuul last week 16:19
bauzasmoving on ?16:19
gibigmann: it seems it is https://zuul.opendev.org/t/openstack/status#81367016:19
gibibauzas: yes16:19
gmannah interesting. 16:20
bauzask, let's continue16:20
bauzasoh, wait then16:20
bauzasgmann: we're all listening to you :)16:20
gmannah,  we can move. I will check in parallel 16:20
bauzasgmann: ok, as you prefer16:21
bauzasmoving on then16:21
bauzas(we can continue to discuss this bug in the open discussion if people want and we have time)16:21
bauzas #topic Release Planning 16:22
bauzas Nova 24.0.0 is now released \o/16:22
bauzasof course, placement too16:22
bauzasthanks to all the people who contributed to it16:22
bauzasnow, we're officially in the Yoga timeframe, even if master is on Yoga since 3 weeks :)16:23
bauzasthat makes me go to the next deadline16:23
bauzas Yoga-1 is due Nova 18th #link https://releases.openstack.org/yoga/schedule.html#y-116:23
bauzasI just wrote something in the PTG agenda about discussing our own deadlines16:25
bauzasand whether we should do feature or specs freeze16:25
bauzasnot wanting to discuss it by now16:25
bauzasbut start to think about this16:25
bauzaswe're a bit early in the cycle so a Spec review day is not really important to discuss by now16:26
bauzasunless people disagree16:26
bauzaswe don't have a lot of open specs by now16:27
bauzasanyone anything ?16:28
bauzasfor release planning ?16:28
gibibauzas: there is two move implemented specs patch https://review.opendev.org/q/project:openstack/nova-specs+status:open16:28
gibione from you and one from brin16:29
opendevreviewJan Hartkopf proposed openstack/nova master: docs: add hint to create folder for new microversion  https://review.opendev.org/c/openstack/nova/+/81367216:29
bauzasoh that's fun16:29
bauzasgibi: good catch, will close mine as it's the newest16:29
bauzasbut I'm not sure brin got all the things16:29
bauzasanyway, let's not discuss this now16:30
gibiack16:31
bauzas #topic Review priorities 16:31
bauzas https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement)+label:Review-Priority%252B116:31
bauzasstill continuing to use this flag for tracking changes I wanna review16:32
bauzasmoving on, we'll discuss the flag usage at the PTG16:32
bauzas #topic PTG Planning 16:33
bauzas every info is in the PTG etherpad #link https://etherpad.opendev.org/p/nova-yoga-ptg16:33
bauzasimportant :16:33
bauzas Last week before the PTG, please prepare your points by Thursday EOB so bauzas can provide a nova PTG agenda16:33
bauzas(that's an ideal thing)16:33
bauzasI'll start moving things around on Friday in order to group related things (if I can)16:34
bauzasof course, we'll surely have last minute topics to discuss :)16:34
bauzas neutron PTL ( lajoskatona ) seemed interested in some x-p session, we need to find a slot16:34
bauzasI just got a email before the meeting from rosmaita about a proposed cinder x-p session slot on Thursday 1pm UTC16:35
bauzaswfy ?16:35
gibiworks for me, early for the US folks16:37
opendevreviewBalazs Gibizer proposed openstack/nova master: DNM: add traces to troubleshoot bug/1946339  https://review.opendev.org/c/openstack/nova/+/81367416:38
bauzasbrian is telling that they have 1pm-2.30pm free16:38
bauzasso I guess dansmith could join at 1.30pm if he wishes but for melwitt, yeah that could be tough16:39
dansmithmeaning 1330Z ?16:39
bauzasyeah we could start at this time16:39
bauzasthe two things he had to discuss was :16:40
dansmithyeah that's the absolute earliest for me without a special arrangement,16:40
bauzashttps://blueprints.launchpad.net/cinder/+spec/add-volume-re-image-api16:40
bauzashttps://blueprints.launchpad.net/nova/+spec/volume-backed-server-rebuild16:40
dansmithbut not sure I need to be there specifically16:40
dansmithah16:40
bauzasI'll try to see if we can get some later slot16:40
bauzasmelwitt could be interested in one of the two topics16:41
dansmithwell, like I say, I'll be available at that time, but.. ack16:41
bauzasif not the two :)16:41
bauzasmelwitt: around ?16:41
bauzasdansmith: ack, we'll see how this goes16:42
bauzasthe problem here is that I think both specs are pretty large16:42
bauzasthey're actually tied16:42
bauzasok, let's move on from this topic, time is flying but I'll see what we can do16:43
bauzasthe fact is, we'll be missing lyarwood and I'd appreciate if some other volume experts to be there16:43
bauzasmoving on16:44
bauzas #topic Stable Branches 16:44
bauzaselodilles: floor is yours16:44
elodilleslet me copy from the meeting page :)16:45
elodillesold stable gates are blocked (stein and older), needs the setuptools pinning patch to unblock: https://review.opendev.org/q/I26b2a14e0b91c0ab77299c3e4fbed5f7916fe8cf16:45
elodillesWallaby (23.1.0), Victoria (22.3.0), Ussuri (21.2.3) were released last week16:46
elodillesUssuri Extended Maintenance transition is in a month (Nov 12) - do we want to prepare another release? -- https://etherpad.opendev.org/p/nova-stable-ussuri-em16:46
elodillesthat was it16:46
elodillessince we just released from stable/ussuri I don't know whether we need another release before the transition16:47
elodillesbut if anyone see any patch that would be good to release then let us know16:47
bauzasdo we have backports to look at ?16:47
elodilleswe have lots of backports on ussuri16:48
elodillesthey are listed on the pad: https://etherpad.opendev.org/p/nova-stable-ussuri-em16:48
bauzasyeah I can see them16:48
bauzashttps://review.opendev.org/q/project:openstack/nova+branch:stable/ussuri+is:open16:48
bauzashumpf16:49
elodillesI went through the patches and commented on the etherpad for some16:49
bauzasI guess one last release would be appreciated16:49
bauzasbefore we call it EM16:49
elodillesthere are some that needs a second +2 only16:49
bauzaselodilles: we can make some kind of status updates at the meetings16:50
elodillesbauzas: sure, I'll prepare with that16:50
bauzaselodilles: and ping me and a few other stable cores to look at ussuri this week and the week after PTG16:50
elodillesack16:50
bauzas #topic Sub/related team Highlights 16:50
bauzas Libvirt (lyarwood)16:51
bauzaslyarwood is not there, nothing to mention16:51
bauzaslet's move straight to the last bit of the agenda16:51
bauzas #topic Open discussion 16:51
bauzas Next week meeting to be cancelled (PTG). AgreedĀ ?16:51
gibiagre16:51
gibie16:51
bauzasanyone disagreeing ?16:51
bauzas1..16:51
bauzas2..16:51
bauzas3..16:52
bauzasOK, that's it, will send the email16:52
bauzasI'm done with the agenda now, anyone wanting to raise anything ?16:52
bauzaswe have 8 mins in the balance :)16:52
gibijust circle back to the placement functional failure16:53
bauzaswfm16:53
gibiit still visible on master16:53
bauzasgmann: you had thoughts ?16:53
gibiand I do belive this causing https://opendev.org/zuul/zuul-jobs/commit/c02c28a982da8d5a9e7b4ca38d30967f6cd1531d16:53
gmannI think this is because of https://opendev.org/zuul/zuul-jobs/commit/c02c28a982da8d5a9e7b4ca38d30967f6cd1531d16:53
gibithat we include the regexp to the call16:53
gmanngibi: ah you are fast16:53
gibiwe simply not need the regexp in that call16:53
gibior we need to quote it16:54
gibibut I don't know how to feed this back to zuul16:54
gibigmann:  :)16:54
gmannI think bext way is not to include the test regex in tox_extra_args  and instead add it as part of command itself16:55
gmannbest16:55
gmannlet me try that16:55
gibigmann: ack, we can even left out the regexp param that is not needed for the --showconfig call I think16:56
bauzasgmann: gibi: thanks for the findings16:56
gibiI have nothing else for today16:56
gmannyeah, tox_extra_args purpose is completely separate then test regex  16:56
gibigmann: ack16:56
sean-k-mooneywe normally do that with posargs like this https://github.com/openstack/placement/blob/master/tox.ini#L4016:57
gmannyeah16:57
sean-k-mooneythen invoke tox like "tox -e fucntional -- <regex goes here>"16:57
sean-k-mooneyi think we can likely close the meeting there16:59
bauzasyeah I was about to say16:59
bauzaswe can continue off meeting16:59
bauzaslet me call it done16:59
bauzas#endmeeting16:59
opendevmeetMeeting ended Tue Oct 12 16:59:40 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:59
opendevmeetMinutes:        https://meetings.opendev.org/meetings/nova/2021/nova.2021-10-12-16.00.html16:59
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/nova/2021/nova.2021-10-12-16.00.txt16:59
opendevmeetLog:            https://meetings.opendev.org/meetings/nova/2021/nova.2021-10-12-16.00.log.html16:59
bauzasthanks all16:59
gibibauzas: thanks!16:59
sean-k-mooneygibi: for  https://bugs.launchpad.net/nova/+bug/1946339  i think we are getting to here but i never see log lings corresponding to https://github.com/openstack/nova/blob/a8d3ab2513c39aeac3393b2154988316dfa2db3a/nova/tests/functional/test_servers_resource_request.py#L2568-L257417:00
bauzasgibi: thanks for spotting the root cause while I was deblatering for some paperwork :)17:00
gibisean-k-mooney: yeah but that becuase we fail at the assert https://github.com/openstack/nova/blob/a8d3ab2513c39aeac3393b2154988316dfa2db3a/nova/tests/functional/test_servers_resource_request.py#L2560\17:06
sean-k-mooney are you sure about that we might be17:08
sean-k-mooneyi did not see that in the log as explcitly failing17:08
sean-k-mooneydid i miss that17:08
gibiI'm looking at this https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b15/713498/27/check/nova-tox-functional-py38/b1582a8/job-output.txt17:09
gibiit has17:09
gibi2021-10-08 01:21:54.733579 | ubuntu-focal |     testtools.matchers._impl.MismatchError: 'UnexpectedResourceProviderNameForPCIRequest' != 'DBNonExistentTable'17:09
sean-k-mooneyah right17:09
sean-k-mooney error_notification = self.notifier.wait_for_versioned_notifications(17:11
sean-k-mooney            'compute.exception')[0]17:11
sean-k-mooneywe are just takign the first error17:11
gibiyepp17:11
gibiI have to drop for today17:11
sean-k-mooneycould it be the order of the excption could change and we get more then one17:12
sean-k-mooneyok17:12
gibiI pushed a small patch with an extra log seeing what triggers the reschedule17:12
gibihttps://review.opendev.org/c/openstack/nova/+/81367417:12
sean-k-mooneyim not sure there is a reschdule17:12
sean-k-mooneybut cool lets see what that shows17:12
sean-k-mooneyill play with a bit locally17:12
gibithis line in the stack trace File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 2263, in _do_build_and_run_instance17:13
gibipoints to a reschedule for me17:13
gibianyhow leaving now. thanks for the shared thinking17:14
gibio/17:14
sean-k-mooneyhum17:19
sean-k-mooneyERROR: Cannot install jsonschema>=3.2.0, openstack-placement==1.0.0 and openstack-placement==1.1.0 because these package versions have conflicting dependencies.17:19
sean-k-mooneyThe conflict is caused by:17:19
sean-k-mooney    The user requested jsonschema>=3.2.017:19
sean-k-mooney    openstack-placement 1.1.0 depends on jsonschema<3.0.0 and >=2.6.017:19
sean-k-mooney    The user requested jsonschema>=3.2.017:19
sean-k-mooney    openstack-placement 1.0.0 depends on jsonschema<3.0.0 and >=2.6.017:19
sean-k-mooney    The user requested (constraint) jsonschema===3.2.017:20
sean-k-mooneythat was form trying to run nova func test on master17:20
sean-k-mooneyit cant create the pip env17:20
sean-k-mooneyplacmenet required >=3.2.0 17:21
sean-k-mooneyhttps://github.com/openstack/placement/blob/master/requirements.txt#L1017:21
sean-k-mooneywhich is the same as uc https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L54717:23
sean-k-mooneyoh17:24
sean-k-mooneythis is the same suds-jurko issue17:24
sean-k-mooneyok i just need to locally comment out oslo.vmware in our test-requirements.txt until that is fixed17:25
opendevreviewGhanshyam proposed openstack/nova master: Define new functional test tox env for placement gate to run  https://review.opendev.org/c/openstack/nova/+/81367917:33
opendevreviewGhanshyam proposed openstack/placement master: Use 'placement-nova-functional-py38' tox env for placement nova job  https://review.opendev.org/c/openstack/placement/+/81368017:36
opendevreviewGhanshyam proposed openstack/nova master: Define new functional test tox env for placement gate to run  https://review.opendev.org/c/openstack/nova/+/81367918:11
opendevreviewGhanshyam proposed openstack/nova master: Define new functional test tox env for placement gate to run  https://review.opendev.org/c/openstack/nova/+/81367918:13
gmanngibi: bauzas these fix the placement-nova functional job - https://review.opendev.org/q/topic:%22fix-placement-gate%22+(status:open%20OR%20status:merged)19:06
opendevreviewsean mooney proposed openstack/nova master: [WIP] adress intermitent failure of functional tests  https://review.opendev.org/c/openstack/nova/+/81369519:08
sean-k-mooneygibi: i think ^ would fix https://bugs.launchpad.net/nova/+bug/194633919:09
*** efried1 is now known as efried19:31
simondodsleyHow would I achieve full disaster recovery of nova instances from one OS cluster to another? I can replicate Cinder volumes to another cluster (sort of), but I can't find a way to replicate the nova instances? I know Stratoscale used to do something like this, but they are dead now. Any other ways to do this?21:09
melwittsimondodsley: I can't answer your question, I'm sure there are a lot of ways to do it, but you might get some ideas from this project https://docs.openstack.org/freezer/latest/ this is/was the openstack project for disaster recovery. it hasn't had activity for the past year or so, it may no longer be maintained https://github.com/openstack/freezer-dr21:21
opendevreviewAde Lee proposed openstack/nova master: Add check job for FIPS  https://review.opendev.org/c/openstack/nova/+/79051921:53
gmannmelwitt: dansmith please check these two to unblock the placement gate https://review.opendev.org/q/topic:%22fix-placement-gate%22+(status:open%20OR%20status:merged)22:04
melwittgmann: hm, not sure I understand the solution. it doesn't seem right to define any placement env in the nova repo? I also don't understand why the current setup is failing22:29
gmannmelwitt: as it is used in nova-placement job we can move the job definition also on nova side but that run only in placement22:31
melwittgmann: I don't understand why the current thing is no longer working, it was intended to be able to use the nova-tox-functional-py38 in other projects right? as a parent job?22:32
melwittwhy do we need to add nova-placement in the nova or placement repo now?22:32
melwittlet me read the commit message and referenced zuul commit again22:35
gmannmelwitt: placement-nova-tox-functional-py38 job is only needed to skip the sample and db tests otherwise same as  nova-tox-functional-py38 22:35
gmannand I think these tests are skipped as they do not use placement_fixture 22:35
melwittyeah, I see that ... trying to understand the bug and why we can't use the parent job as intended22:36
melwittso a recent commit added tox_extra_args back, apparently previously it wasn't being used even though it was defined in the placement .zuul.yaml I guess22:37
melwittand now that it's being used, there's a syntax error happening22:37
gmannmelwitt: it is now started failing because test regex to skip the test is defined in  tox_extra_args which is now added in 'Get tox envlist config; tasks  https://opendev.org/zuul/zuul-jobs/commit/c02c28a982da8d5a9e7b4ca38d30967f6cd1531d 22:37
gmannyes22:37
gmannI do not think tox_extra_args  should be used for tests regex formation instead we should define the test path/regex etc in tox env command itself22:38
gmannand as tox env has to live in nova and job is defined  in placement, we need these two repo changes to fix it22:39
melwittoh... at least to me it looks like it makes sense for the regex string to be an extra arg to tox22:39
gmannbut that fail when we take tox_extra_args  in tox env with no tests run like this https://opendev.org/zuul/zuul-jobs/commit/c02c28a982da8d5a9e7b4ca38d30967f6cd1531d22:40
gmannwhich is generic to add tox_extra_args  because it can have more config things for tox22:41
gmannmelwitt: easy way is to just run nova-tox-functional-py38 and do not skip tests as it is periodic weekly job22:41
melwittsorry, I got lost at the "no tests run" and "skip tests"22:42
gmannor define the placement-nova-tox-functional-py38 job in nova but use in placement only 22:42
gmannmelwitt: no test run when tox is run to show config --showconfig  that is where it is failing when  tox_extra_args  has the test regex string 22:45
melwittok, I see now the "name: Run tox without tests" that is what doesn't work with the regex as an extra arg I take it22:45
gmannmelwitt: here too when for cinfig https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/tox/tasks/siblings.yaml#L2222:46
melwittyeah... I see now22:47
melwittthanks22:48
gmannif having tox env but not used on nova is confusing then we can move job definition too in nova but run in placement only22:50
gmannotherwise running nova-tox-functional-py38 itself is not so costly as this will be run periodic weekly only22:50
melwittI dunno if it's confusing to most, it was just unintuitive to me because normally we can keep things separate and inherit from one project to another but I see now why this isn't possible with the regex we need to use. none of the zuul tox role variables are appropriate for this22:52
gmannyeah22:54
melwittI think the way you have proposed probably makes the most sense22:55
melwitt(what you have already proposed patches)22:55
gmannok, I added about which job is using this tox in its description to make it clear for future 22:56
melwittyeah that is good ++22:57
gmannok23:01
dansmithgmann: each tox env is just a new target that does what we want the other job to do, is that right? 23:07
dansmithmeaning, nova defines a placement env so placement can run nova tests "the placement way" ?23:08
dansmithI think it would be super less confusing if the tox env name didn't have the other project in it, but described the difference, like "functional-all-but-foo-tests" or something23:08
melwittthat's a good point ^23:09
gmanndansmith: ok23:09
gmannplacement does not need to be in name itself23:09
dansmithI mean, just MHO, but.. seems more intuitive to me23:10
gmannhow about this 'functional-skip-sample-db-tests'23:11
melwittyeah maybe like functional-without-sample-tests-py38 or something23:11
melwittI agree, I think it would be more intuitive23:11
dansmithcha23:11
melwittgmann: that sounds good, add -py38 at the end though right since it's 3.8 only?23:12
gmannfunctional-without-sample-db-tests ? as it skip db test too and remove py38 thing as it can lengthy the name23:12
melwittoh23:12
gmannmelwitt: i was thinking just to run it on available py version as goal is not test py version in this job ?23:12
gmannand we do not need to rename when we move to py39 or so23:13
melwittgmann: yeah true23:13
melwittyeah that makes sense23:13
gmannok, let me rename it. 23:13
melwittk. will re +2 them after23:14
gmanndansmith: melwitt do you think to move job definition also to nova side or it is ok? 23:14
melwittgmann: I think the job def in placement is better, personally23:15
gmannok23:16
opendevreviewGhanshyam proposed openstack/placement master: Use 'placement-nova-functional-py38' tox env for placement nova job  https://review.opendev.org/c/openstack/placement/+/81368023:23
opendevreviewGhanshyam proposed openstack/nova master: Define new functional test tox env for placement gate to run  https://review.opendev.org/c/openstack/nova/+/81367923:23
gmannmelwitt: dansmith ^^ updated23:25
dansmithI didn't see a job def change23:26
gmanndansmith: you mean for tox env rename or you were talking to change job name too ?23:28
gmannhttps://review.opendev.org/c/openstack/placement/+/813680/2/.zuul.yaml23:28
gmanndansmith: ^^23:28
dansmithgmann: I dunno you asked what I thought about a job def change.. I think the placement-runs-nova-functional job probably best belongs in placement itself23:34
dansmithwhich it looks like is the case23:35
gmannyeah.23:35
gmannlet's keep it there23:35
dansmithdoesn't the placement one need to depends-on the nova one?23:35
opendevreviewGhanshyam proposed openstack/placement master: Use 'functional-without-sample-db-tests' tox env for placement nova job  https://review.opendev.org/c/openstack/placement/+/81368023:37
gmanndansmith: yeah, it is depends-on23:37
gmannmelwitt: ^^ updated commit msg23:37
dansmithsorry, I dunno what I was missing23:38
dansmithgmann: sorry had the first patch open in two tabs I think23:38

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!